Troubleshooting HTTPS connections with mitmproxy
Home Grown Extensions
However, more than this, mitmproxy
also accepts DIY Python scripts as extensions. As an example, Listing 2 shows a script in Python 3, as required by mitmproxy
, which picks up the URLs of all incoming responses and stores them together with the length of the corresponding web content in bytes in a newly created dump.log
file.
Listing 2
URLDumper.py
01 #!/usr/bin/env python3 02 from mitmproxy import ctx 03 import re 04 05 class URLDumper: 06 def __init__(self): 07 ctx.log.warn( "URLDumper ready" ) 08 09 def request(self, flow): 10 ctx.log.warn( "URLDumper request" ) 11 flow.request.anticache() 12 13 def append_to_dump(self, url, content): 14 with open("dump.log", "a") as f: 15 f.write("%s (%d bytes)\n" % 16 (url, len(content))) 17 18 def response(self, flow): 19 url = flow.request.url 20 self.append_to_dump(url, 21 flow.response.content) 22 23 addons = [ 24 URLDumper() 25 ]
To do this, it imports the ctx
context object from the mitmproxy
package in line 2, to be used later for calls to the console logging function. The package doesn't have to be installed anywhere; it's just that mitmproxy
magically finds it when it imports the script, because it manipulates the Python interpreter's package search paths to its own installation.
By design, the proxy jumps to the request()
method in line 9 before each request. This method uses the flow
variable to field an object for the current web request and, for consistency, calls the request object's anticache()
method. This simply discards headers like If-modified-since
on every request to make sure the browser fetches items anew every time. Nevertheless, browsers sometimes still use a cache regardless and don't even bother to ask the server if an image has changed. If you run into that situation, hit Shift+Reload to force it to reload everything on the current page.
In addition, the proxy jumps to the response()
method in Line 18 for each incoming response. There Listing 2 first extracts the URL of the request, then retrieves the content of the retrieved web resource as a string with a content
attribute, and passes both to the append_to_dump()
method from line 13 for logging. The function opens the dump.log
file in append mode and appends the URL as well as the length of the content string in readable format. To see the script in action, run
$ mitmproxy --mode regular -s URLDumper.py\ --set console_eventlog_verbosity="warn"
which passes the name of the new Python script to the proxy and shows a message reading URLDumper ready in the proxy window footer shortly after launching. This is just a brief confirmation that the add-on's initialization function completed successfully.
For convenience, the proxy will be monitoring the Python script. If it changes at run time, the proxy notices this and reinitializes the script immediately. Figure 9 shows the results of the script in action: After a browser session that had localhost:8080
set as proxy and which visited the Linux Magazine website, the entries shown in Figure 9 were found in dump.log
.

Looking for Trouble …
If an error occurs in the Python script, either during compilation or at run time, the console window displays an ominous message, namely that details are "in the event log." Some searching on the Internet solved the mystery: To view the log, you need to type
:console.view.eventlog
in the console window. The window then switches to the error messages log for which you were looking. Usually, it's easy to discover what was causing Python to complain.
Another killer mitmproxy
application is the ability to log batteries of request sequences, in order to replay them later and expose the server to a preconfigured test case. The man pages – for operating both the console and the API – can be found on mitmproxy.org. The content looks a bit unkempt, but patient searching will usually reveal the information you desire.
Oh, and before I forget: If you do not want to leave a gaping vulnerability on your system after testing with mitmproxy
, you need to remove the CA certificate you added for test purposes after completing the tests. Better safe than sorry!
Infos
- Linux binaries for mitmproxy: https://mitmproxy.org/downloads/#4.0.4/
- mitmproxy how-to: https://docs.mitmproxy.org/stable/concepts-howmitmproxyworks/
- Listings for this article: ftp://ftp.linux-magazine.com/pub/listings/linux-magazine.com/224/
« Previous 1 2
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Direct Download
Read full article as PDF:
Price $2.95
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
News
-
An All-Snap Version of Ubuntu is In The Works
Along with the standard deb version of the open-source operating system, Canonical will release an-all snap version.
-
Mageia 9 Beta 2 Ready for Testing
The latest beta of the popular Mageia distribution now includes the latest kernel and plenty of updated applications.
-
KDE Plasma 6 Looks to Bring Basic HDR Support
The KWin piece of KDE Plasma now has HDR support and color management geared for the 6.0 release.
-
Bodhi Linux 7.0 Beta Ready for Testing
The latest iteration of the Bohdi Linux distribution is now available for those who want to experience what's in store and for testing purposes.
-
Changes Coming to Ubuntu PPA Usage
The way you manage Personal Package Archives will be changing with the release of Ubuntu 23.10.
-
AlmaLinux 9.2 Now Available for Download
AlmaLinux has been released and provides a free alternative to upstream Red Hat Enterprise Linux.
-
An Immutable Version of Fedora Is Under Consideration
For anyone who's a fan of using immutable versions of Linux, the Fedora team is currently considering adding a new spin called Fedora Onyx.
-
New Release of Br OS Includes ChatGPT Integration
Br OS 23.04 is now available and is geared specifically toward web content creation.
-
Command-Line Only Peropesis 2.1 Available Now
The latest iteration of Peropesis has been released with plenty of updates and introduces new software development tools.
-
TUXEDO Computers Announces InfinityBook Pro 14
With the new generation of their popular InfinityBook Pro 14, TUXEDO upgrades its ultra-mobile, powerful business laptop with some impressive specs.