Troubleshooting HTTPS connections with mitmproxy
Home Grown Extensions
However, more than this, mitmproxy
also accepts DIY Python scripts as extensions. As an example, Listing 2 shows a script in Python 3, as required by mitmproxy
, which picks up the URLs of all incoming responses and stores them together with the length of the corresponding web content in bytes in a newly created dump.log
file.
Listing 2
URLDumper.py
01 #!/usr/bin/env python3 02 from mitmproxy import ctx 03 import re 04 05 class URLDumper: 06 def __init__(self): 07 ctx.log.warn( "URLDumper ready" ) 08 09 def request(self, flow): 10 ctx.log.warn( "URLDumper request" ) 11 flow.request.anticache() 12 13 def append_to_dump(self, url, content): 14 with open("dump.log", "a") as f: 15 f.write("%s (%d bytes)\n" % 16 (url, len(content))) 17 18 def response(self, flow): 19 url = flow.request.url 20 self.append_to_dump(url, 21 flow.response.content) 22 23 addons = [ 24 URLDumper() 25 ]
To do this, it imports the ctx
context object from the mitmproxy
package in line 2, to be used later for calls to the console logging function. The package doesn't have to be installed anywhere; it's just that mitmproxy
magically finds it when it imports the script, because it manipulates the Python interpreter's package search paths to its own installation.
By design, the proxy jumps to the request()
method in line 9 before each request. This method uses the flow
variable to field an object for the current web request and, for consistency, calls the request object's anticache()
method. This simply discards headers like If-modified-since
on every request to make sure the browser fetches items anew every time. Nevertheless, browsers sometimes still use a cache regardless and don't even bother to ask the server if an image has changed. If you run into that situation, hit Shift+Reload to force it to reload everything on the current page.
In addition, the proxy jumps to the response()
method in Line 18 for each incoming response. There Listing 2 first extracts the URL of the request, then retrieves the content of the retrieved web resource as a string with a content
attribute, and passes both to the append_to_dump()
method from line 13 for logging. The function opens the dump.log
file in append mode and appends the URL as well as the length of the content string in readable format. To see the script in action, run
$ mitmproxy --mode regular -s URLDumper.py\ --set console_eventlog_verbosity="warn"
which passes the name of the new Python script to the proxy and shows a message reading URLDumper ready in the proxy window footer shortly after launching. This is just a brief confirmation that the add-on's initialization function completed successfully.
For convenience, the proxy will be monitoring the Python script. If it changes at run time, the proxy notices this and reinitializes the script immediately. Figure 9 shows the results of the script in action: After a browser session that had localhost:8080
set as proxy and which visited the Linux Magazine website, the entries shown in Figure 9 were found in dump.log
.
Looking for Trouble …
If an error occurs in the Python script, either during compilation or at run time, the console window displays an ominous message, namely that details are "in the event log." Some searching on the Internet solved the mystery: To view the log, you need to type
:console.view.eventlog
in the console window. The window then switches to the error messages log for which you were looking. Usually, it's easy to discover what was causing Python to complain.
Another killer mitmproxy
application is the ability to log batteries of request sequences, in order to replay them later and expose the server to a preconfigured test case. The man pages – for operating both the console and the API – can be found on mitmproxy.org. The content looks a bit unkempt, but patient searching will usually reveal the information you desire.
Oh, and before I forget: If you do not want to leave a gaping vulnerability on your system after testing with mitmproxy
, you need to remove the CA certificate you added for test purposes after completing the tests. Better safe than sorry!
Infos
- Linux binaries for mitmproxy: https://mitmproxy.org/downloads/#4.0.4/
- mitmproxy how-to: https://docs.mitmproxy.org/stable/concepts-howmitmproxyworks/
- Listings for this article: ftp://ftp.linux-magazine.com/pub/listings/linux-magazine.com/224/
« Previous 1 2
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
Fedora 40 Beta Released Soon
With the official release of Fedora 40 coming in April, it's almost time to download the beta and see what's new.
-
New Pentesting Distribution to Compete with Kali Linux
SnoopGod is now available for your testing needs
-
Juno Computers Launches Another Linux Laptop
If you're looking for a powerhouse laptop that runs Ubuntu, the Juno Computers Neptune 17 v6 should be on your radar.
-
ZorinOS 17.1 Released, Includes Improved Windows App Support
If you need or desire to run Windows applications on Linux, there's one distribution intent on making that easier for you and its new release further improves that feature.
-
Linux Market Share Surpasses 4% for the First Time
Look out Windows and macOS, Linux is on the rise and has even topped ChromeOS to become the fourth most widely used OS around the globe.
-
KDE’s Plasma 6 Officially Available
KDE’s Plasma 6.0 "Megarelease" has happened, and it's brimming with new features, polish, and performance.
-
Latest Version of Tails Unleashed
Tails 6.0 is based on Debian 12 and includes GNOME 43.
-
KDE Announces New Slimbook V with Plenty of Power and KDE’s Plasma 6
If you're a fan of KDE Plasma, you'll be thrilled to hear they've announced a new Slimbook with an AMD CPU and the latest version of KDE Plasma desktop.
-
Monthly Sponsorship Includes Early Access to elementary OS 8
If you want to get a glimpse of what's in the pipeline for elementary OS 8, just set up a monthly sponsorship to help fund its continued existence.
-
DebConf24 to be Held in South Korea
Busan will be the location of the latest DebConf running July 28 through August 4