Troubleshooting HTTPS connections with mitmproxy
Programming Snapshot – mitmproxy
Finding the data zipping back and forth between the browser and server is not only interesting for snooping spies, but also for debugging developers. Mike Schilli gets you started with mitmproxy and shows how to customize it using Python scripts.
If things just don't work when you are developing a web application, the question immediately arises as to what data the browser and web server are actually exchanging. Tools for snooping on the network such as Wireshark (as well as proxies that sit between the client and the server) leave both requests and responses untouched, while logging them for inspection.
mitmproxy
, which stands for man in the middle (MITM) proxy, is the king of the hill in this category; it makes the impossible possible by logging encrypted HTTPS requests. But first, let's look at the simplest, unencrypted case, for which my aging website perlmeister.com that still uses plain old unencrypted HTTP is a great choice.
Figure 1 shows how the browser retrieves the requested page's HTML text, along with some images and JavaScript snippets from the server. The mitmproxy
tool, which is available for download as binary from [1], sports a terminal user interface (UI), which displays a double arrow to the left of the current request, called Flow
in mitmproxy
parlance. When you press the Enter key, the detailed request data come up, as shown in Figure 2.
Using the cursor keys (or, in typical Vi-style: h, j, k, l), you can move to the left/down/up/right and jump between Request
, Response
, and Details
on the request detail page. You can always go back to the next level up with Q. On the main screen listing the flows, use the cursor keys (or J for down and K for up) to select a specific one. Press D to clear the current flow (marked >>
), and hammer D repeatedly to clear the whole screen.
Command-Line Option
There are several options to place mitmproxy
between the client and the server in order to start snooping on the data exchanged. The simplest approach is configuring it on the client side, where applications typically provide options to configure HTTP or SOCKS proxies. To start the proxy, call mitmproxy
from the command line, which will render the currently running terminal black, and log intercepted data, while waiting for user commands from the keyboard.
If you try to persuade Chromium on Ubuntu to use a proxy, officially located under Settings | Advanced/Security | Open Proxy Configuration, you are in for a surprise, because the desktop application won't allow it and outputs an error message (Figure 3). Fortunately, the program accepts the command-line option
$ chromium-browser --proxy-server="localhost:8080"
telling it to use a proxy server on the host (localhost
) and port (8080), which is where mitmproxy
listens by default. The important thing is to run Chromium as a single instance on the Ubuntu desktop. If you already have another instance running, you might find yourself tearing your hair out in frustration because no requests show up in the logs. If it's configured correctly, however, when you access URLs in the browser, the data will appear in the proxy UI for further analysis.
Breaking Encryption
So far, so good. But what happens if the browser sends an HTTPS request? By design, the protocol will prevent sniffing attacks from men in the middle. And, lo and behold, a Chromium browser configured to use mitmproxy
reports that an attack is in progress (Figure 4).
But if you then go to Advanced and click on the Proceed (unsafe) link, Chromium lifts the barrier and lets you continue at your own risk, while mitmproxy
in turn will eagerly log the connection data. This may seem like a silly question, but how does it do that if the client and server are using an encrypted connection?
By using a revolutionary trick, mitmproxy
identifies itself to the client as the server, and to the server as the client (Figure 5). Instead of sending back the actual server response, it sends the client a fake certificate. This leads to the client – assuming it doesn't check or validate the certificate – opening an encrypted connection to the proxy instead of the server, if the user continues despite the warning.
By being in possession of the encryption keys, the proxy can now easily decrypt and log the data going over the wire (Figure 6) that it then encrypts again for forwarding to the server, which thinks it is talking to the client. In other words, a classic MITM attack.
Hard Cases
If the client does not allow communication over obviously hacked channels even if the user tries to force it, the sys admin can get in on the data heist and install a spoofed Certificate Authority (CA) certificate on the client, which mitmproxy
conveniently provides. However, many clients (e.g., mobile phones) do not allow proxy configurations at all or simply ignore them. For these cases, mitmproxy
offers the option of a transparent proxy mode, in which the data are routed to the proxy instead of to the server using lower-level network tools such as iptables. The documentation on [2] illustrates solutions for these trickier cases.
Home-grown programs written in Go rely on the system's CA store for encrypted communications. On Ubuntu, it comes with the ca-certificates package. Out of the box, it doesn't contain the counterfeit mitmproxy
CA certificate, of course, but the sys admin can add it. As an example, Listing 1 [3] submits an HTTPS request to get the TLS-protected website of a well-known online giant. If mitmproxy
is running, the program aborts the request with the following error message if the client is rerouted via the proxy server by setting the HTTP_PROXY
environment variable:
Listing 1
client.go
01 package main 02 03 import ( 04 "fmt" 05 "io/ioutil" 06 "net/http" 07 ) 08 09 func main() { 10 resp, err := http.Get("https://amazon.com") 11 if err != nil { 12 panic(err) 13 } 14 defer resp.Body.Close() 15 body, err := ioutil.ReadAll(resp.Body) 16 fmt.Println(string(body)) 17 }
$ go build client.go $ HTTP_PROXY=localhost:8080 ./client panic: Get https://usarundbrief.com: x509: certificate signed by unknown authority
What happened? Instead of Amazon's certificate, the client received a self-signed certificate, injected by mitmproxy
, and the client rightfully complains about it. Now, to convince this pesky client to accept the certificate anyway (for testing purposes only, of course), the sys admin copies it into the Ubuntu system collection using the command sequence in Figure 7. And, sure enough, the following call to
$ HTTP_PROXY=localhost:8080 ./client
displays the TLS-protected Amazon website without any complaints, while mitmproxy
logs the transaction as an MITM.
By the way, Chromium on Ubuntu does not use the system store for trusted CAs; instead, it comes with its own collection. In order to add the counterfeit certificate to this collection, a few tricks are required, which mitmproxy
explains when the user accesses the magic mitm.it
domain in the browser with the proxy configured, after clicking the button with the desired operating system (Figure 8). It works similarly on Firefox, which offers a configuration menu for adding a new certificate below Options | Privacy & Security | View Certificates | Authorities | Import.
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
Canonical Bumps LTS Support to 12 years
If you're worried that your Ubuntu LTS release won't be supported long enough to last, Canonical has a surprise for you in the form of 12 years of security coverage.
-
Fedora 40 Beta Released Soon
With the official release of Fedora 40 coming in April, it's almost time to download the beta and see what's new.
-
New Pentesting Distribution to Compete with Kali Linux
SnoopGod is now available for your testing needs
-
Juno Computers Launches Another Linux Laptop
If you're looking for a powerhouse laptop that runs Ubuntu, the Juno Computers Neptune 17 v6 should be on your radar.
-
ZorinOS 17.1 Released, Includes Improved Windows App Support
If you need or desire to run Windows applications on Linux, there's one distribution intent on making that easier for you and its new release further improves that feature.
-
Linux Market Share Surpasses 4% for the First Time
Look out Windows and macOS, Linux is on the rise and has even topped ChromeOS to become the fourth most widely used OS around the globe.
-
KDE’s Plasma 6 Officially Available
KDE’s Plasma 6.0 "Megarelease" has happened, and it's brimming with new features, polish, and performance.
-
Latest Version of Tails Unleashed
Tails 6.0 is based on Debian 12 and includes GNOME 43.
-
KDE Announces New Slimbook V with Plenty of Power and KDE’s Plasma 6
If you're a fan of KDE Plasma, you'll be thrilled to hear they've announced a new Slimbook with an AMD CPU and the latest version of KDE Plasma desktop.
-
Monthly Sponsorship Includes Early Access to elementary OS 8
If you want to get a glimpse of what's in the pipeline for elementary OS 8, just set up a monthly sponsorship to help fund its continued existence.