Troubleshooting HTTPS connections with mitmproxy
Programming Snapshot – mitmproxy
Finding the data zipping back and forth between the browser and server is not only interesting for snooping spies, but also for debugging developers. Mike Schilli gets you started with mitmproxy and shows how to customize it using Python scripts.
If things just don't work when you are developing a web application, the question immediately arises as to what data the browser and web server are actually exchanging. Tools for snooping on the network such as Wireshark (as well as proxies that sit between the client and the server) leave both requests and responses untouched, while logging them for inspection.
mitmproxy
, which stands for man in the middle (MITM) proxy, is the king of the hill in this category; it makes the impossible possible by logging encrypted HTTPS requests. But first, let's look at the simplest, unencrypted case, for which my aging website perlmeister.com that still uses plain old unencrypted HTTP is a great choice.
Figure 1 shows how the browser retrieves the requested page's HTML text, along with some images and JavaScript snippets from the server. The mitmproxy
tool, which is available for download as binary from [1], sports a terminal user interface (UI), which displays a double arrow to the left of the current request, called Flow
in mitmproxy
parlance. When you press the Enter key, the detailed request data come up, as shown in Figure 2.
Using the cursor keys (or, in typical Vi-style: h, j, k, l), you can move to the left/down/up/right and jump between Request
, Response
, and Details
on the request detail page. You can always go back to the next level up with Q. On the main screen listing the flows, use the cursor keys (or J for down and K for up) to select a specific one. Press D to clear the current flow (marked >>
), and hammer D repeatedly to clear the whole screen.
Command-Line Option
There are several options to place mitmproxy
between the client and the server in order to start snooping on the data exchanged. The simplest approach is configuring it on the client side, where applications typically provide options to configure HTTP or SOCKS proxies. To start the proxy, call mitmproxy
from the command line, which will render the currently running terminal black, and log intercepted data, while waiting for user commands from the keyboard.
If you try to persuade Chromium on Ubuntu to use a proxy, officially located under Settings | Advanced/Security | Open Proxy Configuration, you are in for a surprise, because the desktop application won't allow it and outputs an error message (Figure 3). Fortunately, the program accepts the command-line option
$ chromium-browser --proxy-server="localhost:8080"
telling it to use a proxy server on the host (localhost
) and port (8080), which is where mitmproxy
listens by default. The important thing is to run Chromium as a single instance on the Ubuntu desktop. If you already have another instance running, you might find yourself tearing your hair out in frustration because no requests show up in the logs. If it's configured correctly, however, when you access URLs in the browser, the data will appear in the proxy UI for further analysis.
Breaking Encryption
So far, so good. But what happens if the browser sends an HTTPS request? By design, the protocol will prevent sniffing attacks from men in the middle. And, lo and behold, a Chromium browser configured to use mitmproxy
reports that an attack is in progress (Figure 4).
But if you then go to Advanced and click on the Proceed (unsafe) link, Chromium lifts the barrier and lets you continue at your own risk, while mitmproxy
in turn will eagerly log the connection data. This may seem like a silly question, but how does it do that if the client and server are using an encrypted connection?
By using a revolutionary trick, mitmproxy
identifies itself to the client as the server, and to the server as the client (Figure 5). Instead of sending back the actual server response, it sends the client a fake certificate. This leads to the client – assuming it doesn't check or validate the certificate – opening an encrypted connection to the proxy instead of the server, if the user continues despite the warning.
By being in possession of the encryption keys, the proxy can now easily decrypt and log the data going over the wire (Figure 6) that it then encrypts again for forwarding to the server, which thinks it is talking to the client. In other words, a classic MITM attack.
Hard Cases
If the client does not allow communication over obviously hacked channels even if the user tries to force it, the sys admin can get in on the data heist and install a spoofed Certificate Authority (CA) certificate on the client, which mitmproxy
conveniently provides. However, many clients (e.g., mobile phones) do not allow proxy configurations at all or simply ignore them. For these cases, mitmproxy
offers the option of a transparent proxy mode, in which the data are routed to the proxy instead of to the server using lower-level network tools such as iptables. The documentation on [2] illustrates solutions for these trickier cases.
Home-grown programs written in Go rely on the system's CA store for encrypted communications. On Ubuntu, it comes with the ca-certificates package. Out of the box, it doesn't contain the counterfeit mitmproxy
CA certificate, of course, but the sys admin can add it. As an example, Listing 1 [3] submits an HTTPS request to get the TLS-protected website of a well-known online giant. If mitmproxy
is running, the program aborts the request with the following error message if the client is rerouted via the proxy server by setting the HTTP_PROXY
environment variable:
Listing 1
client.go
01 package main 02 03 import ( 04 "fmt" 05 "io/ioutil" 06 "net/http" 07 ) 08 09 func main() { 10 resp, err := http.Get("https://amazon.com") 11 if err != nil { 12 panic(err) 13 } 14 defer resp.Body.Close() 15 body, err := ioutil.ReadAll(resp.Body) 16 fmt.Println(string(body)) 17 }
$ go build client.go $ HTTP_PROXY=localhost:8080 ./client panic: Get https://usarundbrief.com: x509: certificate signed by unknown authority
What happened? Instead of Amazon's certificate, the client received a self-signed certificate, injected by mitmproxy
, and the client rightfully complains about it. Now, to convince this pesky client to accept the certificate anyway (for testing purposes only, of course), the sys admin copies it into the Ubuntu system collection using the command sequence in Figure 7. And, sure enough, the following call to
$ HTTP_PROXY=localhost:8080 ./client
displays the TLS-protected Amazon website without any complaints, while mitmproxy
logs the transaction as an MITM.
By the way, Chromium on Ubuntu does not use the system store for trusted CAs; instead, it comes with its own collection. In order to add the counterfeit certificate to this collection, a few tricks are required, which mitmproxy
explains when the user accesses the magic mitm.it
domain in the browser with the proxy configured, after clicking the button with the desired operating system (Figure 8). It works similarly on Firefox, which offers a configuration menu for adding a new certificate below Options | Privacy & Security | View Certificates | Authorities | Import.
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
Canonical Releases Ubuntu 24.04
After a brief pause because of the XZ vulnerability, Ubuntu 24.04 is now available for install.
-
Linux Servers Targeted by Akira Ransomware
A group of bad actors who have already extorted $42 million have their sights set on the Linux platform.
-
TUXEDO Computers Unveils Linux Laptop Featuring AMD Ryzen CPU
This latest release is the first laptop to include the new CPU from Ryzen and Linux preinstalled.
-
XZ Gets the All-Clear
The back door xz vulnerability has been officially reverted for Fedora 40 and versions 38 and 39 were never affected.
-
Canonical Collaborates with Qualcomm on New Venture
This new joint effort is geared toward bringing Ubuntu and Ubuntu Core to Qualcomm-powered devices.
-
Kodi 21.0 Open-Source Entertainment Hub Released
After a year of development, the award-winning Kodi cross-platform, media center software is now available with many new additions and improvements.
-
Linux Usage Increases in Two Key Areas
If market share is your thing, you'll be happy to know that Linux is on the rise in two areas that, if they keep climbing, could have serious meaning for Linux's future.
-
Vulnerability Discovered in xz Libraries
An urgent alert for Fedora 40 has been posted and users should pay attention.
-
Canonical Bumps LTS Support to 12 years
If you're worried that your Ubuntu LTS release won't be supported long enough to last, Canonical has a surprise for you in the form of 12 years of security coverage.
-
Fedora 40 Beta Released Soon
With the official release of Fedora 40 coming in April, it's almost time to download the beta and see what's new.