Home network monitoring with pfSense, Protectli, and a screen scraper
Programming Snapshot – Protectli

What is making the lights on the router flicker so excitedly? An intruder? We investigate with pfSense on a Protectli micro appliance and a screen scraper to email the information.
It's a shame that no routers simply display the network packet addresses that pass through them on an LED display. Because I'm curious about what's going on in my home network, on the advice of a work colleague, I bought a micro appliance from the Chinese company Protectli (Figure 1) [1], which runs the FreeBSD-based open source firewall pfSense. The box is about four by four inches in size and passively cooled, so there is absolutely no fan or other noise.
The installation is a piece of cake – simply load the distribution from the pfSense Community Edition website [2] onto a bootable USB stick, insert an mSATA disk and RAM into the Protectli's small case, and after booting, say yes to the installation prompts. Badda-bing badda-boom, pfSense's web GUI is up and running (Figure 2).
Guardian at the Gate
The Protectli appliance is directly connected to the Internet-facing interface (in my case, a DSL modem to the ISP). On the LAN side, it provides access to the Internet for all devices connected to my home network (in my case, a series of routed subnets).
Equipped with a four-core Celeron, it's powerful enough to look at every single packet, create statistics or even intervene when needed, and block certain communication attempts according to predefined firewall rules. If I want to know why router lights are flickering, I only need to call the pfSense GUI to see who is streaming Spotify, watching Netflix, or ordering on Amazon (Figure 2).
In addition to traditional terminal-based tools such as pftop
, the firewall GUI also offers very elegant add-on packages such as ntopng
, so you can browse through pie charts and HTML tables to find out who uses the most bandwidth or contacts computers in dubious countries (Figure 3).
Unfortunately, there is no official API for the GUI, only a FauxAPI [3], which runs as an add-on package on the pfSense distribution and provides limited access to the firewall's internals.
Keys for Protectli
To check at regular intervals what is happening on the Protectli box, I thought it would be easy to write a screen scraper [4] that logs in periodically and automatically to the box's login page (Figure 4) and scans and mails the data displayed on the dashboard.
However, the first hurdle between a command-line client and the juicy network data is the login page so defiantly presented by pfSense. A look at the HTML code (Figure 5) reveals that the two fields for accepting the username and password are dubbed usernamefld
and passwordfld
, and the submit button goes by the name login
. The Python scraper quickly thrown together in Listing 1 [5], which uses the selenium
module installed using pip3
to simulate a browser, searches for and finds these elements using the find_element_by_name()
function.
Listing 1
dash-scraper.py
For the webdriver.Firefox()
call to work with the system's Firefox browser, the Linux distribution needs the geckodriver
program, which is available as a TAR file [6]. You need to unpack this and dump the binary that falls out of the archive into a path that can be found somewhere in $PATH
. The script opens the browser, takes you to the login page, autofills the form fields, and then clicks on the Login button. The selenium
module is often used for testing WebGUIs and makes it really easy to simulate a user sitting in front of a web user interface.
After the pfSense login screen, the pfSense dashboard page (Figure 6) with the firewall overview data is saved in the saved.png
file by the new screen scraper by calling save_screenshot()
as a trigger. The values to be filled in are read by the script from the creds.yaml
file, which is read from disk; the data is then stored as a username and password accessible in the creds
dictionary (Figure 7).
One Man Went to Mow
Listing 2 is used to periodically send the collected data to an email address. It bundles the PNG file created by Listing 1 into an email as an attachment and sends it via an SMTP server. To read the security-relevant username and password variables for the SMTP server, line 10 retrieves the same YAML file as before and stores its contents in the creds
dictionary.
Listing 2
mail.py
The script then builds an HTML body – with introductory text and an IMG link to the attached image, so that a webmail client can display it graphically. As of line 31, Listing 2 establishes a connection to the SMTP server, whose address is also retrieved from the creds.yaml
file as the smtp_server: mail.provider.net
entry. The script uses port 587 and transmits the data in TLS-encrypted form. It appends the screenshot in MIME format and adds an email content ID header with the name of an imaginary file in square brackets.
Figure 8 shows how the mail arrives in Gmail. Called as a cron job once a day, this keeps the home owner up to date with what's happening on the local network.
Infos
- Protectli micro appliance: https://www.amazon.com/gp/product/B01GIVQI3M
- pfSense Community Edition download: https://www.pfsense.org/download/
- pfSense FauxAPI: https://github.com/ndejong/pfsense_fauxapi
- Jarmul, Katharine, and Richard Lawson. Python Web Scraping, 2nd ed.Packt Publishing, 2017
- Listings for this article: ftp://ftp.linux-magazine.com/pub/listings/linux-magazine.com/208/
- geckodriver: https://github.com/mozilla/geckodriver/releases/tag/v0.19.0
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you've found an article to be beneficial.
News
-
The GNU Project Celebrates Its 40th Birthday
September 27 marks the 40th anniversary of the GNU Project, and it was celebrated with a hacker meeting in Biel/Bienne, Switzerland.
-
Linux Kernel Reducing Long-Term Support
LTS support for the Linux kernel is about to undergo some serious changes that will have a considerable impact on the future.
-
Fedora 39 Beta Now Available for Testing
For fans and users of Fedora Linux, the first beta of release 39 is now available, which is a minor upgrade but does include GNOME 45.
-
Fedora Linux 40 to Drop X11 for KDE Plasma
When Fedora 40 arrives in 2024, there will be a few big changes coming, especially for the KDE Plasma option.
-
Real-Time Ubuntu Available in AWS Marketplace
Anyone looking for a Linux distribution for real-time processing could do a whole lot worse than Real-Time Ubuntu.
-
KSMBD Finally Reaches a Stable State
For those who've been looking forward to the first release of KSMBD, after two years it's no longer considered experimental.
-
Nitrux 3.0.0 Has Been Released
The latest version of Nitrux brings plenty of innovation and fresh apps to the table.
-
Linux From Scratch 12.0 Now Available
If you're looking to roll your own Linux distribution, the latest version of Linux From Scratch is now available with plenty of updates.
-
Linux Kernel 6.5 Has Been Released
The newest Linux kernel, version 6.5, now includes initial support for two very exciting features.
-
UbuntuDDE 23.04 Now Available
A new version of the UbuntuDDE remix has finally arrived with all the updates from the Deepin desktop and everything that comes with the Ubuntu 23.04 base.