Detecting when you need to system rescue
System Rescue
Kurt provides some tips and recommends some tools to help you detect signs of network intrusion and data corruption.
System rescue – it's definitely an important topic with lots of considerations. Do you go with "bare-metal restore" or just back up the data and all the configs? What about your database? Do you snapshot it, or replicate it and keep a transaction log? What about all the new NoSQL things? More to the point, how do you know when you need to do a system rescue?
Sometimes it's pretty obvious, like when some water spilled onto one of my machines; I stared in horror as the machine made a loud "pop" and the power supply killed the motherboard and then itself. Luckily, I didn't lose any data. Sometimes, however, it's not so clear when you have to do a system rescue. For example, if you find a corrupted file on your system, do you have other corrupted files? Short of opening them all and checking them, you don't know whether you have just one bad file or a completely corrupted filesystem.
File Integrity to the Rescue
Such problems have plagued administrators, well, since computers have had read/write data storage. The good news is that several mature tools can help you address the problems of managing files and ensuring that they are not modified or corrupted. Certain strategies are also helpful when designing and architecting systems to make things more robust. Ultimately, the goal is to prevent data corruption or improper modification as much as possible – by using file permissions, robust filesystems with journaling, and so on. Then, you need to ensure that you can detect file corruption and improper modification and, finally, restore things to a known good state. The two main tools for these tasks are Open Source Tripwire [1] and AIDE [2]. Neither has undergone major changes for a few years, mostly because they are fairly feature complete.
Tripwire
Tripwire, first written in 1992, is the granddaddy of file integrity tools. It quickly became popular and was eventually taken commercial, with an open source version remaining available. Open Source Tripwire hasn't undergone an update since late 2011. As I mentioned, it's pretty feature complete – except for hashing algorithms: Open Source Tripwire supports CRC-32 (trivial for an attacker to bypass), HAVAL (weaknesses were found as far back as 2004, so it's probably not a good choice), MD5, and SHA (both of which are showing their age).
Basically Open Source Tripwire doesn't support any modern hashing algorithms (e.g., SHA256 or SHA512). Although MD5 and SHA are hard to break, the skills of attackers keep improving, and it's unlikely that Open Source Tripwire will ever get support for modern hashing algorithms. It also seems to lack support for checking extended file attributes (xattr). Although it can check the basic file permissions (user, group, other), it can't check xattrs, meaning attackers can potentially add themselves to a file or directory and remain undetected. As such, if you have strong security requirements, you should probably consider moving away from Open Source Tripwire. Commercial versions of Tripwire are available, but I've never tried them because I'm not a big fan of closed source security.
AIDE
Luckily, you have a second option, AIDE. AIDE was created as a replacement for Tripwire and has had somewhat more active development. AIDE does support modern hashing algorithms such as SHA256 and SHA512, so the chances of an attacker modifying a file and managing to keep the hash the same on it are pretty nonexistent at this time (and probably for the next 10-20 years). AIDE also supports extended attributes, which is pretty important, because most Linux distributions now default to filesystems like ext4, XFS, and Btrfs, all of which support xattr by default.
Open Source Tripwire and AIDE operate in largely the same manner. You configure them to check certain files and directories, and they create a database of the file and directory permissions, ownership, size, access and modification times, a hash value of the data (if it's a file), and so on. You then run these tools periodically, and they recheck all the files to see whether anything has changed. If it has, the changes are logged, and you can configure the tools to email you a report.
I won't go into installation, because the tools are available as packages for virtually every distribution. Also, I won't cover configuration, because they have pretty solid default policies. I will, however, discuss where things can go horribly wrong and how to prevent that.
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
TUXEDO Computers Unveils Linux Laptop Featuring AMD Ryzen CPU
This latest release is the first laptop to include the new CPU from Ryzen and Linux preinstalled.
-
XZ Gets the All-Clear
The back door xz vulnerability has been officially reverted for Fedora 40 and versions 38 and 39 were never affected.
-
Canonical Collaborates with Qualcomm on New Venture
This new joint effort is geared toward bringing Ubuntu and Ubuntu Core to Qualcomm-powered devices.
-
Kodi 21.0 Open-Source Entertainment Hub Released
After a year of development, the award-winning Kodi cross-platform, media center software is now available with many new additions and improvements.
-
Linux Usage Increases in Two Key Areas
If market share is your thing, you'll be happy to know that Linux is on the rise in two areas that, if they keep climbing, could have serious meaning for Linux's future.
-
Vulnerability Discovered in xz Libraries
An urgent alert for Fedora 40 has been posted and users should pay attention.
-
Canonical Bumps LTS Support to 12 years
If you're worried that your Ubuntu LTS release won't be supported long enough to last, Canonical has a surprise for you in the form of 12 years of security coverage.
-
Fedora 40 Beta Released Soon
With the official release of Fedora 40 coming in April, it's almost time to download the beta and see what's new.
-
New Pentesting Distribution to Compete with Kali Linux
SnoopGod is now available for your testing needs
-
Juno Computers Launches Another Linux Laptop
If you're looking for a powerhouse laptop that runs Ubuntu, the Juno Computers Neptune 17 v6 should be on your radar.