Detecting when you need to system rescue
System Rescue
Kurt provides some tips and recommends some tools to help you detect signs of network intrusion and data corruption.
System rescue – it's definitely an important topic with lots of considerations. Do you go with "bare-metal restore" or just back up the data and all the configs? What about your database? Do you snapshot it, or replicate it and keep a transaction log? What about all the new NoSQL things? More to the point, how do you know when you need to do a system rescue?
Sometimes it's pretty obvious, like when some water spilled onto one of my machines; I stared in horror as the machine made a loud "pop" and the power supply killed the motherboard and then itself. Luckily, I didn't lose any data. Sometimes, however, it's not so clear when you have to do a system rescue. For example, if you find a corrupted file on your system, do you have other corrupted files? Short of opening them all and checking them, you don't know whether you have just one bad file or a completely corrupted filesystem.
File Integrity to the Rescue
Such problems have plagued administrators, well, since computers have had read/write data storage. The good news is that several mature tools can help you address the problems of managing files and ensuring that they are not modified or corrupted. Certain strategies are also helpful when designing and architecting systems to make things more robust. Ultimately, the goal is to prevent data corruption or improper modification as much as possible – by using file permissions, robust filesystems with journaling, and so on. Then, you need to ensure that you can detect file corruption and improper modification and, finally, restore things to a known good state. The two main tools for these tasks are Open Source Tripwire [1] and AIDE [2]. Neither has undergone major changes for a few years, mostly because they are fairly feature complete.
Tripwire
Tripwire, first written in 1992, is the granddaddy of file integrity tools. It quickly became popular and was eventually taken commercial, with an open source version remaining available. Open Source Tripwire hasn't undergone an update since late 2011. As I mentioned, it's pretty feature complete – except for hashing algorithms: Open Source Tripwire supports CRC-32 (trivial for an attacker to bypass), HAVAL (weaknesses were found as far back as 2004, so it's probably not a good choice), MD5, and SHA (both of which are showing their age).
Basically Open Source Tripwire doesn't support any modern hashing algorithms (e.g., SHA256 or SHA512). Although MD5 and SHA are hard to break, the skills of attackers keep improving, and it's unlikely that Open Source Tripwire will ever get support for modern hashing algorithms. It also seems to lack support for checking extended file attributes (xattr). Although it can check the basic file permissions (user, group, other), it can't check xattrs, meaning attackers can potentially add themselves to a file or directory and remain undetected. As such, if you have strong security requirements, you should probably consider moving away from Open Source Tripwire. Commercial versions of Tripwire are available, but I've never tried them because I'm not a big fan of closed source security.
AIDE
Luckily, you have a second option, AIDE. AIDE was created as a replacement for Tripwire and has had somewhat more active development. AIDE does support modern hashing algorithms such as SHA256 and SHA512, so the chances of an attacker modifying a file and managing to keep the hash the same on it are pretty nonexistent at this time (and probably for the next 10-20 years). AIDE also supports extended attributes, which is pretty important, because most Linux distributions now default to filesystems like ext4, XFS, and Btrfs, all of which support xattr by default.
Open Source Tripwire and AIDE operate in largely the same manner. You configure them to check certain files and directories, and they create a database of the file and directory permissions, ownership, size, access and modification times, a hash value of the data (if it's a file), and so on. You then run these tools periodically, and they recheck all the files to see whether anything has changed. If it has, the changes are logged, and you can configure the tools to email you a report.
I won't go into installation, because the tools are available as packages for virtually every distribution. Also, I won't cover configuration, because they have pretty solid default policies. I will, however, discuss where things can go horribly wrong and how to prevent that.
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
Gnome 47.1 Released with a Few Fixes
The latest release of the Gnome desktop is all about fixing a few nagging issues and not about bringing new features into the mix.
-
System76 Unveils an Ampere-Powered Thelio Desktop
If you're looking for a new desktop system for developing autonomous driving and software-defined vehicle solutions. System76 has you covered.
-
VirtualBox 7.1.4 Includes Initial Support for Linux kernel 6.12
The latest version of VirtualBox has arrived and it not only adds initial support for kernel 6.12 but another feature that will make using the virtual machine tool much easier.
-
New Slimbook EVO with Raw AMD Ryzen Power
If you're looking for serious power in a 14" ultrabook that is powered by Linux, Slimbook has just the thing for you.
-
The Gnome Foundation Struggling to Stay Afloat
The foundation behind the Gnome desktop environment is having to go through some serious belt-tightening due to continued financial problems.
-
Thousands of Linux Servers Infected with Stealth Malware Since 2021
Perfctl is capable of remaining undetected, which makes it dangerous and hard to mitigate.
-
Halcyon Creates Anti-Ransomware Protection for Linux
As more Linux systems are targeted by ransomware, Halcyon is stepping up its protection.
-
Valve and Arch Linux Announce Collaboration
Valve and Arch have come together for two projects that will have a serious impact on the Linux distribution.
-
Hacker Successfully Runs Linux on a CPU from the Early ‘70s
From the office of "Look what I can do," Dmitry Grinberg was able to get Linux running on a processor that was created in 1971.
-
OSI and LPI Form Strategic Alliance
With a goal of strengthening Linux and open source communities, this new alliance aims to nurture the growth of more highly skilled professionals.