Detecting when you need to system rescue
System Rescue

Kurt provides some tips and recommends some tools to help you detect signs of network intrusion and data corruption.
System rescue – it's definitely an important topic with lots of considerations. Do you go with "bare-metal restore" or just back up the data and all the configs? What about your database? Do you snapshot it, or replicate it and keep a transaction log? What about all the new NoSQL things? More to the point, how do you know when you need to do a system rescue?
Sometimes it's pretty obvious, like when some water spilled onto one of my machines; I stared in horror as the machine made a loud "pop" and the power supply killed the motherboard and then itself. Luckily, I didn't lose any data. Sometimes, however, it's not so clear when you have to do a system rescue. For example, if you find a corrupted file on your system, do you have other corrupted files? Short of opening them all and checking them, you don't know whether you have just one bad file or a completely corrupted filesystem.
File Integrity to the Rescue
Such problems have plagued administrators, well, since computers have had read/write data storage. The good news is that several mature tools can help you address the problems of managing files and ensuring that they are not modified or corrupted. Certain strategies are also helpful when designing and architecting systems to make things more robust. Ultimately, the goal is to prevent data corruption or improper modification as much as possible – by using file permissions, robust filesystems with journaling, and so on. Then, you need to ensure that you can detect file corruption and improper modification and, finally, restore things to a known good state. The two main tools for these tasks are Open Source Tripwire [1] and AIDE [2]. Neither has undergone major changes for a few years, mostly because they are fairly feature complete.
Tripwire
Tripwire, first written in 1992, is the granddaddy of file integrity tools. It quickly became popular and was eventually taken commercial, with an open source version remaining available. Open Source Tripwire hasn't undergone an update since late 2011. As I mentioned, it's pretty feature complete – except for hashing algorithms: Open Source Tripwire supports CRC-32 (trivial for an attacker to bypass), HAVAL (weaknesses were found as far back as 2004, so it's probably not a good choice), MD5, and SHA (both of which are showing their age).
Basically Open Source Tripwire doesn't support any modern hashing algorithms (e.g., SHA256 or SHA512). Although MD5 and SHA are hard to break, the skills of attackers keep improving, and it's unlikely that Open Source Tripwire will ever get support for modern hashing algorithms. It also seems to lack support for checking extended file attributes (xattr). Although it can check the basic file permissions (user, group, other), it can't check xattrs, meaning attackers can potentially add themselves to a file or directory and remain undetected. As such, if you have strong security requirements, you should probably consider moving away from Open Source Tripwire. Commercial versions of Tripwire are available, but I've never tried them because I'm not a big fan of closed source security.
AIDE
Luckily, you have a second option, AIDE. AIDE was created as a replacement for Tripwire and has had somewhat more active development. AIDE does support modern hashing algorithms such as SHA256 and SHA512, so the chances of an attacker modifying a file and managing to keep the hash the same on it are pretty nonexistent at this time (and probably for the next 10-20 years). AIDE also supports extended attributes, which is pretty important, because most Linux distributions now default to filesystems like ext4, XFS, and Btrfs, all of which support xattr by default.
Open Source Tripwire and AIDE operate in largely the same manner. You configure them to check certain files and directories, and they create a database of the file and directory permissions, ownership, size, access and modification times, a hash value of the data (if it's a file), and so on. You then run these tools periodically, and they recheck all the files to see whether anything has changed. If it has, the changes are logged, and you can configure the tools to email you a report.
I won't go into installation, because the tools are available as packages for virtually every distribution. Also, I won't cover configuration, because they have pretty solid default policies. I will, however, discuss where things can go horribly wrong and how to prevent that.
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Direct Download
Read full article as PDF:
Price $2.95
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Find SysAdmin Jobs
News
-
OpenMandriva Lx 23.03 Rolling Release is Now Available
OpenMandriva "ROME" is the latest point update for the rolling release Linux distribution and offers the latest updates for a number of important applications and tools.
-
CarbonOS: A New Linux Distro with a Focus on User Experience
CarbonOS is a brand new, built-from-scratch Linux distribution that uses the Gnome desktop and has a special feature that makes it appealing to all types of users.
-
Kubuntu Focus Announces XE Gen 2 Linux Laptop
Another Kubuntu-based laptop has arrived to be your next ultra-portable powerhouse with a Linux heart.
-
MNT Seeks Financial Backing for New Seven-Inch Linux Laptop
MNT Pocket Reform is a tiny laptop that is modular, upgradable, recyclable, reusable, and ships with Debian Linux.
-
Ubuntu Flatpak Remix Adds Flatpak Support Preinstalled
If you're looking for a version of Ubuntu that includes Flatpak support out of the box, there's one clear option.
-
Gnome 44 Release Candidate Now Available
The Gnome 44 release candidate has officially arrived and adds a few changes into the mix.
-
Flathub Vying to Become the Standard Linux App Store
If the Flathub team has any say in the matter, their product will become the default tool for installing Linux apps in 2023.
-
Debian 12 to Ship with KDE Plasma 5.27
The Debian development team has shifted to the latest version of KDE for their testing branch.
-
Planet Computers Launches ARM-based Linux Desktop PCs
The firm that originally released a line of mobile keyboards has taken a different direction and has developed a new line of out-of-the-box mini Linux desktop computers.
-
Ubuntu No Longer Shipping with Flatpak
In a move that probably won’t come as a shock to many, Ubuntu and all of its official spins will no longer ship with Flatpak installed.