On the Trail

Sophisticated software can help you keep an eye out for malware, but by the time it's spotted, it's probably too late, costing you the time to repair and rebuild your system, as well as a loss of revenue if it's a production system. Chkrootkit is a proven, mature, and highly sophisticated tool that helps you detect a variety of malware in an ongoing process.

A True Story

A year or two after the turn of the millennium, I lost a server through the disastrous compromise of a popular web-based application. Thankfully, although the server was in use, there were no services on it that couldn't wait a few days before being restored.

The first clue that something was wrong was some strange filesystem behavior while logged in over SSH. I checked the Apache error logs, and – lo and behold – I found some output detailing a wget transaction. The wget file transfer was showing an illegitimate download from a nasty-looking URL.

Although the file name was unusual, what struck me was that all of the elements of the transaction were running under the apache user (hence, the owner of the error.log in question) and using the /var/log/apache directory. This server was not without security, but after some forensic analysis once the server was taken off the network to avoid further local network damage, I discovered that a web application written in PHP had been compromised. PHP, of course, being run by the apache user.

After gaining access via local user apache, the miscreants had downloaded a rootkit without root permissions and managed to spawn a binary that gave them remote SSH access. From there, all hell broke loose.

First, when I accessed the server I noticed an unusual open port  – not 22 or a port that I used for the SSH server (e.g., port 2222). Moreover, the /tmp directory contained some slightly odd-looking temporary files. I suspect this is where the rootkit dropped its initial installation files after it had "broken free" of Apache's user, because /tmp is usually writeable by all.

Second, when I ran a binary file, such as listing files in a directory, I noticed the server was a little clunky and unresponsive. It turns out the hacker had dropped alternative executables of system commands that fired up in their place.

Analysis later showed that at least 20 common binaries had been overwritten by a rootkit. These ranged from the simple ls to more obscure commands. The nefarious attacker had put these alternatives in place to hide their presence on the system. Logs were also manipulated to show fewer details of the attacker's activities. However the attacker had either forgotten about Apache's error log or thought it was abstract enough not to be checked.

Seemingly simple but clever tricks, such as replacing the w command, were employed. That command would normally offer some information on which users were logged on to the system and who was doing what. With the new version of the command, though, the logged-in hacker was essentially invisible until I dug deeper.

To avoid a full system rebuild, I downloaded a number of stock binaries from a source I trusted and overwrote those present on the filesystem. I then watched and waited patiently. Suddenly, I saw some network traffic to an IP address in Brazil and activity on the server that wasn't legitimate.

I repeated that procedure about three times, overwriting binaries again and again and adding firewall rules, but still something on the server was phoning home and letting the miscreants back in. Eventually I gave up, put a brand new disk in the box, and rebuilt the server from scratch. Incidentally, I would not have even tried dropping replacement binaries onto a production box; it is simply too risky. You've got to rip it up and start again if that ever happens on an important server.

One final, alarming, fact is that forensics proved that at midnight any login passwords (and users) were being covertly emailed back to an IP address in Brazil. That's not a good thing I'm sure that you'll agree.

In this article, I'll look at rootkits and chkrootkit, a stalwart tool that helps identify machines that have been infected.

Roto-Rooter

In simple terms, rootkits essentially permit illegitimate access to a system against the owner's wishes. They come in all shapes and sizes and have most recently been discussed in relation to the UEFI (Unified Extensible Firmware Interface) "Secure Boot" functionality recently introduced. Secure Boot is meant to prevent the installation of rootkits into what used to be called the BIOS (Basic Input/Output System), so that your server or workstation isn't compromised before it has even booted up. To be clear, I'll be looking at operating system rootkits, as opposed to the boot-time equivalent.

One statement that probably needs to be reiterated is that if you lose a machine to a rootkit, you shouldn't risk your data for the sake of one hour's extra work; simply start again with a fresh build. The point I'm trying to make clear is that by the time you've spotted the evidence of a rootkit, it's probably too late. However, by using a clever tool to assist you, you at least can keep an eye on config mistakes that you make and potentially monitor unusual activity on your machines, even if they're not rootkit related. Whichever way you decide to use these sophisticated security tools, however, it should be a great learning experience. Now I'll look the venerable Chkrootkit utility.

Table  1 lists a sample of malware that the chkrootkit [1] tool can detect. This substantial list should be extremely worrying; just look at all the rootkit variants that have been discovered in the past. The Chkrootkit website also notes which tests are run to detect these kits (Table  2).

LKMs (loadable kernel modules) are loaded dynamically into memory and interface directly with the kernel and its subsystems. When this type of access is successful, you have usually reached the point of no return. To see which kernel modules are in use, enter one of the following commands:

lsmod
cat /proc/modules

The modules file in the /proc pseudo-filesystem contains all modules loaded into the kernel (Listing  1).

The commands to query LKMs are useful if the rootkit in question is determined to include the source code of some nefarious kernel modules within its payload. Of course, if a compiler is not installed on your system, then the rootkit's code becomes a serious challenge to compile, and that's just what you want. It means the hacker would need to pull down precompiled binaries to your machine, which might not work correctly on your system for a number of reasons.

Take note that some distributions seem to insist that you leave a compiler like GCC installed. You can check for GCC's presence with:

whereis gcc make

Entering gcc -v reveals the version. If safe to do so (i.e., nothing on your system requires that GCC remain there), you can remove the GCC compiler packages from a Debian distro with the following commands:

apt-get remove gcc
apt-get remove build-essential

The build-essential package drops about 20 tools onto your system that allow you to compile from source. I would never leave these packages installed on a production server for any period other than the five minutes every three months that I needed them.

Using Chkrootkit

To install Chkrootkit, you can use the usual method:

apt-get install chkrootkit

However, even from source [2], Chkrootkit is very simple to compile and execute the following:

make sense
./chkrootkit

After installing Chkrootkit using apt-get on my Debian "Jessie" server, I ran it and received the output shown in Listing  2, abbreviated for space reasons.

Each rootkit shown in Table  1 is searched for by using a variety of techniques, such as checking for process names and other known footprints. If nothing is found, infected, or suspect, you're probably in good shape. Figure  1 shows the results you can get if you run chkrootkit without the -x (expert) or -q (quiet) options.

Figure 1: Possible return values for the chkrootkit command.

If your system command binaries have not been overwritten, you will also see:

Checking 'grep'...          not infected
Checking 'su'...            not infected
Checking 'ifconfig'...      not infected
Checking 'killall'...       not infected
Checking 'top'...           not infected
Checking 'w'...             not infected

Notice midway down Listing  2 that suspicious files were found. You might be a little surprised to learn that I'm not worried about these results. Quite commonly you will receive false positives from Chkrootkit. My guess is that these particular files are flagged because the .path directories are hidden (i.e., they start with a dot).

If you're uncertain about suspicious output from Chkrootkit, you can use your favorite online search engine to investigate further or run Chkrootkit on a completely clean (offline) installation to confirm that the same alarms are triggered. If you check the bundled README.Debian file for Chkrootkit, you receive some reassurance not to panic (Listing  3).