Root out rootkits

On the Trail

© Lead Image © silense, 123RF.com

© Lead Image © silense, 123RF.com

Article from Issue 180/2015
Author(s):

Detect rootkits, worms, and other malware with chkrootkit.

Sophisticated software can help you keep an eye out for malware, but by the time it's spotted, it's probably too late, costing you the time to repair and rebuild your system, as well as a loss of revenue if it's a production system. Chkrootkit is a proven, mature, and highly sophisticated tool that helps you detect a variety of malware in an ongoing process.

A True Story

A year or two after the turn of the millennium, I lost a server through the disastrous compromise of a popular web-based application. Thankfully, although the server was in use, there were no services on it that couldn't wait a few days before being restored.

The first clue that something was wrong was some strange filesystem behavior while logged in over SSH. I checked the Apache error logs, and – lo and behold – I found some output detailing a wget transaction. The wget file transfer was showing an illegitimate download from a nasty-looking URL.

Although the file name was unusual, what struck me was that all of the elements of the transaction were running under the apache user (hence, the owner of the error.log in question) and using the /var/log/apache directory. This server was not without security, but after some forensic analysis once the server was taken off the network to avoid further local network damage, I discovered that a web application written in PHP had been compromised. PHP, of course, being run by the apache user.

After gaining access via local user apache, the miscreants had downloaded a rootkit without root permissions and managed to spawn a binary that gave them remote SSH access. From there, all hell broke loose.

First, when I accessed the server I noticed an unusual open port  – not 22 or a port that I used for the SSH server (e.g., port 2222). Moreover, the /tmp directory contained some slightly odd-looking temporary files. I suspect this is where the rootkit dropped its initial installation files after it had "broken free" of Apache's user, because /tmp is usually writeable by all.

Second, when I ran a binary file, such as listing files in a directory, I noticed the server was a little clunky and unresponsive. It turns out the hacker had dropped alternative executables of system commands that fired up in their place.

Analysis later showed that at least 20 common binaries had been overwritten by a rootkit. These ranged from the simple ls to more obscure commands. The nefarious attacker had put these alternatives in place to hide their presence on the system. Logs were also manipulated to show fewer details of the attacker's activities. However the attacker had either forgotten about Apache's error log or thought it was abstract enough not to be checked.

Seemingly simple but clever tricks, such as replacing the w command, were employed. That command would normally offer some information on which users were logged on to the system and who was doing what. With the new version of the command, though, the logged-in hacker was essentially invisible until I dug deeper.

To avoid a full system rebuild, I downloaded a number of stock binaries from a source I trusted and overwrote those present on the filesystem. I then watched and waited patiently. Suddenly, I saw some network traffic to an IP address in Brazil and activity on the server that wasn't legitimate.

I repeated that procedure about three times, overwriting binaries again and again and adding firewall rules, but still something on the server was phoning home and letting the miscreants back in. Eventually I gave up, put a brand new disk in the box, and rebuilt the server from scratch. Incidentally, I would not have even tried dropping replacement binaries onto a production box; it is simply too risky. You've got to rip it up and start again if that ever happens on an important server.

One final, alarming, fact is that forensics proved that at midnight any login passwords (and users) were being covertly emailed back to an IP address in Brazil. That's not a good thing I'm sure that you'll agree.

In this article, I'll look at rootkits and chkrootkit, a stalwart tool that helps identify machines that have been infected.

Roto-Rooter

In simple terms, rootkits essentially permit illegitimate access to a system against the owner's wishes. They come in all shapes and sizes and have most recently been discussed in relation to the UEFI (Unified Extensible Firmware Interface) "Secure Boot" functionality recently introduced. Secure Boot is meant to prevent the installation of rootkits into what used to be called the BIOS (Basic Input/Output System), so that your server or workstation isn't compromised before it has even booted up. To be clear, I'll be looking at operating system rootkits, as opposed to the boot-time equivalent.

One statement that probably needs to be reiterated is that if you lose a machine to a rootkit, you shouldn't risk your data for the sake of one hour's extra work; simply start again with a fresh build. The point I'm trying to make clear is that by the time you've spotted the evidence of a rootkit, it's probably too late. However, by using a clever tool to assist you, you at least can keep an eye on config mistakes that you make and potentially monitor unusual activity on your machines, even if they're not rootkit related. Whichever way you decide to use these sophisticated security tools, however, it should be a great learning experience. Now I'll look the venerable Chkrootkit utility.

Table  1 lists a sample of malware that the chkrootkit [1] tool can detect. This substantial list should be extremely worrying; just look at all the rootkit variants that have been discovered in the past. The Chkrootkit website also notes which tests are run to detect these kits (Table  2).

Table 1

Malware Currently Detectable by Chkrootkit.*

Rootkits

   AjaKit

    shv5

   Ambient's rootkit (ARK)

    SK

   Anonoying rootkit

    Solaris rootkit

   Aquatica

    SucKIT

   Bobkit

    TRK

   dsc-rootkit

    t0rn (and variants);

   duarawkz

    t0rn v8.0

   Ducoci rootkit

    Volc

   ESRK

    zaRwT

   FreeBSD rootkit

    ZK

   Fu rootkit

   George

Worms

   Gold2

    55808.A

   Hidrootkit

    Adore

   Illogic

    Lion

   Kenga3

    LPD

   kenny-rk

    Lupper.Worm

   Linux Rootkit 64Bit

    Omega

   LOC rootkit

    Ramen

   LRK3, LRK4, LRK5, LRK6 (and variants)

    ShitC

   Madalin

    TC2

   Maniac-RK

    Wormkit

   MithRa's rootkit

    x.c

   Monkit

   OpenBSD rk v1

LKMs

   Optickit

    Adore

   Pizdakit

    Enye

   rh[67]-shaper

    knark

   RK17

    sebek

   Romanian rootkit

   Romanian rootkit

Others

   rootedoor

    Operation Windigo

   RSHA

    OSX.RSPlug.A

   Shkit

    RST.b trojan

   Showtee

    Scalper

   shv4

    Slapper A, B, C, and D

*From the Chkrootkit website [1].

Table 2

Tests Run Against a System by Chkrootkit

aliens

ifconfig

rpcinfo

amd

inetd

rshd

asp

inetdconf

scalper

basename

init

sendmail

biff

killall

slapper

bindshell

ldsopreload

slogin

chfn

lkm

sniffer

chkutmp

login

sshd

chsh

ls

su

cron

lsof

syslogd

crontab

mail

tar

date

mingetty

tcpd

dirname

named

tcpdump

du

netstat

telnetd

echo

OSX_RSPLUG

timed

egrep

passwd

top

env

pidof

traceroute

find

pop2

vdir

fingerd

pop3

w

gpm

ps

w55808

grep

pstree

write

hdparm

rexedcs

wted

identd

rlogind

z2

LKMs (loadable kernel modules) are loaded dynamically into memory and interface directly with the kernel and its subsystems. When this type of access is successful, you have usually reached the point of no return. To see which kernel modules are in use, enter one of the following commands:

lsmod
cat /proc/modules

The modules file in the /proc pseudo-filesystem contains all modules loaded into the kernel (Listing  1).

Listing 1

/proc/modules

 

The commands to query LKMs are useful if the rootkit in question is determined to include the source code of some nefarious kernel modules within its payload. Of course, if a compiler is not installed on your system, then the rootkit's code becomes a serious challenge to compile, and that's just what you want. It means the hacker would need to pull down precompiled binaries to your machine, which might not work correctly on your system for a number of reasons.

Take note that some distributions seem to insist that you leave a compiler like GCC installed. You can check for GCC's presence with:

whereis gcc make

Entering gcc -v reveals the version. If safe to do so (i.e., nothing on your system requires that GCC remain there), you can remove the GCC compiler packages from a Debian distro with the following commands:

apt-get remove gcc
apt-get remove build-essential

The build-essential package drops about 20 tools onto your system that allow you to compile from source. I would never leave these packages installed on a production server for any period other than the five minutes every three months that I needed them.

Using Chkrootkit

To install Chkrootkit, you can use the usual method:

apt-get install chkrootkit

However, even from source [2], Chkrootkit is very simple to compile and execute the following:

make sense
./chkrootkit

After installing Chkrootkit using apt-get on my Debian "Jessie" server, I ran it and received the output shown in Listing  2, abbreviated for space reasons.

Listing 2

Chkrootkit Output on a Systemd Server

 

Each rootkit shown in Table  1 is searched for by using a variety of techniques, such as checking for process names and other known footprints. If nothing is found, infected, or suspect, you're probably in good shape. Figure  1 shows the results you can get if you run chkrootkit without the -x (expert) or -q (quiet) options.

Figure 1: Possible return values for the chkrootkit command.

If your system command binaries have not been overwritten, you will also see:

Checking 'grep'...          not infected
Checking 'su'...            not infected
Checking 'ifconfig'...      not infected
Checking 'killall'...       not infected
Checking 'top'...           not infected
Checking 'w'...             not infected

Notice midway down Listing  2 that suspicious files were found. You might be a little surprised to learn that I'm not worried about these results. Quite commonly you will receive false positives from Chkrootkit. My guess is that these particular files are flagged because the .path directories are hidden (i.e., they start with a dot).

If you're uncertain about suspicious output from Chkrootkit, you can use your favorite online search engine to investigate further or run Chkrootkit on a completely clean (offline) installation to confirm that the same alarms are triggered. If you check the bundled README.Debian file for Chkrootkit, you receive some reassurance not to panic (Listing  3).

Listing 3

README.Debian File for Chkrootkit

 

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy Linux Magazine

SINGLE ISSUES
 
SUBSCRIPTIONS
 
TABLET & SMARTPHONE APPS
Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

  • chkrootkit

    Linux can be infected by rootkit malware that is hidden and hard to detect. The chkrootkit program can help find rootkit infections.

  • Expert Security Intro

    Internet intruders have many ingenious ways of escalating privileges and hiding their presence once they get inside your system. The best protection is to keep them out in the cold.

  • Host-Based IDS

    A host-based intrusion detection system is a simple but powerful tool for finding traces of an attacker's footprint.

  • Kernel Rootkits

    Rootkits allow attackers to take complete control of a computer. We describe the tricks intruders use to gain access to the Linux kernel and provide guidelines on hardening the kernel against such attacks.

  • On the Hunt

    The Rootkit Hunter script efficiently checks for malware, with the potential to detect over 240 rootkits.

comments powered by Disqus
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters

Support Our Work

Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.

Learn More

News