Originally, I intended to write an article about the current state of rootkits and the tools that could be used to detect them. But I ran into a slight problem – the more modern rootkits tend to be really good at avoiding detection. By really good, I mean that you're unlikely to detect them unless you take action, such as a detailed analysis of a system memory dump, for example, comparing the actual kernel image with the expected.

History Lesson

Traditional rootkits were relatively simplistic programs, often running as a standalone daemon providing backdoor access. These were generally easy to detect by looking for a new process or newly installed software, which led attackers to start subverting system binaries. In turn, this led to attackers installing modified system binaries, such as hacked versions of OpenSSH that have a hard-coded administrative username and password to get root-level access. With the advent of tools such as Tripwire and the increasingly common use of package managers that can verify the integrity of installed files, such as RPM and dpkg, these became easy to detect [1].

Kernel-Based Rootkits

Soon attackers realized that more sophisticated hiding and subversion methods were needed to control a system, which led to kernel-based rootkits. By modifying the system call table, an attacker can avoid detection easily because, simply put, they control what you are seeing and how your programs are executing.

Typically attackers use one of two methods to modify the system kernel: either loading a malicious kernel module (e.g., heroin) or patching the in-memory kernel by writing to the special device /dev/kmem (e.g., SucKIT). Because these attacks live in memory only, their disadvantage is that they typically do not survive a reboot.

Difficult as they can be to detect, these rootkits can be found by comparing the current system call table with the expected (i.e., by examining the file System.map). Dumps of system memory can be taken and used to verify that the kernel in memory is correct.

So what are attackers to do? Go deeper, of course.

Hardware-Based Rootkits and the Virtualized OS

Released in 2006 at Black Hat in Vegas, the first publicized hardware-based rootkit was called "Blue Pill" [2]. Modern CPU's from AMD and Intel include a number of features that support virtualization of operating systems. Because they no longer need to modify the operating system to work, these rootkits are harder to detect, so checking your system call table won't work. However, these rootkits do replace the Interrupt Descriptor Table (IDT), which is held within a CPU register (the IDTR) [3].

Because two IDTRs (the real one and the one being presented to the compromised operating system) now exist, the one being presented to the compromised operating system will be at a different memory location than usual. Fortunately, the privileged instruction Store Interrupt Descriptor Table (SIDT) can be run from user space and will reliably return the contents of the IDTR being presented to the operating system (which isn't very helpful because it has been compromised) and, more importantly, the memory location (which won't be in the normal location).

This appears to be a stalemate: The attackers have created new methods to hide rootkits, and the defenders have found ways to detect them.