Virtualizing rootkits and the future of system security
A new generation of rootkits avoids detection by virtualizing the compromised system – and the user doesn't notice a thing.
In the typical cat-and-mouse game of attackers and defenders, the aim of the game is to gain or keep control of the operating system (see Figure 1). Legacy malware tries to escalate privileges and, if possible, to run in ring 0, the operating system's kernel mode. Once it gets there, the exploit, and thus the attacker, can manipulate the system.
Virtualization is often heralded as a big advance for system security. Multiple virtual systems can run on the same hardware without the ability to influence each other. This isolation prevents a number of standard attack techniques, but today's virtualization technologies also open a whole new frontier for attacks that never would have been possible in the past. Experts are already talking about a new generation of rootkits that will exploit the powers of virtualization to avoid detection.
Rootkits let an attacker secretly sustain privileged access to a computer. A rootkit can hide processes, network connections, files, and directories to remotely control the victim's PC, install backdoors, sniff network packets, or log keystrokes. Once the rootkit is running in kernel mode, it can filter and manipulate system call return values and very effectively hide files, directories, and processes.
A rootkit with access to kernel mode can easily terminate applications run in user mode (ring 3) by any normal user, including root. Once it has conquered the kernel, the rootkit is extremely difficult to identify and remove. Of course, the legitimate owner of the computer can also use kernel mode to set up an effective line of defense.
Virtualization essentially acts as another ring with even higher privileges than ring 0. Anyone who compromises the virtualization environment practically controls the whole physical environment on which the system runs. Malware hiding in this layer is even more difficult to discover and to remove than malware in kernel mode.
Researchers at the University of Michigan and from Microsoft Research demonstrated an initial proof of concept rootkit dubbed SubVirt  in March 2006, thus spawning the first generation of rootkits to exploit virtualization. After infecting a computer, the rootkit installs itself below the existing system and runs on a virtual machine after rebooting.
To allow this to happen, SubVirt also modifies the boot sequence so that the BIOS no longer loads the Master Boot Record (MBR) belonging to the operating system directly but instead starts a virtual machine. The virtual machine then executes the BIOS and launches the operating system copied into the virtual environment via the MBR.
While users carry on working on their – virtual – operating systems, unaware of what has happened, SubVirt launches a second instance and performs all kinds of nasty tricks. The rootkit cannot be terminated or uninstalled by the guest system because the rootkit controls the virtual machine on which the victim's guest system is running. Security researchers refer to this technique as a virtual machine-based rootkit (VMBR).
Figure 2 shows the new situation; the gray areas are occupied by the rootkit. The attacker's ability to control the victim's system also improves because the rootkit can now use the Virtual Machine Monitor (VMM) to manipulate, forward, or block arbitrary data and hardware characteristics en route to the guest operating system, without leaving the slightest trace of evidence that could be detected by legacy methods.
The researchers demonstrated their ability to compromise both Windows XP and Linux machines, implementing proof of concept attacks with four different vectors, including a phishing web server, a keyboard logger, and spyware that scans the infected system for confidential data.
The technology used by the Windows version of SubVirt is based on Microsoft's Virtual PC software, and the Linux version is based on VMware. However, you do need administrative privileges for the system to install the rootkit, although an attacker could use any number of approaches to gain administrative status.
Virtualization technologies such as VMware or Xen are so widespread that discovering an operating system is running in a virtual environment does not necessarily mean you have found a rootkit. Most diagnostic tools demonstrate the existence of the virtual environment on the basis of anomalies. They measure response times, with the assumption that the same command should take longer to complete in a virtual environment than natively – assuming identical hardware and an identical installation. The effect is caused by the virtual machine consuming CPU cycles itself.
This kind of automated timekeeping might be fine to detect a legitimate virtual machine; however, it does not rule out the existence of a rootkit because the rootkit would also control the internal clock. Also, the idea of using external hardware to measure response times manually does not scale very well.
What really gives away an infected system is anomalies in the visible hardware configuration, which are typical for a virtual environment and particularly true of graphics and network adapters. By comparing the known physical configuration with the output from commands like system-info, hwinfo, or the /proc filesystem, you can discover the differences (Figure 3). Windows admins will have to use the device manager or third-party tools.
The free disk space, or free memory, could also point to the existence of a virtual environment. For example, if you are unable to use the total physical size of your hard disk, the host system of a virtual machine might need this space itself. But be careful: The host system might also manipulate this data, in that it can control any kind of important output to the guest system.
Boot Externally and Scan the Disk
The rootkit described earlier, SubVirt, resides persistently on the hard disk; however, these changes are very difficult to identify on the running system. To reliably identify an infection, you might need to switch off the machine, boot from a different medium, and analyze the hard disk – of course, this method is problematic for many servers.
Tools developed specifically for this purpose give administrators another approach to detecting the existence of a virtualized operating system. For example, Joanna Rutkowska released Red Pill  late in 2004. It works because the SIDT, SGDT, and SLDT instructions executed by virtual systems return values different from those returned by a native CPU. For example, the SIDT instruction returns the address of the interrupt table.
As an alternative, Tobias Klein's Scoopy doo  and Jerry  tools will detect a VMware environment. If you are sure that you are running a system without virtualization software, positive findings by these tools are a real indication of an active VMBR.
This new generation of virtualizing rootkits might be dangerous, but, as you might expect, this technique also has some weaknesses. For example, the rootkit needs to reboot to become active, and the reboot is easy to detect. Rootkit programmers have cooked up other techniques -- some based on the more recent development of hardware-based virtualization.
Either the whole system is virtualized – this is the case with IBM's logical partitions (LPAR ), for example – or virtualization is restricted to individual components, such as the processor via Intel VT (Virtualization Technology; formerly Vanderpool ) or AMD Virtualization (formerly Pacifica ).
System or operating system virtualization relies on a VMM accepting instructions intended for the hardware from the guest systems (the virtual machines). Without processor support, the VMM has to capture and modify certain ring 0 instructions from the guest system, for example, to protect its own memory management against guest access.
In contrast, AMD and Intel's processor virtualization allows the VMM to send instructions of this kind directly to the processor. The CPU itself takes care of keeping the guest system's and the VMM's processes apart because their logic is inaccessible even to ring 0 processes. The ability to do without modification steps in the VMM helps the system achieve better performance.
Some researchers have started using hardware-based virtualization as a role model for a new generation of rootkits that benefit from the processor technology that allows them to insert an additional hypervisor between the visible hardware and the software. The hypervisor takes control of the system and converts the original operating system into a virtual guest on the fly. In contrast to software-based virtualization, this kind of hijacking does not need a restart, and that makes it all the more difficult to detect the intrusion.
Some rootkits use this kind of nesting technology, such as Blue Pill  by Joanna Rutkowska, which was released in 2006 for AMD-V, or Vitriol , which is suitable for Intel VT thanks to Dino Dai Zovi. In 2007, Rutkowska and Alexander Tereshkin relaunched Blue Pill, completely rewriting detect and adding a number of features . Once the new Blue Pill is running with administrative privileges, it enables SVM (Secure Virtual Machine) mode on the more recent AMD CPUs and set up the VMCB (Virtual Machine Control Block), which takes control of the infected OS in guest mode.
Until the next reboot, the rootkit itself works at a level below the hypervisor layer. In contrast to SubVirt, Blue Pill does not reside permanently on disk and thus does not survive a cold start. On the other hand, it doesn't leave behind any traces that could be discovered offline in the course of the forensic investigation.
The Internet is full of bulky and controversial discussions about how easy it is to identify a second-generation rootkit. Of course, command run-time measurements like those described earlier will be less reliable here because the overhead is smaller (or nonexistent) thanks to the host system. Because Joanna Rutkowska originally announced her Blue Pill as "undetectable Malware," people were quick to prove her wrong. Many suggestions as to detecting the rootkit by means of timing analysis were put forward. The approaches at best only managed to confirm that the operating system was running in a virtual environment, which does not necessarily mean a rootkit infection.
To avoid discovery, Rutkowska and Tereshkin have additionally developed a program called Blue Chicken , which detects timing analyses and temporarily removes the malware from virtual memory, thus preventing any timing anomalies. The race between the tortoise and the hare – that is, the game of hiding and discovering rootkits – is very much in full swing.
Buy this article as PDF
New release targets Linux professionals.
The Fedora project adds Wayland and Gnome 3.22
CeBIT 2017: Open Source Forum Call for Papers
Long-time Linux antagonist joins the revolution.
Major bug affects Debian/Ubuntu distributions.
Canonical releases the minimal edition for embedded devices, Internet of Things, and cloud deployments.
The new release features improvements across the board, from performance to security.
Two out of three of the new members are women.
More than 5,000 people attended the event.
Linux Magazine will include the best of both magazines.