Any Black Hat who finds a previously unknown vulnerability in the Linux kernel has hit the jackpot. Potentially millions of servers and embedded devices are suddenly open to attack, and the attacker can usually gain root privileges. Users clearly don't want this to happen, and kernel makers try to prevent such events.

Based on pure theory, strict coding standards and a sophisticated software quality management system ensure that loopholes are found immediately and eliminated before the release. A shining and rare example is OpenBSD, which in 20 years' time has only had two significant security breaches [1]. Although I am certainly a supporter of the vigilant approach followed by OpenBSD, we have to be realistic: The Linux kernel contains mountains of code that no one can review with the required depth; dependencies vary, and thus so do the possible attack vectors. (See the box entitled "Harmless Start" for a complex example.)

The complexity of the Linux kernel means that it is likely to carry legacy ballast and bugs for an indefinite period of time. At the end of 2010 [4], Jonathan Corbet checked how long the safety-relevant bugs eliminated in that year had existed until discovered: 22 of the 80 loopholes examined had been in the code for more than five years!

Practical experience leads to an approach that simultaneously makes attacks more difficult and reduces the consequences of exploitable code weaknesses. This two-pronged approach is the goal of the Kernel Self-Protection [5] project.

Break-In Technology for Everyone

Viewed through the looking glass with sufficient hindsight, most attacks on programs work in a similar way. An attacker tries to add new program code to a running process, which the hijacked process then executes with its privileges. The added code can be SQL or shell commands, or typically, binary code in kernel attacks. In order to inject this code, attackers exploit programming errors that allow them to determine memory contents and manipulate the program counter.

The program counter is a CPU register that points to the next instruction to be executed. For Intel processors, the 16-bit programs had the Instruction Pointer (IP), the 32-bit world had the Extended IP (EIP), and the 64-bit world has a somewhat-morbid Relative Instruction Pointer (RIP). Almost all attacks aim to change the contents of this register in such a way that it points to one of the attacker's commands instead of the next command intended by the programmer.

Direct write access to this register is virtually impossible, which makes a small detour necessary. When a program calls a function, the program copies the return address (i.e., the point at which the program will continue running after calling the function) to the stack. Since the stack also contains local function variables that the attacker may be able to manipulate through input, this return address is a popular destination.

A Leap Back to Ruin

The attacker has several options for changing the address. A buffer overflow is a bit rough: The attacker simply overfills a variable with excessive amounts of data and floods the return address (Figure 1). Slightly more subtle attacks rely on format string vulnerabilities, which make it possible to both read and manipulate the stack.

Figure 1: If an attack succeeds in storing more data in a buffer on the stack than the programmer had intended, the attacker can overwrite the valid return address in the buffer overflow.

Integer overflows are another possibility; for example, an attacker might exploit the fact that signed and unsigned integers have different value ranges. The unsigned number 128 could become a -128. If the program only checks whether a certain upper limit is complied with, for example, because there is a maximum of 100 values on the stack, -128 is technically OK. Instead of occupying memory locations 0 to 100 as the programmer intended, the range now extends from -128 to 100, which can trigger an overflow situation.

Run for Cover

An attacker can attempt to manipulate the stack or heap. Both are storage areas in which data is normally stored. Therefore, a popular avoidance strategy, which Kernel Self-Protection also advocates, is to mark memory areas as either executable or non-executable (in other words, to declare some areas as code and others as data). Newer CPUs have a bit for this in the page descriptor. AMD refers to this as the No Execute (NX) and Intel as the Execute Disable (XD) bit. Linux kernel supports write protection in this context.

Although these access rights do not provide 100 percent protection, they at least make the attack more difficult. For example, advanced attackers could still gain control of the system with a Return to Libc attack, which writes the entry address of execve() to the stack along with suitable parameters [6] [7].

Successful attackers first only manage to influence the memory of a hijacked process. It's a good thing that modern processors contain functions that separate the kernel memory from application memory. In this case, a kernel function must not execute any commands that have been injected into the user space. This protection also makes attacks more difficult.