Applying updates to an active kernel with Ksplice
Piece of Cake
![© anjapepunkt, photocase.com © anjapepunkt, photocase.com](/var/linux_magazin/storage/images/linux-magazine.com/issues/2008/95/piece-of-cake/pc_afzv9qr6apkt_anjapepunkt.png/429284-1-eng-US/pc_afzv9qr6apkt_anjapepunkt.png_medium.png)
© anjapepunkt, photocase.com
Uptime is often just as important as updates. But doesn't a kernel patch require a reboot? Ksplice lets you have your cake and eat it too.
Many administrators don't relish the thought of installing a new kernel, so most distributions come with package management and installation tools that simplify the task. After you load and install the new kernel and register it with the bootloader, the reboot should take just a couple of minutes. But a couple of minutes of downtime is too long for some applications.
If you manage a system used for number crunching, such as in weather simulations, or a system that has to manage a large number of network connections, such as a phone server or online shop, you probably would prefer to avoid a reboot.
On the other hand, administrators are also responsible for the security of the systems they manage. Surveys have revealed that developers discover a Linux kernel bug every three weeks on average. The open development model means that patches appear often.
Jeffrey Brian Arnold from the Massachusetts Institute of Technology established in a survey [1] that patches were very simple in most cases. Eighty percent of them comprised fewer than 15 lines of code, and more than half were little more than one-liners. Problems are often caused by "off-by-one errors," in which a developer has simply miscounted a parameter such as an array boundary. Bugs of this type are very easy to resolve. For example, Listing 1 shows a patch for the prctl() system call, which resolves the vulnerability listed as CVE-2006-2451. The problem, for which exploits exist, takes just one line of code to fix.
Listing 1
Patch for CVE-2006-2451
Arnold developed the Ksplice package [2] with the idea of patching the active kernel directly, without the need for a reboot. The program just needs the source code for the current kernel, the configuration data, and the symbol table. The best thing is that you do not even need to prepare the running system to use Ksplice. The program can modify any kernel as of version 2.6.8.
Linux distributions offer the kernel source code, or, if you built your own kernel, you will find it in /usr/src. Distributions typically store the two files config and System.map in /boot. On top of this, Ksplice obviously needs a patch file or one or multiple files with changes. The program builds two new kernels: pre designates a version of the current system, and post designates the kernel after the updates.
Spot the Difference
After building both kernels, Ksplice looks for differences in the object code (see Figure 1). If Ksplice were to analyze the source code directly, it would need to emulate all of the compiler's decisions, which is far too complicated for the task in hand. For this reason, Ksplice uses the GNU BFD library [3] to search for functions in the object code that have changed in the new kernel. Ksplice then adds the new code to modules and inserts trampoline jumps at the start of the original functions that then point to the new versions. When the administrator finally enables the changes, Ksplice loads two kernel modules on the running system that then perform the modifications (see Figure 2).
![](/var/linux_magazin/storage/images/linux-magazine.com/issues/2008/95/piece-of-cake/figure-1/429287-1-eng-US/Figure-1_large.png)
![](/var/linux_magazin/storage/images/linux-magazine.com/issues/2008/95/piece-of-cake/figure-2/429290-1-eng-US/Figure-2_large.png)
One critical issue is timing as to when Ksplice is allowed to install the trampoline jumps. Trouble occurs if a kernel thread is currently running one of the functions that needs to be replaced. To avoid this, the program calls stop_machine_run() to stop the thread execution because the function creates a high-priority process for each CPU. The Ksplice module now checks to see whether the change candidate contains threads. If it does, the module waits for a while and then retries. This approach will not work for some functions, such as the scheduler, because the scheduler will always be doing something. In that case, Ksplice gives up; in all other cases, it installs the jump addresses. From now on, the Linux kernel executes the patched version of the functions.
Inner Values
Ksplice has to find the right functions and vector addresses in relocatable code. The program is capable of detecting changes in relative jump addresses where the function itself is not affected by a modification, thanks to the new length of the patched function. The kernel typically enters functions implemented in C at the start only. In contrast, the program has to search for the vector in the case of assembler code.
If the program were to use a different compiler to create the pre kernel, it could lead to incorrect assumptions about the running system. Ksplice relies on substantial logic to analyze the symbol tables, which many distributions store in the /boot/Symbol.map file. On top of this, it builds the two kernels with a number of compiler options that assign a separate ELF text segment to each function to make it easier to identify modified, relative jumps.
Because it handles code like a black box, Ksplice can't detect changes in data structures. For example, if a patch adds a new attribute to a data structure or changes its layout, unpredictable issues will occur. Trampoline jumps will typically pick up function pointers, but there is no guarantee Ksplice will handle more complex pointer arithmetic or "creative" typecasting gracefully in each case.
Off Limits
In the documentation, the author emphasizes that he mainly designed the tool for minor security patches and that it is the system administrator's own responsibility to read, understand, and evaluate the patch before applying it. In other words, you need a great deal of kernel expertise to apply the tool; otherwise, the effect might be far more destructive than a bit of reboot downtime.
Because Ksplice cannot make semantic changes to a running kernel, the administrator's pipe dream of measuring uptime in years is just that because most changes between one kernel release and the next add some new functionality.
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
![Learn More](https://www.linux-magazine.com/var/linux_magazin/storage/images/media/linux-magazine-eng-us/images/misc/learn-more/834592-1-eng-US/Learn-More_medium.png)
News
-
NVIDIA Released Driver for Upcoming NVIDIA 560 GPU for Linux
Not only has NVIDIA released the driver for its upcoming CPU series, it's the first release that defaults to using open-source GPU kernel modules.
-
OpenMandriva Lx 24.07 Released
If you’re into rolling release Linux distributions, OpenMandriva ROME has a new snapshot with a new kernel.
-
Kernel 6.10 Available for General Usage
Linus Torvalds has released the 6.10 kernel and it includes significant performance increases for Intel Core hybrid systems and more.
-
TUXEDO Computers Releases InfinityBook Pro 14 Gen9 Laptop
Sporting either AMD or Intel CPUs, the TUXEDO InfinityBook Pro 14 is an extremely compact, lightweight, sturdy powerhouse.
-
Google Extends Support for Linux Kernels Used for Android
Because the LTS Linux kernel releases are so important to Android, Google has decided to extend the support period beyond that offered by the kernel development team.
-
Linux Mint 22 Stable Delayed
If you're anxious about getting your hands on the stable release of Linux Mint 22, it looks as if you're going to have to wait a bit longer.
-
Nitrux 3.5.1 Available for Install
The latest version of the immutable, systemd-free distribution includes an updated kernel and NVIDIA driver.
-
Debian 12.6 Released with Plenty of Bug Fixes and Updates
The sixth update to Debian "Bookworm" is all about security mitigations and making adjustments for some "serious problems."
-
Canonical Offers 12-Year LTS for Open Source Docker Images
Canonical is expanding its LTS offering to reach beyond the DEB packages with a new distro-less Docker image.
-
Plasma Desktop 6.1 Released with Several Enhancements
If you're a fan of Plasma Desktop, you should be excited about this new point release.