Facebook releases its own OOM implementation
Facebook Not Amused
Compared to the apparent erratic way the badness()
function worked until 2010, the current OOM implementation is easy to understand and sensibly designed. But not all companies that use Linux on a large scale see it that way.
One very prominent critic of the current OOM implementation is Facebook: The company is known to rely on Linux in its data centers, albeit in a heavily modified form.
The Facebook developers summarized their problems with the kernel's own OOM killer on the net, stating that the OOM killer reacted unpredictably. The possibilities for influencing its function were too limited, and the configuration options inadequate. And because the OOM killer resided in the kernel, it was also extremely sluggish: There can be a pretty long wait between the time when the kernel realizes that there may be a memory problem, and the time when it actually does something about it and releases memory (Figure 4).

The kernel kills the processes one after the other, re-evaluates the situation, and has to empty the page cache in the meantime before looking at the next process, so the cycle is continually restarted until the system is running stably once again.
In the worst case, according to Facebook, livelock situations lasting more than 30 minutes can occur. As a reminder: A deadlock is a situation in which all components of a system are waiting for another component on the same system. Livelocks work in the same way, but the relations between the waiting components change at regular intervals. While the kernel could currently be waiting for the remains of a program to disappear from the page cache, it might be waiting for the OOM score to be calculated for a certain process a short while later.
Facebook's annoyance with this situation is understandable: During the 30 minutes that such a scenario allegedly takes, the system is practically unusable. Only a hard reboot is an effective measure, but that is exactly what Facebook wants to prevent.
OOM as a Holistic Measure
OOMd [1], which Facebook presented in August 2018, cuts off old braids and certainly takes a very bold approach to the subject of OOM. The biggest difference with the existing OOM implementation in Linux is probably the fact that Facebook's OOM killer does not run in kernel space, but as a normal application in user space. This makes OOMd a real novelty, as OOM has undisputedly been part of the kernel domain until now.
OOMd doesn't have big technical disadvantages – ultimately, if it is running with sys admin privileges, it can kill a process simply by issuing a SIGKILL
. The OOM killer in the kernel doesn't do anything wildly different at the end of the day.
In return, the advantages of the userspace implementation are obvious: In particular, OOMd is far more flexible than the kernel implementation could ever be.
Adding PSI
If you want to try out OOMd right after reading this article, be warned: You still need the kernel. The functions do not reside in the kernel in OOMd's case. But OOMd in user space depends on receiving as much information as possible about the current state of the system from the kernel. A suitable interface already exists in the form of Linux-PSI [2], a component that reports on kernel memory, CPU, and IO pressure metrics. Linux PSI is not yet in the kernel. If you want to use OOMd, you have to build PSI into your kernel, a task that is made easier because PSI is available as a kernel module.
« Previous 1 2 3 4 Next »
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Direct Download
Read full article as PDF:
Price $2.95
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Find SysAdmin Jobs
News
-
KDE Plasma 5.27 Beta is Ready for Testing
The latest beta iteration of the KDE Plasma desktop is now available and includes some important additions and fixes.
-
Netrunner OS 23 Is Now Available
The latest version of this Linux distribution is now based on Debian Bullseye and is ready for installation and finally hits the KDE 5.20 branch of the desktop.
-
New Linux Distribution Built for Gamers
With a Gnome desktop that offers different layouts and a custom kernel, PikaOS is a great option for gamers of all types.
-
System76 Beefs Up Popular Pangolin Laptop
The darling of open-source-powered laptops and desktops will soon drop a new AMD Ryzen 7-powered version of their popular Pangolin laptop.
-
Nobara Project Is a Modified Version of Fedora with User-Friendly Fixes
If you're looking for a version of Fedora that includes third-party and proprietary packages, look no further than the Nobara Project.
-
Gnome 44 Now Has a Release Date
Gnome 44 will be officially released on March 22, 2023.
-
Nitrux 2.6 Available with Kernel 6.1 and a Major Change
The developers of Nitrux have officially released version 2.6 of their Linux distribution with plenty of new features to excite users.
-
Vanilla OS Initial Release Is Now Available
A stock GNOME experience with on-demand immutability finally sees its first production release.
-
Critical Linux Vulnerability Found to Impact SMB Servers
A Linux vulnerability with a CVSS score of 10 has been found to affect SMB servers and can lead to remote code execution.
-
Linux Mint 21.1 Now Available with Plenty of Look and Feel Changes
Vera has arrived and although it is still using kernel 5.15, there are plenty of improvements sure to please everyone.