Always on Your Guard

OOMd is also modular. In addition to PSI, the service relies on cgroups v2 under-the-hood; this is a kernel-based function for limiting and assigning resources and permissions. OOMd does not force the implementation of cgroups but uses cgroups v2 only as a second data source for its own monitoring of the system with a focus on RAM.

The dominant principle behind OOMd is that caution is far better than indulgence. Monitoring plays an important role: Unlike the kernel implementation of the memory killer, OOMd is designed to take action before a system-wide OOM situation occurs. To this end, the service continuously scans its main data sources – cgroups and PSI – and, in the event of an issue, automatically takes action before a genuine OOM situation actually occurs (i.e., before a malloc() call fails).

Because OOMd resides in user space and not in the kernel space, it benefits from far greater flexibility, especially with regard to extensions. It is thus not surprising that Facebook has designed the OOMd with a plugin interface that can be used to load external functionality. The plugins are directly integrated into the OOMd workflows – but the only plugin that comes with the OOMd is the one for killing processes. However, many options are conceivable, such as the option of performing various other tasks on a system before killing a process.

Simple Configuration

OOMd is written in C++ and uses the JSON notation as its configuration format. The results are very clear: The cgroups directive allows different procedures to be defined for individual cgroups, where target stands for the target group to which the entry refers. The other entries in each cgroup directive determine, among other things, which handlers and actions are used for OOM events. Figure 5 provides an example that makes it quite clear how flexible OOMd is in practice.

Figure 5: The OOMd configuration file is neatly organized and offers levers and switches for all important parameters in JSON format.

The example specifies that, when an OOM situation occurs, the chef service should be stopped first if it accounts for more than 60 percent of the total RAM load or has allocated more than 100MB of RAM for itself. The developers, however, have given the OpenSSH daemon (sshd) a survival guarantee; the max_usage:inf entry specifies that OOMd must never shut down the service under any circumstances.

This is also very useful, because without sshd, only a VNC or KVM console would remain for the remote login, and in the case of the kernel OOM, it is quite conceivable that the sshd would get into the firing line as one of many memory users. If you use OOMd, on the other hand, you can completely rule this out.

The Facebook developers heap praises on their own solution, by the way. In the documentation on GitLab, they state that the new OOMd has completely eliminated the previously described livelock situation that resulted in waits of 30 minutes or more.

Conclusions

Facebook brings an urgently needed and pleasantly fresh breeze into the OOM debate with its in-house solution: It now becomes clear that OOM is not a task that the kernel must handle. And it is also clear that functionality and flexibility can be achieved in user space that would be unthinkable in kernel space. The userspace OOM killer published by Facebook is interesting in this respect and will hopefully see rapid development.

And that additional development necessary: Currently, OOMd's functionality is limited to little more than what the existing OOM killer in the kernel can do. Nevertheless, Facebook explains that the tool works far better in everyday life than the original Linux solution. It is therefore almost impossible to predict what will happen if OOMd attracts a larger fan base and is available in packaged form for the usual distributions. It's quite possible that Facebook has succeeded in a real coup. One thing is certain: If you are looking for a virtual contract killer, you now have a choice.