Preventing the Kernel from Tainting

Matthew Garrett recently posted a patch to allow users to select at compile time whether the kernel would be "tainted" when loading unsigned modules. In the Linux world, tainting refers to whether the kernel is entirely open source or not. For example, if a user loads a binary-only module from a third-party vendor, it would taint the kernel because the module is not open source. The reason this is significant is that the Linux developers don't want to respond to bug reports that could be related to closed-source blobs of code to which they don't have access. Typically if a user sends in a bug report using a tainted kernel, the Linux developers will ask that user to reproduce the bug using an untainted kernel.

It wasn't immediately clear why Matthew wanted this code added to the kernel. He only said, "Distributions may wish to provide kernels that permit loading of unsigned modules based on certain policy decisions."

Rusty Russell said Matthew's explanation was too vague and asked for clarification.

Matthew pointed out that the kernel also included a run-time license checker that reads a variable included in each module, specifically identifying the license under which the module was released. Anything other than an open source license, he said, would taint the kernel. In which case, he reasoned, there was no need to taint unsigned modules, since the license checker would do it anyway.

This didn't make sense to Rusty, since it didn't answer the question of why one would want to add Matthew's patch in the first place. So Matthew offered this explanation: "A distribution may ship a kernel with signed modules. In some configurations, the signatures are irrelevant – there's no mechanism to verify that the correct kernel was loaded in the first place, so for all you know the signature validation code has already been removed at run time. In that scenario you're fine with users loading unsigned kernel modules, and there's no benefit in tainting the kernel. But the same kernel may be booted under circumstances where it is possible to validate the kernel, and in those circumstances, you want to enforce module signatures, and so sig_enforce is set."

Matthew went on to say that a main motivation for this patch was "in order to avoid dealing with user questions about why loading locally built modules now taints the kernel."

Jessica Yu came into the discussion at this point, saying that she still didn't see the point of going to all this trouble to disable a useful feature. She said, "I find it a bit contradictory to have CONFIG_MODULE_SIG enabled and at the same time expect the kernel to behave as if the option wasn't enabled."

But Matthew was still unable to give a clear explanation of why the patch would be useful. At one point, Ben Hutchings speculated, "The current state of affairs is that Debian doesn't have the mechanism in place to sign modules with a trusted key. If we were to allow third parties to add signatures in some way (I think that's what Matthew's interested in doing), we would have to enable CONFIG_MODULE_SIG, but that would cause modules to be tainted by default."

Matthew didn't clarify, and the thread petered out.

This kind of thing happens occasionally. A company will want to construct some sort of loophole in Linux policy and will try to justify it either vaguely or with made-up scenarios that aren't what the company really wants to do.

Encrypting printk() Output

Dan Aloni recently posted a patch to encrypt all printk() output (i.e., all kernel messages to the console). In this way, Dan hoped to eliminate possible attack vectors, by preventing hostile users from seeing what the kernel was up to. He said, "any information leak has the potential to make exploitation easier," and pointed out that he didn't use any special encryption scheme in his code, only the normal cryptographic keys that the kernel already used to sign firmware blobs and kernel modules.

Steven Rostedt replied that this did not seem like a very useful feature. Apparently, the only real users might be device manufacturers, hiding kernel output from regular users as well as from attackers. Steven said he wasn't opposed to a feature like this in principle, but only if folks like Google and other phone producers said it would truly be useful.

While not working for Google, Daniel Micay, nonetheless an android security engineer, pointed out that Android already used SELinux to prevent regular users from seeing whatever printk() sent to dmesg. He added, "Unprivileged processes including apps can't access dmesg, debugfs, sysfs (other than a tiny set of exceptions), procfs outside of /proc/net and /proc/PID (but it uses hidepid=2), and a few whitelisted files, etc. That's mostly done with SELinux along with using it for ioctl command whitelisting to have per-device, per-domain whitelists of commands instead of using their global seccomp-bpf policy for it."

To Steven, this seemed to indicate that Android would not enable Dan's feature, since printk() output was already restricted via SELinux. Absent any further information from device engineers, he said, Dan's patch didn't seem worth it.

The thread died there. It's interesting, because it leaves open the question of why the patch was developed in the first place. Was it to stop attackers from breaking through security? Was it to stop regular users from rooting their device? It's unclear.

New Kernel Bug Reporting Bot

Dmitry Vyukov announced the new syzbot open source automated bug identification system tailored for the Linux kernel. The idea behind this project was to automate the entire process, from bug identification, to sending the reports into the Linux developers, tracking the progress of each bug as the developers submit patches to fix them, and testing those proposed patches.

The problem was in the bug tracking portion of syzbot. There was currently no way to associate a given bug report with a future patch that might be intended to fix it. And so far, Dmitry had not discovered any way to purely automate a solution – he felt he would need collaboration from the developer community.

One possibility he had considered would be to use the ancient and still functioning Kernel.org Bugzilla system to track everything. Another would be for developers to tag Git commits with an ID provided by syzbot.

Greg Kroah-Hartman was opposed to using Bugzilla, but wanted to stick to using the official Linux Kernel Mailing List if possible.

Andrey Ryabinin suggested adding the ID number to the From: line of the syzbot emails. For example, From: syzbot-{hash} <syzkaller@googlegroups.com>. This way the tag would be likely to be kept via the normal kernel development sign-off process.

Linus Torvalds liked Andrey's idea, but offered a slight modification:

Put the hash in the original email instead, and use "+" instead of "-," because that's the standard email ("latter part doesn't matter"). Skip the proper name part entirely, since it doesn't really add anything.

And then make it show up in the Cc: part or Reported-by:, so it would just look something like Reported-by: syzkaller+1234abc@googlegroups.com and you're all done.

Dmitry liked this idea and implemented it in syzbot.

Meanwhile, Eric Biggers suggested that syzbot should do more to expose its internal data to developers. He wanted a dashboard of some kind, so that developers could see the status of each bug, when it was last seen, how frequently, and so on. He also felt it would be useful for syzbot to automatically send out reminders about old bugs that hadn't been fixed yet.

And in the case of bugs that were still technically unsolved, but that hadn't been reproduced in awhile, Eric felt they should be automatically invalidated, to prevent bug reports from piling up without bound, yet also without usefulness.

Dmitry replied, saying he'd like to expose the UI, as soon as he'd done some more work to spruce it up a bit. But he added that all the data was accumulating, so it would all be visible as soon as the dashboard came online.

Dmitry was reluctant to automate email reminders though, saying he was afraid they wouldn't be warmly received by the developers.

Regarding a public UI, Ozgur Karatas remarked that a graphical UI probably wouldn't be necessary and wouldn't be as valuable as a command-line interface.

But Dmitry said the web-based UI was already partially implemented so it would probably appear ahead of a command-line tool. Also, syzbot was implemented in "the cloud," so it didn't lend itself naturally to the command line. However, the whole thing was open source, so Dmitry invited Ozgur, and anyone else who pleased, to contribute patches to the UI or any other part of the code.

Meanwhile, Eric W. Biederman voiced some issues with syzbot's overall value. He said, "It tells us things are wrong but not how to reproduce the problem. Apparently syzbot will test fixes, but that doesn't help when more information is needed to track down the problem. The long of the short of it is that I don't care about bug reports that no one can reproduce and no human cares about." Eric also pointed out that "syzbot is written in a language (Go) that switches which kernel thread things run in at arbitrary times. That is absolutely not productive to understanding what is happening when things break. I have heard too many complaints from container run times that they can't make what should be a couple of lines change but is completely non-trivial because someone chose Go for their implementation language. Whatever benefits Go has, it is not a programming language I would choose for fine and reproducible control of kernel interfaces."

Dmitry replied that syzbot did try to include information about how to reproduce bugs. But he said, "Unfortunately, localizing kernel bugs is hard and is not possible in all cases. The root cause of this is actually in the kernel itself, not in syzbot. Things would be much simpler if we would work on a single-threaded, deterministic user-space library. Then we would get precise reproducers in 100% of cases. But kernel is a concurrent, parallel, non-deterministic system that constantly accumulates state."

Dmitry also said that in a lot of cases, the bug was trivial to understand simply by looking at the crash report – missing input checks, off-by-one errors, and so on.

As far as the programming language issue, Dmitry replied, "this is not true. The part of syzkaller that actually executes syscalls is written in C++ from day one." He continued, "It does explicit, manual thread scheduling; compiled as static binary to avoid any variance due to dynamic loading; does not use C++ run-time support nor malloc to avoid unexpected mmap calls. This is mostly for the reasons you outlined."

The discussion petered out around there. But the essential issues seemed to have been resolved – bug tracking. Beyond that, the rest will probably evolve over time to suit the developers' needs.