Zack's Kernel News

Retargeting the Magic SysRq Key

Andrzej Pietrasiewicz pointed out that not all computers had a "Magic SysRq" key on their keyboard. The SysRq key is generally the same key as Print Screen (PrtScn) on QWERTY keyboards, and the Linux kernel uses it to perform various operations even when the computer is completely locked up. It's invoked as Alt+SysRq+another key, where that other key is how you tell Linux what you want done. For example, Alt+SysRq+B will immediately reboot the system, without unmounting any drives or syncing any data.

Andrzej's patch allowed the user to configure a different key to have those magic powers instead of the SysRq key. By default, he felt that the F10 key would be appropriate.

He had submitted a similar patch in June 2020, which got a lukewarm reception, and now Andrzej was trying it out again, rebased onto the latest Linux kernel.

Randy Dunlap asked if Andrzej had tested the patch, because Randy noticed one of the lines of C code was missing the trailing semicolon. Andrzej replied that this was an error from the conflict-resolution phase of generating the patch and that he'd fix it. He quickly posted an updated patch.

Greg Kroah-Hartman's autoresponder email bot replied to Andrzej's patch, saying it appeared to be updating an earlier patch, but without specifying in the commit log what had changed since the previous version.

Elsewhere, Pavel Machek asked if F10 was the best key to replace SysRq on keyboards that had no SysRq of their own. He suggested Alt+Shift+Esc as a better alternative.

Meanwhile, Maciej W. Rozycki pointed out that "Lenovo in their infinite wisdom have placed the <PrintScreen> key (which in a traditional PS/2-keyboard manner produces <SysRq> when combined with <Alt>) in their keyboards between the right <Alt> and <Ctrl> keys. With thumbs not being as accurate as other fingers (and the overall misdesign of the keyboard and touchpad interface) you can imagine how often I have inadvertently hit <SysRq> combined with a letter key, wreaking havoc to my system (and of course I want to keep the key enabled for times when I do need it)."

Greg, however, caught up with the discussion and said, "The fact that this patch adds a 'new' sysrq key no matter what is a non-starter, please think through the consequences of such a change… So no, as-is, this change is not acceptable at all, and I would be amazed if anyone would ship such a thing." To which Andrzej replied that the patch didn't add a new SysRq key "no matter what." He added that, instead, "It does so only when the input device (keyboard) does _not_ have SysRq key at all. So I would say that this patch adds a replacement SysRq key if the SysRq key proper is _physically_ absent. Which seems not such a bad thing to me. The problem I'm trying to solve is exactly this: what to use as SysRq if there's no SysRq?" In a subsequent email he added, "Is 'connect an external keyboard' the _only_ choice Linux wants to offer to its users in case of devices such as e.g. Chromebooks?"

Greg was in no mood. He said, "it's an RFC, which can't be applied to the tree so I'll wait to review it 'for real' when you feel comfortable enough to submit it for inclusion."

And that was the end of the discussion.

It's unusual for an RFC to get stomped quite that hard by anyone other than Linus Torvalds, especially when there's an actual group of users for whom a given important feature is missing, such as the physical existence of the SysRq key.

On the other hand, this does seem to be a feature that Andrzej had already proposed in years gone by, so it's possible that with no significant changes to his proposal, the higher-ups such as Greg have less patience than they might have had earlier.

Either way, as features go, it doesn't seem particularly stomp-worthy. Although maybe there's a lot more hiding behind Greg's cryptic "think about the consequences" statement. Maybe the top kernel people feel that it's crucially important to protect users from accidentally triggering unexpected SysRq actions by hitting key combinations that didn't seem to involve that key.

It seems like something that will continue to be debated, especially if Chromebooks have no SysRq key and Lenovo has placed it between Ctrl and Alt. Clearly the hardware manufacturers are not up to speed on the importance of SysRq. Either way, Linux does like to support everyone, so it's likely that Linus would want Chromebooks and Lenovo laptops to have reasonable SysRq options, if possible.

The Seventh Circle of Bug-Tracking Hell

Thorsten Leemhuis wanted the Linux kernel documentation to stop sending users to bugzilla.kernel.org to file bug reports. He said Bugzilla wasn't working very well and wasn't the preference of most kernel developers. He also summarized "the good, the bad, and the ugly" aspects of Bugzilla, by way of explaining his suggestion.

Among the good items, he said that roughly 15 people listed as kernel maintainers actually did use Bugzilla, both to track and fix bugs reported by others and to report bugs themselves. Various other subsystems, said Thorsten, also use Bugzilla. He named Greg Kroah-Hartman as one of the active Bugzilla users, though he said Greg's replies on Bugzilla were often simply to counsel the person reporting the bug to send the report to a particular distribution or mailing list instead.

Thorsten also affirmed that sometimes if multiple people reported the same bug in Bugzilla, the tool would help them find and merge their reports and then identify the right maintainer to bring the bug report to. So to that extent, he said, it was useful.

And as the final "good part" of Bugzilla, Thorsten said it offered a file-sharing capability for bugs that needed memory dumps or other files to go with them.

Among the bad items, Thorsten said that Bugzilla had a very out-of-date list of kernel components and maintainers, causing bug reports to alert the wrong people or get included in the wrong sections of the tool. Thorsten went into detail about exactly how many different components and maintainers were listed wrong or left out completely in the tool.

And among the ugly items, Thorsten pointed out that Bugzilla itself was never the official place to report kernel bugs. Nor does Bugzilla's front web page make that clear – or point to the Documentation/admin-guide/reporting-bugs.rst file in the kernel documentation, which does give guidance on how to report kernel bugs.

Combined with "the bad," Thorsten said, "that's the reasons why quite a few (a lot?) reports never get a reply from someone. During a randomly selected 2 week window at the end of November 2020 there were 60 public bugs and a bit more than half of them by the end of the year never got a single comment by anyone except maybe the reporter."

He added, "it's irrelevant at all who's to blame for the state of bugzilla.kernel.org; it for sure was set up with good intentions, it just didn't work out very well in the end. The situations just needs to be improved, ideally quickly; blaming it on someone isn't helping at all."

In terms of specifically excluding people from blame, Thorsten said, "But there is one aspect that should be noted here: The situation can't be blamed on the kernel.org admins. They are doing a good job at keeping the bugzilla.kernel.org up and the bugzilla codebase up2date. But as admins it's not their job to maintain the list of products and components."

He then considered the question of why he proposed ditching Bugzilla rather than fixing it. He said, "It's well known for years now that bugzilla.kernel.org is not working that well, but nobody ever stepped up to improve the situation. Maybe this commit gets something rolling. If that's the case this change can be reverted. For now the change is an improvement that was agreed on during the maintainers summit 2017 in a session discussion [on] regression tracking."

Randy Dunlap had a nit to pick in Thorsten's analysis. Thorsten had said that a lot of the email addresses used by Bugzilla actually pointed to domains that don't exist, causing bug reports sent to those addresses to simply fail. Randy pointed out that these were actually "virtual" email addresses that were handled specifically by Bugzilla. So bug reports sent to those addresses wouldn't fail at all but instead would go to whoever they were configured to go to. So this was not a problem, although Randy acknowledged that the virtual addresses probably should be updated so more bug reports would go to more of the right people.

Thorsten bopped himself on the head for missing that, but he also asked who Randy believed would update that list of virtual addresses. Randy had said "we" should do it, so Thorsten asked:

"Who is 'we'? We as in 'the kernel community'? Or is there actually a smaller group of people you are referring to which is actively maintaining the list of products and components on bugzilla.kernel.org?

"Just trying to understand things better here, as there are other things that look strange to me and were mentioned in the patch description. For example: Why are there only 200 products and components on bugzilla.kernel.org (some of them for historic things like the ac-kernels) while the MAINTAINERS file has more than 2200 entries?"

By way of background, Thorsten added, "FWIW: I don't care too much about this whole thing, the whole idea for the approach I'm currently driving forward started when I did regression tracking in 2017. Back then I noticed quite a lot of bug reports on bugzilla.kernel.org never got a single reply, even if they were good and looked valid. That's why I brought this forward on the maintainers summit (https://lwn.net/Articles/738216/) and there it was discussed to basically go the route I'm taking currently. But I'm totally [fine] to adjust that route if there are good reasons, especially as that discussion happened some time ago."

Konstantin Ryabitsev from the Linux Foundation, and one of the kernel.org system administrators, remarked, "My general comment on this is that bug triage sucks and nobody really wants to do it for any extended period of time. :) There were times in the past when this or that person did step up and kept an eye on all incoming new bugs, properly routing them to the proper product/component, but they quickly burned out or found a less thankless occupation. Understandably."

He added:

"I want to encourage you and the rest of the developers to complain about this to the TAB [Linux Foundation's Technical Advisory Board]. It is entirely in their power to come to the Linux Foundation with the suggestion that perhaps bug triage should be a paid position. It's not a given that such a position would then be created and funded, but this for sure won't happen if these complaints don't reach People In Charge Of Funds at the LF.

"(FYI, this person shouldn't be me – every time I've come to the Foundation, I was asked that the proper way to go about it is through the TAB.)

"TBH, bug triage sounds like a great kernel developer semi-retirement gig. :)"

And he suggested:

"I'm not sure there's any single solution that will solve the problem. If we properly organize products/components, many people will just get lost in them and create all bug reports in 'other' (or 'helpdesk', as is the case lately).

"The sanest approach would be to have a simple web gateway to bug reporting:

– which distribution are you using?

– if they choose a distribution, show them where to report bugs for that distribution, because most bugs should start there, really

– on that page, also give a link: 'I'm a distribution maintainer and I want to report this bug upstream'

– if they click that link, let them fill out a freeform bug report that will create a new bug entry on bugzilla.kernel.org in 'Other/Other'

– creating a bug there will email the designated person in charge of initial bug triage

– that designated person or persons will then assign proper product/component, or simply forward the bug report to the proper maintainer if they are able to ascertain that

"This is far from perfect and still hinges on finding a person willing to do bug triage. However, it should hopefully improve the workflow without making it too complicated."

Close by, Thorsten remarked, "Sure, the LF or someone else could hire someone [...]; but I wonder if we have more pressing issues where the money would better be [spent] better. And even if not: getting that money and hiring someone would take some time…."

And in terms of finding someone among the kernel developers who'd be willing to do bug triage just out of love, Thorsten said, "Not me. ;-) That bugzilla.kernel.org is not working to well is known for years now, without anyone stepping up to improve the situation for real. Maybe my work/this discussion gets something rolling. But I guess until I see that happen I continue working towards discouraging people from using bugzilla.kernel.org, as otherwise things will just stay as they are."

At that point the discussion ended with no particular conclusion or decision coming out of it … until 10 months later, when Artem S. Tashkinov decided the moment was right to "express an utter dissatisfaction and even contempt for this proposal" – that is, the proposal to deprecate or remove Bugzilla from kernel development.

He said, "most open source projects are managed via bug trackers, I see no reason the Linux kernel should be exempt." He added that in spite of many developers and maintainers saying that they preferred to handle bugs via direct email, "it's an utterly preposterous, nonworking and broken idea." Specifically, said Artem, "Tons of messages in various kernel related mailing lists have zero replies and are not acted upon in any way or shape." And in such an email-based system, there was "no way to see what are the current issues, what's resolved or not." He added that "Users have an extremely hard time looking for bug reports which are spread along God knows where," and "It's impossible to follow up on such messages except when you were subscribed to the original mailing list."

He listed several things he felt could be done to resuscitate Bugzilla, including updating the various components listed in the interface and having someone go through all the bug reports and sort them into the right groups. He concluded, "I really hate how this bugzilla is treated as an afterthought which no one really cares about. If no one cares, why does it exist in the first place? Let's shut it down then. Let's move to mailing lists and create a total mayhem of lack of accountability. 'Bugs? Sorry, I've had no time to read emails'. [...] AFAIK kernel.org is a The Linux Foundation project. The organization is heavily sponsored by tons of companies. It would be great if we had a person actually invested in bugzilla.kernel.org."

Matthew Wilcox, in accordance with tradition, responded to Artem by saying, "Thank you for volunteering to take over administration of bugzilla.kernel.org. Can you lay out your plans for making kernel developers care about it?"

To which Artem answered, "Last time I checked the Linux foundation is not exactly living from hand to mouth and has enough financial backing from major corporations. Is this really about funding?"

The discussion died out shortly thereafter.

The thing I find fascinating about the discussion, and about the problem itself, is that bug tracking is a much-prized, highly desired aspect of pretty much any software project. And here is Linux, with tens of thousands of developers working to improve it all the time – most of whom do it for love, while many also do it as their full-time job. There is the well-funded Linux Foundation which supports Linux and open source in a not-for-profit capacity.

And yet even for Linux, the problem of bug tracking remains unsolved.

It's slightly reminiscent of the time when revision control was not solved for Linux, so Linus Torvalds took a couple weeks off kernel development and solved it for the entire world by creating Git. Maybe this is another such "if you build it they will come" situation.