Zack's Kernel News

Submitting Fixes to Various Trees

Greg Kroah-Hartman revealed a schism in the development process when he complained that he had more than 170 patches marked for the stable series that he couldn't apply because they hadn't gotten into Linus's tree yet.

One of the rules for the stable tree is that it only accepts patches after they are already in the main tree. This process may seem odd, but it's intended to prevent too much divergence between the trees. The thinking is also that anything good enough for the stable tree is good enough for the main tree, and the way to ensure that it actually goes into the main tree is to make that an explicit requirement.

Linus's development process has evolved quite a bit over the years, but at the moment it seems to start with a brief "merge window," during which time developers submit their patches. Then, it goes through a series of stabilizing release candidates – from -rc1 to -rc7 or -rc8 or thereabouts. After that, Linus releases a new official kernel version, and the process begins again.

Linus is very strict about only accepting important fixes during the -rc series. Additionally, his strictness increases with each release candidate, as does the volume of his rejection of patches that don't fit his idea of what's acceptable for a given release candidate. He is so strict, in fact, that, as Dave Jones points out, as soon as -rc1 is released, many developers pull their heads down into their tortoise shells and wait for the next merge window.

Thus, a lot of developers have begun sending patches to Greg for the stable series instead, relying on him to pass things to Linus at the appropriate time. This approach, however, results in a delay leading to big piles of unapplied patches, as I mentioned.

In response to Greg's complaint, a number of developers said that they didn't have a clear idea of what sorts of patches to submit to Linus as opposed to Greg, and at what point in the -rc release schedule it was still OK to send those patches. There were written guidelines in Documentation/stable_kernel_rules.txt, but several folks pointed out that those guidelines were not clear enough.

Greg and Linus both tried to explain their expectations to people, but their explanations seemed to conflict. Linus said that, after -rc3, he wanted only regressions, crash fixes, and security fixes; whereas Greg said that, for his stable tree, he wanted patches that fixed real problems. Of course, "real" problems could include a lot of things that Linus's explanation excluded, and that could explain the situation of developers sending more patches to Greg and not sending enough to Linus.

A number of folks tried to introduce new rules to clarify the situation; for example, Willy Tarreau suggested that in many cases developers should tag their patches as fixing a bug from a particular kernel version, rather than simply tagging it as being for the stable tree. Then, only important stability fixes could be tagged as being specifically for the stable tree.

Theodore Ts'o and Guenter Roeck both liked that idea, but Greg didn't like it. As he said, "if something 'fixes' an issue, then I want it in stable, just like Linus wants that in his tree. Don't add another tag that requires users to dig for fixes that we are just too lazy to be including for all users, that way is crazy."

So, the mixed message seemed to be that Greg wanted all fixes to come to the stable tree, whereas Linus wanted only regression, crash, and security fixes.

However, as Greg clarified, his problem was that people were sending him things that were not fixes at all. For example, adding a newline to debugging output. True, that might be a fix for a group of developers who wanted better debugging output, but it wasn't a good enough fix to justify sending to the stable tree. But, Greg also added, "It's a grey area, which is good, let's keep it that way."

At one point, Linus said to Greg, "the reason you get a lot of stable patches seems to be that you make it easy to act as a door-mat." He added, "You may need to learn to shout at people."

Shouting, Culture, and Abuse

At a certain point in the discussion about which patches should be sent to the stable tree and the various -rc releases, the discussion shifted to whether to shout at people. These comments led to a bit of a flame war.

Typically, Linus will give someone a dressing-down when he thinks they're doing something bad that they should know better than to do. His comments may include all-caps, curses, and threats to refuse future patches.

One reason developers are reluctant to send patches to Linus during the -rc series is because they're afraid he'll yell at them. Meanwhile, Greg doesn't yell at anyone for sending patches to the stable tree, and so he's tended to get a growing pile of patches that need to be sent to Linus before they can be accepted into the stable tree.

Linus suggested that Greg learn to yell at people more. At this point, however, Sarah Sharp said, "Seriously, guys? Is this what we need in order to get improve -stable? Linus Torvalds is advocating for physical intimidation and violence." She added, "Violence, whether it be physical intimidation, verbal threats or verbal abuse is not acceptable. Keep it professional on the mailing lists."

She posted some examples of Linus being particularly vociferous and said, "I'm not going to put up with that shit any more."

Linus replied: The fact is, people need to know what my position on things are. And I can't just say "please don't do that", because people won't listen. I say "On the Internet, nobody can hear you being subtle," and I mean it.

And I definitely am not willing to string people along, either. I've had that happen too – not telling people clearly enough that I don't like their approach, they go on to re-architect something, and get really upset when I am then not willing to take their work.

Sarah, first off, I don't have that many tools at hand. Secondly, I simply don't believe in being polite or politically correct. And you can point at all those cultural factors where some cultures are not happy with confrontation (and feel free to make it about gender too – I think that's almost entirely cultural too). And please bring up "cultural sensitivity" while at it. And I'll give you back that same "cultural sensitivity". Please be sensitive to _my_ culture too.

Linus added, "Some of the above is written tongue-in-cheek, but all of it is also serious. I really fundamentally believe that being honest and open about your emotions about core/process is good. And because it's damn hard to read people over email, I think you need to be *more* honest and *more* open over email."

Sarah replied, "You can tell developers in no uncertain terms that you're not willing to take their work *without* verbally tearing them apart. You're Linus Torvalds, for crying out loud! A simple, 'No, that's a bad idea, stop working on this RIGHT now,' is more than enough from you." She added, "I've seen you be polite, and explain to clueless maintainers why there's no way you can revert their merge that caused regressions, and ask them to fix it without resorting to tearing them down emotionally."

Sarah also said, "You just don't want to take the time to be polite to everyone. Don't give me the 'I'm not polite' card. Go write some documentation about what's acceptable for stable. While you're at it, write some more documentation about why it's impossible for you to revert merges, so maintainers know not to send you crap." And, she said, "*No one* deserves to be yelled at IN ALL CAPS in email, or publicly ridiculed. It doesn't matter if they are a minority or not. You are in a position of power. Stop verbally abusing your developers."

Rob Landley said, "Not _all_ of us lose our voice when yelled at by Linus's lieutenants. Some of us just post updates to the same darn patch series for 5 years (yes, really; my Perl removal series started in 2008 and was applied earlier this year), on the theory it's useful to the people actually applying it to their own trees (at one point, Gentoo), and that someday the stars might be right and Cthulhu will arise from the deep and accept the patch series into his tree. (Or in my case, Andrew Morton.)"

Linus said to Sarah: Oh, I'll be polite when it's called for.

But when people who know better send me crap, I'll curse at them.

I suspect you'll notice me cursing *way* more at top developers than random people on the list. I expect more from them, and conversely I'll be a lot more upset when they do something that I really think was not great.

For example, my latest cursing explosion was for the x86 maintainers, and it comes from the fact that I *know* they know to do better. The x86 tip pulls have generally been through way more testing than most other pulls I get (not just compiling, but even booting randconfigs etc.). So when an x86 pull request comes in that clearly missed that expected level of quality, I go to town.

Similarly, you will see fireworks if some long-term maintainer makes excuses for breaking user space etc. That will make me go into incoherent rages.

The "polite Linus" example that you point to? That was a maintainer asking for direction for when things went wrong and *before* sending me something dubious. Of course I'm polite then.

Sarah, I don't have Tourettes syndrome. You seem to think that my cursing is uncontrolled and random. I argue that it has causes. Big difference.

Willy Tarreau also said: Most of us have already been scolded by Linus, and while it usually is an unpleasant moment, I do think that it's efficient and (it might surprise you) probably a mark of respect. Please re-read some of the famous public flames from Linus. When he tells you "stop saying such idiocies, you're a f*cking moron", he doesn't really mean that, he means that he's very disappointed that *that person* says this or that, so he takes the time to say it to that person. The proof is that most often in the next mail he explains to the person how to do the thing right. He just tries to ensure that the person he's telling words to understands that he/she has crossed a line.

Sure it can be hard for newcomers but I don't remember having read him scold a newcomer. So that's probably not that much of a problem in the end, and helps getting the things done in time. I'm much more concerned by the "administrative" development mode that we're taking in fact and that some people seem to have expressed in this thread (what patch flow to follow, when to send/not to send, etc. …).

Sarah said to Linus: It does not matter if your cursing fits have causes. The fact is that if you misjudge someone's emotional state for the day, you yelling at them is not productive.

In karate, or any other sport, if your opponent is motionless on the floor, you stop. You can't see the person you're emailing. You can't see if the first conversation-disabling blow has completely knocked them out. You can't see if you've misjudged their mental strength for the day and completely wiped out their ability to use their brain to correct the technical mistake you're trying to get them to fix.

Ask them to fix their mistake. If they protest, then lay into them. If you *know* they don't take verbal abuse well, don't.

Let's get this personal baggage out of the way right now, before anyone else brings it up: I've been through verbal abuse before. I won't take that shit from you, or any of the other Linux kernel developers. Tell me, politely, what I have done wrong, and I will fix it. You don't need to SHOUT, call me names, or tell me to SHUT THE FUCK UP!

I'm not the only one that won't take verbal abuse. Stop abusing your developers.

This made sense to Kees Cook, who added, "Even if we accept that some people can "correctly" choose when to be abusive, it creates an atmosphere where other people will come to think that kind of thing is okay." He went on:

People's mistakes can be pointed out without the kind of abuse I've read on lkml. People need to know the severity of problems they create, and almost never are those problems _intentional_ (which would still require one to accept that it's okay to be abusive as a form of "self defense"). Expecting people to change their behaviors, methods, or practices in order to avoid mistakes seems like a reasonable thing. This is how I've tried to fix my stupid mistakes when I encounter them or they're pointed out.

When someone cuts me off in traffic, I assume they're oblivious rather than malicious. If I drove 8 hours a day, I'm sure my resolve to accept and understand these mistakes would erode over time, but I still think it would be more productive to let them know it was uncool with a short honk rather than trying to ram them. :)

Linus replied to Sarah: "The thing is, different people act and react differently. On both sides. And I think we should recognize that and also *allow* for that. And sometimes it means, for example, that people interact primarily with certain people that they like more – because they are a better "fit."

I think we actually do it very naturally, simply because we are human, and this is how people interact in real life too. Sometimes we do it consciously – the way we have people at various companies that act as go-betweens – but most of the time we do it just because humans are all about social interactions and we don't even think about what we do and why.

For example, you work mostly through Greg. I don't think either of you *planned* it that way, but it's likely because you guys work well together.

See what I'm saying? People are different. I'm not polite, and I get upset easily but generally don't hold a grudge – I have these explosive emails. And that works well for some people. And it probably doesn't work well with you.

And you know what? That's fine. Not everybody had to get along or work well with each other. But the fact that it doesn't work with you doesn't make it "wrong."

This isn't all that different from working around language issues etc. by having certain people work as in-betweens on that front.

And where we differ is in thinking either side has to necessarily change. You think people need to act "nicer". While I think it's *natural* that people have different behavior – and different expectations. We all have issues somewhere and don't all like each other. There are certain people I refuse to work with, for example. They may be good engineers, but they just aren't people I can work with.

And, hey, I don't actually think we've personally even had any problems. And I realize that you may react very strongly and get nervous about us having problems, but realistically, do you actually expect to like all the other kernel engineers?

And equally importantly, not everybody has to like you, or necessarily think they have to be liked by you. OK?

So, as far as I'm concerned, the discussion is about "how to work together DESPITE people being different". Not about trying to make everybody please each other. Because I can pretty much guarantee that I'll continue cursing. To me, the discussion would be about how to work together despite these kinds of cultural differences, not about "how do we make everybody nice and sing songs sound the campfire".

Do you think you might be interested in *that* kind of discussion instead of the "you are abusing me" kind of discussion?

Because if you want me to "act professional", I can tell you that I'm not interested. I'm sitting in my home office wearing a bathrobe. The same way I'm not going to start wearing ties, I'm *also* not going to buy into the fake politeness, the lying, the office politics and backstabbing, the passive aggressiveness, and the buzzwords. Because THAT is what "acting professionally" results in: People resort to all kinds of really nasty things because they are forced to act out their normal urges in unnatural ways.

The discussion went on for quite a while and included more considerations of the pros and cons of professionalism. At one point, Sarah, Linus, and others started making plans for a panel discussion at the upcoming Kernel Summit. And some folks started discussing the boundaries of acceptable behavior for kernel developers.