Linus Torvalds and Greg Kroah-Hartman discuss the state of Linux

Kernel View

Article from Issue 219/2019
Author(s):

Linux creator Linus Torvalds describes some recent challenges and opportunities for the Linux kernel – and stable branch maintainer Greg Kroah-Hartman joins in with thoughts on diversity and competition.

In what's becoming an annual event, I sat down with Linus Torvalds at the Open Source Summit, Vancouver (Canada) to talk about the state of Linux. We chatted for over an hour and Greg Kroah-Hartman (maintainer of the stable branch of the Linux Kernel) also joined us. We talked about a wide range of topics, including the sustainability of Linux, threats to Linux (both from inside and outside), the state of security in the kernel world, the success of Linux on the desktop, his outbursts on the Linux kernel mailing list, privacy, diversity, and much more.

Following is an edited version of my interview with Linus, along with a special appearance from Greg Kroah-Hartman.

[UCC:intervewer-intervewee]Swapnil Bhartiya:[/UCC] It's been almost 27 years since Linux came to life. How are things in the Linux world?

[UCC:intervewer-intervewee]Linus Torvalds:[/UCC] It's been normal. Our model of doing development really hasn't changed for over 10 years. It's standardized, and we continue to make releases about every two to three months. One of the big changes this year has been dealing with all the hardware bugs (Spectre and Meltdown [1]). We had to spend a fair amount of efforts for workarounds, and that's been somewhat stressful, but I'm hoping that we're starting to see the tail end of that.

[UCC:intervewer-intervewee]SB:[/UCC] Are hardware bugs more problematic than software bugs?

[UCC:intervewer-intervewee]LT:[/UCC] Hardware bugs have been pretty problematic for the kernel community. It's not because the workarounds have to be in the kernel – it's also because of the secrecy around them, which means that our normal workflow doesn't work.

It's not just about bringing everybody in and talking about it on the mailing list; it's about not being able to use the open infrastructure that we have for testing and validation. Whenever we try to do development without all that infrastructure, things become very painful. So, it has really been a big issue for the kernel community.

[UCC:intervewer-intervewee]SB:[/UCC] But there have been software bugs, too.

[UCC:intervewer-intervewee]LT:[/UCC] Yes, but that's nothing new. We have always had software bugs [2]. What may have changed is that, over the last few years, people have gotten way more vocal about security issues. It's not necessarily because they are now more serious about them; it could be because there are more processes in place for them. But we have always had software bugs. Hardware bugs are also not new, but they are different for us in the way they affect our workflow.

[UCC:intervewer-intervewee]SB:[/UCC] Linux played a critical role in establishing open source as a successful software development model. Do you see any risk that companies who have opened up their code might go back to a proprietary model to protect their investment?

[UCC:intervewer-intervewee]LT:[/UCC] From a development angle, I think people do realize how powerful it is to have open code and let people be part of the development process. So the only situation where, I think, people might go back to the old-fashioned proprietary model is for niche products. These could be those cases where you actually have a hard time finding that general population that wants to help you with code. And I don't think those cases matter that much.

I also think that if you worry about just open source in general, you should worry more about the whole experience. When you do everything on the web, and as a service, you can use all those services with open source applications in a regular Chromium browser, but you don't see what's going [on] behind the curtain. I think that's what a lot of people are worried about. You can always walk into a situation where the software you run on your machine may be open source, but what you use on that machine ends up running in a proprietary cloud system anyway.

[UCC:intervewer-intervewee]SB:[/UCC] There is a genuine reason to worry. My desktop is free, but my data and applications (which run in the cloud) are not. I am curious, personally, what do you care about more – just that the code has to be open or also about the business practices as well?

[UCC:intervewer-intervewee]LT:[/UCC] Personally, I only care about the code. When I say maybe there are people who worry about walled gardens and cloud providers who take ownership of your data, I am not one of those people. That's not what I actually care about. That's not what I do. What I do is code. What I care about is code. Even in some of those walled gardens and cloud services, they are running open source code. What these companies have is all the data on their users, which is obviously their bread and butter. Maybe it is a real issue for many people. I do understand that a lot of people care about open source because of the whole traditional "freedom" thing, but they will find it much more worrying when they know how much computing is happening inside of walled gardens.

[UCC:intervewer-intervewee]SB:[/UCC] You come from Europe, and they have very strict laws to protect privacy, which we don't see here in the US. Don't you care about your privacy?

[UCC:intervewer-intervewee]LT:[/UCC] I don't worry too much about my own privacy; I'm a very public person. I do all my work in public. I have very little that I care about.

At the same time, I think it's inevitable. There's going to be a lot of data. People are going to know a lot about you. People who worry about your DNA being available in various databases are right; it's going to be available in a lot of databases. Even if you are careful, the fact that you have family members who were not that careful means there will be a lot of data about you out there. I don't necessarily think that's a problem, but it could be problematic in certain circumstances.

I think at some point, the US will wake up to realize the problem and make strict data privacy rules. There will be some kind of protection, but let's face it, data is cheap and it's not going away. Whoever thinks that it will go away is deluding themselves.

[UCC:intervewer-intervewee]SB:[/UCC] While we are talking about privacy and looking at the ruckus your emails create on LKML, do you wish there was a private channel where top maintainers could discuss things candidly without prying eyes of bloggers who are looking for sensational stories?

[UCC:intervewer-intervewee]LT:[/UCC] Honestly, no. We do have private mailing lists for security issues. We also email each other or a group of people to discuss things. Usually, these private discussions are not about code; they are about things like maintainership issues.

But whenever such private conversations happen, I find them to be problematic, because then you don't have any archive that you can look back at. When you bring in new people, they miss the whole context. So, I am much happier having all the discussions in the open. It doesn't always work that way; it does mean that then when people get heated, everything they say becomes public, but I think that's healthier than closed room discussions.

[UCC:intervewer-intervewee]SB:[/UCC] Has there ever been any moment where you felt burnt out and said I will go and do something else?

[UCC:intervewer-intervewee]LT:[/UCC] I've never been burned out in that sense. Things do get stressful sometimes. The last merge window was fairly painful. There were two weeks of not just the regular work of a merge window, which is my busiest time anyway, but there were all other security related issues going on.

When I finally closed the window I said, "okay, I'm going to take a breather." But it's not like burnout. It's more like, "Okay, it's good. I have a calm week coming up. I have some travel. I will just step back for a couple of days and then I'm fine again."

[UCC:intervewer-intervewee]SB:[/UCC] But you don't travel much.

[UCC:intervewer-intervewee]LT:[/UCC] No. It was just this time. I don't travel very much for conferences. But my point is even normally I can just say, "okay, I'll do something else for three days," and that's fine. It's not where I feel this project is not worth it. It's more like, wow, okay, now I just need to have an extended weekend to take a break from all that stress.

Honestly, it's not that common. Most of the time, I am not stressed. We have a very efficient workflow. Even when I'm busy, it's seldom really stressful.

[UCC:intervewer-intervewee]SB:[/UCC] Okay, and we all want you to be around forever. That said, what have you done to ensure sustainability of Linux when we all are gone, given that you have set high standards when it comes to code quality?

[UCC:intervewer-intervewee]LT:[/UCC] We have a very big and a great community. When it comes to open source, the kernel is not just unusual: It's a complete outlier. As far as I know, there are no other open source projects that literally have a thousand people participating in every release. We have hundreds of maintainers and tens of pretty high-level maintainers. We have both depth and breadth in our development community. We have a very efficient workflow.

(Note: The interview was conducted before Linus took a break in the aftermath of the events surrounding the new Linux code of conduct. Greg Kroah-Hartman stepped in and everything moved forward normally, which implies that the Linux kernel does indeed have a very efficient workflow that would ensure the longevity of the project. Coincidentally, that was about the time when Greg Kroah-Hartman joined us).

[UCC:intervewer-intervewee]SB:[/UCC] We often hear that the kernel community has a mono culture; that it is not as diverse as the rest of our world? Does that worry you?

Linus looks at Greg.

[UCC:intervewer-intervewee]Greg Kroah-Hartman:[/UCC] There's two things here. One nice thing about Linux is that we've been doing this for so long that we all enjoy the experience and we don't go away. People may change companies and jobs, but they continue to work on the kernel. So we have the same crowd of mostly white males who got involved with the project at the early stage. At the same time, we are getting a lot of new people.

It's a big community, and we don't know everyone. I don't know the person whose patches I took. I met someone here who said that "you took my patch when I was 12 years old." In most cases, we don't know every committer; we don't know their age or their background. In itself, that's a good thing too. Anyone can submit a patch, regardless of who they are. But, we agree that diversity is important. It's really important to have people who bring different perspectives, different approaches to solving a problem. We are aware of the problem, and many people are working on it. We know we have a long way to go.

We participate in the Outreachy program [3], two times a year. Outreachy is really good because most of the people that come through Outreachy easily get jobs. A lot of kernel developers at ARM came through Outreachy. We participate in the Google Summer of Code, and most participants are from Asia or India. So, we are trying to do our best.

[UCC:intervewer-intervewee]SB:[/UCC] But you can do only so much; you are not a company where you go and hire people.

[UCC:intervewer-intervewee]LT:[/UCC] True. It's very easy to talk about diversity, but at the same time, it's really hard to find people. There are cultural and historical reasons. It's social. It's about who has access to computers. I think girls are discouraged from getting into the whole math and hard technical stuff, so there's a very small percentage coming in.

[UCC:intervewer-intervewee]GKH:[/UCC] It's also up to companies. They are the ones that assign people to work on the kernel. I was at IBM, and they have a large number of women at IBM who work on the kernel. They are participating, but sometimes we just don't realize it.

[UCC:intervewer-intervewee]SB:[/UCC] Is there anything that still worries you and keeps you awake at night?

[UCC:intervewer-intervewee]LT:[/UCC] The technology doesn't worry me. I do worry about the community and maintainership issues. There are certain people that I worry about burning out. These people are very central to the code. I am concerned about David Miller [4], who looks after the networking code. Sometimes I get the feeling that he may not be burning out, but he does get frustrated. I am also concerned about Greg as the last merge window was very stressful for all of us.

So there are areas in the kernel where I worry about maintainership, but on the whole, we've never actually had a big problem. We never really had huge maintainership issues, even though it's actually what worries me most.

The real problems we've had have been when we needed to change the flow of patches or when we changed from BitKeeper to Git. That was really very painful.

[UCC:intervewer-intervewee]GKH:[/UCC] I have talked to a bunch of maintainers here about how to do that job better. There are maintainers like Dan Williams [5] who are working on a document to help other maintainers.

I talked to someone who was a co-maintainer, and he says that's the best thing ever. Now he can go away for a week and lean on the other person. Working together really helps.

[UCC:intervewer-intervewee]SB:[/UCC] I was talking to Jonathan Corbet, the editor of LWN.net, and he said that what he worries about sometimes are the new projects like Zephyr and Fuchsia [6], which come from companies that don't understand the importance of the GPL license.

[UCC:intervewer-intervewee]LT:[/UCC] Companies do what companies do. If you don't like the GPL and choose another license, if you need to create a competitor to the kernel, go ahead do it. We can compete. We like competition.

[UCC:intervewer-intervewee]SB:[/UCC] It's good to hear that you want competition, because Linux, in a positive way, has a monopoly: Everyone is using it (laughter).

[UCC:intervewer-intervewee]GKH:[/UCC] It's not a monopoly, because it's free. Competition is good. Just look at the GCC and Clang communities [7]. They are both happy. It sped up their development process. They are now bouncing ideas at each other, and everyone benefits.

Zephyr is a different case, as it tried to solve a problem where there wasn't any good solution. But then if you look at what Microsoft is doing with Linux and Azure Sphere OS, they have slimmed down Linux even tinier for some ARM system. We are really getting close to that edge. Maybe we do need to run Linux on these tiny devices.

[UCC:intervewer-intervewee]LT:[/UCC] I don't follow those projects, but I have an opinion that most projects fail. It's true not just of projects; it's also true of branching, too. I don't get too upset when somebody makes a branch and says, I will rewrite something. I am fine with that. Go ahead and prove us wrong. The problem with "tinyfication" efforts – and there have been multiple – is that they usually end up doing a hatchet job. They just cut off pieces in ways that are not back mergeable. They go off for a few years, and they cut off all the pieces they don't care about. Then we are left with a situation where we would like to be more modular, and we'd like to be better at supporting a smaller model, but the way they did it makes it hard to merge it back. They chopped things without taking into account other cases.

[UCC:intervewer-intervewee]GKH:[/UCC] But there have also been cases where the branched project was so successful that communities worked together to make it mergeable, but it takes a lot of time and efforts.

[UCC:intervewer-intervewee]SB:[/UCC] Let's switch the gear from serious topics to a lighter one. Linux has conquered the world, but the desktop still remains an unrequited dream. I am curious if the Linux desktop really matters anymore, now that most of our workload has moved to the cloud?

[UCC:intervewer-intervewee]LT:[/UCC] I think it still matters. Not everybody uses the cloud; there are a lot of people who still use the desktop. I still wish we were better at having a standardized desktop that goes across all the distributions. It's not a kernel issue, but it's more of a personal annoyance to see how the fragmentation of the different vendors have held the desktop back a bit. I do see some progress with things like Flatpack.

[UCC:intervewer-intervewee]SB:[/UCC] True, but there is a new kind of fragmentation now – there is AppImage, Flatpack, and Snap (package tools). There is no single platform that developers can target?

[UCC:intervewer-intervewee]LT:[/UCC] Yes (shakes his head in disappointment). And maybe that's what Chromebook ends up doing – turning into a de facto standard for desktop applications. Once Chromebooks start running Debian packages, we will see. I am still optimistic after 25 years. It's going to be another few years, but the Linux desktop is not there yet.

Actually, Chromebooks are getting pretty good. The Crouton tool has been around for a while now, and it lets you run Linux distributions on a Chromebook. Now they have official support for running Debian apps. I have not actually used it yet, because I only use Chromebook for calendaring.

[UCC:intervewer-intervewee]SB:[/UCC] Why you don't use Chromebook for your work?

[UCC:intervewer-intervewee]LT:[/UCC] I would actually not mind having a Chromebook, but right now my main problem is even when you can run native Linux on Chromebooks with Crouton and official support for Debian, you can't do the kernel testing. That's what I care about. Chromebooks have reached the point where I could see myself using a Chromebook in a few years, but it is not there yet.

But, in general it seems that Chromebooks and Android are the paths towards Linux on desktop.

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy Linux Magazine

SINGLE ISSUES
 
SUBSCRIPTIONS
 
TABLET & SMARTPHONE APPS
Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

comments powered by Disqus
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters

Support Our Work

Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.

Learn More

News