Three in One

The Fedora project began in 2003. At the time, Red Hat had just canceled the old Red Hat Linux distro and was getting started with the commercial product that would be known as Red Hat Enterprise Linux (RHEL). The company wanted to continue to work with the open source community on a free Linux edition, and they started Fedora as a community-driven upstream contributor to RHEL.

Because Linux was open source, however, Red Hat could not distribute RHEL without making the source code available to others in the Linux community. A new category of distros emerged around the practice of removing trademarked material from RHEL and then compiling the source code to create a new, independent distribution. The most famous and most popular of these RHEL spin-offs was CentOS, which existed as an independent project for several years. Then in 2014, Red Hat surprised many experts by announcing that it would hire the CentOS developers and take over sponsorship of CentOS, thus making CentOS an in-house Red Hat project that was very close to RHEL but available for no cost.

The result is a constellation of three Linux distributions – two community-based and one commercial, each with a slightly different role, but all fitting together somehow to achieve a greater purpose for Red Hat. Swap sat down with Jim Perrin, Community Platform Engineering Manager, CentOS/Fedora at Red Hat, to sort through the complicated relationships of Fedora, CentOS, and RHEL.

Jim Perrin has been involved with the CentOS project for over 14 years and has spent eight years as a member of the CentOS Governing Board. When Red Hat acquired CentOS back in 2014, Perrin joined Red Hat as an employee. Perrin currently works at Red Hat as Community Platform Engineering Manager of CentOS and Fedora. He seems to have a complicated role. His primary focus is on the CentOS and Fedora communities, but as an engineering manager, he is also responsible for Fedora and CentOS infrastructure. In the past, he has been involved with the 64-bit ARM port for CentOS.

CentOS vs. Fedora: Friend or Foe

You could think of CentOS as a project that's eating RHEL's lunch just the way Linux Mint is eating Ubuntu's dinner. But the reality is, despite it being a clone of RHEL, CentOS maintains a very friendly and healthy relationship with its cousin Fedora.

"We tried to work as closely with the Fedora community as we could," said Perrin. "One of the most successful Fedora projects is the Extra Packages for Enterprise Linux (EPEL) repository for CentOS. Some of that code comes from Fedora. Most of the people who sit on the EPEL council are CentOS representatives, but the project itself belongs in Fedora. So that's one of the many areas where the two projects are collaborating with each other."

CentOS and Fedora are also sharing their Continuous Integration (CI) infrastructure. At the moment the CI initiative is more about continuous testing and not continuous integration, although they are working towards the goal of CI. The infrastructure helps in testing every time there are any updates. "We have a suite of tests for different projects that we run to try and validate as much as we can," he said. The project started at CentOS some two years ago, and today even Fedora is using some of that infrastructure for their test cases.

I Am Your Father

The popular belief is that CentOS and Fedora are like cousins, but there is more to the story. They also share code with each other. Rationally, we draw a conclusion that RHEL is upstream for CentOS and Fedora is upstream for RHEL, so technically Fedora is also upstream for CentOS. But things are not as they appear. "Both Fedora and CentOS are upstream for RHEL," said Perrin.

CentOS has a system of Special Interest Groups (SIGs) – groups that work on new topics or features to add functionality to CentOS that it doesn't inherit directly from RHEL. The SIG system allows for the upstream development of products that target enterprise Linux. In some cases, it is better to develop new community-drive features for RHEL through the CentOS SIG system than through Fedora. If you want to develop something for the current version of RHEL, using Fedora means you are running Linux kernel 4.14 or 4.15, whereas RHEL 7.x (and CentOS) are running a 3.10 based kernel. In that case, "Using Fedora as a development platform to target RHEL isn't going to work," said Perrin.

The better course would be to do that upstream development on CentOS, because it has an equivalent kernel version. As a result, there is a dual – downstream and upstream – relationship between CentOS and RHEL. "Fedora is entirely upstream in all ways, as it should be, but on the CentOS side, we are upstream for some things and downstream for others," said Perrin.

Whereas RHEL development is fully controlled by the company, anyone can come and contribute to Fedora and CentOS. A good example is the Xen virtualization project that has not been part of RHEL since version 5. But Xen still has an active community, and they are still contributing as a member of the CentOS virtualization SIG. "They still provide us new code, and we still build it. They control their release cycle for what they're putting out, and they're valued members of the community. Anybody can come and contribute. The code is not controlled by Red Hat," said Perrin.

CentOS creates an opportunity for enterprise developers to target RHEL. One might draw the conclusion that enterprise Linux developers should target CentOS instead of Fedora, but that would be a mistake.

"If someone wants to contribute code that's targeted at enterprise users of RHEL, they should look at both options because what happens in Fedora is what you see in the next version of RHEL. Fedora is the upstream for all the community," reminded Perrin.

If somebody is trying to target a specific version of enterprise Linux and looks at CentOS, they must consider what happens to that work when the next version of RHEL or CentOS comes out. If you skip Fedora, that code won't be in RHEL or CentOS. So as tempting as it may sound, developers should look at the larger picture. "If they're only focusing on what exists today, they're ignoring what's coming tomorrow."

But it's complicated. Targeting CentOS is justifiable in a case when someone is looking at those layered projects that focus on enterprise Linux. A good example is RDO, OpenStack customized for CentOS. The life cycle for Fedora and RDO is reasonably short. Running that code on Fedora is not wise. It makes more sense to target the code for CentOS, which has a longer life cycle; it's supported for years and not months.

He warned again that developers must keep an eye on what's coming next instead of focusing on the 3.10 kernel that exists in CentOS 7/RHEL 7 today.

Changing Gears: Going Fast with Project Atomic

Earlier this year, Red Hat acquired CoreOS, the company that pioneered the concept of atomic updates. Red Hat responded to CoreOS by releasing RHEL Atomic Host; SUSE came out with SUSE CaaS Platform, and Canonical introduced Ubuntu Core. As more and more workload is becoming containerized, is it possible that atomic distros will replace traditional distros? Is it an evolution of operating systems, just the way virtualization and containerization are the evolution of application development and delivery?

"I don't think you can equate those two [traditional distros and atomic distros]," said Perrin. It's the same code that powers Atomic, RHEL, CentOS, and Fedora. The difference lies in the delivery and update mechanism.

When we are talking about a certain scale of deployment across 40 to 50 thousand servers, running yum update or dnf update across a network of that size may lead to some failure. "That's where the atomic method makes sense, because if something fails, it rolls back to the previous version. It's automatic. You don't end up with a corrupted Yum database and other problems that keep sys admins awake at night,"

That also doesn't mean that the bare metal approach with existing CentOS or RHEL customers has any less value. "It's just a different method of delivering the same code to deal with how the world of technology moves forward in a containerized world with different things," he said. "In some case, atomic makes sense, whereas in many others cases, traditional approaches are more appropriate. Atomic doesn't mean RHEL, CentOS, or Fedora are any less important. It's just a different method of delivering the same code."