Scramble at the Start

For many years, the Linux kernel started the init process as its first official act and assigned it a process ID of 1. From that point, the init process spawned all other processes running on the system, acting as a kind of "daemon-starting daemon" that initiated the processes necessary for getting the system working at the required runlevel. SysV init reads a list of configured processes in the /etc/inittab file and then shuffles through the runlevels (Figure 1).

Figure 1: Init goes through various runlevels in the SysV system, all of which are set by inittab, and executes some init scripts in each.

At each runlevel, init – supported by init scripts – launches various services and does not stop until it reaches the originally envisaged target runlevel for the system. In the case of a Debian system with a runlevel target of 1, for instance, the user has no network connection or graphical interface but can still use local applications. When you shut down the computer, init terminates any running processes to leave the system in a safe shutdown state.

The time-tested init has been around for years, but it has been showing some signs of age. Some promising alternatives have appeared recently to challenge the uniformity of the all-init Linux universe.

From simpleinit to SysV init

As the name suggests, SysV init did not come from a vacuum; it was instead based on the runlevel system in System V, AT&T's commercial Unix system from 1983. Miquel van Smoorenburg ported SysV init to Minix in 1992; Peter Ørbæk developed a version for Linux, which debuted in April 1992 in version 1.0 of admutil [1].

However, SysV Init was neither the first nor the only init system for Linux, which first used simpleinit [2], an init-based system that Ørbæk had developed with the help of Werner Almesberger. It was a little simpler than SysV init, because it did not use runlevels, among other things, and it resided in the poeigl archive.

At the end of 1992, after the release of admutil-1.4, a discussion ensued on Usenet [3] about the appropriate init system. The starting point was Ørbæk's plan to use the Getty tool to write the logout records. A user pointed out that you should probably leave this job to init in the best Unix fashion and mentioned his good experience with SysV init. Ørbæk agreed to make SysV init the default in the next version of his toolkit; simpleinit had already grown "too big" he said. Then, plans changed because SysV init author Miquel van Smoorenburg in December 1992 took over the development of the tool on Linux and released version 2.0 [4]. He had purchased a 386, installed Linux, and completely rebuilt the SysV init code.

The original Minix support was lost, but to make up for this, SysV init was added to the kernel and, soon after, became the boot system for Softlanding Linux, the first Linux distribution ever. Smoorenburg supervised SysV init for 12 years, until he stopped the project in 2004.

Rearguard Action

Smoorenburg left, and an abandoned project that was still being used remained at version number 2.86. Debian developer Petter Reinholdtsen unofficially handled bug reports for SysV init, added patches, and released patched versions, which was also used by Ubuntu.

For five years, this frustrating status continued until Reinholdtsen, in July 2009, took over the project along with some helpers [5] and summarily declared the newly released version 2.87dsf of SysV init to be the new upstream.

The change of maintainer was also accompanied by a change in the repository: Ever since, Savannah [6] has hosted the code. Reinholdtsen certainly saw the irony in this takeover, because some distributions were thinking by this time about switching to Ubuntu's new init system.

Upstart

Upstart, a brainchild of Ubuntu developer Scott James Remnant, was designed to iron out the disadvantages of SysV init, which at that point was getting on in years and had been developed with a completely different model of computer in mind: Desktop computers that were powered up and then ran for quite a while without major changes to the hardware. In 2006, when Remnant began to develop Upstart, USB sticks, mobile hard drives, and external USB devices were already in common use.

Upstart supports flexible, event-based starting and stopping of services and thus speeds up the boot process. SysV init, on the other hand, delays the boot process occasionally because dependent services are not available, so it does not execute the other scripts until after a timeout.

Upstart does not define boot targets for starting services; instead they react to certain events. By defining a job, the admin ensures that Apache starts when a network is available. If needed, an event can trigger multiple jobs that run in parallel. The syntax is quite simple compared with SysV init.

Fedora, which integrated Upstart version 10 [7] in late 2008, wanted especially to get rid of the numerous Bash scripts associated with SysV init.

Debian saw Upstart as the answer to an increasingly event-based kernel and wanted to solve the problems that a sequential boot process occasionally brought with it. Reinholdtsen proposed that Debian switch to Upstart [8], but it was not until the end of 2012 that a package landed in Debian unstable.

In Ubuntu, Upstart has been around since October 2006, but this had little effect initially, because the developers emulated the behavior of SysV init. It was not until the 9x versions that the Ubuntu project replaced numerous init scripts with Upstart jobs (Figure 2).

Figure 2: Excerpt from a boot chart generated under Ubuntu 9.04.

Upstart Reloaded

In 2011, Upstart suffered a temporary setback when Remnant left the project. He joined Google to work on Chrome OS, which also relies on Upstart. Over time, his contributions to the mailing list dwindled significantly, and his place was taken by Canonical's James Hunt, a former IBM employee. Hunt still maintains and releases new versions of Upstart and took some time out to answer a few of my questions about the project.

According to Hunt, currently about 20 people work on Upstart, mainly Canonical employees. The software is not only part of all the Ubuntu installations worldwide but also runs under Chrome OS and on older versions of Fedora. Communication is handled via the mailing list and IRC channel, and the code and bug reports are managed on Ubuntu's Launchpad platform.

One of the changes initiated by Hunt involves setting up a repository that builds Upstart versions daily. Code changes are automatically posted on the mailing list. In addition to Scan-Build and Smatch, Coverity Scan handles static code analysis. The aim is to rely more on automated testing in future.

One of the greatest conflicts in recent project history, in Hunt's view, is the debate about Canonical's Contribution Agreement, out of which the Contributor License Agreement (CLA) arose. This still allows Canonical to license the code for Upstart and other projects commercially, but the developers reserve the right to sublicense and distribute freely.

Systemd

Another surprise event also happened during Remnant's tenure: systemd. Lennart Poettering, the brains behind PulseAudio and Avahi, and a Red Hat developer by trade, announced the alternative systemd init system in 2010 on his blog [9]. As he writes in the email interview, he was already working privately on an init system called Babykit before the appearance of Upstart, but he postponed it after the Upstart announcement.

At Linux Plumbers Conference 2009, he and colleague Kay Sievers decided that Upstart was taking the wrong approach, because it expected user input up front to determine the minimum number of required services. So, Poettering dusted off his plans, took a close look at Solaris's SMF and Apple's launchd, and started in November 2009 to develop code for systemd.

Future Plans

Technically, systemd wants to create massively parallelized services by enabling them in different ways: through sockets, buses, devices, paths, or timers. Traditionally, daemons wait under Linux for the sockets of other daemons to start receiving before they send requests to them. Systemd wants to change this by activating all sockets before it starts its daemons. The requests land in the socket buffer, where the kernel manages them.

Once the daemon is online, it processes the queue. If a client expects a response to its synchronous request, it has to wait until the daemon is running, but this step does not block other services. Because all sockets are active, dependencies between daemons are unimportant.

The plans go further, however; for example, the systemd developers, mainly Sievers, Zbigniew Jedrzejewski-Szmek, and Poettering, want systemd to handle disk management at system startup time, which involves tasks such as mounting, health checks, space management or swapping.

Additionally, the team wants to track processes via the cgroups feature in the kernel to keep an eye on services and restart them if they crash – or at least collect crash information. The intent is also for systemd to create logfiles for daemons; bind to Luks, SELinux, and PAM; manage locales; set up consoles and keyboards; and much more. His project, says Poettering, has "extended the focus of systemd somewhat." A complete list of the tasks planned for systemd is available online [10] (Figure 3).

Figure 3: In a very long table (this is only a short excerpt), Lennart Poettering lists on his blog numerous functions that systemd will assume in the future.

For many developers, the feature richness of systemd raises concern. Poettering defends his plans: He sees the project as an opportunity to abolish the various proprietary solutions and scripts used by distributions by allowing systemd to integrate new Linux features intelligently and unify the start system.

Many Linux users still adhere to the old Unix mantra of "Do one thing and do it well." Opponents accuse him of making life difficult for smaller distros that want to do without systemd by linking a range of services to his init system [11]. A number of Gentoo developers founded eudev [12], a fork of udev, after systemd integrated the udev code into its own codebase.

Poettering finds such accusations exaggerated and refutes them in his blog, but his offensive manner polarizes, and the nonchalance with which he drops what is purportedly obsolete does not necessarily suit everybody.

The project itself is harmonious, however, writes Poettering. Seventeen people have commit permission for systemd; the repository contains patches by 400 people, and 600 people are on the mailing list. All use the mailing list for communication, along with several hack festivals a year, and he regularly phones his co-developer Kay Sievers.

The systemd code is managed on Git; bugs are tracked by the Fedora bugtracker and Freedesktop.org. Currently, systemd is used by Fedora, openSUSE, Arch Linux, Mandriva, Mageia, Tizen, and Mer, a fork of MeeGo.

Although systemd packages exist for Debian, it still uses SysV init. Also, Linux Mint has not changed, maybe because Ubuntu is sticking to Upstart. Mark Shuttleworth gave his reasons for this in his blog, saying that Upstart was carefully designed and tested, along with a clear task assignment [13].

Future

The developers of Upstart or systemd all justify their new developments with technical arguments. Whereas Upstart replaced the somewhat poorly maintained and no longer up-to-date SysV init, the systemd makers justify their new build by stating that they want to turn Upstart's concept around, putting it on its feet instead of its head.

The developers would probably deny any entrepreneurial reasons behind the decisions. Nevertheless, Canonical has a CLA for Upstart, so it can provide the necessary software under a proprietary license if needed, such as on mobile devices. Upstart is also a well-adjusted part of Ubuntu after a long period of development, and Canonical pays most of its developers. In other words, Canonical is unlikely to throw the init program overboard without good reason.

The developers of systemd, however, work for Red Hat. The company has not officially commented on the plans, but it certainly is practical to have the experts for the new system on your own payroll. Canonical will not change for the time being because of reasons stated. Fedora also has a unique feature that cannot be readily reconstructed because of the complexity of systemd. Whether this technology contest – which also affects Wayland and Mir, as well as Unity and Gnome – benefits or harms the Linux community is still uncertain. In particular, the supporters and developers of smaller distributions will be observing the conflict closely.

Some systems have already opted for systemd (such as Arch Linux), but others are just staying with SysV init (Debian) or using their own versions of the boot system, such as OpenRC (Gentoo). Thanks to the free licenses of the two boot systems, a retroactive change is also possible in a pinch.