Unification

Since 1993, the Filesystem Hierarchy Standard (HS) [1] has been the guideline for Unix-like directory structures. It requires the root directory partition to contain all the files the system needs for booting and mounting additional partitions.

This results in 14 directories or symbolic links to them: /bin, /boot, /dev, /etc, /lib, /media, /mnt, /opt, /run, /sbin, /srv, /tmp, /usr, and /var. Only the /opt, /usr, and /var folders can be located on other partitions. In addition, FHS considers the home directory to be optional, along with /root, /lib32, and /lib64.

In 2015, FHS was integrated into the Linux Standard Base (LSB). FHS is now maintained under the umbrella of the Linux Foundation.

Starting in 2012, several distributions have chosen to merge the /usr directories [2], thus modifying the FHS (and are therefore no longer LSB-compliant). Starting with Debian 10 Buster, Debian has now joined the /usr merge ranks.

Root Canal Treatment

In Debian 10 Buster, Debian makes changes to the root directory for new installations. The /usr merge aims to move the contents of the /bin, /sbin, /lib, and /lib64 directories to matching directories below /usr. For compatibility reasons, only symbolic links remain in the root directory. Thus, two directories for binary files and two directories for libraries are merged into one (Figure 1).

Figure 1: This is how the /usr merge is represented schematically in the file tree.

Debian comes to the /usr merge party relatively late compared to other distributions. As early as 2012, Fedora system developers Harald Hoyer and Kay Sievers implemented the /usr merge in Fedora 17 [3]; Solaris took the leap two years earlier with version 11. Arch Linux also started unifying the directories in 2012 and merging the /usr directories in 2013. Arch Linux also merged /bin and /sbin, which Debian doesn't intend to do at this time.

Lennart Poettering, creator of PulseAudio, Avahi, and systemd, has supported this change, because it facilitates stateless systems [4]. Following the /usr merge, it is possible to mount the immutable parts of the installed operating system with write protection, to update them atomically, or even to distribute them to several hosts or containers.

Before the /usr merge, historical partitioning was needed to mount /usr as a separate partition or across a network. The system mounts the /usr directory early on in the boot process, long before the initial RAM disk (initrd) [5].

Historically Conditioned

The roots of the historical division, where some directories are duplicated both directly below root and in /usr, date back to the beginning of Unix development in the 1970s. When Ken Thompson and Dennis Ritchie started developing Unix in 1969, storage space was limited.

In the beginning Thompson and Ritchie used floppy disks and then switched to a Digital Equipment Corporation [6] PDP-11 computer in about 1971. The PDP-11 already had a hard disk with a capacity of up to 512KB and supported a removable RK05 DECpack [7] magnetic media drive with 2.5MB, which was larger but far slower.

When the Unix root system grew beyond half a megabyte, the largest directory with the applications and tools was stored on the RK05-DECpack, which until then had contained only user data. Based on this practice, they named the mount point /usr (other sources claim that the acronym stands for "Unix System Resources"). Probably both are correct, because after the introduction of a third hard disk, the user data migrated there, making renaming meaningful.

Since the root directory was now distributed over two hard disks due to space constraints, a successful boot process required that everything on the first disk be kept accessible in order to boot the system to an extent where /usr could be mounted. This led to directories such as /bin, /lib, and /sbin being found both directly below the root and again below /usr.

Despite the advent of hard disks with sufficient capacity for the entire root system, this setup continued for many years and flowed into the FHS. Today, there is no reason to duplicate these directories. (More information on this topic can be found in an essay by Rob Landley [8]).

Penultimate

Canonical has delivered a unified /usr directory since Ubuntu 19.04. With the Debian Buster release, openSUSE will be the only major distributions that is not merging the /usr directories.

Debian started the migration in 2016, when the usrmerge package became available in the unstable branch [9]. usrmerge contains a Perl script that converts an existing system into one with a merged /usr directory. The developers also made a change to the debootstrap package so that it creates the required symbolic links itself before installing packages in a chroot.

Due to bugs that could not be fixed in time for Debian 9, the developers shifted the unification of / and /usr to Debian 10. However, during further development, other errors were made since packages created after unification would not run on unconverted systems.

Most of these bugs are now a thing of the past, or patches are available [10]. However, Debian developer Ian Jackson believes that the change could lead to more serious bugs and called on the Debian Technical Committee (TC) to postpone the /usr merge again [11]. In March, however, the TC rejected this request, so Debian 10 features the completed /usr merge [12].

"Testing" Reality

As an explanation, the TC discussed the advantages and disadvantages, as well as the future planning for Debian 11 Bullseye. People who install Debian 10 Buster or derivatives thereof can look forward to a system with a unified /usr directory and symbolic links of /bin, /sbin, /lib to the /usr/bin, /usr/sbin, and /usr/lib directories (Figure 2).

Figure 2: Before /usr merge (above): The /bin, /lib, /lib64, and /sbin directories reside directly below root and are duplicated in the /usr directory. After the conversion (below): The /bin, /lib, /lib32, /lib64, and /sbin directories exist as symbolic links to the physical directories in /usr.

The setup does not affect existing systems unless this is desired by the user. In this case, you just need to install the usrmerge package, which moves the corresponding directories and makes further adjustments. During the install, Debian explicitly asks for confirmation, since the change cannot be reversed.

After the change, check to see whether the /etc/dpkg/dpkg.cfg.d/usrmerge directory exists and if it contains package names. If it does, this means that there are still conflicts with these remaining packages. If it is empty or does not exist, you can remove usrmerge; otherwise wait until the list is empty (Figure 3).

Figure 3: The root directory in the terminal also reflects the new relationships and displays the symbolic links.

Unlike Debian, other distributions switched to a unified /usr directory on deadline. Debian's transition issues largely resulted from the decision by developers to leave older systems untouched and give users a choice. For an in-depth analysis of why Debian has had more problems with this migration than other distributions, see LWN.net [13].

Conclusions and Outlook

Switching to a merged /usr directory eliminates the historical duplication of directories and paves the way for modern techniques, such as stateless systems and atomic updates, thus facilitating the handling of snapshots. However, most desktop users are unlikely to notice any changes. The symbolic links to the subdirectories in /usr, on the other hand, will probably remain with us for some years. For Debian 11, the aim is for Debian packages that can be built on systems with and without the /usr merge and that still will work on the other system without restrictions.