Custom Repair Toolkit

Computers crash. Whether because of human error, software bugs, or hardware dying from old age, workstations and servers alike will eventually break. For this reason, every system administrator should have a digital first-aid kit to get broken systems back to working condition as soon as possible. For this task, I keep SystemRescue in my toolkit.

SystemRescue, formerly known as SystemRescueCd, is a Live Linux distribution designed to repair broken systems and handle a number of maintenance tasks. It includes tools to recover deleted files, repair broken boot managers, fix inconsistent filesystems, and more. The default set of installed programs is impressive (see Table 1 for a sample of included tools or visit the SystemRescue website [1] for the complete list). If one of your workstations or servers is rendered unbootable, you can simply boot your SystemRescue DVD or USB and have a full recovery environment available in a matter of minutes. With SystemRescue, you can repair damaged boot managers, partition tables, or filesystems. More importantly, you can extract your files from the failing computer and safely save them to a USB-attached hard drive or over a network.

Despite SystemRescue's impressive feature set, you will eventually find yourself in a situation where the default features fall short. SystemRescue's maintainer has recognized this possibility and made it easy to create your own SystemRescue solutions to suit your needs. This article explains how to create a custom SystemRescue environment.

The Need for Customization

Two real-life scenarios call for creating a custom SystemRescue-based distribution. The most common scenario involves needing a tool that SystemRescue does not include by default. Because SystemRescue is based on Arch, it has access to Arch's repositories. If you ever need a tool, you can simply install it in your rescue environment as you would on a regular operating system. In practical terms, you can use the pacman package manager from within a running SystemRescue environment. For example, if you prefer Emacs over the default Vim, you can install it with the following commands:

# pacman -Sy
# pacman -S emacs

Packages installed this way will be placed in a filesystem overlay in RAM. The packages will be available only to the rescue environment and won't touch the computer's hard drive that you are rescuing. On the downside, you will have to manually install these packages every single time you boot SystemRescue, because packages you install with pacman are not kept across reboots. Therefore, if you use an application often or expect to be performing maintenance tasks without access to Arch's repositories, you will need to include the desired packages in your custom version of SystemRescue.

Another scenario that calls for a custom SystemRescue involves a recovery environment that allows running custom scripts, performing tasks automatically, or executing services without the intervention of an administrator. For example, a server at your office breaks down and the responsible system administrator is out of town, requiring the sys admin to instruct an untrained co-worker to perform recovery via phone, which may not end well. An easier solution would be to have the sys admin fix the problem remotely by telling the co-worker to grab a recovery DVD from the shelf, put it in the server's DVD tray, reboot, and have SystemRescue offer a shell to the sys admin.

USB or DVD?

The easiest way to create a custom SystemRescue is to install it on a USB drive using a backing store [2], which is an area on the USB drive where changes to the running SystemRescue environment are written. You can install SystemRescue to a USB with a backing store, boot it, install all the packages you need, and configure all the desired scripts, and then shut it down. Your changes will be preserved across reboots this way.

While this approach is serviceable, I personally avoid using USB drives for recovery environments for a number of reasons. DVDs offer a reasonable guarantee that you will get a pristine recovery environment every time you boot, and there is value in that. (Your little brother is more likely to borrow a USB and replace your valuable data with B-movies.)

Because I want my rescue environment stored to read-only media, I can't use a backing store. Instead, I will create an ISO image containing my custom software and configurations and then burn it to a DVD.

Adding SSH Access and Persistent Keys

To illustrate how to include additional files and custom auto-started services on a SystemRescue DVD, I'll use the example of a bunch of servers in a remote office. One of the servers breaks, and the sys admin needs to make it boot SystemRescue and offer an SSH shell to connect to it from home.

While SystemRescue includes an SSH server, it is not accessible by default for two reasons: A proper password needs to be configured first, and the default firewall blocks incoming SSH connections. To remedy this, you need to create a custom SystemRescue image that does the following:

• Set a password to enable login over SSH
• Remove the firewall
• Use a persistent set of SSH fingerprints (because I want to verify the SSH server's identity when connecting to it remotely)

This custom SystemRescue image will then allow the senior sys admin to instruct a junior sys admin in the server room to boot SystemRescue, which will allow the sys admin to connect to the rescue environment without the junior admin performing any complex task.

To implement this setup, you need to download a copy of SystemRescue using a standard Linux system:

$ curl -L https://sourceforge.net/projects/systemrescuecd/files/sysresccd-x86/10.00/systemrescue-10.00-amd64.iso/download > systemrescue-10.00-amd64.iso

A helpful script by the SystemRescue creator lets you unpack the ISO image you just downloaded. You can use the script to extract the contents of the ISO file and modify them before repackaging. You can download the script as follows:

$ curl -L https://gitlab.com/systemrescue/systemrescue-sources/-/raw/main/airootfs/usr/share/sysrescue/bin/sysrescue-customize?inline=false> sysrescue-customize

Once the script, sysrescue-customize, has been downloaded, extract the image's contents with the following command:

$ ./sysrescue-customize --unpack --source=systemrescue-10.00-amd64.iso --dest=custom

Including Custom Configuration Files

Unpacking the ISO image will give you the contents shown in Figure 1.

Figure 1: An extracted SystemRescue image: Custom configuration files can be placed under the sysrescue.d directory, and SystemRescueModules under sysresccd.

Customary of Live CDs, SystemRescue's behavior can be modified at boot by using cheat codes, which are keywords and flags passed at the boot prompt before the system is booted (Figure 2). See the SystemRescue website for some supported cheat codes [3].

Figure 2: SystemRescue accepts cheat codes at boot time that enable or disable features. Among other things, cheat codes let you set a root password for the rescue environment and enable or disable SystemRescue's firewall.

Passing the cheat codes rootcryptpass and nofirewall enables SSH access upon boot (see the "Root Passwords in Configuration Files" box). You also need to pass loadsrm=y in order to load custom modules (which I will describe later). SystemRescue supports applying these options via YAML files (see Listing 1 for an example), which is much more elegant than hacking the bootloaders to bake them in. Save Listing 1 in the sysrescue.d folder of the image you just extracted. Beware, the YAML format is inflexible regarding indentation: Don't indent using the tab key; use the space bar instead.

$ python3 -c 'import crypt; print(crypt.crypt("$password",crypt.mksalt(crypt.METHOD_SHA512)))'

global:
  loadsrm: y
  nofirewall: true
  rootcryptpass: "$6$W9Lhg.3mr54J0mxe$4rudPwTGXLc9xgQotiE9HINide7NjO4lRzhZmsLOSwrrwXhSDr6BT2mhgfml0XvUfz6my8fULs.gm3u9THXVb0"

Modifying with cheat codes is not enough. SystemRescue, by default, creates a set of SSH keys every time it boots, and the keys are not preserved across reboots. If you want SystemRescue to use the same set of SSH keys every time, which lets you verify SystemRescue's fingerprints when connecting remotely, you need to pre-generate these keys and preload them. The ideal way to accomplish this is to create a SystemRescueModule (SRM), which is just a SquashFS filesystem that contains a set of files to be incorporated in your custom SystemRescue, similar to Tiny Core Linux extensions [4].

Generating a new set of keys is easy. On a different computer, boot SystemRescue with its default options. It will generate a set of SSH host keys under /etc/ssh at boot. You can use utilities included in SystemRescue to create an SRM that includes the SSH keys:

# mksquashfs /etc/ssh sshd_keys.srm -no-strip

This command will generate a module file called sshd_keys.srm, which you can transfer to the computer where you are doing the remaster. This file may be placed in the unpacked image's sysresccd directory. Keep in mind that this file's contents are sensitive; any evil entity that has access to these files could perform a man-in-the-middle (MITM) attack against your SSH sessions.

You can also use SRMs to include programs. Utilities for creating modules from Arch packages also exist. You can create a module including your desired programs by booting a clean SystemRescue DVD, installing the required applications using pacman, and then creating the SRM using cowpacman2srm:

# cowpacman2srm my_module.srm

The resulting SRM my_module.srm will include all the Arch packages you installed.

Putting Everything Together

Once you have placed your YAML configuration files under sysrescue.d and loaded your desired modules under sysresccd, you can create your custom ISO image with the command:

$ sysrescue-customize --rebuild --source=custom --dest=custom.iso

You can now burn this image to DVD using any of the usual tools.

Beyond Simple Modifications

SystemRescue features one advanced customization capability: the ability to download and execute scripts from the web upon boot. You can place a script on a web server, and a customized SystemRescue can be configured to download that script and execute it right after boot. Scripts may be fetched over HTTPS, and SystemRescue is capable of performing certificate validation for the TLS connection, so in theory the process is reasonably secure.

This groundbreaking feature allows SystemRescue to be used to provision services. You can create a custom SystemRescue image configured to download a script from a given URL and execute it. Then you can set a web server to deliver a different script depending on the IP from which the request is made. The end result is that you can use a SystemRescue DVD to boot a server, which will fetch the script and use it to download and install a set of packages and launch a set of services. However, if you boot the same image from a server assigned a different IP, it may be given a different script and end up downloading a different set of packages and executing a different set of services. This makes it possible to create a simple server provisioning system without effort. The only thing you need to take care of is configuring the network's DHCP server to assign the appropriate IP to each server.

Listing 2 shows a proof of concept web server written in Perl, based on an example provided by Perl's documentation [5]. It will return a Bash script if it receives a request from IP 192.168.3.50, and it will return a different one to queries made from any other address. Listing 3 displays an example script that creates an instantaneous FTP server which offers files stored in one of the machine's filesystems as anonymous downloads.

01 #!/usr/bin/perl
02 use IO::Socket qw(AF_INET AF_UNIX SOCK_STREAM SHUT_WR);
03
04 # Create a listening socket on all interfaces and port 8080
05 my $server = IO::Socket->new(
06   Domain => AF_INET,
07   Type => SOCK_STREAM,
08   Proto => 'tcp',
09   LocalHost => '0.0.0.0',
10   LocalPort => 8080,
11   ReusePort => 1,
12   Listen => 5,
13 ) or die "Can't open socket: $IO::Socket::errstr";
14
15 # Infinite loop that listens for requests and records the
16 # source IP from clients connecting to the server.
17 while (1) {
18   # Wait for connection
19   my $client = $server->accept();
20
21   # Obtain IP from the client.
22   my $client_address = $client->peerhost();
23
24   # If the IP is 192.168.3.50, read script_1.sh
25   # from the local filesystem and send it over.
26   # In any other case, return script_2.sh.
27
28   my $script;
29
30   if ( $client_address eq "192.168.3.50" ) {
31     $script = 'script_1.sh';
32   } else {
33     $script = 'script_2.sh';
34   }
35
36   open (INFILE, $script) || die "Couldn't open $script";
37   $client->send("HTTP/1.0 200 OK\r\n\r\n");
38   while (<INFILE>) {
39         $client->send($_);
40   }
41   close (INFILE);
42
43     # Notify the client of connection closure
44     $client->shutdown(SHUT_WR);
45 }
46
47 $server->close();

01 #!/bin/bash
02
03 # Mount a filesystem in the local machine
04 mount UUID=7bd471e8-cb11-474b-9901-e7ab6bfe2890 /srv/ftp
05
06 # Report an error and exit if the mount fails
07 if [ "$?" -ne "0" ]; then
08   echo "Could not mount on /srv/ftp"
09   exit 1
10 fi
11
12 # Run an ftp server in the background
13 systemctl enable ftpd.service
14 systemctl start ftpd.service

Listing 4 shows an example YAML configuration for SystemRescue to fetch a script from your web server and execute it at boot. Keep in mind that you may place as many YAML configuration files within SystemRescue's sysrescue.d as you like: They get loaded and read in canonical order, and all of their options applied. In case of conflicting options, the configuration of latter YAML files overwrites the ones from earlier files.

autorun:
  exec:
    200:
      url: "http://192.168.3.100:8080/"

Conclusion

Despite SystemRescue's usefulness for fixing broken systems and performing maintenance tasks, you may need to add tools. SystemRescue's creator has ensured you can add as many programs to your rescue environment as needed. This flexibility makes it easy to configure SystemRescue to start services upon boot and perform complex tasks. You can even use SystemRescue to perform server provisioning in simple environments.

SystemRescue advertises the ability to download YAML configuration files at boot time and load their configuration dynamically. I found during my tests that this feature does not work as advertised, because SystemRescue attempts to fetch the YAML configuration file from the web before the network interfaces are configured.

Figure 3: For safe storage on a DVD, you need to generate a hash of your password. Ensure you do this in a secure environment where nobody else can check your shell history or your list of running processes.

The Author

Rubén Llorente is a mechanical engineer who ensures that the IT security measures for a small clinic are both legally compliant and safe. In addition, he is an OpenBSD enthusiast and a weapons collector.