Computers crash. Whether because of human error, software bugs, or hardware dying from old age, workstations and servers alike will eventually break. For this reason, every system administrator should have a digital first-aid kit to get broken systems back to working condition as soon as possible. For this task, I keep SystemRescue in my toolkit.

SystemRescue, formerly known as SystemRescueCd, is a Live Linux distribution designed to repair broken systems and handle a number of maintenance tasks. It includes tools to recover deleted files, repair broken boot managers, fix inconsistent filesystems, and more. The default set of installed programs is impressive (see Table 1 for a sample of included tools or visit the SystemRescue website [1] for the complete list). If one of your workstations or servers is rendered unbootable, you can simply boot your SystemRescue DVD or USB and have a full recovery environment available in a matter of minutes. With SystemRescue, you can repair damaged boot managers, partition tables, or filesystems. More importantly, you can extract your files from the failing computer and safely save them to a USB-attached hard drive or over a network.

Despite SystemRescue's impressive feature set, you will eventually find yourself in a situation where the default features fall short. SystemRescue's maintainer has recognized this possibility and made it easy to create your own SystemRescue solutions to suit your needs. This article explains how to create a custom SystemRescue environment.

The Need for Customization

Two real-life scenarios call for creating a custom SystemRescue-based distribution. The most common scenario involves needing a tool that SystemRescue does not include by default. Because SystemRescue is based on Arch, it has access to Arch's repositories. If you ever need a tool, you can simply install it in your rescue environment as you would on a regular operating system. In practical terms, you can use the pacman package manager from within a running SystemRescue environment. For example, if you prefer Emacs over the default Vim, you can install it with the following commands:

# pacman -Sy
# pacman -S emacs

Packages installed this way will be placed in a filesystem overlay in RAM. The packages will be available only to the rescue environment and won't touch the computer's hard drive that you are rescuing. On the downside, you will have to manually install these packages every single time you boot SystemRescue, because packages you install with pacman are not kept across reboots. Therefore, if you use an application often or expect to be performing maintenance tasks without access to Arch's repositories, you will need to include the desired packages in your custom version of SystemRescue.

Another scenario that calls for a custom SystemRescue involves a recovery environment that allows running custom scripts, performing tasks automatically, or executing services without the intervention of an administrator. For example, a server at your office breaks down and the responsible system administrator is out of town, requiring the sys admin to instruct an untrained co-worker to perform recovery via phone, which may not end well. An easier solution would be to have the sys admin fix the problem remotely by telling the co-worker to grab a recovery DVD from the shelf, put it in the server's DVD tray, reboot, and have SystemRescue offer a shell to the sys admin.

USB or DVD?

The easiest way to create a custom SystemRescue is to install it on a USB drive using a backing store [2], which is an area on the USB drive where changes to the running SystemRescue environment are written. You can install SystemRescue to a USB with a backing store, boot it, install all the packages you need, and configure all the desired scripts, and then shut it down. Your changes will be preserved across reboots this way.

While this approach is serviceable, I personally avoid using USB drives for recovery environments for a number of reasons. DVDs offer a reasonable guarantee that you will get a pristine recovery environment every time you boot, and there is value in that. (Your little brother is more likely to borrow a USB and replace your valuable data with B-movies.)

Because I want my rescue environment stored to read-only media, I can't use a backing store. Instead, I will create an ISO image containing my custom software and configurations and then burn it to a DVD.

Adding SSH Access and Persistent Keys

To illustrate how to include additional files and custom auto-started services on a SystemRescue DVD, I'll use the example of a bunch of servers in a remote office. One of the servers breaks, and the sys admin needs to make it boot SystemRescue and offer an SSH shell to connect to it from home.

While SystemRescue includes an SSH server, it is not accessible by default for two reasons: A proper password needs to be configured first, and the default firewall blocks incoming SSH connections. To remedy this, you need to create a custom SystemRescue image that does the following:

• Set a password to enable login over SSH
• Remove the firewall
• Use a persistent set of SSH fingerprints (because I want to verify the SSH server's identity when connecting to it remotely)

This custom SystemRescue image will then allow the senior sys admin to instruct a junior sys admin in the server room to boot SystemRescue, which will allow the sys admin to connect to the rescue environment without the junior admin performing any complex task.

To implement this setup, you need to download a copy of SystemRescue using a standard Linux system:

$ curl -L https://sourceforge.net/projects/systemrescuecd/files/sysresccd-x86/10.00/systemrescue-10.00-amd64.iso/download > systemrescue-10.00-amd64.iso

A helpful script by the SystemRescue creator lets you unpack the ISO image you just downloaded. You can use the script to extract the contents of the ISO file and modify them before repackaging. You can download the script as follows:

$ curl -L https://gitlab.com/systemrescue/systemrescue-sources/-/raw/main/airootfs/usr/share/sysrescue/bin/sysrescue-customize?inline=false> sysrescue-customize

Once the script, sysrescue-customize, has been downloaded, extract the image's contents with the following command:

$ ./sysrescue-customize --unpack --source=systemrescue-10.00-amd64.iso --dest=custom