Most home users, and I dare say some system administrators, lack a backup policy. Their family pictures, music collections, and customer data files live on their hard drives and are never backed up to offline storage, only to be lost when the hard drive eventually crashes. A few users know to keep backups and regularly copy their files over to a safe storage medium. But even these conscientious people may find their strategy lacking when the time comes to recover from a system crash and they discover corrupted backup data. A successful backup strategy must involve checking for corrupted data.

Silent Data Corruption

The small number of users who keep copies of their important files only keep a single backup. Often, they use an external storage system, such as Tarsnap or a Nextcloud instance, periodically or continuously synchronizing the important files on their computers with the cloud. While a comfortable approach for end users, a single backup suffers a number of problems. Most importantly, single backups are vulnerable to silent data corruption.

Take for example a folder called Foals, which is full of pictures of happy young horses. My backup strategy consists of weekly copying the entire folder over to USB mass storage with a tool such as rsync [1]:

$ rsync -a --delete --checksum Foals/ /path/to/usb/

The rsync tool synchronizes the contents of /path/to/usb with the contents of Foals.

This strategy works until one of the pictures in Foals gets corrupted. Files get damaged for a number of reasons, such as a filesystem failing to recover properly after an unclean shutdown. Files also may be lost because of human error – you intend to delete Foals/10_foal.jpg but end up removing Foals/01_foal.jpg instead without realizing the mistake. If a file gets corrupted or lost and you don't detect the issue before the next backup cycle, rsync will overwrite the good copy in USB storage with bad data. At this point, all the good copies of the data cease to exist, destroyed by the very backup system intended to protect them.

To mitigate this threat, you can establish a long term storage policy for backups which involves saving your backup to a different folder each week within the USB mass storage. I could therefore keep a current backup of Foals in a folder called Foals_2022-01-30, an older backup in Foals_2022-01-23, and so on. When the backup storage becomes full, I could just delete the older folders to make room for the newer ones. With this strategy, if data corruption happens and it takes me a week to discover it, I may be able to dig up good copies of the files from an older snapshot (Figure 1). See the boxout "The rsync Time Machine" for instructions on how to set up this multi-week backup system.

Figure 1: A damaged image in the Foals folder results in corrupted data in the current backup directory (Foals_2022-01-30). Luckily, an undamaged version can be retrieved from an earlier backup (Foals_2022-01-23).

$ mkdir /mnt/Foals_2022-01-23
$ rsync -a Foals/ /mnt/Foals_2022-01-23

$ mkdir /mnt/Foals_2022-01-30
$ rsync -a --link-dest /mnt/Foals_2022-01-23 Foals/ /mnt/Foals_2022-01-30

Unfortunately, long term storage only works if the data corruption is discovered in time. If your storage medium only has room for storing four snapshots, a particular version of a file will only exist in the backup for four weeks. On the fifth week, the oldest snapshot will be deleted in order to make room for new copies. If the data corruption is not detected within this time window, the good copies of the data will be gone and you will no longer be able to retrieve them from a backup.

Solving for Silent Data Corruption

The first step in guaranteeing a good backup is to verify that you are backing up only uncorrupted data, which is easier said than done. Fortunately, a number of tools exist to help you preserve your data integrity.

Filesystems with checksum support (such as ZFS) offer a reasonable degree of protection against corruption derived from hardware errors. A checksum function takes data, such as a message or a file, and generates a string of text from it. As long as the function is passed the same data, it will generate the same string. If the data gets corrupted in the slightest, the generated string will be different.

ZFS [2], in particular, can verify if a data block is correct upon reading it. If it is not (e.g., as a result of a hard drive defect), ZFS either repairs the data block or throws an error for the user to see.

However, ZFS cannot protect data against human error: If you delete a file by accident with

rm Foals/01_foal.jpg

ZFS has no way of knowing this is a mistake instead of a legitimate operation. If a bogus image editor accidentally damages the picture using valid system calls, ZFS can not differentiate changes caused by software bugs from changes intended by the user. While ZFS is often praised as the ultimate guarantee for data integrity, its impressive capabilities fall short in my opinion.

Protection from Userspace

To verify that the data being backed up is correct, I suggest relying on userspace utilities. While many userspace programs are superb at locating damaged files, they are not easily executable from an arbitrary recovery environment. In a system crash scenario, you may find yourself using something like an obsolete SystemRescue DVD (perhaps from an old Linux Magazine) instead of your normal platform. In keeping with the KISS principle, you should choose userspace tools that are portable and easy to use from any platform.

If your distribution includes the GNU coreutils package (which the vast majority do), you need no fancy tooling.

Ideally, you should verify the files' integrity immediately before the backup is performed. The simplest way of ensuring a given file has not been modified, accidentally or otherwise, is to calculate its checksum and compare the result with the checksum it threw from a known good state (Figure 2). Thus, the first step towards protecting a given folder against corruption is by calculating the checksum of every file in the folder:

$ cd Foals
$ find . -type f ! -name '*.md5' -print0 | xargs -0 md5sum | sort -k 2 > md5sums_`date -I`.md5

Figure 2: Checksums can be used to locate files that have been modified, intentionally or otherwise.

(See the "Creating a Checksum" box for a more detailed explanation.)

find . -type f ! -name '*.md5' -print0

xargs -0 md5sum

sort -k 2

If a week later I want to verify that the files are fine, I can issue the same command to generate a new list. Then, it would be easy to check the differences between the state of the Foals folder on the previous date and the state of the Foals folder on the current date with the following command:

$ diff md5sums_2022-01-23.md5 md5sums_2022-01-30.md5

The diff command generates a list of differences between the two files, which will make it easy to spot which files have been changed, added, or removed from Foals (Figure 3). If a file has been damaged, this command will expose the difference.

Figure 3: The diff utility will compare the checksum files, but the output will be messy if there have been multiple changes between the current and the last backups.

Using diff is only practical if the dataset is small. If you are backing up several files, there are better ways to check that your data is not corrupted. For instance, you can use grep to list the entries that exist in the old checksum file but not in the new one. In other words: grep will list the files that have been modified or removed since the last time you performed a check:

$ grep -Fvf md5sums_2022-01-30.md5 md5sums_2022-01-23.md5

The -f md5sums_2022-01-30.md5 option instructs grep to treat every line of md5sums_2022-01-30.md5 as a target pattern. Any line in md5sums_2022-01-23.md5 that coincides with any of these patterns will be regarded as a match. The -F option forces grep to consider patterns as fixed, instead of as regular expressions. Therefore, for a match to be registered, it must be exact. Finally, -v inverts the matching: Only lines from md5sums_2022-01-23.md5 that match no pattern will be printed.

You can also list the files that have been added since the check was last run with the shell magic in Listing 1.

awk '{print $2}' < md5sums_2022-01-30.md5 | while read -r file; do
    if (! grep $file md5sums_2022-01-23.md5 > /dev/null); then
      echo "$file is new.";
    fi;
  done

With these tools, an integrity verification policy falls into place. In order to ensure you don't populate your backups with corrupted files, you must do the following:

• Generate a list of the files in the dataset and its checksums before initiating the backup.
• Verify this list against the list you generated at the last known good state.
• Identify which changes have happened between the last known good state and the current state, and check if they suggest data corruption.
• If the data is good, back up your files.

A great advantage of this method is that the checksum files can be used to verify the integrity of the backups themselves. For example, if you dumped the backup to /mnt/Foals_2022-01-23, you could just use a command such as:

$ cd /mnt/Foals_2022-01-23
$ md5sum --quiet -c md5sums_2022-01-23.md5

If any file was missing from the backup or had been modified, this command would reveal the issue right away.