Command Line – zstd

Unix-like systems have been around long enough that replacements are available for time-honored commands. For instance, tree is a substitute for ls, while apt has unified apt-get and the most popular of its associated scripts. Since 2015, one of the most popular substitutes has been Zstandard (zstd) [1], a compression tool that is a simplification of gzip that is significantly faster than standard archiving tools such as tar, zip, and bzip.

An LZ77 lossless data compression algorithm [2] gives zstd its speed. Algorithms in the LZ77 family are a form of sliding window compression, so-called because they encode in chunks of customizable sizes as they copy and verify results. Chunks that are too small do nothing to increase speed, but ones that are too large start to slow compression because they take longer to verify. Finding the right balance maximizes the speed of an operation. To help obtain the most efficient setting, zstd can build and use a dictionary of settings for different types of files (see the "Using a Dictionary" section).

To LZ77 compression, zstd adds two modern, high-speed entropy encoders [3], which are used at the end of compression. Huffman, with its out-of-order (OoO) execution, offers high speed operations, while Finite State Entropy (FSE), a more recent entropy encoder, is designed to ensure the accuracy of compression at high speeds. Armed with LZ77 compression and these two entropy encoders, zstd easily outperforms other archiving commands, especially when the command options are carefully chosen.

Closely resembling gzip, zstd has associated commands that are the equivalent of some common options. For example, unzstd is the equivalent of zstd -d, which decompresses files. However, zstd differs from gzip in several ways that make it more user-friendly. To start with, zstd does not delete original files by default like gzip does. Instead, the original files are only deleted if the --remove option is added to the command. In addition, by default, zstd displays progress notifications for single files and displays a help file when an error occurs, behaviors that are turned off when the --quiet (-q) option is used.

Command Structure

Integers in zstd options can be specified in kilobytes (KiB, Ki , K, or KB) or megabytes (MiB, Mi, M, or MB). All these abbreviations should come immediately after the integer, with no space between them.

Files compressed with zstd have a .zst extension. Using a standard command structure, zstd compresses a file with:

zstd INPUT-FILE

An archive will be created in the same directory as the original file (Figure 1). If you add the --verbose (-v) option, you can see the compression details. You can compress multiple files using either regular expressions or by listing them after the command in a space-separated list. To create the compressed file in some other location, use the following format:

zstd -INPUT-FILE -o=OUTPUT-FILE

Figure 1: Using the verbose option shows what the simplest zstd command structure does.

If you want to store multiple files in the same archive, you need to add tar to the command. The simplest use of tar involves no zstd options:

tar --zstd -cf DIRECTORY.tar.zst DIRECTORY

If you want to use any zstd options, you need to use tar's -I option and place the zstd command and any of its options inside quotes (Figure 2). For example:

tar -I 'zstd --ultra -22' -cf DIRECTORY.tar.zst Directory/

Figure 2: Compressing multiple files into one archive requires the use of the tar command.

At the most basic level, all that is needed is often a single option: --compress (-z) or the command zstd to compress files; -d, --decompress, --uncompress, or the command unzstd to decompress files. If no other options are given, zstd uses its defaults, which might not be the most efficient choices but could be sufficient for general purposes.

If you want more control over compression, other options exist. Using --list (-l), you can view information about compressed files, such as their size, compression ratio, and checksum (Figure 3). Additional information can be had by adding the --verbose (-v) option. If you do not want to use the default compression ratio of 3, you can specify -#=1-19, with a lower number offering greater speed of operation and a high number greater compression. To give a sense of performance, at 3, zstd compresses an average Linux kernel slightly faster than gzip, while at 19 it is 26 times slower. However, the biggest gain in speed is in decompression, for which an archive made at level 3 is about 25 percent faster than gzip, and one made at 19 is about 60 percent faster than gzip. From these figures, you can deduce that zstd's priorities are the degree of compression and the speed of decompression. By contrast, the speed of compression seems of secondary importance.

Figure 3: The --list option gives detailed information about compressed files. Here, compression is minimal due to compression level and file type.

The speed of operation can be affected by the other options used. Using --fast=NUMBER, you can add faster speeds of -5 to 2, while --ultra=NUMBER allows compression of 20 or greater. However, note that --fast and --ultra are not automatically enabled because they may have unintended effects. For instance, --fast is more likely to produce a file that will not fit on an external drive, while a file produced using --ultra may take an unacceptably long time to decompress. You may prefer to use --adapt=min=NUMBER,max=NUMBER to have zstd set the compression level as it judges best. In the same way, --rsyncable can be specified to make zstd adjust to work more efficiently when rsync is used to connect to remote machines. If zstd is compiled with multithread support, still another way to affect speed is to add the option --threads=#NUMBER (-T#NUMBER), which, when set to 0, prompts zstd to detect the number of available cores. Alternatively, the --single-thread option restricts zstd to one thread.

By default, zstd defaults to its own .zst format. However, it can also be used with several other common compression formats, including .gzip, .xz, .lzma, and .lz4. These can be specified with the option --format=FORMAT. Once a compressed file is created, zstd runs an integrity test that compares it to the original file.

Benchmarking

With all the options that affect zstd's performance, you may want to experiment to find the most efficient command structure for compressions that you do regularly. zstd can be compiled with several options for benchmarking, although only the long help, available with the -H option, lists them all. With -b#NUMBER, you can test a compression level. Rather than test compression levels one at a time, you can specify a starting level with -e#NUMBER and the end of a range with -b#NUMBER. You can also set the time taken to benchmark in seconds with -i#SECONDS or cut the archive into blocks, specifying -i#SIZE, which can be useful if you secure files in a cloud by storing them in several pieces. Should you want to know exactly how long an operation takes, you can add the option --priority=rt (real-time).

Using a Dictionary

If you compress the same type of file regularly, you could be able to squeeze more compression from zstd by creating a dictionary. Even though the files might be small, you might still be able to compress or decompress more efficiently with a dictionary, because zstd will not have to read each file separately. However, the effort to create a dictionary could fail because the sample size is too small for general patterns to be observed or because zstd is unable to find a means to streamline operations with a particular file type. Generally speaking, an effective dictionary requires several thousand samples, although you might manage to produce a dictionary with less. The only way to know is to try.

To create a dictionary, create a directory and add to it files of the same format. Then use the command structure:

zstd --train ##SAMPLE-DIRECTORY/*

If an error occurs, zstd will stop and suggest how to correct it (Figure 4). If a dictionary is created, its default name will be dictionary, but you can give it a more specific name with -o FILE. Other options are listed in the long form help (-H) but are seriously under-documented. A sample dictionary is available [4], as well as an industry standard [5] and a guide for creating a third-party dictionary builder [6], although none seem to exist yet. However you create a dictionary, you can use it when either compressing or decompressing by adding the -D DICTIONARY option.

Figure 4: Dictionary training is under-documented, but zstd does explain different ways to correct errors in what can be a complicated process.