A modern compression tool
Command Line – zstd

© Lead Image © Lembit Ansperi, 123RF.com
Like other modern replacement commands, zstd offers significantly faster file compression than the standard archiving tools.
Unix-like systems have been around long enough that replacements are available for time-honored commands. For instance, tree
is a substitute for ls
, while apt
has unified apt-get
and the most popular of its associated scripts. Since 2015, one of the most popular substitutes has been Zstandard (zstd
) [1], a compression tool that is a simplification of gzip
that is significantly faster than standard archiving tools such as tar
, zip
, and bzip
.
An LZ77 lossless data compression algorithm [2] gives zstd
its speed. Algorithms in the LZ77 family are a form of sliding window compression, so-called because they encode in chunks of customizable sizes as they copy and verify results. Chunks that are too small do nothing to increase speed, but ones that are too large start to slow compression because they take longer to verify. Finding the right balance maximizes the speed of an operation. To help obtain the most efficient setting, zstd
can build and use a dictionary of settings for different types of files (see the "Using a Dictionary" section).
To LZ77 compression, zstd
adds two modern, high-speed entropy encoders [3], which are used at the end of compression. Huffman, with its out-of-order (OoO) execution, offers high speed operations, while Finite State Entropy (FSE), a more recent entropy encoder, is designed to ensure the accuracy of compression at high speeds. Armed with LZ77 compression and these two entropy encoders, zstd
easily outperforms other archiving commands, especially when the command options are carefully chosen.
Closely resembling gzip
, zstd
has associated commands that are the equivalent of some common options. For example, unzstd
is the equivalent of zstd -d
, which decompresses files. However, zstd
differs from gzip
in several ways that make it more user-friendly. To start with, zstd
does not delete original files by default like gzip
does. Instead, the original files are only deleted if the --remove
option is added to the command. In addition, by default, zstd
displays progress notifications for single files and displays a help file when an error occurs, behaviors that are turned off when the --quiet
(-q
) option is used.
Command Structure
Integers in zstd
options can be specified in kilobytes (KiB, Ki , K, or KB) or megabytes (MiB, Mi, M, or MB). All these abbreviations should come immediately after the integer, with no space between them.
Files compressed with zstd
have a .zst
extension. Using a standard command structure, zstd
compresses a file with:
zstd INPUT-FILE
An archive will be created in the same directory as the original file (Figure 1). If you add the --verbose
(-v
) option, you can see the compression details. You can compress multiple files using either regular expressions or by listing them after the command in a space-separated list. To create the compressed file in some other location, use the following format:
zstd -INPUT-FILE -o=OUTPUT-FILE
If you want to store multiple files in the same archive, you need to add tar
to the command. The simplest use of tar
involves no zstd
options:
tar --zstd -cf DIRECTORY.tar.zst DIRECTORY
If you want to use any zstd
options, you need to use tar
's -I
option and place the zstd
command and any of its options inside quotes (Figure 2). For example:
tar -I 'zstd --ultra -22' -cf DIRECTORY.tar.zst Directory/
At the most basic level, all that is needed is often a single option: --compress
(-z
) or the command zstd
to compress files; -d
, --decompress
, --uncompress,
or the command unzstd
to decompress files. If no other options are given, zstd
uses its defaults, which might not be the most efficient choices but could be sufficient for general purposes.
If you want more control over compression, other options exist. Using --list
(-l
), you can view information about compressed files, such as their size, compression ratio, and checksum (Figure 3). Additional information can be had by adding the --verbose
(-v
) option. If you do not want to use the default compression ratio of 3
, you can specify -#=1-19
, with a lower number offering greater speed of operation and a high number greater compression. To give a sense of performance, at 3
, zstd
compresses an average Linux kernel slightly faster than gzip
, while at 19
it is 26 times slower. However, the biggest gain in speed is in decompression, for which an archive made at level 3
is about 25 percent faster than gzip
, and one made at 19
is about 60 percent faster than gzip
. From these figures, you can deduce that zstd
's priorities are the degree of compression and the speed of decompression. By contrast, the speed of compression seems of secondary importance.

The speed of operation can be affected by the other options used. Using --fast=NUMBER
, you can add faster speeds of -5
to 2
, while --ultra=NUMBER
allows compression of 20
or greater. However, note that --fast
and --ultra
are not automatically enabled because they may have unintended effects. For instance, --fast
is more likely to produce a file that will not fit on an external drive, while a file produced using --ultra
may take an unacceptably long time to decompress. You may prefer to use --adapt=min=NUMBER,max=NUMBER
to have zstd
set the compression level as it judges best. In the same way, --rsyncable
can be specified to make zstd
adjust to work more efficiently when rsync
is used to connect to remote machines. If zstd
is compiled with multithread support, still another way to affect speed is to add the option --threads=#NUMBER
(-T#NUMBER
), which, when set to 0
, prompts zstd
to detect the number of available cores. Alternatively, the --single-thread
option restricts zstd
to one thread.
By default, zstd
defaults to its own .zst
format. However, it can also be used with several other common compression formats, including .gzip
, .xz
, .lzma
, and .lz4
. These can be specified with the option --format=FORMAT
. Once a compressed file is created, zstd
runs an integrity test that compares it to the original file.
Benchmarking
With all the options that affect zstd
's performance, you may want to experiment to find the most efficient command structure for compressions that you do regularly. zstd
can be compiled with several options for benchmarking, although only the long help, available with the -H
option, lists them all. With -b#NUMBER
, you can test a compression level. Rather than test compression levels one at a time, you can specify a starting level with -e#NUMBER
and the end of a range with -b#NUMBER
. You can also set the time taken to benchmark in seconds with -i#SECONDS
or cut the archive into blocks, specifying -i#SIZE
, which can be useful if you secure files in a cloud by storing them in several pieces. Should you want to know exactly how long an operation takes, you can add the option --priority=rt
(real-time).
Using a Dictionary
If you compress the same type of file regularly, you could be able to squeeze more compression from zstd
by creating a dictionary. Even though the files might be small, you might still be able to compress or decompress more efficiently with a dictionary, because zstd
will not have to read each file separately. However, the effort to create a dictionary could fail because the sample size is too small for general patterns to be observed or because zstd
is unable to find a means to streamline operations with a particular file type. Generally speaking, an effective dictionary requires several thousand samples, although you might manage to produce a dictionary with less. The only way to know is to try.
To create a dictionary, create a directory and add to it files of the same format. Then use the command structure:
zstd --train ##SAMPLE-DIRECTORY/*
If an error occurs, zstd
will stop and suggest how to correct it (Figure 4). If a dictionary is created, its default name will be dictionary
, but you can give it a more specific name with -o FILE
. Other options are listed in the long form help (-H
) but are seriously under-documented. A sample dictionary is available [4], as well as an industry standard [5] and a guide for creating a third-party dictionary builder [6], although none seem to exist yet. However you create a dictionary, you can use it when either compressing or decompressing by adding the -D DICTIONARY
option.
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Direct Download
Read full article as PDF:
Price $2.95
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Find SysAdmin Jobs
News
-
OpenMandriva Lx 23.03 Rolling Release is Now Available
OpenMandriva "ROME" is the latest point update for the rolling release Linux distribution and offers the latest updates for a number of important applications and tools.
-
CarbonOS: A New Linux Distro with a Focus on User Experience
CarbonOS is a brand new, built-from-scratch Linux distribution that uses the Gnome desktop and has a special feature that makes it appealing to all types of users.
-
Kubuntu Focus Announces XE Gen 2 Linux Laptop
Another Kubuntu-based laptop has arrived to be your next ultra-portable powerhouse with a Linux heart.
-
MNT Seeks Financial Backing for New Seven-Inch Linux Laptop
MNT Pocket Reform is a tiny laptop that is modular, upgradable, recyclable, reusable, and ships with Debian Linux.
-
Ubuntu Flatpak Remix Adds Flatpak Support Preinstalled
If you're looking for a version of Ubuntu that includes Flatpak support out of the box, there's one clear option.
-
Gnome 44 Release Candidate Now Available
The Gnome 44 release candidate has officially arrived and adds a few changes into the mix.
-
Flathub Vying to Become the Standard Linux App Store
If the Flathub team has any say in the matter, their product will become the default tool for installing Linux apps in 2023.
-
Debian 12 to Ship with KDE Plasma 5.27
The Debian development team has shifted to the latest version of KDE for their testing branch.
-
Planet Computers Launches ARM-based Linux Desktop PCs
The firm that originally released a line of mobile keyboards has taken a different direction and has developed a new line of out-of-the-box mini Linux desktop computers.
-
Ubuntu No Longer Shipping with Flatpak
In a move that probably won’t come as a shock to many, Ubuntu and all of its official spins will no longer ship with Flatpak installed.