A modern compression tool
Command Line – zstd

© Lead Image © Lembit Ansperi, 123RF.com
Like other modern replacement commands, zstd offers significantly faster file compression than the standard archiving tools.
Unix-like systems have been around long enough that replacements are available for time-honored commands. For instance, tree
is a substitute for ls
, while apt
has unified apt-get
and the most popular of its associated scripts. Since 2015, one of the most popular substitutes has been Zstandard (zstd
) [1], a compression tool that is a simplification of gzip
that is significantly faster than standard archiving tools such as tar
, zip
, and bzip
.
An LZ77 lossless data compression algorithm [2] gives zstd
its speed. Algorithms in the LZ77 family are a form of sliding window compression, so-called because they encode in chunks of customizable sizes as they copy and verify results. Chunks that are too small do nothing to increase speed, but ones that are too large start to slow compression because they take longer to verify. Finding the right balance maximizes the speed of an operation. To help obtain the most efficient setting, zstd
can build and use a dictionary of settings for different types of files (see the "Using a Dictionary" section).
To LZ77 compression, zstd
adds two modern, high-speed entropy encoders [3], which are used at the end of compression. Huffman, with its out-of-order (OoO) execution, offers high speed operations, while Finite State Entropy (FSE), a more recent entropy encoder, is designed to ensure the accuracy of compression at high speeds. Armed with LZ77 compression and these two entropy encoders, zstd
easily outperforms other archiving commands, especially when the command options are carefully chosen.
Closely resembling gzip
, zstd
has associated commands that are the equivalent of some common options. For example, unzstd
is the equivalent of zstd -d
, which decompresses files. However, zstd
differs from gzip
in several ways that make it more user-friendly. To start with, zstd
does not delete original files by default like gzip
does. Instead, the original files are only deleted if the --remove
option is added to the command. In addition, by default, zstd
displays progress notifications for single files and displays a help file when an error occurs, behaviors that are turned off when the --quiet
(-q
) option is used.
Command Structure
Integers in zstd
options can be specified in kilobytes (KiB, Ki , K, or KB) or megabytes (MiB, Mi, M, or MB). All these abbreviations should come immediately after the integer, with no space between them.
Files compressed with zstd
have a .zst
extension. Using a standard command structure, zstd
compresses a file with:
zstd INPUT-FILE
An archive will be created in the same directory as the original file (Figure 1). If you add the --verbose
(-v
) option, you can see the compression details. You can compress multiple files using either regular expressions or by listing them after the command in a space-separated list. To create the compressed file in some other location, use the following format:
zstd -INPUT-FILE -o=OUTPUT-FILE
If you want to store multiple files in the same archive, you need to add tar
to the command. The simplest use of tar
involves no zstd
options:
tar --zstd -cf DIRECTORY.tar.zst DIRECTORY
If you want to use any zstd
options, you need to use tar
's -I
option and place the zstd
command and any of its options inside quotes (Figure 2). For example:
tar -I 'zstd --ultra -22' -cf DIRECTORY.tar.zst Directory/
At the most basic level, all that is needed is often a single option: --compress
(-z
) or the command zstd
to compress files; -d
, --decompress
, --uncompress,
or the command unzstd
to decompress files. If no other options are given, zstd
uses its defaults, which might not be the most efficient choices but could be sufficient for general purposes.
If you want more control over compression, other options exist. Using --list
(-l
), you can view information about compressed files, such as their size, compression ratio, and checksum (Figure 3). Additional information can be had by adding the --verbose
(-v
) option. If you do not want to use the default compression ratio of 3
, you can specify -#=1-19
, with a lower number offering greater speed of operation and a high number greater compression. To give a sense of performance, at 3
, zstd
compresses an average Linux kernel slightly faster than gzip
, while at 19
it is 26 times slower. However, the biggest gain in speed is in decompression, for which an archive made at level 3
is about 25 percent faster than gzip
, and one made at 19
is about 60 percent faster than gzip
. From these figures, you can deduce that zstd
's priorities are the degree of compression and the speed of decompression. By contrast, the speed of compression seems of secondary importance.

The speed of operation can be affected by the other options used. Using --fast=NUMBER
, you can add faster speeds of -5
to 2
, while --ultra=NUMBER
allows compression of 20
or greater. However, note that --fast
and --ultra
are not automatically enabled because they may have unintended effects. For instance, --fast
is more likely to produce a file that will not fit on an external drive, while a file produced using --ultra
may take an unacceptably long time to decompress. You may prefer to use --adapt=min=NUMBER,max=NUMBER
to have zstd
set the compression level as it judges best. In the same way, --rsyncable
can be specified to make zstd
adjust to work more efficiently when rsync
is used to connect to remote machines. If zstd
is compiled with multithread support, still another way to affect speed is to add the option --threads=#NUMBER
(-T#NUMBER
), which, when set to 0
, prompts zstd
to detect the number of available cores. Alternatively, the --single-thread
option restricts zstd
to one thread.
By default, zstd
defaults to its own .zst
format. However, it can also be used with several other common compression formats, including .gzip
, .xz
, .lzma
, and .lz4
. These can be specified with the option --format=FORMAT
. Once a compressed file is created, zstd
runs an integrity test that compares it to the original file.
Benchmarking
With all the options that affect zstd
's performance, you may want to experiment to find the most efficient command structure for compressions that you do regularly. zstd
can be compiled with several options for benchmarking, although only the long help, available with the -H
option, lists them all. With -b#NUMBER
, you can test a compression level. Rather than test compression levels one at a time, you can specify a starting level with -e#NUMBER
and the end of a range with -b#NUMBER
. You can also set the time taken to benchmark in seconds with -i#SECONDS
or cut the archive into blocks, specifying -i#SIZE
, which can be useful if you secure files in a cloud by storing them in several pieces. Should you want to know exactly how long an operation takes, you can add the option --priority=rt
(real-time).
Using a Dictionary
If you compress the same type of file regularly, you could be able to squeeze more compression from zstd
by creating a dictionary. Even though the files might be small, you might still be able to compress or decompress more efficiently with a dictionary, because zstd
will not have to read each file separately. However, the effort to create a dictionary could fail because the sample size is too small for general patterns to be observed or because zstd
is unable to find a means to streamline operations with a particular file type. Generally speaking, an effective dictionary requires several thousand samples, although you might manage to produce a dictionary with less. The only way to know is to try.
To create a dictionary, create a directory and add to it files of the same format. Then use the command structure:
zstd --train ##SAMPLE-DIRECTORY/*
If an error occurs, zstd
will stop and suggest how to correct it (Figure 4). If a dictionary is created, its default name will be dictionary
, but you can give it a more specific name with -o FILE
. Other options are listed in the long form help (-H
) but are seriously under-documented. A sample dictionary is available [4], as well as an industry standard [5] and a guide for creating a third-party dictionary builder [6], although none seem to exist yet. However you create a dictionary, you can use it when either compressing or decompressing by adding the -D DICTIONARY
option.
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you've found an article to be beneficial.
News
-
Fedora 39 Beta is Now Available for Testing
For fans and users of Fedora Linux, the first beta of release 39 is now available, which is a minor upgrade but does include GNOME 45.
-
Fedora Linux 40 to Drop X11 for KDE Plasma
When Fedora 40 arrives in 2024, there will be a few big changes coming, especially for the KDE Plasma option.
-
Real-Time Ubuntu Available in AWS Marketplace
Anyone looking for a Linux distribution for real-time processing could do a whole lot worse than Real-Time Ubuntu.
-
KSMBD Finally Reaches a Stable State
For those who've been looking forward to the first release of KSMBD, after two years it's no longer considered experimental.
-
Nitrux 3.0.0 Has Been Released
The latest version of Nitrux brings plenty of innovation and fresh apps to the table.
-
Linux From Scratch 12.0 Now Available
If you're looking to roll your own Linux distribution, the latest version of Linux From Scratch is now available with plenty of updates.
-
Linux Kernel 6.5 Has Been Released
The newest Linux kernel, version 6.5, now includes initial support for two very exciting features.
-
UbuntuDDE 23.04 Now Available
A new version of the UbuntuDDE remix has finally arrived with all the updates from the Deepin desktop and everything that comes with the Ubuntu 23.04 base.
-
Star Labs Reveals a New Surface-Like Linux Tablet
If you've ever wanted a tablet that rivals the MS Surface, you're in luck as Star Labs has created such a device.
-
SUSE Going Private (Again)
The company behind SUSE Linux Enterprise, Rancher, and NeuVector recently announced that Marcel LUX III SARL (Marcel), its majority shareholder, intends to delist it from the Frankfurt Stock Exchange by way of a merger.