File Compression for Modern Computing
Command Line – zstd
In an effort to meet modern computing needs, zstd offers a greater degree of compression at a faster compression rate, with unique options to enhance performance.
Many standard Linux tools have been around so long that second-generation tools are being developed to meet modern needs. For instance, Neovim is an update of the Vim text editor, and apt
is a rearrangement of the basic tools for apt-get
, the Debian package manager. Similarly, Zstandard (zstd
) [1] is a revision of compression tools like tar
and gzip
, except with higher degrees of compression at a faster rate. Additionally, zstd
includes several unique tools for enhanced performance, such as advanced compression features, compression levels and strategies, and dictionaries.
zstd
was written by Facebook employee Yann Collet and released in August 2016. Briefly, it is a lossless compression algorithm based loosely on the earlier LZ77 algorithm [2]. The command's syntax is deliberately similar to that of gzip
, down to variations on the basic command that are the equivalent of popular options. For example, zstdmt
is the same as zstd -T0
(use the same number of threads as detected cores), whereas unzstd
is the same as zstd -d
(decompress), and zstdcat
is the same as zstd -dcf
(decompress, force write to standard output, and overwrite without prompt).
The Basics
Getting started with zstd
is as simple as typing:
zstd FILE
Multiple files can be specified using a space-separated list. Unless you add --rm
as an option, the original file is not deleted. A progress bar is displayed as a single file is compressed; unless -q
is added to the command, an error produces a short help page. Unless otherwise specified, level 3 compression is used along with four threads (see below), and a data-integrity check is done on the original file before compression. The result is a file with the same name as the original file, but ending in .zst
(Figure 1).
To decompress, type:
zstd -d FILE
and a decompressed file is created without the .zst
extension. If you specify more than one .zst
file to uncompress, all the files are decompressed into a single file. Another option is to run --test
(-t
) to check the integrity of compressed files without creating or deleting any files.
In any operation, you can specify file size as needed in kilobytes (using KiB, Ki, K, or KB) or in megabytes (using MiB, Mi, M, or MB).
Basic Options
Most of zstd
's basic behaviors can be modified by options. To start, zstd
has both verbose (-v
) and quiet (-q
) modes for running the command. You can also use -o FILE
to specify any file name you want for an output file, placing the option after the original file's name, instead of directly after the basic command with the rest of the options. Additionally, if you are aware that a compressed file of the same name as the output file already exists, you can add --force
(-f
) to overwrite any file of the same name without confirming the operation first.
Several options help speed up commands. You can save time by turning off the integrity check during compression with --no-check
. The increased speed, of course, comes with the possibility that the compressed file might not be usable. A somewhat safer option to increase the speed is to enable --sparse
, which reduces the number of zeroes in the output file, which can add a couple more percentage points of compression when dealing with a text file. For a graphics file, however, --sparse
saves so little that it hardly seems worth using unless you are determined to save every bit of hard drive space possible.
As a recently created utility, zstd
can also be compiled to use multiple CPU cores to make compression faster. By default, only one core is used, but you can adjust the number with the option -T=NUMBER
(--threads=NUMBER
).
If the value is
, then zstd
will detect the number of cores and try to use all of them. Should the online help appear, you will know that the zstd
version you are using was compiled without threading.
Compression Levels, Strategies, and Advanced Options
zstd
approaches compression in two different ways. The more conventional tactic is to specify a specific compression level using --compress
(-z
or -#LEVEL
). The default level 3
can be overridden with any number from 1
to 19
, with 1
being the quickest and least compressed, and 19
the slowest and most compressed. To give a sense of the choices involved, a compression setting of 1
reduced the size of a 42MB .png
file by five percent in about six seconds, whereas a setting of 19
compresses the same file by just under eight percent in about 20 seconds. With a plain text file of 4,600 bytes, level 1 compression produces an archive file 55 percent smaller, wereas level 19 compression creates a file that is 59 percent smaller, both requiring only a few seconds. This difference between graphic and text files is typical.
You also have the option of adding the --ultra
option to enable the high, more memory intensive compression of levels 20-22. However, when used by themselves, the advanced compression levels are no more efficient than level 19 compression. To get the most from the ultra-compression levels, you need to experiment with the advanced options.
The advanced options for compression are defined in the option:
--zstd=OPTION=SETTING,OPTION=SETTING
The easiest to use is strategy=
(strat=
). This option can be completed with a number from
to 7
, in which
is the fastest and 7
the most compressed. Each strategy contains a number of methods and searches the file being compressed for an opportunity to use them. This search greatly increases both the time and the memory required to compress the file. However, the use of 7
can double the compression for a file.
Other advanced options for compression can override any of the options used in zstd
's compression algorithm. For instance, hashLog=BITS
sets the maximum number of bits for a hash table, making compression faster. Unfortunately, the man page lists the algorithm options with only a brief explanation of what they do, so most users will have to experiment blindly or else find other sources of information to understand what is being adjusted. Any algorithm option not specifically altered will use its default settings.
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
Canonical Bumps LTS Support to 12 years
If you're worried that your Ubuntu LTS release won't be supported long enough to last, Canonical has a surprise for you in the form of 12 years of security coverage.
-
Fedora 40 Beta Released Soon
With the official release of Fedora 40 coming in April, it's almost time to download the beta and see what's new.
-
New Pentesting Distribution to Compete with Kali Linux
SnoopGod is now available for your testing needs
-
Juno Computers Launches Another Linux Laptop
If you're looking for a powerhouse laptop that runs Ubuntu, the Juno Computers Neptune 17 v6 should be on your radar.
-
ZorinOS 17.1 Released, Includes Improved Windows App Support
If you need or desire to run Windows applications on Linux, there's one distribution intent on making that easier for you and its new release further improves that feature.
-
Linux Market Share Surpasses 4% for the First Time
Look out Windows and macOS, Linux is on the rise and has even topped ChromeOS to become the fourth most widely used OS around the globe.
-
KDE’s Plasma 6 Officially Available
KDE’s Plasma 6.0 "Megarelease" has happened, and it's brimming with new features, polish, and performance.
-
Latest Version of Tails Unleashed
Tails 6.0 is based on Debian 12 and includes GNOME 43.
-
KDE Announces New Slimbook V with Plenty of Power and KDE’s Plasma 6
If you're a fan of KDE Plasma, you'll be thrilled to hear they've announced a new Slimbook with an AMD CPU and the latest version of KDE Plasma desktop.
-
Monthly Sponsorship Includes Early Access to elementary OS 8
If you want to get a glimpse of what's in the pipeline for elementary OS 8, just set up a monthly sponsorship to help fund its continued existence.