File Compression for Modern Computing

Compression Dictionaries

A dictionary is a file that stores the compression settings for small files. A dictionary is assembled from a group of typical small files that contain similar information, preferably over 100 files. For greatest efficiency, their combined size should be about one hundred times the size of the dictionary produced from them. If the files used are fewer or smaller in size than recommended, zstd will display a warning but still allow the dictionary to be created (Figure 2).

Figure 2: A dictionary of related files can be specified to help zstd compress other files more efficiently. zstd outputs warnings, but leaves users to select the relevant files.

To create a dictionary, use the command:

zstd --train FILES

The dictionary will be saved with the default name dictionary, and a default size of 112,640KB. To give the dictionary its own name, add the name to the train option; for example, a dictionary called quick would be named using the option --train-quick. You can also force the dictionary to use the most compressed files by specifying the number of files to use after the name; for example,

--train-quick=k=NUMBER OF FILES

A specific size can be added with the option --maxdict=SIZE, and a specific ID with --dicID=NUMBER, which makes communication with the dictionary faster than relying on the name. To use a dictionary, add the option -D FILE to the command. Nothing in the output will indicate that the dictionary is in use.

In general, the smaller the file, the greater the improvement in compression. According to the man page, a dictionary can only increase the compression of a 64KB file by 10 percent, compared with a 500 percent improvement for a file of less than 1KB.

Benchmarking

To use zstd to its full potential requires experimentation. To use the advanced compression options, you probably will need to research the compression algorithm. However, with the methods listed here, zstd is sure to be efficient.

But how efficient? More particularly, how does zstd compare with other compression tools? zstd provides its own answer with a small selection of benchmarking options. To start, you can use the option -bLEVEL to set the compression level to test. Alternately, you can use -bLEVEL to indicate the start of a range of compression levels and use -eLEVEL to indicate the end of the range (Figure 3). You can also change the default of three seconds for the length of the testing with -iSECONDS. Of course, you can also make notes as you gain experience with zstd.

Figure 3: A benchmark of compression rate and compression speed for the same file at different levels of compression.

zstd has been released recently enough that, in many ways, it is still an expert's tool. However, although the documentation can be spotty for the advanced features, there is still enough to make zstd an alternative tool for any level of user, especially those who want a compression tool designed for modern computing.

The Author

Bruce Byfield is a computer journalist and a freelance writer and editor specializing in free and open source software. In addition to his writing projects, he also teaches live and e-learning courses. In his spare time, Bruce writes about Northwest coast art. You can read more of his work at http://brucebyfield.wordpress.com

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy Linux Magazine

SINGLE ISSUES
 
SUBSCRIPTIONS
 
TABLET & SMARTPHONE APPS
Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

  • BorgBackup

    In Star Trek, the Borg adds individuals to its collective consciousness, an apt metaphor for any backup application that stores individual files in an archive. BorgBackup creates folder repositories for multiple archives, making it an especially befitting description of assimilation.

  • Command Line: gzip, bzip2, tar

    A short command is all it takes to pack your data or extract it from an archive.

  • FLAC: The premier digital audio codec

    With a little effort, you can create digital audio files with CD quality sound.

  • Resettable Lab Accounts

    Let your lab users play, and restore the original settings for the next login.

  • Command Line: Archives

    Gzip and bzip2 not only compress files, they also provide lean and powerful tools for viewing, searching, and comparing text files.

comments powered by Disqus

Direct Download

Read full article as PDF:

Price $2.95

News