Building Frames
Building Frames
FFmpeg is good not only for converting and fusing videos together, it can also generate streams on the fly, which you can then use for compositing and effects.
The casual user might only use the FFmpeg multimedia framework for converting from one audio or video format to another, but FFmpeg can do much more than that. I'll take a look at one of FFmpeg's most powerful secret weapons: the lavfi
virtual device.
Using lavfi
FFmpeg's lavfi
(short for libavfilter) virtual device sounds more complicated than it really is. Instead of using prerecorded video or audio files as streams, lavfi
lets you create streams out of thin air. You can use these streams on the fly and combine them with clips and other dynamically generated streams (e.g., from a webcam or microphone) to create your output file.
Maybe it is better explained with an example:
ffmpeg -f lavfi -i color=c=red:size=640x480:rate=30 -t 10 red.webm
Most basic ffmpeg
instructions you have seen probably have at least one input file, but the instruction above has none because "input" is coming from the lavfi
virtual device, not from a file. The content the virtual device generates is described by what goes after the -i
parameter, that is, color=c=red:size=640x480:rate=30
. Broken up, color
tells FFmpeg what sort of stream it should expect – in this case, a simple color video stream. After the first equals sign comes the color
parameters:
c
sets the color of the frame; you can browse a complete list of colors that FFmpeg understands [1] and use one of those (e.g.,red
here), or you can use the #RRGGBB[AA] notation.size
establishes the frame size.rate
sets the frames per second.
The -t 10
parameter makes the resulting clip 10 seconds long.
Making a 10-second video of a flat color is probably the simplest thing you can do with lavfi
, but making a video of a test pattern is also easy:
ffmpeg -f lavfi -i testsrc=s=640x480 -t 10 testscr.webm
Combining lavfi
-generated streams with other streams is what makes it really powerful. As you can see in Listing 1, lavfi
streams can be used the same way you use regular streams, as described in last month's issue [2]. In this case, you are blending a generated test card stream with a clip taken from the Big Buck Bunny [3] movie (Figure 1). If you would like to see what it will look like before rendering a new video file, check out the "Going Live" box and the ffplay
utility.
Going Live
FFmpeg comes with a simple media player that allows you to preview the results of your experiments live. Substitute ffplay
in place of ffmpeg
, and the result will be output directly to a window on your desktop instead of a file – a great way to check that you are achieving the effect you want to achieve without waiting for the whole clip to be rendered.
For example, to play the clip of red frames as it is generated, you can run:
ffplay -f lavfi -i "color=c=red:size=640x480:rate=30"
To stop the script, press Ctrl+C in the terminal window in which you are running FFmpeg.
To show the test card, run:
ffplay -f lavfi -i "testsrc=s=640x480"
Listing 1
Superimposing a Generated Stream
Meme Factory
All of the above is well and good, but lavfi
-generated streams do have some real-world applications, such as in animated GIF building.
GIFs are a really old and not very efficient graphical format, but they have become very popular in the last years. Twitter, Facebook, WhatsApp, and Telegram all implement ways for users to convey their deepest thoughts using animated clips of celebrity facepalms and cats falling off of things.
However, not all GIFs are equal; in fact, it is easy to distinguish a "professional" GIF artisan from a wannabe artist by the caliber of their pixels. Let me explain with Figure 2.
You are looking at a compound image of two different GIFs, each generated using a different technique. On the left, the image looks smooth, with a gradual gradient between each shade of gray. On the right, you can clearly see the pixels, and the transition between areas with different shades of gray is abrupt and jagged. Believe it or not, both are the exact same resolution, and the smooth, good-looking version on the left weighs a whole 2MB less than the one on the right.
To understand why this happens, you must look at how GIF uses palettes. GIF images use a palette of only 256 colors, which is not enough to cover the whole spectrum of a full-color video and make it look good; at least, it is not good enough by today's standards. The easiest and laziest way of generating a GIF from a clip is to enter:
ffmpeg -i clip.mp4 image.gif
However, this is not optimum by a long shot. All of the colors in all of the frames in your clip must be matched to one of the colors in the default palette used by FFmpeg (Figure 3). Not surprisingly, the final result can look grainy.
However, you can create a palette that draws its colors from the clip you are converting itself:
ffmpeg -i clip.mp4 -filter_complex "fps=15,scale=320:-1:flags=lanczos,palettegen" clip.palette.png
Here, FFmpeg creates a palette called clip.palette.png
(Figure 4) with the colors from clip.mp4
. Notice how the colors are softer and more pastel than those of the default palette and how they match the general palette of the clip much more closely.
Once you have your custom palette, you can then use it to generate your GIF:
ffmpeg -i clip.mp4 -i clip.palette.png -filter_complex "paletteuse,fps=10" clip.gif
Apart from using the palette, you are also reducing the frame rate (fps=10
) to make a smaller, lighter clip.
If your clip still doesn't look very good, remember you still only have 256 colors. The palette in Figure 4 does not look like a uniform gradient, which is a sure sign your GIF will probably look splotchy. If your original clip has a large variety of colors (e.g., with frames shot at night and bright, colorful frames shot during the day), your custom palette will be more uneven and your GIF will look grainier.
Filters and algorithms let you compensate for this effect, but none are simple, nor do they tend to improve the quality that much. Instead, "professional" GIF creators have come up with something else.
256 Shades of Gray
You might have noticed that many GIFs are rendered in shades of gray. This is because it is easier to get smooth transitions from one shade of gray to the next with a palette of 256 grays than with a palette trying to cover a wider array of colors.
To prepare a clip for the "GIFification" process, you need to reduce it to shades of gray:
ffmpeg -i clip.mp4 -filter_complex "format=gray,scale=640:-1" clip_gray.mp4
It's important to notice how you should resize your clip to the size you want your GIF to be (scale=640:-1
). If you mess with the size when converting to GIF, it will mess with your palette.
By converting your clip to gray, you would seemingly have made matters worse. The default palette in Figure 3, has very little in the way of gray. Instead of being able to pick from 256 colors, if FFmpeg were to use the default palette, it would have to choose from 10 or 15 colors, making the splotchiness even worse than if you left the clip in full color.
However, if you generate a palette with the following,
ffmpeg -i clip_gray.mp4 -filter_complex "fps=15,scale=320:-1: flags=lanczos,palettegen" clip_gray.palette.png
you'll see something like Figure 5.
If you use this palette to generate your GIF,
ffmpeg -i clip_gray.mp4 -i clip_gray.palette.png -filter_complex "paletteuse,fps=10" clip_gray.gif
open clip_gray.gif
in an image visualizer and zoom, you will notice how the transitions between colors are much smoother.
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
Gnome 47.2 Now Available
Gnome 47.2 is now available for general use but don't expect much in the way of newness, as this is all about improvements and bug fixes.
-
Latest Cinnamon Desktop Releases with a Bold New Look
Just in time for the holidays, the developer of the Cinnamon desktop has shipped a new release to help spice up your eggnog with new features and a new look.
-
Armbian 24.11 Released with Expanded Hardware Support
If you've been waiting for Armbian to support OrangePi 5 Max and Radxa ROCK 5B+, the wait is over.
-
SUSE Renames Several Products for Better Name Recognition
SUSE has been a very powerful player in the European market, but it knows it must branch out to gain serious traction. Will a name change do the trick?
-
ESET Discovers New Linux Malware
WolfsBane is an all-in-one malware that has hit the Linux operating system and includes a dropper, a launcher, and a backdoor.
-
New Linux Kernel Patch Allows Forcing a CPU Mitigation
Even when CPU mitigations can consume precious CPU cycles, it might not be a bad idea to allow users to enable them, even if your machine isn't vulnerable.
-
Red Hat Enterprise Linux 9.5 Released
Notify your friends, loved ones, and colleagues that the latest version of RHEL is available with plenty of enhancements.
-
Linux Sees Massive Performance Increase from a Single Line of Code
With one line of code, Intel was able to increase the performance of the Linux kernel by 4,000 percent.
-
Fedora KDE Approved as an Official Spin
If you prefer the Plasma desktop environment and the Fedora distribution, you're in luck because there's now an official spin that is listed on the same level as the Fedora Workstation edition.
-
New Steam Client Ups the Ante for Linux
The latest release from Steam has some pretty cool tricks up its sleeve.