Building Frames

Building Frames

Article from Issue 207/2018

FFmpeg is good not only for converting and fusing videos together, it can also generate streams on the fly, which you can then use for compositing and effects.

The casual user might only use the FFmpeg multimedia framework for converting from one audio or video format to another, but FFmpeg can do much more than that. I'll take a look at one of FFmpeg's most powerful secret weapons: the lavfi virtual device.

Using lavfi

FFmpeg's lavfi (short for libavfilter) virtual device sounds more complicated than it really is. Instead of using prerecorded video or audio files as streams, lavfi lets you create streams out of thin air. You can use these streams on the fly and combine them with clips and other dynamically generated streams (e.g., from a webcam or microphone) to create your output file.

Maybe it is better explained with an example:

ffmpeg -f lavfi -i color=c=red:size=640x480:rate=30 -t 10 red.webm

Most basic ffmpeg instructions you have seen probably have at least one input file, but the instruction above has none because "input" is coming from the lavfi virtual device, not from a file. The content the virtual device generates is described by what goes after the -i parameter, that is, color=c=red:size=640x480:rate=30. Broken up, color tells FFmpeg what sort of stream it should expect – in this case, a simple color video stream. After the first equals sign comes the color parameters:

  • c sets the color of the frame; you can browse a complete list of colors that FFmpeg understands [1] and use one of those (e.g., red here), or you can use the #RRGGBB[AA] notation.
  • size establishes the frame size.
  • rate sets the frames per second.

The -t 10 parameter makes the resulting clip 10 seconds long.

Making a 10-second video of a flat color is probably the simplest thing you can do with lavfi, but making a video of a test pattern is also easy:

ffmpeg -f lavfi -i testsrc=s=640x480 -t 10 testscr.webm

Combining lavfi-generated streams with other streams is what makes it really powerful. As you can see in Listing 1, lavfi streams can be used the same way you use regular streams, as described in last month's issue [2]. In this case, you are blending a generated test card stream with a clip taken from the Big Buck Bunny [3] movie (Figure 1). If you would like to see what it will look like before rendering a new video file, check out the "Going Live" box and the ffplay utility.

Figure 1: You can combine lavfi-generated streams to achieve all sorts of interesting effects.

Going Live

FFmpeg comes with a simple media player that allows you to preview the results of your experiments live. Substitute ffplay in place of ffmpeg, and the result will be output directly to a window on your desktop instead of a file – a great way to check that you are achieving the effect you want to achieve without waiting for the whole clip to be rendered.

For example, to play the clip of red frames as it is generated, you can run:

ffplay -f lavfi -i "color=c=red:size=640x480:rate=30"

To stop the script, press Ctrl+C in the terminal window in which you are running FFmpeg.

To show the test card, run:

ffplay -f lavfi -i "testsrc=s=640x480"

Listing 1

Superimposing a Generated Stream

ffmpeg -i BBB_flowers.mp4 -f lavfi -i testsrc=s=1280x720[out] -t 5 -filter_complex "[1:v]setsar=sar=1[red];[0:v]setsar=sar=1,format=rgba[bg];[bg][red]blend=all_mode=addition:all_opacity=0.5,format=yuva422p10le" -codec:a copy testscr.mp4

Meme Factory

All of the above is well and good, but lavfi-generated streams do have some real-world applications, such as in animated GIF building.

GIFs are a really old and not very efficient graphical format, but they have become very popular in the last years. Twitter, Facebook, WhatsApp, and Telegram all implement ways for users to convey their deepest thoughts using animated clips of celebrity facepalms and cats falling off of things.

However, not all GIFs are equal; in fact, it is easy to distinguish a "professional" GIF artisan from a wannabe artist by the caliber of their pixels. Let me explain with Figure 2.

Figure 2: Smooth pixels make a high-quality GIF animation.

You are looking at a compound image of two different GIFs, each generated using a different technique. On the left, the image looks smooth, with a gradual gradient between each shade of gray. On the right, you can clearly see the pixels, and the transition between areas with different shades of gray is abrupt and jagged. Believe it or not, both are the exact same resolution, and the smooth, good-looking version on the left weighs a whole 2MB less than the one on the right.

To understand why this happens, you must look at how GIF uses palettes. GIF images use a palette of only 256 colors, which is not enough to cover the whole spectrum of a full-color video and make it look good; at least, it is not good enough by today's standards. The easiest and laziest way of generating a GIF from a clip is to enter:

ffmpeg -i clip.mp4 image.gif

However, this is not optimum by a long shot. All of the colors in all of the frames in your clip must be matched to one of the colors in the default palette used by FFmpeg (Figure 3). Not surprisingly, the final result can look grainy.

Figure 3: The default FFmpeg palette used when generating GIFs.

However, you can create a palette that draws its colors from the clip you are converting itself:

ffmpeg -i clip.mp4 -filter_complex "fps=15,scale=320:-1:flags=lanczos,palettegen" clip.palette.png

Here, FFmpeg creates a palette called clip.palette.png (Figure 4) with the colors from clip.mp4. Notice how the colors are softer and more pastel than those of the default palette and how they match the general palette of the clip much more closely.

Figure 4: A palette taken from a five-second clip lifted out of Big Buck Bunny.

Once you have your custom palette, you can then use it to generate your GIF:

ffmpeg -i clip.mp4 -i clip.palette.png -filter_complex "paletteuse,fps=10" clip.gif

Apart from using the palette, you are also reducing the frame rate (fps=10) to make a smaller, lighter clip.

If your clip still doesn't look very good, remember you still only have 256 colors. The palette in Figure 4 does not look like a uniform gradient, which is a sure sign your GIF will probably look splotchy. If your original clip has a large variety of colors (e.g., with frames shot at night and bright, colorful frames shot during the day), your custom palette will be more uneven and your GIF will look grainier.

Filters and algorithms let you compensate for this effect, but none are simple, nor do they tend to improve the quality that much. Instead, "professional" GIF creators have come up with something else.

256 Shades of Gray

You might have noticed that many GIFs are rendered in shades of gray. This is because it is easier to get smooth transitions from one shade of gray to the next with a palette of 256 grays than with a palette trying to cover a wider array of colors.

To prepare a clip for the "GIFification" process, you need to reduce it to shades of gray:

ffmpeg -i clip.mp4 -filter_complex "format=gray,scale=640:-1" clip_gray.mp4

It's important to notice how you should resize your clip to the size you want your GIF to be (scale=640:-1). If you mess with the size when converting to GIF, it will mess with your palette.

By converting your clip to gray, you would seemingly have made matters worse. The default palette in Figure 3, has very little in the way of gray. Instead of being able to pick from 256 colors, if FFmpeg were to use the default palette, it would have to choose from 10 or 15 colors, making the splotchiness even worse than if you left the clip in full color.

However, if you generate a palette with the following,

ffmpeg -i clip_gray.mp4 -filter_complex "fps=15,scale=320:-1: flags=lanczos,palettegen" clip_gray.palette.png

you'll see something like Figure 5.

Figure 5: A grayscale palette generated from a black-and-white clip taken from Big Buck Bunny.

If you use this palette to generate your GIF,

ffmpeg -i clip_gray.mp4 -i clip_gray.palette.png -filter_complex "paletteuse,fps=10" clip_gray.gif

open clip_gray.gif in an image visualizer and zoom, you will notice how the transitions between colors are much smoother.

Buy Linux Magazine

Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

  • Gobbling Up

    Most video editors supply you with a generic catalog of transitions, usually in the shape of tired wipes and fades. But what if you wanted something a little more special? FFmpeg to the rescue.

  • Video Editor Roundup

    In a comparison test, we checked out nine free video editing programs: Cinelerra, Flowblade, Kdenlive, Kino, Lightworks, LiVES, OpenShot, Pitivi, and Shotcut.

  • Motion Detection

    The motion detector software, Motion, monitors the video signal from one or multiple cameras and is able to detect whether a significant part of the picture has changed, record and track movement, or launch arbitrary external commands to trigger other actions.

  • Bitparade: Screencast Tools

    Screenshots provide a static impression of an active program, whereas desktop video sequences show the software hard at work. We’ll introduce you to some of the major tools for creating screencasts.

  • Web Videos

    We’ll show you how to convert your videos to FLV format and play them from your website with FlowPlayer.

comments powered by Disqus

Direct Download

Read full article as PDF:


njobs Europe
Njobs Netherlands Njobs Deutschland Njobs United Kingdom Njobs Italia Njobs France Njobs Espana Njobs Poland
Njobs Austria Njobs Denmark Njobs Belgium Njobs Czech Republic Njobs Mexico Njobs India Njobs Colombia