Full Potential

For awhile, graphics cards have contained dedicated components for decoding and encoding video formats such as MPEG2, H.264, and H.265. In chip developer jargon, these embedded units are called application-specific integrated circuits (ASICs) [1].

CPUs without ASICs have a hard time decoding videos. However, graphics cards with their specialized circuits can decode video even on low-powered systems like the Raspberry Pi without jerking and with lower power consumption, thus shifting some of the load off the main processor. Lower power consumption means longer battery life and less heat generated, which in turn makes the system quieter.

Interface

On Linux, programs access the graphics chip's acceleration functions via the Video Acceleration API (VA-API) [2]. The vainfo tool (included in the vainfo or libva-utils packages) shows you the algorithms supported by the graphics card. The VLD suffix indicates the decoding capability, while EncSlice denotes the encoding capability.

For example, the integrated Intel 5500 HD graphics chip in the Intel CPU i7 5600U processor (based on the Broadwell architecture) supports decoding and encoding of MPEG2 and H.264, but not VP9 (Listing 1). Intel's eighth generation Core microprocessor ("Coffee Lake") – for example, an i7 9700 with UHD Graphics 630 – can already decode and encode VP9 (Listing 2). An older AMD graphics card only speeds up decoding of H.264, but this GPU does not yet work with VP8 and VP9 (Listing 3).

§§number
$ vainfo
[...]
vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 20.2.0 ()
vainfo: Supported profile and entrypoints
[...]
    VAProfileMPEG2Simple : VAEntrypointVLD
    VAProfileMPEG2Simple : VAEntrypointEncSlice
[...]
    VAProfileH264Main : VAEntrypointVLD
    VAProfileH264Main : VAEntrypointEncSlice
    VAProfileH264Main : VAEntrypointFEI
[...]
    VAProfileVP8Version0_3 : VAEntrypointVLD

§§number
$ vainfo
libva info: VA-API version 1.3.0
[...]
vainfo: Driver version: Intel i965 driver for Intel(R) Coffee Lake - 2.3.0.pre1 (2.3.0.pre1)
[...]
    VAProfileVP9Profile0 : VAEntrypointVLD
    VAProfileVP9Profile0 : VAEntrypointEncSlice
    VAProfileVP9Profile2 : VAEntrypointVLD

$ vainfo
[...]
vainfo: Driver version: Mesa Gallium driver 20.0.1 for AMD OLAND (DRM 3.39.0, 5.8.0.mx64.339-13250-g66bc99e8bbd14, LLVM 9.0.1)
    VAProfileMPEG2Simple : VAEntrypointVLD
    VAProfileMPEG2Main : VAEntrypointVLD
    VAProfileVC1Simple : VAEntrypointVLD
    VAProfileVC1Main : VAEntrypointVLD
    VAProfileVC1Advanced : VAEntrypointVLD
    VAProfileH264ConstrainedBaseline: VAEntrypointVLD
    VAProfileH264Main : VAEntrypointVLD
    VAProfileH264High : VAEntrypointVLD
    VAProfileNone : VAEntrypointVideoProc

Because not all required software components are under a free license, you have to enable VA-API support on Ubuntu or Debian (for example) for fifth generation Intel chips (Broadwell) by installing intel-media-va-driver-non-free from the package sources.

Then you use the mpv media player to check if the graphics card is used during decoding. To do this, call the player with the --hwdec=vaapi option. If the program reports

VO: [...] vaapi

when it starts playing back a video, the interface is working (Listing 4). Involving the dedicated decoder chip reduces the entire system's energy consumption. For example, PowerTOP now reports only 6.32W (Figure 1) instead of 7.87W (Figure 2).

§§number
$ mpv --hwdec=vaapi /<path>/<to>/video.mkv
[...]
VO: [gpu] 1920x1080 vaapi[nv12]
VO: [gpu] 1920x1080 yuv420p
[...]

Figure 1: Launched with the options --vo=gpu --hwdec=vaapi, mpv accesses the specialized decoder chips on the graphics card.

Figure 2: The system's energy consumption decreases noticeably when VA-API is activated. Without the optimization, the system draws 1.5W more battery power.

Web Browser

Today many Internet sites – not least YouTube – integrate videos, resulting in web browsers often functioning as video playback programs. Users can benefit in particular if the browser can access the chips on the graphics card during decoding. With the WebRender [3] compositor, Mozilla laid the foundation for using the graphics card to display web pages. WebRender lets you use hardware acceleration in addition to the GL Compositor. However, even the current Firefox 80 only enables the function for Windows 10 [4] by default, because Linux developers complain that there are too many problems with the graphics card drivers.

As is so often the case, a Red Hat developer has taken the Linux desktop another step forward with Linux graphics card drivers. Martin Stransky programmed the changes for Firefox; from Fedora 31 on, they are implemented in Firefox 77.0. In his blog post [5], Stransky reports that thanks to Wayland, many errors in the Linux graphics card drivers have now been discovered and fixed, as Wayland makes intensive use of the acceleration functions. For hardware-accelerated decoding and encoding, the developer resorts to the proven FFmpeg [6], for which Firefox writes the material directly to the graphics card memory via the dma-buf subsystem, thus removing the need to copy the data. By default, the functions are disabled.

In a Wayland desktop environment, calling

MOZ_ENABLE_WAYLAND=1 MOZ_WEBRENDER=1 firefox

from the terminal activates WebRender, as well as the VA-API interface. After appropriate configuration, Listing 5 shows the browser call with VA-API in an X11 environment, which Firefox has supported since version 80. The MOZ_LOG variable controls the output messages.

§§number
$ MOZ_X11_EGL=1 MOZ_WEBRENDER=1 MOZ_LOG="PlatformDecoderModule:5" firefox
[...]
[Child 19296: MediaPDecoder #3]: D/PlatformDecoderModule Choosing FFmpeg pixel format for VA-API video decoding.
[Child 19296: MediaPDecoder #3]: D/PlatformDecoderModule Requesting pixel format VAAPI_VLD
[h264 @ 0x7f01f5720000] Format vaapi_vld chosen by get_format().
[h264 @ 0x7f01f5720000] Format vaapi_vld requires hwaccel initialization.
[h264 @ 0x7f01f5720000] Considering format 0x3231564e -> nv12.
[h264 @ 0x7f01f5720000] Picked nv12 (0x3231564e) as best match for yuv420p.
[...]

Next, open the configuration editor with the pseudo-URL about:config and enable VA-API support by double-clicking on media.ffmpeg.vaapi.enabled to switch from false to true. To find out if WebRender works, check the URL about:support in the Graphics section. This is where Firefox should output information about WebRender.

Currently, YouTube delivers VP9-encoded movies by default due to the smaller size for the same quality. For hardware-accelerated VP8/VP9 playback in Firefox, do not load the ffvpx library provided by Firefox, because it cannot use the VA-API interface. Setting the media.ffvpx.enabled key to false in about:config tells Firefox to use the system FFmpeg libraries [7]. Some distributions already configure Firefox accordingly.

If the graphics card does not support H.264, the Firefox h264ify [8] or enhanced-h264ify [9] extensions block formats encoded with VP8 and VP9. If you find messages in the logs such as

D/PlatformDecoderModule DMA-Buf/VA-API can't be used, WebRender/DMA-Buf is disabled

please check the settings again.

Troubleshooting

If problems occur when playing videos, you should first check whether the errors still occur in the nightly build of the browser with the latest changes. If the bugs are confirmed, you can contribute to the project by reporting the problem to Mozilla's bug tracker [10], appending the output of MOZ_LOG="PlatformDecoderModule:5".

Download the current nightly build [11] for Linux to your computer as shown in Listing 6 and unpack the archive; the last command starts the browser. Make sure that you close the regular Firefox instance completely before doing this. After completing testing, you can delete the firefox/ directory created by unpacking.

§§number
### Download 64-bit version:
$ wget 'https://download.mozilla.org/?product=firefox-nightly-latest-ssl&os=linux64&lang=en-US' -O firefox-nightly64.tar.bz2
### Download 32-bit version:
$ wget 'https://download.mozilla.org/?product=firefox-nightly-latest-ssl&os=linux&lang=en-US' -O firefox-nightly32.tar.bz2
### Unzip and call:
$ tar xf firefox-nightly*.tar.bz2
$ firefox/firefox

Outlook

Use the turbostat tool to determine whether the energy consumption when playing videos using VA-API is now lower (see the "Consumption Display" box). Table 1 shows the averaged values when playing a video via the mpv media player and for Firefox on a Dell Latitude E7250 with the Broadwell chipset (Intel Core i5 5300U). While mpv obviously benefits from the optimized decoder chip, the savings in Firefox with WebRender are somewhat lower. PowerTOP reports a discharge rate of 7.60W (Figure 3) for VA-API; without it, the system draws 8.56W of power from the notebook battery when playing the video in the browser (Figure 4).

Figure 3: The Firefox web browser can also use hardware acceleration, but support for this function is not yet standard under Linux.

Figure 4: Without VA-API, the system draws nearly 1W more power from the laptop's battery when playing back the video in Firefox's current nightly build.

Thanks to Red Hat's commitment, Firefox on Linux will soon use the graphics card for decoding videos by default, allowing Firefox to finally catch up with the Microsoft Windows 10 version in terms of functionality. Default VA-API integration removes another obstacle to achieving the year of the Linux desktop (if that year ever happens).

As videoconferencing needs increase, encoding videos (of yourself) becomes just as important as decoding the received video data. Many image streams are encoded with VP8, which many graphics cards support. Firefox 80 or 81 can use VA-API for WebRTC [12] after enabling media.ffmpeg.low-latency.enabled in about:config. Hopefully, Firefox will soon enable VA-API integration by default, doing away with all of the manual work.

§§number
$ turbostat --quiet -i 3 -s 'PkgWatt,CorWatt,GFXWatt,RAMWatt'
PkgWatt CorWatt GFXWatt RAMWatt
4.21 1.55 0.13 1.02
4.21 1.55 0.13 1.02
[...]