If you are an ordinary user, you may barely notice systemd. However, if you administer a system, even if only your own, the time is overdue to learn a thing or two about systemd. A useful place to start is with its journals, partly because systemd handles journals in a very organized way, and partly because reading logs is essential to troubleshooting.

Systemd's journals are a replacement for the traditional syslogs. The traditional syslog daemon is gone from a systemd system and replaced by journald, although you might not have noticed because, for now, many distributions are forwarding systemd journals through traditional utilities in a default form similar to those with which you are familiar. You might not even have seen any new boot messages, because, at least initially, many distributions are shipping with the systemd journal not enabled.

If the journal is running, you will have a /var/log/journal file. If you do not, create the file and run the following command to set the proper permissions:

setfacl -R -nm g:adm:rx,d:g:adm:rx /var/log/journal

You can further modify the journal's behavior by modifying /etc/systemd/journald.conf. In particular, you should uncomment SystemMaxFileSize= and set a limit so that some day in the future you do not wake up with an unexpectedly full hard drive.

Once the journal is running, you can either view the entries for a specific user or, as root user, entries for the entire system. Unlike traditional Linux logs, systemd's journals are binary, and at best you will be able to read only part of their files if you try to open them in a text editor. Linus Torvalds has complained about this change as an unnecessary complication, but if you are using a virtual terminal, you can easily copy and paste, whereas at the command line you can redirect output to a text file. In any case, journalctl, the command for viewing the journal, has plenty of commands to control both the format and content of the display.

Formatting Options

One of the unusual features of systemd's journals is the variety of formatting options available. To view events for which you have permission, the command journalctl is all you need. The unmodified command shows a list that starts in less with the oldest recorded event. Each event displays on a separate line. From left to right (Figure 1), the event shows the time it occurred, the machine it occurred on, the originator of the event, and a brief description. The description is truncated if it is longer than the space allotted for it.

Figure 1: The default display for systemd's journal is similar to that for the old logs.

This default is the default because it is the closest to the traditional syslog entries. All the same, it is only moderately useful unless you have just enabled the journal and have only one bootup's worth of entries. For one thing, it contains perhaps one-sixth of all fields that may be in an entry, most of which only display if you add the option --output=verbose (Figure 2).

Figure 2: The verbose output displays all the information for a journal entry.

Almost certainly, too, you will want to use -r (or --reverse), which inverts the default order and shows the most recent entries first. You probably also want to use --no-full, so that the ends of lines are not cut off and you can read each entry in full. Additionally, -x (--catalog) will display any hints included in an event, although they are limited in their usefulness by the fact that few entries include any hints.

You can change this entry display with --output=FORMAT. The default entry is short; --output= can be completed with several other formats. Besides verbose, the most useful completion is short-precise, which displays times down to microseconds.

Other completions of --output= are three versions of JSON, a format developed for systemd [1]. Of these, json and json-sse are intended for other software and are detailed in the man pages "Journal JSON Format" and "Server Sent Events." If you choose to use JSON, json-pretty is intended for human use (Figure 3).

Figure 3: JSON has several formats. Pictured here is the one for people rather than machines.

At other times, you might just want to look at the entry messages, saving the details for when you have narrowed down those that interest you. In these cases, you can use --output=cat.

Filtering Journal Entries

Systemd has as many ways to filter journal entries as it does to format them. Many of these filters are for a specific information source. For example, specifying a path after the basic command displays messages from the executable listed; similarly,

• -k or --dmesg displays only entries that originate with the kernel.
• --system shows entries from system services.
• -M (--machine=) shows entries from local containers.
• --PID= shows entries from the ID for an executable.

Other options let you specify the message priority to display (-P, --priority) or the journal file to search (--file=FILE-PATH GLOB).

Systemd also introduces the concept of units, or types of objects. For example, a unit can be a mount unit for each filesystem, a service unit for the CUPS printing system, or a device unit for mounting a sound card. Unsurprisingly, you can also use -u (--unit=) to search by unit or --user-unit to track down messages from a particular part of the system.

Other filters take a different approach. With --list-boots, journalctl displays the system boots it has recorded (Figure 4). You can then use the identifying string in the second field for a boot to search for messages that were sent during a particular boot using -b (--boot=).

Figure 4: Before you can search for a particular boot, you need a list of boots in the log. Shown here is an entry for the first boot after the journal was enabled.

Still other filters involve a combination of the options; -n and lines=NUMBER show the most recent events, setting a limit on the number of events displayed. Another way to limit the display is to use --since, --until, or both, specifying a time and date using the YYYY:MM:DD:HH:MM:SS format or a completion like 10 minutes ago.

Another useful option is -f (-follow), which is borrowed from the tail command, just as the journal viewer borrows from less. As in tail, these options display the most recent entries, adding new entries as they occur, making it an essential member of the toolkit when you are trying to track down a problem while going about your work.

Finally, if you have not specified a maximum file size for a journal in the configuration file, you might decide to clean house before searching – or vacuum, to use journalctl's term. The option --vacuum-size= can be completed with a size followed by a unit such as megabytes or terabytes to reduced the journal down to the size specified. Alternatively, --vacuum-time= can be completed by a number and a unit of time, such as weeks to remove all data later than the time you specify. With either form of vacuuming, you might have a better chance of finding whatever you are looking for.

Judging the Journal

Personally, I still wonder why the journal in systemd needs be in binary format. However, after working with systemd, I have learned a certain respect for it. Its naming is systematic, with binaries ending in *ctl and configuration files in *.conf. Option names, too, are precisely chosen, so you can usually tell from the long versions what each one does.

Moreover, the options give a thorough selection of formats and filters, borrowing from other commands as needed to cover a wide variety of needs.

Whether systemd as a whole was worth implementing may take a few years for free software users to decide, but for now, one thing is clear: Its journal is not only genuinely useful but an improvement on the logs that it replaces. Take some time to explore systemd's journal, and you are likely to conclude that it is far from the worse change that might have happened.