Log Rolling

Linux log files do not take up much space. Even with systemd, many logs continue to be text files. All the same, if left unattended, log files would eventually fill their entire filesystem – and, if that filesystem is root, crash the system. To avoid this eventuality, Linux uses logrotate, a system of multiple logs that are switched regularly, either at a specified time or when the file size reaches the designated limit.

Logrotate is installed by default in every distribution of which I am aware. You can usually take for granted that logrotate is installed, but you can check by running the command ls /var/log. If you see multiple files with numbered suffixes, then logrotate is installed. Alternatively, in some distributions, you can run less /var/lib/logrotate.status to see the latest logrotate activity.

In fact, logrotate is required by a Linux system, because many standard applications use logs by default, including apache, aptitude, cups-daemon, and dpkg. Ordinarily, such logs exist as much for the system as the administrator, but they can be invaluable for troubleshooting. You may want to modify the default setups for a log or rotate a log for your own scripts.

For example, on my system with a solid state drive, I have a script with a log for fstrim, the utility run weekly to discard unused blocks and to keep the drive running at top efficiency. By setting up the fstrim script for logrotate, I can keep its size from ballooning out of control without any further action on my part.

Logrotate works with several files:

• /etc/logrotate.conf: The global configuration file.
• /etc/logrotate.d: A directory of logrotate settings for particular applications. Entries in this directory override the configuration in /etc/logrotate.conf.
• /etc/cron.daily/logrotate: The scheduler in which an entry for logrotate is entered so that all the applications with an entry in /etc/logrotate.d are processed. Usually, the scheduler is daily, but it can be hourly, monthly, or weekly instead.

The logs themselves are kept in /var/log (Figure 1).

Figure 1: Logrotate uses a series of logs to prevent the size of logs becoming large enough to destabilize the system.

Note that, unlike many commands, the logrotate command does not override the settings in /etc/logrorate.conf or /etc/logrotate.d. Instead, the command's options affect only how the command is run.

Setting Log Options

Values for log rotation can be set globally in /etc/logrotate.conf (Figure 2) or in a separate file in /etc.logrotate.d (Figure 3). Fields can be set in any order in the file, but if fields contradict each other, the one closest to the bottom of the field is used. Otherwise, fields can be in any order, with one per line, although the usual arrangement is to have global settings at the start of the file, and values that apply to specific logs contained within curly braces. For example, /etc/logrotate.d/ defines one log this way:

/var/log/apt/term.log {
  rotate 12
  monthly
  compress
  missingok
  notifempty
}

Figure 2: The /etc/logrotate.conf file contains the global options for logrotate.

Figure 3: The configuration file for the CUPS daemon from /etc/logrotate.d/.

The same file, incidentally, defines another apt-related log in the same file. Most distributions further organize the settings by adding a comment above each field so that it is easily read. Field values can also be disabled by adding a hashtag (#) at the start of the line.

Regardless of where the log options are set, a couple of options must be set. Each set of values must have a regularity, such as monthly, weekly, or daily, corresponding to the cronjob in which it will run. Although not required, each set should also have a maximum number of rotations, which is set using rotate NUMBER.

Alternatively, logs can be rotated whenever the current one reaches a current size with size SIZE. Other options are maxsize SIZE or minsize SIZE. The size can be set using k, M, or G (kilobytes, megabytes, gigabytes). Because logs are plain text, you can probably use kilobytes.

Regardless of which method is used to determine rotation, when a log is rotated, it is renamed by adding a number as a suffix, so that the current log becomes log.1, and the previous one log.2, and so on. When the maximum number of rotations is reached, the oldest log is deleted and replaced by the next oldest. If rotate is set to 0, then the older log is deleted, instead of being temporarily stored. If you prefer, with dateext, older logs can have a date extension such as YYYYMMDD, with the exact date format set using dateformat. No matter what the naming method, there is no requirement about how many rotations to use, except that, the more important the information, the more rotations should generally be used.

Ordinarily, rotated logs are stored in the same directory as the current one – usually, /var/log. However, you also have the option of using olddir Directory to store a log rather than deleting it when its turn come. Another choice is to mail a log being rotated out of existence instead of renaming it, using mail ADDRESS. If you prefer, use mailfirst to mail the just-rotated file. If you want to be sure that no log is mailed, use nomail instead.

By default, old logs are compressed using gzip. However, you can store old logs without depression by using nocompress. You can also use another command for archiving, such as tar using compresscmd, which requires a matching uncompresscmd so that the logs can be properly accessed. With any compression method, compressionoptions passes on to logrotate the options to use during compression. For example, when using gzip, passing on the option -6 will maximize compression at the price of speed.

Still other options include missingok, which ignores a missing log file and continues processing, and shred and shredcycles NUMBER, which overwrites deleted logs, making them unreadable. Further flexibility can be added to how a particular set of options is handled by adding scripts that begin with postrotate, prerotate, firstaction, or lastaction on a separate line, each of which ends with endscript.

Choosing Command Options

When logrotate runs, it works with all the logs defined in /etc/logrotate.conf and /etc/logrotate.d/. The options for the command are about how it runs, not how individual sets of logs are handled. If you mail any logs when they are rotated, use --mail (-mail) COMMAND to change the email application from the default /usr/bin/mail -s.

When you add new configuration entries, or if you have removed files manually instead of using logrotate, you may need to use --force (-f). In these situations or with any other troubleshooting, I suggest using --verbose (-v), especially when testing, so that you can be sure that the command and the configuration files are interacting in the way that you hope.

Setting Up a Cronjob

Although you might test your command as you set up logrotate, its main purpose is to add it as a cronjob so you can forget about log rotation. Typically, you will set it as a daily cronjob, running a command to confirm the existence of log rotation, then running logrotate itself, as shown in Figure 4. This command can be easily modified if you want to run logrotate at another frequency.

Figure 4: Once you add logrotate as a cronjob, you may not need to edit it again.