Manage Logs with logrotate
The Bookkeeper
Take charge of your installation’s logfiles with logrotate.
Special Thanks: This article was made possible by support from Linux Professional Institute
Perhaps one of the most underreported yet regular tasks of a system administrator is to parse logfiles. Scrolling through logfiles isn’t fun, but they are an indispensable resource for helping troubleshoot an issue. Seasoned admins will tell you that they almost always instinctively pull up the logs whenever they face any issues with their installations.
Linux, for its part, is ever vigilant and makes careful logs of everything that takes place within your system. Most logfiles under Linux reside under the /var/log/ directory. These are all system and service logs, which come in handy when troubleshooting system-wide issues. Different apps also write app-specific logs, which are kept in a configurable location usually under a user’s home directory.
You don’t necessarily need to use any special tools to read logfiles, since they are stored as plain text. Not only can you search the logfiles, you can also create scripts to traverse them and then perform any action based on the results. While you can usually read logfiles without leaving the command-line interface (CLI) (see the box titled “View Logfiles Using the CLI”), many distributions also ship with a graphical tool to help you with the task (Figure 1).
View Logfiles Using the CLI
Most logfiles' unending nature means often using tools like grep and tail to view only the information you need instead of traversing through the entire file each time. As an example, an authorization tool's logfiles prompt for user passwords (such as sudo), SSH sessions, and such. Their usage is tracked by the /var/log/auth.log file. Depending on how your system is accessed, this file will have many entries. So if you only want to look at information related to sshd logins, you can use:
grep sshd /var/log/auth.log | less
Another useful CLI command to prune the logfile is the tail command. For instance,
tail /var/log/auth.log
will display the last 10 lines of the auth.log file. If that doesn’t provide enough information to diagnose your problem, you can specify a number along with the -n option to make tail print the specified number of lines starting from the end. For instance,
tail -n 20 /var/log/auth.log
will print the last 20 lines.
The idea of using tail with logfiles is that if an application malfunctions, any information pertaining to the failure will be at the bottom of the logfile, since the application is no longer running and nothing new is added to the logfile. If, however, the application is running and the logfile is being updated, you can still use tail to monitor updates to the file in real time with the follow (-f) option. The command
tail -f /var/log/auth.log
displays the last 10 entries and then waits to print any new ones as soon as they are written to the logfile.
Archived Logs
Looking through the /var/log/ directory, you may notice that some logfiles have a number suffix, such as auth.log.1, auth.log.2.gz, etc. These are rotated logfiles. Over a period of time, just about all logfiles grow too large and become difficult to read. When that happens, the original logfile is renamed, and a new logfile is started. Log rotation is the process that renames a current logfile, such as changing auth.log to auth.log.1, and sets up a new logfile, auth.log, for new log entries. Depending on the importance of the underlying tool or service, some logfiles will have older iterations with higher numbers, while others will have lower numbers. The system might also compress the old logfile before creating a new one to save space, which is especially true for logfiles that take up too much disk space.
Rotating logfiles is important for several reasons. First, you probably don’t want older logfiles eating up too much of your disk space. Second, you don’t want the logfiles to be extremely large, because that’ll make it even more cumbersome when you need to analyze the data within them. Log rotation, like any system administration task, is an involved process. You’ll have to decide which logfiles are to be rotated, when and how often, and whether the rotated logfiles need to be compressed, along with several other questions. The logrotate utility makes log rotation fairly straightforward by helping you automate the process. It offers enough dexterity, and you can easily configure the tool to rotate logs per your requirements, as I’ll demonstrate in this tutorial.
Getting Started with logrotate
Chances are your favorite distribution already ships with logrotate right out of the box, but if it doesn’t, you can easily pull it from the official repositories with a simple apt, yum, or dnf command. The logrotate utility, which is governed by the /etc/logrotate.conf file, is responsible for creating new logfiles, as well as renaming and compressing the old logfiles. Also of note is the /etc/logrotate.d directory, which contains configuration files for individual logfiles such as apt, dpkg, etc.
When logrotate runs, it reads its configuration files to decide where to find the logfiles that it needs to rotate, how often the files should be rotated, how many archived logs to keep, and other details. There are primarily two ways to write a logrotate script. You can either tweak its default configuration file /etc/logrotate.conf or create a separate configuration file for each app or service that logs data inside the /etc/logrotate.d directory. By default, the tool uses a combination of both of these methods. The /etc/logrotate.conf file typically looks like Listing 1.
Listing 1
Sample /etclogrotate.conf File
# rotate logfiles weekly weekly # use the syslog group by default, since this is the owning group # of /var/log/syslog. su root syslog # keep 4 weeks worth of backlogs rotate 4 # create new (empty) logfiles after rotating old ones create # use date as a suffix of the rotated file #dateext # uncomment this if you want your logfiles compressed #compress # packages drop log rotation information into this directory include /etc/logrotate.d
In Listing 1, we’ve set several global options. Remember, however, that local definitions, which we’ll get to in a bit, override global ones. The weekly option asks the tool to attempt to rotate the logs on a weekly basis. You can change the frequency by using either the daily or the monthly option. The rotate 4 option specifies that the tool should keep the last four rotated logs. When it’s time to rotate the fifth one, logrotate will delete the oldest log and rename the others to maintain the sequential file names. The create keyword is used to create a new logfile while renaming the old one. The su root syslog specified at the top of the file determines the logfiles' permissions. Here, I tell logrotate to archive the logs using the specific user (root) and group (syslog) to prevent issues with permissions.
Besides these, there are a couple of commented out options as well. If enabled, dateext will use the current date as the old logfiles' extension when rotating files. Then there’s the compress option, which asks the utility to compress old rotated logs. By default, it uses gzip, but you can also specify another compression utility. For instance,
compresscmd /bin/bzip2
asks logrotate to use the bzip2 compression utility and together with the option
compressext .bz2
appends the specified extension to the compressed files.
If you need to read a rotated logfile that’s been compressed, you’ll first have to uncompress it. This is rather inconvenient if you need to use rotated files regularly. In such a case, you can use the delaycompress option that asks the utility to not compress the rotated file until the next rotation cycle. This will ensure that the most recent rotated logfile is in an uncompressed state and can be used without extra effort.
At the end of Listing 1, the include option asks logrotate to pull in the specified file's contents, which in this case is the entire /etc/logrotate.d directory. The directory contains logrotate configuration files for various utilities; this is where you can add a configuration file for a service or tool per your requirements. As an example, take a look at the contents of the Uncomplicated Firewall (UFW) configuration file in Listing 2.
Listing 2
/etc/logrotate.d/ufw
/var/log/ufw.log { rotate 12 monthly missingok notifempty compress delaycompress sharedscripts postrotate invoke-rc.d rsyslog rotate >/dev/null 2>&1 || true endscript }
For the most part, Listing 2 uses the same options as the global configuration file, but it also introduces some new ones. For starters, instead of weekly rotations, I am doing them once every month for UFW. Secondly, I am preserving the logs for a year by keeping them for 12 rotations. As I mentioned earlier, in case both the global configuration file and the local configuration file mention different values for the same option, logrotate will follow the values set in the files under /etc/logrotate.d. This is why it’ll stick to the rotation policy (monthly and rotate 12) set in this file instead of the ones set in the global file (weekly and rotate 4). The missingok option asks logrotate to continue working even if there is no logfile. The default option, nomissingok, asks the tool to throw an error if it doesn’t find an existing logfile. The notifyempty option asks logrotate to not to rotate a logfile if it is empty.
One important option is postrotate, which lists custom commands and scripts that will be run after the logs have been rotated. When used together with the sharedscripts option, logrotate will run only once when all the configuration files that write to the same log directory have been rotated. To force rsyslog to reopen the files it’s using to write logs, use the command:
invoke-rc.d rsyslog rotate > /dev/null
This is required, because rsyslog will continue to write to the file that first called it, which has since been rotated. So this command ensures that rsyslog writes to the current logfile. You can also use postrotate to reload services, like the Apache web server:
postrotate /bin/systemctl restart apache2.service > /dev/nullendscript
Here, I reload the Apache web server, after its logfiles have been rotated, to make sure it writes to the newly created logfiles. There’s also the prerotate option, which will execute the mentioned commands/scripts before the log rotation process. When either the prerotate/postrotate option is triggered, it’ll continue executing the commands until it runs into the endscript keyword.
I’ve only mentioned a handful of the most frequently used logrotate options. There are several others that you can read about on the logrotate’s manpage.
Running logrotate
After you’ve defined or tweaked a logrotate configuration file, it’s best to use the debug (-d) option to execute a dry run to see what the utility would do when it’s actually executed (Figure 2).
While you can invoke logrotate manually, it’s best to run it through cron. By default, the installation of logrotate creates a cron job file named /etc/cron.daily/logrotate. You can copy the logrotate cron job inside any other cron directory (there’s /etc/cron.hourly, /etc/cron.weekly, or /etc/cron.monthly) to change the execution frequency. The /var/lib/logrotate/status file is updated whenever the logrotate cron job is executed. The status file shows the date and time when each of the logfiles was last rotated.
There is no disputing the importance of logs in the life of a system administrator. A utility like logrotate helps you take charge of the management of such a crucial resource. It’s an elegant solution that offers lots of configurational flexibility while still being easy to comprehend and deploy.
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
Latest Cinnamon Desktop Releases with a Bold New Look
Just in time for the holidays, the developer of the Cinnamon desktop has shipped a new release to help spice up your eggnog with new features and a new look.
-
Armbian 24.11 Released with Expanded Hardware Support
If you've been waiting for Armbian to support OrangePi 5 Max and Radxa ROCK 5B+, the wait is over.
-
SUSE Renames Several Products for Better Name Recognition
SUSE has been a very powerful player in the European market, but it knows it must branch out to gain serious traction. Will a name change do the trick?
-
ESET Discovers New Linux Malware
WolfsBane is an all-in-one malware that has hit the Linux operating system and includes a dropper, a launcher, and a backdoor.
-
New Linux Kernel Patch Allows Forcing a CPU Mitigation
Even when CPU mitigations can consume precious CPU cycles, it might not be a bad idea to allow users to enable them, even if your machine isn't vulnerable.
-
Red Hat Enterprise Linux 9.5 Released
Notify your friends, loved ones, and colleagues that the latest version of RHEL is available with plenty of enhancements.
-
Linux Sees Massive Performance Increase from a Single Line of Code
With one line of code, Intel was able to increase the performance of the Linux kernel by 4,000 percent.
-
Fedora KDE Approved as an Official Spin
If you prefer the Plasma desktop environment and the Fedora distribution, you're in luck because there's now an official spin that is listed on the same level as the Fedora Workstation edition.
-
New Steam Client Ups the Ante for Linux
The latest release from Steam has some pretty cool tricks up its sleeve.
-
Gnome OS Transitioning Toward a General-Purpose Distro
If you're looking for the perfectly vanilla take on the Gnome desktop, Gnome OS might be for you.