Audit Your Linux Box
Core Technology
Look for intruders and study the health of your system with Linux auditing tools.
No one enjoys being tracked. In Free Software and Linux, we take privacy very seriously. Yet, we sometimes set surveillance cameras to watch the back yard. We hardly ever look at the recordings, unless things go wrong. Then we could use videos to learn who broke that window.
Audit in Linux works much the same way. It captures security-related events, such as file access, system calls, user logins, or system reboots. Then it stores these logs safely and lets you search through them. This process doesn't add any security by itself, but it helps to track intruders. Having this is a prerequisite to Common Criteria certification, and it's a good way to peek into the system's operation for learning, fun, and profit.
The Big Picture
The Linux audit framework spans multiple components, both in userspace and in the kernel (Figure 1).
A story begins when something generates an audit event. For instance, when you open a file, the kernel can learn about it using the very same mechanism covered previously [1]. Or, if you want to trace system calls, the kernel raises a flag on a process descriptor, which causes Linux to fire an event when it returns from the call.
Not all audit events originate from the kernel, though. When the system boots, switches run levels, starts or stops daemons, etc., an init
process such as systemd logs these life cycle events. This also happens when a user logs in or out. Technically, such messages end up in the kernel as well, so all other audit framework components have a single events source.
The kernel reports audit events via a Netlink socket. In userspace, a daemon called auditd
reads them and stores them in audit log file. auditd
can also forward events to a dispatcher daemon, audisp
. This component acts as an event multiplexer for programs, which want to analyze events in real time, such as Intrusion Detection/Prevention Systems (IDS/IPS). You can also use audisp
to send events to storages other than local disk, such as remote servers. This helps keep them secure. Even though audit logs are inaccessible to users other than root, an intruder with sufficient permissions can still tamper or destroy them.
Many things are happening in your system at any given moment. Logging all of them would probably be overkill and a waste of resources. How does the kernel know which one to remember and which one to forget? Audit rules tell it. These rules define which files, directories, and system calls the kernel should monitor. You also use rules to select processes to trace or users whose actions you want to follow. auditctl
communicates rules to the kernel, either during the system startup or dynamically in run time.
Now, with events recorded somewhere, how do you get a sense of them? aureport
tool comes to the rescue. It produces a summary or detailed report upon your request. There are also ways to visualize these reports, as you'll see shortly. If you look for a specific event, use the ausearch
tool. A common pattern is to find an event of interest in aureport
(Figure 2), and then take its event ID and retrieve the details with ausearch
.
Getting Your Feet Wet
For starters, let's look at how all these pieces work together using vanilla Kubuntu 16.04 LTS as an example. Once you boot the system, the audit log, which resides in /var/log/audit.log
, should already have some events (Listing 1).
Listing 1
audit.log Sample
Listing 1 shows a so-called raw log format: auditd
decodes kernel messages and persists them in the same form they arrive. There are no other options available out of the box, but you can ask audisp
to redirect messages to the syslog
if you wish.
You may now wonder what happens if auditd
is not running. It depends on the setting called "failure mode." If you are serious about security, the kernel may shut down the whole system. The default behavior is just to put a warning in the dmesg log, though. Similar options are available for when the system is low on disk space or a disk error occurs [2].
Despite the name, raw log format is mostly self-explanatory. As you see in Listing 1, each audit record consists of several key-value pairs or fields. type
sets the record type. Suppose a SYSTEM_BOOT
event occurs when the system boots, and DAEMON_START
fires when auditd
itself comes to life. Note it happens before the system is booted.
Different message types have slightly different fields yet many of them are common. Say, msg
stores both the timestamp and the message ID, colon-delimited. To query a record by its ID, you can do the following (#
denotes root command prompt):
# ausearch -a 1665 -i ---- type=DAEMON_START msg=audit(11/11/2016 18:18:58.665:1665) : auditd start, ver=2.4.5 format=raw kernel=4.4.0-47-generic auid=unset pid=406 subj=unconfined res=success
Here the -a
option tells ausearch
the ID of the record you are interested in, and -i
prescribes to convert numeric values, such as user identifier (UIDs) or timestamps, to their human-readable representations. Sometimes, this search yields more than one event record. Events related to a single operation (e.g., a system call) share the same message ID. Note that message IDs don't persist across system reboots.
You've probably guessed what pid
, uid
, gid,
etc. are for. auid
stands for Audit user ID and is also known as "login ID." You get one when you login, and it stays unchanged during the session. comm
is the command, in kernel's parlance. As you see, it could be different from the executable (exe
). success
, of course, tells the operation status, and the key
field acts as an administrator-defined tag; I'll show you an example.
To get an events summary, use the aureport
tool (Figure 2). The command also accepts a number of switches to build reports for specific event groups. For example, aureport --auth
reports authentication attempts (either successful or not), aureport --file
tells about files access, and aureport --comm
does the same for commands. Note that each detailed report is a table, normally having "event" as the last column. That's the ID you can use to lookup event details with ausearch
.
aureport
has some filtering capabilities built-in. You can limit the timespan with --start
and --end
switches. Both accept the date and the time (in this order) and are locale aware. Here, 11/12/2016 and 12.11.2016 are both valid, depending on which part if the globe you are, and refer to the same point in time. Shortcuts like now, today, recent (not more than 10 minutes ago), or this-week are also supported. You may also filter successful operations only with --success
[3].
The ausearch
querying capabilities are, as per name, more advanced. You can filter by message type or types (comma-separated), pid, command name, and user ID, including auid (aka login ID). Free-text search is available for string-based fields, such as file names, and ausearch
is clever enough to match whole words if needed. The later means systemd matches systemd, but not systemd-update-utmp, for instance. You can also filter by date range or operation status the same way you do in aureport
.
It is possible to pipe ausearch
output into aureport
, to make the latter read it instead of audit log. For example, this is how you build a neat summary for the events of interest:
# ausearch ... --raw | aureport --summary --file
Note the --raw
argument to ausearch
, which makes it output event records in raw form, as in the logs. (See the "Audit Beyond Commands" box for more information.)
Audit Beyond Commands
Userspace audit tools rely on two libraries, libaudit and libauparse, to do the low-level stuff. These libraries constitute the public API. This means you can use them in your programs as well. This is not something unusual. For example, when an init daemon wants to generate a SYSTEM_BOOT
event, it calls audit_log_user_message()
or a similar function from the libaudit
.
Administrators rarely code in C. They use scripting languages such as Bash or Python. For the former, userspace audit tools should cover most needs. If you code in Python, you'll be happy know that both libraries provide official bindings for Python 2 and 3. They are generated with SWIG and follow the native C API closely. There is no separate documentation, unfortunately: You have to consult libaudit
and libauparse
man pages directly. The good news is that these bindings should be available in your package manager as python-audit or something similar.
Your Game, Your Rules
As you can see, the default configuration already captures some interesting things. Yet there are many possibilities for fine tuning. The Linux audit framework lets you install rules that define events of interest. Static rules reside in /etc/audit/audit.rules
or /etc/audit/audit.rules.d
. When auditd
starts, its init script calls auditctl
to load these rules, so they are always in effect. You may also use the auditctl
command to install custom dynamic rules that you don't want to survive a system reboot. The syntax stays the same in both cases: You either store auditctl
command-line switches in the rules file or supply them to the command directly.
auditctl
is a multitool. It can do global configuration: disable audit, lock the subsystem so no one (even root) can change it, flush rules, or set the failure mode. It can also list rules or query audit status. And, of course, it can also manage your own custom audit rules!
Rules reside in one of four kernel lists and have one of two actions: never
, which suppresses event generation, or always
, which induces it. The most used list is exit
, which stores rules to run at the end of a system call. The user
list applies rules to events coming from the userspace. Recall that many userspace components, such as an init daemon, generate audit events, often unconditionally. This way, you can iron out these events. The exclude
list is used to filter events by type. The last list, task
, is rarely seen in practice.
Each rule also has a set of preconditions. They come in form of C-like comparisons (=
, !=
, &=,
and so on) either against fixed values (-F
) or between event fields themselves (-C
). All conditions must be met for an action to trigger. Finally, you can attach an arbitrary string (called a "key") to a rule. This helps to filter events produced by the rule: ausearch -k
does just that.
Remember that only processes with CAP_AUDIT_WRITE
capability (this means root in general) can send audit events from the userspace [4]. That's how you silence them for a while:
# auditctl -a user,never
The -a
option tells auditctl
to append a rule to the list; -A
prepends it, and -d
deletes. The order is important because rules are evaluated until the first match. Check that the rule was installed with auditctl -l
. Now, if you generate a userspace event with auditctl -m
, the kernel will happily ignore it.
Okay, this was an artificial example. Normally, you use auditctl
to install filesystem rules, called watches, or system call tracers.
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
New Steam Client Ups the Ante for Linux
The latest release from Steam has some pretty cool tricks up its sleeve.
-
Gnome OS Transitioning Toward a General-Purpose Distro
If you're looking for the perfectly vanilla take on the Gnome desktop, Gnome OS might be for you.
-
Fedora 41 Released with New Features
If you're a Fedora fan or just looking for a Linux distribution to help you migrate from Windows, Fedora 41 might be just the ticket.
-
AlmaLinux OS Kitten 10 Gives Power Users a Sneak Preview
If you're looking to kick the tires of AlmaLinux's upstream version, the developers have a purrfect solution.
-
Gnome 47.1 Released with a Few Fixes
The latest release of the Gnome desktop is all about fixing a few nagging issues and not about bringing new features into the mix.
-
System76 Unveils an Ampere-Powered Thelio Desktop
If you're looking for a new desktop system for developing autonomous driving and software-defined vehicle solutions. System76 has you covered.
-
VirtualBox 7.1.4 Includes Initial Support for Linux kernel 6.12
The latest version of VirtualBox has arrived and it not only adds initial support for kernel 6.12 but another feature that will make using the virtual machine tool much easier.
-
New Slimbook EVO with Raw AMD Ryzen Power
If you're looking for serious power in a 14" ultrabook that is powered by Linux, Slimbook has just the thing for you.
-
The Gnome Foundation Struggling to Stay Afloat
The foundation behind the Gnome desktop environment is having to go through some serious belt-tightening due to continued financial problems.
-
Thousands of Linux Servers Infected with Stealth Malware Since 2021
Perfctl is capable of remaining undetected, which makes it dangerous and hard to mitigate.