Audit Your Linux Box

Core Technology

Article from Issue 195/2017
Author(s):

Look for intruders and study the health of your system with Linux auditing tools.

No one enjoys being tracked. In Free Software and Linux, we take privacy very seriously. Yet, we sometimes set surveillance cameras to watch the back yard. We hardly ever look at the recordings, unless things go wrong. Then we could use videos to learn who broke that window.

Audit in Linux works much the same way. It captures security-related events, such as file access, system calls, user logins, or system reboots. Then it stores these logs safely and lets you search through them. This process doesn't add any security by itself, but it helps to track intruders. Having this is a prerequisite to Common Criteria certification, and it's a good way to peek into the system's operation for learning, fun, and profit.

The Big Picture

The Linux audit framework spans multiple components, both in userspace and in the kernel (Figure  1).

Figure 1: The Linux audit framework: major components and their interactions. Arrows indicate possible data flow.

A story begins when something generates an audit event. For instance, when you open a file, the kernel can learn about it using the very same mechanism covered previously [1]. Or, if you want to trace system calls, the kernel raises a flag on a process descriptor, which causes Linux to fire an event when it returns from the call.

Not all audit events originate from the kernel, though. When the system boots, switches run levels, starts or stops daemons, etc., an init process such as systemd logs these life cycle events. This also happens when a user logs in or out. Technically, such messages end up in the kernel as well, so all other audit framework components have a single events source.

The kernel reports audit events via a Netlink socket. In userspace, a daemon called auditd reads them and stores them in audit log file. auditd can also forward events to a dispatcher daemon, audisp. This component acts as an event multiplexer for programs, which want to analyze events in real time, such as Intrusion Detection/Prevention Systems (IDS/IPS). You can also use audisp to send events to storages other than local disk, such as remote servers. This helps keep them secure. Even though audit logs are inaccessible to users other than root, an intruder with sufficient permissions can still tamper or destroy them.

Many things are happening in your system at any given moment. Logging all of them would probably be overkill and a waste of resources. How does the kernel know which one to remember and which one to forget? Audit rules tell it. These rules define which files, directories, and system calls the kernel should monitor. You also use rules to select processes to trace or users whose actions you want to follow. auditctl communicates rules to the kernel, either during the system startup or dynamically in run time.

Now, with events recorded somewhere, how do you get a sense of them? aureport tool comes to the rescue. It produces a summary or detailed report upon your request. There are also ways to visualize these reports, as you'll see shortly. If you look for a specific event, use the ausearch tool. A common pattern is to find an event of interest in aureport (Figure 2), and then take its event ID and retrieve the details with ausearch.

Figure 2: Sample aureport output, which shows a detailed system call report on a vanilla Kubuntu 16.04 LTS box.

Getting Your Feet Wet

For starters, let's look at how all these pieces work together using vanilla Kubuntu 16.04 LTS as an example. Once you boot the system, the audit log, which resides in /var/log/audit.log, should already have some events (Listing 1).

Listing 1

audit.log Sample

 

Listing 1 shows a so-called raw log format: auditd decodes kernel messages and persists them in the same form they arrive. There are no other options available out of the box, but you can ask audisp to redirect messages to the syslog if you wish.

You may now wonder what happens if auditd is not running. It depends on the setting called "failure mode." If you are serious about security, the kernel may shut down the whole system. The default behavior is just to put a warning in the dmesg log, though. Similar options are available for when the system is low on disk space or a disk error occurs [2].

Despite the name, raw log format is mostly self-explanatory. As you see in Listing 1, each audit record consists of several key-value pairs or fields. type sets the record type. Suppose a SYSTEM_BOOT event occurs when the system boots, and DAEMON_START fires when auditd itself comes to life. Note it happens before the system is booted.

Different message types have slightly different fields yet many of them are common. Say, msg stores both the timestamp and the message ID, colon-delimited. To query a record by its ID, you can do the following (# denotes root command prompt):

# ausearch -a 1665 -i
----
type=DAEMON_START msg=audit(11/11/2016 18:18:58.665:1665) : auditd start, ver=2.4.5 format=raw kernel=4.4.0-47-generic auid=unset pid=406 subj=unconfined res=success

Here the -a option tells ausearch the ID of the record you are interested in, and -i prescribes to convert numeric values, such as user identifier (UIDs) or timestamps, to their human-readable representations. Sometimes, this search yields more than one event record. Events related to a single operation (e.g., a system call) share the same message ID. Note that message IDs don't persist across system reboots.

You've probably guessed what pid, uid, gid, etc. are for. auid stands for Audit user ID and is also known as "login ID." You get one when you login, and it stays unchanged during the session. comm is the command, in kernel's parlance. As you see, it could be different from the executable (exe). success, of course, tells the operation status, and the key field acts as an administrator-defined tag; I'll show you an example.

To get an events summary, use the aureport tool (Figure 2). The command also accepts a number of switches to build reports for specific event groups. For example, aureport --auth reports authentication attempts (either successful or not), aureport --file tells about files access, and aureport --comm does the same for commands. Note that each detailed report is a table, normally having "event" as the last column. That's the ID you can use to lookup event details with ausearch.

aureport has some filtering capabilities built-in. You can limit the timespan with --start and --end switches. Both accept the date and the time (in this order) and are locale aware. Here, 11/12/2016 and 12.11.2016 are both valid, depending on which part if the globe you are, and refer to the same point in time. Shortcuts like now, today, recent (not more than 10 minutes ago), or this-week are also supported. You may also filter successful operations only with --success [3].

The ausearch querying capabilities are, as per name, more advanced. You can filter by message type or types (comma-separated), pid, command name, and user ID, including auid (aka login ID). Free-text search is available for string-based fields, such as file names, and ausearch is clever enough to match whole words if needed. The later means systemd matches systemd, but not systemd-update-utmp, for instance. You can also filter by date range or operation status the same way you do in aureport.

It is possible to pipe ausearch output into aureport, to make the latter read it instead of audit log. For example, this is how you build a neat summary for the events of interest:

# ausearch ... --raw | aureport --summary --file

Note the --raw argument to ausearch, which makes it output event records in raw form, as in the logs. (See the "Audit Beyond Commands" box for more information.)

Audit Beyond Commands

Userspace audit tools rely on two libraries, libaudit and libauparse, to do the low-level stuff. These libraries constitute the public API. This means you can use them in your programs as well. This is not something unusual. For example, when an init daemon wants to generate a SYSTEM_BOOT event, it calls audit_log_user_message() or a similar function from the libaudit.

Administrators rarely code in C. They use scripting languages such as Bash or Python. For the former, userspace audit tools should cover most needs. If you code in Python, you'll be happy know that both libraries provide official bindings for Python 2 and 3. They are generated with SWIG and follow the native C API closely. There is no separate documentation, unfortunately: You have to consult libaudit and libauparse man pages directly. The good news is that these bindings should be available in your package manager as python-audit or something similar.

Your Game, Your Rules

As you can see, the default configuration already captures some interesting things. Yet there are many possibilities for fine tuning. The Linux audit framework lets you install rules that define events of interest. Static rules reside in /etc/audit/audit.rules or /etc/audit/audit.rules.d. When auditd starts, its init script calls auditctl to load these rules, so they are always in effect. You may also use the auditctl command to install custom dynamic rules that you don't want to survive a system reboot. The syntax stays the same in both cases: You either store auditctl command-line switches in the rules file or supply them to the command directly.

auditctl is a multitool. It can do global configuration: disable audit, lock the subsystem so no one (even root) can change it, flush rules, or set the failure mode. It can also list rules or query audit status. And, of course, it can also manage your own custom audit rules!

Rules reside in one of four kernel lists and have one of two actions: never, which suppresses event generation, or always, which induces it. The most used list is exit, which stores rules to run at the end of a system call. The user list applies rules to events coming from the userspace. Recall that many userspace components, such as an init daemon, generate audit events, often unconditionally. This way, you can iron out these events. The exclude list is used to filter events by type. The last list, task, is rarely seen in practice.

Each rule also has a set of preconditions. They come in form of C-like comparisons (=, !=, &=, and so on) either against fixed values (-F) or between event fields themselves (-C). All conditions must be met for an action to trigger. Finally, you can attach an arbitrary string (called a "key") to a rule. This helps to filter events produced by the rule: ausearch -k does just that.

Remember that only processes with CAP_AUDIT_WRITE capability (this means root in general) can send audit events from the userspace [4]. That's how you silence them for a while:

# auditctl -a user,never

The -a option tells auditctl to append a rule to the list; -A prepends it, and -d deletes. The order is important because rules are evaluated until the first match. Check that the rule was installed with auditctl -l. Now, if you generate a userspace event with auditctl -m, the kernel will happily ignore it.

Okay, this was an artificial example. Normally, you use auditctl to install filesystem rules, called watches, or system call tracers.

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy Linux Magazine

SINGLE ISSUES
 
SUBSCRIPTIONS
 
TABLET & SMARTPHONE APPS
Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

  • Security Lessons: auditd

    The auditd tool can provide system logging capabilities to satisfy even the most paranoid users.

  • Integrity Measurement Architecture

    The Integrity Measurement Architecture adds important details to your audit logs, making it easier to track an intruder's footprints.

  • SELinux

    SELinux provides a comprehensive Mandatory Access Control system for Linux, if you are ready for all the details.

  • FreeIPA

    FreeIPA offers integrated identity management and big ideas for the future.

comments powered by Disqus
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters

Support Our Work

Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.

Learn More

News