Crime scene: the server room…The thief doesn't need a key card or the protection of darkness – an intruder can use the Internet to come and go. But despite the secret entrance, the attacker still leaves behind some telltale traces. Finding and interpreting this evidence is the top priority of criminal investigators.

This month's cover story explores the world of computer forensics. We'll show you some tools the experts use to find clues, recover deleted files, and root out hidden evidence. We start with a study of the open source Sleuth Kit forensics toolkit. We also look at Foremost and Scalpel – a pair of tools for finding and restoring deleted files. We show you how to examine Windows disks with Linux tools, and we end with a look at the Open Computer Forensics Architecture, a freely available collection of forensics tools and libraries developed for the Dutch police.

But if you're not really going to trial and you just want to catch the intruder on your system, you might not want to go to all the trouble of launching a full forensics investigation. The following sections describe some tips for finding intruders on the system using standard Linux utilities.

In Your Face, or Clandestine?

One of the first questions a forensics investigator must ask is whether the investigation should be performed openly – which means that it will be visible to the attacker, too – or the attacker should be kept unaware of the investigation.

A computer under forensic investigation is very similar to a particle in quantum mechanics: just looking at it changes the state.

An attacker might see a ps command, and running find against the hard disk typically overwrites valuable atime records on the filesystem, removing evidence of a user's last access.

Despite the possible complications of working in the open, the need to get to the bottom of illicit activities is sometimes more important than taking elaborate steps to avoid notice.

Also, keep in mind that most attacks are launched through automated scripts and programs, thus, it is unusual to catch an attacker red-handed at the console. The following tips are primarily for cases in which you don't really care about concealing your activities or keeping a paper trail.

To avoid overlooking details, a systematic approach is useful. The idea of following a hot trail is often very seductive, but if it takes you to a dead end, you will be disappointed.

For example, if you investigate a list of processes using the command

ps gauxwww

you should store and work through the full list. The command shows all of the active processes and their command-line arguments, including full options.

Of course, if your system has been compromised, it is always possible an intruder has installed trojaned versions of system utilities, such as ps, to conceal the break-in.

A small shell script does the same job by reading the /proc data (see Listing 1).

01 #!/bin/sh
02
03 cd /proc
04 for p in [0-9]*
05 do
06     proc=$(cat $p/cmdline)
07     user=$(ls -ld $p | cut -d\  -f3)
08
09     echo "$user $p $proc"
10 done

Individual extensions are easily added to a script like this and can be particularly useful if you can't trust ps.

For a good sanity check, you need to crosscheck the results using a tool similar to the popular pstree. Forensic investigators will also remember that programs can change argv[], their argument list, programmatically (see Listing 2).

01 /*
02    Build as follows
03    gcc -o changecommand changecommand.c
04    Waits for three seconds, then changeas its
05    command line, waits another three seconds
06    and terminates. The specified commands
07    are not executed, but they do frighten
08    the administrator running the ps command.
09
10 */
11
12 void overwrite(char *arg, char *new) {
13     char w;
14
15     while (*arg)
16     {
17         if (*new)
18             w = *new++;
19         else
20             w = 0x00;
21         *arg++ = w;
22     }
23 }
24
25 int main(int argc, char **argv)
26 {
27     char a0[] = "/bin/rm";
28     char a1[] = "-fr";
29     char a2[] = "*";
30
31     usleep(3000000);
32
33     overwrite(argv[0], a0);
34     overwrite(argv[1], a1);
35     overwrite(argv[2], a2);
36
37     usleep(3000000);
38
39     return 0;
40 }

Kernel

A simple trick like searching for processes is powerless against a kernel rootkit. Rootkits modify the kernel to prevent it from delivering information about certain processes to the /proc filesystem or other information mechanisms.

On the other hand, it is very surprising how little trouble some attackers take to cover their tracks, so it might be worth trying.

Network Connections

Besides processes, network connections can also reveal clues, such as the attack vector and the address the attacker used to connect to the system.

Issuing netstat --ip -pan on Linux shows you all the local IP sockets, their protocols (TCP or UDP), and possibly the communication partners for connected sockets – unless the command or the kernel happen to have been manipulated.

Setting the -n option in netstat prevents DNS from resolving IP addresses and giving you the matching hostnames. This is a good idea because it avoids unnecessary and suspicious network traffic to the name server.

If necessary, you can always resolve the IP addresses later.

The whois and traceroute commands display more information about IP addresses. whois queries one of several Internet databases to reveal the registration data for the network scope.

Typically, these details are very trustworthy and difficult for attackers to spoof without the cooperation of an Internet service provider.