Connection Origin

One final thing that forensic investigators should not forget is that the origin of a TCP or UDP connection does not necessarily match the attacker's location. Some attackers use hijacked systems as a starting point for their work.

If the connection originates with a system that is very close to your own network, you should be extremely cautious. A short list of hops in the output from tcpdump target-address will tell you more. If you can rule out a regular user, you have fairly convincing evidence that an attacker has made inroads into your network. An attacker who is close is much more dangerous than one who is far away. (Sniffing passwords off the same subnet is much easier than off the Internet.)

Looking for Clues

If the forensics expert has discovered an unknown process or program running on the system, the next question is: "What does the malware actually do?"

Is it just a jumping-off point for more attacks? If so, an unknown process will probably have created an entry in the socket list.

Does it sniff data off the network? If so, look for a network interface in promiscuous mode.

Typically, this creates an audit entry in the kernel ring buffer, and you can run dmesg to find out:

device eth0 entered promiscuous mode
audit(1408381411.504:2): dev=eth0 prom=256 old_prom=0 auid=4267295
device eth0 left promiscuous mode
audit(1408381413.144:3): dev=eth0 prom=0 old_prom=256 auid=4297295

What are you supposed to do if you see an active process but don't know what it does? To start, it makes sense to start by backing up the process itself. To locate the executable, type ps gauxwww or check /proc/PID/cmdline.

What are you supposed to do if an attacker has launched a tool, immediately deleted it, and overwritten the disk sectors? While the program is running, there is hope – the kernel keeps a virtual symlink to the executable in /proc/PID/exe, even if the attacker has deleted it from the filesystem. If the response team saves this file somewhere safe, analysis is often possible at a later stage.

Poking in the Binary Trash

One simple but effective approach is to take a closer look at the binary itself. The strings -a binary command searches a file for printable characters. If the malware connects to an FTP or web server that requires a password, you might be able to find the password in the program code. But you will need a modicum of intuition to distinguish digital bread crumbs from binary trash.