Ssssuper

Incidentally, for comparison, another open port lookup tool succinctly called ss (apparently, socket statistics) only took 0.054 seconds (Listing 2).

The frisky systems tool ss is super-lightweight but examining its innards with an autopsy are for another day, so I'll get back to lsof as promised.

Clearly, I have a bias toward lsof, but in the next example netstat isn't too far behind in terms of output. Figure 2 shows the output for netstat -a, which shows both listening and non-listening socket information, piped through grep.

Figure 2: Netstat somewhat crudely (and slowly) searching for the keyword ssh.

Grepping with ssh outputs the service name. Adding the -n switch to netstat would show non-translated services and IP addresses. If you then grep for port  22, the output is far noisier than lsof, as Figure  3 shows.

The noisy output in Figure 3 helps demonstrate why I still always find myself turning to lsof. Admittedly, a combination of switches, with which I am as yet unfamiliar, could probably make netstat much quieter.

Figure 3: Netstat can be noisy, even if you try to limit the output.

The Proof Is in the Pudding

Don't get me wrong; as I've already said, there's little doubt that if you just run lsof without adding switches, it's a pretty noisy little utility, too. (Try running the lsof command to see ALL the open files on your system if you want proof.) Having looked at netstat for a moment, I'll now look at some exceptionally useful crime-fighting functionality, courtesy of the friendly neighbourhood lsof.

In Figure 1, you saw the output after asking lsof to check TCP port 22. Now I'll reverse engineer that process and imagine that I'm troubleshooting an SSH daemon problem. Suppose you've discovered a suspicious instance of SSH running on a port that you definitely didn't set up and that might be the result of a system compromise. Also imagine that port 22 is really 22222, for example. In such a scenario, the first step would be spotting SSH in the process table, as you see in Figure 4 using the ever-faithful ps -ef to show process information with just the right level of detail.

Figure 4: Finding the PID of the SSH daemon.

Now that you're armed with the magic number (494 is the PID in this case), you can ask lsof to tell you what it knows about everything on the filesystem relating to that PID. And, boy, can it tell you some things (Figure 5). You should ignore the .gvfs hidden file error at the top, which apparently relates to a bug in FUSE not allowing even the superuser to view the mountpoint located at /home/chris/.gvfs (Figure 6).

Figure 5: Everything relating to PID 494.

Figure 6: A FUSE mountpoint that apparently contributes to the error generated in Figure 5.

Not only can you see in Figure 5 the TCP ports (the SSH daemon has both an IPv6 and an IPv4 instance) at the bottom, but the executable that fired up the daemon is also mentioned (third significant line down with the name /usr/sbin/sshd). Finally – and this will hopefully appear more relevant later as I explain further – you can see all the open files relating to that process and the significant libraries.

A more relaxed way of displaying this information can be achieved by showing all processes executing a command called ssh:

# lsof -c ssh

In Figure 7, the output is relatively verbose, hence the tiny font. I hope you can still make out the difference between this output and that in Figure 5.

Figure 7: Looking for all processes executing a command starting with 'ssh'.

The additional content is the result of running the command on an Ubuntu desktop, which is why ssh-agent, usually associated with X sessions and less commonly found on servers, is listed.

Less Is More

The formidable lsof just keeps on giving, and at a controlled level of detailed output that limits eye strain. I'll look at a few of the other options available now.

So far, I've looked at ports, PIDs, and process names, but what about per user information, which is the same as owner or, more accurately, who spawned the process? I have chosen the user daemon that, on a desktop as well as a server, should give some useful output. In this case, /var/spool/cron/atjobs and /usr/sbin/atd show that the atd command is dutifully run by the user daemon (see Figure 8).

Figure 8: The lsof -u daemon command lists a number of files marked as being open for the user daemon.

With the mention of the ever-popular cron, you would be correct in thinking that atd does indeed schedule tasks to run, as the cron daemon does. The main difference between the two is apparently that atd jobs just run once, and to re-use them, you have to start them again. Cron however just keeps on trucking, repeatedly, until told to stop re-running its scheduled tasks.