/proc Talk

The proc filesystem [1] (procfs for short), is a Linux pseudo-filesystem that provides an interface to the operating system's kernel data structures. Procfs leverages the well-known concept of "in Unix, everything is a file" [2] to provide the same uniform interface of Unix file I/O (e.g., open, read, write, close, etc.) for getting kernel- and OS-related information. This uniformity makes it easier for the Linux programmer or system administrator to learn about the kernel, with fewer interfaces to learn.

Procfs is usually mounted at /proc. Many kinds of information about the operating system (OS) and processes running in it are exposed via pseudo-files in procfs. By reading data from /proc files, you can learn a lot of useful information about the system.

This article shows some ways of getting information from procfs using custom Python programs and Linux commands.

For comprehensive details about the information each pseudo-file in procfs provides, please refer to the man page for the /proc filesystem [3]. I will focus on showing practical examples, along with a few fun shell features and tricks.

More on procfs

Procfs is automatically mounted by the kernel at boot time under the /proc mountpoint.

The fragment (since Linux 3.3)

hidepid=n

specifies a mount option. The values for n can be  , 1, or 2, with   being the most lenient, and 1 and 2 being progressively stricter modes of access with respect to security and privacy of information. Mode   is the default.

Many categories of files and directories reside under the /proc hierarchy. Each category serves a different purpose.

Table 1 shows the categories of proc files and what the kind of information each type provides. (In Table 1 and in the examples below, [pid] is a placeholder for the process ID of the process for which you want information.)

Procfs provides two kinds of OS information: process-specific and non-process-specific, or general. The first three items in Table 1 are process-specific information, and the last item is an example of general OS information.

In the rest of this article, I reveal the details of the kind of information each of the categories in Table 1 contains, as well as some examples of what you can do with this information.

Get the Number of Numerically Named Directories

Procfs contains a numerically named subdirectory for each running process; the subdirectory is named by the process ID (PID):

/proc/[pid]

The number of numerically named directories (each representing a running process named by the process's PID) is an indication of the number of running processes:

$ ls -ld /proc/[0-9]* | wc -l
84

The command means: list (ls) the files matching the given file name pattern, and count the number of lines in the output (wc -l). Only the directory names (the d in -ld) are listed in long format (the l in -ld). Only files with names made up of just one or more digits ([0-9]*) are counted.

Get the Command Line of a Process

Files of the form

/proc/[pid]/cmdline

hold the complete command line for the process. The components of the command line appear in this file as a sequence of strings, each terminated by a null byte (ASCII code 0, which is character \0).

Figure 1 shows how to get the command line of a process with a Python script. The get_proc_cmdline.py script is shown in Listing 1. The script takes a PID as input and outputs the command line for the process, as referenced in the /proc/[pid]/cmdline file. As you can see in Listing 1, get_proc_cmdline.py refers to proc_info.py (Listing 2) and error_exit.py (Listing 3).

Figure 1: Getting the command line of a process using a Python script.

01 # A program to get the command lines of processes, given their PIDs.
02
03 from __future__ import print_function
04 import sys
05 from proc_info import read_proc_cmdline
06
07 from error_exit import error_exit
08
09 def main():
10     if len(sys.argv) < 2:
11         error_exit("{}: Error: Need at least one PID to process.\n".format(sys.argv[0]))
12     pids = sys.argv[1:]
13     print("Getting command lines for these PIDs:\n{}".format(' '.join(pids)))
14     for pid in pids:
15         proc_filename = "/proc/{}/cmdline".format(pid)
16         ok, result = read_proc_cmdline(proc_filename)
17         if ok:
18             sys.stdout.write("PID: {} Command line: {}\n".format(pid, result))
19         else:
20             sys.stderr.write("PID: {} Error: {}\n".format(pid, result))
21
22 if __name__ == '__main__':
23     main()

01 # A module with functions to get information from files
02 # in the /proc pseudo file system for various purposes.
03
04 from __future__ import print_function
05 import sys
06 import string
07 import pwd
08
09 from error_exit import error_exit
10
11 def read_proc_cmdline(proc_filename):
12     """
13     Function to read the command-line of a process, given its proc_filename.
14     Returns a tuple: first item of tuple is a boolean, True if successful,
15     False if not; second item of tuple is the result (cmdline) if first item
16     is True, or an error message if first item is False.
17     """
18
19     try:
20         with open(proc_filename, 'r') as proc_fil:
21             # Read cmdline value.
22             data = proc_fil.read()
23             # Make it printable.
24             ret_val = (True, data.replace('\0', ' '))
25     except IOError as ioe:
26         ret_val = (False, "IOError while opening proc file: {} ".format(str(ioe)))
27     except Exception as e:
28         ret_val = (False, "Exception: {}".format(str(xe)))
29     finally:
30         return ret_val
31
32 def read_proc_environ(proc_filename):
33     """
34     Function to read the environment of a process, given its proc_filename.
35     Returns a tuple: first item of tuple is a boolean, True if successful,
36     False if not; second item of tuple is the result (environ) if first item
37     is True, or an error message if first item is False.
38     """
39     try:
40         with open(proc_filename, 'r') as proc_fil:
41             # Read environ value.
42             data = proc_fil.read()
43             # Make it printable.
44             ret_val = (True, data.replace('\0', '\n'))
45             return ret_val
46     except IOError as ioe:
47         ret_val = (False, "IOError while opening proc file: {}".format(str(ioe)))
48     except Exception as e:
49         ret_val = (False, "Exception: {}".format(str(ioe)))
50     finally:
51         return ret_val
52
53 def read_proc_status(proc_filename):
54
55     """
56     Function to read the status of a process, given its proc_filename.
57     Returns a tuple: first item of tuple is a boolean, True if successful,
58     False if not; second item of tuple is the result (status) if first item
59     is True, or an error message if first item is False.
60     """
61     try:
62         with open(proc_filename, 'r') as proc_fil:
63             # Read desired status fields and values.
64             proc_status = {}
65             for lin in proc_fil:
66                 if lin.startswith(('Name:', 'State:', 'Pid:', 'PPid:', 'Uid:', 'Gid:')):
67                     parts = lin.split()
68                     assert len(parts) > 1
69                     proc_status[parts[0]] = parts[1]
70             assert 'Uid:' in proc_status
71             # Get username for uid.
72             uid = proc_status['Uid:']
73             pwent = pwd.getpwuid(int(uid))
74             proc_status["Username:"] = pwent.pw_name
75             ret_val = (True, proc_status)
76     except ValueError as ve:
77         ret_val = (False, "ValueError in read_proc_status(): {}".format(str(ve)))
78     except IOError as ioe:
79         ret_val = (False, "IOError in read_proc_status(): {}".format(str(ioe)))
80     except Exception as e:
81         ret_val = (False, "Exception in read_proc_status(): {}".format(str(ioe)))
82     finally:
83         return ret_val

01 # error_exit.py
02
03 # Purpose: This module, error_exit.py, defines a function with
04 # the same name, error_exit(), which takes a string message
05 # as an argument. It prints the message to sys.stderr, or
06 # to another file object open for writing (if given as the
07 # second argument), and then exits the program.
08 # The function error_exit can be used when a fatal error condition occurs,
09 # and you therefore want to print an error message and exit your program.
10
11 import sys
12
13 def error_exit(message, dest=sys.stderr):
14     dest.write(message)
15     sys.exit(1)
16
17 def main():
18     error_exit("Testing error_exit with dest sys.stderr (default).\n")
19     error_exit("Testing error_exit with dest sys.stdout.\n",
20         sys.stdout)
21     with open("temp1.txt", "w") as fil:
22         error_exit("Testing error_exit with dest temp1.txt.\n", fil)
23
24 if __name__ == "__main__":
25     main()

The next example uses echo_args.py, a Python program that echoes its arguments to the standard output

To see get_proc_cmdline.py at work, consider the example script in Listing 4, echo_args.py, which is a test script that echoes the command-line arguments used to call the script to standard output, then sleeps for a while, giving the user some time to run another program that reads the command line of the process running echo_args.py. Run echo_args.py with arguments, using some combinations of backslashes and quoting to show how they are interpreted by the shell before the Python program receives them. A trailing ampersand & after the command starts a new background process and outputs the process ID. For example,

$ python echo_args.py arg1 "arg 2" arg\ 3 'arg 4' "arg\ 5" &
[1] 2943
$ echo_args.py|arg1|arg 2|arg 3|arg 4|arg\ 5

outputs the PID 2943.

01 # echo_args.py
02 # Echoes the command-line arguments to the standard output.
03
04 from __future__ import print_function
05
06 import sys
07 import time
08
09 print("Arguments separated by | signs:")
10 print('|'.join(sys.argv))
11
12 # Sleep for 10 minutes; any reasonable time will do.
13 # It just needs to be enough for us to go to another terminal
14 # and run the command to read this process's arguments via /proc.
15 time.sleep(600)

You can then use get_proc_cmdline.py to get the command line for the process with PID 2943:

$ python get_proc_cmdline.py 2943
Getting command lines for these PIDs:
2943
PID: 2943 Command line: python echo_args.py arg1 arg 2 arg 3 arg 4 arg\ 5

Tip: The value of the $! built-in shell variable is the PID of the last background process started, so if you are sure that the command python echo_args.py is the last background process run on this terminal, you can use $! instead of the literal value 2943 for the PID.

Get the Environment of a Process

Another useful trick is to list the set of environment variables and their values. Files of the form /proc/[pid]/environ contain the initial environment that was set up when the process with PID [pid] was started. Caveat: It is a snapshot, so if the process changes some of the environment variables later using the putenv (C) or os.putenv (Python) library functions or equivalent, those changes will not be reflected in the file's contents. The entries are of the form var=value and are terminated by null bytes (\0). So, to print out the environment of process 1234, you could enter

$ strings /proc/1234/environ

where strings is a Linux command that prints only the strings of printable characters in any file (including binary files); for example:

# strings /proc/$$/environ
TERM=xterm
MAIL=/var/mail/root
HOME=/root
SHELL=/bin/bash
USER=root
LOGNAME=root
(some lines of output deleted)

(See man strings on your local Linux system for more on the strings command.) The get_proc_environ.py script in Listing 5 gets the environment of a process and is similar to get_proc_cmdline.py. The format of the data they both retrieve is the same (both use null-terminated strings), the overall program structure is the same, and the output and error messages are similar. The script refers to proc_info.py (Listing 2), as well as read_proc_environ.py (Listing 6).

01 # A program to get the environments of processes, given their PIDs.
02
03 from __future__ import print_function
04 import sys
05 from proc_info import read_proc_environ
06
07 from error_exit import error_exit
08
09 def main():
10     if len(sys.argv) < 2:
11         error_exit("{}: Error: Need at least one PID to process.\n".format(sys.argv[0]))
12     pids = sys.argv[1:]
13     print("Getting environment for these PIDs:\n{}".format(' '.join(pids)))
14     for pid in pids:
15         proc_filename = "/proc/{}/environ".format(pid)
16         ok, result = read_proc_environ(proc_filename)
17         if ok:
18             sys.stdout.write("PID: {}\nEnvironment:\n{}".format(pid, result))
19         else:
20             sys.stderr.write("PID: {} Error: {}\n".format(pid, result))
21
22 if __name__ == '__main__':
23     main()

01 function
02 import sys
03
04 pid = sys.argv[1]
05 print("Trying to get environment for process with PID:", pid)
06
07 try:
08     #print(open('/proc/{}/cmdline'.format(pid), 'r').read().replace('\0', ' '))
09     filename = '/proc/{}/environ'.format(pid)
10     fil = open(filename, 'r')
11     env_with_nulls = fil.read()
12     env_with_newlines = env_with_nulls.replace('\0', '\n')
13     print("Process with PID: {} has command-line:\n{}".format(pid, env_with_newlines))
14 except IOError as ioe:
15     sys.stderr.write("Caught IOError while opening file {}:\n{}\n".format(filename, str(ioe)))
16 except Exception as e:
17     sys.stderr.write("Caught Exception: {}".format(str(e)))

To get the environment of a process, run the get_proc_environ.py script:

$ python get_proc_environ.py $$
Getting environment for these PIDs:
3807
PID: 3807
Environment:
USER=vram
HOME=/home/vram
MAIL=/var/mail/vram
SHELL=/bin/bash
TERM=xterm
(some lines of output deleted)

Note that the command uses $$ to represent the PID of the current shell instance.

If you try to get the environment for the init process (PID 1) or other processes that require superuser access, you'll need to use su:

$ su
Password:
# python get_proc_environ.py 1
Getting environment for these PIDs:
1
PID: 1
Environment:
HOME=/
TERM=linux
PATH=/sbin:/usr/sbin:/bin:/usr/bin
PWD=/
(some lines of output deleted)

Getting the Process Status

Files of the form /proc/[pid]/status describe the status of the running process. The Linux ps command itself uses this file to get process status information.

The status of a process contains some of the same information provided with the ps command, but in this case, you are getting it programmatically, and you also get other information, such as the username of the user in whose name the process is running.

The /proc/[pid]/status file for a process contains many field_name: field_value pairs. Each pair gives information about some aspect of the process.

The following are a few useful pieces of status data:

• the process name (the Name: line)
• the process ID (Pid:)
• the parent process ID (PPid:)
• the process user ID (Uid:)
• the process group ID (Gid:)
• the process user name (this field is not in the status information; I get it from the /etc/passwd file using the user ID as the key)

The awk script that fetches the status data is shown in Listing 7. The script gets the required fields and their values from any file of the form /proc/[pid]/status, with the file name given as a command-line argument.

01 # proc_status.awk
02 FILENAME != OLDFILENAME {print ""; OLDFILENAME = FILENAME}
03 /^Name:/ {print "Name:", $2}
04 /^State:/ {print "State:", $2}
05 /^Pid:/ {print "Pid:", $2}
06 /^PPid:/ {print "PPid:", $2}
07 /^Uid:/ {print "Uid:", $2}
08 /^Gid:/ {print "Gid:", $2}

The first line in Listing 7 prints a blank line as a separator whenever the input file name changes. FILENAME is a built-in awk variable representing the current input file name. OLDFILENAME is a variable defined by me. Because OLDFILENAME is compared with FILENAME, which is a string variable, OLDFILENAME is also treated as a string variable.

String variables are initialized by default to an empty string (""), so from the result of the evaluation of the pattern, that first script line prints a blank line in all cases, whether you have one input file name or many. (print "" prints an empty line.) Also, each time the input file name changes, that first line updates OLDFILENAME to be equal to the new value of FILENAME.

I use the status field names as patterns inside slash characters, and the corresponding actions in braces are run for each line in the input (to be shown) where the pattern matches the line. The caret (^) at the start of each pattern tells awk to anchor the pattern to the beginning of the line.

The patterns match the desired lines from the input, and the second field ($2) of each line is printed with an appropriate text label before it.

To get the status information for the current process, the line

$ awk -f proc_status.awk /proc/$$/status

outputs:

Name: bash
 State: S
 Pid: 2123
 PPid: 2122
 Uid: 1002
 Gid: 1004

The State field in the output shows the current state of the process. It can have one of the following values: R (running), S (sleeping), D (disk sleep), T (stopped), t(tracing stop), Z (zombie), or X (dead).

The -f option of awk says to read the script (to be run) from the file name that follows the option. The next argument to awk is the file name from which to get the input. In this case, it is the procfs pseudo-file that holds the status information from the current process ($$).

Adding the username in the output is slightly complex (Listing 8).

01 # proc_status_with_username.awk
02 FILENAME != OLDFILENAME { print ""; OLDFILENAME = FILENAME }
03 /^Name:/ { print "Name:", $2 }
04 /^State:/ { print "State:", $2 }
05 /^Pid:/ { print "Pid:", $2 }
06 /^PPid:/ { print "PPid:", $2 }
07 /^Uid:/ {
08     print "Uid:", $2;
09     uid=$2;
10     printf "Username: "
11     system("awk -F: ' $3 == '"uid"' { print $1; exit } ' /etc/passwd")
12 }
13 /^Gid:/ { print "Gid:", $2 }

The main change between Listing 7 and Listing 8 is the change to the line with the pattern ^Uid:; instead of printing just "Uid:", $2, it has a block of four statements between braces (lines 7-12). Those statements are executed instead of the previous single statement whenever the pattern matches.

Awk's -F: option (line 11) sets the input field delimiter to a different value from the default – here a colon, because that is the delimiter used in the password file.

Those four statements between the braces (in Listing 8) do the following, with the last statement performing a bit of shell quoting magic:

• print the text Uid: and $2, the second field (the user ID);
• set variable uid equal to $2;
• print the text Username: (with a space after) without a newline; hence, the printf, not print, because printf does not add a newline unless asked to);
• use the awk built-in function system to run another instance of awk as a child process; that instance has a pattern that tries to match the third field of the current line of input (the user ID) from the password file with the value of variable uid, and if it matches, it prints the first field of that line (the user name) and then exits.

The end result, therefore, is a call to an inner awk script in the middle of the outer awk script, which fetches the user name and inserts it into the middle of the outer script's output.

Because awk supports multiple file name arguments (as any well-written Linux filter should), you can run the script in Listing 8 with more than one file name.

Figure 2 shows how to get the status of processes using awk.

Figure 2: Getting the status of processes using awk.

You can also get the status output using a Python program. See the get_proc_status.py program with the listings for this article at the Linux Magazine website [4].

Conclusion

In this article, I described how to get useful information from the Linux /proc filesystem using shell commands, Python scripts, and awk. A couple of ideas for further exploration of procfs are:

• Use the files named /proc/[pid]/io to get information about I/O being performed by a process (e.g., the progress of a file tree copy).
• Get some of the /proc filesystem information remotely from another machine using a distributed computing technology, such as HTTP REST calls or XML-RPC.

Exploring the /proc filesystem will give you a deeper understanding of Linux, and along the way, you'll get some useful practice with scripting and classic command-line tools.