Watching Grampa

Some excellent technologies and packages are available for monitoring computer hardware. For medium to large systems, the Simple Network Monitoring Protocol (SNMP) approach is usually the preferred solution. However if you have a smaller system with older or low-end servers, some excellent lightweight command-line monitoring utilities can be used instead.

These command-line utilities can be run remotely over Secure Shell (SSH) and the output parsed to return only the key data values, which can then be displayed graphically in a Node-RED web dashboard (Figure 1). In this article, I demonstrate examples that use the iostat utility [1] to monitor CPU utilization and the lm-sensors package [2] and hddtemp utility [3] to monitor temperatures on dashboards.

Figure 1: Sending commands over SSH to a Node-RED dashboard.

CPU Utilitization

The iostat utility is part of the sysstat package and is probably already loaded on your older systems. If not, it can be installed by:

sudo apt-get install sysstat

The iostat utility generates a report for CPU, device, and filesystem utilization. The output from this command can be parsed with some Bash statement to return just the key value of interest.

Listing 1 shows an example of how to grab the fourth line of output with sed (line 7) and parse the %idle value at the end of the line with awk (line 11) to get the sixth string item (line 12).

01 pete@lubuntu: ~$ iostat -c
02 Linux 4.15.0-72-generic (lubuntu)   2020-05-02  i686    (4 CPU)
03 avg-cpu: %user  %nice %system %iowait  %steal  %idle
04           8.37   0.03  3.33      1.29          86.97
05
06 # Get 4th line of the iostat output
07 pete@lubuntu : ~$ iostat -c | sed -n 4p
08           8.36   0.03  3.33      1.29  0.00    86.99
09
10 # Now get the 6th string
11 pete@lubuntu : ~$ iostat -c | sed -n 4p | awk '{print $6}'
12 87.02

Chip-Based Temperature

To install the hardware sensors in the lm-sensors package, enter:

sudo apt-get install lm-sensors

After the package is installed, the software needs to detect which sensors are available for monitoring:

sudo sensors-detect

This step presents a number of prompts about which sensors need to be scanned. Once the scan step is complete, the sensors command returns results for all the hardware it found.

Specific sensors can be shown with the command:

sensors <chip name>

Listing 2 shows an example of the sensors command being used to look at the dell_smm-virtual-0 chipset. The CPU temperature value (in line 5) can be parsed with grep (line 10), which looks for the line that contains "CPU'; then, awk (line 14) outputs the second item of the string (line 15).

01 pete@lubuntu: ~$ sensors dell_smm-virtual-0
02 dell_smm-virtual-0
03 Adapter: Virtual device
04 Processor Fan: 2687 RPM
05 CPU:            +40.0°C
06 Ambient:        +34.0°C
07 SODIMM:         +33.0°C
08
09 # Get the CPU temperature
10 pete@lubuntu: ~$ sensors dell_smm-virtual-0 | grep 'CPU'
11 CPU:            +40.0°C
12
13 # Return just the temperature
14 pete@lubuntu: ~$ sensors dell_smm-virtual-0 | grep 'CPU' | awk '{print $2}'
15 +40.0°C

Hard Drive Temperature

The hddtemp hard drive temperature monitoring package is installed with:

sudo apt-get install hddtemp

By default, hddtemp requires superuser rights, so to make the results available to non-superusers use:

sudo chmod u+s /usr/sbin/hddtemp

To see the temperature of a hard drive, enter its device name. For example, to see /dev/sda, use:

$ hddtemp /dev/sda
/dev/sda: WDC WD3200BPVT-75JJ5T0: 34°C

Again, you can use the awk command to parse the output to get just the temperature. For this example, the temperature is the fourth item of the string, so the command displays that value as shown here:

$ hddtemp /dev/sda | awk '{print $4}'
34°C

Remote Commands

Rather than having remote Linux servers send data to a central monitoring node, the central node can periodically poll the remote servers for data.

SSH can be used to run remote commands; however, SSH needs you to enter a password. Although this is fine for manual testing, for automated polling it is a problem. Two solutions to this problem are:

1. ssh-keygen, which generates an SSH key pair that is stored in a user directory, enabling the standard SSH client to run without you having to enter a password.
2. sshpass, which is an SSH client that includes the username/password as a command-line option.

The ssh-keygen approach is recommended for most applications because it does not expose passwords. For testing, I use the sshpass method, and in the Node-RED project that follows, I use the ssh-keygen approach. The sshpass command is included in many standard distributions. To install it, enter:

sudo apt-get install sshpass

Listing 3 shows how to get CPU (line 1) and hard drive (line 13) temperatures.

01 $ sshpass -p mypassword ssh username@192.168.0.116 sensors
02 dell_smm-virtual-0
03 Adapter: Virtual device
04 Processor Fan: 3044 RPM
05 CPU:            +31.0°C
06 Ambient:        +33.0°C
07 SODIMM:         +33.0°C
08
09 acpitz-virtual-0
10 Adapter: Virtual device
11 temp1:        +51.5°C  (crit = +107.0°C)
12
13 $ sshpass -p mypassword ssh username@192.168.0.116 hddtemp /dev/sda
14 /dev/sda: HTS548040M9AT00: 39°C

As you saw earlier, grep and awk calls can be used to show just the required values for the CPU:

$ sshpass -p mypassword ssh username@192.168.0.116 sensors | grep temp1 | awk '{print $2}'
+30.5°C

and for the hard drive:

$ sshpass -p mypassword ssh username@192.168.0.116 hddtemp /dev/sda | awk '{print $3}'
39°C

At this point, a mechanism is available to get key monitoring values from remote Linux nodes. The next step is to pass these values to a Node-RED dashboard.

Node-RED Dashboard

Node-RED [4] is a web-based visual programming environment with a wide variety of nodes that can be wired together to create logic. On the Raspberry Pi platform, Node-RED is pre-installed in Raspbian images. To install Node-RED on other systems see the Get Started docs [5].

For this project I installed two nodes: bigssh, an SSH node that saves and uses ssh-keygen credentials, and bigtimer, a timer node that schedules data polling. These components can be installed either manually or with the Manage palette menu option (Figure 2).

Figure 2: Installing nodes from the Manage palette menu.

A basic test circuit to poll a Linux server manually and return a temperature would use an Inject, a bigssh, and a Debug node (Figure 3). The Inject node is used to pulse the logic.

Figure 3: Basic polling test circuit with Inject, Big SSH, and Debug nodes (left to right).

By double-clicking on the bigssh node, an Edit configuration window opens. In the myssh field, you enter the username, host, and ssh-keygen password. The bigssh node passes its first output (commandLine) to a Debug node. The Debug node sends message and payload information to the Debug tab. If the Debug node does not show the expected SSH command result, then the bigssh third output (minError) shows the error output.

After I was able to get all my SSH commands working successfully in the test logic, I changed the logic to use a bigtimer node for scheduling and some dashboard nodes for display (Figure 4). The bigtimer node has a good selection of scheduling and timing functions. Of the three Big Timer outputs, the middle output pin generates a pulse every minute.

Figure 4: Monitoring data sent to Node-RED dashboards.

You have the choice of a number of dashboard presentations. For a simple first pass, I used gauge and chart nodes. Double-click on each of these dashboard nodes to edit properties like labels and scales.

After the logic is complete, click the Deploy button on the right side of the menubar to run your updated project. The Node-RED web dashboard is at http://<your-node-red-ip>:1880/ui/ (Figure 5).

Figure 5: Node-RED dashboard showing data from old Linux servers.

Final Comment

In this article, I only looked at three command-line utilities; however, many others commands could use this technique.