Detect and restart hanging programs with Go
Programming Snapshot – Automated Restarts with Go
![© Lead Image © xyzproject, 123RF.com © Lead Image © xyzproject, 123RF.com](/var/linux_magazin/storage/images/issues/2022/261/pipe-cleaner/123rf-xyzproject_cables-binary.png/807114-1-eng-US/123RF-xyzproject_cables-binary.png1_medium.png)
© Lead Image © xyzproject, 123RF.com
Detecting programs where the standard output has frozen can require a deep dive into terminal emulation basics. Go plumber Mike Schilli builds a plunger to free up the pipe works.
If the browser stops halfway while loading a website and then freezes, experienced users know that usually all it takes is clicking the reload button to make it work like clockwork on the next attempt. Or, if an rsync transfer suddenly stalls because the server has fallen asleep, admins will intuitively press Ctrl+C to abort, only to immediately restart the command and see it finish without a hitch in most cases. Scenarios where humans have to manually take control over running processes just because the computer fails to understand that an automatic restart would solve the problem are one of the last hurdles to a fully automated world.
The yoyo
Go program presented in this issue supervises programs entrusted to it and will rejigger them like a yo-yo (as you know, yo-yos also need to be pulled up by a human hand to keep them moving). To do this, the utility monitors a supervised program's standard output (along with its standard error), which typically features frequently changing patterns – such as a progress bar that indicates that something is still happening. If the display freezes, for example, because of a network outage or because the server has lost interest, yoyo
detects this and restarts the program after a configurable timeout, in hopes of somehow fixing the problem this way.
Feels Like a Terminal
Easy, right? But programs behave differently, depending on whether or not they think they are running in a terminal. Typing git push
in a terminal, for example, continuously outputs the transfer progress as a percentage. And this gives the calling user, especially when they need to commit large files, some idea of how long the whole process is going to take (Figure 1, top).
But if git push
fails to find a terminal, for example, because its stdout
and stderr
channels have both been redirected to an out
file (Figure 1, bottom), it will not output any progress messages at all while the files are being transferred to the git
server. Instead, it just outputs a message at the end after everything is done. As you can easily determine from the git
source code on GitHub, git
uses the standard C isatty(2)
function to check if the error output (file descriptor 2
) is connected to a terminal and stops the output if isatty(2)
returns 0
.
Taciturn Without a Terminal
So without some kind of trickery, it would be impossible for a simple yo-yo controller program like the one in Listing 1 to track the progress of git push
. As soon as it taps into the stdout
and stderr
of the program it is monitoring, the associated terminal vanishes; git
notices this and switches to silent mode. This is why Listing 1 only prints a brief status message after the git
command exits. This just won't work at all for monitoring the process to restart it in case of stalled progress.
Listing 1
capture.go
Fake Terminals
As a workaround, the supervisor has to provide the monitored program with an environment that makes it think it is running in a terminal. Luckily, Unix offers programmers pseudo-terminals for this purpose. These kernel structures trick executed programs into thinking they are running in a real terminal, while allowing the monitor to control the input (stdin
) and intercept the output (stdout
, stderr
) of the program under their control. Standard Linux utilities such as ssh
, screen
, tmux
, and script
(for recording shell sessions) make heavy use of pseudo-terminals.
The test script in Listing 2 simulates a monitored program with a terminal sensor. It first checks via the shell's -t
command in line 2 whether standard output (file descriptor number 1
) is connected to a terminal. If this test fails, the script outputs an error message and stops running in line 5. Otherwise, the for
loop starting in line 7 counts to five in one-second steps, after which line 12 (optionally) sleeps for 31 seconds to allow the monitor to schedule a restart after 30 seconds of inactivity.
Listing 2
test.sh
The supervising yoyo
Go program presented in Listing 3 manages to provide this fake environment to its monitored entities just fine, as Figure 2 and Figure 3 illustrate. In the first case, yoyo
receives individual messages every second and outputs them until it detects that the script to be monitored has exited. This is normal and fine – no need to restart anything. But when Listing 2 enables the sleep 31
command, you get the situation shown in Figure 3: The yoyo
monitor patiently waits for 30 seconds on updates on the stalled output channel before pulling the ripcord and restarting the script.
Listing 3
yoyo.go
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
![Learn More](https://www.linux-magazine.com/var/linux_magazin/storage/images/media/linux-magazine-eng-us/images/misc/learn-more/834592-1-eng-US/Learn-More_medium.png)
News
-
NVIDIA Released Driver for Upcoming NVIDIA 560 GPU for Linux
Not only has NVIDIA released the driver for its upcoming CPU series, it's the first release that defaults to using open-source GPU kernel modules.
-
OpenMandriva Lx 24.07 Released
If you’re into rolling release Linux distributions, OpenMandriva ROME has a new snapshot with a new kernel.
-
Kernel 6.10 Available for General Usage
Linus Torvalds has released the 6.10 kernel and it includes significant performance increases for Intel Core hybrid systems and more.
-
TUXEDO Computers Releases InfinityBook Pro 14 Gen9 Laptop
Sporting either AMD or Intel CPUs, the TUXEDO InfinityBook Pro 14 is an extremely compact, lightweight, sturdy powerhouse.
-
Google Extends Support for Linux Kernels Used for Android
Because the LTS Linux kernel releases are so important to Android, Google has decided to extend the support period beyond that offered by the kernel development team.
-
Linux Mint 22 Stable Delayed
If you're anxious about getting your hands on the stable release of Linux Mint 22, it looks as if you're going to have to wait a bit longer.
-
Nitrux 3.5.1 Available for Install
The latest version of the immutable, systemd-free distribution includes an updated kernel and NVIDIA driver.
-
Debian 12.6 Released with Plenty of Bug Fixes and Updates
The sixth update to Debian "Bookworm" is all about security mitigations and making adjustments for some "serious problems."
-
Canonical Offers 12-Year LTS for Open Source Docker Images
Canonical is expanding its LTS offering to reach beyond the DEB packages with a new distro-less Docker image.
-
Plasma Desktop 6.1 Released with Several Enhancements
If you're a fan of Plasma Desktop, you should be excited about this new point release.