Detect and restart hanging programs with Go


Now you may be wondering what the implementation of yoyo with its dark magic terminal trickery looks like. Setting up a pseudo-terminal pair requires some non-trivial boilerplate code, but fortunately there is already a project on GitHub named Expectre [1] that implements the Linux expect tool in Go. The yoyo program simply recycles Expectre's pseudo TTY code: In line 5 of Listing 3, it imports the library from GitHub (see the "Compiling Yoyo" box for more information).

Compiling Yoyo

To compile the yoyo binary from the source code, the Go compiler needs the Expectre library used in Listing 3 from GitHub. The following three steps

$ go mod init yoyo
$ go mod tidy
$ go build yoyo.go

resolve the dependencies and build an executable binary from the Go code in Listing 2.

In line 24, yoyo creates a new expectre object, which then calls Spawn, with the process parameters passed to yoyo as arguments on the command line. In Go, os.Args[0] traditionally contains the name of the called binary (yoyo). The flag package courtesy of Go's standard library will gobble up all command-line flags and arguments and later provides the agruments in the array slice returned by flag.Args(). In the case at hand, the slice contains a single string, "./", which is the script that you want yoyo to run and supervise. Line 25 passes it to Spawn().

The Spawn() function from the expectre package then starts a process with the supervised program and lulls it into a pseudo-terminal pair. The controlled process will then assume it is running in a regular terminal and behave accordingly. As soon as the process is running, the for loop starting in line 30 jumps into a select statement that waits for one of four different events:

  • A line of output printed by the launched process reaching the exp.Stdout channel
  • A line of output printed by the launched process reaching the exp.Stderr channel
  • The timer in line 36 expiring
  • The exp.Released channel signalling that the process launched for monitoring has just terminated and no longer needs any supervision

A select statement like this is typical of Go programs that wait for events. Each case waits either for messages from any number of monitored channels or for a timer to expire – and all concurrently without the computer having to do any active work.

Slow Traffic

Ultimately, yoyo distinguishes between two cases. In the first case, the process terminates itself because it has reached the end of its instructions. This is perfect, because it means yoyo does not have to do anything and can also terminate. But, in the second case, if the timer in line 36 elapses, yoyo has to kill the monitored process; this is done neatly in line 39 by the Cancel() function from the expectre package.

In this case, however, line 38 sets the triggered variable to a true value. Once the monitored process terminates, a exp.Released() message arrives, and line 43 uses continue restart to resume the outer for loop in line 22 with the restart label. The supervised program comes back online when Spawn() in line 25 restarts the process.

Brief Connection

Every application is different – while one application might want to be reminded to get back to work after 30 seconds, another might need a shorter timeout. You also want yoyo to set a limit for the number of startup attempts. If something goes wrong, you wouldn't want it to indefinitely request restarts and potentially annoy the owners of the services it keeps contacting that way. The --timeout and --maxtries flags allow the user to set appropriate values to control this behavior, and Go's flag package takes care of fielding input from the command line and syntax checking the arguments.

Figure 4 shows a yoyo run with a two-second timeout and at most two restarts of the test script from Listing 2. After the second restart also times out, yoyo terminates as instructed. In Figure 5, during a longish commit, yoyo's monitoring of the git push command shows that the commit falls asleep after some time because the server is slow to respond. After 30 seconds, yoyo detects this, terminates the frozen process, and then wakes it up again. And, lo and behold, the process continues.

Figure 4: A yoyo run with a two-second timeout and a maximum of two attempts.
Figure 5: After the git push stalls, yoyo revitalizes it.

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy Linux Magazine

Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

  • Command Line: Processes

    Innumerable processes may be running on your Linux system. We’ll show you how to halt, continue, or kill tasks, and we’ll examine how to send the remnants of crashed programs to the happy hunting grounds.

  • Command Line: Process Control

    What is happening on your Linux machine? Various shell commands give you details about system processes and help you control them.

  • Elixir 1.0

    Developers will appreciate Elixir's ability to build distributed, fault-tolerant, and scalable applications.

  • System Monitoring Center

    The System Monitoring Center combines all the important information you need to monitor a computer in a single state-of-the-art interface.

  • Fsniper

    Every day, computers are inundated with hundreds of files. Fsniper welcomes the new arrivals and processes them according to rules that you define.

comments powered by Disqus
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters

Support Our Work

Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.

Learn More