Harder than scripting, but easier than programming in C

Processes, Threads, and Goroutines

Traditionally, Unix systems implement concurrency with processes, but their resource consumption is enormous because of memory duplication. Languages such as Java or C++ let programmers handle this with threads that share the memory and are therefore far more lightweight. However, a few hundred thousand threads running in parallel will also overwhelm the processor.

Go adds another abstraction layer on top of the thread model, sending many goroutines per thread into the field and scheduling them with its own scheduler, which actually lets you run millions of them simultaneously. Using the syntax

go func() {...}

Go programmers fire off new goroutines, which the processor apparently executes simultaneously together with the rest of the program flow.

Of course, this causes problems during synchronization. How does one goroutine wait for another, how do they exchange data, and how can the main program call back and shut down all the goroutines started to date?

Channels: Synchronizing Communication

Various concurrent program components in Go often exchange messages via channels whose function goes beyond that of airmail-style Unix pipes. In fact, the sender and receiver often use channels to mutually sync in an elegant way without needing hard-to-handle software structures such as semaphores.

When a Go program reads from a channel with nothing written to it, the reading goroutine blocks the program flow until something arrives in the channel. And if a goroutine tries to write into a channel when no one is reading from it, it also blocks until a recipient is found to read from the pipe.

If you try to read from a channel and then write to it in a Go program, or vice versa, you will end up writing the most boring Go program in the world. It will just block permanently (Listing 7). And if nothing else is going on apart from what's shown there, the Go runtime detects a deadlock, aborts the program, and outputs an error:

fatal error: all goroutines are
asleep - deadlock!

Listing 7

block.go

package main
func main() {
  ch := make(chan bool)
  ch <- true // blocking
  <-ch
}

Read and write instructions for a channel therefore always need to be happening in parallel, usually in different, concurrent goroutines. By way of an example, Listing 8 generates two channels, ping and ok, that transfer messages of the bool type (true or false). After the channels are created, the program's main function (which is a goroutine in itself) fires up another goroutine that tries to read from the ping channel, causing it to block.

Listing 8

chan.go

package main
import (
  "fmt"
)
func main() {
  ping := make(chan bool)
  ok := make(chan bool)
  go func() {
    select {
    case <-ping:
      fmt.Printf("Ok!\n")
      ok <- true
    }
  }()
  fmt.Printf("Ping!\n")
  ping <- true
  <-ok
}

Meanwhile, the main program continues and writes a Boolean value to the ping channel after announcing "Ping!" to the user. As soon as the parallel goroutine on the other end starts to listen to the channel, the write goes through, and the main program advances to the next statement, which now waits by reading from the ok channel at the end of Listing 8.

The previously started parallel goroutine, which also has access to the channel via the ok variable, meanwhile advances beyond the read statement from ping and now writes a Boolean value to the ok channel. This prompts the last line of the main program to terminate its blocking read statement, and the program ends – a perfect handshake that allows two goroutines, one from the main program and the additional one started in line 11, to talk to each other – that is, to synchronize.

The output of the binary compiled from Listing 8 is "Ping!" and "Ok!", in exactly that order and never out of order, because the channel arrangement shown here categorically rules out any dreaded race conditions.

No More than Two

Normally, channels do not buffer the input they receive, as shown by the example in Listing 7, which simply blocked the program flow. Usually, if you want a reader to be able to read without blocking, you have to make sure that a writer is writing to the channel in parallel. On the other hand, buffered channels can store data, so a writer can write to them without blocking, even if no one is reading yet. If a reader connects at some point, it can retrieve the data held in a buffer.

Buffered channels also provide a tool for limiting the maximum number of concurrently running goroutines. This avoids overloading the CPU with a single application when doing compute-intensive work. Listing 9 fires off 10 goroutines in a for loop, but a buffered channel allows only two to run at any given time. How does this work?

Listing 9

limit.go

01 package main
02
03 import (
04   "fmt"
05   "time"
06 )
07
08 func main() {
09   limit := make(chan bool, 2)
10
11   for i := 0; i < 10; i++ {
12     go func(i int) {
13       limit <- true
14       work(i)
15       <-limit
16     }(i)
17   }
18
19   time.Sleep(10 * time.Second)
20 }
21
22 func work(id int) {
23   fmt.Printf("%d start\n", id)
24   time.Sleep(time.Second)
25   fmt.Printf("%d end\n", id)
26 }

The size of the channel buffer (specified as the second optional argument in the make statement) determines the maximum number of writes into the channel that can be held without a reader present. In Listing 9, it also defines the maximum number of goroutines passing through the gated section in parallel. Any goroutine that seeks entry to the section with the work() call in line 14 will first attempt to write to the channel limit. If there is still buffer space available (initially there are two slots), then there are not too many goroutines running yet, and the channel will let the current goroutine write to continue running without blocking.

On the other hand, if the buffer is already full, no more goroutines are allowed to enter the protected area, and the channel blocks all attempts by incoming guests to write to the buffer. At the other end of the protected area, outflowing goroutines read a piece of data from the channel, freeing up a slot in the buffer. This affects the flow at the start of the protected area, where the channel then allows a write action and lets one of the inflowing goroutines pass. In this way, a buffered channel effortlessly limits the maximum number of goroutines running in parallel through a protected area.

Figure 2 shows that the goroutines with the index values i=0 and i=3 get in first (this is random). After this, 3 leaves the area, and 9 pushes to the front. Then   says goodbye, and 4 makes its way in, and so on.

Figure 2: There are only ever two goroutines running concurrently.

By the way, watch out for for loops – such as the one in line 11 of Listing 9 – that fire off goroutines using a loop counter such as i. The i variable changes in each round of the loop. Since all goroutines share this variable, they would all display the same value (from the latest round of the loop) if they simply printed i. To allow each goroutine to pick up and display its own copy of the current state of i, line 12 passes in the i variable as a parameter to the go func() call, and now everything works as desired, because i remains local to the function and is separate from the shared value.

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy Linux Magazine

SINGLE ISSUES
 
SUBSCRIPTIONS
 
TABLET & SMARTPHONE APPS
Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

  • Rat Rac

    If program parts running in parallel keep interfering with each other, you may have a race condition. Mike Schilli shows how to instruct the Go compiler to detect these conditions and how to avoid them in the first place.

  • Simultaneous Runners

    In the Go language, program parts that run simultaneously synchronize and communicate natively via channels. Mike Schilli whips up a parallel web fetcher to demonstrate the concept.

  • Fighting Chaos

    When functions generate legions of goroutines to do subtasks, the main program needs to keep track and retain control of ongoing activity. To do this, Mike Schilli recommends using a Context construct.

  • Motion Sensor

    Inotify lets applications subscribe to change notifications in the filesystem. Mike Schilli uses the cross-platform fsnotify library to instruct a Go program to detect what's happening.

  • Progress by Installments

    Desktop applications, websites, and even command-line tools routinely display progress bars to keep impatient users patient during time-consuming actions. Mike Schilli shows several programming approaches for handwritten tools.

comments powered by Disqus
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters

Support Our Work

Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.

Learn More

News