Programming Snapshot – Bootable USB with Go

To test new Linux distributions on real hardware, a bootable USB stick with a downloaded image in ISO format will help with bootstrapping the installation. Rebooting the computer with the stick plugged in will often bring up a Live system that can be played around with to your heart's content, possibly after having to make some changes to the boot order in the BIOS.

How does the ISO file get onto the stick? Ultimately, this is done quite simply with a dd command that expects the ISO file as the input (if) and the device entry of the stick (for example /dev/sdd) as the output (of).

Tools like Ubuntu's Startup Disk Creator make things even more convenient with a graphical UI, but there are some concerns. Under no circumstances would you want the tool to have a bug that accidentally overwrites the next hard disk in the device tree instead of the stick.

How hard would it be to write a similar tool in Go – one that actively waits for the USB stick to be plugged in and then asks the user for permission to copy the ISO file to it? As always, by writing tools yourself, you'll learn a few new techniques that you might find helpful going forward to solve new everyday tasks.

The Art of Copying

It is not trivial to copy an ISO file weighing in at several gigabytes to another filesystem like the USB stick. Utilities like cp and dd do not read all the data in the source file from the disk in a single pass – this would take up a huge amount of precious RAM without shortening the process. Instead, such copy tools read the data from the source file, typically in megabyte-sized chunks, and keep writing them to the target file opened at the same time.

This is exactly what the code from Listing 1 does. The cpChunks() function expects the names of the source and the target files as well as an open Go channel as parameters. The caller taps into the latter as a source of information to see how far the copy process has progressed. To do this, cpChunks() sends a percentage value to the channel after each copied chunk, which indicates the fraction of the bytes already copied in relation to the total number. Starting off, the function obtains the total number of bytes to be copied via the os.Stat() system function, which gets the source file size from the filesystem.

01 package main
02
03 import (
04   "bufio"
05   "os"
06   "io"
07 )
08
09 func cpChunks(src, dst string, percent chan<- int) error {
10
11   data := make([]byte, 4*1024*1024)
12
13   in, err := os.Open(src)
14   if err != nil {
15     return err
16   }
17   reader := bufio.NewReader(in)
18   defer in.Close()
19
20   fi, err := in.Stat()
21   if err != nil {
22     return err
23   }
24
25   out, err := os.OpenFile(dst, os.O_WRONLY, 0644)
26   if err != nil {
27     return err
28   }
29   writer := bufio.NewWriter(out)
30   defer out.Close()
31
32   total := 0
33
34   for {
35     count, err := reader.Read(data)
36     total += count
37     data = data[:count]
38
39     if err == io.EOF {
40       break
41     } else if err != nil {
42       return err
43     }
44
45     _, err = writer.Write(data)
46     if err != nil {
47       return err
48     }
49
50     percent <- int(int64(total) * int64(100) / fi.Size())
51   }
52
53   return nil
54 }

To allow Go programmers to funnel data between different functions without writing too much glue code, many libraries accept the standard Reader and Writer interfaces. A library function expects a pointer to a Reader object from the caller and then uses Read() to draw data from it chunk by chunk.

The Reader/Writer interface builds an abstraction on the data, regardless of its origin, whether it's a JSON stream from a web server or blocks from the local filesystem read in via a file descriptor. The advantage is that things are kept flexible; you don't need to change the code just because the data source changes, because the interface remains the same.

Go Design: Reader/Writer

For example, Listing 1 opens the source file and receives an object of the os.File type from the Open() call. This object is passed to NewReader() from the bufio package, which returns a reader that the caller can in turn use to extract the bytes from the file gradually. Accordingly, the code obtains a writer to the target file, which already exists in the application as a stick device entry – but on Unix, practically everything is a file.

Calling os.OpenFile() with the O_WRONLY option in line 25 opens the filesystem entry for writing. As it is a device entry that must already exist (instead of being created by the function), the O_CREATE option, which is normally used for files, is deliberately missing here. Line 29 creates a new writer object from the file object, and the copying process can begin.

In the for loop starting in line 34, the reader now fetches 4MB data chunks to match the data buffer previously defined in line 11. However, the Read() function does not always return 4MB, because at the end of the file, there may be less bytes available while scraping the bottom of the barrel. It is important to shorten the data slice in line 37 to reflect the actual number of bytes retrieved and to discard garbage at the end of the buffer.

If the function were to simply pass the buffer to the writer, the writer would write the whole buffer out to the target file. A 5MB source file would be read as two 4MB chunks, written out to create an 8MB target file, with the last 3MB consisting of uninitialized garbage.

Percentages Through the Pipe

Since the data are read in small chunks and line 20 has determined the size of the source file in advance using os.Stat(), the function knows in each loop pass how far it has progressed with copying and how much remains to be done. Line 50 writes this ratio as a percentage integer between   and 100 to the Go channel passed to the function as percent by the caller. The caller later reads incoming values from the channel and can move a progress bar to the right while the function is still doing its job – true multitasking.

Now, how does the Flasher detect that the newly connected USB stick has appeared? In Listing 2, the driveWatch() function, starting in line 14, calls devices() from line 61 on to see which device entries are visible on the system below /dev/sd*. Usually the first hard drive is listed there as /dev/sda, while /dev/sdb and higher mark other SATA devices. USB sticks usually appear in /dev/sdd on my system, but this may vary elsewhere.

01 package main
02
03 import (
04   "bytes"
05   "errors"
06   "fmt"
07   "os/exec"
08   "path/filepath"
09   "strconv"
10   "strings"
11   "time"
12 )
13
14 func driveWatch(
15   done chan error) chan string {
16   seen := map[string]bool{}
17   init := true
18   drivech := make(chan string)
19   go func() {
20     for {
21       dpaths, err := devices()
22       if err != nil {
23         done <- err
24       }
25       for _, dpath := range dpaths {
26         if _, ok := seen[dpath]; !ok {
27           seen[dpath] = true
28           if !init {
29             drivech <- dpath
30           }
31         }
32       }
33       init = false
34       time.Sleep(1 * time.Second)
35     }
36   }()
37   return drivech
38 }
39
40 func driveSize(
41   path string) (string, error) {
42   var out bytes.Buffer
43   cmd := exec.Command("sfdisk", "-s", path)
44   cmd.Stdout = &out
45   cmd.Stderr = &out
46
47   err := cmd.Run()
48   if err != nil {
49     return "", err
50   }
51
52   sizeStr := strings.TrimSuffix(out.String(), "\n")
53   size, err := strconv.Atoi(sizeStr)
54   if err != nil {
55     return "", err
56   }
57
58   return fmt.Sprintf("%.1f GB", float64(size)/float64(1024*1024)), nil
59 }
60
61 func devices() ([]string, error) {
62   devices := []string{}
63   paths, _ := filepath.Glob("/dev/sd*")
64   if len(paths) == 0 {
65     return devices,
66       errors.New("No devices found")
67   }
68   for _, path := range paths {
69     devices = append(devices, path)
70   }
71   return devices, nil
72 }

The standard globbing library in Go, similar to the one used by the shell to convert wildcards like * into matching filesystem entries, does not report an error if the file path is invalid. It only complains about incorrect glob expressions. Because of this, if a glob expression does not find anything in Go, it is a good idea to look for other causes, such as incorrect paths or missing access permissions.

This is why devices() searches through all entries starting in line 61, and driveWatch() adopts these paths to initialize the seen map variable with the entries that were found. This search is performed asynchronously, because driveWatch() uses go func() to start a Go routine in parallel in line 19. Meanwhile, the function proceeds to the end and returns the newly created drivech Go channel to the caller to report newly discovered drives after the init phase.