Programming Snapshot – Go File Retrieval

I have lots of PDFs of scanned books on my Google Drive and often download what I need from my digital bookshelf to my hard drive [1]. The browser interface on drive.google.com is very useful for this (Figure 1). However, Google Drive could be easier and faster to use when searching for books by listing the results and downloading matches immediately. The Go program presented in this issue does this at the command line, which goes down well with programmers who feel at home in the terminal window and are reluctant to ever leave it.

Figure 1: The browser shows scanned programming books as PDF files in Google Drive.

Building the source code from Listings 1 to 4 [2] and calling the generated binary that accepts a search string like algorithms-in-cpp is shown in Figure 2. On the user's Google Drive, the program finds the PDF for the book Algorithms in C++. It offers up the file for selection and downloads it after confirmation. While the PDF, which is around 150MB in size, is crossing the wire, the Go program displays a slow or fast progress bar, depending on the Internet connection, to give an impression on the number of bytes received in relation to the expected total number.

Figure 2: The Go program shown in Listings 1 to 4 searches Google Drive for files and downloads them if requested.

Owners Only

Of course, I don't let every Tom, Dick, or Harry have access to my treasured and paid-for books. Therefore, a newly written client like the gd program shown in Listing 1 first has to identify itself to Google Drive as personally authorized by me. This is not done by means of a username and password, but rather by following an OAuth 2 flow and then by means of access tokens, which the client can renew from locally cached refresh tokens after the former have expired.

01 package main
02
03 import (
04   "flag"
05   "fmt"
06   "golang.org/x/oauth2/google"
07   "google.golang.org/api/drive/v3"
08   "io/ioutil"
09   "log"
10   "os"
11 )
12
13 func main() {
14   flag.Parse()
15   if flag.NArg() != 1 {
16     log.Fatalf(fmt.Sprintf("usage: %s partial-name", os.Args[0]))
17   }
18   query := flag.Arg(0)
19
20   b, err := ioutil.ReadFile("credentials.json")
21   if err != nil {
22     log.Fatalf("Error reading client secret file: %v", err)
23   }
24   config, err := google.ConfigFromJSON(b, drive.DriveReadonlyScope)
25   if err != nil {
26     log.Fatalf("Error parsing config: %v", err)
27   }
28
29   client := oauth2Client(config)
30   srv, err := drive.New(client)
31   if err != nil {
32     log.Fatalf("Error retrieving gdrive client: %v", err)
33   }
34
35   err = pickNGet(srv, query)
36   if err != nil {
37     log.Fatalf("Error retrieving document: %v", err)
38   }
39 }

The main program in Listing 1 uses the flag package to read the search term from the command line; the pattern provided by the user needs to match one or more files in Google Drive. In the version shown in this issue, the program searches for matches with file names. A full-text search would also be possible with a minor change to the source code.

Before sending the user through the OAuth 2 flow, line 20 parses the credentials.json file, which defines the client secrets (i.e., the data with which Google identifies the application – the Go program, not the user). The user's consent is later obtained by a browser dialog, started by the user by cutting and pasting a URL produced by the Go program, when started for the first time. Equipped with valid credentials, pickNGet() in line 35 then sets out to offer up the user documents that match the search term and downloads them upon request.

What the client is ultimately allowed to do after the user has given their permission is determined by the so-called OAuth scope, which developers set in their applications in advance and tell Google when they register the application. When the user later agrees to let the client have access, Google displays the scope to make sure the user knows what's at stake. Possible scope values range from access to meta-information of the files stored on the drive to actual reading and download rights, up to unlimited write access. Listing 1 defines DriveReadonlyScope for the Go program; this allows the client to query the file names and download the content in question.

Second-Class Citizen Go

While the developer pages for the Google Drive API show examples in Java, Python, and Node.js (Figure 3), Google doesn't seem to consider Go to be a first-class citizen.

Figure 3: Except for Java, Python, and Node.js, Google offers very little help in the use of the Google Drive API.

I was rubbing my eyes in disbelief, while scrolling through auto-generated spaghetti code, having to reverse engineer how to call the API functions with their correct signatures. On Stack Overflow, you will find five-year-old requests from perplexed programmers, and only a few brave developers have ever found an answer to handling even the most trivial of tasks.

Of course, you could also access the web API directly in Go, but if Google already provides an SDK, then they should document it and maintain it to keep it up-to-date.

OAuth, Take Two

So where does the credentials.json file come from? It is the result of the developer registering the client program with Google as an API application. On the Developers Console for the Google API [3], you first need to enable the Google Drive API with a valid Google account (Figure 4). On the same page, you then have to register the Go client as a desktop application and define how it will be represented later when Google asks the user whether they want to grant the app access to their Google Drive data.

Figure 4: Developers can click to enable the API on Google Drive.

Google lets you download the registration data in JSON format under the file name client-secret* (Figure 5), but you should rename it, because the client in Listing 1 expects the data as credentials.json. Keep in mind though that the app is not given access to user data on the basis of these credentials yet; instead, it's only for registration purposes of the app with Google.

Figure 5: The JSON download after creating a desktop client on the API console, in the Credentials tab.

When the Go program is later launched for the first time, it finds the JSON credentials file in the same directory and prints a link to standard output (Figure 6), which the end user is instructed to copy into the URL field of a web browser before being directed to the OAuth 2 flow.

Figure 6: On first launch, the program from Listing 1 expects the credentials.json file; it outputs a URL, which guides the user through the Google OAuth 2 flow in the browser.