Programming Snapshot – Go GPS Data Retrieval

As we all know, the past year and a half were completely lost to the pandemic. Because of various lockdowns, there wasn't much left to do in your leisure time in terms of outdoor activity. Playing soccer was prohibited and jogging with a face covering too strenuous. This prompted my wife and me to begin exploring areas of our adopted city, San Francisco. We hiked hidden pathways, previously unknown to us, every evening on hour-long neighborhood walks. To our amazement, we discovered that even 25 years of living in a city is not enough to explore every last corner. We found countless little hidden stairways, unpaved paths, and sights completely unknown to major travel guides.

Memorizing all the turnoffs for these newly invented, winding city hiking trails is almost impossible, but fortunately a mobile phone can step in as a brain extension here. Hiking apps plan your tours, record your progress during the walk, display traveled paths on a online map, and allow for sharing completed tours with friends (Figure 1). One of the best-known hiking (and biking) apps is the commercial komoot, which is based on OpenStreetMap data and remains free of charge as long as the user gets invited by another (even new) user and limits themselves to one local hiking area.

Figure 1: Practical: The app shows the trail and helps navigate it.

Staying Safe

This cheap subscription should suffice for basic needs, but I ended up buying the World Pack. The one-time payment of US $30 is not exactly a pittance (Figure 2), but I thought I'd support the komoot whippersnappers, helping them to keep the lights on in the datacenter. But then it dawned on me: What happens if komoot goes out of business at some point? What happens to my painstakingly created trails in that case? Fortunately, komoot lets you export the GPS data from the routes you hiked, and you can restore the tours from these data in an emergency. However, seeing that I have dozens of saved trails (Figure 3), manual downloads would be too labor-intensive, not to mention the discipline required to back up new trails on a regular basis – after all, you never know when an app will fail or be abandoned.

Figure 2: Cost of using komoot.

Figure 3: Archived city tours in the komoot app.

For this reason, a program that logs into komoot once a week via a cron job and stores tours that have not yet been saved locally in a backup directory would be just the thing. Komoot does offer an API for script-driven retrieval of user data but not for ordinary people like me. When I asked about getting an API key, the support team there referred me to a B2B department that only deals with business partners. But with a bit of tweaking, a web scraper can also scrape the GPS data from the web page. That's exactly what the Go program presented in this article does, based on some reverse-engineering work published on GitHub [1].

Cron Mirror

The scraper negotiates the login process on the website, queries all the tours stored under the account, downloads their JSON data, and converts them into the GPX format supported by all GPS trackers [2]. Now, if komoot were to lose the saved routes for some reason, the tour collection could be restored from the backup because the GPX data represents the hikes as a set of GPS coordinates with timestamps, independently of a particular app.

Listing 1 [3] shows the main Go program. A cron job calls the program once a week, and Go downloads the .gpx files of all tours that exist in a komoot account and saves them in a subdirectory named tours/. To do this, the Go program logs into the komoot web page with a username and password in the kc.kLogin() function, uses kc.kTours() to query a list of all tours saved under that account, and downloads only the .gpx files of the tours it doesn't already have in a local directory.

01 package main
02
03 import (
04   "fmt"
05   "io/ioutil"
06   "log"
07   "os"
08 )
09
10 const saveDir = "tours"
11
12 func main() {
13   kc := NewkColl()
14
15   _ = os.Mkdir(saveDir, 0755)
16
17   err := kc.kLogin()
18   if err != nil {
19     log.Fatalf("Login returned %v", err)
20     return
21   }
22
23   jdata, err := kc.kTours()
24
25   if err != nil {
26     log.Fatalf("Fetching tour ids returned %v", err)
27     return
28   }
29
30   ids := tourIDs(jdata)
31
32   for _, id := range ids {
33     gpxPath := fmt.Sprintf("%s/%s.gpx", saveDir, id)
34     if _, err := os.Stat(gpxPath); err == nil {
35       fmt.Printf("%s already saved\n", gpxPath)
36       continue
37     }
38
39     jdata, err = kc.kTour(id)
40
41     if err != nil {
42       log.Fatalf("Fetching tour %s returned %v", id, err)
43       return
44     }
45
46     gpx := toGpx(jdata)
47
48     fmt.Printf("Saving %s\n", gpxPath)
49     err := ioutil.WriteFile(gpxPath, gpx, 0644)
50     if err != nil {
51       panic(err)
52     }
53   }
54 }

The call to toGpx() in line 46 converts the JSON data from komoot to the platform-independent GPX tracker format, while the code starting in line 49 saves newly converted data to a .gpx file in the tours/ subdirectory. The procedure is gentle on the komoot servers and should not bother anyone there, and it helps the user to maintain ownership of routes they created themselves.

Caution, Password

It wouldn't be good style to hard code the username and password into the program, so Listing 2 offloads this to the creds.yaml file. You don't want to check the file into a GitHub repo; just keep it local. The web scraper later reads this YAML file before fetching your tour data and uses the komoot email, the associated password, and the numeric user ID stored there in RAM only while the program is running. Sample values are shown in Listing 3. For active use of the program, replace them with the values of your komoot account.

01 package main
02
03 import (
04   "gopkg.in/yaml.v2"
05   "io/ioutil"
06   "os"
07 )
08
09 var credsPath = "creds.yaml"
10
11 func readCreds() map[string]string {
12   creds := map[string]string{}
13
14   f, err := os.Open(credsPath)
15   if err != nil {
16     panic(err)
17   }
18   defer f.Close()
19
20   bytes, err := ioutil.ReadAll(f)
21   if err != nil {
22     panic(err)
23   }
24
25   err = yaml.Unmarshal(bytes, &creds)
26   if err != nil {
27     panic(err)
28   }
29
30   return creds
31 }

01 email: "foo@bar.com"
02 password: "hunter123"
03 client_id: "2014254181621"

To parse the YAML in creds.yaml, Listing 2 retrieves the gopkg.in/yaml.v2 package from GitHub, which – in the exported Unmarshal() function in line 25 – unravels the YAML data structure from Listing 3 and converts it into an internal Go hashmap. The data structure returned as creds contains the email address serving as the username for the komoot account used in the email key; password is unsurprisingly the password, and the client_id is the numeric ID of the user account with which komoot accesses the user's data. For an active account, the browser displays the numeric user ID in the URL field after you log in (Figure 4), from where it can be easily copied into the YAML file.

Figure 4: The browser displays the komoot account's numeric user ID.