Distributed weather monitoring in gardens

The Things Network

The Things Network provides a public network server for LoRaWAN applications, registration services for LoRaWAN nodes (sensors), security key generation, packet examination facilities, and, most important, an API for data extraction for several languages. Use of this service is free, but by default, The Things Network does not store your data. If you don't extract your data in a timely manner, it is simply discarded.

The first step is to register your gateway with The Things Network in the gateway console [16]. You can give your gateway any name you like (Figure 6) and choose to make it public, so other people can use it, or private, for your own use only. All data is strongly encrypted, so it is secure, even on a public gateway.

Figure 6: Gateway console and application registration in The Things Network console.

The next step is to register your application. The application unique ID (EUI) that is generated must be added to the sensor's code for the device to be considered for registration.

Finally, you can register one or more devices with a name you select by selecting register device in the Devices pane: I call the devices for this application weatherstation1, weatherstation2, …, and so on. Devices on The Things Network must have a unique ID (in the case of my microprocessor, the EUI is a unique 64-bit number programmed by the manufacturer into its flash memory), which is entered in the Device EUI field on the registration page (Figure 7).

Figure 7: The settings for weatherstation1.

Once registered, the generated application key must be added to the device's code, as well. When the device is powered up, it sends a registration request with a tuple of device EUI, application EUI, and application key.

To view the data, use the Data tab (Figure 8). The payload (application data) has been decrypted and is displayed in hexadecimal format. In this case, the data is a comma-separated list of five floating-point numbers representing temperature, humidity, pressure, intensity, and battery voltage. The metadata provides some details about the packet transmission, the most useful of which is timestamp, representing the time at which the packet was received.

Figure 8: Data view.

Extracting the Data

I chose to write my client for The Things Network in golang, although SDKs for many other languages are provided, as well [17]. Data is made available through an MQTT protocol [18], and an SDK for golang is provided. A simple example and GoDoc is provided online [19]. Only a few lines of golang subscribes to the service, lists the registered devices, and waits for data.

In the first example (Listing 1), I just access my data in The Things Network and write it to a file. The program must provide the application ID and the application access key (lines 24 and 25) to register successfully with The Things Network. Note that the time stamp is extracted from the metadata, which will be important when it comes time to plot the data as time series graphs. To make the data truly useful, however, you need to store it in something more sophisticated than a flat file.

Listing 1

ttn-client.go

01 package main
02
03 import (
04         "os"
05         "fmt"
06         "time"
07         "strings"
08         ttnsdk "github.com/TheThingsNetwork/go-app-sdk"
09 )
10
11 const (
12         sdkClientName = "lora-weather-station"
13 )
14
15 func main() {
16
17         // open a file to write results to
18         f, err := os.OpenFile("weather.dat", os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
19         if err != nil {
20                 fmt.Println(err)
21         }
22
23         defer f.Close()
24
25         appID := "loramini2"
26         appAccessKey := "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXxxx"
27
28         config := ttnsdk.NewCommunityConfig(sdkClientName)
29
30         // create a client to 'the things network. the access key has been obscured
31         client := config.NewClient(appID, appAccessKey)
32         defer client.Close()
33
34         devices, err := client.ManageDevices()
35         if err != nil {
36                 fmt.Printf("lora-weather-station: could not get device manager\n")
37         }
38
39         // create a list of known devices
40         deviceList, err := devices.List(10, 0)
41         if err != nil {
42                 fmt.Printf("lora-weather-station: could not get devices\n")
43         }
44         fmt.Printf("lora-weather-station: found device(s)\n")
45         for _, device := range deviceList {
46                 fmt.Printf("%s\n", device.DevID)
47         }
48
49         // subscribe to events from all the devices
50         pubsub, err := client.PubSub()
51         if err != nil {
52                 fmt.Printf("lora-weather-station: could not get application pub/sub\n")
53         }
54
55         defer pubsub.Close()
56
57         allDevicesPubSub := pubsub.AllDevices()
58         defer allDevicesPubSub.Close()
59         uplink, err := allDevicesPubSub.SubscribeUplink()
60         if err != nil {
61                 fmt.Printf("lora-weather-station: could not subscribe to uplink messages")
62         }
63
64     // wait for data from each device and save it to the file. loop forever
65         fmt.Printf("waiting for data\n")
66         for message := range uplink {
67                 timestamp := time.Time(message.Metadata.Time).Unix()
68                 measurements := strings.Split(string(message.PayloadRaw), ",")
69
70                 if len(measurements)<4 {
71                         fmt.Printf("lora-weather-station: uplink message malformed, ignored")
72                         continue
73                 }
74
75                 data := fmt.Sprintf("%s %d %s %s %s %s %s\n", message.HardwareSerial,
76                 timestamp,measurements[0],measurements[1],measurements[2],measurements[3],,measurements[4])
77
78                 fmt.Printf(data)
79
80                 if _, err := f.WriteString(data); err != nil {
81                         fmt.Printf("lora-weather-station: could not write data")
82                 }
83         }
84 }

Prometheus

Prometheus is a free and open source multidimensional time series database for event monitoring and alerting that runs on Linux. You can find many excellent tutorials about Prometheus installation online, so you should choose one that is most appropriate for your distro. (For my Ubuntu setup, I chose a video tutorial on the LinOxide website [20].) Once installed, you should be able to browse to the Prometheus home page. However, to be of real use, you need to be able to insert data into Prometheus, so it's back to the golang code to add Prometheus metrics (Listing 2).

Listing 2

ttn-client-prom.go

001 package main
002
003 import (
004         "flag"
005         "fmt"
006         ttnsdk "github.com/TheThingsNetwork/go-app-sdk"
007         "github.com/prometheus/client_golang/prometheus"
008         "github.com/prometheus/client_golang/prometheus/promhttp"
009         "net/http"
010         "strconv"
011         "strings"
012         "time"
013 )
014
015 // create a bunch of prometheus vectors, one for each measurement
016 var (
017         temperatureGauge = prometheus.NewGaugeVec(
018                 prometheus.GaugeOpts{
019                         Name: "temperature",
020                         Help: "temperature reported by a weather station over time in degrees Celcius",
021                 },
022                 []string{"hardware_id"})
023
024         humidityGauge = prometheus.NewGaugeVec(
025                 prometheus.GaugeOpts{
026                         Name: "humidity",
027                         Help: "humidity reported by a weather station over time in percent",
028                 },
029                 []string{"hardware_id"})
030
031         pressureGauge = prometheus.NewGaugeVec(
032                 prometheus.GaugeOpts{
033                         Name: "pressure",
034                         Help: "barometric reported by a weather station over time",
035                 },
036                 []string{"hardware_id"})
037
038         intensityGauge = prometheus.NewGaugeVec(
039                 prometheus.GaugeOpts{
040                         Name: "intensity",
041                         Help: "light intensity reported by a weather station over time",
042                 },
043                 []string{"hardware_id"})
044
045         batteryGauge = prometheus.NewGaugeVec(
046                 prometheus.GaugeOpts{
047                         Name: "battery",
048                         Help: "battery voltage reported by a weather station over time",
049                 },
050                 []string{"hardware_id"})
051
052         addr = flag.String("prom-server-address", ":8080", "the address of the server for prometheus metrics")
053
054         appID         = "loramini2"
055         appAccessKey  = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
056         sdkClientName = "lora-weather-station"
057 )
058
059 func main() {
060
061         flag.Parse()
062
063         // register the prometheus vectors
064         prometheus.MustRegister(temperatureGauge)
065         prometheus.MustRegister(humidityGauge)
066         prometheus.MustRegister(pressureGauge)
067         prometheus.MustRegister(intensityGauge)
068         prometheus.MustRegister(batteryGauge)
069
070         // publish the prometheus vectors to the premetheus can 'scrape' them for recording
071         // this starts a mini http server
072         go func() {
073                 http.Handle("/metrics", promhttp.HandlerFor(
074                         prometheus.DefaultGatherer,
075                         promhttp.HandlerOpts{},
076                 ))
077
078                 fmt.Printf("starting prometheus server on %s\n", *addr)
079
080                 err := http.ListenAndServe(*addr, nil)
081                 if err != nil {
082                         fmt.Printf("unable to start prometheus server\v")
083                 }
084         }()
085
086         // connect to 'the things network'
087         config := ttnsdk.NewCommunityConfig(sdkClientName)
088
089         client := config.NewClient(appID, appAccessKey)
090         defer client.Close()
091
092         devices, err := client.ManageDevices()
093         if err != nil {
094                 fmt.Printf("could not get device manager\n")
095         }
096
097         // List the first 10 devices
098         deviceList, err := devices.List(10, 0)
099         if err != nil {
100                 fmt.Printf("could not get devices\n")
101         }
102         fmt.Printf("found device(s)\n")
103         for _, device := range deviceList {
104                 fmt.Printf("%s\n", device.DevID)
105         }
106
107         fmt.Println()
108
109         // subscribe to events from the devices
110         pubsub, err := client.PubSub()
111         if err != nil {
112                 fmt.Printf("could not get application pub/sub\n")
113         }
114
115         defer pubsub.Close()
116
117         // listen for measurement events and update the prometheus counters. loop forever
118         allDevicesPubSub := pubsub.AllDevices()
119         defer allDevicesPubSub.Close()
120         uplink, err := allDevicesPubSub.SubscribeUplink()
121         if err != nil {
122                 fmt.Printf("could not subscribe to uplink messages\n")
123         }
124         fmt.Printf("waiting for data\n")
125         for message := range uplink {
126                 timestamp := time.Time(message.Metadata.Time).Format(time.RFC3339)
127                 measurements := strings.Split(string(message.PayloadRaw), ",")
128
129                 if len(measurements) < 4 {
130                         fmt.Printf("uplink message malformed, ignored\n")
131                         continue
132                 }
133
134                 fmt.Printf("measurement from %s at %s : %s %s %s %s %s\n", message.HardwareSerial, timestamp, measurements[0], measurements[1], measurements[2], measurements[3], measurements[4])
135
136                 temperature, err := strconv.ParseFloat(measurements[0], 64)
137
138                 if err == nil {
139                         temperatureGauge.With(prometheus.Labels{"hardware_id": message.HardwareSerial}).Set(temperature)
140                 } else {
141                         fmt.Printf("error parsing temperature: %s\n", err)
142                 }
143
144                 humidity, err := strconv.ParseFloat(measurements[1], 64)
145
146                 if err == nil {
147                         humidityGauge.With(prometheus.Labels{"hardware_id": message.HardwareSerial}).Set(humidity)
148                 } else {
149                         fmt.Printf("error parsing humidity: %s\n", err)
150                 }
151
152                 pressure, err := strconv.ParseFloat(measurements[2], 64)
153
154                 if err == nil {
155                         pressureGauge.With(prometheus.Labels{"hardware_id": message.HardwareSerial}).Set(pressure)
156                 } else {
157                         fmt.Printf("error parsing pressure: %s\n", err)
158                 }
159
160                 intensity, err := strconv.ParseFloat(measurements[3], 64)
161
162                 if err == nil {
163                         intensityGauge.With(prometheus.Labels{"hardware_id": message.HardwareSerial}).Set(intensity)
164                 } else {
165                         fmt.Printf("error parsing intensity: %s\n", err)
166                 }
167
168                 battery, err := strconv.ParseFloat(measurements[4], 64)
169
170                 if err == nil {
171                         batteryGauge.With(prometheus.Labels{"hardware_id": message.HardwareSerial}).Set(battery)
172                 } else {
173                         fmt.Printf("error parsing battery: %s\n", err)
174                 }
175         }
176 }

Immediately after the metrics section, the code declares a set of two-dimensional Prometheus vectors, one for each measurement: hardware_id and time are two dimensions. Therefore, when you insert data, it is keyed on the ID of the sensor in question, so you can plot time-based curves from all the sensors on one graph. As you add sensors, their data will be stored without the need to change this code. The program's entry point, main (line 58), registers these vectors with Prometheus (lines 61-65) and starts an HTTP server to serve the metrics (lines 67-71).

Prometheus will query this HTTP endpoint on a regular basis to scrape data. The rest of the code is similar to the previous version, but instead of saving data to a file, the data is written to the Prometheus vectors. Again, note that you have to supply the hardware ID along with the measurement (e.g., line 131 for temperature), so that the data ends up in the correct time series.

Once this program is started (on the same machine as Prometheus), you can query and visualize the data (Figure 9). One of the vectors is simply called temperature, so if you type that in the query box, press Execute, and then select the Graph tab, you can see measurements from three sensors for the last hour. To see a different time range, you can play with the time range slider.

Figure 9: Querying the temperature data.

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy Linux Magazine

SINGLE ISSUES
 
SUBSCRIPTIONS
 
TABLET & SMARTPHONE APPS
Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

  • LoRa Long Range Radio

    WiFi is convenient for devices that are in the same house. If you want to extend the distance, give LoRa a call.

  • Prometheus

    Legacy monitoring solutions are fine for small-to-medium-sized networks, but complex environments benefit from a different approach. Prometheus is an interesting alternative to classic tools like Nagios.

  • Automated Irrigation

    An automated watering system comprising a Raspberry Pi Zero W, an analog-to-digital converter, and an inexpensive irrigation kit can help keep your potted plants from dying of thirst.

  • Electronic Weighing

    Create your own weighing device with easily available components and open source software.

  • Ren'Py

    Ren'Py helps you create Android, Linux, macOS, Windows, and HTML5 games and apps.

comments powered by Disqus

Direct Download

Read full article as PDF:

Price $2.95

News