RabbitMQ

For Internet of Things (IoT) projects, there are a lot of different ways that sensors, devices, and client interfaces can be connected together. For many projects, using a simple Message Queue Telemetry Transport (MQTT) broker is all that you need. However, if you're trying to merge and build IoT projects that use both MQTT and Advanced Message Queue Protocol (AMQP), or you require a REST API, then you should take a look at RabbitMQ.

RabbitMQ [1] is an open source middleware solution that natively uses AMQP communications, but it has a good selection of plugins to support features like MQTT, MQTT WebSockets, HTTP REST API, and server-to-server communications (Figure 1).

Figure 1: RabbitMQ overview.

In this article, I will set up a RabbitMQ server, and I will look at some of the differences between MQTT and AMQP messaging. Finally, an example of an Arduino MQTT message will be presented as both an MQTT and an AMQP item in a Node-RED dashboard.

Getting Started Locally

RabbitMQ can be installed on Windows, Linux, and macOS systems, and there are also some cloud-based offerings. For small systems, lower-end hardware like a Raspberry Pi can be used. Complete RabbitMQ installation instructions are available online [2]. To install and run RabbitMQ on an Ubuntu system, enter:

sudo apt-get update
sudo apt-get install rabbitmq-server
sudo service rabbitmq-server start

The next step is to add some plugins. For my project, I loaded the MQTT and web administration plugins:

sudo rabbitmq-plugins enable rabbitmq_mqtt
sudo rabbitmq-plugins enable rabbitmq-management

The rabbitmqctl command-line tool allows you to configure and review the RabbitMQ server. To add a user (admin1) with a password (admin1) that has config, write, and read rights for management and administrator access, enter:

sudo rabbitmqctl add_user admin1 admin1
sudo rabbitmqctl set_permissions -p / admin1 ".*" ".*" ".*"
sudo rabbitmqctl set_user_tags admin1 management administrator

After you've defined an administrative user, the RabbitMQ web management plugin can be accessed at http://ip_address:15672 (Figure 2).

Figure 2: RabbitMQ web administration.

The RabbitMQ web management tool offers an overview of the present system load, connections, exchanges, and queues.

The RabbitMQ web management tool is excellent for small manual changes. However if you are looking at a doing a large number of additions or changes, then rabbitmqadmin, you can use the command-line management tool. Install the tool by entering:

# Get the cli and make it available to use.
wget http://127.0.0.1:15672/cli/rabbitmqadmin
sudo chmod +x rabbitmqadmin

Comparing MQTT and AMQP

It's useful to comment about some of the differences between MQTT and AMQP.

MQTT is a lightweight publish- and subscribe-based messaging protocol that works well with lower-end hardware and limited bandwidth. For Arduino-type applications where you only need to pass some sensor data, MQTT is an excellent fit.

AMQP has more overhead than MQTT, because it is a more advanced protocol that includes message orientation, queuing, routing, reliability, and security. Presently, there are no mainstream AMQP Arduino libraries, but numerous programming options for Raspberry Pi, Linux, Windows, and macOS systems exist. An AMQP IoT example would be to send sensor failures and alarms to dedicated maintenance and alarm queues.

MQTT and AMQP Queues

One of the biggest differences in queues is that MQTT queues are designed to show you the last available message, where as AMQP will store multiple messages in a queue.

A published MQTT message contains a message body, a retain flag, and a quality of service (QoS) value.

An AMQP message can be published with added properties, such as time stamp, type of message, and expiration information. AMQP messages also support the addition of custom header values. Listing 1 is a Python publish example that defines the message type to be "Pi Sensor", and it includes custom headers for status and alarm state.

01 #!/usr/bin/env python
02 import pika
03
04 node = "192.168.0.121"
05 user = "pete"
06 pwd = "pete"
07
08 # Connect to a remote AMQP server with a username/password
09 credentials = pika.PlainCredentials(user, pwd)
10 connection = pika.BlockingConnection(pika.ConnectionParameters(node,
11         5672, '/', credentials))
12 channel = connection.channel()
13
14 # Create a queue if it doesn't already exist
15 channel.queue_declare(queue='Rasp_1',durable=True)
16
17 # Define the properties and publish a message
18 props = pika.BasicProperties(
19     headers= {'status': 'Good Quality',"alarm":"HI"},
20     type ="Pi Sensor")
21 channel.basic_publish(exchange='',
22     routing_key='Rasp_1',body='99.5', properties = props)
23
24 connection.close()

The results from the Listing 1 example can be examined using the Queue | Get Message option in the RabbitMQ web management interface (Figure 3).

Figure 3: AMQP queue with custom properties.

RabbitMQ Messaging

There are a variety of ways to which AMQP messages can be published and subscribed to (Figure 4). The simplest way is to create a queue, and then messages can be published and subscribed to from that queue.

Figure 4: RabbitMQ messaging overview.

To help with the distribution and filtering of messages, AMQP supports a number of different exchange types. Messages in an exchange use bindings based on a routing key to link them to a queue.

The main types of exchanges are direct, fanout, headers, and topic. An IoT example of a direct exchange would be if a group of Raspberry Pi sensor values were going into a Rasp Pi sensor exchange. When the Rasp Pi publishes a sensor result to the exchange, the message also includes a routing key to link the message to the correct queue (Figure 5).

Figure 5: RabbitMQ direct exchange routing.

An IoT example of a fanout exchange would be critical sensor failures. A sensor failure message is sent to Bill and Sam's work queues and the All Maintenance Points queue all at the same time (Figure 6).

Figure 6: RabbitMQ fanout exchange routing.

Connecting MQTT

After the MQTT plugin is installed, RabbitMQ can act as a standalone MQTT broker.

For an MQTT project, any ESP8266-supported Arduino hardware can be used. There are a number of MQTT Arduino libraries that are available. For this project, I used the PubSubClient library that can be installed using the Arduino Library Manager.

As a test project, I used a low cost MQ-2 Smoke Gas Sensor ($3) [3] that measures a combination of LPG, alcohol, propane, hydrogen, CO, and even methane. Note to fully use this sensor, some calibration is required. On the MQ-2 sensor, the analog signal is connected to Arduino pin A0 and the analogRead(thePin) function is used to read the sensor value.

Listing 2 is an Arduino example that reads the MQ-2 sensor and publishes the result to the RabbitMQ MQTT broker with a topic name of mq2_mqtt.

01 #include <ESP8266WiFi.h>
02 #include <PubSubClient.h>
03
04 // Update these with values suitable for your network.
05 const char* ssid = "your_ssid";
06 const char* password = "your_password";
07 const char* mqtt_server = "192.168.0.121";
08 const char* mqtt_user = "admin1";
09 const char* mqtt_pass= "admin1";
10
11 const int mq2pin = A0; //the MQ2 analog input pin
12
13 WiFiClient espClient;
14 PubSubClient client(espClient);
15
16 void setup_wifi() {
17   // Connecting to a WiFi network
18   WiFi.begin(ssid, password);
19   while (WiFi.status() != WL_CONNECTED) {
20     delay(500);
21     Serial.print(".");
22   }
23   Serial.println("WiFi connected");
24   Serial.println("IP address: ");
25   Serial.println(WiFi.localIP());
26 }
27
28 void reconnect() {
29   // Loop until we're reconnected
30   Serial.println("In reconnect...");
31   while (!client.connected()) {
32     Serial.print("Attempting MQTT connection...");
33     // Attempt to connect
34     if (client.connect("Arduino_Gas", mqtt_user, mqtt_pass)) {
35       Serial.println("connected");
36     } else {
37       Serial.print("failed, rc=");
38       Serial.print(client.state());
39       Serial.println(" try again in 5 seconds");
40       delay(5000);
41     }
42   }
43 }
44
45 void setup() {
46   Serial.begin(9600);
47   setup_wifi();
48   client.setServer(mqtt_server, 1883);
49 }
50
51 void loop() {
52   char msg[8];
53   if (!client.connected()) {
54     reconnect();
55   }
56
57   sprintf(msg,"%i",analogRead(mq2pin));
58   client.publish("mq2_mqtt", msg);
59   Serial.print("MQ2 gas value:");
60   Serial.println(msg);
61
62   delay(5000);
63 }

Once the MQTT value is published, any MQTT client can subscribe to it. Listing 3 is a Python MQTT subscribe example.

01 import paho.mqtt.client as mqtt
02
03 def on_message(client, userdata, message):
04     print ("Message received: "  + message.payload)
05
06 client = mqtt.Client()
07 client.username_pw_set("admin1", password='admin1')
08 client.connect("192.168.0.121", 1883)
09
10 client.on_message = on_message       #attach function to callback
11
12 client.subscribe("mq2_mqtt")
13 client.loop_forever()                 #start the loop

Read MQTT Messages Using AMQP

MQTT clients can subscribe to MQTT messages directly, or it's possible to configure RabbitMQ to have the MQTT data accessible using AMQP. The routing of MQTT messages to AMQP queues is done using the amp.topic direct exchange (Figure 7).

Figure 7: Routing MQTT messages to AMQP queues.

To configure RabbitMQ to forward MQTT, the following steps are required:

1. Create a new RabbitMQ queue.
2. Create a binding between the MQTT exchange and the queue.

These two steps can be done in a number of ways: manually, in the RabbitMQ config file, using the rabbitmqadmin command-line tool, or via a program. To use rabbitmqadmin enter:

./rabbitmqadmin declare queue name=mq2_amqp durable=true
./rabbitmqadmin declare binding source=amq.topic \
  destination_type=queue destination=mq2_amqp routing_key=mq2_mqtt

rabbitmqadmin can also be used to check the queue for messages (Listing 4).

01 ./rabbitmqadmin get queue=mq2_amqp
02 +-------------+-----------+---------------+---------+
03 | routing_key | exchange  | message_count | payload |
04 +-------------+-----------+---------------+---------+
05 | mq2_mqtt    | amq.topic | 77            | 157     |
06 +-------------+-----------+---------------+---------+

Node-RED Dashboard

Node-RED [4] is a visual programming environment that allows users to create applications by dragging and dropping nodes on the screen. Logic flows are then created by connecting the different nodes together.

Node-RED has been preinstalled on Raspbian Jesse since November 2015. Node-RED can also be installed on Windows, Linux, and Mac OS X. To install and run Node-RED on your specific system see [5].

To install the AMQP components, select the Manage palette option from the right side of the menubar. Then search for "AMQP" and install node-red-contrib-amqp (Figure 8). If your installation of Node-RED does not have dashboards installed, search for and install node-red-dashboard.

Figure 8: Installing AMQP on Node-RED.

For this Node-RED MQTT and AMQP example, I will use a mqtt and a amqp node from the input palette group, along with two gauge nodes from the dashboard group (Figure 9).

Figure 9: Node-RED dashboard logic.

Nodes are added by dragging and dropping them into the center Flow sheet. Logic is created by making connection wires between inputs and outputs of a node. After the logic is laid out, double-click on each of the nodes to configure their specific properties. You will need to specify the MQTT and AMQP definitions of your RabbitMQ IP address, user rights, MQTT topic, and AMQP queue name. You will also need to double-click on the gauge nodes to configure the look-and-feel of the web dashboard.

After the logic is complete, hit the Deploy button on the right side of the menubar to run the logic. The Node-RED dashboard user interface can be accessed at http://ipaddress:1880/ui.

For my project, I used a number of different MQ sensors and inputs. Figure 10 is a picture of the Node-RED web dashboard that I created with the same MQTT value being shown natively and as a AMQP queued value.

Figure 10: Node-RED dashboard with RabbitMQ data.

Final Comments

I found that RabbitMQ was easy to install and the web administration plug-in, along with rabbitmqadmin, made the system very easy to maintain.

If you're just looking to show sensor values, then a basic MQTT broker might be all you need. However, if you're looking at some future applications like alarm, maintenance, or task lists, then AMQP exchanges and queues make RabbitMQ an interesting option.