Scaling

My company has now been using Sensu for more than a year on a dedicated server with state-of-the-art hardware, and we see no need for additional capacity. Although the infrastructure of the managed hosting platform is steadily increasing, the team has not once had such severely delayed checks as were experienced with Nagios using comparable hardware.

However, we still feed run-time metrics directly from the client to Graphite; if we were using Sensu as a metric distribution center, the traffic on the message bus would certainly increase significantly. We would probably then have to devote ourselves actively to the topic of scaling. Fortunately, Sensu is also well prepared for this.

The Sensu server as a central distribution point will, at some point, become the performance bottleneck with the increasing number of events. This problem could not be easier to solve: Simply connect other Sensu servers to RabbitMQ and Redis – and you're done. The Sensu servers automatically elect the master instance among themselves and distribute event processing – #monitoringlove! (See the "Who Monitors the Monitor?" box for more information.)

Anyone using the Sensu API so intensively that it starts to moan under the load can counteract this problem with conventional load-balancing strategies, because Sensu uses a simple, stateless HTTP front end. Also, the RabbitMQ documentation describes strategies for scaling clusters that use the RabbitMQ message bus [10]. These approaches all aim to increase performance. You will have to consider each individual component separately if you want to make the entire Sensu system highly available [11].

Things become a little more complex if Sensu has to monitor large infrastructures that are distributed across multiple sites. Of course, a centralized approach is your first option (Figure 2). All Sensu clients perform their checks locally and deliver their results to the remote message bus via WAN.

Figure 2: The structure of a solution for distributed monitoring with a centralized Sensu instance.

However, intensive use of WAN connections also has a downside – in the form of much additional traffic. An unstable network connection can easily lead to a flood of host-down alarms.

An architecture with distributed instances of RabbitMQ, as shown in Figure 3, can help ensure that no events are lost on the (WAN) link. The Federation plugin [12] and the Shovel plugin [13] for RabbitMQ can help you implement this setup.

Figure 3: An approach for a distributed Sensu architecture.

This approach favors the characteristics of availability and partition tolerance over consistency; however, it also suffers from a weakness that can cause keepalive alerts in case of connection problems. You can handle this problem by setting up check dependencies.

Reliance on a stable WAN connection is lifted if you run not just the message bus but the complete Sensu system locally at each site. A common view of the overall state of our infrastructure is maintained thanks to the fact that the Sensu dashboard can represent the states of multiple Sensu installations.

Figure 4 graphically depicts this aggregation approach. This redundancy may mean slightly higher installation overhead, but users can leave this to a configuration management system such as Chef.

Figure 4: Aggregation approach for Sensu in geographically distributed environments.

Conclusions

Sensu is highly suited for its role as the central point of a comprehensive monitoring system. The Sensu monitoring tool is easy to operate and can easily reuse existing checks thanks to the Nagios plugin interface. Sensu also lets you integrate a steadily growing number of external services, and it is well equipped for the future through its scalability. #monitoringsucks is finally a thing of the past.