Sensu — A powerful and scalable monitoring solution
Scaling
My company has now been using Sensu for more than a year on a dedicated server with state-of-the-art hardware, and we see no need for additional capacity. Although the infrastructure of the managed hosting platform is steadily increasing, the team has not once had such severely delayed checks as were experienced with Nagios using comparable hardware.
However, we still feed run-time metrics directly from the client to Graphite; if we were using Sensu as a metric distribution center, the traffic on the message bus would certainly increase significantly. We would probably then have to devote ourselves actively to the topic of scaling. Fortunately, Sensu is also well prepared for this.
The Sensu server as a central distribution point will, at some point, become the performance bottleneck with the increasing number of events. This problem could not be easier to solve: Simply connect other Sensu servers to RabbitMQ and Redis – and you're done. The Sensu servers automatically elect the master instance among themselves and distribute event processing – #monitoringlove! (See the "Who Monitors the Monitor?" box for more information.)
Who Monitors the Monitor?
From my own experience, I can say that it is a serious setback if, after a number of months full of #monitoringlove, you determine that errors have gone undetected for hours because Sensu has frozen. It is therefore important to monitor the function of Sensu itself. For example, a check, for which a single supervisor such as Monit is sufficient, should find out whether the central Sensu server process is running. Another check should investigate the Ready queue in RabbitMQ to identify processing backlogs.
Anyone using the Sensu API so intensively that it starts to moan under the load can counteract this problem with conventional load-balancing strategies, because Sensu uses a simple, stateless HTTP front end. Also, the RabbitMQ documentation describes strategies for scaling clusters that use the RabbitMQ message bus [10]. These approaches all aim to increase performance. You will have to consider each individual component separately if you want to make the entire Sensu system highly available [11].
Things become a little more complex if Sensu has to monitor large infrastructures that are distributed across multiple sites. Of course, a centralized approach is your first option (Figure 2). All Sensu clients perform their checks locally and deliver their results to the remote message bus via WAN.
However, intensive use of WAN connections also has a downside – in the form of much additional traffic. An unstable network connection can easily lead to a flood of host-down alarms.
An architecture with distributed instances of RabbitMQ, as shown in Figure 3, can help ensure that no events are lost on the (WAN) link. The Federation plugin [12] and the Shovel plugin [13] for RabbitMQ can help you implement this setup.
This approach favors the characteristics of availability and partition tolerance over consistency; however, it also suffers from a weakness that can cause keepalive alerts in case of connection problems. You can handle this problem by setting up check dependencies.
Reliance on a stable WAN connection is lifted if you run not just the message bus but the complete Sensu system locally at each site. A common view of the overall state of our infrastructure is maintained thanks to the fact that the Sensu dashboard can represent the states of multiple Sensu installations.
Figure 4 graphically depicts this aggregation approach. This redundancy may mean slightly higher installation overhead, but users can leave this to a configuration management system such as Chef.
Conclusions
Sensu is highly suited for its role as the central point of a comprehensive monitoring system. The Sensu monitoring tool is easy to operate and can easily reuse existing checks thanks to the Nagios plugin interface. Sensu also lets you integrate a steadily growing number of external services, and it is well equipped for the future through its scalability. #monitoringsucks is finally a thing of the past.
Infos
- Sensu: https://github.com/sensu
- Sensu admin: https://github.com/sensu/sensu-admin
- Chef cookbook: https://github.com/sensu/sensu-chef
- Puppet module: https://github.com/sensu/sensu-puppet
- Sensu shell helper: https://github.com/solarkennedy/sensu-shell-helper
- Sensu plugins: https://github.com/sensu/sensu-community-plugins
- Sensu CLI: https://github.com/agent462/sensu-cli
- Hubot: https://github.com/sensu/sensu-hubot
- Lita: https://www.lita.io
- RabbitMQ, Cluster documentation: https://www.RabbitMQ.com/clustering.html
- "High availability monitoring with Sensu" by Jean-Francois Theroux: http://failshell.io/sensu/high-availability-sensu/
- Federation plugin: https://www.RabbitMQ.com/federation.html
- Shovel plugin: https://www.RabbitMQ.com/shovel.html
« Previous 1 2 3
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
So Long Neofetch and Thanks for the Info
Today is a day that every Linux user who enjoys bragging about their system(s) will mourn, as Neofetch has come to an end.
-
Ubuntu 24.04 Comes with a “Flaw"
If you're thinking you might want to upgrade from your current Ubuntu release to the latest, there's something you might want to consider before doing so.
-
Canonical Releases Ubuntu 24.04
After a brief pause because of the XZ vulnerability, Ubuntu 24.04 is now available for install.
-
Linux Servers Targeted by Akira Ransomware
A group of bad actors who have already extorted $42 million have their sights set on the Linux platform.
-
TUXEDO Computers Unveils Linux Laptop Featuring AMD Ryzen CPU
This latest release is the first laptop to include the new CPU from Ryzen and Linux preinstalled.
-
XZ Gets the All-Clear
The back door xz vulnerability has been officially reverted for Fedora 40 and versions 38 and 39 were never affected.
-
Canonical Collaborates with Qualcomm on New Venture
This new joint effort is geared toward bringing Ubuntu and Ubuntu Core to Qualcomm-powered devices.
-
Kodi 21.0 Open-Source Entertainment Hub Released
After a year of development, the award-winning Kodi cross-platform, media center software is now available with many new additions and improvements.
-
Linux Usage Increases in Two Key Areas
If market share is your thing, you'll be happy to know that Linux is on the rise in two areas that, if they keep climbing, could have serious meaning for Linux's future.
-
Vulnerability Discovered in xz Libraries
An urgent alert for Fedora 40 has been posted and users should pay attention.