Sensu — A powerful and scalable monitoring solution
Scaling
My company has now been using Sensu for more than a year on a dedicated server with state-of-the-art hardware, and we see no need for additional capacity. Although the infrastructure of the managed hosting platform is steadily increasing, the team has not once had such severely delayed checks as were experienced with Nagios using comparable hardware.
However, we still feed run-time metrics directly from the client to Graphite; if we were using Sensu as a metric distribution center, the traffic on the message bus would certainly increase significantly. We would probably then have to devote ourselves actively to the topic of scaling. Fortunately, Sensu is also well prepared for this.
The Sensu server as a central distribution point will, at some point, become the performance bottleneck with the increasing number of events. This problem could not be easier to solve: Simply connect other Sensu servers to RabbitMQ and Redis – and you're done. The Sensu servers automatically elect the master instance among themselves and distribute event processing – #monitoringlove! (See the "Who Monitors the Monitor?" box for more information.)
Who Monitors the Monitor?
From my own experience, I can say that it is a serious setback if, after a number of months full of #monitoringlove, you determine that errors have gone undetected for hours because Sensu has frozen. It is therefore important to monitor the function of Sensu itself. For example, a check, for which a single supervisor such as Monit is sufficient, should find out whether the central Sensu server process is running. Another check should investigate the Ready queue in RabbitMQ to identify processing backlogs.
Anyone using the Sensu API so intensively that it starts to moan under the load can counteract this problem with conventional load-balancing strategies, because Sensu uses a simple, stateless HTTP front end. Also, the RabbitMQ documentation describes strategies for scaling clusters that use the RabbitMQ message bus [10]. These approaches all aim to increase performance. You will have to consider each individual component separately if you want to make the entire Sensu system highly available [11].
Things become a little more complex if Sensu has to monitor large infrastructures that are distributed across multiple sites. Of course, a centralized approach is your first option (Figure 2). All Sensu clients perform their checks locally and deliver their results to the remote message bus via WAN.
However, intensive use of WAN connections also has a downside – in the form of much additional traffic. An unstable network connection can easily lead to a flood of host-down alarms.
An architecture with distributed instances of RabbitMQ, as shown in Figure 3, can help ensure that no events are lost on the (WAN) link. The Federation plugin [12] and the Shovel plugin [13] for RabbitMQ can help you implement this setup.
This approach favors the characteristics of availability and partition tolerance over consistency; however, it also suffers from a weakness that can cause keepalive alerts in case of connection problems. You can handle this problem by setting up check dependencies.
Reliance on a stable WAN connection is lifted if you run not just the message bus but the complete Sensu system locally at each site. A common view of the overall state of our infrastructure is maintained thanks to the fact that the Sensu dashboard can represent the states of multiple Sensu installations.
Figure 4 graphically depicts this aggregation approach. This redundancy may mean slightly higher installation overhead, but users can leave this to a configuration management system such as Chef.
Conclusions
Sensu is highly suited for its role as the central point of a comprehensive monitoring system. The Sensu monitoring tool is easy to operate and can easily reuse existing checks thanks to the Nagios plugin interface. Sensu also lets you integrate a steadily growing number of external services, and it is well equipped for the future through its scalability. #monitoringsucks is finally a thing of the past.
Infos
- Sensu: https://github.com/sensu
- Sensu admin: https://github.com/sensu/sensu-admin
- Chef cookbook: https://github.com/sensu/sensu-chef
- Puppet module: https://github.com/sensu/sensu-puppet
- Sensu shell helper: https://github.com/solarkennedy/sensu-shell-helper
- Sensu plugins: https://github.com/sensu/sensu-community-plugins
- Sensu CLI: https://github.com/agent462/sensu-cli
- Hubot: https://github.com/sensu/sensu-hubot
- Lita: https://www.lita.io
- RabbitMQ, Cluster documentation: https://www.RabbitMQ.com/clustering.html
- "High availability monitoring with Sensu" by Jean-Francois Theroux: http://failshell.io/sensu/high-availability-sensu/
- Federation plugin: https://www.RabbitMQ.com/federation.html
- Shovel plugin: https://www.RabbitMQ.com/shovel.html
« Previous 1 2 3
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
Canonical Bumps LTS Support to 12 years
If you're worried that your Ubuntu LTS release won't be supported long enough to last, Canonical has a surprise for you in the form of 12 years of security coverage.
-
Fedora 40 Beta Released Soon
With the official release of Fedora 40 coming in April, it's almost time to download the beta and see what's new.
-
New Pentesting Distribution to Compete with Kali Linux
SnoopGod is now available for your testing needs
-
Juno Computers Launches Another Linux Laptop
If you're looking for a powerhouse laptop that runs Ubuntu, the Juno Computers Neptune 17 v6 should be on your radar.
-
ZorinOS 17.1 Released, Includes Improved Windows App Support
If you need or desire to run Windows applications on Linux, there's one distribution intent on making that easier for you and its new release further improves that feature.
-
Linux Market Share Surpasses 4% for the First Time
Look out Windows and macOS, Linux is on the rise and has even topped ChromeOS to become the fourth most widely used OS around the globe.
-
KDE’s Plasma 6 Officially Available
KDE’s Plasma 6.0 "Megarelease" has happened, and it's brimming with new features, polish, and performance.
-
Latest Version of Tails Unleashed
Tails 6.0 is based on Debian 12 and includes GNOME 43.
-
KDE Announces New Slimbook V with Plenty of Power and KDE’s Plasma 6
If you're a fan of KDE Plasma, you'll be thrilled to hear they've announced a new Slimbook with an AMD CPU and the latest version of KDE Plasma desktop.
-
Monthly Sponsorship Includes Early Access to elementary OS 8
If you want to get a glimpse of what's in the pipeline for elementary OS 8, just set up a monthly sponsorship to help fund its continued existence.