System monitoring for a new generation with Prometheus
Push Gateway
All services cannot be meaningfully run with exporters. This is particularly problematic for services that generate data on the fly and need to offload the data immediately. One classic example is Cron, which handles a specific task and generates output while doing so, which Prometheus then needs to process. Prometheus has the push gateway to handle this kind of situation.
The gateway runs as a separate service with a RESTful interface and waits for the required values to arrive. If you want to deliver the Cron job output to the push gateway, all you need to do is append a pipe and a Curl call to the Cron job. The REST interface on the push gateway accesses the target host for Curl.
Of course, these gateways will scale horizontally; in other words you can run multiple gateways in parallel in the same installation. The Prometheus servers pick up the stored data from the gateways in exactly the same way they do with normal exporters. But, there is one slight drawback: The data output by the Cron job needs to exist in a format that the push gateway actually understands. In other words, you may need to tune the Cron output.
Alert Manager
If something is going wrong in the cluster, administrators would naturally expect the monitoring system to tell them. Prometheus has two possible approaches. Prometheus itself offers a basic alerting function: if the value of a metric is outside of defined boundaries over a defined period of time, Prometheus sends a message to a previously defined address.
However, the developers expressly point out that this function is only basic alerting, and the number of tweaks required for modifying the alerting style to meet your local needs limits flexibility.
But Prometheus offers a second approach: instead of handing alerting itself, Prometheus can forward alerts directly to the Alert Manager. The Alert Manager is a separate component in the Prometheus universe (Figure 4), and it is probably one of the biggest construction sites in Prometheus right now.
When this issue went to press, the Alert Manager GitHub page [8] featured a statement to the effect that the version in the master branch is a totally new development that is no longer compatible with the current stable version. If you want to take a look at, or test, Prometheus, you will still want to go for this version: the legacy version will disappear in the foreseeable future.
As an interface to Prometheus, the Alert Manager has its own API. When the Alert Manager receives an alert from Prometheus, it can process that alert in a variety of ways: the manager can mail a warning or forward the alert to the Pager Duty API.
The pager, in turn, supports various modules. Out of the box are Pager Duty modules for Slack, Hipchat, and various other chat protocols. But you could hitch up Pager Duty to a text service with a matching SMS module.
PromDash
PromDash (Figure 5) is a powerful tool for rendering the metrics stored in Prometheus in a meaningful way. The tool's developers describe PromDash as a dashboard builder, thus setting it apart from standard dashboards.
PromDash does not try to give users prebuilt templates for charts but instead gives administrators a tool for creating these templates themselves. PromDash can thus rightly be described as a graphical helper that facilitates the use of the PromQL language.
After setting up the solution, you need to add the servers that will act as data sources in PromDash, then sit back and enjoy the ride. One of PromDash's features is the ability to annotate graphs or integrate them directly into other pages. For example, you could easily implement a status dashboard for the entire platform that would give you a centralized overview of all the systems you need to monitor.
The solution for configuring PromDash itself is also very smart: The PromDash module stores the configured dashboards in a MySQL database in the background. This design means the configuration and program are separate, so you can run any number of PromDash instances on the same host without you needing to manually synchronize the configurations.
« Previous 1 2 3 4 Next »
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
Canonical Bumps LTS Support to 12 years
If you're worried that your Ubuntu LTS release won't be supported long enough to last, Canonical has a surprise for you in the form of 12 years of security coverage.
-
Fedora 40 Beta Released Soon
With the official release of Fedora 40 coming in April, it's almost time to download the beta and see what's new.
-
New Pentesting Distribution to Compete with Kali Linux
SnoopGod is now available for your testing needs
-
Juno Computers Launches Another Linux Laptop
If you're looking for a powerhouse laptop that runs Ubuntu, the Juno Computers Neptune 17 v6 should be on your radar.
-
ZorinOS 17.1 Released, Includes Improved Windows App Support
If you need or desire to run Windows applications on Linux, there's one distribution intent on making that easier for you and its new release further improves that feature.
-
Linux Market Share Surpasses 4% for the First Time
Look out Windows and macOS, Linux is on the rise and has even topped ChromeOS to become the fourth most widely used OS around the globe.
-
KDE’s Plasma 6 Officially Available
KDE’s Plasma 6.0 "Megarelease" has happened, and it's brimming with new features, polish, and performance.
-
Latest Version of Tails Unleashed
Tails 6.0 is based on Debian 12 and includes GNOME 43.
-
KDE Announces New Slimbook V with Plenty of Power and KDE’s Plasma 6
If you're a fan of KDE Plasma, you'll be thrilled to hear they've announced a new Slimbook with an AMD CPU and the latest version of KDE Plasma desktop.
-
Monthly Sponsorship Includes Early Access to elementary OS 8
If you want to get a glimpse of what's in the pipeline for elementary OS 8, just set up a monthly sponsorship to help fund its continued existence.