System monitoring for a new generation with Prometheus
Push Gateway
All services cannot be meaningfully run with exporters. This is particularly problematic for services that generate data on the fly and need to offload the data immediately. One classic example is Cron, which handles a specific task and generates output while doing so, which Prometheus then needs to process. Prometheus has the push gateway to handle this kind of situation.
The gateway runs as a separate service with a RESTful interface and waits for the required values to arrive. If you want to deliver the Cron job output to the push gateway, all you need to do is append a pipe and a Curl call to the Cron job. The REST interface on the push gateway accesses the target host for Curl.
Of course, these gateways will scale horizontally; in other words you can run multiple gateways in parallel in the same installation. The Prometheus servers pick up the stored data from the gateways in exactly the same way they do with normal exporters. But, there is one slight drawback: The data output by the Cron job needs to exist in a format that the push gateway actually understands. In other words, you may need to tune the Cron output.
Alert Manager
If something is going wrong in the cluster, administrators would naturally expect the monitoring system to tell them. Prometheus has two possible approaches. Prometheus itself offers a basic alerting function: if the value of a metric is outside of defined boundaries over a defined period of time, Prometheus sends a message to a previously defined address.
However, the developers expressly point out that this function is only basic alerting, and the number of tweaks required for modifying the alerting style to meet your local needs limits flexibility.
But Prometheus offers a second approach: instead of handing alerting itself, Prometheus can forward alerts directly to the Alert Manager. The Alert Manager is a separate component in the Prometheus universe (Figure 4), and it is probably one of the biggest construction sites in Prometheus right now.

When this issue went to press, the Alert Manager GitHub page [8] featured a statement to the effect that the version in the master branch is a totally new development that is no longer compatible with the current stable version. If you want to take a look at, or test, Prometheus, you will still want to go for this version: the legacy version will disappear in the foreseeable future.
As an interface to Prometheus, the Alert Manager has its own API. When the Alert Manager receives an alert from Prometheus, it can process that alert in a variety of ways: the manager can mail a warning or forward the alert to the Pager Duty API.
The pager, in turn, supports various modules. Out of the box are Pager Duty modules for Slack, Hipchat, and various other chat protocols. But you could hitch up Pager Duty to a text service with a matching SMS module.
PromDash
PromDash (Figure 5) is a powerful tool for rendering the metrics stored in Prometheus in a meaningful way. The tool's developers describe PromDash as a dashboard builder, thus setting it apart from standard dashboards.
PromDash does not try to give users prebuilt templates for charts but instead gives administrators a tool for creating these templates themselves. PromDash can thus rightly be described as a graphical helper that facilitates the use of the PromQL language.
After setting up the solution, you need to add the servers that will act as data sources in PromDash, then sit back and enjoy the ride. One of PromDash's features is the ability to annotate graphs or integrate them directly into other pages. For example, you could easily implement a status dashboard for the entire platform that would give you a centralized overview of all the systems you need to monitor.
The solution for configuring PromDash itself is also very smart: The PromDash module stores the configured dashboards in a MySQL database in the background. This design means the configuration and program are separate, so you can run any number of PromDash instances on the same host without you needing to manually synchronize the configurations.
« Previous 1 2 3 4 Next »
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Direct Download
Read full article as PDF:
Price $2.95
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Find SysAdmin Jobs
News
-
MNT Seeks Financial Backing for New Seven-Inch Linux Laptop
MNT Pocket Reform is a tiny laptop that is modular, upgradable, recyclable, reusable, and ships with Debian Linux.
-
Ubuntu Flatpak Remix Adds Flatpak Support Preinstalled
If you're looking for a version of Ubuntu that includes Flatpak support out of the box, there's one clear option.
-
Gnome 44 Release Candidate Now Available
The Gnome 44 release candidate has officially arrived and adds a few changes into the mix.
-
Flathub Vying to Become the Standard Linux App Store
If the Flathub team has any say in the matter, their product will become the default tool for installing Linux apps in 2023.
-
Debian 12 to Ship with KDE Plasma 5.27
The Debian development team has shifted to the latest version of KDE for their testing branch.
-
Planet Computers Launches ARM-based Linux Desktop PCs
The firm that originally released a line of mobile keyboards has taken a different direction and has developed a new line of out-of-the-box mini Linux desktop computers.
-
Ubuntu No Longer Shipping with Flatpak
In a move that probably won’t come as a shock to many, Ubuntu and all of its official spins will no longer ship with Flatpak installed.
-
openSUSE Leap 15.5 Beta Now Available
The final version of the Leap 15 series of openSUSE is available for beta testing and offers only new software versions.
-
Linux Kernel 6.2 Released with New Hardware Support
Find out what's new in the most recent release from Linus Torvalds and the Linux kernel team.
-
Kubuntu Focus Team Releases New Mini Desktop
The team behind Kubuntu Focus has released a new NX GEN 2 mini desktop PC powered by Linux.