Filtering log messages with Splunk
Search with Remote Control
The post()
method in line 57 sends a search command to the Splunk server. In contrast to the web GUI, searches submitted via the API need to start with the search
command. Besides the NOT eventtype=chatter
filter that I already described, it defines the restriction earliest=-24h
; that is, it only asks for events in the past 24 hours. As defined by the output_mode
parameter, I want Splunk to return the results in JSON format. If you prefer to avoid lengthy emails, you will also want to restrict the number of hits to 50 using limit=50
.
The from_json()
function from the JSON CPAN module then converts the results to Perl data structures one line at a time in line 82. Three fields are crucial for the mail to be sent: the time stamp of the log entry with the _time
key, the logfile in source
, and the original log line in _raw
.
The Net::SMTP module from CPAN sends the mail with the results of the search to the target defined in $to_email
. The SMTP server $smtp_server
was set previously in line 23.
For Dinosaurs and Hipsters
Complex HTML messages annoy old codgers like me that use text-based email readers such as Pine. Conversely, a plain text email is too old school for young, dynamic Outlook and Thunderbird mouse pushers. To mediate between the two worlds, Listing 1 formats the tabular results of the query in ASCII using the CPAN Text::ASCIITable module. To prevent the timestamp column from becoming too long, and to wrap it instead, line 76 limits its width to a maximum of 10 characters. The same thing applies to the column with the log entry, which wraps to a line length of 34, keeping messages readable, even on mobile phones.
Some modern mail readers prefer HTML, and to satisfy them, line 96 calls the CPAN Email::MIME module. It wraps the existing ASCII text in inline HTML, surrounded by simple pre
tags. Thus, the results are acceptable both in Alpine (Figure 7) and in Gmail (Figure 8).
The script can be easily extended to include tests that compare the values found with previously set limits and then only send messages when the limit is exceeded. This can happen once per day as a summary or at five-minute intervals for rapid-alert email messages.
Some open source alternatives to Splunk, such as Logstash [6] [7] and Graylog2 [8], are also available, but so far they do not come close to Splunk in terms of ease of use and scalability.
Infos
- Splunk: http://www.splunk.com
- Hadoop: http://hadoop.apache.org
- "Giant Data: MapReduce and Hadoop" by Thomas Hornung, Martin Przyjaciel-Zablocki, and Alexander Schätzle: http://www.admin-magazine.com/HPC/Articles/MapReduce-and-Hadoop
- Listings for this article: ftp://ftp.linux-magazin.de/pub/listings/magazine/155
- "Splunk: Intro REST API tutorial": http://dev.splunk.com/view/SP-CAAADQT
- Logstash: http://logstash.net
- "Centralized Log Archiving with Logstash" by Martin Loschwitz, Linux Magazine, June 2013, p. 60: http://www.linux-magazine.com/Issues/2013/151/Logstash/(language)/eng-US
- Graylog2: http://graylog2.org
« Previous 1 2 3
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
![Learn More](https://www.linux-magazine.com/var/linux_magazin/storage/images/media/linux-magazine-eng-us/images/misc/learn-more/834592-1-eng-US/Learn-More_medium.png)
News
-
NVIDIA Released Driver for Upcoming NVIDIA 560 GPU for Linux
Not only has NVIDIA released the driver for its upcoming CPU series, it's the first release that defaults to using open-source GPU kernel modules.
-
OpenMandriva Lx 24.07 Released
If you’re into rolling release Linux distributions, OpenMandriva ROME has a new snapshot with a new kernel.
-
Kernel 6.10 Available for General Usage
Linus Torvalds has released the 6.10 kernel and it includes significant performance increases for Intel Core hybrid systems and more.
-
TUXEDO Computers Releases InfinityBook Pro 14 Gen9 Laptop
Sporting either AMD or Intel CPUs, the TUXEDO InfinityBook Pro 14 is an extremely compact, lightweight, sturdy powerhouse.
-
Google Extends Support for Linux Kernels Used for Android
Because the LTS Linux kernel releases are so important to Android, Google has decided to extend the support period beyond that offered by the kernel development team.
-
Linux Mint 22 Stable Delayed
If you're anxious about getting your hands on the stable release of Linux Mint 22, it looks as if you're going to have to wait a bit longer.
-
Nitrux 3.5.1 Available for Install
The latest version of the immutable, systemd-free distribution includes an updated kernel and NVIDIA driver.
-
Debian 12.6 Released with Plenty of Bug Fixes and Updates
The sixth update to Debian "Bookworm" is all about security mitigations and making adjustments for some "serious problems."
-
Canonical Offers 12-Year LTS for Open Source Docker Images
Canonical is expanding its LTS offering to reach beyond the DEB packages with a new distro-less Docker image.
-
Plasma Desktop 6.1 Released with Several Enhancements
If you're a fan of Plasma Desktop, you should be excited about this new point release.