Centralized log management with Graylog
Configuring Elasticsearch
The Elasticsearch tool lets you index, organize, and search on the log messages in the Graylog message database. To configure the Elasticsearch component, you need to set up on the es-master
and es-node1
VMs with the following commands:
$ wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.5.2.deb $ sudo dpkg -i elasticsearch-1.5.2.deb
To access the logging server, you will need the web interface. Set this up on the graylog-web-interface
VM:
$ wget https://packages.graylog2.org/repo/packages/graylog-1.0-repository-debian7_latest.deb $ sudo dpkg -i graylog-1.0-repository-debian7_latest.deb $ sudo apt-get install apt-transport-https $ sudo apt-get update $ sudo apt-get install graylog-web
For the master VM, make the following changes to the configuration file /etc/elastic-search/elasticsearch.yml
:
cluster.name: graylog-production
– Unique identifier of the cluster for the Elasticsearch component.node.name: es-master
– Unique name of the node in the cluster of the Elasticsearch component.node.master: true
--The node acts as a master in the Elasticsearch cluster.node.data: true
– The node (Elasticsearch component) stores data.index.number_of_shards: 2
– Seeelasticsearch_shards
in the Graylog server configuration.index.number_of_replicas: 1
– Seeelasticsearch_replicas
in the Graylog server configuration.discovery.zen.ping.multicast.enabled: false
– Disable the multicast discovery function to avoid sending multicast requests to determine the nodes in the cluster.discovery.zen.ping.unicast.hosts: ["es-master:9300", "es-node1:9300"]
– A list of nodes that make up the Graylog production cluster. Replace the names in/etc/hosts
or use the IP addresses for your systems accordingly.
You will find the complete and detailed description of each configuration parameter in the documentation [1].
Make all the same changes to the /etc/elasticsearch/elasticsearch.yml
configuration file for the es-node1
VM except for the following:
node.name: es-node1
– Unique name of the node in the cluster of the Elasticsearch component.node.master: false
– The node does not act as master in the cluster.
Setting Up the Load Balancer
The Zen load balancer will distribute the log message traffic among the Graylog servers. We used the Zen load balancer community edition [2]. The current stable version at the time of this article was version 3.05.
You can use the Zen administration panel web interface to configure the load balancer. The web interface is reachable at https://IP_address_of_load_balanceer:444. The username and the password are admin by default.
To configure the load balancer so log messages are split across two Graylog servers, you need to create a farm in the web interface. A farm is a profile that contains the configuration for a specific network protocol (such as TCP, UDP, or HTTP) and an algorithm for load balancing. After you have created a new farm, adjust the additional configuration parameters by adding the Graylog servers that receive the log messages and entering their IP addresses and ports.
This example assumes the load balancer is configured to balance the load between two Graylog servers that receive log messages from clients via UDP with nxlog
and syslog
. The IP addresses and associated ports of the systems are as follows:
IP address "graylog-lb": 192.168.15.86 IP address "graylog-ms": 192.168.15.86 IP address "graylog-node1": 192.168.15.86 UDP port "nxlog": 12201 UPD port "syslog": 1514
Each UDP port has a farm. The names for the farms are GraylogL4xNAT-UDP-12201 and GraylogL4xNAT-UDP-1514. The configuration parameters for the GraylogL4xNAT-UDP-12201 farm are shown in Figure 3. Note that you select the UDP network protocol as the Protocol type. For load balancing, I have set the algorithm to Weight connection linear dispatching by weight as an example. Load distribution depends on the weighting; you set up the weighting in the next step for the two Graylog servers. Then enter the IP addresses and the corresponding ports of the two Graylog servers and set the weighting or priority according to the load distribution. (See the detailed description of each configuration parameter in the Zen documentation [3].)
Transferring Log Messages
Once you get the Graylog server up and running, you'll need a way for the other systems to forward their log messages to Graylog. Syslog (via TCP or UDP) is a useful choice as a client tool because it is available on most Linux systems and is typically supported by managed network devices such as routers, switches, and firewalls.
For systems that do not use syslog by default (e.g., Windows), you'll need the NXLog client software. NXLog Community Edition [4] supports multithreaded log management and various log message formats (syslog, CSV, GELF, JSON, XML, Windows EventLog). In addition to several Windows platforms, NXLog runs on several versions of Linux, as well as BSD and Android. NXLog, an open source program available free of charge [5], is a good option for mixed networks with a both Window and Linux clients. Linux – Debian 7 (wheezy) here – offers an up-to-date version available as a DEB package" (nxlog-ce-x.x.x_debian-wheezy.deb
). Use the following command to install:
$ sudo dpkg -i nxlog-ce-x.x.x_debian-wheezy.deb
The NXLog configuration syntax is identical on Windows and Linux. On Windows platforms, the configuration file usually is located under C:\Program Files(x86)\nxlog\conf\nxlog.conf
. The default installation configuration file is below /etc/nxlog/nxlog.conf
on Debian 7. To transfer all the log messages stored in the event log of a Windows 7 client to the log server, you need an nxlog.conf
configuration file that looks like Listing 3.
GELF [6] offers a number of advantages over syslog. See the entry for the IP address (192.168.15.86) and UDP port (12201) of the graylog-lb
load balancer in Listing 3 (lines 20 and 21).
Listing 3
nxlog.conf
To transfer all the log messages stored in the Debian Linux logfile /var/log/messages
to the Graylog server, add the entries in Listing 4 to the nxlog.conf
configuration file.
Listing 4
nxlog.conf Additions
For details on the individual configuration parameters, see the documentation for NXLog [4].
« Previous 1 2 3 Next »
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
Canonical Bumps LTS Support to 12 years
If you're worried that your Ubuntu LTS release won't be supported long enough to last, Canonical has a surprise for you in the form of 12 years of security coverage.
-
Fedora 40 Beta Released Soon
With the official release of Fedora 40 coming in April, it's almost time to download the beta and see what's new.
-
New Pentesting Distribution to Compete with Kali Linux
SnoopGod is now available for your testing needs
-
Juno Computers Launches Another Linux Laptop
If you're looking for a powerhouse laptop that runs Ubuntu, the Juno Computers Neptune 17 v6 should be on your radar.
-
ZorinOS 17.1 Released, Includes Improved Windows App Support
If you need or desire to run Windows applications on Linux, there's one distribution intent on making that easier for you and its new release further improves that feature.
-
Linux Market Share Surpasses 4% for the First Time
Look out Windows and macOS, Linux is on the rise and has even topped ChromeOS to become the fourth most widely used OS around the globe.
-
KDE’s Plasma 6 Officially Available
KDE’s Plasma 6.0 "Megarelease" has happened, and it's brimming with new features, polish, and performance.
-
Latest Version of Tails Unleashed
Tails 6.0 is based on Debian 12 and includes GNOME 43.
-
KDE Announces New Slimbook V with Plenty of Power and KDE’s Plasma 6
If you're a fan of KDE Plasma, you'll be thrilled to hear they've announced a new Slimbook with an AMD CPU and the latest version of KDE Plasma desktop.
-
Monthly Sponsorship Includes Early Access to elementary OS 8
If you want to get a glimpse of what's in the pipeline for elementary OS 8, just set up a monthly sponsorship to help fund its continued existence.