Configuring Elasticsearch

The Elasticsearch tool lets you index, organize, and search on the log messages in the Graylog message database. To configure the Elasticsearch component, you need to set up on the es-master and es-node1 VMs with the following commands:

$ wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.5.2.deb
$ sudo dpkg -i elasticsearch-1.5.2.deb

To access the logging server, you will need the web interface. Set this up on the graylog-web-interface VM:

$ wget https://packages.graylog2.org/repo/packages/graylog-1.0-repository-debian7_latest.deb $ sudo dpkg -i graylog-1.0-repository-debian7_latest.deb
$ sudo apt-get install apt-transport-https
$ sudo apt-get update
$ sudo apt-get install graylog-web

For the master VM, make the following changes to the configuration file /etc/elastic-search/elasticsearch.yml:

• cluster.name: graylog-production – Unique identifier of the cluster for the Elasticsearch component.
• node.name: es-master – Unique name of the node in the cluster of the Elasticsearch component.
• node.master: true --The node acts as a master in the Elasticsearch cluster.
• node.data: true – The node (Elasticsearch component) stores data.
• index.number_of_shards: 2 – See elasticsearch_shards in the Graylog server configuration.
• index.number_of_replicas: 1 – See elasticsearch_replicas in the Graylog server configuration.
• discovery.zen.ping.multicast.enabled: false – Disable the multicast discovery function to avoid sending multicast requests to determine the nodes in the cluster.
• discovery.zen.ping.unicast.hosts: ["es-master:9300", "es-node1:9300"] – A list of nodes that make up the Graylog production cluster. Replace the names in /etc/hosts or use the IP addresses for your systems accordingly.

You will find the complete and detailed description of each configuration parameter in the documentation [1].

Make all the same changes to the /etc/elasticsearch/elasticsearch.yml configuration file for the es-node1 VM except for the following:

• node.name: es-node1 – Unique name of the node in the cluster of the Elasticsearch component.
• node.master: false – The node does not act as master in the cluster.

Setting Up the Load Balancer

The Zen load balancer will distribute the log message traffic among the Graylog servers. We used the Zen load balancer community edition [2]. The current stable version at the time of this article was version 3.05.

You can use the Zen administration panel web interface to configure the load balancer. The web interface is reachable at https://IP_address_of_load_balanceer:444. The username and the password are admin by default.

To configure the load balancer so log messages are split across two Graylog servers, you need to create a farm in the web interface. A farm is a profile that contains the configuration for a specific network protocol (such as TCP, UDP, or HTTP) and an algorithm for load balancing. After you have created a new farm, adjust the additional configuration parameters by adding the Graylog servers that receive the log messages and entering their IP addresses and ports.

This example assumes the load balancer is configured to balance the load between two Graylog servers that receive log messages from clients via UDP with nxlog and syslog. The IP addresses and associated ports of the systems are as follows:

IP address "graylog-lb": 192.168.15.86
IP address "graylog-ms": 192.168.15.86
IP address "graylog-node1": 192.168.15.86
UDP port "nxlog": 12201
UPD port "syslog": 1514

Each UDP port has a farm. The names for the farms are GraylogL4xNAT-UDP-12201 and GraylogL4xNAT-UDP-1514. The configuration parameters for the GraylogL4xNAT-UDP-12201 farm are shown in Figure 3. Note that you select the UDP network protocol as the Protocol type. For load balancing, I have set the algorithm to Weight connection linear dispatching by weight as an example. Load distribution depends on the weighting; you set up the weighting in the next step for the two Graylog servers. Then enter the IP addresses and the corresponding ports of the two Graylog servers and set the weighting or priority according to the load distribution. (See the detailed description of each configuration parameter in the Zen documentation [3].)

Figure 3: Configuration parameters for a farm – the UDP transport protocol is an important setting.

Transferring Log Messages

Once you get the Graylog server up and running, you'll need a way for the other systems to forward their log messages to Graylog. Syslog (via TCP or UDP) is a useful choice as a client tool because it is available on most Linux systems and is typically supported by managed network devices such as routers, switches, and firewalls.

For systems that do not use syslog by default (e.g., Windows), you'll need the NXLog client software. NXLog Community Edition [4] supports multithreaded log management and various log message formats (syslog, CSV, GELF, JSON, XML, Windows EventLog). In addition to several Windows platforms, NXLog runs on several versions of Linux, as well as BSD and Android. NXLog, an open source program available free of charge [5], is a good option for mixed networks with a both Window and Linux clients. Linux – Debian 7 (wheezy) here – offers an up-to-date version available as a DEB package" (nxlog-ce-x.x.x_debian-wheezy.deb). Use the following command to install:

$ sudo dpkg -i nxlog-ce-x.x.x_debian-wheezy.deb

The NXLog configuration syntax is identical on Windows and Linux. On Windows platforms, the configuration file usually is located under C:\Program Files(x86)\nxlog\conf\nxlog.conf. The default installation configuration file is below /etc/nxlog/nxlog.conf on Debian 7. To transfer all the log messages stored in the event log of a Windows 7 client to the log server, you need an nxlog.conf configuration file that looks like Listing 3.

GELF [6] offers a number of advantages over syslog. See the entry for the IP address (192.168.15.86) and UDP port (12201) of the graylog-lb load balancer in Listing 3 (lines 20 and 21).

To transfer all the log messages stored in the Debian Linux logfile /var/log/messages to the Graylog server, add the entries in Listing 4 to the nxlog.conf configuration file.

For details on the individual configuration parameters, see the documentation for NXLog [4].