The sys admin's daily grind: RSyslog

Where To Next?

Author(s):

Well-used services write reams of log information to disk, which is not only bothersome from a storage perspective but also pushes grep and the usual group of statistics tools to their limits. Will hitching the syslog daemon up to a database help?

Who said length doesn't matter? My spam filters alone give me a 3GB logfile daily, which would be fine if I just needed the beast to check up on the occasional error. Because I need to extract a whole bunch of statistics about spam and virus threats from the file, grepping such enormous files takes ages, besides creating far too much I/O overhead.

RSyslog [1] took me a giant leap closer to finding a solution – it logs directly to a MySQL or PostgreSQL database, meaning that I can replace my grep commands with fast SQL statements. RSyslog is included with many distributions and is the default application in Fedora 8, for example. By default, my Ubuntu lab environment runs sysklogd, making it easy for me to switch to RSyslog. For the time being, I can even keep my old syslog.conf. For the most part, RSyslog's configuration file is the same as the legacy format, although it does support a couple of additional options for linking up with the database.

First I need to set up a database. To do so, I run the createDB.sql script, which is provided with the RSyslog package:

mysql -u root -pPasswort < ./createDB.sql

This command line automatically creates a database called Syslog along with the required tables.

At the MySQL prompt, I then create a user and assign privileges:

> grant ALL ON Syslog.* to rsyslog@localhost
 identified by 'secret';
> flush privileges;

Next, telling RSyslog to use the database is refreshingly simple – just two lines in /etc/rsyslog.d/mysql.conf:

$ModLoad MySQL
mail.* >localhost,Syslog,rsyslog,secret

The first line loads the module that RSyslog needs to access the database. The second line defines the log facility containing the entries RSyslog will push to the database. The mail facility data are all I need to create spam filter statistics, followed by the access parameters for the MySQL database: hostname, database name, MySQL user name, and password.

After rebooting, I was pleased to see the RSyslog daemons filling the database (Figure 1). All done! This won't magically improve your spam statistics, but at least it will get the job done faster. Incidentally, the example here works with the current 2.0.2. stable release. RSyslog's author, Rainer Gerhards, is working hard on a 3x version, which he promises will add even more neat features.

Figure 1: RSyslog feeds log information to a database instead of a file. The database provides the back end for statistics, such as the spam information shown here.
Figure 2: If your system doesn't come with rsyslog, you can download the latest version

Infos

  1. RSyslog: http://www.rsyslog.com

The Author

Charly Kühnast is a Unix System Manager at the data center in Moers, near Germany's famous River Rhine. His tasks include ensuring firewall security and availability and taking care of the DMZ (demilitarized zone).

Read full article as PDF:

063-063_charly.pdf  (245.00 kB)

Related content

  • Security Lessons: Rsyslog

    Might as well do it properly – rsyslog.

  • Charly's Column

    Using SQL to sift syslog data out of a database is an admittedly universal, but also fairly convoluted approach. phpLogCon, with its web interface, gives admins an easier option.

  • Security Lessons: Windows Logging

    Windows 7 is pretty good at logging, but what do you do with all those log files? We look at some monitoring tools that can help you get the most out your logging data.

  • Charly's Column: lsof

    The shorter a command, the longer the list of support parameters. This rule applies to lsof, one of Charly’s favorite commands.

  • Charly's Column

    Worms, mail bombs, and users who send multiple megabyte Powerpoint files across the wire give Postfix administrators plenty of reasons to view their charges with a critical eye from time to time.

comments powered by Disqus

Direct Download

Read full article as PDF:

063-063_charly.pdf  (245.00 kB)

News

njobs Europe
What:
Where:
Country:
Njobs Netherlands Njobs Deutschland Njobs United Kingdom Njobs Italia Njobs France Njobs Espana Njobs Poland
Njobs Austria Njobs Denmark Njobs Belgium Njobs Czech Republic Njobs Mexico Njobs India Njobs Colombia