Automatic Apache

A more serious use for incron would be to monitor a server's configuration files and order a reboot if anything changes, such as a modified Apache web server httpd.conf or apache2.con file. Begin by finding out which user has privileges to stop and restart Apache on your system and edit their incrontab.

For this exercise, assume that user is root, become superuser, and open root's incrontab for editing:

$ su
# incrontab -e

If you get an error that reads user 'root' is not allowed to use incron, edit /etc/incron.allow and add root to the list of allowed users. Then, insert the following line into root's incrontab file:

/etc/apache2/apache2.conf IN_MODIFY /etc/init.d/apache2 restart

After modifying httpd.conf, or apache2.conf (e.g., by changing the Timeout value), and saving the file, look at /var/log/apache2/error.log (Figure 1). You'll see that Apache was restarted automatically when you hit the Save button (or typed :wq).

Figure 1: Apache is restarted automatically every time you modify its main config file.

However, you probably know that it's been a long time since Apache has had only one configuration file. Apache's configuration has been modularized and sometimes approaches tens of files spread out over several nested directories. To monitor the whole lot, you can add the following to your root's incrontab:

/etc/apache2/conf.d/ IN_CREATE /etc/init.d/apache2 restart

This method is not very subtle, but, again, it'll work. You've probably figured out that this code restarts the web server every time a new file is added to the /etc/apache2/conf.d/ directory. It doesn't check whether it is a *.conf file or not, but because Apache's conf.d directory was created for those kinds of files, it doesn't really matter.

Furthermore, you're going to want to monitor the files within the conf.d directory to see whether any of the files change. To do that, you can modify the incrontab to:

/etc/apache2/conf.d/ IN_CREATE,IN_CLOSE_WRITE /etc/init.d/apache2 restart

As you can see, if you want to monitor more than one event, you have to separate the triggers with commas. In this case, you are monitoring for both the  IN_CREATE and IN_CLOSE_WRITE events, because you also want to restart Apache when a configuration file closes after it has been modified.

Note that your root's incrontab file is now two lines long, one for each directory. You're probably thinking that using two lines for a directory and one of its subdirectories is inefficient, and that it would be more elegant if you had a way to monitor the upper directory and then drill down.

You'd be wrong.

Incron does not support recursive directory monitoring, nor are there any plans for it to do so, and there's a good reason for this decision: infinite recursive loops.

To Infinity and Beyond

Imagine that you want to monitor the files in a directory and write an entry in a log each time a file is modified. You save the log for convenience in the same directory you're monitoring.

The incrontab could look like this:

</path/to/directory/> IN_CLOSE_WRITE log_changes $#

where log_changes is the following Bash script:

#!/bin/bash
echo "`date` File $1 modified" >> </path/to/directory/>my.log

Do you start to see the problem? You've created an infinite loop.

When you finish writing the entry in my_log, the IN_CLOSE_WRITE event fires because, you know, my_log is a file in the monitored directory, which was closed after being written to. A new entry is written in my_log, and the IN_CLOSE_WRITE event fires again, because my_log is a file in the monitored directory, which was closed after being written to. IN_CLOSE_WRITE fires again and  … . You get the picture. Bad things would happen if you let something like that run unchecked.

To avoid this kind of situation, every sys admin worth his salt has two tools available: (1) common sense and (2) the IN_NO_LOOP wildcard event. This element blocks the incrontab instruction until it's completely finished, avoiding the other event from firing in the same iteration, and thus avoiding loops.

Although the following is a lousy, lazy, shoddy solution (the best move would be to store your logfile elsewhere), you can solve the problem described above with the following incrontab line:

</path/to/directory/> IN_CLOSE_WRITE, IN_NO_LOOP log_changes $#

So, you can see that drilling down into subdirectories of subdirectories of subdirectories (including soft-linked directories), if it were implemented in incron, would increase significantly the chances of infinite loops, so no recursive monitoring is allowed in incron.

Silent Alarm

In the next example, I'm going to implement a security system against intruders who try to access the system and directories containing sensitive documents. I'm not talking about that folder called secret_CIA_documents that everybody has in their home directory.

If you have learned anything from the likes of the NSA, LulzSec, and Anonymous, it is that the files most coveted by attackers are email messages. Email contains usernames, passwords, bank accounts, credit card numbers, personal images, business contracts, and on and on. A frequently used email folder is a true treasure trove for your friendly neighborhood snooper.

However, monitoring a real email folder would be a nightmare, with so many writes, rewrites, deletions, and changes making it impossible to filter out false positives. Also, are you sure you want an attacker to get as far as being able to read your messages? No, you don't. What you'll do instead is construct a honeypot by creating a .Mail directory hanging off your home directory and filling it with messages culled from, say, your spam folder. That should make for fun reading. If your email client already points to that folder to store messages, CHANGE IT! It's way too exposed.

Incron will monitor your honeypot directory as before, but with a twist: Your action will depend on what the intruder does. If the intruder opens a file for reading, you'll activate the silent alarm and warn the system administrator. However, if the intruder starts writing in files, you'll disconnect the machine from the network with, for example

ifconfig eth0 down

The problem with this solution is that, apparently, you need two lines to call two different scripts, depending on the triggered event. Something like:

/home/<user>/.Mail IN_ACCESS warn_admin.sh
/home/<user>/.Mail IN_CLOSE_WRITE close_connection.sh

But that won't work because incron doesn't allow you to monitor the same thing twice. If you try to run these lines to trigger one of the events, you will get an error in /var/log/cron that says:

Jul 24 21:17:54
host
incrond[9454]:
cannot create
watch for user root:
(16) Device or resource busy

To solve this problem, you can combine both events onto one line and use the predefined incron variable $%:

/home/paul/.Mail IN_ACCESS, IN_CLOSE_WRITE access_control.sh $%

The $% wildcard is passed to your script as an argument, telling it which event was triggered, so you can then deal with it as shown in Listing 1.