Better Logging

As I mentioned, I had disabled SELinux instead of leaving it in permissive mode. With hard-disk space relatively cheap now, it rarely makes sense to forego audit logs to save disk space on most systems.

Additionally, with the relative ease of setting up network logging for Syslog, there is no excuse not to centrally log events (thus making it much harder for an attacker to erase any trail they leave). However, note that many attacks will leave little if any meaningful messages in the logs.

Often, a message saying that a server program has crashed will be all that is left, which hardly warrants a full-scale investigation each time it occurs.

Having a trustworthy audit trail can mean the difference between needing to rebuild an entire group of machines and only needing to clean off one. Logging also is important in providing the information needed to fix a system. Simply restoring a system to its original state won't prevent the attacker from breaking in again.

Backups

Often backups are a neglected side of information security. Backups are crucial for ensuring that systems can be rebuilt properly and data restored if necessary. If you back up nothing else, at least back up your data and make sure to store it offsite. Also, having restoration systems that work is just as important.

Too often, I have seen backups work perfectly until they were needed, bare-metal recovery images that were incompatible with replacement hardware, or exports of data that were corrupted or missing data. For example, I have a backup script that was quietly failing for three months because I forgot the "overwrite older files" option, so all the changed files had not been backed up in some time.

Pre-Mortems

One of the most powerful practices I've seen is conducting a "pre-mortem." Essentially, you sit down and assume the system has already failed. Having an idea of what to do in this scenario will help prevent confusion if it ever does happen and can offer insight into preemptive action to take, such as loading host-based firewalls on every computer, regardless of whether the machine is attached to a public network.