6. Intrusion Detection

This section will deal with how to get early warning, how to be alerted after the fact, and how to clean up from intrusion attempts.

6.1. Intrusion Detection Systems (IDS)

Intrusion Detection Systems (IDS for short) are designed to catch what might have gotten past the firewall. They can either be designed to catch an active break-in attempt in progress, or to detect a successful break-in after the fact. In the latter case, it is too late to prevent any damage, but at least we have early awareness of a problem. There are two basic types of IDS: those protecting networks, and those protecting individual hosts.

For host based IDS, this is done with utilities that monitor the filesystem for changes. System files that have changed in some way, but should not change -- unless we did it -- are a dead give away that something is amiss. Anyone who gets in, and gets root, will presumably make changes to the system somewhere. This is usually the very first thing done. Either so he can get back in through a backdoor, or to launch an attack against someone else. In which case, he has to change or add files to the system.

This is where tools like tripwire (http://www.tripwire.org) play a role. Tripwire is included beginning with Red Hat 7.0. Such tools monitor various aspects of the filesystem, and compare them against a stored database. And can be configured to send an alert if any changes are detected. Such tools should only be installed on a known "clean" system.

For home desktops and home LANs, this is probably not an absolutely necessary component of an overall security strategy. But it does give peace of mind, and certainly does have its place. So as to priorities, make sure the Steps 1, 2 and 3 above are implemented and verified to be sound, before delving into this.

We can get somewhat the same results with rpm -Va, which will verify all packages, but without all the same functionality. For instance, it will not notice new files added to most directories. Nor will it detect files that have had the extended attributes changed (e.g. chattr +i, man chattr and man lsattr). For this to be helpful, it needs to be done after a clean install, and then each time any packages are upgraded or added. Example:

 # rpm -Va > /root/system.checked


Then we have a stored system snapshot that we can refer back to.

Another idea is to run chkrootkit (http://www.chkrootkit.org/) as a weekly cron job. This will detect common "rootkits".

6.2. Have I Been Hacked?

Maybe you are reading this because you've noticed something "odd" about your system, and are suspicious that someone was gotten in? This can be a clue.

The first thing an intruder typically does is install a "rootkit". There are many prepackaged rootkits available on the Internet. The rootkit is essentially a script, or set of scripts, that makes quick work of modifying the system so the intruder is in control, and he is well hidden. He does this by installing modified binaries of common system utilities and tampering with log files. Or by using special kernel modules that achieve similar results. So common commands like ls may be modified so as to not show where he has his files stored. Clever!

A well designed rootkit can be quite effective. Nothing on the system can really be trusted to provide accurate feedback. Nothing! But sometimes the modifications are not as smooth as intended and give hints that something is not right. Some things that might be warning signs:

Sometimes the intruder is not so smart and forgets about root's .bash_history, or cleaning up log entries, or even leaves strange, leftover files in /tmp. So these should always be checked too. Just don't necessarily expect them to be accurate. Often such left behind files, or log entries, will have obvious script kiddie sounding names, e.g. "r00t.sh".

Packet sniffers, like tcpdump (http://www.tcpdump.org), might be useful in finding any uninvited traffic. Interpreting sniffer output is probably beyond the grasp of the average new user. snort (http://www.snort.org), and ethereal (http://www.ethereal.com), are also good. Ethereal has a GUI.

As mentioned, a compromised system will undoubtedly have altered system binaries, and the output of system utilities is not to be trusted. Nothing on the system can be relied upon to be telling you the whole truth. Re-installing individual packages may or may not help since it could be system libraries or kernel modules that are doing the dirty work. The point here is that there is no way to know with absolute certainty exactly what components have been altered.

We can use rpm -Va |less to attempt to verify the integrity all packages. But again there is no assurance that rpm itself has not been tampered with, or the system components that RPM relies on.

If you have pstree on your system, try this instead of the standard ps. Sometimes the script kiddies forget about this one. No guarantees though that this is accurate either.

You can also try querying the /proc filesystem, which contains everything the kernel knows about processes that are running:

 # cat /proc/*/stat | awk '{print $1,$2}'

This will provide a list of all processes and PID numbers (assuming a malicious kernel module is not hiding this).

Another approach is to visit http://www.chkrootkit.org, download their rootkit checker, and see what it says.

Some interesting discussions on issues surrounding forensics can be found at http://www.fish.com/security/. There is also a collection of tools available, aptly called "The Coroner's Toolkit" (TCT).

Read below for steps on recovering from an intrusion.

6.3. Reclaiming a Compromised System

So now you've confirmed a break-in, and know that someone else has root access, and quite likely one or more hidden backdoors to your system. You've lost control. How to clean up and regain control?

There is no sure fire way of doing this short of a complete re-install. There is no way to find with assurance all the modified files and backdoors that may have been left. Trying to patch up a compromised system risks a false sense of security and may actually aggravate an already bad situation.

The steps to take, in this order:

At this time, any rootkit cleanup tools that may be available on-line are not recommended. They probably do work just fine most of the time. But again, how to be absolutely sure that all is well and all vestiges of the intrusion are gone?