It all started when the system rebooted...
I had been having reliability problems with my system for over a month. It would run fine for up to a week or so then it would crash with wierd symptoms. I know it's unusual to trust in your software these days, but I had faith that Linux was not the culprit. Only operating systems produced by large companies have to be rebooted every day.
I took the motherboard out of the system and drove down to the supplier. The guy behind the counter had the standard "electronic supplier salesperson disease". He thought I was A. an idiot, B. trying to rip him off or C. trying to ruin his day/profit margin. I explained the problem, told him how it gave different symptoms each time it died, and how I had swapped out parts. After about 20 minutes he had no more arguments and he gave me a new motherboard.
I took it home and put it back into the case. I was back up in a few minutes and I put the system back into service. After almost three weeks of blissfull operation it rebooted itself and started back up without a problem. I didn't even know about it until I saw the system log file a day later. ARGGG! The **** thing is broken again...
I studied the logs and found that odd things had happened. The web server process log was filled with total nonsense. The system log had stopped working shortly after the reboot. I felt that a power failure had caused the odd log messages and possibly damaged the system logging program.
As I began looking at the other logs I found that someone had transferred copies of some of my files to a system I had never heard of before. This was serious! I had been violated! I didn't have hardware problems, some sleazoid-weasel had broken into my system! I had previously been over the system carefully trying to eliminate all the security holes. I hadn't been careful enough!
I copied off every log file I could find and immediately changed all the passwords on the system. If they had gotten in and copied the password file they could eventually crack the encoding on their own system and they would have all the passwords.
I sent off a message to the system administrator of the system that the files had been sent to. With a little time at a search engine site I found that this system was located in Chicago. I later found out from the site's system administrator that this guy had somehow broken through the security in one of their systems routers. Once into the router he installed a packet sniffer. This program reads the data packets that go across the net and records anything that looks like a password.
I had been connecting to my system remotely to get mail from it. I have since found out that the POP3 protocol used to get mail sends your account password in clear text (unencrypted) when getting your mail. This sleazy booger's packet sniffer probably captured my password when I was getting my mail. The rlogin, rsh, rexec, rlp, telnet, adn FTP protocols also send passwords in clear text by the way!
I went through the '/etc/services' file one more time and found that I had not disabled the 'rlogin' service as I had first thought. This service runs on port 513 but is not called 'rlogin'. I went through and disabled every service that starts with an 'r'. These are the remote services programs that a cracker can use to get into your system. I disabled all file sharing and all protocols except tcp/ip. I disabled the telnet service altogether since there is a better replacement. I also made sure that NFS and RPC were disabled since there was supposed to be a security hole in these too.
Well, not a lot had been done to my system, other than the reboot after the break-in. One nagging thing was that the system logging no longer worked. After goofing around with it for a day or so I finally noticed what should have been obvious. The 'syslogd' program had been replaced with another program with the same name.
I haven't verified it but I believe this program is another copy of the packet sniffer the cracker used in the router. When you do a 'ps' to see what's running you wouldn't think anything about it since this program should be running all the time. I replaced the 'syslogd' program with the correct one and it worked like a champ again.
While poking around in my /tmp directory I found a copy of the 'bash' shell with the SUID bit set. WHOA! What's this? With this little baby you can become root by simply running it. When I happened to mention this to a fine gentleman [Jim Dennis, The Answer Guy --Editor] who was helping me try to get it working he immediately remembered the security hole associated with this. There's a bug with the 'sendmail' program that allows you to make an SUID copy of your shell in the /tmp directory. If you don't have version 8.8.3 or later of the sendmail program you're vulnerable too! (go to http://www.sendmail.org for the latest stuff).
So, what have I learned from all this?
best of luck to you!