Thoughts On ...

May 28, 2005

Where The Site Went

For anyone who has been checking, it would've been obvious that this site has been unavailable for the past few weeks. I thought I'd take a moment to explain why.

In short: The web server was hacked.

The first I knew about it was the emails I began receiving from sysadmins that my machine was attacking their machine with brute-force ssh login attempts. A quick inspection of my machine (plus log snippets from the sysadmins and my host) confirmed that the system had been compromised, and I took the box offline to investigate.

After much exploration (log analysis, sniffer traces, find and date modified searches, etc) I discovered that someone had compromised my machine, installed a rootkit and some automated scripts to launch the attacks. While the damage to my machine was minimal (a couple of log files were truncated), it was clear that root had been compromised, necessitating a complete reinstall of the machine.

Before that was possible, I had to (for peace of mind) determine how the attacker got in. With some great help from Alex Jackson (one of my colleagues at ThoughtWorks) we found out that the breach occurred via a relatively new exploit in awstats -- a freely available and widely used web statistics tool. It was clear from the logs that the attackers were looking for it randomly, and once found were able to execute code on my machine (to download and start a telnet daemon). From there, they got their root kit aboard, and the rest is history.

So what's been taking so long? Well, despite the fact that the hole was a 3rd party cgi script (and cgi was obviously insecure on my machine), I decided to do a complete security audit and implement a more robust security strategy for the site this time around. While the details would themselves make a great article, I'd rather not describe the security layout in detail.

However, general lessons learned (and implemented):

  1. Take security seriously (I thought I was already!).
  2. Secure the cgi-bin.
  3. Be obsessively minimal about what is installed and/or publicly accessible on the server.
  4. Keep track of all security updates for everything that is installed -- if that is not possible, that software doesn't belong on the box.
  5. Find, read, absorb and implement all the security information available for the OS/distribution in use. Many of the major linux distros have howtos on this topic. Read them.
  6. Use Security in Depth principles (i.e. more than one layer of security). Limit access, limit software, keep up to date, install intrusion detection, use firewalls, etc.).
  7. Security of a system must be appropriate to its intended use -- too much creates an unnecessary configuration burden (which itself is a security risk) too little drives the risk of compromise above the cost of securing the box.
  8. Be Paranoid. Security time spent now saves a lot of restoration time spent later.
  9. Be Prepared. Backup data (I was lucky here). Assume the site will be compromised at some point and keep a system image ready to go in an offline location (I wasn't so lucky here). Secure that image *before* and back it up *before* bringing the system online.
  10. Use Webalizer instead of Awstats (Webalizer doesn't even optionally use cgi, and I like its reports better anyway).

One good thing that came from this is that I have learned a lot more about security than I knew before. Its always been a topic of interest to me, but nothing like getting hit close to home to turn an academic subject into a practical one.

Given the hiatus, you'd think I'd have a boatload of entries saved up ready to post -- well you'd be half right. I have been doing some writing, I do have plenty of drafts saved up (and even more entry ideas) but very little is cut-and-paste ready. However, fear not! While its been tough going these past few weeks, the site is now officially back online, and additional entries are forthcoming.

Thank you for your patience through this time.

I also would like to extend a thank you to Martin Norland (one of the sysadmins my machine attacked) for his help in identifying the attack, his understanding and support. I would also like to thank Matthew Bloch and the rest of the gang at Bytemark who have been great through all of this. These guys are the best site hosts I have ever exprienced -- if you want to run your own web server, I couldn't recommend them and their service enough (if you sign up feel free to give me as a reference -- they offer a small incentive, and given the amount of lost time, I could sure use it :-) ).

Finally, for those interested in learning more about securing a web server, here are some of the articles I read while working on this site (Google and the references in these articles provided many more):

Posted by wcaputo at May 28, 2005 04:29 PM
Comments