They tried to get my server to join a BotNet

I first noticed a mysterious connection on a netstat:

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 1 mrtg.sampas.net:42321 223-118-92-213.server:49153 SYN_SENT

I also noticed Apache had opened a shell:

F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
1 S apache 18005 1 0 76 0 - 1282 - Apr10 ? 00:00:00 sh -i

Soon after, perl became a runaway process, consuming 100% of my CPU time. And I thought /sbin/nologin meant user Apache couldn't just get a shell. I updated zlib from 1.2.2 to 1.2.3 to fix a security hole. up2date -u reports everything is up-to-date. (It did that for my old zlib, too.) I don't see any new holes in my applications, MT and Gallery. I did a Nessus scan with recent updates, and all it showed no holes and one warning. I ran clamscan and it didn't find anything, either. Rkhunter found nothing, and nikto gave me the following output:

+ Server: Apache/2.0.52 (Red Hat)
+ Allowed HTTP Methods: GET,HEAD,POST,OPTIONS,TRACE
+ Apache/2.0.52 appears to be outdated (current is at least Apache/2.2.3). Apache 1.3.33 is still maintained and considered secure.
+ / - TRACE option appears to allow XSS or credential theft. See http://www.cgisecurity.com/whitehat-mirror/WhitePaper_screen.pdf for details (TRACE)
+ /usage/ - Webalizer may be installed. Versions lower than 2.10-09 vulnerable to Cross Site Scripting (XSS). CA-2000-02. (GET)

So now I'm stuck looking through my apache access logs, because that's the only thing exposed to the outside world.

I did a capture just while open my firewall for a couple of minutes, and I saw it try to log in to an IRC channel. Ouch. I've been pwned. Fortunately, my firewall stops my server from being used for attacks, and I was able to block the port range used by the IRC bot.

Finally, Red Hat released a lot of new patches for PHP, and I set PerlTaintCheck On in /etc/httpd/conf.d/perl.conf, which was the real problem: user Apache had started listening on port 80 using Perl, so I couldn't even restart httpd.

Next time, I need to check the logs closer and post a network capture of the login process.

I graduate from grad school, get a new job, a new house, and become a father.

I graduated from Grad School with a Master's in Information Systems Technology, focusing on Management Information Systems.

It's official. George Washington University sent me my degree in the mail. They took three and a half months to get it out. Even the registrar didn't change my status until March after a couple of phone calls. A lot has happened since then: I moved into a new (old) house, started a new job, and am about to become a father.

What did I really learn in grad school anyway? I learned a lot, but every class covered, to some degree, entity-relationship diagrams (ERDs), data-flow diagrams (DFDs), and object-oriented diagrams, which can be state charts, class diagrams, and use cases, to name a few. Some classes went so far as to cover the theory behind them. Every class covered the relational database model, which hasn't changed much in thirty years and is still useful and relevant to just about every information system I've ever worked with.

Since IS grad school is part of the business school, we learned to work in teams. It's not about writing code -- it's about finishing projects on time. The funny part is the professors don't teach much about team projects -- they just expect you to manage yourselves.