Has my new Linode been compromised already?
Aug 27 01:49:26 zembla kernel: atack[25830]: segfault at 0 ip
0000000008048e33 sp 00000000ffdb41b0 error 4 in atack[8048000+c0000]
Aug 27 01:49:26 zembla kernel: atack[25836]: segfault at 0 ip 0000000008048e33 sp 00000000ffdb41b0 error 4 in atack[8048000+c0000]
Aug 27 01:49:26 zembla kernel: atack[25835]: segfault at 0 ip 0000000008048e33 sp 00000000ffdb41b0 error 4 in atack[8048000+c0000]
...
Aug 27 01:49:31 zembla kernel: __ratelimit: 53 callbacks suppressed
...
Aug 27 01:49:34 zembla kernel: nf_conntrack: table full, dropping packet.
...
Lots of these "atack: segfault" lines, lots of _ratelimit, and lots of nfconntrack: table full lines. These all started at 1:49:26 this morning. It all stopped about 23 minutes later with this line:
Aug 27 02:12:53 zembla kernel: atack[13100]: segfault at 0 ip 0000000008048e33 sp 00000000ffbdf7e0 error 4 in atack[8048000+c0000]
Now as far as I can tell, what the logs seem to be saying is that a process named "atack" ran a ton of times in this 23 minute time period, segfaulted tons of times, and caused lost of network connections that resulted in network connection rate limiting.
This smells alot like someone managed to run something unauthorized on my system. I can't find a binary named 'atack' anywhere on my system so I can only suspect that it was removed already by the attacker somehow.
I searched google for "atack" and find nothing. Does anyone have any clue whatsoever what this is all about?
23 Replies
My kernel:
[root@zembla ~]# uname -a
Linux zembla.ischo.com 2.6.29-x86_64-linode6 #1 SMP Thu Apr 2 15:14:25 EDT 2009 x86_64 Intel(R) Xeon(R) CPU L5520 @ 2.27GHz GenuineIntel GNU/Linux
My services which listen for connections and could be compromised by a remote attacker:
[root@zembla ~]# netstat -anp | grep LISTEN
tcp 0 0 0.0.0.0:993 0.0.0.0:* LISTEN 1430/dovecot
tcp 0 0 0.0.0.0:143 0.0.0.0:* LISTEN 1430/dovecot
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 3452/sshd
tcp 0 0 0.0.0.0:25 0.0.0.0:* LISTEN 3545/master
tcp 0 0 :::80 :::* LISTEN 3634/httpd
These are associated with the following versions of software:
dovecot 1.2.1-1
openssh 5.2p1-1
postfix 2.6.2-1
apache 2.2.11-3
Additionally, my apache server has:
php 5.2.10-3
squirrelmail 1.4.19-1
gallery 2.3-1
@bji:
This smells alot like someone managed to run something unauthorized on my system.
Aug 27 01:46:45 scrith sshd[10582]: Did not receive identification string from 72.14.189.48
Aug 27 01:49:27 scrith sshd[10583]: Invalid user globus from 72.14.189.48
Aug 27 01:49:27 scrith sshd[10586]: Invalid user condor from 72.14.189.48
Aug 27 01:49:27 scrith sshd[10588]: Invalid user tomcat from 72.14.189.48
Aug 27 01:49:27 scrith sshd[10590]: Invalid user global from 72.14.189.48
Aug 27 01:49:28 scrith sshd[10592]: Invalid user upload from 72.14.189.48
Aug 27 01:49:28 scrith sshd[10594]: Invalid user jboss from 72.14.189.48
Aug 27 01:49:28 scrith sshd[10596]: User postmaster from 72.14.189.48 not allowed because not listed in AllowUsers
Aug 27 01:49:28 scrith sshd[10598]: Invalid user demo from 72.14.189.48
Aug 27 01:49:28 scrith sshd[10600]: Invalid user apache from 72.14.189.48
Aug 27 01:49:28 scrith sshd[10604]: Invalid user postgres from 72.14.189.48
Aug 27 01:49:28 scrith sshd[10608]: Invalid user mysql from 72.14.189.48
Aug 27 01:49:28 scrith sshd[10610]: Invalid user tester from 72.14.189.48
Aug 27 01:49:28 scrith sshd[10612]: Invalid user testing from 72.14.189.48
Aug 27 01:49:28 scrith sshd[10616]: Invalid user test from 72.14.189.48
Aug 27 01:49:29 scrith sshd[10618]: Invalid user photo from 72.14.189.48
Aug 27 01:49:29 scrith sshd[10620]: Invalid user oracle from 72.14.189.48
Aug 27 01:49:29 scrith sshd[10622]: Invalid user feedback from 72.14.189.48
Aug 27 01:49:29 scrith sshd[10624]: Invalid user sameer from 72.14.189.48
Aug 27 01:49:29 scrith sshd[10628]: Invalid user temp from 72.14.189.48
Aug 27 01:49:29 scrith sshd[10630]: Invalid user testuser from 72.14.189.48
Aug 27 01:49:29 scrith sshd[10632]: Invalid user portal from 72.14.189.48
$ dig zembla.ischo.com
; <<>> DiG 9.4.3-P3 <<>> zembla.ischo.com
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41423
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;zembla.ischo.com. IN A
;; ANSWER SECTION:
zembla.ischo.com. 86400 IN A 72.14.189.48
I'd say yes you have been compromised.
caker
Linode Staff
You want to select "Latest 2.6 Stable" (i386 or x86_64) in your profile, save, and reboot -- in addition to your other cleaning up.
-Chris
@caker:
Also, the kernel you're running has the local root exploit.
http://blog.linode.com/2009/08/17/new-k … abilities/">http://blog.linode.com/2009/08/17/new-kernels-fixed-vulnerabilities/ You want to select "Latest 2.6 Stable" (i386 or x86_64) in your profile, save, and reboot -- in addition to your other cleaning up.
-Chris
OK, I did that. Now I am at a complete loss as to how someone got in. I can see from my Linode performance graphs that in that 23 minute range, there was tons of network activity. And then it all goes dead.
Now, there are exactly two users who have a login shell on my server: myself, and root. root's password is quite strong, as is mine. So I don't think that anyone got in directly via ssh.
I'm suspecting that they must have gotten in via a service like apache (most likely), or dovecot or postfix (less likely). In any case, I think they would have been isolated to running with priveleges of the http, postfix, or dovecot user. Unless they took advantage of the local root exploit you mentioned, but if they had, wouldn't I have seen more activity than I did?
I'm just trying to figure out how bad it's likely to be.
Once years ago I got some kind of Apache virus but because it was limited to the apache user, it was relatively easy to clean out. I wonder if I am in the same situation now, or if my entire host is somehow so compromised that I can't trust root anymore …
What I really don't understand is why it stopped after 23 minutes. Why wouldn't it have just kept running indefinitely? Maybe it exhausted its ssh attacks against all other hosts on the local network (sorry everyone!)? And since it didn't have anything else to do without root access, it just quit?
@bji:
Once years ago I got some kind of Apache virus but because it was limited to the apache user, it was relatively easy to clean out. I wonder if I am in the same situation now, or if my entire host is somehow so compromised that I can't trust root anymore …
If you suspect your host is compromised at all, start over with a fresh install. It might seem like a lot of work, but unless you have a reliable way to tell what has been changed (like tripwire) then you are just asking for more trouble. You have no idea what the attacker could have modified and what type of backdoors he could have put in.
By the way, it looks like "atack" is a program to brute force attack ssh on other machines.
It appears to also go by the "brute" name:
I think what may have happened is that someone attempted a DoS or DDoS against your webserver. The conntrack logs support this. Your system's connection table filled up. If there are tons of connections that the system is attempting to track, the system only has finite resources to track connections…if there are a massive amount of connections and the attack is long-lived, your system will eventually run out of resources and crap itself (dunno if your system locked up or recovered on its own).
Typically, we see this with firewalls attempting to track massive infections such as Slammer or Blaster. While the FW itself isn't infected, the FW will eventually bog down and/or crash. Note that this can happen with any type of connection flood…there doesn't have to be a compromise or intrusion for this to happen.
I'd look at your webserver logs (or your SSH logs, since that port was also showing in your netstat output, although I doubt its an SSH attack that happened).
I conduct first-, second-, and third-level attack and compromise analysis for a living. IMO, kernel logs don't always mean what people think they mean (at least regarding compromise). You can't just run some exploit against a kernel without having a vector (avenue of attack, which is more than likely a network service). Kernel logs don't define the endgame of security, unless it is actually a kernel-level attack, which would suggest that someone is logged in locally to the machine (although this may NOT be the case with the OP). conntrack is a kernel-level function of network connectivity, which means that anyone conducting a DoS will affect connection tracking.
You're doing this admin a disservice if you think that what he described is an actual compromise.
Looking at 'atack segfault' under Google shows the following:
Almost ALL of those links suggest a denial of service and have similar content as the OP posted. If it quacks and waddles like a duck, its either a relative to the duck or IS a duck, IMO.
Here's a GOOD one (from
doing a "ping -f -l 3" on my host towards my board on linus tree as of
Friday results in lots of:
nf_conntrack: table full, dropping packet.
nf_conntrack: table full, dropping packet.
nf_conntrack: table full, dropping packet.
__ratelimit: 11 callbacks suppressed
nf_conntrack: table full, dropping packet.
nf_conntrack: table full, dropping packet.
nf_conntrack: table full, dropping packet.
nf_conntrack: table full, dropping packet.
for ucc_geth on a MPC832x.
This really looks strange to me, ideas?
If this type of activity is directed against an application, there's where the segfault comes in. Regarding the 'atack' tag, I've no idea where that is coming from but to suggest that someone injected that onto the system is to pull suppositions out of thin air. People in my field actually need data to prove that something like this occurred. That's where application logs
All of this is circumstantial, since there are only kernel logs that were provided. Still…
Secondly, there's the issue of him reinstalling (because he was told to do so) without understanding the nature of the attack. There's a chance it may happen, again, within hours/days of his reinstall if he doesn't understand the who/what/where/why of the prior attack. I've seen machine owners reinstall 3-4 times and get compromised 3-4 times afterward…all because the admins didn't understand the underlying issue.
There are several other people that have had similar messages appear in their logs and they were all compromised too. I posted a link to one earlier. Here is another that found atack running in "/var/tmp/ ……. / . /unixcod/atack":
While I agree he needs to figure out how he got compromised in order to patch and protect against it in the future, telling him that he was simply DoS'd and to not worry about it is not very good advice either.
@unixfool:
You're doing this admin a disservice if you think that what he described is an actual compromise.
Maybe you missed the strange coincidence that within one second of the OP's segfault log, MY logs (posted earlier) show his system starting an ssh attack against my linode.
Someone got at least user-level access on his system and was looking for more hosts to breach. I don't claim to do this for a living, but if he wasn't compromised then I don't know what the word means.
@Stever:
@unixfool:You're doing this admin a disservice if you think that what he described is an actual compromise.
Maybe you missed the strange coincidence that within one second of the OP's segfault log, MY logs (posted earlier) show his system starting an ssh attack against my linode.Someone got at least user-level access on his system and was looking for more hosts to breach. I don't claim to do this for a living, but if he wasn't compromised then I don't know what the word means.
I agree - unixfool, I think you must have just missed the part of my logs where I listed the lines indicating that a program named "atack" had run locally on my system (and segfaulted quite alot).
What I think happened is that this atack program forked itself tons of times to issue SSH attacks on other hosts on the network simultaneously. My Linode control panel graphs showed that in this 23 minute time period, my system went to 100% CPU usage, tons of network activity, and all of my memory got used up. I suspect that all of the seg faults are due to fork calls failing when the system was out of memory and the atack program not gracefully handling that.
There are three big questions from my perspective:
1. How did they get in? I have searched for known exploits on the handful of services that I am running, and I can't find anything. So maybe there are exploits in PHP or apache or some other software that just isn't known globally yet but is known to some hackers. Or more likely, I just didn't look hard enough when searching for known exploits.
2. Did they acquire any privileges above and beyond the user process that the atack program ran as? Unfortunately, I can't tell what user ran 'atack'. And unfortunately again, I was apparently running a kernel susceptible to the local root explot that was recently fixed in the newer kernels. And yet, I don't see any evidence of having been rooted. It is possible that some process is sitting there on my system completely hidden from me somehow, but I've done pretty exhaustive searches and I don't find anything. netstat shows nothing. process accounting (via acct, which I installed after this happened) shows nothing. All of the find commands and manual file examination shows nothing. The linode control panel shows that my box isn't suffering the CPU, memory, or network load of running any kind of active explot. I basically have no evidence that the user access was elevated to root access by the attacker. I know, the truly paranoid would say that I shouldn't trust anything and should just re-install. And maybe that is a reasonable level of paranoia. But for the time being, I'm watching and waiting to see if there is ever any other evidence.
3. Is my system still vulnerable? Since I don't know how they got in, I don't know if I've "fixed" the problem. I have installed process accounting to help me watch for any further shenanigans … any other suggestions for stuff to install to help me watch out for a recurrence of this attack?
Use the find command to find all files created since that day. If there's too much in that output, try limiting it to all executable files created since that day.
Do you use rpm, apt, or something else? With RPM, you can verify all files against the md5sum from the database to see if any binaries were modified from the packaged version. I'm sure apt has a similar feature.
Find and install chkrootkit.
Find all occurences of '.bash_history', and make sure there's nothing funny in any of those.
Many "hackers" using rootkits will patch netstat, but not lsof. Use it to see if there's any ports listening that shoudn't. If it's already installed, remove it, and reinstall it from a binary package.
Hopefully, by this time you have some files to work with. Whatever you do, don't run them. Copy them off to a different host (like a throwaway VM), and inspect them there. File ownership might point you in the direction of which daemon was exploited.
Hope some of that helps!
My machine does nightly backups and I compared a nightly backup from before the break-in to one afterwards. I found some "interesting" files in /tmp:
-rwxr-xr-x 3 88 88 464140 2007-09-20 10:04 k-rad3
-rwxr-xr-x 3 88 88 8550 2008-02-18 05:33 2.6.20
-rwxr-xr-x 3 88 88 11048 2008-02-18 05:34 2.6.23
-rwxr-xr-x 3 88 88 11523 2008-02-18 05:36 2.6.24
-rwxr-xr-x 3 88 88 10137 2008-02-18 05:37 2.6.21
-rwxr-xr-x 3 88 88 240642 2008-02-18 05:48 2.2.4
drwxr-xr-x 3 88 88 4096 2009-08-25 02:15
drwx–---- 5 88 88 4096 2009-08-25 07:00 -
Also some "hidden" files:
./ /.time:
total 636
-rw-r--r-- 1 88 88 328 2009-08-27 03:00 72.14.189.48.user
-rw-r--r-- 3 88 88 328 2009-08-25 02:12 72.14.189.48.user2
-rw-r--r-- 1 88 88 328 2009-08-27 03:00 72.14.189.48.user3
-rwxr-x--x 3 88 88 317 2006-10-29 22:15 autorun
-rwxr-x--x 3 88 88 492135 2006-10-29 22:15 bash
-rw-r--r-- 3 88 88 47 2009-08-25 02:11 cron.d
-rwxr-x--x 3 88 88 9175 2009-08-25 02:11 inst
-rw-r--r-- 3 88 88 171 2009-08-25 02:13 LinkEvents
-rw-r--r-- 3 88 88 14 2009-08-25 02:11 mech.dir
-rwxr-x--x 3 88 88 22882 2006-10-29 22:15 m.help
-rw-r--r-- 1 88 88 1043 2009-08-27 03:00 m.lev
-rw------- 3 88 88 6 2009-08-25 02:11 m.pid
-rw-r--r-- 1 88 88 2127 2009-08-27 03:00 m.ses
-rw-r--r-- 3 88 88 2783 2009-08-25 02:11 m.set
-rw-r--r-- 1 88 88 7699 2009-08-27 03:30 ning.seen
-rw-r--r-- 1 88 88 11738 2009-08-27 03:30 niu.seen
drwxr-x--x 2 88 88 4096 2007-05-23 20:00 r
-rwxr-x--x 3 88 88 29 2006-10-29 22:15 run
-rwxr-x--x 3 88 88 752 2008-06-01 12:06 start
-rwxr--r-- 3 88 88 169 2009-08-25 02:11 update
-rw-r--r-- 3 88 88 13 2009-08-25 02:11 vhosts
-rwxr-x--x 3 88 88 28489 2006-10-29 22:15 xh
-rw-r--r-- 3 88 88 0 2009-08-25 02:12 zhou.seen
On the host that has the backups, user 88 and group 88 does not exist. But on my Linode, from which the backup was made, 88 is:
postgres
So it looks like they exploited some way to get into postgres.
One thing I discovered today is that I had some users that didn't allow interactive ssh (I set the shell for all users to /bin/false in /etc/passwd, except for me and root) but I didn't realize that they could still be used to create SSH tunnels. So it's possible that the attacker was able to use one of these accounts to SSH tunnel into my local postgres which only listens on 127.0.0.1 and has an easy password.
I have now added an AllowUsers line to /etc/ssh/sshd_config to disallow SSH access from anyone except me and root.
Another interesting fact:
It looks like the earliest file created was on
2009-08-25 07:00. So I guess that this exploit had been around for a few days before it ended up being "used" to run an SSH attack.
Also - after a reboot, these files were cleaned out and no longer exist on my Linode.
I believe that the attacker did not get root access; if they did, they probably would have removed all evidence of their attack from /tmp.
@jsr:
Do you have the postgres running and its port (5432) open to the world? If so, you should setup some firewall rules to block it off.
It is running but it is only listening on 127.0.0.1, so it cannot be accessed from outside.
It does have a weak password, but I assumed that because it was only listening locally, it could not be exploited from afar.
However, if someone was able to set up an SSH tunnel from the local postgres port to some remote host, because they were able to guess the password for one of my email users who shouldn't have been allowed to SSH at all (I've fixed this), then they could get to it.
I will change my postgres password to be more secure in any case.
@jsr:
All signs point to him being compromised though. The atack program is a ssh cracking tool and part of the unixcod package. You can find it at
http://www.bart-design.co.uk/unixcod/
> There are several other people that have had similar messages appear in their logs and they were all compromised too. I posted a link to one earlier. Here is another that found atack running in "/var/tmp/ ……. / . /unixcod/atack":/
http://ibot.rikers.org/%23debian/20080502.html.gz
I didn't know that, even after googling for 'atack'. Thanks!
> While I agree he needs to figure out how he got compromised in order to patch and protect against it in the future, telling him that he was simply DoS'd and to not worry about it is not very good advice either.
In no way did I say that it wasn't anything to worry about. I specificially stated that with the information that was provided, the issue was difficult to analyze. Up to the point where I chimed in, there was nothing in the thread that suggested what you did up above. I live in a world where what's in your face is what you act upon…no data to act on hinders the process of analysis. Based on the facts provided at that time, it appeared to be a DoS to me, especially looking at the conntrack log entry.
As I said before, just posting an obscure kernel log entry by itself doesn't mean that a machine has been cracked.
Thanks for updating this thread with further info, though…it helps a ton in understanding what's going on.
@Stever:
@unixfool:You're doing this admin a disservice if you think that what he described is an actual compromise.
Maybe you missed the strange coincidence that within one second of the OP's segfault log, MY logs (posted earlier) show his system starting an ssh attack against my linode.Someone got at least user-level access on his system and was looking for more hosts to breach. I don't claim to do this for a living, but if he wasn't compromised then I don't know what the word means.
You didn't exactly say what those logs were, either (although I'd wondered WTF they were and what they had to do with the issue). I'd assumed that you were seeing something similar to what he was…I didn't factor in that he was your network neighbor, but then again, why should I if you didn't specifically mention it?
Thanks, though.
@bji:
OK, so I was able to find some more out.
My machine does nightly backups and I compared a nightly backup from before the break-in to one afterwards. I found some "interesting" files in /tmp:
-rwxr-xr-x 3 88 88 464140 2007-09-20 10:04 k-rad3
-rwxr-xr-x 3 88 88 8550 2008-02-18 05:33 2.6.20
-rwxr-xr-x 3 88 88 11048 2008-02-18 05:34 2.6.23
-rwxr-xr-x 3 88 88 11523 2008-02-18 05:36 2.6.24
-rwxr-xr-x 3 88 88 10137 2008-02-18 05:37 2.6.21
-rwxr-xr-x 3 88 88 240642 2008-02-18 05:48 2.2.4
drwxr-xr-x 3 88 88 4096 2009-08-25 02:15
drwx–---- 5 88 88 4096 2009-08-25 07:00 -
Also some "hidden" files:
./ /.time:
total 636
-rw-r--r-- 1 88 88 328 2009-08-27 03:00 72.14.189.48.user
-rw-r--r-- 3 88 88 328 2009-08-25 02:12 72.14.189.48.user2
-rw-r--r-- 1 88 88 328 2009-08-27 03:00 72.14.189.48.user3
-rwxr-x--x 3 88 88 317 2006-10-29 22:15 autorun
-rwxr-x--x 3 88 88 492135 2006-10-29 22:15 bash
-rw-r--r-- 3 88 88 47 2009-08-25 02:11 cron.d
-rwxr-x--x 3 88 88 9175 2009-08-25 02:11 inst
-rw-r--r-- 3 88 88 171 2009-08-25 02:13 LinkEvents
-rw-r--r-- 3 88 88 14 2009-08-25 02:11 mech.dir
-rwxr-x--x 3 88 88 22882 2006-10-29 22:15 m.help
-rw-r--r-- 1 88 88 1043 2009-08-27 03:00 m.lev
-rw------- 3 88 88 6 2009-08-25 02:11 m.pid
-rw-r--r-- 1 88 88 2127 2009-08-27 03:00 m.ses
-rw-r--r-- 3 88 88 2783 2009-08-25 02:11 m.set
-rw-r--r-- 1 88 88 7699 2009-08-27 03:30 ning.seen
-rw-r--r-- 1 88 88 11738 2009-08-27 03:30 niu.seen
drwxr-x--x 2 88 88 4096 2007-05-23 20:00 r
-rwxr-x--x 3 88 88 29 2006-10-29 22:15 run
-rwxr-x--x 3 88 88 752 2008-06-01 12:06 start
-rwxr--r-- 3 88 88 169 2009-08-25 02:11 update
-rw-r--r-- 3 88 88 13 2009-08-25 02:11 vhosts
-rwxr-x--x 3 88 88 28489 2006-10-29 22:15 xh
-rw-r--r-- 3 88 88 0 2009-08-25 02:12 zhou.seen
On the host that has the backups, user 88 and group 88 does not exist. But on my Linode, from which the backup was made, 88 is:
postgres
So it looks like they exploited some way to get into postgres.
One thing I discovered today is that I had some users that didn't allow interactive ssh (I set the shell for all users to /bin/false in /etc/passwd, except for me and root) but I didn't realize that they could still be used to create SSH tunnels. So it's possible that the attacker was able to use one of these accounts to SSH tunnel into my local postgres which only listens on 127.0.0.1 and has an easy password.
I have now added an AllowUsers line to /etc/ssh/sshd_config to disallow SSH access from anyone except me and root.
Another interesting fact:
It looks like the earliest file created was on
2009-08-25 07:00. So I guess that this exploit had been around for a few days before it ended up being "used" to run an SSH attack.
Also - after a reboot, these files were cleaned out and no longer exist on my Linode.
I believe that the attacker did not get root access; if they did, they probably would have removed all evidence of their attack from /tmp.
This is AWESOME information!
Looks like they loaded and installed a kernel-level local exploit (
Man, your machine was jacked!
@unixfool:
@Stever:
@unixfool:You're doing this admin a disservice if you think that what he described is an actual compromise.
Maybe you missed the strange coincidence that within one second of the OP's segfault log, MY logs (posted earlier) show his system starting an ssh attack against my linode.Someone got at least user-level access on his system and was looking for more hosts to breach. I don't claim to do this for a living, but if he wasn't compromised then I don't know what the word means.
You didn't exactly say what those logs were, either (although I'd wondered WTF they were and what they had to do with the issue). I'd assumed that you were seeing something similar to what he was…I didn't factor in that he was your network neighbor, but then again, why should I if you didn't specifically mention it?
Thanks, though.
I don't relish carrying this particular thread of discussion any further but … he included "dig" output next to his logs to show that the host that was issuing the SSH attacks was my host (zembla.ischo.com). So all that you needed to know to make the correlation was in his posts.
That being said, I paid alot closer attention to his post than you probably did because I was the one who was compromised and so I had a more than intense interest in the thread. I can see how, if you were only reading casually, you might have missed it.
@unixfool:
This is AWESOME information!
Looks like they loaded and installed a kernel-level local exploit (
http://www.xfocus.net/tools/200512/k-rad3.c ). It looks like they loaded up several exploits that targets different kernel versions. I also see what looks like an IRC bot (mech.dir hints to this andhttp://www.webhostingtalk.com/showthread.php?p=5323420 shows an example). The *.seen files look to be bot log files of bot users.Man, your machine was jacked!
Yes, I examined all of the files that were present and you are right, the k-rad and kernel version files were executables which attempt to gain root access via various outdated Linux kernel bugs. None of them would have succeeded against my kernel, though.
Apparently I was running a kernel that is susceptible to the newest Linux kernel exploit, but they didn't run that exploit. I guess their scripts were a little too old.
I have compared SHA-1 hashes of all files that existed on my system before and after the compromise. There were no modified files aside from those that I listed above from /tmp. That, combined with the fact that there is no other evidence whatsoever of continued abuse of my system, leads me to feel fairly confident that the attack has been neutralized simply by rebooting into the latest Linux kernel, removing all files from /tmp (which the reboot accomplished), and putting more stringent controls for sshd (via AllowUsers).