Locating the source of a malicious script/program

It may not be a hard-coded script, it might be a proxy server has been set up on your linode and someone is directing the attack from another location through your linode.

I wonder if someone is accessing your linode at the exact same times as the remote logs you posted.

When I typed that user-agent from the logs into google, one of the results came up with something about Net::Proxy. Not sure if it's related, but it is one place to look if you're not finding a hard-coded script anywhere on your box.

Have you tried installing http://www.chkrootkit.org/?

No luck with Net::Proxy or User agent search…

This stands out from chkrootkit!

Checking `bindshell'… INFECTED (PORTS: 465)

The bind shell one is possibly a false positive see http://www.chkrootkit.org/faq/#7

What does netstat -lpntu show?

It shows this :) (The server is running WHM/cPanel)

Thanks for all your help - much appreciated!

root@server1 [~]# netstat -lpntu
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
tcp        0      0 0.0.0.0:993                 0.0.0.0:*                   LISTEN      2996/dovecot        
tcp        0      0 0.0.0.0:2082                0.0.0.0:*                   LISTEN      22406/cpsrvd - wait 
tcp        0      0 0.0.0.0:2083                0.0.0.0:*                   LISTEN      22406/cpsrvd - wait 
tcp        0      0 0.0.0.0:995                 0.0.0.0:*                   LISTEN      2996/dovecot        
tcp        0      0 0.0.0.0:2086                0.0.0.0:*                   LISTEN      22406/cpsrvd - wait 
tcp        0      0 0.0.0.0:2087                0.0.0.0:*                   LISTEN      22406/cpsrvd - wait 
tcp        0      0 0.0.0.0:3306                0.0.0.0:*                   LISTEN      3388/mysqld         
tcp        0      0 0.0.0.0:110                 0.0.0.0:*                   LISTEN      2996/dovecot        
tcp        0      0 0.0.0.0:2095                0.0.0.0:*                   LISTEN      22406/cpsrvd - wait 
tcp        0      0 127.0.0.1:783               0.0.0.0:*                   LISTEN      15006/spamd child   
tcp        0      0 0.0.0.0:143                 0.0.0.0:*                   LISTEN      2996/dovecot        
tcp        0      0 0.0.0.0:2096                0.0.0.0:*                   LISTEN      22406/cpsrvd - wait 
tcp        0      0 0.0.0.0:80                  0.0.0.0:*                   LISTEN      534/httpd           
tcp        0      0 0.0.0.0:465                 0.0.0.0:*                   LISTEN      15574/exim          
tcp        0      0 0.0.0.0:21                  0.0.0.0:*                   LISTEN      15535/pure-ftpd (SE 
tcp        0      0 178.79.154.28:53            0.0.0.0:*                   LISTEN      2795/named          
tcp        0      0 178.79.145.220:53           0.0.0.0:*                   LISTEN      2795/named          
tcp        0      0 127.0.0.1:53                0.0.0.0:*                   LISTEN      2795/named          
tcp        0      0 0.0.0.0:25                  0.0.0.0:*                   LISTEN      15574/exim          
tcp        0      0 127.0.0.1:953               0.0.0.0:*                   LISTEN      2795/named          
tcp        0      0 0.0.0.0:443                 0.0.0.0:*                   LISTEN      534/httpd           
tcp        0      0 0.0.0.0:2077                0.0.0.0:*                   LISTEN      22400/cpdavd - acce 
tcp        0      0 0.0.0.0:2078                0.0.0.0:*                   LISTEN      22400/cpdavd - acce 
tcp        0      0 :::465                      :::*                        LISTEN      15574/exim          
tcp        0      0 :::21                       :::*                        LISTEN      15535/pure-ftpd (SE 
tcp        0      0 :::22                       :::*                        LISTEN      2815/sshd           
tcp        0      0 :::25                       :::*                        LISTEN      15574/exim          
udp        0      0 178.79.154.28:53            0.0.0.0:*                               2795/named          
udp        0      0 178.79.145.220:53           0.0.0.0:*                               2795/named          
udp        0      0 127.0.0.1:53                0.0.0.0:*                               2795/named          
udp        0      0 0.0.0.0:68                  0.0.0.0:*                               2662/dhclient       
udp        0      0 178.79.154.28:123           0.0.0.0:*                               2829/ntpd           
udp        0      0 178.79.145.220:123          0.0.0.0:*                               2829/ntpd           
udp        0      0 127.0.0.1:123               0.0.0.0:*                               2829/ntpd           
udp        0      0 0.0.0.0:123                 0.0.0.0:*                               2829/ntpd           
udp        0      0 ::1:123                     :::*                                    2829/ntpd           
udp        0      0 :::123                      :::*                                    2829/ntpd

If it's running whm then yes it is a false positive.

However there are a few "problems" you'll want to address.

1) SSH has password authentication enabled

2) MySQL is listening on the public interface

3) You're using apache 2.0 you should really be using 2.2 by now.

4) You have front page extensions enabled these are no longer supported by microsoft.

Have a check in /var/log/auth.log for unauthorized access attempts via ssh.

Great, I've fixed all those problems (apache 2.2 still compiling)

/var/log/auth.log doesn't exist - so can't see what attempts were made :s

I have however found that a bunch of perl scripts were eating my CPU and file IO owned by a website which I have found to be hacked into and malicious scripts had been uploaded - thinking this could have been the proxy to which the hackers were using my server!

Thanks again obs, eld101 and haus :)

Wait… you have to compile Apache 2.2?

What distro (and version) are you running?

Nothing serious, WHM has a module called easyapache which does it all for me :) Compiles against the apache modules rather than having them included at runtime.

CENTOS 5.5 ;)

http://docs.cpanel.net/twiki/bin/view/EasyApache3/

If you've a hacked site then yes that's probably the cause of the problem, suspend the site and do further investigation of the box, cpanel has a tendency of changing the system quite heavily, there will be (or should be) some form of log in /var/log which contains ssh attempts.

You should also inspect the malicious perl scripts to see if they managed to edit any files, do you know what user the scripts were running under?

Yes I do know which user was running the perl scripts, that's what helped me find out which site was running the processes :)

Look for the cpanel SSH log on CentOS in /var/log/secure (that's where mine is on another server). If someone got bruteforced you should be using something like CSF/LFD or fail2ban. If they got the password through another means…not much you can do if you're going to allow other users on your box.

I'd love to know more about those scripts they uploaded.

You can also set up an "ftpcheck" script to monitor FTP uploads and look for anything suspicious, though if you have customers uploading files that might go against privacy concerns. The problem now is that once your box is compromised, you never quite know if you've gotten it all out or not. Hopefully the damage was limited to that one user account and a handful of perl scripts.

I assume that account doesn't have sudo access if it does you'll want to check the logs for sudo uses.

You could also run find / -user username -exec ls -lh {} \;

to list all files owned by the user and their details i.e. last modified etc, so you can see if any new files have been created.

@obs:

I assume that account doesn't have sudo access if it does you'll want to check the logs for sudo uses.

You could also run find / -user username -exec ls -lh {} \;

to list all files owned by the user and their details i.e. last modified etc, so you can see if any new files have been created.

Doesn't matter, the logs can be modified. As soon as root is compromised, or a sudoers user is compromised, the box is too compromised to save, and must be rebuilt.

No users other than root are sudo'ers and the user that the script ran on only has write permissions on it's own home directory which compromises of the website and email files :)