help! massive outgoing bandwidth spikes. possible ddos issue
so I have a linode running several small wordpress sites and one small joomla site. Several days ago, I saw there were several huge spikes in the outgoing bandwidth traffic graph and the sites were timing out until I rebooted the server. I shut down the postfix server cause a buddy of mine who had a quick moment to look at the system said one of the sites was sending out a ton of mail. So we killed the mail and started the task of setting a new server on a seperate linode. In the meantime, a few times now, there was HUGE bandwidth spikes (like enough that I got a TOS violation ticket because of them) but my buddy is gone for a few weeks and I have no idea how to figure out where they are coming from.
He is the kicker, I moved half of my wordpress sites over to the new linode, patched them and checked them all for vulnerabilities. The original site that started the trouble was a victim of the timthumb.php exploit so I made sure to patch all the other sites. I also deleted all the crazy extra .htaccess files that ended up in each of the vhost directories. Everything was going fine till about 8pm tonight while I was out when suddenly, the new linode I setup had another massive 2 hour outgoing bandwidth spike.
So, my question is, based on my very limited knowledge of linux and the such, how I can track down which of the sites is actually causing the spikes, if it one of the sites at all. I poured through all the logs and all I could see out of the ordinary was some russian bot scanning of the sites looking for setup files for phpmyadmin and the such.
So, please and thanks for any advice on how to help narrow down the problem. I get the feeling I just ended up copying over the issue from one linode to the next and the patching/upgrading didn't fix it.
Cheers!
10 Replies
Does your new setup also have postfix running on it? Try shutting it down again and see what happens to your outgoing bandwidth. There could be a malicious script on your server that does nothing but send thousands and thousands of spam.
The timthumb exploit allowed arbitrary code to be uploaded, so your site may still contain the spamming script even if you patched timthumb itself. If the attacker was able to place .htaccess files in arbitrary locations, any file in your site, even WordPress itself, could have been contaminated. Unfortunately, it is very difficult to fix this without reinstalling WordPress and all plugins from official sources. You can't just copy the files over, because you'd also copy the contaminated file. This can be a huge pain in the ass if you have lots of attachments (such as images), which you probably do since you've been using an image processing plugin.
is there anyway to help isolate which possible wordpress site is effected so i can patch it up first before getting to the other sites?
@Tamerax:
is there anyway to help isolate which possible wordpress site is effected so i can patch it up first before getting to the other sites?
I assume there's nothing suspicious in the web server logs? Well, here's a really simple method if you're okay with some downtime. Disable your virtual hosts one by one, for about 15 minutes each, or long enough for any difference to show up in your dashboard. If the bandwidth suddenly drops, you'll know that you just disabled the affected site.
It will be more difficult to tell, however, if more than one site is affected. You said that you found extra .htaccess files in each vhost. So the attacker definitely had access to every site at some point. (You're not using anything fancy like suexec to separate the privileges, are you? That would have contained the damage to one site.)
What webserver are you on? Are you sure it isn't just a users site going nuts?
Finding ing out what is wrong and fixing it will probably be a good education for you. I know very little, but learnt it all on these or other forums when stuff wasn't working
@tentimes:
Harden your webserver. Read the expereiences of a guy on this forum (this week) who was having DDoS attacks. If not using apache then have a look at Nginx ge-location module and ban all traffic from russia.
I'm not sure if that will help. This doesn't seem to be an incoming DDoS attack like the other thread. The Russian bots aren't DDoSing the OP's server. Rather, it seems that the OP's server is DDoSing and/or spaming someone else, which is a much more tricky situation.
By the way, OP: Could it be something other than a malicious PHP script on your site? If the answer to any of the following questions is "yes", the attacker might have gained SSH access to your server. (a) Does your server allow root login via SSH? (b) Do you use a short or easily guessable password to log in? © Do you use FTP to manage your sites?
@hybinet:
@Tamerax:is there anyway to help isolate which possible wordpress site is effected so i can patch it up first before getting to the other sites?
I assume there's nothing suspicious in the web server logs? Well, here's a really simple method if you're okay with some downtime. Disable your virtual hosts one by one, for about 15 minutes each, or long enough for any difference to show up in your dashboard. If the bandwidth suddenly drops, you'll know that you just disabled the affected site.
It will be more difficult to tell, however, if more than one site is affected. You said that you found extra .htaccess files in each vhost. So the attacker definitely had access to every site at some point. (You're not using anything fancy like suexec to separate the privileges, are you? That would have contained the damage to one site.)
I found one php file that was definitely not supposed to be there and i got rid of it on friday. I'm not using suexec and yeah, there was an extra htaccess file in each directory redirecting to some russian sites (which is how I found out it was the timthumb exploit).
I would try diabling each vhost but the spikes never happen at any regular interval. Like today, it only happened once at around 6pm. So I can't really turn everything off and wait cause that could mean days of downtime for those sites.
@tentimes:
Harden your webserver. Read the expereiences of a guy on this forum (this week) who was having DDoS attacks. If not using apache then have a look at Nginx ge-location module and ban all traffic from russia.
What webserver are you on? Are you sure it isn't just a users site going nuts?
Finding ing out what is wrong and fixing it will probably be a good education for you. I know very little, but learnt it all on these or other forums when stuff wasn't working
;)
I'm using apache with nginx running as a reverse proxy. I don't mind learning how to do things on my own but I need some advice more than anything
At this point though, you're probably best off recovering what you can, wiping clean, and rebuilding.
@hybinet:
By the way, OP: Could it be something other than a malicious PHP script on your site? If the answer to any of the following questions is "yes", the attacker might have gained SSH access to your server. (a) Does your server allow root login via SSH? (b) Do you use a short or easily guessable password to log in? © Do you use FTP to manage your sites?
The first server that got infected didn't allow root login and used a 7 character alpha numeric password and I used SFTP connecting to a non-standard port for all my uploading.
The second server does use root (didn't have a chance to chance that setting yet), and the pass is 15 character alphanumeric so I think that is pretty good. Also uses STFP on port 22.
@Tamerax:
@hybinet:By the way, OP: Could it be something other than a malicious PHP script on your site? If the answer to any of the following questions is "yes", the attacker might have gained SSH access to your server. (a) Does your server allow root login via SSH? (b) Do you use a short or easily guessable password to log in? © Do you use FTP to manage your sites?
The first server that got infected didn't allow root login and used a 7 character alpha numeric password and I used SFTP connecting to a non-standard port for all my uploading.
The second server does use root (didn't have a chance to chance that setting yet), and the pass is 15 character alphanumeric so I think that is pretty good. Also uses STFP on port 22.
You might want to try using SSH keys.