I found some old MX records for Google Gmail in my DNS Manager for one of my websites that I was trying to setup and gave up on several weeks ago. I took them out and I haven't had any speed issues since. Crossing fingers that was the only issue.

update nope that didn't help - having issue again and just noticed another "www-data" process taking up over 10% of the CPU.

What do your access logs say?

@Azathoth:

What do your access logs say?

Hi Azathoth, thanks for the reply.

I'm a Linode newbie I admit. So when you say access logs, do you mean for each website (access.log)? They are huge, but I really don't see anything out of the ordinary when looking at them.

@phowell32:

@Azathoth:

What do your access logs say?

Hi Azathoth, thanks for the reply.

I'm a Linode newbie I admit. So when you say access logs, do you mean for each website (access.log)? They are huge, but I really don't see anything out of the ordinary when looking at them.

Another thing you might try is iftop (apt-get install iftop) and then when the bandwidth goes up if you run it with sudo it should show you the IPs that are causing it. Might be able to narrow it down with the access.log files.

@phowell32:

They are huge, but I really don't see anything out of the ordinary when looking at them.

Nothing out of the ordinary during the peak bandwidth times? There must be something out of the ordinary if your site suddenly jumps from virtually nothing to 50 Mbps, and according to your process list sample there are 4 very active Apache processes.

Edit: What I'm trying to say is, could be some rogue script or wp module doing something nasty. This looks a lot like a situation I had not too long ago, a site had a "send to friend" feature which allowed basically any kind of email be sent anywhere. When the spammers got hold of that, they hammered the server with thousands of POSTs (which was out of the ordinary in the logs). 40k emails sent in under an hour.

@Nuvini:

@phowell32:

@Azathoth:

What do your access logs say?

Hi Azathoth, thanks for the reply.

I'm a Linode newbie I admit. So when you say access logs, do you mean for each website (access.log)? They are huge, but I really don't see anything out of the ordinary when looking at them.

Another thing you might try is iftop (apt-get install iftop) and then when the bandwidth goes up if you run it with sudo it should show you the IPs that are causing it. Might be able to narrow it down with the access.log files.

Ok yes, iftop shows a ton of bandwidth from a few sources, it just doesn't stop. I'm going through the access logs for each site to see if I can find those IP's in the log. Will update once I find something.

@Azathoth:

@phowell32:

They are huge, but I really don't see anything out of the ordinary when looking at them.

Nothing out of the ordinary during the peak bandwidth times? There must be something out of the ordinary if your site suddenly jumps from virtually nothing to 50 Mbps, and according to your process list sample there are 4 very active Apache processes.

Edit: What I'm trying to say is, could be some rogue script or wp module doing something nasty. This looks a lot like a situation I had not too long ago, a site had a "send to friend" feature which allowed basically any kind of email be sent anywhere. When the spammers got hold of that, they hammered the server with thousands of POSTs (which was out of the ordinary in the logs). 40k emails sent in under an hour.

Here are my firewall rules which should be blocking email:

> *filter

Allow all loopback (lo0) traffic and drop all traffic to 127/8 that doesn't use lo0

-A INPUT -i lo -j ACCEPT

-A INPUT ! -i lo -d 127.0.0.0/8 -j REJECT

Accept all established inbound connections

-A INPUT -m state –state ESTABLISHED,RELATED -j ACCEPT

Allow all outbound traffic - you can modify this to only allow certain traffic

-A OUTPUT -j ACCEPT

Allow HTTP and HTTPS connections from anywhere (the normal ports for websites and SSL).

-A INPUT -p tcp --dport 80 -j ACCEPT

-A INPUT -p tcp --dport 443 -j ACCEPT

Allow SSH connections

#

The -dport number should be the same port number you set in sshd_config

#

-A INPUT -p tcp -m state --state NEW --dport 22 -j ACCEPT

Allow ping

-A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT

Log iptables denied calls

-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7

Reject all other inbound - default deny unless explicitly allowed policy

-A INPUT -j REJECT

-A FORWARD -j REJECT

COMMIT

You could also grep out only the traffic during those peak load times from your access logs into a separate file, then run webalizer on that file, just be careful where you point its output and/or incremental (you don't need it for this), so you don't overwrite your regular stats.

Here's an example that extracts 4pm traffic of May 25th, and produces stat graphs in /var/www/incident_stats:

grep "25/May/2012:16" access.log > incident.log
webalizer -i -b -o /var/www/incident_stats incident.log

@phowell32:

Here are my firewall rules which should be blocking email:

Nothing about email in those rules, and I meant outgoing not incoming.

I know this may sound weird but while watching "iftop" I notice I have much less traffic when I'm not connected via Putty as Root. I connected via LISH Ajax Console and there is much less traffic but when I connect via Putty with root I see a ton more and also the sites start to slow down.

I removed a bunch of old wp plugins on my sites and the traffic seems to have died down once again (doesn't mean it won't come roaring back though). I'll just watch it tonight and see if anything happens. I also turned KeepAlive back ON because I read that it saves your CPU at the cost of memory. Since CPU was getting destroyed I thought I would retry it.

Thanks for everyone's help!

I wanted to give an update to this issue and I think I have found what was causing the problem.

While reviewing the accesslogs at a certain time when CPU and bandwidth went nuts I noticed one of my sites was getting a lot of access to plugins and every post over and over and over again. Also, there were a few old videos on the site that I completely forgot about and those were also being accessed. I'm assuming it was a bot because it all happens within a few seconds. I deleted the videos and moved the site off my Linode to another host.

From that point going forward I no longer had anymore issues.

Lesson learned, if you have issues with bandwidth, review your access logs!

Thanks to everyone for your help!