Very low traffic web server but Apache is very slow.

5.44Mb/s is a ton of traffic. Take a look at access.log (or run it through some stats package) and see what the traffic is.

ycs2000,

It sounds like you may be getting unwanted traffic. Check your logs to see if you're getting http traffic. If not, perhaps you're getting unwanted DNS (port 53) or NTP (port 123) traffic. Both of these services can get amplification attacks.

MSJ

Try running nethogs. iptraf and and iotop are useful too, for iotop you might want to supply -n to suppress DNS lookups to reduce output clutter.

Re: access.log file, there seems to be traffic from Google, Bing, etc crawlers and some IP addresses (so I'm definitely getting more traffic than I thought) but nothing seemed like an attack.

I'll check nethogs, iptraf and iotop and report back.

I just ran nethogs, iptraf, and iotop. I'm not that familiar with these tools but nothing appears to be going on that is out of the ordinary. Running an iotop I do see something strange:

2993 be/4 tomcat7 0.00 B/s 10.82 K/s 0.00 % 0.00 % java -Djava.util.logging.config.file=/var/lib/tomcat7/conf/logg~/tomcat7-tomcat7-tmp org.apache.catalina.startup.Bootstrap start

tomcat7 shouldn't be running so I have no idea what's going on here.

@ycs2000:

I just ran nethogs, iptraf, and iotop. I'm not that familiar with these tools but nothing appears to be going on that is out of the ordinary. Running an iotop I do see something strange:

2993 be/4 tomcat7 0.00 B/s 10.82 K/s 0.00 % 0.00 % java -Djava.util.logging.config.file=/var/lib/tomcat7/conf/logg~/tomcat7-tomcat7-tmp org.apache.catalina.startup.Bootstrap start

tomcat7 shouldn't be running so I have no idea what's going on here.

If you run tomcat from the repo then it may set up a service, run "sudo initctl list | grep tomcat" and see if it's listed.

On Ubuntu use update-rc.d to change its run state if you don't want it to run when the system starts up.

Tomcat is currently not running for certain but I'm still experiencing slowness. Any other ideas? Now I'm even considering Worker but I'm not sure if I'm a good candidate for it. I still have no idea what's causing the current performance issues so it's hard to tell if Worker would do any good.

Can't connect (tried earlier today same thing).

If you're getting hammered, and can figure out what's causing all that traffic, you've probably been compromised.

Might be time to start fresh, ONLY install/config the services you need for the one site, get that sorted out, then (and only then) add more services.

@vonskippy:

Might be time to start fresh, ONLY install/config the services you need for the one site, get that sorted out, then (and only then) add more services.

I second this. If that machine is serving 5Mbs of something and it's not your traffic you have to reinstall the machine, there is no other way to be sure.

For all you know it could be serving kiddie porn.

Unfortunately I think you guys might be right.

The site is inaccessible this morning because I shutdown apache last night to see if that helped with my high/unusual incoming/outgoing. Oddly enough, it did. So somehow Apache is causing super weird and high incoming/outgoing.

In light of the fact that incoming/outgoing traffic went way down after shutting down apache do we still think that wiping my linode is the best option? Are there no other apache config options I can explore?

Do you know WHAT is causing that much traffic?

All you now know is that it's using Apache. So Apache, or one of it's processes is probably compromised?

Personally, my rule of thumb is Four Hours. If I can't figure out the problem in four hours, then it's time to start over. YMMV

At some point, you are in a losing game. Between downtime and your time futzing around trying to fix it, you've exceeded the time it would take to start from scratch.

That's why build logs (i.e. how did you set the server up - exactly) and known good data backups are key.

But remember, all of this is pretty much "blind advice" - you're the man onsite, you're the sysadmin, and you're the only one that can decide what's best for your system.

Perhaps you're not compromised, perhaps your Public IP is poisoned? But without knowing where the traffic is coming from, or what the traffic is, all of this is just a guess.

@vonskippy:

Do you know WHAT is causing that much traffic?

All you now know is that it's using Apache. So Apache, or one of it's processes is probably compromised?

Personally, my rule of thumb is Four Hours. If I can't figure out the problem in four hours, then it's time to start over. YMMV

At some point, you are in a losing game. Between downtime and your time futzing around trying to fix it, you've exceeded the time it would take to start from scratch.

That's why build logs (i.e. how did you set the server up - exactly) and known good data backups are key.

But remember, all of this is pretty much "blind advice" - you're the man onsite, you're the sysadmin, and you're the only one that can decide what's best for your system.

Perhaps you're not compromised, perhaps your Public IP is poisoned? But without knowing where the traffic is coming from, or what the traffic is, all of this is just a guess.

You bring up a good point. At this point it doesn't appear I'm going to be able to find a solution, and my buddies do need their site up, so I need to just wipe the linode and move on. Unfortunately, my sysadmin skills are severely lacking; I'm a tenderfoot at best. I'm afraid to move forward until I find out what is causing the issues, whether it be a compromise or misconfiguration, because what if this happens in the future. But the good thing about this being a favor and being a tenderfoot is I'm not expected to be an expert so if it happens again I can just re-wipe and move on with my life.

Thanks to everyone who has assisted with this. I appreciate everything.

When I'm off work I'm going to install tcptrack, re-start apache and try to see if I can find what is causing the high traffic. This will be my last task before wiping.