Disk IO Spikes, Crashes My Linode Server

Sounds like you might be hitting swap (which will destroy your performance).

Are you running munin?

That much swap is pointless. If you're thrashing even 100M of swap the box will be unusable anyhow, so reduce your swap image back to 256M or something sane. I'd rather have a box go OOM and reboot itself than thrash and be unusable for eternity. Check out this post:

http://www.linode.com/wiki/index.php/RebootingonOOM

Check your Apache setup: Identify which MPM you're running with: "apache2 -V | grep MPM". Find that section in apache2.conf. What are the values? If you see numbers three digits long, you're in trouble.

Linode.com's looks like this: <ifmodule mpm_prefork_module="">StartServers 5 MinSpareServers 5 MaxSpareServers 10 MaxClients 75 MaxRequestsPerChild 1000</ifmodule>
I'd recommend setting MaxClients lower to start out with (say 20).

Hope that helps,

-Chris

@JshWright:

Are you running munin?
No I'm not but seems like a good idea to add a monitoring application like that - thanks for the heads up and I'm looking into it.

@caker:

That much swap is pointless. If you're thrashing even 100M of swap the box will be unusable anyhow, so reduce your swap image back to 256M or something sane.
Ok, will do, I had originally decided on 2GB as both Ubuntu and CentOS documentation suggest 2x your RAM. I did find this though in the Ubuntu docs which is more like what you recommend.

@caker:

Check your Apache setup: Identify which MPM you're running with: "apache2 -V | grep MPM".

Server MPM:     Prefork
 -D APACHE_MPM_DIR="server/mpm/prefork"

@caker:

Find that section in apache2.conf. What are the values? If you see numbers three digits long, you're in trouble.
This is what I have:

 <ifmodule mpm_prefork_module="">StartServers          5
    MinSpareServers       5
    MaxSpareServers      10
    MaxClients          150
    MaxRequestsPerChild   0</ifmodule> 

MaxClients to 20 is a great start…

@MichaelE:

@caker:

That much swap is pointless. If you're thrashing even 100M of swap the box will be unusable anyhow, so reduce your swap image back to 256M or something sane.
Ok, will do, I had originally decided on 2GB as both Ubuntu and CentOS documentation suggest 2x your RAM. I did find this though in the Ubuntu docs which is more like what you recommend.
The general 2x recommendation isn't necessarily unreasonable on a dedicated machine, since while disk is significantly slower than memory, the swapping is still only on behalf of the single environment running on the machine. It does depend on whether you're truly going to be overcomitted (and then thrashing, in which case even a little can be disastrous), or just have a larger allocation than working set. It's still probably reasonable to also have some absolute maximums independent of machine size (e.g., 2GB is probably extreme for a 1GB machine in any case), unless you know a lot about how your working set operates and you're ok with a hit when switching among activities.

Importantly, in a VPS scenario, disk I/O is, in general, more contentious and performance impacting. Unlike memory (which with Xen based systems is guaranteed availability for a given VPS), disk is shared among all other machines on that host. So the already performance compromising disk I/O performance is slowed even further by the VPS contention for host resources.

The bottom line is that in a VPS environment, you want to do whatever you can to keep everything in memory, and there's little benefit to permitting significant swapping as your performance will likely tank horribly.

With respect to the original log, the binary method log would seem to indicate that the source address is connecting to you and just spewing binary data at the server. If you see some consistent addresses in the logs doing that, couldn't hurt to filter them out, though as long as you get your Apache configuration under control so having lots of hits won't overload the machine, having some sporadic screwy connections shouldn't hurt that much.

– David

Big thanks to caker and db3l - I've set MaxClients to 20 as well as reduced my swap.

Just one part of my question that I'm still curious about. I know that my setting for swap and MaxClients may have left me susceptible to the Disk I/O rate going high.

But why did Disk I/O get that high in the first place?

Was that IP address hitting me with a DDoS attack?

@MichaelE:

But why did Disk I/O get that high in the first place?
Hard to be absolutely certain after the fact, but given that you were permitting Apache to service up to 150 clients simultaneously, it's certainly a reasonable guess that you exceeded memory and did begin swapping, which could certainly spike the disk I/O if you started thrashing between the processes.

> Was that IP address hitting me with a DDoS attack?
Also tough to say for sure. If you have a gazillion similar looking entries from one (or a few) IP addresses, then perhaps. But if you just have random assortments of those sorts of hits, or sporadic as opposed to really heavy load, it's more likely it's just your average trolling that exists nowadays on the public net where people try to find hosts with exposures.

Note that even if it wasn't a DDoS attack, just having enough of the hits in a short enough period of time could cause the swapping under your original configuration since Apache would try to service them all simultaneously rather than queueing them up with the smaller client limit.

– David