swap of death

Make sure you've got mysql and apache configured appropriately for the amount of RAM you have; your maxclients in Apache, if using mpm_prefork (the default I believe), should not be greater than the RAM available to Apache divided by max ram per Apache process.

Other than that, throw money at the problem and upgrade the RAM of the box until you have time to fix it. It's all pro-rated to the day, and the upgrade takes a few minutes.

Guspaz is correct, but it doesn't look like swap is your major problem; the swap line only spikes twice on the graph.

What are you running every hour that hits the disk so hard? If it's a web logfile analyzer, try deleting older logfiles or moving them out of the way to another location. The quiet periods in between seem to indicate that your normal workload doesn't require gobs of I/O.

You could also put ionice -c 3 at the front of your cron jobs; this will not reduce their disk usage, but will ensure that disk requests from the web and database server get priority. Note that it may extend the time that the jobs take to run - since your jobs appear to be taking 40 minutes or so, this could cause them to run for longer than an hour, meaning overlapping jobs would be kicked off before the previous ones completed. Try to take care of the massive disk usage aspect first.

I don't know what exactly is running on your server or how it's configured so I can only guess.

If I was you I'd go though the various crontabs and see what can be removed or tuned. See how many worker processes you have for your various daemons and maybe eliminate a few. Check PHP-APC, memcached, or a database isn't eating large amounts of ram.

Throwing memory at the problem is the quickest way to fix the performance if you are spending someone else's money.

Thanks everyone - definitely cron jobs are the culprit here. I've rescheduled jobs a little more evenly and it helped somewhat.

I think the cron jobs are maybe causing some big full-table scans (the DB is much larger than my available RAM) but i can't quite pick them up in the slowlog. The jobs do need to run and mostly performance isn't an issue - until suddenly, everything falls into the swap cascade and thrashes for an hour.

The cron jobs don't do any disk stuff outside of DB calls (and maybe queueing email, but email queues are low), so i think if anything is swapping, it might be the database server, can't ionice it without affecting interactive users of the web application(?).

Is there a way of penalising the kernel scheduling of tasks undergoing high swap? Or maybe when IO load gets high, suspend a set of processes and run them one at a time instead of all at once? Then the jobs would actually proceed albeit slowly. (I'd write a script to do it, but when the server goes into swap of death, running another process seems counter-intuitive) Googling for a kernel scheduler that penalises processes with high swap gives me results about an IBM patent.