What can be done about the load spikes?

Hi all. I've been a Linode customer for about a year and a half now. I've experienced load spikes of varying severity pretty much the entire time; I started on host5 which had them very consistently at a certain time of the night (due to that host being set up before caker's scripts were moving crontab jobs around), then I was moved to host31 when that host was available. Both are Linode64 hosts. I've seen a steady increase in the load spikes on host31 over the past 6 months and it's getting a bit unbearable. My munin graphs show a load spike of greater than 3 (I've found that a load spike of 3 or above renders the machine effectively unuseable; between 2 and 3 the spike slows the machine down but it is still basically useable) about 2 - 3 times per day on average.

My subjective experience mirrors this too - I'd say once every couple of days I happen to be trying to do something with my Linode at the time of a spike and I find that my Linode is effectively unreachable. It's basically the equivalent in effect to a network outage several times a day every day. If there were a network outage at the data center this frequently, the terms of service would indicate that a rebate is in order. These load spikes have the same effect but apparently don't invoke the same terms of service.

I'm trying to figure out what the best hope of avoiding these spikes is. I have watched my Linode's performance for several months and have found that my host has used io tokens only a couple of times - and never gone below 50% io tokens available (except on the once or twice that I tested the io token functionality by purposely running the io token count down). So it's not my Linode that is causing the load spikes, which are generally acknowledged to be due to one or more Linodes on the host hammering the disk. Unless - maybe the load spikes are sometimes just a natural consequence of 30+ Linodes all doing a bit more io than normal at the same time, but not enough for any of them to start using io tokens?

Would it be possible to tune a host to have an even stricter io token limit? One that is more tuned to hosts which are mostly idle (my Linode is basically just a web server and mail server and uses very little resources)? If so, I would gladly move to that host if it meant a slightly slower overall performance but a reduction (elimination hopefully) of the load spikes.

Also, how much do newer kernels with io load balancing help? Do these kernels have to be installled on the host or the Linodes or both? I am still running a 2.4 kernel because I haven't gotten around to upgrading yet. I don't think that my kernel has anything to do with the spikes though, it's other Linodes that are causing it as far as I can tell.

I know that other people have complained of the same problem - I've read some other posts where people say that their Linodes are inaccessible during certian times of the day, or randomly - and I attribute it all to these load spikes.

I'll post my most recent munin graphs for load average here so you can see what I am talking about.

Here's my daily graph for today. There was a 15+ load spike early this morning. It was so bad that my other munin graphs have a gap in them at that time because munin didn't run as scheduled.

~~http://www.ischo.com/load_graphs/load-daily.png" />

Weekly:

~~http://www.ischo.com/load_graphs/load-weekly.png" />

Monthly: (here you can really see the worsening of the load spike problem over the last couple of weeks)

~~http://www.ischo.com/load_graphs/load-monthly.png" />

Yearly: (here we see that even though the average load is low, there are so many load spikes that the entire graph is filled with them)

~~http://www.ischo.com/load_graphs/load-yearly.png" />

I love Linode's service otherwise, the load spike issue has been a real problem now for the entire time that I've used this service. During those periods when the load spikes are relatively small and infrequent, I am a very happy customer indeed. But unfortunately the load spikes have been getting worse and worse, and over the course of my subscription to the Linode service, have more often than not been a real problem for me.

Any suggestions on what can be done by me with my Linode or by Linode itself to solve this problem would be very, very welcome. Thank you!~~~~