host5 performance issues 1:25 am PST?

Actuallly I think I might have been misreading the host load from my members page. It is possible that the host load was listed as high before and not low as I previously reported; it's certainly high right now in any case.

My Linode's load is still listed at from 2.5 to 3.5, but is much more responsive than before. It's now almost 1:40 am PST …

This is a nightly occurrence on the first few hosts, since they have Linodes with distro images deployed before I removed from cron most of what causes the high disk contention (updatedb, etc). I did pause each Linode for a few moments to determine if it was a specific Linode thrashing.

I've been working on a 2.6 host kernel and am preparing to deploy it fairly soon. 2.6 has improvements to the I/O system, and I'm looking forward to getting it online.

-Chris

Well in the spirit of trying to help mitigate this problem through cooperation of the various Linodes on host5, I'm posting my /etc/crontab listings for hourly, weekly, daily, etc here. If you have a Linode on host5 could you do the same so that we can all try to find times that do not coincide?

04,09,14,19,24,29,34,39,44,49,54,59 * * * * root run-parts /etc/cron.fiveminutely

07 * * * * root run-parts /etc/cron.hourly

17 4 * * * root run-parts /etc/cron.daily

27 4 * * 1 root run-parts /etc/cron.weekly

47 4 2 * * root run-parts /etc/cron.monthly

Notes:

I made my own "cron.fiveminutely" to run stuff every five minutes. I use this for webalizer.

My hourly times are on seven minutes after the hour.

My daily times are at 4:17 am eastern time.

My weekly times are at 4:27 am eastern time on Monday morning.

My monthy times are at 4:47 am eastern time on the second day of each month.

I run backups at 10:04a, 10:12a, 2:04p, 2:12p, 6:04p, 6:12p, 10:04p, and 10:12p all Pacific Time. Unfortunately, these are initiated from my home machine which currently is not synced with a time server. I'll try to remedy that shortly.

Why backups that frequently and why pairs 8 minutes apart? What backup method are you using? Even if it is incremental this puts quite a load on the server to check files and determine which have changed.

@mikegrb:

Why backups that frequently and why pairs 8 minutes apart? What backup method are you using? Even if it is incremental this puts quite a load on the server to check files and determine which have changed.

It is odd to have them in pairs like that, but there probably is a reason; maybe backing up a database before some operation is done to it and then backing it up afterwards too?

I don't think that the backups that this person is doing is putting an appreciable load on host5. I have never noticed performance problems, and no one else seems to be reporting them, at the times that this person is doing backups.

The real problem seems to be all of the updatedb cron jobs firing at the same time late at night (eastern time). Hopefully people who have left their updatedb cron jobs to their default times will see these messages and remedy this.

@mikegrb:

Why backups that frequently and why pairs 8 minutes apart? What backup method are you using? Even if it is incremental this puts quite a load on the server to check files and determine which have changed.

The paired backup operations go in different directions, server to home and home to server. I separated them just to preserve scheduling flexibility.

The backups use hard links and rsync, to maintain multiple snapshots of the other machine. They get issued every four hours because I don't want to lose more than four hours of data :-)!

As bji suggested, this should not be a substantial load. They take no more than a couple minutes each.