Linode Alert - disk io rate

If you look carefully at the graphs on the dashboard, you can usually tell exactly when the I/O spike occurred, give or take 5 minutes. Then you can check your log files for those specific minutes.

But perhaps your I/O threshold is set too low? The default values are relics of the UML days when there was a hardcoded I/O limit of around 500, but these values don't seem to work well with Xen nodes. Everyone's I/O requirements are different, so you can change those thresholds in the linode manager to meet your needs (as long as you're not thrashing the drives too hard). If you're not doing anything out of the ordinary and your I/O is still hovering around the threshold, you may want to set the threshold higher.

OK - it looks like the spike happened at the same time cron.daily is set to run.

I'll have to look in the cron.daily directory and see what all is in there. It may have been something like prebind or updatedb running for the first time.

Also I would suggest not doing updates automatically via cron. What if you update something that has a bug or a library becomes incompatible with the system you are building, it is too much out of your control.

@cherring:

Also I would suggest not doing updates automatically via cron. What if you update something that has a bug or a library becomes incompatible with the system you are building, it is too much out of your control.

CentOS updates are RHEL update and RHEL updates are very thoroughly tested before they are pushed, with the exception of immediate security concerns.

RHEL does not change the library ABI/API except when they make a point release update, and even then they almost never do it (RHEL 5.2 did switch to FireFox 3 - but that took some major discussion and waited until a point release)

If it was Ubuntu or Fedora, nightly check for updates could be problematic, or if I was using Dag/rpmforge/etc (I'm not) - but RHEL/CentOS is really fairly safe to do that with, updates are not pushed unless there is a damn good reason to push them, and then - they are bug/security fixes - and have been through RHEL scrutiny.

There's a reason I chose CentOS for my image.

It's also the distro I have installed on three boxes here at home.

Oh - and yum excludes php* from the updates, php is my own build, I update it and test it as I see fit.

I also know that their update server got hacked fairly recently.

@cherring:

I also know that their update server got hacked fairly recently.

That was Fedora, not RHEL and not CentOS

edit:

correction - rhel was impacted, centos was not, as centos uses their own key to sign packages.

@FunkyRes:

If it was Ubuntu or Fedora, nightly check for updates could be problematic, or if I was using Dag/rpmforge/etc (I'm not) - but RHEL/CentOS is really fairly safe to do that with, updates are not pushed unless there is a damn good reason to push them, and then - they are bug/security fixes - and have been through RHEL scrutiny.

There's a reason I chose CentOS for my image.

It's also the distro I have installed on three boxes here at home.

I was about to say pretty much exactly the same thing, except with "Debian" in place of "RHEL/CentOS".

Yay distro wars! :D

I like Debian - I use to run it on an SE/30.

I stick with CentOS largely because I'm fairly intimately familiar with rpm (use to create packages for a living, er, that was a major part of the job) - but seriously, look at the dependencies of RPM - I like it, it works well, but it isn't KISS.

Then there's yum - an awesome tool, but look at the dependency trail that thing has …

Debian is much simpler with their packaging system, at least dependency wise, without a doubt. But I've too many RPMs in my bloodstream.

-=-=-

The point about automated updates is valid - in a perfect world, you have a test server and absolutely no updates get pushed until you have applied them on your test server and verified that nothing broke.

I don't have that, I do have several CentOS installs but all of them are differently configured than the linode, nor do I particularly wish to dedicate the time and effort to setting one up, my linode is for hobby stuff.

Thus - the only way to be sure an update isn't going to break my linode server is to install it. Might as well let cron do that, and fix things that break after the breakage.

I can tell you though, RHEL is not going to push an API/ABI change to either MySQL or Apache and those apps are going to get some heavy testing before either of them have updates pushed.

I don't see there being an issue, and I have better things to do then set up a test server for my hobby related website.

People get hacked from lack of updated software far far far more often than a hacker gets ahold of an rpm signing key.

I get the warning email each time I do a database dump (and my database is < 10MB at the moment). Recently I started to treat them as confirmations that my backing-up cron job is working ;-)

Not a true problem, but it would make sense if linode offered some guidance while picking correct limits.

Also, as many people tend to have backup/update/… jobs run at the specific moments, would make sense if the warning level was not fixed for the whole day ("I run heavy jobs at [daily \/] at [04:00 AM]").

@Mekk:

I get the warning email each time I do a database dump (and my database is < 10MB at the moment). Recently I started to treat them as confirmations that my backing-up cron job is working ;-)

Not a true problem, but it would make sense if linode offered some guidance while picking correct limits.

Also, as many people tend to have backup/update/… jobs run at the specific moments, would make sense if the warning level was not fixed for the whole day ("I run heavy jobs at [daily \/] at [04:00 AM]").

Only 10mb :D

I have mine set to 1500 I/O's my server bounces from 500 to 3000 pretty often thanks to cron.