Disk IO Problems
Then one day it started spiking like this:
~~http://www.lykaon.com/diskio.jpg
Previously, it was pretty much the same as the low peaks all the time.
I don't know when it started or why. I have another linode with nearly the same configuration and it doesn't have these problems. I think it started some time after I did some updates to the some of the software.
I'm running lighttpd and mysql. That's about it. Traffic to the site has not increased at all, so it's not just heavier load.
Everything is running fine in general, and I raised my threshold so Linode would stop sending me emails about it, but it still bothers me.
Anyone seen this before? Any ideas on things I could do hunt down the culprit?~~
4 Replies
You seem to be having 3 spikes per hour (every 20 minutes), except between midnight and 6am when the interval is slightly longer. Do you have any cron jobs that run every 20 minutes during the day? Do you have a control panel that might perform periodic updates, etc?
@hybinet:
Install iotop and see which process is using the most I/O during one of those spikes.
You seem to be having 3 spikes per hour (every 20 minutes), except between midnight and 6am when the interval is slightly longer. Do you have any cron jobs that run every 20 minutes during the day? Do you have a control panel that might perform periodic updates, etc?
Thanks for the tips.
Cron isn't even installed, so it's not a cron job. As far as I know, I don't have anything that would run periodically like that. But obviously, something is.
I installed iotop, but it won't run:
Could not run iotop as some of the requirements are not met:
- Python >= 2.5 for AF_NETLINK support: Found
- Linux >= 2.6.20 with I/O accounting support: Not found
Any way around that?
This is what I'm seeing with a ps aux:
/sbin/udevd --daemon
dhclient3 -e IF_METRIC=100 -pf /var/run/dhclient.eth0.pid -lf /var/lib/dhcp3/dhclient.eth0.leases eth0
/usr/sbin/ntpd -p /var/run/ntpd.pid -u 109:114 -g
/sbin/syslogd -u syslog
/bin/dd bs 1 if /proc/kmsg of /var/run/klogd/kmsg
/sbin/klogd -P /var/run/klogd/kmsg
/usr/sbin/sshd
/usr/bin/mysqld_safe
/usr/sbin/mysqld
logger -p daemon.err -t mysqld_safe -i -t mysqld
/usr/sbin/lighttpd -f /etc/lighttpd/lighttpd.conf
/usr/bin/php5-cgi
/usr/bin/php5-cgi
/usr/bin/php5-cgi
/usr/bin/php5-cgi
/sbin/getty 38400 tty1
/usr/bin/php5-cgi
/usr/bin/php5-cgi
/usr/bin/php5-cgi
/usr/bin/php5-cgi
Do any of those look like they could be doing something every 20 minutes?
@Lykaon:
Could not run iotop as some of the requirements are not met: - Python >= 2.5 for AF_NETLINK support: Found - Linux >= 2.6.20 with I/O accounting support: Not foundAny way around that?
Run Latest Paravirt. It wants a Linux version higher than 2.6.20, and Latest Stable is 2.6.18.
Our spikes are also 3 per hour. After getting iotop going, I found that syslogd writes a lot to disk. A bit of Google searching said that its default is to run every 20 minutes…..
Now what? Any suggestions?