Up and down, up an down...

It is always at the same time of day that the errors happen?

Adam

@Drowned:

My server runs fine almost all day. The whole day it'll be below 1 load average, but at but around 4-5 times a day my load will show numbers like 14 to 64!! Everything halts and I can't even login to my shell account cause the server is so loaded :(

I checked everything and my configuration seems to be fine, so that only leaves the other people in host10. Will other account distro installations cause this kind of problem?

I seriously doubt that any other activity on a Linode host system would show up in your Linode as load. Even if all of the other Linodes on your Linode host are running 100% CPU processes, I'd expect your Linode to show a load level of 0, since the load level reflects the number of processes running on your Linode that want CPU, not the number running on the host system.

That being said, load on the host system will make your Linode run more slowly, as the processes in your Linode get less actual CPU time when they have to share with other processes on the host. Thus a high load count on your Linode could indicate that the amount of CPU that your Linode is getting is not sufficient for the amount of work that it needs to do, and perhaps processes are building up as users queue up requests faster than the Linode can process them.

My other thought is that there could be some kind of I/O problem. Chris has mentioned several different kinds of problems with the kernel's handling of I/O, maybe this is affecting you. I'm on host5 and have never noticed any freezing up problems at all, although once in a while it does get a bit slow. But in general the performance is quite good. I just run a mail server and web server though, I'm not loading my Linode up with lots of processes that all want the CPU. Remember that even a Linode 128 is only guaranteed 128 Mhz of "virtual CPU" - basically meaning that if the host system is completely loaded due to every Linode on it using as much CPU as it can, all at the same time (quite unlikely if you ask me), then you will see performance of your Linode that is a fraction of what you would see on an unloaded Linode - about 128 Mhz equivalent, which is quite slow for anything other than the very lightest of loads. I guess that when everyone's cron jobs all fire at the same time, the load could very easily spike. I have moved my cron jobs to unusual minutes, and I recommend that everyone else do the same.

The real question is, do any other Linode users on your host system (I think from what I have read in the past, that it is host7?) experience the same very high load averages at the same time that you do? If not, then it is either just that you are overloading your Linode with too much work, and the run level is snowballing, or, like I said, maybe there is some kind of I/O problem that is holding up processes on your Linode and causing them to spin excessively.

Either way, I don't have any answers, but I hope that I have provided a little bit of insight …

At the moment, there are only six other Linode accounts on host10. But, just to be sure, I'll gather vmstat output for a few days.

-Chris

I'm on host10, the host you are now on. What times of day do you notice this? I use irssi, an irc client, via ssh through out the day to provide support in #xbox-linux. I haven't seen anything like this yet. Well I did have something slightly similiar. Yesterday evening about 10-11 pm EDT I did a few update set uid=xxxx where uid=yyyy in mysql. My load only got up to about 4 or 5 at max This table had half a million lines and around 100,000 where affected by this. The thing is though, the responsiveness problems I had while this was going on were inline with what I was doing on my linode. Like I said, I haven't had anything like this happen except at that time.

Well, the reason I asked about server load from other accounts affecting mine is because the load increases out of nowhere… without a reason. I don't change a thing and traffic is always the same (around 40 forum users all day). It just it suddenly peaks and everything halts. A few minutes later it goes back to normal.

As for now, It has been working fine all afternoon. In the morning the server peaked 2 times. Sometimes it happens in the afternoon and sometimes all day around 5 times.

@mikegrb:

I'm on host10, the host you are now on. What times of day do you notice this? I use irssi, an irc client, via ssh through out the day to provide support in #xbox-linux. I haven't seen anything like this yet. Well I did have something slightly similiar. Yesterday evening about 10-11 pm EDT I did a few update set uid=xxxx where uid=yyyy in mysql. My load only got up to about 4 or 5 at max This table had half a million lines and around 100,000 where affected by this. The thing is though, the responsiveness problems I had while this was going on were inline with what I was doing on my linode. Like I said, I haven't had anything like this happen except at that time.

Well, with that answer I guess I have to be the one making the problem. I guess I'll go back to checking my config :( I just can't seem to find what the source of the problem is :(

This isn't directly related to your problem, but I've noticed that various apps make Linode go nuts.

I've been on two hosts, and on both of the hosts running screen for over ~24 hous from the Debian Small distro makes the linode practically rape the CPU. The odd part was it wasn't noticed by top, or another userland command.

The only bit of help I can offer is the following script … just set the two variables to your liking … and run (you may wish to redirect the output to a file ..). This script is very tiny so that it makes minimal impact on your linode

#!/bin/sh

INTERVAL=
LIST_SIZE=

while [ true ] ; do
    date
    echo "Top $LIST_SIZE memory eating apps ... "
    ps -eo %mem,command | grep -v '%MEM' | sort +1n | tail -n $LIST_SIZE

    echo "Top $LIST_SIZE CUP eating apps ... "
    ps -eo %cpu,command | grep -v '%CPU' | sort +1n | tail -n $LIST_SIZE

    echo ""
    sleep "${INTERVAL}s"

done

HTH

Sunny Dubey

I've just made available 2.4.23-pre8 kernel for the Linodes – I encourage those of you who are having performance problems give this kernel a try (edit your configuration profile, select from kernel list, save, reboot). It contains many important patches to the VM system which I've had success with so far on host9 and 10.

It is interesting to note that UML kernels inherit their features and bugs alike from 'real' kernel versions. I've found I/O performance to be lacking in the last few 2.4 kernels.

I'm very curious to find out if the new 2.4.23-pre8 Linode kernel improves things.

-Chris

EDIT: I was wrong about the Anticipatory I/O Scheduler being included in recent 2.4.23 kernels. It isn't :o

@caker:

I've just made available 2.4.23-pre8 kernel for the Linodes – I encourage those of you who are having performance problems give this kernel a try (edit your configuration profile, select from kernel list, save, reboot). It contains many important patches to the VM system which I've had success with so far on host9 and 10.

It is interesting to note that UML kernels inherit their features and bugs alike from 'real' kernel versions. I've found I/O performance to be lacking in the last few 2.4 kernels.

I'm very curious to find out if the new 2.4.23-pre8 Linode kernel improves things.

-Chris

EDIT: I was wrong about the Anticipatory I/O Scheduler being included in recent 2.4.23 kernels. It isn't :o

Testing it as we speak :D :D :D :D :D :D

Well, with the new kernel I haven't had a single load problem all morning :D :D :D :D :D :D :D :D

Right now I could say it's fixed :D

I'll let you know how it works in the afternoon ;)

BTW, I have 61 users online right now and the load average for the past 15 mins is 0.77 :D