PHP Slow, initially..

I've no idea why this is happening, but the quick and dirty workaround is a cron job that requests dynamic output from your Web server and pipes it to null once a minute.

..yuck!

Usually this is because you've filled up your RAM, and after a minute or so of inactivity, said processes might be swapped out for something else, and it adds a good amount of time to swap it back in when it's needed again. This is a fairly general answer though, and may not be the case at all, but you didn't give us much detail to really go on. Some statistics on system resources could be helpful: output of 'cat /proc/meminfo', iolimiter info ('cat /proc/iostatus'), a few of the top lines from top, etc.

Thanks for the responses! I think I've risen to the level of my incompetence as far as sys admin goes.

Output below. top sorted by memory indicates that mysql is the largest memory hog. Memory is on the low-ish side. Perhaps I should cut the number of mysql processes?

free -m:

total used free shared buffers cached

Mem: 288 275 13 0 39 29

-/+ buffers/cache: 207 81

Swap: 64 9 55

cat /proc/io_status:

iocount=2732964 iorate=4 iotokens=400000 tokenrefill=512 token_max=400000

cat /proc/meminfo:

total: used: free: shared: buffers: cached:

Mem: 302915584 288636928 14278656 0 40415232 37089280

Swap: 68149248 10403840 57745408

MemTotal: 295816 kB

MemFree: 13944 kB

MemShared: 0 kB

Buffers: 39468 kB

Cached: 30412 kB

SwapCached: 5808 kB

Active: 36740 kB

Inactive: 38972 kB

HighTotal: 0 kB

HighFree: 0 kB

LowTotal: 295816 kB

LowFree: 13944 kB

SwapTotal: 66552 kB

SwapFree: 56392 kB

top:

top - 06:25:14 up 7 days, 16:44, 1 user, load average: 0.00, 0.00, 0.00

Tasks: 51 total, 1 running, 50 sleeping, 0 stopped, 0 zombie

Cpu(s): 0.0% user, 0.0% system, 0.0% nice, 100.0% idle

Mem: 295816k total, 282396k used, 13420k free, 39780k buffers

Swap: 66552k total, 10160k used, 56392k free, 30460k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

1106 mysql 9 0 25300 24m 4076 S 0.0 8.6 0:00.84 mysqld

1122 mysql 8 0 25300 24m 4076 S 0.0 8.6 0:00.74 mysqld

1123 mysql 9 0 25300 24m 4076 S 0.0 8.6 0:00.00 mysqld

1124 mysql 9 0 25300 24m 4076 S 0.0 8.6 0:00.11 mysqld

1125 mysql 9 0 25300 24m 4076 S 0.0 8.6 0:00.05 mysqld

1128 mysql 9 0 25300 24m 4076 S 0.0 8.6 0:00.00 mysqld

1164 mysql 9 0 25300 24m 4076 S 0.0 8.6 0:00.29 mysqld

1165 mysql 9 0 25300 24m 4076 S 0.0 8.6 0:00.30 mysqld

1166 mysql 9 0 25300 24m 4076 S 0.0 8.6 0:00.00 mysqld

1169 mysql 9 0 25300 24m 4076 S 0.0 8.6 0:04.24 mysqld

1179 mysql 9 0 25300 24m 4076 S 0.0 8.6 0:00.00 mysqld

28062 nobody 9 0 17684 13m 6228 S 0.0 4.7 0:05.18 httpd

28078 nobody 9 0 17460 13m 6352 S 0.0 4.7 0:03.44 httpd

28111 nobody 9 0 17564 13m 6204 S 0.0 4.7 0:03.27 httpd

28063 nobody 9 0 17372 13m 6144 S 0.0 4.6 0:03.91 httpd

28106 nobody 9 0 17224 13m 6256 S 0.0 4.6 0:02.98 httpd

28105 nobody 9 0 17200 13m 6092 S 0.0 4.6 0:02.66 httpd

28061 nobody 9 0 17056 13m 6056 S 0.0 4.6 0:09.86 httpd

28065 nobody 9 0 17092 12m 5984 S 0.0 4.5 0:03.07 httpd

28064 nobody 9 0 16900 12m 5752 S 0.0 4.3 0:02.54 httpd

28060 nobody 9 0 15084 11m 4104 S 0.0 3.9 0:05.71 httpd

25937 root 8 0 9284 7992 7308 S 0.0 2.7 0:01.67 httpd

1155 ntp 9 0 3672 3672 2760 S 0.0 1.2 0:00.01 ntpd

1250 postfix 9 0 2468 2468 1368 S 0.0 0.8 0:00.01 qmgr

Most of the memory shown as being used by MySQL will be shared between all the instances of the program.

I think that configuring MySQL and Apache to use less memory will only be marginally beneficial on a Linode 300 using a 2.6 kernel. The kernel will not necessarily stop swapping out parts of running applications even if you reduce memory usage below the size of physical memory - it may well create a bigger disk cache. Disk caching on a Linode is of questionable value because caching is also done by the host system kernel. Your Linode has ~10% of memory (30412 kB - more that MySQL or Apache are using) in use for disk caching and is swapping out parts of running applications when thet are idle.

My suggestion: adjust the kernel's 'swappiness' value. The swappiness defaults to 60, on a scale of 0 (prefer shrinking disk cache) to 100 (prefer swapping out inactive pages). Try reducing your swappiness value to between 0 and 20: echo 10 > /proc/sys/vm/swappiness
This will encourage the kernel to shrink the disk buffers before swapping out applications.

@pclissold:

My suggestion: adjust the kernel's 'swappiness' value. The swappiness defaults to 60, on a scale of 0 (prefer shrinking disk cache) to 100 (prefer swapping out inactive pages). Try reducing your swappiness value to between 0 and 20: echo 10 > /proc/sys/vm/swappiness
This will encourage the kernel to shrink the disk buffers before swapping out applications.

I'm looking at this now. This may be a silly question, but do I have to login via lish to write to the vm/ directory? Currently there is no swappiness file on my gentoo distribution and trying to write to it results in:

localhost ~ # echo 10 > /proc/sys/vm/swappiness
bash: /proc/sys/vm/swappiness: No such file or directory

Thanks again for the help.

@nurikabe:

This may be a silly question, but do I have to login via lish to write to the vm/ directory?
No, you just have to be root.

@nurikabe:

Currently there is no swappiness file on my gentoo distribution and trying to write to it results in:

localhost ~ # echo 10 > /proc/sys/vm/swappiness
bash: /proc/sys/vm/swappiness: No such file or directory

What kernel are you using?

@pclissold:

What kernel are you using?

Twould be 2.4.29-linode39-1um.

2.4.x kernels aren't tunable for swappiness - it was introduced in 2.6. It's probably worth your while to upgrade - the VM code in 2.6 is generally better. My feeling - no hard data though - is that I've had less memory/swap issues since switching to 2.6.

@pclissold:

My feeling - no hard data though - is that I've had less memory/swap issues since switching to 2.6.

Okay, finally had a chance to do this. (I was gearing up to recompile the kernel myself.. and then remembered that I can just switch it from the Linode control panel. How I do love Linode.)

The result: with 2.6 and reduced swappiness, PHP seems to be much less groggy when awakened. There's still room for improvement, but we've gone from delays of dozens of seconds to three or four. Thanks again for all your help!

Evan

@nurikabe:

Okay, finally had a chance to do this. (I was gearing up to recompile the kernel myself.. and then remembered that I can just switch it from the Linode control panel. How I do love Linode.)
Evan - with linode, as with any VPS service, you're forced into using the kernels that are provided for you by the host. As such, trying to compile and run your own kernel is an excercise in futility.