Linode keeps OOMing; High IOWAIT; High load average - help?

My first impression is that this is a grossly overpowered box for what you're serving. But maybe your Wordpress application really is that inefficient.

What's probably happening here is that your PHP processes which normally take ~1GB do something so that they take up ~2GB or more, so you spiral into swap use and start IO thrashing.

But you need better data. Run vmstat, iotop, htop or something and report data from the times that your box goes into swap, what is it doing then?

Maybe install something like munin to monitor your system, maybe you'll see some patterns there.

Maybe you can use mod_php instead of fastcgi? Maybe 80% of your traffic could be handled by a reverse squid proxy?

Well, the thing is, he's using prefork.

If he switched to mod_fastcgi, good, but still it's the prefork that sucks so much ram up. Keep fastcgi, switch to worker, and seriously, MaxClients down to 100-120 is totally enough. Especially after you cut down KeepAliveTimeout to 5.

coughwordpresssuckscough

````
[Mon Sep 20 11:51:13 2010] [error] server reached MaxClients setting, consider raising the MaxClients setting

````
Do not ever listen to what this error message says. It is very misleading in a VPS environment! When you're running out of memory, you want a lower MaxClients setting, not higher.

Some rules of thumb, assuming mpmprefork and modphp:

Linode 512: MaxClients 25 or less

Linode 1024: MaxClients 50 or less

Linode 1536: MaxClients 75 or less

Linode 2048: MaxClients 100 or less

Anything more and you're likely to swap.

Also, install the WordPress Super Cache plugin. This solves the problem 95% of the time. I wouldn't even start looking for another source of trouble until this was installed.

Thanks everyone for responding… the system start thrashing again today around 1:30PM EST (I wasn't near by to deal with it, so I don't have any good details)…

I was running iostat -x 1 when it took place and got this (not very useful to me) -- last few entries:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.77    0.00    7.79   91.32    0.12    0.00

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
xvda             19.00     0.00 1157.00    5.00 33456.00    40.00    28.83   220.60  218.08   5.02 583.00
xvdb            415.00   206.00  246.00 1519.00  5056.00 13824.00    10.70   981.47  636.18   3.47 611.90

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.96    0.00    5.76   93.16    0.12    0.00

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
xvda              0.67     1.01  116.95    0.17  3393.29     4.03    29.01    42.31  357.67   6.29  73.61
xvdb             32.72    24.16   14.77  164.93   408.05  1551.68    10.91   125.80  726.63   4.20  75.42

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.41    0.00   11.54   86.92    0.12    0.00

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
xvda              1.56     0.00   82.58    0.31  2397.51     7.47    29.01    27.58  242.03   4.11  34.11
xvdb             26.75    11.35   15.40   95.02   364.54   831.10    10.83    52.70  481.36   3.16  34.93

I had tried Worker MPM and w/ FastCGI but it was still happening. In fact, I just changed it to prefork, mod_php this past Friday as an attempt to fix the issue.

I also have W3 Total Cache installed, which I personally think out performs Super Cache.

I'll try lowering MaxClients tonight.

I know what I have is over-kill for the traffic being generated, this is why I am utterly confused. I manage several servers and this is the only one giving me grief.

Thanks again.

Your numbers clearly say that you're out of memory and swapping hard. In the default configuration, /dev/xvdb is the swap partition.

@lewayotte:

I also have W3 Total Cache installed, which I personally think out performs Super Cache.
IIRC, if you use Total Cache, you still need to go through the PHP interpreter. Total Cache has truckloads of features, but it comes at the expense of having to use PHP. (After all, its features are written in PHP.) At least that's how I remember it; recent versions may have changed a bit.

In contrast, Super Cache tinkers with your .htaccess file so that most pages completely bypass PHP. The two plugins have different performance characteristics. You have to judge not only by raw speed but also take your low-memory situation into account. I think the "bypass PHP altogether" approach has clear benefits in this regard, but YMMV.

I would suggest using Apache worker with mod_fastcgi. Something like StartServers 4, ThreadsPerChild 32 (or even 64). Apache would then eat much less memory. FcgidMaxProcesses maybe to 64? And I would (for a testing purposes) disable caching unless you are hitting too much I/O (CPU is rather irrelevant). You will serve pages slower but you should be able to serve them more.

Using prefork, you would save next to nothing since PHP engine is loaded even for serving static content.

I advise against Apache if you don't need any Apache specific features. Lighty or Nginx will do just fine.

I am running a low-profile website (average 1200 visitors, 400-450 thousand hits per day), in php. Using worker and mod_fastcgi. With a bit of tuning, it all runs in near-constant memory space (my usual "used RAM" readout with all running services is ~170MB, of which the whole set of apache and php-cgi processes is maybe about 100).

It's way easier than trying to convert all the apache dependencies into lighttpd's or nginx's syntax, setting up proxing, and/or (in case of nginx) compiling the whole mess by hand to get anything resembling "recent version".

If anyone's interested…

apc.shm_size = 64
(...)
    StartServers          2
    MaxClients          250
    MinSpareThreads      25
    MaxSpareThreads      75
    ThreadsPerChild      25
    MaxRequestsPerChild   0
    ThreadStackSize 2097152 # Linux default is 8MB, a ton of wasted VMem... 2MB seems totally safe, probably could be reduced further.
(...)
        FastCgiConfig \
                -idle-timeout 120 \
                -initial-env PHP_FCGI_CHILDREN=24 \
                -initial-env PHP_FCGI_MAX_REQUESTS=500 \
                -killInterval 100000 \
                -listen-queue-depth 300 \
                -maxClassProcesses 1 \
                -singleThreshold 0

I know it's a very blunt force solution, but I'd honestly suggest just backing up your database and wordpress configs, and reinstalling the server. Then triple check your backed up configs, and toss them back on. Odds are, you accidentally deleted a # in a random config file while tuning things, and uncommented "eat-all-swap-space=1" or something.

All the power to you if you want to debug it, but a rebuild would probably be quite a bit faster.

@akerl:

Odds are, you accidentally deleted a # in a random config file while tuning things, and uncommented "eat-all-swap-space=1" or something.
I doubt it 8)

@lewayotte:

StartServers 8

MinSpareServers 5

MaxSpareServers 20

ServerLimit 300

MaxClients 300

MaxRequestsPerChild 4000
Three hundred prefork processes, each fully loaded with the PHP engine. No surprise the box is OOMing.