[SOLVED] Need to figure out why Linode is crashing
It hosts two web sites , the one with all the traffic is Wordpress running around 6000 hits/hour. I had been experiencing sporadic hangs system-wide and so did some tuning. I -
turned on WP caching
cut down the number of queries displayed on the WP / page
attempted to move to apache2-mpm-worker, but rolled back to prefork (reinstalling phplib5 in the process)
at apache2.conf, I took timeouts from 300 to 1 second
Every 4 hours or so, the Linode panics and crashes:
~~http://kleptonomicon.com/images/graphs2.jpg
Apache /server-status
Apache Server Status for
Server Version: Apache/2.2.14 (Ubuntu) PHP/5.3.2-1ubuntu4.7 with Suhosin-Patch mod_ssl/2.2.14 OpenSSL/0.9.8k
Server Built: Nov 18 2010 21:17:19
Current Time: Saturday, 05-Feb-2011 04:53:39 UTC
Restart Time: Saturday, 05-Feb-2011 03:57:23 UTC
Parent Server Generation: 0
Server uptime: 56 minutes 16 seconds
11 requests currently being processed, 7 idle workers
KKKWKK_K…..KK.K….K………………………………
……………………………………………………….
……………………………………………………….
……………………………………………………….
Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process
PID Key:
3383 in state: K , 3371 in state: K , 3323 in state: K
3388 in state: W , 3376 in state: K , 3389 in state: K
3390 in state: _ , 3391 in state: _ , 3392 in state: _
3375 in state: _ , 3360 in state: K , 3393 in state: _
3394 in state: _ , 3395 in state: _ , 3367 in state: K
3368 in state: K , 3370 in state: K , 3298 in state: K
vmstat 1 20
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 0 161908 21384 250356 0 0 13 4 92 20 3 0 95 1
0 0 0 161684 21396 250344 0 0 0 312 1024 184 6 0 94 0
0 0 0 160568 21396 250356 0 0 0 0 227 42 0 0 99 0
1 0 0 158212 21404 250352 0 0 0 0 2110 245 11 2 87 0
0 0 0 159288 21436 250336 0 0 0 0 2927 509 15 2 83 0
0 0 0 159040 21436 250472 0 0 0 0 1683 267 9 1 90 0
0 0 0 159032 21436 250472 0 0 0 52 113 38 0 0 100 0
0 0 0 157032 21448 250472 0 0 0 104 222 44 0 0 99 0
0 0 0 157032 21448 250472 0 0 0 0 154 32 0 0 100 0
1 1 0 156328 21452 250468 0 0 4 0 1285 285 5 0 95 0
2 0 0 140572 21496 250464 0 0 40 0 1502 232 6 1 91 1
1 0 0 140076 21496 250484 0 0 28 56 2219 475 14 1 84 0
0 0 0 142928 21496 250516 0 0 0 0 341 53 2 0 98 0
0 0 0 142680 21516 250508 0 0 0 576 961 218 5 0 95 0
1 0 0 130892 21516 250576 0 0 0 0 545 81 2 0 98 0
0 0 0 129892 21520 250572 0 0 0 0 1734 357 10 1 89 0
0 0 0 130264 21520 250632 0 0 0 736 1188 184 6 0 94 0
0 0 0 130264 21520 250680 0 0 0 0 137 46 0 0 100 0
0 0 0 130264 21520 250680 0 0 0 0 51 32 0 0 100 0
0 0 0 145144 21528 250676 0 0 0 124 98 37 0 0 100 0
free -m
total used free shared buffers cached
Mem: 751 609 141 0 21 245
-/+ buffers/cache: 343 407
Swap: 255 0 255
ALSO:
I put a value of 30 into /proc/sys/kernel/panic but the most recent crash did not reboot after the panic. I also put 1 into /proc/sys/kernel/paniconoops …anybody know why that isn't working?
-r~~
5 Replies
@warmowski:
- attempted to move to apache2-mpm-worker, but rolled back to prefork (reinstalling phplib5 in the process)
What is MaxClients set to in your apache2.conf? As a rough guess, for prefork I'd suggest starting at around 30 on a Linode 768 and adjusting as memory usage patterns indicate.
@Vance:
@warmowski:
- attempted to move to apache2-mpm-worker, but rolled back to prefork (reinstalling phplib5 in the process)
What is MaxClients set to in your apache2.conf? As a rough guess, for prefork I'd suggest starting at around 30 on a Linode 768 and adjusting as memory usage patterns indicate.
MaxClients is at 30. In which direction should adjustment go?
In the end I reinstalled PHP minus the patch and that seems to cure it. Can't say it will do for you but something to consider if everything else fails.
I can't remember the link I followed, but this looks like it's the correct direction
Went from
Latest 2.6 Stable 2.6.18.80-linode10
to
2.6.37-linode 30
Thanks caker and everybody else!
-r