Thrashing, Yelling, Crying - Advice Needed
At least once a week my server goes "down" and when I go in with Lish all I see is apache processes getting killed.
While the server is down my SWAP and IO is in the RED and no other metrics are out of normal.
Linode 1536
Ubuntu 10.10
2.6.32.16-linode28
512MB SWAP
~~![](<URL url=)
apache2 -l
Compiled in modules:
core.c
modlogconfig.c
mod_logio.c
prefork.c
http_core.c
mod_so.c
apache2.conf
MinSpareServers 3
MaxSpareServers 5
ServerLimit 48
MaxClients 48
MaxRequestsPerChild 4000
keepalive OFF
/etc/mysql/my.cnf
key_buffer = 16M
maxallowedpacket = 16M
thread_stack = 192K
threadcachesize = 8
myisam-recover = BACKUP
skip-innodb
table_cache = 128
querycachelimit = 1M
querycachesize = 16M
expirelogsdays = 10
maxbinlogsize = 100M
maxallowedpacket = 16M
key_buffer = 16M
php.ini
memory_limit = 256M
Any recommendations appreciated!~~
6 Replies
If you do nothing else, try dropping MaxClients significantly. You may slow down your peak request rate somewhat, but you should prevent the system from getting into such a bad state. Finding the best value is heavily application and load dependent (and also addressed more fully in past forum threads), but simply making that number much smaller is a quick 'n dirty first step to ensuring you stick within the resources you have available.
– David
I've been beating the linode forum and other website threads..
trying this.. trying that.. I can't seem to find the right mix of configurations..
I really wish someone would look at my my.cnf / php.ini and apache2.conf and tell me it looks decent or if they see a glaring issue…
Overall the system seems to be big enough for my LAMP load and the SWAP issue seems to be random with no rhyme or reason. I'll push the maxclient down some more and see.. but all my tweaks seem to do NO help at all.
PHP Memory Limit 256M
48 x 256M = 12G of RAM!!!
There's no way you can fit all of that in your mid-sized Linode. The only reason your Linode works at all is because very few PHP scripts use anywhere near 256M of RAM. But even if each script used only 32M of RAM (typical for heavyweight CMSs), you'd run out of memory just the same. Throw in a couple of image processing add-ons, and you're going to hit the swap.
You need to do two things: cut down MaxClients to half of the current value (24), and cut down the memory limit to no more than a quarter of the current value (64M). That should give you a relatively safe starting point. Then enable the error log in php.ini. If some scripts require more RAM, errors will show up in the log file. Adjust the memory limit to make sure that all your scripts work, or get rid of add-ons that consume more memory than they're worth. Every time you need to increase the memory limit, decrease MaxClients to compensate. If your server works fine without hitting swap for a few weeks, try increasing MaxClients a little bit. It takes trial and error to find the sweet spot. (Good call on disabling KeepAlive, but giving it a very low value such as 1-2 wouldn't hurt, either.)
By the way, you have two keybuffer lines in your my.cnf. Delete the second one. While you're at it, try increasing keybuffer a bit, and see if it helps your pages load faster. Since your site seems to get a significant amount of traffic, consider transitioning to the InnoDB storage engine. But that's for the long term, and you'll need to budget a bit more RAM if you want to use InnoDB. Right now, you should focus on getting your memory usage under control.
Now this is what I needed to here.. thank you for helping tie this together for me.
Many thanks for your time.
max_connections = 100
table_cache = 128
thread_concurrency = 10
key_buffer = 16M
maxallowedpacket = 16M
thread_stack = 192K
threadcachesize = 8
[isamchk]
key_buffer = 16M
@kutu:
So you want want me to remove the "keybuffer" located under [isamchk] ?? Will do.. just for my brain, why? Oh, that was under a different section? Then don't delete it. In your first post it looked like you had the same line twice in the same section. (By the way, the one that you should try adjusting is the keybuffer line in the [mysqld] section.)