Thrashing, Yelling, Crying - Advice Needed

I'm at my wits end, and need your advice.

At least once a week my server goes "down" and when I go in with Lish all I see is apache processes getting killed.

While the server is down my SWAP and IO is in the RED and no other metrics are out of normal.

Linode 1536

Ubuntu 10.10

2.6.32.16-linode28

512MB SWAP

~~![](<URL url=)http://fantasyknuckleheads.com/wp-conte … pture2.jpg">http://fantasyknuckleheads.com/wp-content/uploads/2011/11/Capture2.jpg" />

apache2 -l

Compiled in modules:

core.c

modlogconfig.c

mod_logio.c

prefork.c

http_core.c

mod_so.c

apache2.conf

StartServers 3

MinSpareServers 3

MaxSpareServers 5

ServerLimit 48

MaxClients 48

MaxRequestsPerChild 4000

keepalive OFF

/etc/mysql/my.cnf

key_buffer = 16M

maxallowedpacket = 16M

thread_stack = 192K

threadcachesize = 8

myisam-recover = BACKUP

skip-innodb

table_cache = 128

querycachelimit = 1M

querycachesize = 16M

expirelogsdays = 10

maxbinlogsize = 100M

maxallowedpacket = 16M

key_buffer = 16M

php.ini

memory_limit = 256M

Any recommendations appreciated!~~

6 Replies

Check through the forum for Apache tuning tips - the odds are very good that at the point you run into problems you end up with more (and/or larger) Apache processes than your system memory can handle and you end up in a death spiral of swapping. There have been a number of threads in the past that address this issue, how you can start to break down your memory usage, and then adjustments you can make to work within your resources.

If you do nothing else, try dropping MaxClients significantly. You may slow down your peak request rate somewhat, but you should prevent the system from getting into such a bad state. Finding the best value is heavily application and load dependent (and also addressed more fully in past forum threads), but simply making that number much smaller is a quick 'n dirty first step to ensuring you stick within the resources you have available.

– David

thanks db3l

I've been beating the linode forum and other website threads..

trying this.. trying that.. I can't seem to find the right mix of configurations..

I really wish someone would look at my my.cnf / php.ini and apache2.conf and tell me it looks decent or if they see a glaring issue…

Overall the system seems to be big enough for my LAMP load and the SWAP issue seems to be random with no rhyme or reason. I'll push the maxclient down some more and see.. but all my tweaks seem to do NO help at all.

MaxClients 48

PHP Memory Limit 256M

48 x 256M = 12G of RAM!!!

There's no way you can fit all of that in your mid-sized Linode. The only reason your Linode works at all is because very few PHP scripts use anywhere near 256M of RAM. But even if each script used only 32M of RAM (typical for heavyweight CMSs), you'd run out of memory just the same. Throw in a couple of image processing add-ons, and you're going to hit the swap.

You need to do two things: cut down MaxClients to half of the current value (24), and cut down the memory limit to no more than a quarter of the current value (64M). That should give you a relatively safe starting point. Then enable the error log in php.ini. If some scripts require more RAM, errors will show up in the log file. Adjust the memory limit to make sure that all your scripts work, or get rid of add-ons that consume more memory than they're worth. Every time you need to increase the memory limit, decrease MaxClients to compensate. If your server works fine without hitting swap for a few weeks, try increasing MaxClients a little bit. It takes trial and error to find the sweet spot. (Good call on disabling KeepAlive, but giving it a very low value such as 1-2 wouldn't hurt, either.)

By the way, you have two keybuffer lines in your my.cnf. Delete the second one. While you're at it, try increasing keybuffer a bit, and see if it helps your pages load faster. Since your site seems to get a significant amount of traffic, consider transitioning to the InnoDB storage engine. But that's for the long term, and you'll need to budget a bit more RAM if you want to use InnoDB. Right now, you should focus on getting your memory usage under control.

@hybinet

Now this is what I needed to here.. thank you for helping tie this together for me.

Many thanks for your time.

So you want want me to remove the "key_buffer" located under [isamchk] ?? Will do.. just for my brain, why?

max_connections = 100

table_cache = 128

thread_concurrency = 10

key_buffer = 16M

maxallowedpacket = 16M

thread_stack = 192K

threadcachesize = 8

[isamchk]

key_buffer = 16M

@kutu:

So you want want me to remove the "keybuffer" located under [isamchk] ?? Will do.. just for my brain, why? Oh, that was under a different section? Then don't delete it. In your first post it looked like you had the same line twice in the same section. (By the way, the one that you should try adjusting is the keybuffer line in the [mysqld] section.)

Reply

Please enter an answer
Tips:

You can mention users to notify them: @username

You can use Markdown to format your question. For more examples see the Markdown Cheatsheet.

> I’m a blockquote.

I’m a blockquote.

[I'm a link] (https://www.google.com)

I'm a link

**I am bold** I am bold

*I am italicized* I am italicized

Community Code of Conduct