CPU spikes crashing my server several times per week

odds are you've got your maxclients set too high in /etc/httpd/conf/httpd.conf

step 1, in httpd.conf, turn off keepalives

step 2, work out how much each apache process is using with ps aux, say it's 50 megs, for example

2000/50 = 40, so set maxclients and serverlimit to 30, to leave room for mysql.

monitor apache logs for warnings about maxclients reached, if you see them, you either need to

tune your php app or database to respond quicker, upgrade memory or even put nginx in front of

apache to serve static content.

Thanks for the reply. It looks like my max clients is too high. If I run ps aux | grep apache I get the following:

apache   10718  8.0  1.9  65848 41796 ?        S    13:56   0:43 /usr/sbin/httpd
apache   10731  5.7  0.7  39088 15256 ?        S    13:57   0:24 /usr/sbin/httpd
apache   10747  6.1  0.6  36624 12768 ?        S    13:59   0:20 /usr/sbin/httpd
apache   10765  8.1  0.6  37388 13524 ?        S    13:59   0:26 /usr/sbin/httpd
apache   10767  6.5  0.8  42276 18536 ?        S    13:59   0:21 /usr/sbin/httpd
apache   10770  4.2  1.4  54832 31276 ?        S    13:59   0:14 /usr/sbin/httpd
apache   10817  6.9  0.9  42780 18920 ?        S    14:01   0:15 /usr/sbin/httpd
apache   10820  7.6  0.6  36784 12912 ?        S    14:01   0:16 /usr/sbin/httpd
apache   10821  6.0  0.7  39376 15468 ?        S    14:01   0:13 /usr/sbin/httpd
apache   10822  8.3  1.4  55784 31460 ?        S    14:01   0:18 /usr/sbin/httpd
apache   10827  9.5  0.7  39712 15612 ?        S    14:01   0:20 /usr/sbin/httpd
apache   10838  4.8  0.6  36784 12904 ?        S    14:02   0:08 /usr/sbin/httpd
apache   10842  8.2  0.6  36768 12844 ?        S    14:02   0:11 /usr/sbin/httpd
apache   10844  6.6  1.3  51884 27624 ?        S    14:03   0:08 /usr/sbin/httpd
apache   10848  9.0  0.6  37076 12952 ?        S    14:03   0:06 /usr/sbin/httpd
apache   10850  7.0  0.6  36784 12872 ?        S    14:04   0:04 /usr/sbin/httpd
apache   10852  3.8  0.6  36784 12776 ?        S    14:04   0:02 /usr/sbin/httpd
apache   10854  7.7  0.6  36768 12824 ?        S    14:04   0:04 /usr/sbin/httpd
apache   10855  4.7  0.0      0     0 ?        Z    14:04   0:02 [httpd] <defunct>apache   10858  5.4  0.6  37076 12956 ?        S    14:04   0:03 /usr/sbin/httpd
apache   10859  8.2  0.6  36792 12844 ?        S    14:04   0:04 /usr/sbin/httpd
apache   10861  6.9  0.6  37324 13372 ?        S    14:04   0:03 /usr/sbin/httpd
apache   10864  0.0  0.2  32348  5856 ?        S    14:04   0:00 /usr/sbin/httpd
apache   10865  8.7  0.8  42760 18852 ?        S    14:04   0:04 /usr/sbin/httpd
apache   10882  5.6  0.6  37068 12784 ?        S    14:04   0:00 /usr/sbin/httpd
apache   10884  7.7  0.5  36776 12524 ?        S    14:04   0:00 /usr/sbin/httpd
apache   10885  6.4  0.6  37068 12788 ?        S    14:04   0:00 /usr/sbin/httpd
apache   10888  9.7  0.5  36784 12556 ?        S    14:04   0:00 /usr/sbin/httpd</defunct> 

If I average that out, I get about 16.5 MB per process. with 2GB, that gives me about 117 Max Clients that I can have, it is currently set for 180. So I am going to try reducing the Max Clients to 100, and see if that does the trick.

Does that seem like the right thing to do here?

Unless you ONLY have Apache running, more like Max Clients = 80 and adjust up or down from there.

the thing to note also is setting maxclients to the right setting won't magically make a server handle more users,

it will stop the server from falling over during high load, but if you have more visitors than maxclients, some

will have to wait for a free process.

setting keepalives off will help it handle more users at once, also tuning mysql (with mysqltuner.pl) will also help it handle

more concurrent users. It might sound backwards, but letting mysql use more ram will usually allow the server to handle more users.

So, keep an eye on the logs for the warning about increasing maxclients, also watch your swap usage, during high load you

don't want it swapping continuously, that will slow it down a lot. It's better to have a lower maxclients and no active

swapping than a higher maxclients and swapping. Having stuff in swap isn't a bad thing, but if you're watching swap and it's

constantly changing, that's bad. you can watch swap with "free" or "vmstat 1"

Thanks for all the info so far, it's been really helpful. I have updates apache and am just watching the logs at this point. I will keep you guys posted.

So I've dropped my max clients to 80, but I am getting this error now:

[error] server reached MaxClients setting, consider raising the MaxClients setting

So what does that mean? My server is still crashing. Sometimes apache just crashes, other time I can't even SSH into my VPS anymore and I need to restart it.

That is just a notice that you are getting enough visitors to hit the MaxClients setting. Nothing major to worry about, as they will wait in the TCP queue until a process becomes free to serve their request. It's only a problem if users have to wait an excessive time; then they will start complaining. In that case, you'll either need to set up a higher-performing system or upgrade to a larger Linode.

Also take a look at how much of your traffic is from robots and consider adding a robots.txt file to your site(s).