Strange Apache issue

@Eidolon Night:

(except an entry saying that the max clients has been reached).

Trying increasing it a little, and see if it spreads the outages some. Look in your access logs and see if you are getting a lot of traffic from a single IP.

What does the memory usage look like when it happens? Are you hitting swap? (Install Munin to find out.)

Max clients is currently at 25. I increased it all the way to 50, but the problem still occurs, randomly as always, and generally at least once per day. memory usage is spiking along with the CPU. As I mentioned, my disk IO takes quite a leap. The strange thing is that despite the memory and CPU usage given by top, I'm able to easily and quickly log into the server and issue an Apache restart command. The server seems responsive and is by no means slow, it's only Apache that is going crazy.

I've installed the DDoS-Deflate thinking that this may be an attack, but the fact that I can simply restart Apache and everything is fine, makes that unlikely.

UPDATE: Wanted to add that I have the Xen swappiness set to 25 as recommended by the Linode wiki.

UPDATE 2: pparadis recommended switching the kernel config to 2.6.28.3-linode17 (which I've done) so I'll give the server 48 hours and post the results here. Thanks again for everyone's help with this.

Nope, problem not solved. Any more ideas?

@hybinet:

Install Munin
Have you done this ?

If you're intent on sticking with Apache, have a look at Caker's config partway down on http://www.linode.com/forums/viewtopic.php?t=4460. Personally, I'd think seriously about switching to Nginx or Lighttpd.

Are you sure that you've tightened up Mysql sufficiently ?

50 max clients with Apache/mod_php on a Linode 720 is unsustainable. Each request takes up ~13MB of memory, so with 50 simultaneous requests you need ~650MB of memory for Apache alone. No wonder your server goes OOM and starts eating up swap (hence the high disk IO).

1. Try setting max clients to something low, such as 15-20.

2. If your site is the only one on this linode, ditch Apache and start using something a little more VPS-friendly, such as lighttpd or nginx. There are plenty of tutorials on the web to help you make the transition, especially with something as popular as Drupal.

Thanks for the help so far everyone. I have used lighttpd in the past, but I much prefer Apache and some of the Drupal modules I use require Apache (unless I want to do a good deal of hackery, which I do not).

My max clients is usually set at 25, I only set it higher for testing purposes. I've done quite a bit of reading over at 2bits and came to use the following settings. Everything was fine until a few weeks ago. In fact, everything is still fine. The site performs wonderfully up until the crashes. The crashes also do not coincide with peak traffic. So it's not like I'm getting traffic spikes that cause this.

MaxKeepAliveRequests 100

KeepAliveTimeout 5

StartServers 2

MinSpareServers 2

MaxSpareServers 5

MaxClients 25

MaxRequestsPerChild 4000

I've also installed the latest eaccelerator which helps a lot with memory usage.

MySQL isn't an issue. It's memory usage remains normal, and restarting only Apache fixes the issue.

I have installed Munin today.

@Eidolon Night:

Thanks for the help so far everyone. I have used lighttpd in the past, but I much prefer Apache and some of the Drupal modules I use require Apache (unless I want to do a good deal of hackery, which I do not).
You can still both have Lighttpd/Nginx and Apache installed, and let the static files served by the first and proxy php to Apache. At least you static file requests won't eat up as much memory as with Apache.

With the proxying mode You can even set a prepend file in php which eliminates some HTTP headers so it can be transparent.

Also try disabling unnecessary Apache modules, those might add memory usage for each Apache handled request.

BTW what Drupal modules do you use that require Apache? I am using Lighttpd for one of my Drupal sites served by Lighty and FastCGI and I had no problems so far.

@melon:

@Eidolon Night:

Thanks for the help so far everyone. I have used lighttpd in the past, but I much prefer Apache and some of the Drupal modules I use require Apache (unless I want to do a good deal of hackery, which I do not).
You can still both have Lighttpd/Nginx and Apache installed, and let the static files served by the first and proxy php to Apache. At least you static file requests won't eat up as much memory as with Apache.

With the proxying mode You can even set a prepend file in php which eliminates some HTTP headers so it can be transparent.

Also try disabling unnecessary Apache modules, those might add memory usage for each Apache handled request.

BTW what Drupal modules do you use that require Apache? I am using Lighttpd for one of my Drupal sites served by Lighty and FastCGI and I had no problems so far.

I honestly don't remember which modules, but I was running lighttpd for awhile ( a few months ago) and the reason I switched to Apache was because of a couple modules. Going through the trouble of splitting up static pages would do little good because the majority of my traffic is from logged-in users.

As I said before, I have zero issues 99.999% of the time. Apache performs well, and the server runs fine. Then out of nowhere, CPU usage hits 100% (according to top) but the server continues to respond like normal. I mean, everything except Apache responds quickly, and doesn't behave like a server under max CPU load.

A simple Apache restart instantly solves the problem. So, I guess what I'm saying is that I would much rather "fix" the problem and the server running the way it was before than run away from it.

Here is the link to my Munin results. I'll post again when there's another crash: http://aionelysium.com/munin/localdomai … omain.html">http://aionelysium.com/munin/localdomain/localhost.localdomain.html

@Eidolon Night:

I honestly don't remember which modules, but I was running lighttpd for awhile ( a few months ago) and the reason I switched to Apache was because of a couple modules. Going through the trouble of splitting up static pages would do little good because the majority of my traffic is from logged-in users.
Even if traffic comes from logged-in users, a simple request to a css or image file involves a php-powered apache process costing around 10-30 MB of memory which might be overkill.

Try to check your access and error logs during CPU spikes.

Also enable munin plugins which gets info from apache /server-status pages to see apache processes and volume. (and of course enable server-status from 127.0.0.1 in apache.)

Your problem might be caused by a drupal module as well. Have you checked your drupal logs about any errors…

@melon:

@Eidolon Night:

I honestly don't remember which modules, but I was running lighttpd for awhile ( a few months ago) and the reason I switched to Apache was because of a couple modules. Going through the trouble of splitting up static pages would do little good because the majority of my traffic is from logged-in users.
Even if traffic comes from logged-in users, a simple request to a css or image file involves a php-powered apache process costing around 10-30 MB of memory which might be overkill.

Try to check your access and error logs during CPU spikes.

Also enable munin plugins which gets info from apache /server-status pages to see apache processes and volume. (and of course enable server-status from 127.0.0.1 in apache.)

Your problem might be caused by a drupal module as well. Have you checked your drupal logs about any errors…

Please read my original post. No entries in the Apache error log and nothing in the Drupal logs as well. I'll look into the munin plugins tonight. The server generally runs with a good amount of free memory.

total used free shared buffers cached

Mem: 705 544 160 0 20 225

-/+ buffers/cache: 297 407

Swap: 511 4 507

AND as I said, the crashes do not coincide with peak traffic times. Out of the blue apache simply goes crazy. I'm also confused about the CPU usage being at 1005 (or 400% according to Linode charts). I log and can move about the server with no delay. It certainly doesn't behave like a server that's under full load. In fact, I'm also running TeamSpeak on the server, and it continues to run fine the entire time.

My bad I haven't read carefully your original post, sorry.

Let's find an another spike and by then you should be able to see what apache did via the munin plugins.

@melon:

My bad I haven't read carefully your original post, sorry.

Let's find an another spike and by then you should be able to see what apache did via the munin plugins.

Just got another crash: http://aionelysium.com/munin/localdomai … omain.html">http://aionelysium.com/munin/localdomain/localhost.localdomain.html

Can't miss the massive CPU spike. I see that MySQL spiked as well, so I'll take a look at that. (http://aionelysium.com/munin/localdomai … eries.html">http://aionelysium.com/munin/localdomain/localhost.localdomain-mysql_slowqueries.html)

BTW I installed munin-plugins-extra package, but do not see anything particular to apache.

UPDATE: Nothing in the MySQL logs. :(

UPDATE 2: OK, had to manually enable the apache plugins. Done.

@Eidolon Night:

UPDATE 2: OK, had to manually enable the apache plugins. Done.

Apache plugins need access to http://localhost/server-status?auto otherwise you won't see any data. It uses the ExtendedStatus feature of Apache.

@melon:

@Eidolon Night:

UPDATE 2: OK, had to manually enable the apache plugins. Done.

Apache plugins need access to http://localhost/server-status?auto otherwise you won't see any data. It uses the ExtendedStatus feature of Apache.

In mods-enabled I have status.conf which contains:

ExtendedStatus On

SetHandler server-status

Order deny,allow

Deny from all

Allow from localhost ip6-localhost

Allow from .aionelysium.com

BUT it doesn't seem to to be working. Am I missing something?

Found this from http://ubuntuforums.org/showthread.php?t=1044117

@Eidolon Night:

In mods-enabled I have status.conf which contains:

ExtendedStatus On

SetHandler server-status

Order deny,allow

Deny from all

Allow from localhost ip6-localhost

Allow from .aionelysium.com

BUT it doesn't seem to to be working. Am I missing something?

Found this from http://ubuntuforums.org/showthread.php?t=1044117

By default the Apache Munin plugins will try to connect to http://127.0.0.1:80. In my case I only have Apache listening on the external IP, and as I wanted to keep that Apache config, I've added this to my Munin config:

[apache*]
env.url http://my.domain-name.com/server-status?auto
env.ports 80 443

Tried the above, but I still get "no apache server-status on ports 80" when attempting to enable the apache plugins.

@hybinet:

50 max clients with Apache/mod_php on a Linode 720 is unsustainable.
This does not correspond to my experience. One Linode 360 I manage runs Ubuntu 8.04 and Apache 2.2.8 just fine with MaxClients set to 50 (and hits that max several times a week). It uses PunBB, which is lightweight compared to, say, Drupal or Wordpress.

Another running CentOS 5.3 and Apache 2.2.3 uses MediaWiki and can only handle a MaxClients of 25 before hitting swap. For whatever reason, Apache seems less memory-efficient on CentOS than on Ubuntu, but this server also runs mailing lists and other services that consume RAM.

> By default the Apache Munin plugins will try to connect to http://127.0.0.1:80. In my case I only have Apache listening on the external IP[…]

You can have apache listen on both IPs. Have apache itself listen on localhost, and have your vhost listen on the external IP.

I'm having the same issues with my linode server running drupal on apache. Were you able to solve the problem? I'm getting spikes about once a day and it's usually in the middle of the night when there is almost no traffic on the site.

Thanks, Tim