Apache slowing down / stalling

Hello All,

I've just switched to Apache2 due to fastcgi issueshowever, I'm finding Apache runs well for a while, then seems to stall. There's nothing showing up in the error log, and my SSH session is responsive, just loading the site takes ages.

I've noticed when the usage slows down the load seems reasonably high, around .5. Looking in htop it seems Apache is the main culprit for processor time.

I've tried a heap of different apache settings, but I'm really just guessing, I'd be very appreciative for any suggestions.

Ubuntu 8.04 LTS

Running Wordpress and SMF forum, around 4,000 visits per day.

Linode 360 with 540MB RAM (that was a stop gap to hope it stayed up during a promo we ran), considering upgrading to a Lindoe 540

> dpkg -l | grep apache

ii apache2 2.2.8-1ubuntu0.15 Next generation, scalable, extendable web se

ii apache2-mpm-prefork 2.2.8-1ubuntu0.15 Traditional model for Apache HTTPD

ii apache2-utils 2.2.8-1ubuntu0.15 utility programs for webservers

ii apache2.2-common 2.2.8-1ubuntu0.15 Next generation, scalable, extendable web se

ii libapache2-mod-php5 5.2.4-2ubuntu5.10 server-side, HTML-embedded scripting languag

rc libapache2-mod-rpaf 0.5-3 module for Apache2 which takes the last IP f

dpkg -l | grep php

ii libapache2-mod-php5 5.2.4-2ubuntu5.10 server-side, HTML-embedded scripting languag

ii php-pear 5.2.4-2ubuntu5.10 PEAR - PHP Extension and Application Reposit

ii php5 5.2.4-2ubuntu5.10 server-side, HTML-embedded scripting languag

ii php5-cli 5.2.4-2ubuntu5.10 command-line interpreter for the php5 script

ii php5-common 5.2.4-2ubuntu5.10 Common files for packages built from the php

ii php5-curl 5.2.4-2ubuntu5.10 CURL module for php5

ii php5-gd 5.2.4-2ubuntu5.10 GD module for php5

ii php5-mcrypt 5.2.3-0ubuntu1 MCrypt module for php5

ii php5-mysql 5.2.4-2ubuntu5.10 MySQL module for php5

ii php5-suhosin 0.9.22-1 advanced protection module for php5

ii php5-xcache 1.2.1-3 Fast, stable PHP opcode cacher

ii phpmyadmin 4:2.11.3-1ubuntu1.3 Administrate MySQL over the WWW
Extract from /etc/apache2/apache.conf
> # Timeout: The number of seconds before receives and sends time out.

#

Timeout 30

#

KeepAlive: Whether or not to allow persistent connections (more than

one request per connection). Set to "Off" to deactivate.

#

KeepAlive On

#

MaxKeepAliveRequests: The maximum number of requests to allow

during a persistent connection. Set to 0 to allow an unlimited amount.

We recommend you leave this number high, for maximum performance.

#

MaxKeepAliveRequests 50

#

KeepAliveTimeout: Number of seconds to wait for the next request from the

same client on the same connection.

#

KeepAliveTimeout 2

#

Server-Pool Size Regulation (MPM specific)

#

prefork MPM

StartServers: number of server processes to start

MinSpareServers: minimum number of server processes which are kept spare

MaxSpareServers: maximum number of server processes which are kept spare

MaxClients: maximum number of server processes allowed to start

MaxRequestsPerChild: maximum number of requests a server process serves

StartServers 3

MinSpareServers 3

MaxSpareServers 8

MaxClients 10

MaxRequestsPerChild 1000

Any suggestions for the next step? I was considering moving to the worker MPM, but now I'm thinking nginx to proxy static files??

Thanks,

Moses.

26 Replies

Here's some munin charts

~~![](<URL url=)http://img28.imageshack.us/img28/6646/inode.png" />

~~![](<URL url=)http://img97.imageshack.us/img97/7047/memoryu.png" />

~~![](<URL url=)http://img28.imageshack.us/img28/8288/postfix.png" />

~~![](<URL url=)http://img192.imageshack.us/img192/9936/netstat.png" />

~~![](<URL url=)http://img28.imageshack.us/img28/7382/mysqlh.png" />

![](" />~~

It's possible that you're hitting the MaxClients limit (10). If you are then Apache will log messages to the error log. Make sure you're checking the main Apache error log and not a domain-specific error log. MaxClient messages will appear in the main error log, probably in /etc/apache2/logs.

Thanks sleddog, didn't realise there was another error.log , this one is at /var/log/apache2/error.log .

I have been getting a few MaxClients warnings, though wont raising this eat into more RAM? I've lifted it to 15 for the moment, will see how that goes overnight. I guess going to nginx will mean less RAM for each apache process, hence more clients can run…

> top - 00:24:20 up 1 day, 1:05, 1 user, load average: 0.35, 0.18, 0.15

Tasks: 95 total, 3 running, 92 sleeping, 0 stopped, 0 zombie

Cpu(s): 7.0%us, 0.4%sy, 0.0%ni, 92.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Mem: 553184k total, 516256k used, 36928k free, 50916k buffers

Swap: 393208k total, 52k used, 393156k free, 288804k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

21144 mysql 15 0 73620 49m 4728 S 2 9.1 5:53.37 mysqld

12354 www-data 16 0 46524 24m 15m R 20 4.5 0:02.01 apache2

12296 www-data 15 0 46284 24m 15m S 0 4.5 0:02.40 apache2

12355 www-data 15 0 46904 24m 14m S 0 4.5 0:01.38 apache2

12285 www-data 15 0 46132 24m 15m S 0 4.5 0:03.16 apache2

11605 www-data 16 0 45620 23m 15m R 5 4.4 0:06.41 apache2

9432 www-data 15 0 44144 22m 15m S 0 4.2 0:18.23 apache2

11714 www-data 15 0 44324 22m 15m S 0 4.2 0:04.59 apache2

12348 www-data 15 0 44152 20m 13m S 0 3.8 0:01.12 apache2

27480 root 18 0 40476 7400 4268 S 0 1.3 0:00.04 apache2

2574 root 15 0 6816 4956 1544 S 0 0.9 0:00.20 munin-node

2685 root 18 0 23728 4488 1776 S 0 0.8 0:00.28 fail2ban-server

22795 moses 15 0 4424 2988 1252 S 0 0.5 0:00.09 bash

I also note there are a bunch of xcache errors in this log, that was cause I had the xcache settings in /etc/php5/apache2/conf.d/xcache.ini as well as /etc/php5/apache2/php.ini

As I was trying to load xcache twice, it took the first settings with just 16MB RAM. I've increased this to 64MB and am hoping this will take some load of apache and mysql??

> [Fri Apr 23 12:03:37 2010] [notice] Apache/2.2.8 (Ubuntu) PHP/5.2.4-2ubuntu5.10 with Suhosin-Patch configured – resuming normal operations

[Fri Apr 23 12:05:29 2010] [notice] caught SIGWINCH, shutting down gracefully

PHP Warning: Module 'XCache' already loaded in Unknown on line 0

[Fri Apr 23 12:05:31 2010] [notice] Apache/2.2.8 (Ubuntu) PHP/5.2.4-2ubuntu5.10 with Suhosin-Patch configured – resuming normal operations

[Fri Apr 23 12:05:36 2010] [error] server reached MaxClients setting, consider raising the MaxClients setting

[Fri Apr 23 12:16:23 2010] [notice] Graceful restart requested, doing restart

PHP Warning: Module 'XCache' already loaded in Unknown on line 0

[Fri Apr 23 12:16:23 2010] [notice] Apache/2.2.8 (Ubuntu) PHP/5.2.4-2ubuntu5.10 with Suhosin-Patch configured – resuming normal operations

[Fri Apr 23 21:14:37 2010] [notice] Graceful restart requested, doing restart

PHP Warning: Module 'XCache' already loaded in Unknown on line 0

[Fri Apr 23 21:14:37 2010] [notice] Apache/2.2.8 (Ubuntu) PHP/5.2.4-2ubuntu5.10 with Suhosin-Patch configured – resuming normal operations

[Fri Apr 23 21:51:39 2010] [notice] caught SIGWINCH, shutting down gracefully

PHP Warning: Module 'XCache' already loaded in Unknown on line 0

[Fri Apr 23 21:51:41 2010] [notice] Apache/2.2.8 (Ubuntu) PHP/5.2.4-2ubuntu5.10 with Suhosin-Patch configured – resuming normal operations

[Fri Apr 23 21:51:45 2010] [error] server reached MaxClients setting, consider raising the MaxClients setting

Thanks again for looking at this,

Moses.

I think you did the right thing by making a small increase in MaxClients. Just monitor the error log and your memory usage for a couple days to see how it goes. Every Apache process is not using an additional 15M of memory – a good part of that is shared memory between all Apache processes.

If you're still getting MaxClients errors try increasing the setting to 20…

Also, have you done any mysql optimization such as enable thread, query and table caches…?

I had made a few changes to mysql, current excerpt is

> # * Fine Tuning

#

key_buffer = 16M

maxallowedpacket = 1M

thread_stack = 64K

threadcachesize = 8

max_connections = 100

table_cache = 64

thread_concurrency = 10

#

* Query Cache Configuration

#

query-cache-type = 1

query-cache-size = 32M

querycachelimit = 1M

querycachesize = 20M

I'm thinking of removing some #'s and changing it to (will do some research first)
> # * Fine Tuning

#

key_buffer = 16M

maxallowedpacket = 1M

thread_stack = 64K

threadcachesize = 8

max_connections = 100

table_cache = 64

thread_concurrency = 10

#

* Query Cache Configuration

#

query-cache-type = 1

query-cache-size = 32M

querycachelimit = 1M

querycachesize = 20M

I re-enabled xcache yesterday afternoon, and this has led to a return of suhosin's canary errors in my site specific log

> [Sat Apr 24 10:27:04 2010] [error] [client 23.23.23.23] ALERT - canary mismatch on efree() - heap overflow detected (attacker '23.23.23.23', file '/home/public_html/greenandgoldrugby.com/public/index.php')

[Sat Apr 24 10:27:07 2010] [error] [client 23.23.23.23] ALERT - canary mismatch on efree() - heap overflow detected (attacker '23.23.23.23', file '/home/public_html/greenandgoldrugby.com/public/wp-admin/index.php')

[Sat Apr 24 10:27:07 2010] [error] [client 23.23.23.23] ALERT - canary mismatch on efree() - heap overflow detected (attacker '23.23.23.23', file '/home/public_html/greenandgoldrugby.com/public/wp-admin/index.php')

[Sat Apr 24 10:27:08 2010] [error] [client 23.23.23.23] ALERT - canary mismatch on efree() - heap overflow detected (attacker '23.23.23.23', file '/home/public_html/greenandgoldrugby.com/public/wp-admin/index.php')

[Sat Apr 24 10:27:08 2010] [error] [client 23.23.23.23] ALERT - canary mismatch on efree() - heap overflow detected (attacker '23.23.23.23', file '/home/public_html/greenandgoldrugby.com/public/wp-admin/index.php')

Read this

http://www.suspekt.org/2008/10/12/suhos … -detected/">http://www.suspekt.org/2008/10/12/suhosin-canary-mismatch-on-efree-heap-overflow-detected/

I seem to have this nagging feeling somewhere that xcache and suhosin don't agree with each other but for the life of me I can't remember why.

Have you tried APC?

G'Day obs,

Thanks for that link, I'm pretty sure I've read it before, though just took the advice of it's last comment at set

suhosin.session.encrypt=Off

I had a look at APC but didn't have a lot of luck installing it, got sort of confused around pecl etc.. then xcache was a simple apt-get install so went with that

apt-get install php-apc

No need for pecl

Although if you do want to use pecl it's

pecl install apc

if you're using php 5.3 it's

pecl install apc-beta

just make sure you add extension=apc.so to your php.ini

Not sure if this has been tried already (I've got slow intertubes right now), but try setting KeepAlives off, or set MaxKeepAliveRequests down to a low number, significantly low than MaxClients; otherwise, you'll have a lot of idle connections just sitting there waiting for time to pass.

@sleddog:

It's possible that you're hitting the MaxClients limit (10). If you are then Apache will log messages to the error log. Make sure you're checking the main Apache error log and not a domain-specific error log. MaxClient messages will appear in the main error log, probably in /etc/apache2/logs.
I noticed that I was hitting the MaxClient at 15 as well, so upped it to 20. Then I went into swap memory so dropped it to 18. Seems better there, but hit swap memory again later today so I've reduced MaxRequestsPerChild from 1000 to 250, hopefully this will keep the 18 apache processes growing too large before respawning.

I'll add the nginx proxy this weekend and that should help a lot to keep apache in check, will go to APC at the same time..

seems the apc package is not in the repository for 8.04 LTS

I tried with pecl however that failed, so ended up installing 3.1.13.p1 from source. Seems pretty good and I really like the stats page it gives

Ah didn't check the 8.04 since I use 9.10, but yeah it's a nice add-on and for php 5.3 you need 3.1 which is "beta" but I've had no problems with it.

You might want to look at upgrading to 10.4 LTS in a month or so.

Well after getting APC to install once on the dev site I'm clueless how to get it working again!

I can install it from source, but when I reload apache get an error about

> PHP Warning: PHP Startup: apc: Unable to initialize module\nModule compiled with module API=20090626, debug=0, thread-safety=0\nPHP compiled with module API=20060613, debug=0, thread-safety=0\nThese options need to match\n in Unknown on line 0

I believe the error is coming from conflicting PHP versions?

My PHP Extension is 20060613

and the release of APC i'm trying to install (3.1.3p1) is 20090626

Any suggestions?

What version of php are you using? Where did you get it from compile or apt? How are you installing apc?

Thanks for the response obs, i'm pretty sure I installed PHP from "apt-get install php5"

> dpkg -l | grep php

ii libapache2-mod-php5 5.2.4-2ubuntu5.10 server-side, HTML-embedded scripting languag

ii php-pear 5.2.4-2ubuntu5.10 PEAR - PHP Extension and Application Reposit

ii php5 5.2.4-2ubuntu5.10 server-side, HTML-embedded scripting languag

ii php5-cli 5.2.4-2ubuntu5.10 command-line interpreter for the php5 script

ii php5-common 5.2.4-2ubuntu5.10 Common files for packages built from the php

ii php5-curl 5.2.4-2ubuntu5.10 CURL module for php5

ii php5-dev 5.2.4-2ubuntu5.10 Files for PHP5 module development

ii php5-gd 5.2.4-2ubuntu5.10 GD module for php5

ii php5-mcrypt 5.2.3-0ubuntu1 MCrypt module for php5

ii php5-mysql 5.2.4-2ubuntu5.10 MySQL module for php5

ii php5-suhosin 0.9.22-1 advanced protection module for php5

ii phpmyadmin 4:2.11.3-1ubuntu1.3 Administrate MySQL over the WWW

> cd /home/source

mkdir apc

cd apc

wget http://pecl.php.net/get/APC-3.1.3p1.tgz

tar -xzvf APC-3.1.3p1.tgz

cd APC-3.1.3p1

phpize

./configure –enable-apc --enable-apc-mmap --with-apxs --with-php-config=/usr/local/bin/php-config

make

make test

make install

echo extension="apc.so" >> /etc/php5/apache2/php.ini

cp apc.php /home/public_html/greenandgoldrugby.com/public/

cp /home/source/apc/APC-3.1.3p1/modules/apc.so /usr/lib/php5/20060613+lfs/apc.so

/etc/init.d/apache2 restart

vi /etc/php5/apache2/conf.d/suhosin.ini

hrmm try this

pear channel-update pecl.php.net

pecl install apc

That "should" work. tbh I'm not exactly sure why your method didn't work since the error says the php version apc is compiled against is different to your php version but you have the same version of php and php-dev.

I went back to APC 3.0.19 and that works, happy days ;)

Should I increase the RAM for wordpress / vbulletin running on a 360?

Nah you should be fine, make sure you use wp super cache, I don't really use forums/bulletin board software since they tend to be full of holes so I can't advise on that part.

Thanks obs, I'm thinking of removing suhosin as I've still had no luck with APC… it's worked twice on the dev site now, and failed about 40 times!

I noticed I'm still hitting maxclients from time to time so have lifted this to 30. Figure I'm not running out of memory, but find it confusing how top shows memory usage as a %.. makes it tough to know how many MB each process is using.

I also get an occasional error where the server prompts one of the other admins to download a .php file rather than serving it. Any thoughts on what could

Hey did you ever get anywhere with this? I'm using Ubuntu 8.04 and trying to install APC with no luck. I can't get it to find the application with the normal get-apt command (or apt-get, whatever).

Didn't want to go to source since that didn't work for you either.

THANKS.

http://lmgtfy.com/?q=ubuntu+8.04+apc

@theatereleven:

Hey did you ever get anywhere with this? I'm using Ubuntu 8.04 and trying to install APC with no luck. I can't get it to find the application with the normal get-apt command (or apt-get, whatever).

Didn't want to go to source since that didn't work for you either.

THANKS.
I did get it going in the end, through a lot of perseverance and testing on a second linode. The latest beta of APC, combined with a switch to nginx for front end proxying has been fantastic and my site has now enjoyed an unprecedented 36 days uptime (touch wood).

Cool thanks! Was the switch to Nigx difficult? I tried the link from the forum member above but it still doesn't work installing APC. Says this:

ERROR: '/tmp/pear/cache.APC-3.0.19/configure –with-apxs' failed

Whatever that means. =(

You might want to Google "Drupal Nginx" before going down that path. Nginx is NOT a drop in replacement for Apache.

Okay thanks - good to know. If it is going to be a hassle there is probably more support out there for Apache so it just will not be worth it.

Reply

Please enter an answer
Tips:

You can mention users to notify them: @username

You can use Markdown to format your question. For more examples see the Markdown Cheatsheet.

> I’m a blockquote.

I’m a blockquote.

[I'm a link] (https://www.google.com)

I'm a link

**I am bold** I am bold

*I am italicized* I am italicized

Community Code of Conduct