Apache slowing down / stalling
I've just switched to Apache2 due to fastcgi issueshowever, I'm finding Apache runs well for a while, then seems to stall. There's nothing showing up in the error log, and my SSH session is responsive, just loading the site takes ages.
I've noticed when the usage slows down the load seems reasonably high, around .5. Looking in htop it seems Apache is the main culprit for processor time.
I've tried a heap of different apache settings, but I'm really just guessing, I'd be very appreciative for any suggestions.
Ubuntu 8.04 LTS
Running Wordpress and SMF forum, around 4,000 visits per day.
Linode 360 with 540MB RAM (that was a stop gap to hope it stayed up during a promo we ran), considering upgrading to a Lindoe 540
> dpkg -l | grep apache
ii apache2 2.2.8-1ubuntu0.15 Next generation, scalable, extendable web se
ii apache2-mpm-prefork 2.2.8-1ubuntu0.15 Traditional model for Apache HTTPD
ii apache2-utils 2.2.8-1ubuntu0.15 utility programs for webservers
ii apache2.2-common 2.2.8-1ubuntu0.15 Next generation, scalable, extendable web se
ii libapache2-mod-php5 5.2.4-2ubuntu5.10 server-side, HTML-embedded scripting languag
rc libapache2-mod-rpaf 0.5-3 module for Apache2 which takes the last IP f
dpkg -l | grep php
ii libapache2-mod-php5 5.2.4-2ubuntu5.10 server-side, HTML-embedded scripting languag
ii php-pear 5.2.4-2ubuntu5.10 PEAR - PHP Extension and Application Reposit
ii php5 5.2.4-2ubuntu5.10 server-side, HTML-embedded scripting languag
ii php5-cli 5.2.4-2ubuntu5.10 command-line interpreter for the php5 script
ii php5-common 5.2.4-2ubuntu5.10 Common files for packages built from the php
ii php5-curl 5.2.4-2ubuntu5.10 CURL module for php5
ii php5-gd 5.2.4-2ubuntu5.10 GD module for php5
ii php5-mcrypt 5.2.3-0ubuntu1 MCrypt module for php5
ii php5-mysql 5.2.4-2ubuntu5.10 MySQL module for php5
ii php5-suhosin 0.9.22-1 advanced protection module for php5
ii php5-xcache 1.2.1-3 Fast, stable PHP opcode cacher
ii phpmyadmin 4:2.11.3-1ubuntu1.3 Administrate MySQL over the WWW
Extract from /etc/apache2/apache.conf
> # Timeout: The number of seconds before receives and sends time out.#
Timeout 30
#
KeepAlive: Whether or not to allow persistent connections (more than
one request per connection). Set to "Off" to deactivate.
#
KeepAlive On
#
MaxKeepAliveRequests: The maximum number of requests to allow
during a persistent connection. Set to 0 to allow an unlimited amount.
We recommend you leave this number high, for maximum performance.
#
MaxKeepAliveRequests 50
#
KeepAliveTimeout: Number of seconds to wait for the next request from the
same client on the same connection.
#
KeepAliveTimeout 2
#
Server-Pool Size Regulation (MPM specific)
#
prefork MPM
StartServers: number of server processes to start
MinSpareServers: minimum number of server processes which are kept spare
MaxSpareServers: maximum number of server processes which are kept spare
MaxClients: maximum number of server processes allowed to start
MaxRequestsPerChild: maximum number of requests a server process serves
StartServers 3 MinSpareServers 3
MaxSpareServers 8
MaxClients 10
MaxRequestsPerChild 1000
Any suggestions for the next step? I was considering moving to the worker MPM, but now I'm thinking nginx to proxy static files??
Thanks,
Moses.
26 Replies
~~![](<URL url=)http://img28.imageshack.us/img28/6646/inode.png
~~![](<URL url=)http://img97.imageshack.us/img97/7047/memoryu.png
~~![](<URL url=)http://img28.imageshack.us/img28/8288/postfix.png
~~![](<URL url=)http://img192.imageshack.us/img192/9936/netstat.png
~~![](<URL url=)http://img28.imageshack.us/img28/7382/mysqlh.png
![](~~
I have been getting a few MaxClients warnings, though wont raising this eat into more RAM? I've lifted it to 15 for the moment, will see how that goes overnight. I guess going to nginx will mean less RAM for each apache process, hence more clients can run…
> top - 00:24:20 up 1 day, 1:05, 1 user, load average: 0.35, 0.18, 0.15
Tasks: 95 total, 3 running, 92 sleeping, 0 stopped, 0 zombie
Cpu(s): 7.0%us, 0.4%sy, 0.0%ni, 92.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 553184k total, 516256k used, 36928k free, 50916k buffers
Swap: 393208k total, 52k used, 393156k free, 288804k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
21144 mysql 15 0 73620 49m 4728 S 2 9.1 5:53.37 mysqld
12354 www-data 16 0 46524 24m 15m R 20 4.5 0:02.01 apache2
12296 www-data 15 0 46284 24m 15m S 0 4.5 0:02.40 apache2
12355 www-data 15 0 46904 24m 14m S 0 4.5 0:01.38 apache2
12285 www-data 15 0 46132 24m 15m S 0 4.5 0:03.16 apache2
11605 www-data 16 0 45620 23m 15m R 5 4.4 0:06.41 apache2
9432 www-data 15 0 44144 22m 15m S 0 4.2 0:18.23 apache2
11714 www-data 15 0 44324 22m 15m S 0 4.2 0:04.59 apache2
12348 www-data 15 0 44152 20m 13m S 0 3.8 0:01.12 apache2
27480 root 18 0 40476 7400 4268 S 0 1.3 0:00.04 apache2
2574 root 15 0 6816 4956 1544 S 0 0.9 0:00.20 munin-node
2685 root 18 0 23728 4488 1776 S 0 0.8 0:00.28 fail2ban-server
22795 moses 15 0 4424 2988 1252 S 0 0.5 0:00.09 bash
I also note there are a bunch of xcache errors in this log, that was cause I had the xcache settings in /etc/php5/apache2/conf.d/xcache.ini as well as /etc/php5/apache2/php.ini
As I was trying to load xcache twice, it took the first settings with just 16MB RAM. I've increased this to 64MB and am hoping this will take some load of apache and mysql??
> [Fri Apr 23 12:03:37 2010] [notice] Apache/2.2.8 (Ubuntu) PHP/5.2.4-2ubuntu5.10 with Suhosin-Patch configured – resuming normal operations
[Fri Apr 23 12:05:29 2010] [notice] caught SIGWINCH, shutting down gracefully
PHP Warning: Module 'XCache' already loaded in Unknown on line 0
[Fri Apr 23 12:05:31 2010] [notice] Apache/2.2.8 (Ubuntu) PHP/5.2.4-2ubuntu5.10 with Suhosin-Patch configured – resuming normal operations
[Fri Apr 23 12:05:36 2010] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Fri Apr 23 12:16:23 2010] [notice] Graceful restart requested, doing restart
PHP Warning: Module 'XCache' already loaded in Unknown on line 0
[Fri Apr 23 12:16:23 2010] [notice] Apache/2.2.8 (Ubuntu) PHP/5.2.4-2ubuntu5.10 with Suhosin-Patch configured – resuming normal operations
[Fri Apr 23 21:14:37 2010] [notice] Graceful restart requested, doing restart
PHP Warning: Module 'XCache' already loaded in Unknown on line 0
[Fri Apr 23 21:14:37 2010] [notice] Apache/2.2.8 (Ubuntu) PHP/5.2.4-2ubuntu5.10 with Suhosin-Patch configured – resuming normal operations
[Fri Apr 23 21:51:39 2010] [notice] caught SIGWINCH, shutting down gracefully
PHP Warning: Module 'XCache' already loaded in Unknown on line 0
[Fri Apr 23 21:51:41 2010] [notice] Apache/2.2.8 (Ubuntu) PHP/5.2.4-2ubuntu5.10 with Suhosin-Patch configured – resuming normal operations
[Fri Apr 23 21:51:45 2010] [error] server reached MaxClients setting, consider raising the MaxClients setting
Thanks again for looking at this,
Moses.
If you're still getting MaxClients errors try increasing the setting to 20…
Also, have you done any mysql optimization such as enable thread, query and table caches…?
> # * Fine Tuning
#
key_buffer = 16M
maxallowedpacket = 1M
thread_stack = 64K
threadcachesize = 8
max_connections = 100
table_cache = 64
thread_concurrency = 10
#
* Query Cache Configuration
#
query-cache-type = 1
query-cache-size = 32M
querycachelimit = 1M
querycachesize = 20M
I'm thinking of removing some #'s and changing it to (will do some research first)
> # * Fine Tuning
#
key_buffer = 16M
maxallowedpacket = 1M
thread_stack = 64K
threadcachesize = 8
max_connections = 100
table_cache = 64
thread_concurrency = 10
#
* Query Cache Configuration
#
query-cache-type = 1
query-cache-size = 32M
querycachelimit = 1M
querycachesize = 20M
> [Sat Apr 24 10:27:04 2010] [error] [client 23.23.23.23] ALERT - canary mismatch on efree() - heap overflow detected (attacker '23.23.23.23', file '/home/public_html/greenandgoldrugby.com/public/index.php')
[Sat Apr 24 10:27:07 2010] [error] [client 23.23.23.23] ALERT - canary mismatch on efree() - heap overflow detected (attacker '23.23.23.23', file '/home/public_html/greenandgoldrugby.com/public/wp-admin/index.php')
[Sat Apr 24 10:27:07 2010] [error] [client 23.23.23.23] ALERT - canary mismatch on efree() - heap overflow detected (attacker '23.23.23.23', file '/home/public_html/greenandgoldrugby.com/public/wp-admin/index.php')
[Sat Apr 24 10:27:08 2010] [error] [client 23.23.23.23] ALERT - canary mismatch on efree() - heap overflow detected (attacker '23.23.23.23', file '/home/public_html/greenandgoldrugby.com/public/wp-admin/index.php')
[Sat Apr 24 10:27:08 2010] [error] [client 23.23.23.23] ALERT - canary mismatch on efree() - heap overflow detected (attacker '23.23.23.23', file '/home/public_html/greenandgoldrugby.com/public/wp-admin/index.php')
I seem to have this nagging feeling somewhere that xcache and suhosin don't agree with each other but for the life of me I can't remember why.
Have you tried APC?
Thanks for that link, I'm pretty sure I've read it before, though just took the advice of it's last comment at set
suhosin.session.encrypt=Off
I had a look at APC but didn't have a lot of luck installing it, got sort of confused around pecl etc.. then xcache was a simple apt-get install so went with that
No need for pecl
Although if you do want to use pecl it's
pecl install apc
if you're using php 5.3 it's
pecl install apc-beta
just make sure you add extension=apc.so to your php.ini
@sleddog:
It's possible that you're hitting the MaxClients limit (10). If you are then Apache will log messages to the error log. Make sure you're checking the main Apache error log and not a domain-specific error log. MaxClient messages will appear in the main error log, probably in /etc/apache2/logs.
I noticed that I was hitting the MaxClient at 15 as well, so upped it to 20. Then I went into swap memory so dropped it to 18. Seems better there, but hit swap memory again later today so I've reduced MaxRequestsPerChild from 1000 to 250, hopefully this will keep the 18 apache processes growing too large before respawning.
I'll add the nginx proxy this weekend and that should help a lot to keep apache in check, will go to APC at the same time..
I tried with pecl however that failed, so ended up installing 3.1.13.p1 from source. Seems pretty good and I really like the stats page it gives
You might want to look at upgrading to 10.4 LTS in a month or so.
I can install it from source, but when I reload apache get an error about
> PHP Warning: PHP Startup: apc: Unable to initialize module\nModule compiled with module API=20090626, debug=0, thread-safety=0\nPHP compiled with module API=20060613, debug=0, thread-safety=0\nThese options need to match\n in Unknown on line 0
I believe the error is coming from conflicting PHP versions?
My PHP Extension is 20060613
and the release of APC i'm trying to install (3.1.3p1) is 20090626
Any suggestions?
> dpkg -l | grep php
ii libapache2-mod-php5 5.2.4-2ubuntu5.10 server-side, HTML-embedded scripting languag
ii php-pear 5.2.4-2ubuntu5.10 PEAR - PHP Extension and Application Reposit
ii php5 5.2.4-2ubuntu5.10 server-side, HTML-embedded scripting languag
ii php5-cli 5.2.4-2ubuntu5.10 command-line interpreter for the php5 script
ii php5-common 5.2.4-2ubuntu5.10 Common files for packages built from the php
ii php5-curl 5.2.4-2ubuntu5.10 CURL module for php5
ii php5-dev 5.2.4-2ubuntu5.10 Files for PHP5 module development
ii php5-gd 5.2.4-2ubuntu5.10 GD module for php5
ii php5-mcrypt 5.2.3-0ubuntu1 MCrypt module for php5
ii php5-mysql 5.2.4-2ubuntu5.10 MySQL module for php5
ii php5-suhosin 0.9.22-1 advanced protection module for php5
ii phpmyadmin 4:2.11.3-1ubuntu1.3 Administrate MySQL over the WWW
> cd /home/source
mkdir apc
cd apc
wget
http://pecl.php.net/get/APC-3.1.3p1.tgz tar -xzvf APC-3.1.3p1.tgz
cd APC-3.1.3p1
phpize
./configure –enable-apc --enable-apc-mmap --with-apxs --with-php-config=/usr/local/bin/php-config
make
make test
make install
echo extension="apc.so" >> /etc/php5/apache2/php.ini
cp apc.php /home/public_html/greenandgoldrugby.com/public/
cp /home/source/apc/APC-3.1.3p1/modules/apc.so /usr/lib/php5/20060613+lfs/apc.so
/etc/init.d/apache2 restart
vi /etc/php5/apache2/conf.d/suhosin.ini
pear channel-update pecl.php.net
pecl install apc
That "should" work. tbh I'm not exactly sure why your method didn't work since the error says the php version apc is compiled against is different to your php version but you have the same version of php and php-dev.
Should I increase the RAM for wordpress / vbulletin running on a 360?
I noticed I'm still hitting maxclients from time to time so have lifted this to 30. Figure I'm not running out of memory, but find it confusing how top shows memory usage as a %.. makes it tough to know how many MB each process is using.
I also get an occasional error where the server prompts one of the other admins to download a .php file rather than serving it. Any thoughts on what could
Didn't want to go to source since that didn't work for you either.
THANKS.
@theatereleven:
Hey did you ever get anywhere with this? I'm using Ubuntu 8.04 and trying to install APC with no luck. I can't get it to find the application with the normal get-apt command (or apt-get, whatever).
Didn't want to go to source since that didn't work for you either.
THANKS.
I did get it going in the end, through a lot of perseverance and testing on a second linode. The latest beta of APC, combined with a switch to nginx for front end proxying has been fantastic and my site has now enjoyed an unprecedented 36 days uptime (touch wood).
ERROR: '/tmp/pear/cache.APC-3.0.19/configure –with-apxs' failed
Whatever that means. =(