Weird problem with seemingly stale apache2 processes.
I used the StackScript to conveniently fire up Debian Lenny with a full Apache2/MySQL/PHP hosting stack and set up all of my virtual hosts like I have always done. Past that initial configuration, I've installed very few things and tinkered very little. All of my packages came directly from Debian repos via apt-get such as zip, phpmyadmin, curl and APC… I'm quite sure that is all I have going on. Certainly nothing that would cause this error:
"Restarting web server: apache2(98)Address already in use: make_sock: could not bind to address [::]:80
(98)Address already in use: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
Unable to open logs
failed!"
So of course the first thing I did when I saw this error was fire up netstat to see what was going on. Using every parameter I know, I could find nothing that insinuated that port 80 was being used so that Apache2 could not bind to it. Obviously restarting Apache was not working, so I rebooted the whole darn server.
Nothing.
The same thing persisted. The odd thing though is that an Apache process was still hanging around but doing nothing… still whining about not being able to bind.
So as a last resort I decided to 'killall -9 apache2' then '/etc/init.d/apache2 restart' and voila - things are right again.
Now the odd thing is that it has happened exactly 3 times in the past two days on a 5 day old disk image - two times in the same hour span yesterday and once today. No warning signs or anything… things just quit responding.
When this happens it won't let me access anything via port 80 or 443 and if I am allowed to login through SSH, it's very slow and unresponsive until a full system reboot.
It's worth noting too that I installed a RapidSSL certificate the same day problems started happening but things worked perfectly for hours and hours before anything like that happened with the port 80 problems.
Any ideas?
13 Replies
I'm on the 512 plan with a 256MB swap running … drumroll please…. Drupal!
Yes, Drupal can be a resource hog but I've been using it for a while now and know how to build them with speed and resources in mind.
The output of 'free -m' shows me nothing terrible (I think):
total used free shared buffers cached
Mem: 498 96 401 0 8 45
-/+ buffers/cache: 42 456
Swap: 255 18 237
That is running fine though. I'll report back if it happens again with new numbers.
It's funny you mention disk I/O too as it has been rather high lately - I've gotten three notices that my average I/O is over 1000 blocks/s but even the email said to take it lightly. Maybe I should look into that more? The site is in active development but I hope this strain wouldn't kill it like this…
Thanks,
So if you're hitting SWAP you need to optimize your processes better.
Changing the size of SWAP has nothing to do with why it's being used in the first place.
Small or Large SWAP size really doesn't matter (of course your wife will debate that endlessly), it just delays when the crash from lack of memory will occur.
@vonskippy:
Small or Large SWAP size really doesn't matter (of course your wife will debate that endlessly), it just delays when the crash from lack of memory will occur.
Hah!
Well I'll certainly keep your advice in mind!
I came back to report a new development! Right before I ever had the chance to change the swap size, it happened again! A 'netstat -lnp | grep :80' shows that apache2 IS in fact listening in but not doing a thing. I don't recall that happening last time but it was late and I was quite tired and frustrated so anything is possible…
I'm not sure what would cause Apache2 to just freeze like that randomly but I'll look into it.
A little looking around the system logs showed this happening a few times:
''[Wed Sep 08 22:16:06 2010] [error] server reached MaxClients setting, consider raising the MaxClients setting"
I know for a fact that the site isn't getting flooded so I assumed PHP may be the culprit. After toying with a few settings and optimizing the memory a bit more responsibly, things seem to be working much smoother and the flood of Apache notices seemed to stop a bit. I'll report back on the status later tonight for those interested. Thanks for the help!
ninjaedit:
@obs:
What kernel are you using?
Debian Lenny with 2.6.32.16
@nanerpuss:
[error] server reached MaxClients setting, consider raising the MaxClients setting"
What is your MaxClients setting? Can you post the mod_prefork section from your /etc/apache2/apache2.conf ?
By the way, using ~20M of swap under normal conditions is perfectly OK. Using a little swap could actually help your server run faster, by freeing up more RAM for buffers/cache.
More information on how to read the output of free -m:
@hybinet:
@nanerpuss:[error] server reached MaxClients setting, consider raising the MaxClients setting"
What is your MaxClients setting? Can you post the mod_prefork section from your /etc/apache2/apache2.conf ?
By the way, using ~20M of swap under normal conditions is perfectly OK. Using a little swap could actually help your server run faster, by freeing up more RAM for buffers/cache.
More information on how to read the output of free -m:
http://www.linuxatemyram.com/
Sure:
MinSpareServers 5
MaxSpareServers 10
MaxClients 19
MaxRequestsPerChild 0
19 does seem a little low doesn't it?
@nanerpuss:
19 does seem a little low doesn't it?
No, 19 looks just fine. Each client needs one process, and each process contains the PHP interpreter which takes up over 10MB of RAM (plus the amount of RAM used by your PHP script, if it's a PHP request). Anything over 20 and you're just increasing the chance of swapping. In a VPS environment, you should normally ignore "server reached MaxClients" messages. At worst, a few clients will have to wait in line for a fraction of a second.
On the other hand, if MaxClients is already below 20, that pretty much rules out the possibility of Apache+PHP eating up all your RAM – unless one of your scripts is a supermassive resource hog. Just to be sure, try reducing MaxClients to 15 for a while. It shouldn't hurt performance too much.
What's the value of memory_limit in /etc/php5/apache2/php.ini ? Do you have any .htaccess directives or PHP scripts that override the default?
How much RAM is MySQL using? Any other big programs?
Can you run htop while you're having one of those slowdowns? (install it first: apt-get install htop) That would show you a lot of interesting realtime statistics, as well as the name of the program that's using the most CPU & RAM at that moment.
The above steps would nearly exhaust the out-of-memory possibilities. If you're not out of memory when the slowdown happens, the problem may be elsewhere.
You said you began to have these problems after you installed SSL. Can you disable SSL for a few days to see how it goes? If the server run fine without SSL, we might need to look more closely at your SSL configuration.
It seems htop is very handy though….
On my machine with 498MB available to use, it's consistently hanging around the 375-425 range!
Apache has about 9 processes sitting about 9% and there are 6 MySQL processes combining for ~13%.
It seems like memory is indeed my issue…
So I'm going to go ahead and lower the max clients as you suggested and throttle the hell out of some Drupal processes. I have no caching enabled whatsoever right now so I'm sure that plays a role but still - that's outrageous!
Using 425MB out of 498MB is surely quite close to running out of memory. But your "free -m" seems to tell a different story. Was that with a fresh reboot?
Anyway, caching is a good idea with monsters such as Drupal. I wouldn't even imagine using a modern CMS without a caching plugin.
It's worth noting that when I turned on full caching, I was hovering around 180-200 MB total usage and a slight spike when clearing a cache or uploading a huge image.
So it seems we found my problem anyways which is a HUGE relief. Thanks a lot for your help!