Memory usage of php-cgi processes is growing steadily
I'm using Debian Lenny 32bit and its lighttpd and php5-cgi packages. Apart from some config changes (listed below), I'm using the stock setup by Debian.
The website is based on Drupal. Using Drupal's debugging module (called devel), I can tell that memory usage of PHP scripts is less than 20KB on average, and it never exceeds 8MB.
Here are the relevant parts from the output of ps aux:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
www-data 29871 0.0 1.7 54552 6368 ? Ss Aug12 0:00 /usr/bin/php-cgi
www-data 29873 0.0 7.4 65808 27468 ? S Aug12 0:00 /usr/bin/php-cgi
www-data 29874 0.0 3.7 55808 13736 ? S Aug12 0:00 /usr/bin/php-cgi
www-data 29875 0.0 4.3 58040 16204 ? S Aug12 0:00 /usr/bin/php-cgi
www-data 29876 0.0 4.4 57444 16288 ? S Aug12 0:00 /usr/bin/php-cgi
www-data 29877 0.0 1.7 54552 6368 ? Ss Aug12 0:00 /usr/bin/php-cgi
www-data 29879 0.0 9.6 67140 35684 ? S Aug12 0:26 /usr/bin/php-cgi
www-data 29880 0.0 6.6 59172 24492 ? S Aug12 0:23 /usr/bin/php-cgi
www-data 29881 0.0 7.1 59784 26388 ? S Aug12 0:22 /usr/bin/php-cgi
www-data 29882 0.0 7.4 60880 27440 ? S Aug12 0:23 /usr/bin/php-cgi
* Is it normal to have php-cgi this large?
Is it possible to estimate php-cgi memory usage based on settings?
Any tips for reducing memory consumption of php-cgi processes?
Searching for known memory leaking bugs didn't yield anything relevant. And I'd be surprised if the default Debian packages/config had such an obvious memory leak. Other users on the same host do not have this problem.
What I've done so far is set PHPFCGIMAX_REQUESTS to a low value so that php-cgi processes are recycled quickly. When I use ab to simulate high load, this works very well. Processes die quickly before they grow higher than 10MB. However, under low-to-medium load, all processes grow steadily (because of load balancing) and most of them consume 28MB+ simultaneously, putting my VPS at risk of swapping. Please note than even without any sort of traffic, the processes grow steadily.
I can reduce the number of php-cgi processes, but this feels like a workaround more than a fix. I'd be surprised if php-cgi normally grew like this.
Also, summing the total RSS numbers for php-cgi processes gives:
$ ps -C php-cgi -o rss= | awk '{s+=$1}END{print s/1024}'
195.738
Yet, free -m gives the following output:
total used free shared buffers cached
Mem: 360 351 8 0 33 190
-/+ buffers/cache: 127 232
Swap: 255 0 255
* Am I missing something? How come the used memory (without buffers) is lower than the total resident memory of php-cgi processes on the host?
I have the following PHP extensions:
php5-cgi php5-common php5-curl php5-gd php5-mysql php5-xcache
xcache.size is set to 24M. It used to be 32M but reducing it didn't help. xcache.var_size is set to 0. The remaining plugins are using the stock configuration. The xcache admin pages shows that xcache is using less than 1MB.
PHP's memory_limit is set to 32M.
Here is my FastCGI config:
fastcgi.server = ( ".php" =>
((
"bin-path" => "/usr/bin/php-cgi",
"socket" => "/tmp/php.socket",
"max-procs" => 2,
"idle-timeout" => 20,
"bin-environment" => (
"PHP_FCGI_CHILDREN" => "4",
"PHP_FCGI_MAX_REQUESTS" => "1000"
),
"bin-copy-environment" => (
"PATH", "SHELL", "USER"
),
"broken-scriptfilename" => "enable"
))
)
I'm using more-or-less the stock lighttpd.conf that ships with Debian.
Please let me know if there is any other data that I can provide.
Any help is appreciated. I've been trying to troubleshoot this for days. I've run out of ideas.
8 Replies
htop tells me I'm using nearly 120MB. This has only just become a problem, I'm still running pretty much the same thing on my server since I first turned it on (as well as dnscache and irssi now) and memory has just shot up over time. I simply don't understand it.
Edit: Well waddaya know… Simply restarting lighty took memory usage to just 80MB. php5-cgi is still gobbling up a bit too much but at least one thing's solved…
My recommendations:
Right now, lighttpd is set to spawn two processes, which will each spawn four children (for a total of 8 children). This isn't a good idea if you're using an opcode cacher (e.g. xcache); set your max-procs to 1. You can then twiddle PHPFCGICHILDREN if you need more actual children, although four should be fine for most purposes.
Removing the idle-timeout may help performance, although it won't impact memory consumption. (The goal is to use memory wisely, not to avoid using it.)
Here's what I'm seeing on an Ubuntu 8.04 (32-bit) system… note that I use spawn-fcgi, run each application as its own username, and use different child limits for each. So, basically, I'm violating my first recommendation above in the name of better isolation
total used free shared buffers cached
Mem: 348 332 16 0 74 112
-/+ buffers/cache: 145 203
Swap: 255 56 199
asterisk 2744 0.0 0.8 37532 2900 ? Ss Aug12 0:00 /usr/bin/php-cgi
asterisk 2745 0.0 0.5 37532 2024 ? S Aug12 0:00 /usr/bin/php-cgi
asterisk 2746 0.0 0.5 37532 2024 ? S Aug12 0:00 /usr/bin/php-cgi
blogs 2748 0.0 0.8 37532 2912 ? Ss Aug12 0:00 /usr/bin/php-cgi
blogs 4632 0.0 4.2 39224 15236 ? S Aug13 0:28 /usr/bin/php-cgi
blogs 5371 0.0 4.4 40520 16012 ? S Aug13 0:27 /usr/bin/php-cgi
blogs 5375 0.0 5.0 40828 17904 ? S Aug13 0:27 /usr/bin/php-cgi
blogs 5378 0.0 4.9 41912 17800 ? S Aug13 0:27 /usr/bin/php-cgi
gallery2 15103 0.0 4.4 41624 15892 ? S Aug13 0:00 /usr/bin/php-cgi
gallery2 2756 0.0 0.8 37532 2908 ? Ss Aug12 0:00 /usr/bin/php-cgi
gallery2 6536 0.0 4.1 40024 14788 ? S Aug13 0:00 /usr/bin/php-cgi
gallery2 6550 0.0 4.0 40012 14344 ? S Aug13 0:00 /usr/bin/php-cgi
The first batch gets nearly zero traffic (n'fact, I don't think it's had any since I started it…); the second and third batches get a lot of traffic. I thought I was using xcache, but I don't have anything mentioning it in php.ini, so perhaps I'm not.
@hoopycat:
Memory usage displayed on ps isn't exactly intuitive. I suspect a lot of it has to do with memory the instances are sharing. In any case, at least from the ps and free you pasted, you're probably in reasonable shape. I wouldn't be worried.
Indeed, I don't fully understand the numbers. Perhaps the RSS values include cache/buffers? Because the sum of the RSS values adds up to more than the total system memory. Yet, the free command reports plenty of free memory.
> My recommendations:
- Right now, lighttpd is set to spawn two processes, which will each spawn four children (for a total of 8 children). This isn't a good idea if you're using an opcode cacher (e.g. xcache); set your max-procs to 1. You can then twiddle PHPFCGICHILDREN if you need more actual children, although four should be fine for most purposes.
Good point, I will make this change and see how it goes. My initial idea was to have 2 process trees so that if one dies, the other would be immediately available. Also, I thought that this would utilize the multi-core system better. Keep in mind that I don't fully understand how requests are handled by FastCGI so it is likely that my ideas are inaccurate or wrong.
Interestingly, the 2 parent processes never grow above 7MB. It is the children who consume most of the memory. Furthermore, the admin interface of xcache reports that all of my PHP pages are cached in 2MB.
> - Removing the idle-timeout may help performance, although it won't impact memory consumption. (The goal is to use memory wisely, not to avoid using it.)
Already done this.
> Here's what I'm seeing on an Ubuntu 8.04 (32-bit) system… note that I use spawn-fcgi, run each application as its own username, and use different child limits for each. So, basically, I'm violating my first recommendation above in the name of better isolation
total used free shared buffers cached Mem: 348 332 16 0 74 112 -/+ buffers/cache: 145 203 Swap: 255 56 199 asterisk 2744 0.0 0.8 37532 2900 ? Ss Aug12 0:00 /usr/bin/php-cgi asterisk 2745 0.0 0.5 37532 2024 ? S Aug12 0:00 /usr/bin/php-cgi asterisk 2746 0.0 0.5 37532 2024 ? S Aug12 0:00 /usr/bin/php-cgi blogs 2748 0.0 0.8 37532 2912 ? Ss Aug12 0:00 /usr/bin/php-cgi blogs 4632 0.0 4.2 39224 15236 ? S Aug13 0:28 /usr/bin/php-cgi blogs 5371 0.0 4.4 40520 16012 ? S Aug13 0:27 /usr/bin/php-cgi blogs 5375 0.0 5.0 40828 17904 ? S Aug13 0:27 /usr/bin/php-cgi blogs 5378 0.0 4.9 41912 17800 ? S Aug13 0:27 /usr/bin/php-cgi gallery2 15103 0.0 4.4 41624 15892 ? S Aug13 0:00 /usr/bin/php-cgi gallery2 2756 0.0 0.8 37532 2908 ? Ss Aug12 0:00 /usr/bin/php-cgi gallery2 6536 0.0 4.1 40024 14788 ? S Aug13 0:00 /usr/bin/php-cgi gallery2 6550 0.0 4.0 40012 14344 ? S Aug13 0:00 /usr/bin/php-cgiThe first batch gets nearly zero traffic (n'fact, I don't think it's had any since I started it…); the second and third batches get a lot of traffic. I thought I was using xcache, but I don't have anything mentioning it in php.ini, so perhaps I'm not.
Thanks for sharing. Your numbers are a bit lower than mine.
By the way, you can grab the xcache admin interface
The key to FastCGI on a small VPS is to keep the number of processes as low as possible without hurting performance. In your case you can probably make do with just 4, and likely no more than 8. Just do the following math:
How many dynamic requests do you handle per second? e.g. 20
How long does it take for the average request to be handled? e.g. 0.2 second
Multiply the two numbers, e.g. 20 x 0.2 = 4
Then you need a minimum of 4 FastCGI processes.
@ah:
Good point, I will make this change and see how it goes. My initial idea was to have 2 process trees so that if one dies, the other would be immediately available. Also, I thought that this would utilize the multi-core system better. Keep in mind that I don't fully understand how requests are handled by FastCGI so it is likely that my ideas are inaccurate or wrong.
The children handle the requests, so they're already nicely split across the multiple cores. PHP is handling all of the multi-processor (and probably some of the redundancy) for you. What a deal!
Speaking of xcache, I did some digging through the config and it looks like xcache.count is a good thing to set to "4". That might eek out some more performance.
> Thanks for sharing. Your numbers are a bit lower than mine.
Different applications and sets of extensions, probably. Keep in mind the php interpreter itself is kinda beefy:
rtucker 25281 0.2 1.9 38280 6904 pts/7 S+ 16:12 0:00 php
That's from just typing "php" at the command line.
> By the way, you can grab the
xcache admin interface
Sweeeet. I have done so, and indeed, it is. I've also done some tuning. Thanks!
I installed Munin a few days ago. It looks like memory usage automagically drops every 20-30 hours without any intervention and without changes in PIDs.
I think I should read up on how memory management in Linux works in order to understand the numbers. But for now, things look fine.