1GB Linode - Apache Worker MPM/FCGID/Max concurrent

Any ideas?

What are you hanging off the FCGID? PHP? Python? Vuvuzelas? Custom C program?

What's the config of that "thing", in regards to memory limits and spawning subprocesses?

(Keep in mind that Apache modfcgid does not pipeline requests, it assumes each of the FCGI processes it spawned can handle one request at a time. If you're using php-cgi* with the FCGICHILDREN option, you should use mod_fastcgi instead.)

• Never used php-fpm, heard the situation is different there though.

Do you require Apache? While it's possible to configure Apache for decent memory use, it's a lot easier to configure lighttpd or nginx for large numbers of consistent connections.

Your config generally seems oriented at much more RAM than you have. For example, you've got FastCGI configured for 25 processes. If we assume 64MB of RAM each (this is closer to the worst case, but that's what you should plan for), that's 1600MB of RAM for PHP right there. Plus, as far as I can tell, you've got Apache configured for up to 16 processes, if we assume 20MB of RAM for that (I've no idea how much RAM Apache uses), that would give us another 320MB. If you've got a SQL server running on this box as well, double trouble.

So you've basically configured Apache/PHP for a server with somewhere between 2 and 3 gigs of RAM.

It sounds like most of your long-lived connections are direct to Apache, not PHP (streaming static MP3 files), so you can probably afford to drop the number of PHP processes way the heck down. Fitting in 750MB of RAM as a worst case is going to take some sacrifice, so you probably can't afford it to be much beyond 8-10 or so. Might need to drop the number of Apache processes too.

Thanks so much for the reply!

The few suggestions you gave really made alot of different information I read on the internet make sense - I was just not sure what was what.

I have since moved Serverlimit down to 8 Maxclients down to 200 and FCGID down to 10 - will keep an eye on it and see how it goes.

I am also considering adding nginx in from of Apache to handle the static content.

I will keep this updated with what I find and how I update things!

Thanks again!

It's no problem getting everything running in 1GB without resorting to nginx, lighttpd, varnish, etc. All those do is add unneeded complexity.

I've got a box running httpd.worker/event (switched to event a week or so ago), mod_fastcgi + PHP, APC, and MySQL, all in around 650MB. The box is configured for 24 dynamic connections (PHP & MySQL) and 300 static (JS, CSS, images, etc.).

Right now MySQL is running 105MB, httpd 176MB, PHP (including 96MB cache in APC) 273MB. The remaining "core" system services eat the other 100MB or so, leaving around 350MB for buffers/cache and MySQL expansion (my max memory usage with MySQL would be ~200MB @ 24 connections).

Unless you need some aspect of Apache (rewrite rules or something), it's better to replace it entirely with nginx (both static and dynamic) rather than a hybrid approach mixing apache/nginx. That just adds unnecessary complexity and memory overhead without any gain, so if possible it should be avoided.

@jasonlitka:

It's no problem getting everything running in 1GB without resorting to nginx, lighttpd, varnish, etc. All those do is add unneeded complexity.

I've got a box running httpd.worker/event (switched to event a week or so ago), mod_fastcgi + PHP, APC, and MySQL, all in around 650MB. The box is configured for 24 dynamic connections (PHP & MySQL) and 300 static (JS, CSS, images, etc.).

Right now MySQL is running 105MB, httpd 176MB, PHP (including 96MB cache in APC) 273MB. The remaining "core" system services eat the other 100MB or so, leaving around 350MB for buffers/cache and MySQL expansion (my max memory usage with MySQL would be ~200MB @ 24 connections).

PHP has a default memory limit of 64MB per script, so the fact you've got 24 PHP processes eating only 273MB of RAM is just luck. Get hit with some heavy load/scripting/usage and those PHP processes will try to consume more RAM than you've got.

In my opinion, it's better to set things up taking the worst case into account so that high load causes a graceful failure (some people get through, some people get load errors, the whole box keeps working) rather than a catastrophic failure (RAM usage shoots up so high the entire sever locks up due to swap-death, OOMs cause the kernel to start killing random processes, nobody gets through).

@kiteplans: kudos on lowering ThreadStackSize to a saner value, but I've missed the part where you've put MaxRequestsPerChild 1000… that's a good idea for prefork, but for worker use a very high number or 0 (unlimited).

This is my setup. All this PLUS MySQL is fitting in under half of Linode 512, leaving rest for cache.

apache.conf, mpm_worker section:

    StartServers          2
    ServerLimit          12
    MinSpareThreads      25
    MaxSpareThreads      75
    ThreadLimit          25
    ThreadsPerChild      25
    MaxClients          300
    MaxRequestsPerChild   0
    ThreadStackSize 1048576

Each of these 25-thread Apache worker processes averages around 8 (eight) MB RAM.

The master, and FCGI process manager add up to another 16 MB or so.

apache.conf, fastcgi section:

        FastCgiConfig \
                -idle-timeout 120 \
                -initial-env PHP_FCGI_CHILDREN=16 \
                -initial-env PHP_FCGI_MAX_REQUESTS=500 \
                -killInterval 100000 \
                -listen-queue-depth 300 \
                -maxClassProcesses 1 \
                -singleThreshold 0

Once again - don't use PHPFCGICHILDREN with modfcgid, as it will think there's one parser and will execute one PHP script at a time. Use modfastcgi instead.

(And without PHPFCGICHILDREN APC is much less useful because it can't share cache between parsers.)

Also, PHPFCGIMAXREQUESTS is equivalent-ish of MaxRequestsPerChild in prefork+modphp. PHP isn't good at memory management, so you want the parsers to respawn relatively often to keep memory leaks under control.

php.ini:

memory_limit = 32M
apc.shm_size = 64

Careful: I'm running a PHP-heavy site, so I'm running many parsers in parallel. They usually average to 16MB each.

But as I /need/ to allow the ability of uploading and thumbnailing large pics, the PHP memory limit is set to 32MB. Which means it might be possible to OOM (or at least swapthrash) it by starting a bunch of such image uploads in parallel.

These values would be safe for a Linode 1024, though, I think.

@Guspaz:

@jasonlitka:

It's no problem getting everything running in 1GB without resorting to nginx, lighttpd, varnish, etc. All those do is add unneeded complexity.

I've got a box running httpd.worker/event (switched to event a week or so ago), mod_fastcgi + PHP, APC, and MySQL, all in around 650MB. The box is configured for 24 dynamic connections (PHP & MySQL) and 300 static (JS, CSS, images, etc.).

Right now MySQL is running 105MB, httpd 176MB, PHP (including 96MB cache in APC) 273MB. The remaining "core" system services eat the other 100MB or so, leaving around 350MB for buffers/cache and MySQL expansion (my max memory usage with MySQL would be ~200MB @ 24 connections).

PHP has a default memory limit of 64MB per script, so the fact you've got 24 PHP processes eating only 273MB of RAM is just luck. Get hit with some heavy load/scripting/usage and those PHP processes will try to consume more RAM than you've got.

In my opinion, it's better to set things up taking the worst case into account so that high load causes a graceful failure (some people get through, some people get load errors, the whole box keeps working) rather than a catastrophic failure (RAM usage shoots up so high the entire sever locks up due to swap-death, OOMs cause the kernel to start killing random processes, nobody gets through).

I've generated heavy load after setting up that dev box, the system is rock stable when being pummeled and does not OOM at all. The highest I've ever seen memory usage during stress testing was a total of ~820MB and that was with 64 clients slapping what turned out to be the most complicated PHP script (as per memorygetpeak_usage). Of course, since only 24 can be serviced at a time, page load time averaged around 5 seconds (with a peak of ~6) instead of the typical 1.0-1.2s, but it still worked.

The point of this is that you need to know your content in order to do a proper job of server tuning.

Thank you to everyone for all the information! It has been so helpful! Got to love the Linode Community!

@rsk - I have taken your advice and set

MaxRequestsPerChild 0

I was also reading more online and found many people saying that you should set the Serverlimit setting to the amount of CPU`s you have - so I set

Serverlimit 8

I also had a situation where someone was trying to download every file on our site and I was lucky enough to be online to monitor the server while it was going down. The server performed really well and never peaked over 650MB memory usage - I was even able to set the mod_fcgid settings a bit higher and saw a great increase in the amount of user that the site could serve.

FcgidMinProcessesPerClass 0 
FcgidMaxProcessesPerClass 35 
FcgidMaxProcesses  40
FcgidIdleTimeout 60 
FcgidProcessLifeTime 120 
FcgidIdleScanInterval 30