Recommendations (other than Apache)

What is using most of your RAM? If it's Apache, then you might want to look at trying out Litespeed. (http://www.litespeedtech.com/)

I use it and it uses next to no RAM for each PHP process spawned and is much faster than Apache.

If MySQL is using most of your RAM, then you should take a look at your my.cnf settings and see what things you can trim down.

Not sure if the values below came from the load test or not looking in the scrollback buffer. MySQL seems reasonable. The load test didn't involve a lot of database work which I have to consider in the future, but my apps aren't to db instensive anyway. Not sure how to interpret php-cgi vs apache2. I assume php-cgi is the spawned process for the interpreter and apache2 sends the results?

top sorted by mem%
 4714 web24     20   0  118m  70m  41m S    7 14.2   0:12.35 php-cgi
 5698 web24     20   0  113m  65m  41m S    0 13.2   0:13.40 php-cgi
 5861 web24     20   0  109m  60m  40m S    0 12.2   0:08.36 php-cgi
 6260 web24     20   0  103m  44m  30m S    0  9.0   0:02.16 php-cgi
 2197 mysql     20   0  116m  21m 4568 S    0  4.3   0:05.39 mysqld
 4265 root      20   0  109m  20m  16m S    0  4.2   0:00.27 apache2
 4712 www-data  20   0  111m  14m 6832 S    0  2.8   0:01.94 apache2
 4348 www-data  20   0  111m  11m 5136 S    0  2.4   0:01.93 apache2
 5551 web33     20   0 93164 9924 6912 S    0  1.9   0:00.03 php-cgi
 5725 www-data  20   0  111m 9104 2308 S    2  1.8   0:01.50 apache2
 5859 www-data  20   0  111m 8816 2288 S    0  1.7   0:01.02 apache2
 5917 www-data  20   0  110m 8744 2280 S    1  1.7   0:00.94 apache2
 6259 www-data  20   0  110m 8456 2200 S    1  1.7   0:00.43 apache2

If I remember right, php-cgi is purely the fcgi backend that Apache uses to process PHP while the Apache 2 processes are handing the static requests.

Which is why his Apache processes aren't using much RAM.

512MB of RAM can handle quite a bit of web load, but you're right that your mail stuff (particularly SpamAssassin it seems) takes up a good chunk of RAM. You don't necessarily have to use two linodes for this, as the costs scale linearly; 1x1024 linodes costs the same as 2x512, and the 1x1024 might be easier to manage. That said, there's something to be said for decoupling services for resiliency purposes, particularly if they're in different datacenters.

There is certain efficiencies to be gained by using Litespeed, Lighttpd, or nginx instead of Apache, but by switching Apache to mpm_worker and fcgi (or equivalent), you've already gotten most of the advantage.

In terms of a PHP memory limit, that depends on your PHP app. Back in the day, the default limit was 16MB, and most everything ran fine with that. These days, the default is higher, and some apps require it go higher still. I don't think there's often occasion to have it higher than 64MB, though, and I think 32 might be the default? I don't remember anymore.

32MB is good enough for most web apps, such as WordPress and Drupal. But heavy add-ons/plugins can cause memory usage to increase beyond that. Image processing scripts are often the worst offenders, as they can use more than 100MB simply to produce a thumbnail from a photograph. Just two or three of those scripts running at the same time can cause your server to run out of memory. Some distributions nowadays ship PHP with 128MB memory limits by default. Insane, in my opinion.

Since you seem to be running one FastCGI family (parent process + one or more children) per website, you should take into account the worst-case scenario where each of those processes use to the maximum permitted amount of memory. Sometimes you simply can't afford to give more than 32MB to each process.

You can always just spin up a copy of imagemagick in your PHP script if you need to do thumbnail stuff.

Good point Guspaz on just doubling the Linode size instead of creating a second one. That would save me the headache of managing a second one.

I have Moodle 2.2 installed and that is the biggest memory hog by far on the system and just double checking they say to decrease the PHP memory limit from 128 to 40.

Thanks!

I also recommend Litespeed especially if you want Apache compatibility, but NGINX is a great performer also.