First access is slow

Something is devouring your memory causing quite a lot of swapping. Check out individual processes with 'top' or 'ps aux'.

My problem is I don't know how much is too much. Here is the list as output from Virtualmin in a csv format.

Well, it's simple really. You just add up numbers until you hit memory maximum and then decide what goes out.

For example, you seem to have 13 PHP processes each 60-70MB large. That's more than 780MB. On a 512 node.

Granted, I'm not familiar with Virtualmin and I don't know what those numbers are, shared virtual or res.

You also seem to have both PHP-CGI and Apache processes take quite a lot of ram. How so? Using both mod_php and fastcgi? Anyway, for those process sizes you can really afford 3 or 4 of each, leaving room for other things.

I just looked at the configuration and it does look like for the older sites the option is set for Apache mod_php (run as Apache's user) and for the newer ones it is set for FCGId (run as virtual server owner). I had some problems getting this set up originally so I hired a guy to do it. So I'll go bug him and see if he can wrangle it back in.

I have a question though. Why don't these processes go away if they aren't being used. I know there are not that many people visiting my sites at any one time.

Because they're set to have minimum number of processes active at any time. Check StartServers, MinSpareServers and MaxClients for Apache. Don't know what you're using for PHP.

Just looked at your CSV and Munin graphs again. The Apache processes all have same amount of RAM and Munin shows just one active really. Can you check with ps -TAfm if those are Apache threads (there'll be one master Apache process with PID)? Using worker_mpm with PHP in fastcgi?

I'm experiencing the same: first bit takes about three seconds. Subsequent page loads (different domains/platforms on the same server) take 0.2 seconds. This makes me doubt the cache theory. I'm thinking along the lines of DNS, even though I have HostnameLookups set to Off.

The following seemed to help, at least it did with swapping.

    StartServers       3
    MinSpareServers    3
    MaxSpareServers    3
    ServerLimit        50
    MaxClients         50
    MaxRequestsPerChild  1000