terrible performance came out of nowhere

@Fangs404:

I'm experiencing some awful performance on my server. (…) Memory usage is fine, swap usage is essentially nil, as is general I/O. CPU usage is really low, too (…)

By awful performance I am assuming page load time? Increase Apache MaxClients and ServerLimit to, say, 32 and observe what happens. Keep in mind, though, that will double the memory consumption of your apache/php processes so make sure there's room for that, or increase in smaller steps. Also fully restart httpd, reload won't catch MPM config changes.

Also install Munin, Apache plugin and corelate number of busy processes with memory/cpu/bandwidth usage. That way you'll know better what's going on.

Yes, awful page load time.

Increasing MaxClients seemed to help quite a bit. I'm setting Munin up now. I'm using quite a bit of swap now that MaxClients is increased (I bumped it to 30). A Linode 512 should be able to handle the kind of load I'm getting just fine.

What sorts of things should I be looking for in Munin? There is a ton of information here!

For starters, with Apache munin plugin you can see the saturation of the Apache processes, as well as what else increases or decreases with traffic.

After you have munin running for a while, you will start to see trends in the charts, like when a backup kicks off, processes start up. I had to tweak the Apache module a bit but I can now see access counts, volume, error counts, it is easy to see when something is happening… pretty good tool.

Via mobile device…

There's a limit to how many simutaneous visitors you can serve with mpm_prefork, especially on a media-heavy website like that. Large downloads clog up the prefork processes, unlike simple pages that quickly get out of the way.

Short-term solution: Disable KeepAlive.

Long-term solution: Offload your /wordpress/wp-content/uploads directory to another web server like nginx (or a CDN). This will leave Apache with some breathing room to serve PHP scripts more efficiently.

I tossed your page into http://www.webpagetest.org/ and see you have a lot of content on your site.

Many of the images you are using are not very optimized for quick loading.

Example:

http://americankpopfans.com/wordpress/w … esmall.jpg">http://americankpopfans.com/wordpress/wp-content/uploads/2012/05/AKPF_Timelinesmall.jpg

Which is a large 800x300 image you use on the main page. It is physically 754kb in size. It is taking 18197 ms to download.

Optimizing the image (still same size in pixels) got it down to 30kb. That is 25x smaller and would load significantly faster.

A lot of your other images are much larger than you are using them on the site. While they are small on the site, you are still loading the full sized images and having the visitors browser resize them down. This is both using more bandwidth (slower download) but also results in less quality images (browser resize based on speed not quality).

Example:

http://americankpopfans.com/wordpress/w … degif1.gif">http://americankpopfans.com/wordpress/wp-content/uploads/2012/03/vikislidegif1.gif

Actual image is 600x880, you are using it on the website as 280x350. At over 500kb, the actual size if you reduced it would be under 20kb.

A little content optimization can go a long way.

Offloading your content as suggested above is a good idea too, but if the images are large, they are going to load slower.

Try google pagespeed, which has some pretty easy-to-fix steps on theoretically making your site less slow:

https://developers.google.com/speed/pag … bile=false">https://developers.google.com/speed/pagespeed/insights#url=http3A2F2Famericankpopfans.com2F&mobile=false

check your logs, see if there are any errors. Did you update your site recently?