Fine tuning Apache and Nginx
Currently the server is only hosting two blogs and a Mumble server. Idle I am sitting around 100-115MB RAM. The two blogs are both running Wordpress. The total traffic per month is around 1,500 unique visitors.
Within the next month I will be adding another website which has substantially more traffic then my current blogs do. Below is a screenshot of the AWStats log of the website which will be hosted on my server in a month:
~~http://imgur.com/Fihve.png
I would imagine the Unique visitors will stay roughly the same for awhile (+/- 10%). It is for a World of Warcraft guild. Not many regular visitors other then members. The new website is heavy on the dynamic side.
I have run AB from my local machine and received the following output:
ab -n100 -c20 http://kastang.com/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking kastang.com (be patient).....done
Server Software: nginx/0.7.65
Server Hostname: kastang.com
Server Port: 80
Document Path: /
Document Length: 87818 bytes
Concurrency Level: 20
Time taken for tests: 22.807 seconds
Complete requests: 100
Failed requests: 0
Write errors: 0
Total transferred: 8806200 bytes
HTML transferred: 8781800 bytes
Requests per second: 4.38 [#/sec] (mean)
Time per request: 4561.495 [ms] (mean)
Time per request: 228.075 [ms] (mean, across all concurrent requests)
Transfer rate: 377.06 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 17 24 3.8 23 37
Processing: 841 4382 1294.3 4515 7343
Waiting: 501 2844 953.0 2779 5522
Total: 859 4406 1295.5 4537 7380
Percentage of the requests served within a certain time (ms)
50% 4537
66% 4675
75% 4709
80% 4803
90% 6234
95% 6715
98% 7171
99% 7380
100% 7380 (longest request)
The server spiked from 110MB idle to around 430MB initially then held steady for the duration of the AB test.
I also ran LoadImpact on the server and received the following:
Below is my Prefork config:
<ifmodule mpm_prefork_module="">StartServers 1
MinSpareServers 1
MaxSpareServers 4
ServerLimit 24
MaxClients 24
MaxRequestsPerChild 1000</ifmodule>
The values I have for prefork are from trial and error. I figured I could lower the values some since Nginx is handling the static content rather then apache.
How do the logs that I provided look? Does anything stand out that should be adjusted? Thanks very much for the input.~~
9 Replies
And if it were, you can always consider just not using Apache, if it's not going to cause lawsuits when a user gets disconnected every now and again you can often find some great alternatives to plug into nginx that are far more lightweight.
Overall, there's probably really not enough info here to form many conclusions on what approach to take just yet though, so I hope you don't mind some questions! I apologize if anything is overly simple for you, I don't mean to be condescending, just not sure where to aim, better to mention and offend than hold off and miss a chance to help.
Anyway, to get a better picture:
* * What size Linode are you using for these sorts of numbers?
* Would you be willing to split to multiple smaller 'nodes if it made sense?
* What will the newer, heavier guild site be running on? Typical LAMP?
* What are you using for caching, if anything?</list>
I think a careful look at those is probably going to provide a better picture of what you need to aim for than worrying too much about other things just yet, they're sort of the low-hanging fruit anyway.
I must admit I haven't run a WordPress install personally for a while so I'm probably a bit behind on more specific details but I doubt the platform as a whole is much different, and you're probably a ways from really needing minor app config tuning and such yet anyway.
Happy to look if you post larger sample/full configs also, but in that, well, workers are always going to be a bit case-by-case and you're saying you've reached those through trial and error, so they're probably well suited to your usage at least. I probably wouldn't let it attempt quite that many but there's not too much meaning there without knowing what size your node is.
This "Mumble", though? Sounds interesting! It sounds like it's a side-band chat app for gaming? When you say you run "Mumble" on this box, are you running the "Murmur" server app for your friends using Mumble to connect to, or the "Mumble" client app for something else?
If it's the server, you might want to look at how much load is typically produced by the Murmur server and how regular it is? Also the bandwidth/quality levels are you running it at? Might need those before deciding exactly what to do, but knowing what exactly each of them needs should help you choose how to balance them… don't have time to read too much into Murmur at the moment but if it's basically VoIP? It feels pretty likely that it'd be using a lot of bandwidth, a little CPU but not much to worry about and not much else worth mentioning?
If that's the case, you may want to look at using memcached or similar and chucking it into the same box alongside your voice app, as it's going to be using a lot of memory but not much anything else… Then you can use it pretty aggressively in caching the main parts of your backend sites. (If you're not already!)
I'm only assuming you're not aware of all this of course but since you didn't mention it, hopefully you aren't?
If that's the case then you're sitting pretty sweet, a good config like this, it seems the server's likely already doing quite well and caching and shuffling servers carefully is usually a very easy free 'upgrade' for most typical sites.
You can then even use nginx to serve pages directly from memcache if you really want it to, it's not always as good as writing a static file cache or other options but it runs pretty well on VPS services from what I've seen so far… and since you're not worried about keeping static files like images in there, even 64MB or 128MB can turn out to be quite a lot of memcache.
If you can describe a bit more detail on this Murmur server and the other minor things there might be able to offer some better advice though, happy to help dig out what I can then if you're interested.
The 110M -> 430M memory spike is probably due to 20 Apache processes being spawned at the same time. (You ran ab with a concurrency of 20, right?) Judging from your LoadImpact graph, and since you also have nginx, you can probably reduce MaxClients & ServerLimit to 10-15 to save RAM without hurting performance. Fewer requests served at the same time, but each request would take less time.
4 requests per second, however, is very very low. Is this a popular CMS such as Drupal or WordPress? If so, install some caching plugins. Is this some custom thing that you've written? Does it make hundreds of database queries per page? Do the queries use indexes? Are you calling external resources (e.g. third-party RSS feeds) with every page view? Cache them all.
@hybinet:
4 requests per second, however, is very very low. Is this a popular CMS such as Drupal or WordPress? If so, install some caching plugins. Is this some custom thing that you've written? Does it make hundreds of database queries per page? Do the queries use indexes? Are you calling external resources (e.g. third-party RSS feeds) with every page view? Cache them all.
The site I tested it on was a fairly stock Wordpress with no Caching plugin installed. I am working on that now.
> You might also consider running a php opcode cache on the apache side. I use APC on my linode with great success. apt-get php-apc
Thanks very much for the suggestion. I will look into that apache module in detail.
> The logs you've offered look pretty typical, but the fact that it's only hitting 4.4 req/s and 377KB/s would probably suggest to me that you've got something in your back-end that's way too slow to be worrying about tuning things like prefork yet anyway…
I am going to go under the assumption the reason for the slow requests is using Wordpress with no cacheing plugin. I will install one tonight and test results again.
> * What size Linode are you using for these sorts of numbers?
Would you be willing to split to multiple smaller 'nodes if it made sense?
What will the newer, heavier guild site be running on? Typical LAMP?
What are you using for caching, if anything?
Currently I am running a Linode 512. The new guild website will be running on the configuration I currently am working on (Apache Dynamic, Nginx static). Currently nothing for caching as I have never had a any sort of load on my server before. I plan on fixing this ASAP.
> This "Mumble", though? Sounds interesting! It sounds like it's a side-band chat app for gaming? When you say you run "Mumble" on this box, are you running the "Murmur" server app for your friends using Mumble to connect to, or the "Mumble" client app for something else?
Mumble is pretty much a free version of Ventrillo. I love it because it provides a lower latency then Vent or TS can offer along with a much better voice quality. We are currently beta testing Murmur (the server) on the Linode. Tests are working out very well with the exception of a few small glitches that I believe can be easily ironed out.
The bandwidth usage isn't to bad. I have monitored the server logs during heavy Murmur use (10/25 man raids) and the server itself is hardly breaking a sweat and uses hardly any resources.
Thank you for the very long and detailed reply, sorry for the short reply in return. I am going to look into some caching options and rerun some tests to see how the server holds up. I will post all new configuration information when it becomes available.
@kastang:
Wordpress
@kastang:I am going to look into some caching options and rerun some tests
WordPress is a massive resource hog. They really should build caching right into the core instead of making frustrated users search for third-party plug-ins, but since it didn't happen with WP3, I guess it won't happen anytime soon.
Note that WP Super Cache
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Server Software: nginx
Server Hostname: wpselect.com
Server Port: 80
Document Path: /
Document Length: 11912 bytes
Concurrency Level: 100
Time taken for tests: 1.598 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 122484177 bytes
HTML transferred: 119465448 bytes
Requests per second: 6258.50 [#/sec] (mean)
Time per request: 15.978 [ms] (mean)
Time per request: 0.160 [ms] (mean, across all concurrent requests)
Transfer rate: 74860.03 [Kbytes/sec] received
@jlevandowski:
Requests per second: 6258.50 [#/sec] (mean)
Wow. On my 540 (at the time), I got about 2,000, but that was merely with WP Super Cache. Did nginx get you to 6,000, or is there other secret sauce in that stack?
@jed:
Wow. On my 540 (at the time), I got about 2,000, but that was merely with WP Super Cache. Did nginx get you to 6,000, or is there other secret sauce in that stack?
This is pure nginx.
My config instructions are at
> Currently I am running a Linode 512. The new guild website will be running on the configuration I currently am working on (Apache Dynamic, Nginx static). Currently nothing for caching as I have never had a any sort of load on my server before. I plan on fixing this ASAP.
> The bandwidth usage isn't to bad. I have monitored the server logs during heavy Murmur use (10/25 man raids) and the server itself is hardly breaking a sweat and uses hardly any resources.
It sounds like with caching and nginx as a front you're going to be more than fine… Been a while since I used Wordpress but you should be able to find caching plugins for it that write static files to a folder as output, then have nginx serve from there, should fly on a 512!
From that point you hardly need to worry about much, I doubt you'll ever manage enough load to scare it if this chat server is as light as it sounds.