New setup very slow

Our of curiosity, what do the ab numbers look like when you match the -c setting to the MaxClients value (-n 1000 -c 10) ?

What does "vmstat 1" look like while you run the test?

-Chris

> Our of curiosity, what do the ab numbers look like when you match the -c setting to the MaxClients value (-n 1000 -c 10) ?

As expected (I suppose) blazing fast!

              min  mean[+/-sd] median   max
Connect:       88   91   1.9     90     104
Processing:   247  322  63.5    304     645
Waiting:      232  304  62.8    286     629
Total:        336  412  63.7    395     736

> What does "vmstat 1" look like while you run the test?
With 10 concurrent connections:

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 0  0   2744  80316  25936 147064    0    0     0     0   66   20  0  0 100  0
 5  0   2744  68012  25936 147064    0    0     0     0 4204 2032 61  3 36  0
 3  0   2744  51532  25936 147064    0    0     0     0 4151 2112 81  5 14  0
 6  0   2744  33916  25936 147064    0    0     0     0 4437 2225 89  5  6  0
 4  0   2744  33496  25936 147068    0    0     0     0 4566 2407 90  5  4  0
 8  0   2744  29704  25936 147068    0    0     0     0 4544 2148 90  4  6  0
 7  0   2744  27348  25944 147076    0    0     0   372 4525 2450 91  5  4  0
 6  0   2744  28464  25944 147076    0    0     0     0 4588 2364 92  5  3  0

The bo ( what is 'bo' btw? Ok, blocks out. :oops: ) spike cycles after that every 5-6 readouts until the test stops.

With 100 concurrent connections:

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 0  0   2744  78828  26096 147164    0    0     0     0   14   16  0  0 100  0
 3  0   2744  65180  26096 147164    0    0     0     0 3065 1327 41  3 57  0
 7  0   2744  55232  26096 147164    0    0     0     0 3660 1908 71  3 26  0
 8  0   2744  43200  26096 147164    0    0     0     0 4597 2319 92  4  3  0
10  0   2744  20212  26096 147164    0    0     0     0 4548 2214 94  5  0  0
 8  0   2744  26908  26096 147168    0    0     0     0 4500 2215 94  4  2  0
10  0   2744  21452  26104 147168    0    0     0   368 4495 2086 95  4  0  0
 6  0   2744  25048  26104 147176    0    0     0     0 4532 2314 94  5  1  0

And the same spikes of 'bo' pattern. CPU usage seems to be higher (also logical) at 93-95%, where as before it was more variable between 89-94%.

That all seems to jive to me. The page you're testing takes, on average, 400ms for everything (connection, processing, rendering, etc). When testing using ab with a -c set beyond MaxClients, for example -c100, those connections are simply queued – so while those 10 Apache client slots are serving up requests, the other 90 are sitting there waiting, influencing your request time results.

Your Linode isn't disk bound. You have some memory left to play with, so you may want to try increasing MaxClients to 15 and running the benchmark again (with -c15) and see if you can sustain 15 clients without a drop in request time. Although from your vmstat output I don't think you're far off from the optimal setting -- since your CPU seems to max even with 10 concurrent clients…

Ask yourself: Where's the bottleneck - DB or apache/php (watch top during the test)? How can I decrease processing time? How to scale number of clients if processing time is fixed? What's more important to you: page rendering time or max number of current clients hitting php pages?

If this is a personal blog, I'd say you're doing just fine. Are you really going to be doing 26M hits/month of php files, not including associated hits for other images/meda (say 100-200M hits?).

-Chris

Thanks caker,

With higher MaxClients I hit swap as soon as I demand a bit from apache.

The bottle-neck is definitely php. Calling a plain file doesn't seem to bother apache much and gives much higher requests per second. So how do I decrease php's footprint? I've thought of killing the modules I don't need. I can always reactivate as needed. Will this benefit performance?

Also I can install a php accelerator. Any suggestions? Is it straight-forward? As in "apt-get install and forget about it" straight-forward?

But this still doesn't answer my question of whether I should switch to the worker MPM or if there is any other way to run apache prefork processes as users (or work-around that).

Again thanks for the help.

Keeping MaxClients low is the key to success with Apache; each active request (especially for dynamic content) eats a surprisingly large amount of memory. There are accelerators and whatnot out there; I think there's a module called SuperCache for WordPress. However, I don't use WordPress (and just barely use PHP), so that's outside my realm of expertise.

I do, however, use lighttpd instead of Apache. There are tradeoffs, but its ability to not go fork-happy to handle a large number of concurrent connections has improved my quality of life. Plus, simple-vhost is simply genius. ;-)

You would have to use php-cgi instead of the built-in module, but it sounds like this might be something you're considering anyway: mod_php is a part of Apache, so it has to run as the same user. You can spawn php-cgi as any user you'd like; I run two instances with different usernames, since one of my payloads has to run as the same uid as another application (and its installation documents recommend setting Apache's uid to the other application's uid :shock: … no, thank you).

I also use memcached, which my primary payload supports quite effectively. This cuts down on a lot of the "thinking" that has to be done for routine hits. It does, however, require heavy integration with the application and (by definition) eats a quantity of memory. Worth it, though.

As others have noted, the average request processing time is going to depend on the # of concurrent requests. What isn't going to change as much as you vary concurrent users (if you have things tuned right) is the average # of requests per second.

I think the biggest thing you can do to improve Wordpress response time, during both periods of light and heavy load; and overall throughput is to use a PHP opcode cache. APC is packaged for Ubuntu. Other distros might package other options, like XCache or Eaccelerator. This eliminates per-request overhead that can easily work out to 50% of wordpress page generation time which will halve response times and double throughput. It comes at the cost of some memory, but that memory is shared among apache processes.

Its true that apache can use a lot of memory, but apache+php is a really well understood combo, so its worth looking at ways to make sure you get the most of the the memory apache+php use. I've done this by putting nginx in front of apache. Nginx is, like lighttpd, a very memory and cpu efficient web server. I have it serving all the static files, which means that I'm not wasting an heavyweight apache process. This is a big deal. My stylesheet takes something like 300ms to send to a client on a DSL connection. That's 300ms where the 20-30MB each apache+php worker uses is basically wasted. In that amount of time, one worker could render ~3 dynamic requests.

It also buffers the responses back from apache, which means that once apache is done generating the page it can move quickly to the next dynamic page request while nginx does the boring work of gzipping the response and sending it out to the client. Again, this is a big win. My home page HTML takes ~250ms to send, enough time to service 2.5 more requests.

Taken together, relying on nginx for static requests and buffering of dynamically generated responses probably means I can get by with 1/10th as many apache processes which leaves a lot more memory free for caching data. This isn't going to be a big deal when my site is lightly loaded.

So, my suggestion to you is install APC, then think about setting up nginx as a front end. I can post my config if you are interested.

Thanks eas,

Yes, I already installed APC and noticed some improvement, but not so much as reported elsewhere; I suppose it depends on your scripts or setup.

My system seems quite stable and handles the traffic it gets very well, with room for an occasional burst. I suppose until a real world™ test (a digg or the sorts) we won't know how well it handles pressure.

I'll keep your Nginx tip in mind if I ever encounter problems or need some extra performance.

> So, my suggestion to you is install APC, then think about setting up nginx as a front end. I can post my config if you are interested.

Please post your config, I'd love to take a look at this as an option. Lighttpd alone just isn't cutting it. An app I use just relies on Apache and the way it does stuff too much.

Ok, so here are the relevant details for how I'm proxying to Apache through nginx.

I'm on ubuntu 9.04. Nginx isn't in the official repositories, but there is a recent version in Universe, so I edited /etc/apt/sources to enable those repositories. I'm running apache2 with mpm-prefork and their packaged php5 and apc modules. I'm sure you can get to the same starting point on other distributions with a little googling.

Starting from the default config, I disabled mod_deflate on Apache, since I want nginx to handle compression. I then enabled the rpaf module so that apache reads the headers nginx sets in order to find out the real client IP for logging, etc.

Next, I edited /etc/apache2/ports.conf to have Apache listen on port 8080, rather than port 80. I will probably go further and just have it listen on localhost.

Similarly, for each VirtualHost entry in /etc/apache2/sites-enabled/*, I set the listening port to 8080.

For nginx, starting with the provided config files:

/etc/nginx/nginx.conf

user www-data;
worker_processes  1;

error_log  /var/log/nginx/error.log;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    access_log    /var/log/nginx/access.log;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;
    tcp_nodelay        on;

    gzip  on;

    server_names_hash_bucket_size 64; # added this because some of my domain names were too long.
    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

The nginx package for ubuntu follows the pattern used for apache (I think this approach comes from Debian) of breaking peices of config out into files and including whole directories, so I took advantage of this to break out my installation wide tweaks for compression and reverse proxying.

/etc/nginx/conf.d/gzip.conf

#by default, nginix only gzips text/html, and maybe text/plain
gzip_types    text/html application/javascript application/x-javascript text/javascript text/css text/xml text/plain application/atom+xml;
gzip_min_length    512;    #probably not that important, but it didn't seem worth even trying to gzip small responses.

/etc/nginx/conf.d/

proxy_redirect          off;
proxy_set_header        Host            $host;
proxy_set_header        X-Real-IP       $remote_addr;
proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_buffering        on;
proxy_connect_timeout   90;
proxy_send_timeout      90;
proxy_read_timeout      90;

/etc/nginx/sites-enabled/mysite:

server {
    listen       72.14.177.137:80;  # your server's public IP address
    server_name  mysite.com www.mysite.com;      # your domain name

    access_log  /var/log/nginx/mysite.access.log;

    location / {
        root   /var/www/mysite;  # absolute path to your WordPress installation
        index  index.php index.html index.htm;

        # makes sure that all .php files are served by apache
        if ($request_filename ~* "\.php$") {
            break;
        }

        # this serves static files that exist without running other rewrite tests
        if (-f $request_filename) {
            #expires 30d;
            break;
        }

        # this sends all non-existing file or directory requests to apache
        if (!-e $request_filename) {
            proxy_pass      http://localhost:8080;
        }
    }

    location ~ \.php$ {
        proxy_pass  http://local   host:8080;   
    }
}

I derived this from someone elses config for fcgi that I found on line. I think it could probably be a little cleaner, but I was deliberately a little cautious.

That's pretty much it. Restart apache, start nginix and you should be good to go. You can check the response headers and the server log files to make sure requests are being handled as expected.

Oh, I did tweak the apache config a little more to reduce the maximum # of apache processes to 6 or 8 or so. I figured at least 4 so there could be one running on each core, and then some extras in case there were any waiting for I/O. Not too many though, because it wastes memory and leads to greater variance in response times.

I also ended up tweaking my apc configuration a little to give it more memory and to enable expiration of cached items (by default it would clear the whole cache if it filled).

There is more tweaking you can do. For example, you can set some cache header policies for both static and dynamic content, and nginx can also cache the output of dynamic pages and serve the cached copy without hitting apache.

I think I found a simpler ruleset for /etc/nginx/sites-enabled/mysite:

server {

    listen   80;
    server_name xxxxxxxx.com;

    access_log /var/www/log/access.log;
    error_log /var/www/log/error.log;

    location / {
        proxy_pass http://127.0.0.1:8080;
    }

    location ~* ^.+.(jpe?g|gif|png|ico|css|zip|tgz|gz|rar|bz2|doc|xls|exe|pdf|ppt|txt|tar|mid|midi|wav|bmp|rtf|js|swf|avi|mp3)$ {
        expires 30d;
        root /var/www/;
    }

}

It seems to work. I disabled Apache and the site went down the images were still being served.

You can't just disable Apache.

These rules are used to proxy requests from nginx @ port 80 to apache @ port 8080.

You must setup apache to listen to port 8080 and you should use module rpaf to pass the real ip to apache logs.

Yes, you can disable Apache — for testing purposes.

sudo /etc/init.d/apache2 stop

I wanted to make sure that nginx was serving up static files using the rules that I mentioned. I stopped Apache. My sites went down, but direct links to static files still worked. Test done.

Sorry if it sounded like I was confused. :D