slow nginx problem

I have 3 wordpress blogs (not very active). I moved one over a year ago to my linode and the page load time (using W3 Total Cache) dropped to between 0.5 and 1 sec, while the 2 remaining at Hostgator had page load times between 1.5 and 2 seconds. This week I moved the other 2. After the first the page load time also dropped to below 1 second, but 1 day after I moved the third page load times for all 3 are over 2 seconds. I have maybe 400 visitors a day (in total). Maybe it is not my server, maybe Linode or my ISP experiences lots of traffic, but I want to start looking what could be wrong.

I can't find any error messages in the log files (nginx, vhosts, php5-fpm).

I have been using http://whichloadsfaster.com/ for the page load times.

My default vhost config:

server {
    listen    80;
    server_name example.net;
    rewrite ^ http://www.example.net$request_uri permanent;
}

server {
    listen   80;
    server_name www.example.net;
    access_log /var/log/nginx/access-www.example.net.log;
    error_log /var/log/nginx/error-www.example.net.log;

    limit_conn zxmplnet 15;
    root /srv/www/example.net/www;
    location / {
        try_files $uri $uri/ /index.php?$args;
    }

    include /etc/nginx/loc_php.inc;

    location = /favicon.ico {
        log_not_found off;
        access_log off;
    }

    location = /robots.txt {
        allow all;
        log_not_found off;
        access_log off;
    }

    location ~ /\. {
        deny all;
        access_log off;
        log_not_found off;
    }

    location ~*  \.(jpg|jpeg|png|gif|css|js|ico)$ {
       expires max;
       log_not_found off;
    }

    rewrite /wp-admin$ $scheme://$host$uri/ permanent;

    location /nginx_status {
        stub_status on;
        access_log off;
    }

    include /srv/www/example.net/www/nginx.conf;

    location ~ /nginx.conf {
        deny all;
        access_log off;
        log_not_found off;
    }
}

the /etc/nginx/loc_php.inc:

location ~ \.php$ {
    include /etc/nginx/fastcgi_params;
    fastcgi_pass unix:/var/run/php5-fpm/www.socket;
        fastcgi_index index.php;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    fastcgi_read_timeout 1800;
    fastcgi_ignore_client_abort on;
}