nginx+php-fpm+wp = cpu spike 504 gateway error

I have a nginx + php+fpm Linode server with apc and running wordpress and w3 total cache.

The server was performing well (for a month without downtime) until yesterday where the CPU suddenly spiked, memory maxed out (1024MB) and load average of 20, 19.6, 18.4 (quad core). I have to restart the php-fpm every 5 minutes to keep it running.

After some meddling and troubleshooting, I have managed to identify the issue with the wordpress database. It is probably due to a corrputed database. The frontend was working fine without any issue, but everytime i save post in the backend, the CPU spike, php5-fpm process took over and crash the server. The problem is, whatever i do, it keeps returning a 504 gateway error.

When I save post in wp: 504 gateway error

when i import database in phpmyadmin: 504 gateway error

I have made plenty of changes to the nginx.conf, php.ini, http://www.conf and my.cnf and increase all the possible timeout and it doesn't seem to solve the 504 gateway error problem.

The situation now is that the site is still live, but I can't touch anything in the WordPress backend as it will cause the cpu to spike and server crash.

What actually went wrong? Can anyone help?

Below is my settings:

nginx.conf

# Generic startup file.
user damien;
worker_processes  4;

error_log  /var/log/nginx/error.log;
pid        /var/run/nginx.pid;

# Keeps the logs free of messages about not being able to bind().
#daemon     off;

events {
        worker_connections  1024;
}

http {
#       rewrite_log on;

        include mime.types;
        default_type       application/octet-stream;
        access_log         /var/log/nginx/access.log;
        sendfile           on;
#       tcp_nopush         on;
        keepalive_timeout  0;
#       tcp_nodelay        on;
#       gzip               on;
        client_max_body_size 200M;
        client_body_buffer_size 128k;
        client_header_buffer_size 256k;
        client_header_timeout 300;
        client_body_timeout 300;
        send_timeout 300;
        index index.php index.html index.htm;

        # Upstream to abstract backend connection(s) for PHP.
        upstream php {
#               server unix:/tmp/php-fpm.sock;
                server 127.0.0.1:9000;
        }
        server {
                location ~* wp-config.php {
                    deny all;
                }

        # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
                location ~ \.php$ {
                    try_files $uri =404;
                    fastcgi_split_path_info ^(.+\.php)(/.+)$;
                    fastcgi_pass 127.0.0.1:9000;
                # With php5-fpm:
                #fastcgi_pass unix:/var/run/php5-fpm.sock;
                    fastcgi_index index.php;
                    include fastcgi_params;
                    fastcgi_param   SCRIPT_FILENAME    $document_root$fastcgi_script_name;
                    fastcgi_param   SCRIPT_NAME        $fastcgi_script_name;
                    fastcgi_read_timeout 1800;
                    fastcgi_send_timeout 1800;
                    fastcgi_connect_timeout 1800;
                    fastcgi_buffer_size 128k;
                    fastcgi_buffers 4 256k;
                    fastcgi_busy_buffers_size 256k;
                    fastcgi_temp_file_write_size 256k;
                    fastcgi_intercept_errors on;
                }

                location ~ /\. {
                     access_log off;
                     log_not_found off;
                     deny all;
                }

                if ($host ~* ^www\.(.*)) {
                    set $remove_www $1;
                    rewrite ^(.*)$ http://$remove_www$1 permanent;
                }

        }

        include sites-enabled/*;
}

http://www.conf

pm = dynamic

pm.max_children = 35

pm.start_servers = 10

pm.min_spare_servers = 2

pm.max_spare_servers = 32

;pm.process_idle_timeout = 10s;

pm.max_requests = 400

my.conf

[client]
port        = 3306
socket        = /var/run/mysqld/mysqld.sock

[mysqld_safe]
socket        = /var/run/mysqld/mysqld.sock
nice        = 0

[mysqld]
user        = mysql
pid-file    = /var/run/mysqld/mysqld.pid
socket        = /var/run/mysqld/mysqld.sock
port        = 3306
basedir        = /usr
datadir        = /var/lib/mysql
tmpdir        = /tmp
lc-messages-dir    = /usr/share/mysql
skip-external-locking
log-error    = /var/log/mysql.err

bind-address        = 127.0.0.1

key_buffer        = 256M
max_allowed_packet    = 16M
thread_stack        = 192K
thread_cache_size       = 20

myisam-recover         = BACKUP
max_connections        = 80
table_cache            = 512
thread_concurrency     = 10

query_cache_limit    = 8M
query_cache_size        = 64M

myisam_sort_buffer_size = 32M
sort_buffer_size = 1M
read_buffer_size = 1M
join_buffer_size = 5M

log_slow_queries    = /var/log/mysql/mysql-slow.log
long_query_time = 2
log-queries-not-using-indexes

# * InnoDB
#
#skip-innodb
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
#
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem

max_heap_table_size  = 80M
tmp_table_size       = 80M

[mysqldump]
quick
quote-names
max_allowed_packet    = 16M

[mysql]
#no-auto-rehash    # faster start of mysql but no tab completition

[isamchk]
key_buffer        = 16M

[myisamchk]
key_buffer = 64M
sort_buffer_size = 64M
read_buffer = 16M
write_buffer = 16M

4 Replies

Would you say it started at around midnight UTC on July 1? If so, reboot. You've hit a bug related to leap-second handling.

Yes, the server problem start occurring at around UTC midnight, but I have already reboot two times and the same problem occur. What's next?

Is it the PHP process that seems to use CPU, or MySQL? Have you watched how things progress with something like htop?

-Tim

Things seem to be working fine now.

Both the mysql and php are causing issue. Even after I fix the leap second issue, the problem still persists.

What I did is to remove the php-apc and reinstall it. Additionally I also installed memcached and everything is back to normal and is running even faster. Seems that the php-apc module is the one causing trouble.

Reply

Please enter an answer
Tips:

You can mention users to notify them: @username

You can use Markdown to format your question. For more examples see the Markdown Cheatsheet.

> I’m a blockquote.

I’m a blockquote.

[I'm a link] (https://www.google.com)

I'm a link

**I am bold** I am bold

*I am italicized* I am italicized

Community Code of Conduct