Had MySQL killed b/c of low memory. PHP5-fpm uses 900+ MB.
I logged into my linode yesterday and realized MySQL wasn't on. It had been killed (from reading dmesg) because it was out of memory. My Swap was also 100%. I believe it was PHP fpm taking up all of the memory. I changed the configs and am curious if I am missing anything else to help performance so this won't happen again, especially at times when more traffic hits. Even with this config, Longview shows php5-fpm using 900MB-1GB of memory and I only have a 1 GB linode.
Current ubuntu status when i just logged in shows memory usage: 55% and swap usage: 14%, 155 processes.
I am currently doing a FPM pool for each subdomain and domain hosted (there's 2 main domains hosted). Is that best or should I actually just use 1?
Any help is appreciated! Sanitized configs below.
Nginx
user www-data;
worker_processes 8;
pid /var/run/nginx.pid;
events {
worker_connections 768;
# multi_accept on;
}
http {
##
# Basic Settings
##
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
# server_tokens off;
# server_names_hash_bucket_size 64;
# server_name_in_redirect off;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# Logging Settings
##
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
gzip_disable "msie6";
# gzip_vary on;
# gzip_proxied any;
# gzip_comp_level 6;
# gzip_buffers 16 8k;
# gzip_http_version 1.1;
# gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
FPM pool 1 (main site)
[domain.com]
listen = /var/run/php5-fpm/domain.com.socket
listen.backlog = -1
; Unix user/group of processes
user = myaccount
group = www-data
; Choose how the process manager will control the number of child processes.
pm = dynamic
pm.max_children = 25
pm.start_servers = 4
pm.process_idle_timeout = 10s
pm.min_spare_servers = 4
pm.max_spare_servers = 10
pm.status_path = /status
pm.max_requests = 500
security.limit_extensions = .php .html .run
; Pass environment variables
env[HOSTNAME] = $HOSTNAME
env[PATH] = /usr/local/bin:/usr/bin:/bin
env[TMP] = /tmp
env[TMPDIR] = /tmp
env[TEMP] = /tmp
PHP fpm pool 2 (admin, low usage subdomain)
[admin.domain.com]
listen = /var/run/php5-fpm/admin.domain.com.socket
listen.backlog = -1
; Unix user/group of processes
user = myaccount
group = www-data
; Choose how the process manager will control the number of child processes.
pm = dynamic
pm.max_children = 25
pm.start_servers = 2
pm.min_spare_servers = 2
pm.max_spare_servers = 20
pm.max_requests = 500
; Pass environment variables
env[HOSTNAME] = $HOSTNAME
env[PATH] = /usr/local/bin:/usr/bin:/bin
env[TMP] = /tmp
env[TMPDIR] = /tmp
env[TEMP] = /tmp
fpm pool 3 (admin subdomain, little more usage, still for public use)
[races.domain.com]
listen = /var/run/php5-fpm/races.domain.com.socket
listen.backlog = -1
; Unix user/group of processes
user = myaccount
group = www-data
; Choose how the process manager will control the number of child processes.
pm = dynamic
pm.max_children = 10
pm.start_servers = 2
pm.min_spare_servers = 2
pm.max_spare_servers = 10
pm.max_requests = 200
security.limit_extensions = .php .html .run
; Pass environment variables
env[HOSTNAME] = $HOSTNAME
env[PATH] = /usr/local/bin:/usr/bin:/bin
env[TMP] = /tmp
env[TMPDIR] = /tmp
env[TEMP] = /tmp
fpm pool domain 2, public use. wordpress
[domain2.com]
listen = /var/run/php5-fpm/domain2.com.socket
listen.backlog = -1
; Unix user/group of processes
user = myaccount
group = www-data
; Choose how the process manager will control the number of child processes.
pm = dynamic
pm.max_children = 25
pm.start_servers = 2
pm.min_spare_servers = 2
pm.max_spare_servers = 10
pm.process_idle_timeout = 10s
pm.max_requests = 500
; Pass environment variables
env[HOSTNAME] = $HOSTNAME
env[PATH] = /usr/local/bin:/usr/bin:/bin
env[TMP] = /tmp
env[TMPDIR] = /tmp
env[TEMP] = /tmp
fpm pool staging domain
[test.domain.com]
listen = /var/run/php5-fpm/test.domain.com.socket
listen.backlog = -1
; Unix user/group of processes
user = myaccount
group = www-data
; Choose how the process manager will control the number of child processes.
pm = dynamic
pm.max_children = 5
pm.start_servers = 1
pm.min_spare_servers = 1
pm.max_spare_servers = 5
pm.max_requests = 100
security.limit_extensions = .php
; Pass environment variables
env[HOSTNAME] = $HOSTNAME
env[PATH] = /usr/local/bin:/usr/bin:/bin
env[TMP] = /tmp
env[TMPDIR] = /tmp
env[TEMP] = /tmp
MySQL my.cnf
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
# Here is entries for some specific programs
# The following values assume you have at least 32M ram
# This was formally known as [safe_mysqld]. Both versions are currently parsed.
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
#
# * Basic Settings
#
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address = 127.0.0.1
#
# * Fine Tuning
#
key_buffer = 16M
max_allowed_packet = 1M
thread_stack = 64K
thread_cache_size = 8
sort_buffer = 64K
net_buffer_length = 2K
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover = BACKUP
#max_connections = 100
table_cache = 4
query_cache_limit = 1M
query_cache_size = 16M
log_error = /var/log/mysql/error.log
expire_logs_days = 10
max_binlog_size = 100M
[mysqldump]
quick
quote-names
max_allowed_packet = 16M
[mysql]
#no-auto-rehash # faster start of mysql but no tab completition
[isamchk]
key_buffer = 16M
#
# * IMPORTANT: Additional settings that can override those from this file!
# The files must end with '.cnf', otherwise they'll be ignored.
#
!includedir /etc/mysql/conf.d/
7 Replies
That said, running multiple pools seems like a mixed bag. Each pool is going to take X MB just to run, and then additional memory for the application code and extensions they load. With multiple pools, each pool might use less memory than they would if everything ran in the same pool, but that's going to be offset or overshadowed by the fact that you have more of them.
The bigger issue though, is that multiple pools make resource management more difficult. With a single pool, you can set a resource cap that limits the total amount of RAM taken up by all the php apps on the system and you can be confident that whatever app(s) are experiencing the highest load at any given time will be able to get as much RAM as possible without taking too much. With multiple pools, you have to make a static allocation of resources among pools. If you do that in a way that guarantees that PHP can never take up too much RAM, then you are also guaranteeing that in some situations, RAM resources will sit idle even when they are needed.
If I were you, I'd probably cut this down to one or two pools (the second for your admin stuff), and set things up so not more than ~24 children were allowed in the main pool, and maybe 4 or so in the admin pool. My thinking is that when I've done my own testing, there doesn't seem to be any point to having more than active 3-4 workers per CPU core (if even that). With that, I see ~100% usage of all cores under a load test. Once CPU utilization is maxed out, having more workers can actually work against you because the extra workers end up taking up more RAM, pushing stuff out of disk cache, which then requires more disk I/O, which slows things down, by requiring the CPUs to wait more for disk.
If you're already maxing things out now you should probably up-size your Linode to 2GB before your regular traffic starts going up, and before you deploy more sites or resources.
MSJ
@xcrunner529:
I finally convinced them to move to a Linode so I could start developing better, more modern apps and have more control over things such as mailing and the resources we get
And you did that with no testing, no first hand knowledge that it will work better, or anything?
A box we control allows me to use more modern platforms, do more server-side caching, run a more efficient web server, etc.
I run ~20 different php websites off one web front end, I have one pool.