Unstable system : Low memory and High Disk I/O

2 days ago, I shifted my wordpress based blog from a shared server to linode. Since shifting, 2-3 times in a day, site becomes unstable and I am not able to login with SSH.

Load average spikes up, no memory left in system, Disk I/O goes high as swap is being used.

I had to restart server from linode login at each occasion.

Server config: Linode 512 with LAMP with Ubuntu 10.04 TLS (Deployed using StackScript: ~~[http://www.linode.com/stackscripts/view/?StackScriptID=10" target="_blank">](http://www.linode.com/stackscripts/view … criptID=10">http://www.linode.com/stackscripts/view/?StackScriptID=10]()

Applications running: Wordpress blog with not more than 2000-3000 page-views in a day.

Sever processes: Apache, Mysql and Postfix

Maybe its just a mysql/apache config issue. Same blog was running fine earlier on a shared host.

Can somebody help me overcome this issue.

MySQL config:

#
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
# 
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html

# This will be passed to all mysql clients
# It has been reported that passwords should be enclosed with ticks/quotes
# escpecially if they contain "#" chars...
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.
[client]
port        = 3306
socket        = /var/run/mysqld/mysqld.sock

# Here is entries for some specific programs
# The following values assume you have at least 32M ram

# This was formally known as [safe_mysqld]. Both versions are currently parsed.
[mysqld_safe]
socket        = /var/run/mysqld/mysqld.sock
nice        = 0

[mysqld]

key_buffer = 144M
sort_buffer_size = 4M
read_buffer_size = 4M
read_rnd_buffer_size = 4M
myisam_sort_buffer_size = 8M
query_cache_size = 28M

skip-locking

skip-innodb

#
# * Basic Settings
#

#
# * IMPORTANT
#   If you make changes to these settings and your system uses apparmor, you may
#   also need to also adjust /etc/apparmor.d/usr.sbin.mysqld.
#

user        = mysql
socket        = /var/run/mysqld/mysqld.sock
port        = 3306
basedir        = /usr
datadir        = /var/lib/mysql
tmpdir        = /tmp
skip-external-locking
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address        = 127.0.0.1
#
# * Fine Tuning
#
max_allowed_packet      = 16M
thread_cache_size       = 8
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover         = BACKUP
#max_connections        = 100
table_cache            = 64
#thread_concurrency     = 10
#
# * Query Cache Configuration
#
#query_cache_limit    = 1M
#query_cache_size        = 16M
#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the log at runtime!
#general_log_file        = /var/log/mysql/mysql.log
#general_log             = 1

log_error                = /var/log/mysql/error.log

# Here you can see queries with especially long duration
slow_query_log_file    = /var/log/mysql/mysql-slow.log
#long_query_time = 2
#log-queries-not-using-indexes
#
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
#       other settings you may need to change.
#server-id        = 1
#log_bin            = /var/log/mysql/mysql-bin.log
expire_logs_days    = 10
max_binlog_size         = 100M
#binlog_do_db        = include_database_name
#binlog_ignore_db    = include_database_name
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
#
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem

[mysqldump]
quick
quote-names
max_allowed_packet    = 16M

[mysql]
#no-auto-rehash    # faster start of mysql but no tab completition

[isamchk]
key_buffer        = 16M

#
# * IMPORTANT: Additional settings that can override those from this file!
#   The files must end with '.cnf', otherwise they'll be ignored.
#
!includedir /etc/mysql/conf.d/

Apache Config:

#
# Based upon the NCSA server configuration files originally by Rob McCool.
#
# This is the main Apache server configuration file.  It contains the
# configuration directives that give the server its instructions.
# See http://httpd.apache.org/docs/2.2/ for detailed information about
# the directives.
#
# Do NOT simply read the instructions in here without understanding
# what they do.  They're here only as hints or reminders.  If you are unsure
# consult the online docs. You have been warned.  
#
# The configuration directives are grouped into three basic sections:
#  1\. Directives that control the operation of the Apache server process as a
#     whole (the 'global environment').
#  2\. Directives that define the parameters of the 'main' or 'default' server,
#     which responds to requests that aren't handled by a virtual host.
#     These directives also provide default values for the settings
#     of all virtual hosts.
#  3\. Settings for virtual hosts, which allow Web requests to be sent to
#     different IP addresses or hostnames and have them handled by the
#     same Apache server process.
#
# Configuration and logfile names: If the filenames you specify for many
# of the server's control files begin with "/" (or "drive:/" for Win32), the
# server will use that explicit path.  If the filenames do *not* begin
# with "/", the value of ServerRoot is prepended -- so "/var/log/apache2/foo.log"
# with ServerRoot set to "" will be interpreted by the
# server as "//var/log/apache2/foo.log".
#

### Section 1: Global Environment
#
# The directives in this section affect the overall operation of Apache,
# such as the number of concurrent requests it can handle or where it
# can find its configuration files.
#

#
# ServerRoot: The top of the directory tree under which the server's
# configuration, error, and log files are kept.
#
# NOTE!  If you intend to place this on an NFS (or otherwise network)
# mounted filesystem then please read the LockFile documentation (available
# at <url:http: httpd.apache.org="" docs-2.1="" mod="" mpm_common.html#lockfile="">);
# you will save yourself a lot of trouble.
#
# Do NOT add a slash at the end of the directory path.
#
ServerRoot "/etc/apache2"

#
# The accept serialization lock file MUST BE STORED ON A LOCAL DISK.
#
# <ifmodule !mpm_winnt.c=""># <ifmodule !mpm_netware.c="">LockFile /var/lock/apache2/accept.lock
#</ifmodule>
#</ifmodule>

#
# PidFile: The file in which the server should record its process
# identification number when it starts.
# This needs to be set in /etc/apache2/envvars
#
PidFile ${APACHE_PID_FILE}

#
# Timeout: The number of seconds before receives and sends time out.
#
Timeout 300

#
# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
#
KeepAlive On

#
# MaxKeepAliveRequests: The maximum number of requests to allow
# during a persistent connection. Set to 0 to allow an unlimited amount.
# We recommend you leave this number high, for maximum performance.
#
MaxKeepAliveRequests 100

#
# KeepAliveTimeout: Number of seconds to wait for the next request from the
# same client on the same connection.
#
KeepAliveTimeout 2

##
## Server-Pool Size Regulation (MPM specific)
## 

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
 <ifmodule mpm_prefork_module="">StartServers 1
    MinSpareServers 1
    MaxSpareServers 5
    ServerLimit 50
    MaxClients 13
    MaxRequestsPerChild 500</ifmodule> 

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
 <ifmodule mpm_worker_module="">StartServers          2
    MinSpareThreads      25
    MaxSpareThreads      75 
    ThreadLimit          64
    ThreadsPerChild      25
    MaxClients          13
    MaxRequestsPerChild   500</ifmodule> 

# event MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
 <ifmodule mpm_event_module="">StartServers          2
    MaxClients          13
    MinSpareThreads      25
    MaxSpareThreads      75 
    ThreadLimit          64
    ThreadsPerChild      25
    MaxRequestsPerChild   500</ifmodule> 

# These need to be set in /etc/apache2/envvars
User ${APACHE_RUN_USER}
Group ${APACHE_RUN_GROUP}

#
# AccessFileName: The name of the file to look for in each directory
# for additional configuration directives.  See also the AllowOverride
# directive.
#

AccessFileName .htaccess

#
# The following lines prevent .htaccess and .htpasswd files from being 
# viewed by Web clients. 
#
 <files ~="" "^\.ht"="">Order allow,deny
    Deny from all
    Satisfy all</files> 

#
# DefaultType is the default MIME type the server will use for a document
# if it cannot otherwise determine one, such as from filename extensions.
# If your server contains mostly text or HTML documents, "text/plain" is
# a good value.  If most of your content is binary, such as applications
# or images, you may want to use "application/octet-stream" instead to
# keep browsers from trying to display binary files as though they are
# text.
#
DefaultType text/plain

#
# HostnameLookups: Log the names of clients or just their IP addresses
# e.g., www.apache.org (on) or 204.62.129.132 (off).
# The default is off because it'd be overall better for the net if people
# had to knowingly turn this feature on, since enabling it means that
# each client request will result in AT LEAST one lookup request to the
# nameserver.
#
HostnameLookups Off

# ErrorLog: The location of the error log file.
# If you do not specify an ErrorLog directive within a <virtualhost># container, error messages relating to that virtual host will be
# logged here.  If you *do* define an error logfile for a <virtualhost># container, that host's errors will be logged there and not here.
#
ErrorLog /var/log/apache2/error.log

#
# LogLevel: Control the number of messages logged to the error_log.
# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
#
LogLevel warn

# Include module configuration:
Include /etc/apache2/mods-enabled/*.load
Include /etc/apache2/mods-enabled/*.conf

# Include all the user configurations:
Include /etc/apache2/httpd.conf

# Include ports listing
Include /etc/apache2/ports.conf

#
# The following directives define some format nicknames for use with
# a CustomLog directive (see below).
# If you are behind a reverse proxy, you might want to change %h into %{X-Forwarded-For}i
#
LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %O" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

#
# Define an access log for VirtualHosts that don't define their own logfile
CustomLog /var/log/apache2/other_vhosts_access.log vhost_combined

# Include of directories ignores editors' and dpkg's backup files,
# see README.Debian for details.

# Include generic snippets of statements
Include /etc/apache2/conf.d/

# Include the virtual host configurations:
Include /etc/apache2/sites-enabled/</virtualhost></virtualhost></url:http:> 

6 Replies

CPU Usage during last 24 hours:

~~![](<URL url=)http://i55.tinypic.com/a9u1iu.png" />

Disk I/O during last 24 hours:

~~![](<URL url=)http://i55.tinypic.com/14jyikp.png" />

Network during last 24 hours:

![](" />~~~~

key_buffer = 144M

Just did a quick read through of your configs, and this line jumped out at me. For comparison, my matching line sets key_buffer at 1M…

@akerl:

key_buffer = 144M

Just did a quick read through of your configs, and this line jumped out at me. For comparison, my matching line sets key_buffer at 1M…

I have made these changes(key_buffer = 1M) and reloaded mysql. Lets see how it goes.

If the WordPress blog has lots of entries, a slightly bigger key_buffer might be appropriate. I think the default is 16M, which is small enough not to cause any issues on today's Linodes. But 1M is also OK for most small blogs.

If you can log in via SSH while one of those low-memory things is happening, run top and press Shift+M to sort your processes by memory usage. Post a screenshot here, and we'll be able to figure out what's eating all your RAM.

I suppose you already have one of the more aggressive caching plugins installed? e.g. Super Cache or Total Cache. You can't run a stable blog without one of those.

MySQL comes with a variety of config files intended for different memory usage levels, so picking the right one of those is a pretty good first step. Most people shouldn't mess around with the config file beyond that.

I'm not an Apache wiz (I use lighttpd), but if you're using mpm_prefork, then your Apache settings look quite reasonable.

I'd suggest more or less what hybinet said, but one trick you can use is to start up top sorted by RAM usage and leave that open in an SSH session. Then, when the server gets into trouble, you'll be able to see what RAM usage is like at the point of failure. Even if you get disconnected, your SSH client will presumably still show you the last thing it saw before the end.

Alternatively, you could start up top over lish, which removes SSH from the equation.

@akerl:

key_buffer = 144M

Just did a quick read through of your configs, and this line jumped out at me. For comparison, my matching line sets keybuffer at 1M… keybuffer normally should be set to 25% of RAM size, so 1M is something…

144M is normal value, pankajbatra.

But!

Other settings:

query_cache_size = 28M
sort_buffer_size = 4M 
read_buffer_size = 4M

all of them will be multiplied *max_connections. And you have default value

max_connections        = 100

Now count: (28M+4M+4M)*100=3.6Gb already, and not all buffers was counted.

So, if my advice not too late, decrease max_connections and use

this script http://www.day32.com/MySQL/tuning-primer.sh to check, if your memory usage by MySQL are correct.

Reply

Please enter an answer
Tips:

You can mention users to notify them: @username

You can use Markdown to format your question. For more examples see the Markdown Cheatsheet.

> I’m a blockquote.

I’m a blockquote.

[I'm a link] (https://www.google.com)

I'm a link

**I am bold** I am bold

*I am italicized* I am italicized

Community Code of Conduct