OOM on Linode1024

Hi, I got up in the morning and noticed that my Linode was down for 6 hours. Linode support has told me that my Linode kernel panicked as a result of running out of memory.

> You'll want to update to the latest 3.0 kernel by editing your active configuration profile and selecting Latest 3.0, then rebooting your Linode.

I did this and rebooted my Linode.

> Please note that this will only alleviate the problem of kernel panicking; you'll want to identify the culprit that caused your Linode to OOM and adjust its configuration to prevent it from happening again.

I'm relatively new to Linux (CentOS) and I would like to ask for some help identifying what the cause of the problem is.

free -m yields this output:

             total       used       free     shared    buffers     cached
Mem:          1004        407        597          0         10        156
-/+ buffers/cache:        239        764
Swap:          255          0        255

ps -eo pmem,pcpu,rss,vsize,args | sort -k 1 -r | less

yields this output:

%MEM %CPU   RSS    VSZ COMMAND
 2.8  6.7 29496 125076 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-external-locking --socket=/var/lib/mysql/mysql.sock
 1.9  0.2 20520  35180 /usr/sbin/httpd
 1.9  0.1 20516  35164 /usr/sbin/httpd
 1.8  0.3 18848  33388 /usr/sbin/httpd
 1.8  0.2 19152  33848 /usr/sbin/httpd
 1.8  0.2 18980  33632 /usr/sbin/httpd
 1.8  0.2 18840  33856 /usr/sbin/httpd
 1.8  0.2 18648  33360 /usr/sbin/httpd
 1.8  0.1 19420  34104 /usr/sbin/httpd
 1.8  0.1 19180  33888 /usr/sbin/httpd
 1.8  0.1 18860  33880 /usr/sbin/httpd
 1.8  0.1 18608  33644 /usr/sbin/httpd
 1.8  0.0 18820  33576 /usr/sbin/httpd
 1.7  0.1 18456  33104 /usr/sbin/httpd
 1.7  0.1 18384  33364 /usr/sbin/httpd
 1.7  0.1 18372  33360 /usr/sbin/httpd
 1.7  0.1 18324  33344 /usr/sbin/httpd
 0.8  0.0  9068  23928 /usr/sbin/httpd
 0.4  0.0  4548   4552 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
 0.3  0.0  3248  10100 sshd: root@ttyp0

Any help would be appreciated.

5 Replies

From what you posted, your server looks OK for now.

What are you running on the Linode, e.g. WordPress, Drupal, phpBB, etc? Does any of that involve image processing, like generating thumbnails from large photos?

Is there anything else running on the server, e.g. dovecot?

Please post your httpd.conf and my.cnf. Those are the usual suspects.

This is my my.cnf

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
# Default to using old password format for compatibility with mysql 3.x
# clients (those using the mysqlclient10 compatibility package).
#old_passwords=1

# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
# symbolic-links=0

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

[mysqld]
# someother settings
socket=/var/lib/mysql/mysql.sock

#added these lines for foreign characters
[client]
socket=/var/lib/mysql/mysql.sock
default-character-set=utf8
[mysqld]
default-character-set = utf8
skip-character-set-client-handshake
character-set-server = utf8
collation-server = utf8_general_ci

init-connect = "SET collation_connection = utf8_general_ci"
init-connect = "SET NAMES utf8"

This is a part of my httpd.conf

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# ServerLimit: maximum value for MaxClients for the lifetime of the server
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
 <ifmodule prefork.c="">StartServers       8
MinSpareServers    5
MaxSpareServers   20
ServerLimit      256
MaxClients       256
MaxRequestsPerChild  4000</ifmodule> 

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
 <ifmodule worker.c="">StartServers         2
MaxClients         150
MinSpareThreads     25
MaxSpareThreads     75
ThreadsPerChild     25
MaxRequestsPerChild  0</ifmodule> 

I am running Wordpress and some custom made sites with Perl. I don't use any image processing at all.

I do a lot of MySQL database operations but I'm not sure if that's the cause.

Any thoughts on my configurations?

You seem to have left out a huge chunk of my.cnf. Where's the part with key_buffer and all those memory usage settings?

But never mind, here's your most likely culprit.

 <ifmodule prefork.c="">StartServers       8
MinSpareServers    5
MaxSpareServers   20
ServerLimit      256
MaxClients       256</ifmodule> 

That should be changed to something like:

 <ifmodule prefork.c="">StartServers       4
MinSpareServers    4
MaxSpareServers    8
ServerLimit       24
MaxClients        24</ifmodule> 

Restart Apache, and you'll probably not see another OOM for a long time.

Thank you so much. I applied these changes and hope the OOM issue is completely gone.

My suggestion is start monitoring the instance so you know it's not responding. Something like nagios will work.

Reply

Please enter an answer
Tips:

You can mention users to notify them: @username

You can use Markdown to format your question. For more examples see the Markdown Cheatsheet.

> I’m a blockquote.

I’m a blockquote.

[I'm a link] (https://www.google.com)

I'm a link

**I am bold** I am bold

*I am italicized* I am italicized

Community Code of Conduct