Out of memory: server halts many times a day!

Hello,

I really don't know what to do so I hope to receive some help from you.

As many users, I have no many experience with unix system, anyway:

I have a linode 1024 with ubuntu 12.04 lts 64bit.

I installed apache2, mysql, php5, drupal, postfix, clamav and so on.

My server goes out of memory, and it happens quite often, 7 8 times a day, and all is stalled, I must reboot from linode.

I ran the lish console and I received a log in which it is clear that many apache processes were killed, like:


Out of memory: Kill process 11685 (apache2) score 24 or sacrifice child

Killed process 11685 (apache2) total-vm:431896kB, anon-rss:12616kB, file-rss:2992kB


Reading and searching in the web, first of all I installed nginx in front of Apache in order to have a webserver that can provide static page faster and allow apache to use less resources.

This works, but the server halts anyway.

Then I reduced some parameters in apache2.conf as follow:

StartServers 1

MinSpareServers 1

MaxSpareServers 5

ServerLimit 64

MaxClients 64

MaxRequestsPerChild 4000

The problem is still there…

I modified my ubuntu configuration in order to reboot any time there is an out of memory, and it happens quite often.

My problem is that I don't know where to find the log that make the server crash, because now if I connect and open the lish, I receive simply the login, in which I must put my ssh credentials in order to enter into the server. but where can I see the log?

At the end, some statistics:

This is the output of the free -m:

total used free shared buffers cached

Mem: 987 977 9 0 1 88

-/+ buffers/cache: 888 99

Swap: 2047 737 1310

and this is a "top" screen…

Tasks: 206 total, 1 running, 205 sleeping, 0 stopped, 0 zombie

Cpu(s): 0.9%us, 0.4%sy, 0.0%ni, 87.7%id, 11.0%wa, 0.0%hi, 0.0%si, 0.0%st

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

4648 www-data 20 0 826m 180m 67m D 8 18.2 5:07.67 apache2

469 root 20 0 0 0 0 S 2 0.0 0:46.87 kswapd0

2569 mysql 20 0 2142m 45m 1352 S 1 4.7 3:20.06 mysqld

1 root 20 0 24340 684 152 S 0 0.1 0:03.17 init

2 root 20 0 0 0 0 S 0 0.0 0:00.01 kthreadd

3 root 20 0 0 0 0 S 0 0.0 0:00.03 ksoftirqd/0

4 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/0:0

5 root 0 -20 0 0 0 S 0 0.0 0:00.00 kworker/0:0H

7 root RT 0 0 0 0 S 0 0.0 0:00.06 migration/0

8 root 20 0 0 0 0 S 0 0.0 0:00.00 rcu_bh

9 root 20 0 0 0 0 S 0 0.0 0:20.04 rcu_sched

10 root RT 0 0 0 0 S 0 0.0 0:00.02 migration/1

11 root 20 0 0 0 0 S 0 0.0 0:00.03 ksoftirqd/1

12 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/1:0

13 root 0 -20 0 0 0 S 0 0.0 0:00.00 kworker/1:0H

14 root RT 0 0 0 0 S 0 0.0 0:00.02 migration/2

15 root 20 0 0 0 0 S 0 0.0 0:00.01 ksoftirqd/2

16 root 20 0 0 0 0 S 0 0.0 0:00.20 kworker/2:0

17 root 0 -20 0 0 0 S 0 0.0 0:00.00 kworker/2:0H


can you help me please?

thank you!

6 Replies

MaxClients of 64 is too much for a 1024, if you could use every bit of ram for apache you'd only allow 16MB per client. Drop it to say 5 and increase it gradually until you reach a good balance between site performance and ram in use.

I put maxclients to 8, and remove the serverlimit since I don't think have sense with prefork, Still server completely locked!

do you know where to look for log file that explain the error?

Check your mysql settings, looks like it's taking up way too much ram

for sql I really don't know how to adjust the settings.

Should I change also the default? If it is important I'm using drupal.

key_buffer = 16M

maxallowedpacket = 16M

thread_stack = 192K

threadcachesize = 8

myisam-recover = BACKUP

max_connections = 100

table_cache = 64

thread_concurrency = 10

Run this https://raw.github.com/major/MySQLTuner … qltuner.pl">https://raw.github.com/major/MySQLTuner-perl/master/mysqltuner.pl to get an idea on what to set your mysql settings to.

I would use nginx in replacement for apache and not on top of it.

If you're looking for a reverse caching option, have a look into Varnish.

It could also be likely that you're over committing various 'aspects' of your config, thus your system eventually runs dry of memory. Also take a look into scripts you're running on your site. A year back I had a caching script and a translating script on joomla that killed my server. I was sure it was a settings issue but it wasn't.

Also try getting some monitoring for your system to see what's happening. Munin is simple and it works well.

Good luck

Reply

Please enter an answer
Tips:

You can mention users to notify them: @username

You can use Markdown to format your question. For more examples see the Markdown Cheatsheet.

> I’m a blockquote.

I’m a blockquote.

[I'm a link] (https://www.google.com)

I'm a link

**I am bold** I am bold

*I am italicized* I am italicized

Community Code of Conduct