Out of memory: server halts many times a day!
I really don't know what to do so I hope to receive some help from you.
As many users, I have no many experience with unix system, anyway:
I have a linode 1024 with ubuntu 12.04 lts 64bit.
I installed apache2, mysql, php5, drupal, postfix, clamav and so on.
My server goes out of memory, and it happens quite often, 7 8 times a day, and all is stalled, I must reboot from linode.
I ran the lish console and I received a log in which it is clear that many apache processes were killed, like:
Out of memory: Kill process 11685 (apache2) score 24 or sacrifice child
Killed process 11685 (apache2) total-vm:431896kB, anon-rss:12616kB, file-rss:2992kB
Reading and searching in the web, first of all I installed nginx in front of Apache in order to have a webserver that can provide static page faster and allow apache to use less resources.
This works, but the server halts anyway.
Then I reduced some parameters in apache2.conf as follow:
MinSpareServers 1
MaxSpareServers 5
ServerLimit 64
MaxClients 64
MaxRequestsPerChild 4000
The problem is still there…
I modified my ubuntu configuration in order to reboot any time there is an out of memory, and it happens quite often.
My problem is that I don't know where to find the log that make the server crash, because now if I connect and open the lish, I receive simply the login, in which I must put my ssh credentials in order to enter into the server. but where can I see the log?
At the end, some statistics:
This is the output of the free -m:
total used free shared buffers cached
Mem: 987 977 9 0 1 88
-/+ buffers/cache: 888 99
Swap: 2047 737 1310
and this is a "top" screen…
Tasks: 206 total, 1 running, 205 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.9%us, 0.4%sy, 0.0%ni, 87.7%id, 11.0%wa, 0.0%hi, 0.0%si, 0.0%st
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4648 www-data 20 0 826m 180m 67m D 8 18.2 5:07.67 apache2
469 root 20 0 0 0 0 S 2 0.0 0:46.87 kswapd0
2569 mysql 20 0 2142m 45m 1352 S 1 4.7 3:20.06 mysqld
1 root 20 0 24340 684 152 S 0 0.1 0:03.17 init
2 root 20 0 0 0 0 S 0 0.0 0:00.01 kthreadd
3 root 20 0 0 0 0 S 0 0.0 0:00.03 ksoftirqd/0
4 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/0:0
5 root 0 -20 0 0 0 S 0 0.0 0:00.00 kworker/0:0H
7 root RT 0 0 0 0 S 0 0.0 0:00.06 migration/0
8 root 20 0 0 0 0 S 0 0.0 0:00.00 rcu_bh
9 root 20 0 0 0 0 S 0 0.0 0:20.04 rcu_sched
10 root RT 0 0 0 0 S 0 0.0 0:00.02 migration/1
11 root 20 0 0 0 0 S 0 0.0 0:00.03 ksoftirqd/1
12 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/1:0
13 root 0 -20 0 0 0 S 0 0.0 0:00.00 kworker/1:0H
14 root RT 0 0 0 0 S 0 0.0 0:00.02 migration/2
15 root 20 0 0 0 0 S 0 0.0 0:00.01 ksoftirqd/2
16 root 20 0 0 0 0 S 0 0.0 0:00.20 kworker/2:0
17 root 0 -20 0 0 0 S 0 0.0 0:00.00 kworker/2:0H
can you help me please?
thank you!
6 Replies
do you know where to look for log file that explain the error?
Should I change also the default? If it is important I'm using drupal.
key_buffer = 16M
maxallowedpacket = 16M
thread_stack = 192K
threadcachesize = 8
myisam-recover = BACKUP
max_connections = 100
table_cache = 64
thread_concurrency = 10
If you're looking for a reverse caching option, have a look into Varnish.
It could also be likely that you're over committing various 'aspects' of your config, thus your system eventually runs dry of memory. Also take a look into scripts you're running on your site. A year back I had a caching script and a translating script on joomla that killed my server. I was sure it was a settings issue but it wasn't.
Also try getting some monitoring for your system to see what's happening. Munin is simple and it works well.
Good luck