[UPDATED] Apparently random crashes

Question

[UPDATED] Apparently random crashes

general

Hi!

In the last few weeks, I've been experimenting some random "crashes" of my Linode 1024. They happen every 9-13 days, at the same hour, about 8:00 GMT, and they cause the CPU, network and HD graphs at my Linode Dashboard to show values close to zero. The server doesn't respond from "outside", and the only possible solution is to reboot it, and this takes about 3 minutes.

It runs Ubuntu 10.04, with LAMP, Tomcat 6, Postfix and Dovecot. All of them are the last versions available, the software is totally up to date. I've checked some "system" logs, like syslog, daemon.log, etc., but they don't show anything relevant, they just "log" things until the crash moment.

I have this server since June, but the problems started in August. In addition, from the moment of the server purchase, I haven't made any software installation or similar. Other than versions changes, the server has the same software that was installed in the first time.

Any ideas?

Thanks in advance!

23 Replies

caker · Answer 1 · Sept. 7, 2011, 10:34 a.m.

caker 13 years, 3 months ago Linode Staff

Hello,

You need to do two things: One, make sure you're running the "Latest 3.0" kernel (Linode Manager -> Linode -> Config Profile -> Kernel drop-down, save, reboot). Second, when this happens, inspect your Lish console before you reboot to see if there has been a kernel BUG or OOM or something, or if your Linode is still 'alive' but something has taken networking down.

Hope that helps,

-Chris

forum:usr01 · Answer 2 · Sept. 7, 2011, 11:02 a.m.

forum:usr01 13 years, 3 months ago

Thanks for your reply. I've just changed the kernel to "Latest 3.0", and now all runs OK. However, about the Linode being "alive" or just the network, I must say that ALL the "system" logs that I've checked were "empty" for the few hours the server was crashed. Not just the network, it was the server, completely.

Should I check one particular log?

And let's suppose this happens again. When I access my server using Lish console at Manager, what should I watch?

Finally, in the case I can't access my Linode from Lish console as said, what could be the issue?

forum:glg · Answer 3 · Sept. 8, 2011, 8 p.m.

forum:glg 13 years, 3 months ago

Check your MaxClients in apache. the default is for systems with many gigabytes of RAM and is not appropriate for a linode. That is the cause of the vast majority of threads like yours.

forum:hybinet · Answer 4 · Sept. 8, 2011, 9:08 p.m.

forum:hybinet 13 years, 3 months ago

The console shows a lot of things that wouldn't be in any log file. For example, a panicked kernel can't write anything to a log file, but it can still spit out a message to the console before going down. Firewall messages often show up on the console, too.

A reboot will probably erase anything that was written to the console, though.

forum:usr01 · Answer 5 · Sept. 9, 2011, 4:03 a.m.

forum:usr01 13 years, 3 months ago

THanks to everyone!

Regarding Apache, I don't think it is a memory problem, because memory consumption is all the time around 30% of the total.

The next time this happens, if it does happen, I will access my server using Lish console and check for warning messages before rebooting, so that they aren't erased.

forum:mnordhoff · Answer 6 · Sept. 9, 2011, 12:58 p.m.

forum:mnordhoff 13 years, 3 months ago

Lish's logview command (detach from the screen session showing your console itself) shows the most recent lines of both your current boot and the previous one, so even if you have rebooted, the information is still there. Unless you've rebooted twice, anyway.

forum:usr01 · Answer 7 · Sept. 9, 2011, 2:03 p.m.

forum:usr01 13 years, 3 months ago

OK.

forum:usr01 · Answer 8 · Sept. 11, 2011, 5:50 a.m.

forum:usr01 13 years, 3 months ago

OK, the server has just crashed, and here is the output from Lish console as you described:

REMOVED. SEE http://lists.xensource.com/archives/htm … 01172.html">http://lists.xensource.com/archives/html/xen-devel/2011-08/msg01172.html FOR INFORMATION

I can't login, just opening the console throws those messages. And I can't do anything. Note that I changed MAC and IP addresses, because that isn't important.

In addition, I'm not the first one with this problem on the VPS:

http://forum.linode.com/viewtopic.php?t=7514

The graphs of my dashboard say that CPU load before crash was 20%, network was about 400 kbps and IO around 3.3 kblocks per sec. And, as I mentioned, memory is always about 30% all the time.

Clearly, I'm not the only customer of Linode that has this error. It would be nice if Linode, as it's a serious company, investigates to solve the problem

Thanks in advance!

forum:Internat · Answer 9 · Sept. 11, 2011, 6:22 a.m.

forum:Internat 13 years, 3 months ago

I dont think you quite understand what your being shown.

This is a kernel crash probably todo with your instance running out of memory. This isnt linode's fault.

Scroll up in that logview and im sure somewhere there will be out of memory error.

This means you need to tune your instance to not eat as much memory. Maybe run ab or a benchmarking utility agaisnt it to see if you can put enough pressure on it to cause the error to happen

Sophira · Answer 10 · Sept. 11, 2011, 6:31 a.m.

Sophira 13 years, 3 months ago

Also, a quick tip - to scroll up in the console where this is being shown, press Ctrl-A and release, then press Escape. You'll then be able to scroll up and down with the cursor keys and Page Up/Page Down. Press Escape again to go back to normal.

forum:usr01 · Answer 11 · Sept. 11, 2011, 6:31 a.m.

forum:usr01 13 years, 3 months ago

I'm sorry, I can't scroll up. How is that done? How can I use "logview"?

If I press Ctrl-A, it shows me a message saying "No other window". If I press Escape, then the message hides and I return to the normal console. I can't scroll with the mouse, or the Page Up/Down keys.

forum:usr01 · Answer 12 · Sept. 11, 2011, 6:46 a.m.

forum:usr01 13 years, 3 months ago

Well, I managed to show the contents of the Lish console:

REMOVED. SEE http://lists.xensource.com/archives/htm … 01172.html">http://lists.xensource.com/archives/html/xen-devel/2011-08/msg01172.html FOR INFORMATION.

It seems like it's an Apache issue… Could anyone translate the above code to English? Is it a memory error? IO? CPU? It seems it's something related to the memory, the stack, bit I don't know it 100% sure.

Thanks in advance!

forum:usr01 · Answer 13 · Sept. 11, 2011, 7:06 a.m.

forum:usr01 13 years, 3 months ago

It appears it's NOT an Apache issue:

http://www.gossamer-threads.com/lists/xen/devel/216324

This user has exactly the SAME problem as me, and… HE/SHE USES LINODE! Apparently, they haven't reached a solution, as they are not able to reproduce the error.

And in most forums in which users report this error, it appears that the problem is related with hardware. However, this is strange, as Linode uses virtual environments…

Finally, it's quite remarkable that, as I described, I have used Linode since June, but problems started in August, and they occur every few days. The forum post I linked is of August.

Hope we find a solution.

forum:hoopycat · Answer 14 · Sept. 11, 2011, 8:16 a.m.

forum:hoopycat 13 years, 3 months ago

I believe Linode might be a couple steps ahead of you, given that the xen-devel thread was started by a Linode engineer. :-)

Might be a good idea to open a ticket, to the attention of psandin, especially if you're willing to be a test pilot.

forum:usr01 · Answer 15 · Sept. 11, 2011, 10:32 a.m.

forum:usr01 13 years, 3 months ago

OK, I opened a ticket, and here are the two answers:

> Hello,

We're still looking in to the issue. We've enabled some more debugging options in the latest kernel builds to try and help locate where the problem is. The current thread can be found at:

http://lists.xensource.com/archives/htm … 01172.html">http://lists.xensource.com/archives/html/xen-devel/2011-08/msg01172.html

As indicated by the last post we're looking for affected customers to help debug the problem. If you would like to volunteer that would be great and we will let you know once we've got a specific plan worked out. If not the 2.6.32 and earlier kernels are known not to be affected.

Regards,

Peter

> Hello,

You are correct that this issue was not caused by a misconfiguration on your behalf. In the majority of cases that we have seen Apache seems to be involved but we haven't been able to pinpoint as to if Apache has something to do with this error or is just a causality. Downgrading to the 2.6.32 kernel will temporarily resolve this issue as we haven't seen this happen with that kernel. However, it will not help track down this bug and ultimately solve the problem at hand. I have notified our System administrators that you would like to help in resolving this issue so that we can take the necessary steps to get that in motion. Feel free to post this information on our forums if you would like. I hope this helps!

Regard

I must note that the first answer was made by psandin, the engineer that opened the forum post I linked. The second one was by lmatos.

The (my) conclusion:

If you have the same problem, notify it to Linode. Use kernel 2.6.32 version or lower to avoid the issue.
The error can't be solved by customers. It must be solved by engineers of Linode.
The cause is unknown. They're working on solving it.

I think it's important to notify it to Linode, in all cases, because they're working on the error and trying to solve it. And solving it will be good for everybody.

Thanks to all the replies.

caker · Answer 16 · Sept. 11, 2011, 11:38 a.m.

caker 13 years, 3 months ago Linode Staff

~~@usr01:~~

It would be nice if Linode, as it's a serious company, investigates to solve the problem
Of course we are… That xen-devel thread was us.

~~@usr01:~~

If you have the same problem, notify it to Linode. Use kernel 2.6.32 version or lower to avoid the issue.
This was bad advice, and I've informed our staff. 2.6.32 is unmaintained and eventually will not even be available. It's better to choose "Latest 3.0 x86_64", save, and reboot, since we know the 64 bit kernel is unaffected. 64 bit kernels can run 32 bit userspace just fine.

~~@usr01:~~

The error can't be solved by customers. It must be solved by engineers of Linode.
Everyone is responsible for solving problems in open source software. Only you can prevent forest fires.

~~@usr01:~~

The cause is unknown. They're working on solving it.
It's a software problem, most likely a race.

forum:usr01 · Answer 17 · Sept. 11, 2011, 12:38 p.m.

forum:usr01 13 years, 3 months ago

~~@caker:~~

~~@usr01:~~

It would be nice if Linode, as it's a serious company, investigates to solve the problem
Of course we are… That xen-devel thread was us.

I made that statement before knowing that thread was opened.

~~@caker:~~

This was bad advice, and I've informed our staff. 2.6.32 is unmaintained and eventually will not even be available. It's better to choose "Latest 3.0 x86_64", save, and reboot, since we know the 64 bit kernel is unaffected. 64 bit kernels can run 32 bit userspace just fine.

Sorry, I'm not an expert. What are the implications of changing a kernel to 64 bits for the rest of the software installed? We're talking about production environments…

~~@caker:~~

It's a software problem, most likely a race.

But the cause is unknown, isn't it?

~~@caker:~~

Everyone is responsible for solving problems in open source software. Only you can prevent forest fires.

Obviously, I didn't mean to say the opposite. What I wanted to say was that the errors weren't caused by the customers. Most people don't have the time/knowledge to directly solve a problem in Xen software's source code. The best we can do is help experts to solve the problem, and I said repeatedly that users with this problem should notify Linode support. Of course everybody must help to prevent forest fires, but only experts should go with the firetruck to extinguish the fire.

However the key question is: why doesn't Linode publish a post about this error instead or correcting/quoting posts written by customers like me? I asked for permission to the support guys before publishing the above tickets in this forum, so I feel a bit uncomfortable with a member of Linode's staff correcting me later.

forum:pclissold · Answer 18 · Sept. 11, 2011, 2:33 p.m.

forum:pclissold 13 years, 3 months ago

~~@usr01:~~

What are the implications of changing a kernel to 64 bits for the rest of the software installed? We're talking about production environments…
There are no implications. 32-bit userland programs will run under a 64-bit kernel without changes. The kernel might take up a little more memory.

forum:usr01 · Answer 19 · Sept. 12, 2011, 7 a.m.

forum:usr01 13 years, 3 months ago

OK, thank you.

forum:goospoos · Answer 20 · Sept. 24, 2011, 12:59 p.m.

forum:goospoos 13 years, 2 months ago

I think the issue is still there. I'm using Linode 512. Got below errors.

Below is logwatch log…

WARNING: Kernel Errors Present

[] ? mmfaulterror+0x130/0x130 …: 4 Time(s)

[] ? error_code+0x5a/0x60 …: 2 Time(s)

My kernel is "Latest 3.0(3.0.4-linode38)"

-Divyang Patel

forum:hoopycat · Answer 21 · Sept. 24, 2011, 1:53 p.m.

forum:hoopycat 13 years, 2 months ago

Could you paste in the full set of log output from that, usually found in /var/log/messages or /var/log/syslog? The kernel spits out a few dozen lines when something weird happens, which is not something logwatch handles well.

forum:goospoos · Answer 22 · Sept. 24, 2011, 2:30 p.m.

I don't know whether this all is useful . But still have a look at it.

Sep 24 11:28:45 li253-233 kernel: apache2 invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0
Sep 24 11:28:45 li253-233 kernel: Pid: 27874, comm: apache2 Not tainted 3.0.4-linode37 #1
Sep 24 11:28:45 li253-233 kernel: Call Trace:
Sep 24 11:28:45 li253-233 kernel: [<c0187ed6>] ? T.677+0x76/0x1b0
Sep 24 11:28:45 li253-233 kernel: [<c01063fb>] ? xen_restore_fl_direct_reloc+0x4/0x4
Sep 24 11:28:45 li253-233 kernel: [<c06bf891>] ? _raw_spin_unlock_irqrestore+0x11/0x20
Sep 24 11:28:45 li253-233 kernel: [<c045c367>] ? ___ratelimit+0x97/0x110
Sep 24 11:28:45 li253-233 kernel: [<c0178ad4>] ? __delayacct_freepages_end+0x24/0x30
Sep 24 11:28:45 li253-233 kernel: [<c018806e>] ? T.676+0x5e/0x150
Sep 24 11:28:45 li253-233 kernel: [<c013e24c>] ? has_capability_noaudit+0xc/0x20
Sep 24 11:28:45 li253-233 kernel: [<c0187d55>] ? oom_badness+0xb5/0x110
Sep 24 11:28:45 li253-233 kernel: [<c018839a>] ? out_of_memory+0x23a/0x300
Sep 24 11:28:45 li253-233 kernel: [<c018bcbb>] ? __alloc_pages_nodemask+0x61b/0x630
Sep 24 11:28:45 li253-233 kernel: [<c01ab477>] ? read_swap_cache_async+0xb7/0xf0
Sep 24 11:28:45 li253-233 kernel: [<c01ab511>] ? swapin_readahead+0x61/0x90
Sep 24 11:28:45 li253-233 kernel: [<c019dc90>] ? do_swap_page+0x4a0/0x5f0
Sep 24 11:28:45 li253-233 kernel: [<c019e01c>] ? handle_pte_fault+0x23c/0x2f0
Sep 24 11:28:45 li253-233 kernel: [<c019e5e4>] ? handle_mm_fault+0xf4/0x1b0
Sep 24 11:28:45 li253-233 kernel: [<c011f261>] ? do_page_fault+0x101/0x3a0
Sep 24 11:28:45 li253-233 kernel: [<c0477115>] ? info_for_irq+0x5/0x20
Sep 24 11:28:45 li253-233 kernel: [<c0477810>] ? evtchn_from_irq+0x10/0x40
Sep 24 11:28:45 li253-233 kernel: [<c0175771>] ? handle_percpu_irq+0x31/0x50
Sep 24 11:28:45 li253-233 kernel: [<c0105c27>] ? xen_force_evtchn_callback+0x17/0x30
Sep 24 11:28:45 li253-233 kernel: [<c0105c27>] ? xen_force_evtchn_callback+0x17/0x30
Sep 24 11:28:45 li253-233 kernel: [<c0106404>] ? check_events+0x8/0xc
Sep 24 11:28:45 li253-233 kernel: [<c01063fb>] ? xen_restore_fl_direct_reloc+0x4/0x4
Sep 24 11:28:45 li253-233 kernel: [<c017792e>] ? rcu_enter_nohz+0x2e/0xb0
Sep 24 11:28:45 li253-233 kernel: [<c011f160>] ? mm_fault_error+0x130/0x130
Sep 24 11:28:45 li253-233 kernel: [<c06c00e6>] ? error_code+0x5a/0x60
Sep 24 11:28:45 li253-233 kernel: [<c06b0000>] ? sctp_err_lookup+0xb0/0x110
Sep 24 11:28:45 li253-233 kernel: [<c011f160>] ? mm_fault_error+0x130/0x130
Sep 24 11:28:45 li253-233 kernel: Mem-Info:
Sep 24 11:28:45 li253-233 kernel: DMA per-cpu:
Sep 24 11:28:45 li253-233 kernel: CPU    0: hi:    0, btch:   1 usd:   0
Sep 24 11:28:45 li253-233 kernel: CPU    1: hi:    0, btch:   1 usd:   0
Sep 24 11:28:45 li253-233 kernel: CPU    2: hi:    0, btch:   1 usd:   0
Sep 24 11:28:45 li253-233 kernel: CPU    3: hi:    0, btch:   1 usd:   0
Sep 24 11:28:45 li253-233 kernel: Normal per-cpu:
Sep 24 11:28:45 li253-233 kernel: CPU    0: hi:  186, btch:  31 usd:  42
Sep 24 11:28:45 li253-233 kernel: CPU    1: hi:  186, btch:  31 usd:  61
Sep 24 11:28:45 li253-233 kernel: CPU    2: hi:  186, btch:  31 usd:  12
Sep 24 11:28:45 li253-233 kernel: CPU    3: hi:  186, btch:  31 usd:   3
Sep 24 11:28:45 li253-233 kernel: active_anon:58856 inactive_anon:58867 isolated_anon:151
Sep 24 11:28:45 li253-233 kernel: active_file:79 inactive_file:159 isolated_file:63
Sep 24 11:28:45 li253-233 kernel: unevictable:0 dirty:0 writeback:121 unstable:0
Sep 24 11:28:45 li253-233 kernel: free:1324 slab_reclaimable:502 slab_unreclaimable:3074
Sep 24 11:28:45 li253-233 kernel: mapped:85 shmem:5 pagetables:1169 bounce:0
Sep 24 11:28:45 li253-233 kernel: DMA free:2108kB min:84kB low:104kB high:124kB active_anon:2012kB inactive_anon:2040kB active_file:0kB inactive_file:16kB unevictable:0kB isolated(anon):16kB isolated(file):0kB present:15808kB mlocked:0kB dirty:0kB writeback:40kB mapped:8kB shmem:0kB slab_reclaimable:20kB slab_unreclaimable:504kB kernel_stack:184kB pagetables:236kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:62 all_unreclaimable? yes
Sep 24 11:28:45 li253-233 kernel: lowmem_reserve[]: 0 500 500 500
Sep 24 11:28:45 li253-233 kernel: Normal free:3188kB min:2816kB low:3520kB high:4224kB active_anon:233412kB inactive_anon:233428kB active_file:316kB inactive_file:620kB unevictable:0kB isolated(anon):588kB isolated(file):252kB present:512064kB mlocked:0kB dirty:0kB writeback:444kB mapped:332kB shmem:20kB slab_reclaimable:1988kB slab_unreclaimable:11792kB kernel_stack:1264kB pagetables:4440kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1356 all_unreclaimable? yes
Sep 24 11:28:45 li253-233 kernel: lowmem_reserve[]: 0 0 0 0
Sep 24 11:28:45 li253-233 kernel: DMA: 27*4kB 114*8kB 68*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2108kB
Sep 24 11:28:45 li253-233 kernel: Normal: 439*4kB 147*8kB 16*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3188kB
Sep 24 11:28:45 li253-233 kernel: 16861 total pagecache pages
Sep 24 11:28:45 li253-233 kernel: 16554 pages in swap cache
Sep 24 11:28:45 li253-233 kernel: Swap cache stats: add 7099735, delete 7083181, find 16078336/16665779
Sep 24 11:28:45 li253-233 kernel: Free swap  = 0kB
Sep 24 11:28:45 li253-233 kernel: Total swap = 262140kB
Sep 24 11:28:45 li253-233 kernel: 133104 pages RAM
Sep 24 11:28:45 li253-233 kernel: 0 pages HighMem
Sep 24 11:28:45 li253-233 kernel: 5833 pages reserved
Sep 24 11:28:45 li253-233 kernel: 5803 pages shared
Sep 24 11:28:45 li253-233 kernel: 124908 pages non-shared
Sep 24 11:28:45 li253-233 kernel: [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Sep 24 11:28:45 li253-233 kernel: [ 1033]     0  1033      592        0   0       0             0 upstart-udev-br
Sep 24 11:28:45 li253-233 kernel: [ 1035]     0  1035      559        0   1     -17         -1000 udevd
Sep 24 11:28:45 li253-233 kernel: [ 1440]     0  1440      558        0   1     -17         -1000 udevd
Sep 24 11:28:45 li253-233 kernel: [ 1471]     0  1471      558        0   0     -17         -1000 udevd
Sep 24 11:28:45 li253-233 kernel: [ 1919]     0  1919      572        0   0       0             0 dhclient3
Sep 24 11:28:45 li253-233 kernel: [ 1971]     0  1971      607       17   0       0             0 cron
Sep 24 11:28:45 li253-233 kernel: [ 1973]     0  1973      900        0   2       0             0 ntpd
Sep 24 11:28:45 li253-233 kernel: [ 1974]   103  1974      874        1   0       0             0 ntpd
Sep 24 11:28:45 li253-233 kernel: [ 1977]   101  1977     7454      262   1       0             0 rsyslogd
Sep 24 11:28:45 li253-233 kernel: [ 2072]     0  2072      460        0   0       0             0 getty
Sep 24 11:28:45 li253-233 kernel: [14026]     0 14026     1400       18   0     -17         -1000 sshd
Sep 24 11:28:45 li253-233 kernel: [20122]     0 20122     2873       40   0       0             0 sendmail-mta
Sep 24 11:28:45 li253-233 kernel: [ 1367]     0  1367     7724       46   2       0             0 apache2
Sep 24 11:28:45 li253-233 kernel: [27704]    33 27704    11908      909   0       0             0 apache2
Sep 24 11:28:45 li253-233 kernel: [27707]    33 27707    11923     1245   0       0             0 apache2
Sep 24 11:28:45 li253-233 kernel: [27874]    33 27874    11912     2305   1       0             0 apache2
Sep 24 11:28:45 li253-233 kernel: [27894]    33 27894    11912     1069   3       0             0 apache2
Sep 24 11:28:45 li253-233 kernel: [27968]   104 27968    36344     1917   0       0             0 mysqld
Sep 24 11:28:45 li253-233 kernel: [27977]    33 27977    12398     1316   2       0             0 apache2
Sep 24 11:28:45 li253-233 kernel: [27995]    33 27995    12392     1509   0       0             0 apache2
Sep 24 11:28:45 li253-233 kernel: [28001]    33 28001    12266     1767   3       0             0 apache2
Sep 24 11:28:45 li253-233 kernel: [28002]    33 28002    12264     2481   0       0             0 apache2
Sep 24 11:28:45 li253-233 kernel: [28005]    33 28005    12268     3518   0       0             0 apache2
Sep 24 11:28:45 li253-233 kernel: [28008]     0 28008      749       34   0       0             0 debian-start
Sep 24 11:28:45 li253-233 kernel: [28014]    33 28014    12392     2217   1       0             0 apache2
Sep 24 11:28:45 li253-233 kernel: [28016]    33 28016    12264     3190   1       0             0 apache2
Sep 24 11:28:45 li253-233 kernel: [28019]    33 28019    11895     2566   2       0             0 apache2
Sep 24 11:28:45 li253-233 kernel: [28020]    33 28020    12268     2783   1       0             0 apache2
Sep 24 11:28:45 li253-233 kernel: [28021]    33 28021    11816     2919   3       0             0 apache2
Sep 24 11:28:45 li253-233 kernel: [28022]    33 28022    12392     2548   0       0             0 apache2
Sep 24 11:28:45 li253-233 kernel: [28030]    33 28030    11560     3383   2       0             0 apache2
Sep 24 11:28:45 li253-233 kernel: [28032]    33 28032    11564     3019   0       0             0 apache2
Sep 24 11:28:46 li253-233 kernel: [28033]    33 28033    12136     3131   1       0             0 apache2
Sep 24 11:28:46 li253-233 kernel: [28040]    33 28040    12200     3504   1       0             0 apache2
Sep 24 11:28:46 li253-233 kernel: [28042]    33 28042    12074     3302   0       0             0 apache2
Sep 24 11:28:46 li253-233 kernel: [28046]    33 28046     8934      863   0       0             0 apache2
Sep 24 11:28:46 li253-233 kernel: [28047]    33 28047    12044     3371   0       0             0 apache2
Sep 24 11:28:46 li253-233 kernel: [28048]    33 28048    11785     3043   0       0             0 apache2
Sep 24 11:28:46 li253-233 kernel: [28050]    33 28050    12173     3452   1       0             0 apache2
Sep 24 11:28:46 li253-233 kernel: [28051]    33 28051    12136     3191   3       0             0 apache2
Sep 24 11:28:46 li253-233 kernel: [28052]    33 28052    12264     3457   0       0             0 apache2
Sep 24 11:28:46 li253-233 kernel: [28055]    33 28055    12264     3812   0       0             0 apache2
Sep 24 11:28:46 li253-233 kernel: [28056]    33 28056    11640     3379   2       0             0 apache2
Sep 24 11:28:46 li253-233 kernel: [28057]    33 28057    11560     2663   0       0             0 apache2
Sep 24 11:28:46 li253-233 kernel: [28059]    33 28059    12173     3741   2       0             0 apache2
Sep 24 11:28:46 li253-233 kernel: [28062]    33 28062    12173     3519   0       0             0 apache2
Sep 24 11:28:46 li253-233 kernel: [28064]    33 28064    11496     2750   0       0             0 apache2
Sep 24 11:28:46 li253-233 kernel: [28065]    33 28065    12173     3656   0       0             0 apache2
Sep 24 11:28:46 li253-233 kernel: [28073]    33 28073    12173     3913   0       0             0 apache2
Sep 24 11:28:46 li253-233 kernel: [28076]    33 28076    12173     3714   0       0             0 apache2
Sep 24 11:28:46 li253-233 kernel: [28077]    33 28077     7834      210   0       0             0 apache2
Sep 24 11:28:46 li253-233 kernel: [28078]    33 28078     7724       76   0       0             0 apache2
Sep 24 11:28:46 li253-233 kernel: [28079]    33 28079     8536     1069   0       0             0 apache2
Sep 24 11:28:46 li253-233 kernel: [28080]    33 28080     8934     1382   0       0             0 apache2
Sep 24 11:28:46 li253-233 kernel: [28081]    33 28081     8721     1256   0       0             0 apache2
Sep 24 11:28:46 li253-233 kernel: [28082]    33 28082     8919     1487   1       0             0 apache2
Sep 24 11:28:46 li253-233 kernel: [28083]    33 28083     7834      222   1       0             0 apache2
Sep 24 11:28:46 li253-233 kernel: [28084]    33 28084     7724       78   0       0             0 apache2
Sep 24 11:28:46 li253-233 kernel: [28098]    33 28098     7724       70   0       0             0 apache2
Sep 24 11:28:46 li253-233 kernel: [28100]    33 28100     7724       61   0       0             0 apache2
Sep 24 11:28:46 li253-233 kernel: [28102]     0 28102     2873      111   0       0             0 sendmail-mta
Sep 24 11:28:46 li253-233 kernel: [28103]    33 28103     7724       75   0       0             0 apache2
Sep 24 11:28:46 li253-233 kernel: [28104]     0 28104      749       35   0       0             0 debian-start
Sep 24 11:28:46 li253-233 kernel: [28107]     0 28107     1119       27   0       0             0 mysql
Sep 24 11:28:46 li253-233 kernel: [28109]    33 28109     7724       67   0       0             0 apache2
Sep 24 11:28:46 li253-233 kernel: [28110]     0 28110       76        2   0       0             0 poweroff
Sep 24 11:28:51 li253-233 kernel: mysql invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Sep 24 11:28:51 li253-233 kernel: Pid: 28107, comm: mysql Not tainted 3.0.4-linode37 #1
Sep 24 11:28:51 li253-233 kernel: Call Trace:
Sep 24 11:28:51 li253-233 kernel: [<c0187ed6>] ? T.677+0x76/0x1b0
Sep 24 11:28:51 li253-233 kernel: [<c01063fb>] ? xen_restore_fl_direct_reloc+0x4/0x4
Sep 24 11:28:51 li253-233 kernel: [<c06bf891>] ? _raw_spin_unlock_irqrestore+0x11/0x20
Sep 24 11:28:51 li253-233 kernel: [<c045c367>] ? ___ratelimit+0x97/0x110
Sep 24 11:28:51 li253-233 kernel: [<c0153fb1>] ? get_task_cred+0x11/0x50
Sep 24 11:28:51 li253-233 kernel: [<c018806e>] ? T.676+0x5e/0x150
Sep 24 11:28:51 li253-233 kernel: [<c013e24c>] ? has_capability_noaudit+0xc/0x20
Sep 24 11:28:51 li253-233 kernel: [<c0187d55>] ? oom_badness+0xb5/0x110
Sep 24 11:28:51 li253-233 kernel: [<c018839a>] ? out_of_memory+0x23a/0x300
Sep 24 11:28:51 li253-233 kernel: [<c018bcbb>] ? __alloc_pages_nodemask+0x61b/0x630
Sep 24 11:28:51 li253-233 kernel: [<c0186bba>] ? filemap_fault+0x21a/0x3f0
Sep 24 11:28:51 li253-233 kernel: [<c0104c7a>] ? xen_batched_set_pte+0x1a/0xf0
Sep 24 11:28:51 li253-233 kernel: [<c0103e95>] ? pte_pfn_to_mfn+0xb5/0xd0
Sep 24 11:28:51 li253-233 kernel: [<c019c426>] ? __do_fault+0x56/0x530
Sep 24 11:28:51 li253-233 kernel: [<c019de80>] ? handle_pte_fault+0xa0/0x2f0
Sep 24 11:28:51 li253-233 kernel: [<c019e5e4>] ? handle_mm_fault+0xf4/0x1b0
Sep 24 11:28:51 li253-233 kernel: [<c011f261>] ? do_page_fault+0x101/0x3a0
Sep 24 11:28:51 li253-233 kernel: [<c01a4555>] ? mmap_region+0x1b5/0x490
Sep 24 11:28:51 li253-233 kernel: [<c01b33ed>] ? filp_close+0x4d/0x80
Sep 24 11:28:51 li253-233 kernel: [<c01b348e>] ? sys_close+0x6e/0xc0
Sep 24 11:28:51 li253-233 kernel: [<c011f160>] ? mm_fault_error+0x130/0x130
Sep 24 11:28:51 li253-233 kernel: [<c06c00e6>] ? error_code+0x5a/0x60
Sep 24 11:28:51 li253-233 kernel: [<c011f160>] ? mm_fault_error+0x130/0x130
Sep 24 11:28:51 li253-233 kernel: Mem-Info:
Sep 24 11:28:51 li253-233 kernel: DMA per-cpu:
Sep 24 11:28:51 li253-233 kernel: CPU    0: hi:    0, btch:   1 usd:   0
Sep 24 11:28:51 li253-233 kernel: CPU    1: hi:    0, btch:   1 usd:   0
Sep 24 11:28:51 li253-233 kernel: CPU    2: hi:    0, btch:   1 usd:   0
Sep 24 11:28:51 li253-233 kernel: CPU    3: hi:    0, btch:   1 usd:   0
Sep 24 11:28:51 li253-233 kernel: Normal per-cpu:
Sep 24 11:28:51 li253-233 kernel: CPU    0: hi:  186, btch:  31 usd:  30
Sep 24 11:28:51 li253-233 kernel: CPU    1: hi:  186, btch:  31 usd:  11
Sep 24 11:28:51 li253-233 kernel: CPU    2: hi:  186, btch:  31 usd:  15
Sep 24 11:28:51 li253-233 kernel: CPU    3: hi:  186, btch:  31 usd:  63
Sep 24 11:28:51 li253-233 kernel: active_anon:58278 inactive_anon:58374 isolated_anon:104
Sep 24 11:28:51 li253-233 kernel: active_file:146 inactive_file:1436 isolated_file:0
Sep 24 11:28:51 li253-233 kernel: unevictable:0 dirty:0 writeback:143 unstable:0
Sep 24 11:28:51 li253-233 kernel: free:1202 slab_reclaimable:500 slab_unreclaimable:3076
Sep 24 11:28:51 li253-233 kernel: mapped:322 shmem:5 pagetables:1155 bounce:0
Sep 24 11:28:51 li253-233 kernel: DMA free:2092kB min:84kB low:104kB high:124kB active_anon:1952kB inactive_anon:2008kB active_file:8kB inactive_file:104kB unevictable:0kB isolated(anon):128kB isolated(file):0kB present:15808kB mlocked:0kB dirty:0kB writeback:16kB mapped:8kB shmem:0kB slab_reclaimable:20kB slab_unreclaimable:504kB kernel_stack:184kB pagetables:232kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:657 all_unreclaimable? yes
Sep 24 11:28:51 li253-233 kernel: lowmem_reserve[]: 0 500 500 500
Sep 24 11:28:51 li253-233 kernel: Normal free:2716kB min:2816kB low:3520kB high:4224kB active_anon:231160kB inactive_anon:231488kB active_file:576kB inactive_file:5640kB unevictable:0kB isolated(anon):288kB isolated(file):0kB present:512064kB mlocked:0kB dirty:0kB writeback:556kB mapped:1280kB shmem:20kB slab_reclaimable:1980kB slab_unreclaimable:11800kB kernel_stack:1256kB pagetables:4388kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:2204 all_unreclaimable? yes
Sep 24 11:28:51 li253-233 kernel: lowmem_reserve[]: 0 0 0 0
Sep 24 11:28:51 li253-233 kernel: DMA: 1*4kB 109*8kB 72*16kB 2*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2092kB
Sep 24 11:28:51 li253-233 kernel: Normal: 367*4kB 148*8kB 4*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2716kB
Sep 24 11:28:51 li253-233 kernel: 19859 total pagecache pages
Sep 24 11:28:51 li253-233 kernel: 18207 pages in swap cache
Sep 24 11:28:51 li253-233 kernel: Swap cache stats: add 7105290, delete 7087083, find 16078947/16666764
Sep 24 11:28:51 li253-233 kernel: Free swap  = 48kB
Sep 24 11:28:51 li253-233 kernel: Total swap = 262140kB
Sep 24 11:28:51 li253-233 kernel: 133104 pages RAM
Sep 24 11:28:51 li253-233 kernel: 0 pages HighMem
Sep 24 11:28:51 li253-233 kernel: 5833 pages reserved
Sep 24 11:28:51 li253-233 kernel: 7342 pages shared
Sep 24 11:28:51 li253-233 kernel: 124750 pages non-shared
Sep 24 11:28:51 li253-233 kernel: [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Sep 24 11:28:51 li253-233 kernel: [ 1033]     0  1033      592        0   0       0             0 upstart-udev-br
Sep 24 11:28:51 li253-233 kernel: [ 1035]     0  1035      559        0   1     -17         -1000 udevd
Sep 24 11:28:51 li253-233 kernel: [ 1440]     0  1440      558        0   1     -17         -1000 udevd
Sep 24 11:28:51 li253-233 kernel: [ 1471]     0  1471      558        0   0     -17         -1000 udevd
Sep 24 11:28:51 li253-233 kernel: [ 1919]     0  1919      572        0   0       0             0 dhclient3
Sep 24 11:28:51 li253-233 kernel: [ 1971]     0  1971      607       17   0       0             0 cron
Sep 24 11:28:51 li253-233 kernel: [ 1973]     0  1973      900        0   2       0             0 ntpd
Sep 24 11:28:51 li253-233 kernel: [ 1974]   103  1974      874        1   0       0             0 ntpd
Sep 24 11:28:51 li253-233 kernel: [ 1977]   101  1977     7454      262   1       0             0 rsyslogd
Sep 24 11:28:51 li253-233 kernel: [ 2072]     0  2072      460        0   0       0             0 getty
Sep 24 11:28:51 li253-233 kernel: [14026]     0 14026     1400       18   0     -17         -1000 sshd
Sep 24 11:28:51 li253-233 kernel: [20122]     0 20122     2873       63   0       0             0 sendmail-mta
Sep 24 11:28:51 li253-233 kernel: [ 1367]     0  1367     7724       52   2       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [27704]    33 27704    11908      958   0       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [27707]    33 27707    11921     1300   0       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [27874]    33 27874    11912     2359   0       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [27894]    33 27894    11912     1057   3       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [27968]   104 27968    36344     1979   0       0             0 mysqld
Sep 24 11:28:51 li253-233 kernel: [27977]    33 27977    12398     1316   2       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [27995]    33 27995    12392     1504   0       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28001]    33 28001    12328     1907   0       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28002]    33 28002    12264     2563   0       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28005]    33 28005    12268     3320   0       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28008]     0 28008      749       34   0       0             0 debian-start
Sep 24 11:28:51 li253-233 kernel: [28016]    33 28016    12264     3154   1       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28019]    33 28019    11880     2658   0       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28020]    33 28020    12268     2683   1       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28021]    33 28021    11831     2994   0       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28022]    33 28022    12392     2543   0       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28030]    33 28030    11560     3178   0       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28032]    33 28032    11564     2849   0       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28033]    33 28033    12202     3191   0       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28040]    33 28040    12264     3367   0       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28042]    33 28042    12137     3135   0       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28046]    33 28046     8919      971   0       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28047]    33 28047    12044     3286   1       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28048]    33 28048    11836     3035   0       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28050]    33 28050    12237     3280   0       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28051]    33 28051    12137     3198   1       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28052]    33 28052    12264     3391   0       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28055]    33 28055    12264     3799   1       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28056]    33 28056    11836     3640   0       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28057]    33 28057    11560     2507   0       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28059]    33 28059    12237     3804   0       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28062]    33 28062    12173     3365   1       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28064]    33 28064    11496     2663   0       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28065]    33 28065    12237     3741   0       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28073]    33 28073    12173     3815   0       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28076]    33 28076    12237     3747   0       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28077]    33 28077     7834      309   0       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28078]    33 28078     7724       76   0       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28079]    33 28079     8572     1164   1       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28080]    33 28080     9181     1742   0       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28081]    33 28081     9117     1718   2       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28082]    33 28082     9181     1702   0       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28083]    33 28083     7834      315   1       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28084]    33 28084     7724       88   0       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28098]    33 28098     7724       70   0       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28100]    33 28100     7724       61   0       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28102]     0 28102     2873      116   0       0             0 sendmail-mta
Sep 24 11:28:51 li253-233 kernel: [28103]    33 28103     7724       75   0       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28104]     0 28104      749       35   0       0             0 debian-start
Sep 24 11:28:51 li253-233 kernel: [28107]     0 28107     1150       29   0       0             0 mysql
Sep 24 11:28:51 li253-233 kernel: [28109]    33 28109     7724       67   0       0             0 apache2
Sep 24 11:28:51 li253-233 kernel: [28110]     0 28110      478       21   0       0             0 poweroff</c011f160></c06c00e6></c011f160></c01b348e></c01b33ed></c01a4555></c011f261></c019e5e4></c019de80></c019c426></c0103e95></c0104c7a></c0186bba></c018bcbb></c018839a></c0187d55></c013e24c></c018806e></c0153fb1></c045c367></c06bf891></c01063fb></c0187ed6></c011f160></c06b0000></c06c00e6></c011f160></c017792e></c01063fb></c0106404></c0105c27></c0105c27></c0175771></c0477810></c0477115></c011f261></c019e5e4></c019e01c></c019dc90></c01ab511></c01ab477></c018bcbb></c018839a></c0187d55></c013e24c></c018806e></c0178ad4></c045c367></c06bf891></c01063fb></c0187ed6>

caker · Answer 23 · Sept. 24, 2011, 4:26 p.m.

caker 13 years, 2 months ago Linode Staff

@goospoos - that's an OOM (out of memory), not a kernel panic. You need to tune your apps to utilize less virtual memory.

FWIW, a fixed kernel was release late last week. Reboot into "Latest 3.0" to pick it up.

http://www.linode.com/kernels/

http://www.linode.com/kernels/rss.xml

-Chris

Compute

Storage

Networking

Databases

Services

Developer Tools

Industries

Pricing

Community

Engage With Us

[UPDATED] Apparently random crashes

23 Replies

Reply

Tips: