[UPDATED] Apparently random crashes
In the last few weeks, I've been experimenting some random "crashes" of my Linode 1024. They happen every 9-13 days, at the same hour, about 8:00 GMT, and they cause the CPU, network and HD graphs at my Linode Dashboard to show values close to zero. The server doesn't respond from "outside", and the only possible solution is to reboot it, and this takes about 3 minutes.
It runs Ubuntu 10.04, with LAMP, Tomcat 6, Postfix and Dovecot. All of them are the last versions available, the software is totally up to date. I've checked some "system" logs, like syslog, daemon.log, etc., but they don't show anything relevant, they just "log" things until the crash moment.
I have this server since June, but the problems started in August. In addition, from the moment of the server purchase, I haven't made any software installation or similar. Other than versions changes, the server has the same software that was installed in the first time.
Any ideas?
Thanks in advance!
23 Replies
You need to do two things: One, make sure you're running the "Latest 3.0" kernel (Linode Manager -> Linode -> Config Profile -> Kernel drop-down, save, reboot). Second, when this happens, inspect your Lish console before you reboot to see if there has been a kernel BUG or OOM or something, or if your Linode is still 'alive' but something has taken networking down.
Hope that helps,
-Chris
Should I check one particular log?
And let's suppose this happens again. When I access my server using Lish console at Manager, what should I watch?
Finally, in the case I can't access my Linode from Lish console as said, what could be the issue?
A reboot will probably erase anything that was written to the console, though.
Regarding Apache, I don't think it is a memory problem, because memory consumption is all the time around 30% of the total.
The next time this happens, if it does happen, I will access my server using Lish console and check for warning messages before rebooting, so that they aren't erased.
REMOVED. SEE
I can't login, just opening the console throws those messages. And I can't do anything. Note that I changed MAC and IP addresses, because that isn't important.
In addition, I'm not the first one with this problem on the VPS:
The graphs of my dashboard say that CPU load before crash was 20%, network was about 400 kbps and IO around 3.3 kblocks per sec. And, as I mentioned, memory is always about 30% all the time.
Clearly, I'm not the only customer of Linode that has this error. It would be nice if Linode, as it's a serious company, investigates to solve the problem
Thanks in advance!
This is a kernel crash probably todo with your instance running out of memory. This isnt linode's fault.
Scroll up in that logview and im sure somewhere there will be out of memory error.
This means you need to tune your instance to not eat as much memory. Maybe run ab or a benchmarking utility agaisnt it to see if you can put enough pressure on it to cause the error to happen
If I press Ctrl-A, it shows me a message saying "No other window". If I press Escape, then the message hides and I return to the normal console. I can't scroll with the mouse, or the Page Up/Down keys.
REMOVED. SEE
It seems like it's an Apache issue… Could anyone translate the above code to English? Is it a memory error? IO? CPU? It seems it's something related to the memory, the stack, bit I don't know it 100% sure.
Thanks in advance!
This user has exactly the SAME problem as me, and… HE/SHE USES LINODE! Apparently, they haven't reached a solution, as they are not able to reproduce the error.
And in most forums in which users report this error, it appears that the problem is related with hardware. However, this is strange, as Linode uses virtual environments…
Finally, it's quite remarkable that, as I described, I have used Linode since June, but problems started in August, and they occur every few days. The forum post I linked is of August.
Hope we find a solution.
Might be a good idea to open a ticket, to the attention of psandin, especially if you're willing to be a test pilot.
> Hello,
We're still looking in to the issue. We've enabled some more debugging options in the latest kernel builds to try and help locate where the problem is. The current thread can be found at:
http://lists.xensource.com/archives/htm … 01172.html">http://lists.xensource.com/archives/html/xen-devel/2011-08/msg01172.html As indicated by the last post we're looking for affected customers to help debug the problem. If you would like to volunteer that would be great and we will let you know once we've got a specific plan worked out. If not the 2.6.32 and earlier kernels are known not to be affected.
Regards,
Peter
> Hello,
You are correct that this issue was not caused by a misconfiguration on your behalf. In the majority of cases that we have seen Apache seems to be involved but we haven't been able to pinpoint as to if Apache has something to do with this error or is just a causality. Downgrading to the 2.6.32 kernel will temporarily resolve this issue as we haven't seen this happen with that kernel. However, it will not help track down this bug and ultimately solve the problem at hand. I have notified our System administrators that you would like to help in resolving this issue so that we can take the necessary steps to get that in motion. Feel free to post this information on our forums if you would like. I hope this helps!
Regard
I must note that the first answer was made by psandin, the engineer that opened the forum post I linked. The second one was by lmatos.
The (my) conclusion:
If you have the same problem, notify it to Linode. Use kernel 2.6.32 version or lower to avoid the issue.
The error can't be solved by customers. It must be solved by engineers of Linode.
The cause is unknown. They're working on solving it.
I think it's important to notify it to Linode, in all cases, because they're working on the error and trying to solve it. And solving it will be good for everybody.
Thanks to all the replies.
@usr01:
It would be nice if Linode, as it's a serious company, investigates to solve the problem
Of course we are… That xen-devel thread was us.
@usr01:
- If you have the same problem, notify it to Linode. Use kernel 2.6.32 version or lower to avoid the issue.
This was bad advice, and I've informed our staff. 2.6.32 is unmaintained and eventually will not even be available. It's better to choose "Latest 3.0 x86_64", save, and reboot, since we know the 64 bit kernel is unaffected. 64 bit kernels can run 32 bit userspace just fine.
@usr01:
- The error can't be solved by customers. It must be solved by engineers of Linode.
Everyone is responsible for solving problems in open source software. Only you can prevent forest fires.
@usr01:
- The cause is unknown. They're working on solving it.
It's a software problem, most likely a race.
@caker:
@usr01:It would be nice if Linode, as it's a serious company, investigates to solve the problem
Of course we are… That xen-devel thread was us.
I made that statement before knowing that thread was opened.
@caker:
This was bad advice, and I've informed our staff. 2.6.32 is unmaintained and eventually will not even be available. It's better to choose "Latest 3.0 x86_64", save, and reboot, since we know the 64 bit kernel is unaffected. 64 bit kernels can run 32 bit userspace just fine.
Sorry, I'm not an expert. What are the implications of changing a kernel to 64 bits for the rest of the software installed? We're talking about production environments…
@caker:
It's a software problem, most likely a race.
But the cause is unknown, isn't it?
@caker:
Everyone is responsible for solving problems in open source software. Only you can prevent forest fires.
Obviously, I didn't mean to say the opposite. What I wanted to say was that the errors weren't caused by the customers. Most people don't have the time/knowledge to directly solve a problem in Xen software's source code. The best we can do is help experts to solve the problem, and I said repeatedly that users with this problem should notify Linode support. Of course everybody must help to prevent forest fires, but only experts should go with the firetruck to extinguish the fire.
However the key question is: why doesn't Linode publish a post about this error instead or correcting/quoting posts written by customers like me? I asked for permission to the support guys before publishing the above tickets in this forum, so I feel a bit uncomfortable with a member of Linode's staff correcting me later.
@usr01:
What are the implications of changing a kernel to 64 bits for the rest of the software installed? We're talking about production environments…
There are no implications. 32-bit userland programs will run under a 64-bit kernel without changes. The kernel might take up a little more memory.
Below is logwatch log…
WARNING: Kernel Errors Present
[
[
My kernel is "Latest 3.0(3.0.4-linode38)"
-Divyang Patel
Sep 24 11:28:45 li253-233 kernel: apache2 invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0
Sep 24 11:28:45 li253-233 kernel: Pid: 27874, comm: apache2 Not tainted 3.0.4-linode37 #1
Sep 24 11:28:45 li253-233 kernel: Call Trace:
Sep 24 11:28:45 li253-233 kernel: [<c0187ed6>] ? T.677+0x76/0x1b0
Sep 24 11:28:45 li253-233 kernel: [<c01063fb>] ? xen_restore_fl_direct_reloc+0x4/0x4
Sep 24 11:28:45 li253-233 kernel: [<c06bf891>] ? _raw_spin_unlock_irqrestore+0x11/0x20
Sep 24 11:28:45 li253-233 kernel: [<c045c367>] ? ___ratelimit+0x97/0x110
Sep 24 11:28:45 li253-233 kernel: [<c0178ad4>] ? __delayacct_freepages_end+0x24/0x30
Sep 24 11:28:45 li253-233 kernel: [<c018806e>] ? T.676+0x5e/0x150
Sep 24 11:28:45 li253-233 kernel: [<c013e24c>] ? has_capability_noaudit+0xc/0x20
Sep 24 11:28:45 li253-233 kernel: [<c0187d55>] ? oom_badness+0xb5/0x110
Sep 24 11:28:45 li253-233 kernel: [<c018839a>] ? out_of_memory+0x23a/0x300
Sep 24 11:28:45 li253-233 kernel: [<c018bcbb>] ? __alloc_pages_nodemask+0x61b/0x630
Sep 24 11:28:45 li253-233 kernel: [<c01ab477>] ? read_swap_cache_async+0xb7/0xf0
Sep 24 11:28:45 li253-233 kernel: [<c01ab511>] ? swapin_readahead+0x61/0x90
Sep 24 11:28:45 li253-233 kernel: [<c019dc90>] ? do_swap_page+0x4a0/0x5f0
Sep 24 11:28:45 li253-233 kernel: [<c019e01c>] ? handle_pte_fault+0x23c/0x2f0
Sep 24 11:28:45 li253-233 kernel: [<c019e5e4>] ? handle_mm_fault+0xf4/0x1b0
Sep 24 11:28:45 li253-233 kernel: [<c011f261>] ? do_page_fault+0x101/0x3a0
Sep 24 11:28:45 li253-233 kernel: [<c0477115>] ? info_for_irq+0x5/0x20
Sep 24 11:28:45 li253-233 kernel: [<c0477810>] ? evtchn_from_irq+0x10/0x40
Sep 24 11:28:45 li253-233 kernel: [<c0175771>] ? handle_percpu_irq+0x31/0x50
Sep 24 11:28:45 li253-233 kernel: [<c0105c27>] ? xen_force_evtchn_callback+0x17/0x30
Sep 24 11:28:45 li253-233 kernel: [<c0105c27>] ? xen_force_evtchn_callback+0x17/0x30
Sep 24 11:28:45 li253-233 kernel: [<c0106404>] ? check_events+0x8/0xc
Sep 24 11:28:45 li253-233 kernel: [<c01063fb>] ? xen_restore_fl_direct_reloc+0x4/0x4
Sep 24 11:28:45 li253-233 kernel: [<c017792e>] ? rcu_enter_nohz+0x2e/0xb0
Sep 24 11:28:45 li253-233 kernel: [<c011f160>] ? mm_fault_error+0x130/0x130
Sep 24 11:28:45 li253-233 kernel: [<c06c00e6>] ? error_code+0x5a/0x60
Sep 24 11:28:45 li253-233 kernel: [<c06b0000>] ? sctp_err_lookup+0xb0/0x110
Sep 24 11:28:45 li253-233 kernel: [<c011f160>] ? mm_fault_error+0x130/0x130
Sep 24 11:28:45 li253-233 kernel: Mem-Info:
Sep 24 11:28:45 li253-233 kernel: DMA per-cpu:
Sep 24 11:28:45 li253-233 kernel: CPU 0: hi: 0, btch: 1 usd: 0
Sep 24 11:28:45 li253-233 kernel: CPU 1: hi: 0, btch: 1 usd: 0
Sep 24 11:28:45 li253-233 kernel: CPU 2: hi: 0, btch: 1 usd: 0
Sep 24 11:28:45 li253-233 kernel: CPU 3: hi: 0, btch: 1 usd: 0
Sep 24 11:28:45 li253-233 kernel: Normal per-cpu:
Sep 24 11:28:45 li253-233 kernel: CPU 0: hi: 186, btch: 31 usd: 42
Sep 24 11:28:45 li253-233 kernel: CPU 1: hi: 186, btch: 31 usd: 61
Sep 24 11:28:45 li253-233 kernel: CPU 2: hi: 186, btch: 31 usd: 12
Sep 24 11:28:45 li253-233 kernel: CPU 3: hi: 186, btch: 31 usd: 3
Sep 24 11:28:45 li253-233 kernel: active_anon:58856 inactive_anon:58867 isolated_anon:151
Sep 24 11:28:45 li253-233 kernel: active_file:79 inactive_file:159 isolated_file:63
Sep 24 11:28:45 li253-233 kernel: unevictable:0 dirty:0 writeback:121 unstable:0
Sep 24 11:28:45 li253-233 kernel: free:1324 slab_reclaimable:502 slab_unreclaimable:3074
Sep 24 11:28:45 li253-233 kernel: mapped:85 shmem:5 pagetables:1169 bounce:0
Sep 24 11:28:45 li253-233 kernel: DMA free:2108kB min:84kB low:104kB high:124kB active_anon:2012kB inactive_anon:2040kB active_file:0kB inactive_file:16kB unevictable:0kB isolated(anon):16kB isolated(file):0kB present:15808kB mlocked:0kB dirty:0kB writeback:40kB mapped:8kB shmem:0kB slab_reclaimable:20kB slab_unreclaimable:504kB kernel_stack:184kB pagetables:236kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:62 all_unreclaimable? yes
Sep 24 11:28:45 li253-233 kernel: lowmem_reserve[]: 0 500 500 500
Sep 24 11:28:45 li253-233 kernel: Normal free:3188kB min:2816kB low:3520kB high:4224kB active_anon:233412kB inactive_anon:233428kB active_file:316kB inactive_file:620kB unevictable:0kB isolated(anon):588kB isolated(file):252kB present:512064kB mlocked:0kB dirty:0kB writeback:444kB mapped:332kB shmem:20kB slab_reclaimable:1988kB slab_unreclaimable:11792kB kernel_stack:1264kB pagetables:4440kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1356 all_unreclaimable? yes
Sep 24 11:28:45 li253-233 kernel: lowmem_reserve[]: 0 0 0 0
Sep 24 11:28:45 li253-233 kernel: DMA: 27*4kB 114*8kB 68*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2108kB
Sep 24 11:28:45 li253-233 kernel: Normal: 439*4kB 147*8kB 16*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3188kB
Sep 24 11:28:45 li253-233 kernel: 16861 total pagecache pages
Sep 24 11:28:45 li253-233 kernel: 16554 pages in swap cache
Sep 24 11:28:45 li253-233 kernel: Swap cache stats: add 7099735, delete 7083181, find 16078336/16665779
Sep 24 11:28:45 li253-233 kernel: Free swap = 0kB
Sep 24 11:28:45 li253-233 kernel: Total swap = 262140kB
Sep 24 11:28:45 li253-233 kernel: 133104 pages RAM
Sep 24 11:28:45 li253-233 kernel: 0 pages HighMem
Sep 24 11:28:45 li253-233 kernel: 5833 pages reserved
Sep 24 11:28:45 li253-233 kernel: 5803 pages shared
Sep 24 11:28:45 li253-233 kernel: 124908 pages non-shared
Sep 24 11:28:45 li253-233 kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Sep 24 11:28:45 li253-233 kernel: [ 1033] 0 1033 592 0 0 0 0 upstart-udev-br
Sep 24 11:28:45 li253-233 kernel: [ 1035] 0 1035 559 0 1 -17 -1000 udevd
Sep 24 11:28:45 li253-233 kernel: [ 1440] 0 1440 558 0 1 -17 -1000 udevd
Sep 24 11:28:45 li253-233 kernel: [ 1471] 0 1471 558 0 0 -17 -1000 udevd
Sep 24 11:28:45 li253-233 kernel: [ 1919] 0 1919 572 0 0 0 0 dhclient3
Sep 24 11:28:45 li253-233 kernel: [ 1971] 0 1971 607 17 0 0 0 cron
Sep 24 11:28:45 li253-233 kernel: [ 1973] 0 1973 900 0 2 0 0 ntpd
Sep 24 11:28:45 li253-233 kernel: [ 1974] 103 1974 874 1 0 0 0 ntpd
Sep 24 11:28:45 li253-233 kernel: [ 1977] 101 1977 7454 262 1 0 0 rsyslogd
Sep 24 11:28:45 li253-233 kernel: [ 2072] 0 2072 460 0 0 0 0 getty
Sep 24 11:28:45 li253-233 kernel: [14026] 0 14026 1400 18 0 -17 -1000 sshd
Sep 24 11:28:45 li253-233 kernel: [20122] 0 20122 2873 40 0 0 0 sendmail-mta
Sep 24 11:28:45 li253-233 kernel: [ 1367] 0 1367 7724 46 2 0 0 apache2
Sep 24 11:28:45 li253-233 kernel: [27704] 33 27704 11908 909 0 0 0 apache2
Sep 24 11:28:45 li253-233 kernel: [27707] 33 27707 11923 1245 0 0 0 apache2
Sep 24 11:28:45 li253-233 kernel: [27874] 33 27874 11912 2305 1 0 0 apache2
Sep 24 11:28:45 li253-233 kernel: [27894] 33 27894 11912 1069 3 0 0 apache2
Sep 24 11:28:45 li253-233 kernel: [27968] 104 27968 36344 1917 0 0 0 mysqld
Sep 24 11:28:45 li253-233 kernel: [27977] 33 27977 12398 1316 2 0 0 apache2
Sep 24 11:28:45 li253-233 kernel: [27995] 33 27995 12392 1509 0 0 0 apache2
Sep 24 11:28:45 li253-233 kernel: [28001] 33 28001 12266 1767 3 0 0 apache2
Sep 24 11:28:45 li253-233 kernel: [28002] 33 28002 12264 2481 0 0 0 apache2
Sep 24 11:28:45 li253-233 kernel: [28005] 33 28005 12268 3518 0 0 0 apache2
Sep 24 11:28:45 li253-233 kernel: [28008] 0 28008 749 34 0 0 0 debian-start
Sep 24 11:28:45 li253-233 kernel: [28014] 33 28014 12392 2217 1 0 0 apache2
Sep 24 11:28:45 li253-233 kernel: [28016] 33 28016 12264 3190 1 0 0 apache2
Sep 24 11:28:45 li253-233 kernel: [28019] 33 28019 11895 2566 2 0 0 apache2
Sep 24 11:28:45 li253-233 kernel: [28020] 33 28020 12268 2783 1 0 0 apache2
Sep 24 11:28:45 li253-233 kernel: [28021] 33 28021 11816 2919 3 0 0 apache2
Sep 24 11:28:45 li253-233 kernel: [28022] 33 28022 12392 2548 0 0 0 apache2
Sep 24 11:28:45 li253-233 kernel: [28030] 33 28030 11560 3383 2 0 0 apache2
Sep 24 11:28:45 li253-233 kernel: [28032] 33 28032 11564 3019 0 0 0 apache2
Sep 24 11:28:46 li253-233 kernel: [28033] 33 28033 12136 3131 1 0 0 apache2
Sep 24 11:28:46 li253-233 kernel: [28040] 33 28040 12200 3504 1 0 0 apache2
Sep 24 11:28:46 li253-233 kernel: [28042] 33 28042 12074 3302 0 0 0 apache2
Sep 24 11:28:46 li253-233 kernel: [28046] 33 28046 8934 863 0 0 0 apache2
Sep 24 11:28:46 li253-233 kernel: [28047] 33 28047 12044 3371 0 0 0 apache2
Sep 24 11:28:46 li253-233 kernel: [28048] 33 28048 11785 3043 0 0 0 apache2
Sep 24 11:28:46 li253-233 kernel: [28050] 33 28050 12173 3452 1 0 0 apache2
Sep 24 11:28:46 li253-233 kernel: [28051] 33 28051 12136 3191 3 0 0 apache2
Sep 24 11:28:46 li253-233 kernel: [28052] 33 28052 12264 3457 0 0 0 apache2
Sep 24 11:28:46 li253-233 kernel: [28055] 33 28055 12264 3812 0 0 0 apache2
Sep 24 11:28:46 li253-233 kernel: [28056] 33 28056 11640 3379 2 0 0 apache2
Sep 24 11:28:46 li253-233 kernel: [28057] 33 28057 11560 2663 0 0 0 apache2
Sep 24 11:28:46 li253-233 kernel: [28059] 33 28059 12173 3741 2 0 0 apache2
Sep 24 11:28:46 li253-233 kernel: [28062] 33 28062 12173 3519 0 0 0 apache2
Sep 24 11:28:46 li253-233 kernel: [28064] 33 28064 11496 2750 0 0 0 apache2
Sep 24 11:28:46 li253-233 kernel: [28065] 33 28065 12173 3656 0 0 0 apache2
Sep 24 11:28:46 li253-233 kernel: [28073] 33 28073 12173 3913 0 0 0 apache2
Sep 24 11:28:46 li253-233 kernel: [28076] 33 28076 12173 3714 0 0 0 apache2
Sep 24 11:28:46 li253-233 kernel: [28077] 33 28077 7834 210 0 0 0 apache2
Sep 24 11:28:46 li253-233 kernel: [28078] 33 28078 7724 76 0 0 0 apache2
Sep 24 11:28:46 li253-233 kernel: [28079] 33 28079 8536 1069 0 0 0 apache2
Sep 24 11:28:46 li253-233 kernel: [28080] 33 28080 8934 1382 0 0 0 apache2
Sep 24 11:28:46 li253-233 kernel: [28081] 33 28081 8721 1256 0 0 0 apache2
Sep 24 11:28:46 li253-233 kernel: [28082] 33 28082 8919 1487 1 0 0 apache2
Sep 24 11:28:46 li253-233 kernel: [28083] 33 28083 7834 222 1 0 0 apache2
Sep 24 11:28:46 li253-233 kernel: [28084] 33 28084 7724 78 0 0 0 apache2
Sep 24 11:28:46 li253-233 kernel: [28098] 33 28098 7724 70 0 0 0 apache2
Sep 24 11:28:46 li253-233 kernel: [28100] 33 28100 7724 61 0 0 0 apache2
Sep 24 11:28:46 li253-233 kernel: [28102] 0 28102 2873 111 0 0 0 sendmail-mta
Sep 24 11:28:46 li253-233 kernel: [28103] 33 28103 7724 75 0 0 0 apache2
Sep 24 11:28:46 li253-233 kernel: [28104] 0 28104 749 35 0 0 0 debian-start
Sep 24 11:28:46 li253-233 kernel: [28107] 0 28107 1119 27 0 0 0 mysql
Sep 24 11:28:46 li253-233 kernel: [28109] 33 28109 7724 67 0 0 0 apache2
Sep 24 11:28:46 li253-233 kernel: [28110] 0 28110 76 2 0 0 0 poweroff
Sep 24 11:28:51 li253-233 kernel: mysql invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Sep 24 11:28:51 li253-233 kernel: Pid: 28107, comm: mysql Not tainted 3.0.4-linode37 #1
Sep 24 11:28:51 li253-233 kernel: Call Trace:
Sep 24 11:28:51 li253-233 kernel: [<c0187ed6>] ? T.677+0x76/0x1b0
Sep 24 11:28:51 li253-233 kernel: [<c01063fb>] ? xen_restore_fl_direct_reloc+0x4/0x4
Sep 24 11:28:51 li253-233 kernel: [<c06bf891>] ? _raw_spin_unlock_irqrestore+0x11/0x20
Sep 24 11:28:51 li253-233 kernel: [<c045c367>] ? ___ratelimit+0x97/0x110
Sep 24 11:28:51 li253-233 kernel: [<c0153fb1>] ? get_task_cred+0x11/0x50
Sep 24 11:28:51 li253-233 kernel: [<c018806e>] ? T.676+0x5e/0x150
Sep 24 11:28:51 li253-233 kernel: [<c013e24c>] ? has_capability_noaudit+0xc/0x20
Sep 24 11:28:51 li253-233 kernel: [<c0187d55>] ? oom_badness+0xb5/0x110
Sep 24 11:28:51 li253-233 kernel: [<c018839a>] ? out_of_memory+0x23a/0x300
Sep 24 11:28:51 li253-233 kernel: [<c018bcbb>] ? __alloc_pages_nodemask+0x61b/0x630
Sep 24 11:28:51 li253-233 kernel: [<c0186bba>] ? filemap_fault+0x21a/0x3f0
Sep 24 11:28:51 li253-233 kernel: [<c0104c7a>] ? xen_batched_set_pte+0x1a/0xf0
Sep 24 11:28:51 li253-233 kernel: [<c0103e95>] ? pte_pfn_to_mfn+0xb5/0xd0
Sep 24 11:28:51 li253-233 kernel: [<c019c426>] ? __do_fault+0x56/0x530
Sep 24 11:28:51 li253-233 kernel: [<c019de80>] ? handle_pte_fault+0xa0/0x2f0
Sep 24 11:28:51 li253-233 kernel: [<c019e5e4>] ? handle_mm_fault+0xf4/0x1b0
Sep 24 11:28:51 li253-233 kernel: [<c011f261>] ? do_page_fault+0x101/0x3a0
Sep 24 11:28:51 li253-233 kernel: [<c01a4555>] ? mmap_region+0x1b5/0x490
Sep 24 11:28:51 li253-233 kernel: [<c01b33ed>] ? filp_close+0x4d/0x80
Sep 24 11:28:51 li253-233 kernel: [<c01b348e>] ? sys_close+0x6e/0xc0
Sep 24 11:28:51 li253-233 kernel: [<c011f160>] ? mm_fault_error+0x130/0x130
Sep 24 11:28:51 li253-233 kernel: [<c06c00e6>] ? error_code+0x5a/0x60
Sep 24 11:28:51 li253-233 kernel: [<c011f160>] ? mm_fault_error+0x130/0x130
Sep 24 11:28:51 li253-233 kernel: Mem-Info:
Sep 24 11:28:51 li253-233 kernel: DMA per-cpu:
Sep 24 11:28:51 li253-233 kernel: CPU 0: hi: 0, btch: 1 usd: 0
Sep 24 11:28:51 li253-233 kernel: CPU 1: hi: 0, btch: 1 usd: 0
Sep 24 11:28:51 li253-233 kernel: CPU 2: hi: 0, btch: 1 usd: 0
Sep 24 11:28:51 li253-233 kernel: CPU 3: hi: 0, btch: 1 usd: 0
Sep 24 11:28:51 li253-233 kernel: Normal per-cpu:
Sep 24 11:28:51 li253-233 kernel: CPU 0: hi: 186, btch: 31 usd: 30
Sep 24 11:28:51 li253-233 kernel: CPU 1: hi: 186, btch: 31 usd: 11
Sep 24 11:28:51 li253-233 kernel: CPU 2: hi: 186, btch: 31 usd: 15
Sep 24 11:28:51 li253-233 kernel: CPU 3: hi: 186, btch: 31 usd: 63
Sep 24 11:28:51 li253-233 kernel: active_anon:58278 inactive_anon:58374 isolated_anon:104
Sep 24 11:28:51 li253-233 kernel: active_file:146 inactive_file:1436 isolated_file:0
Sep 24 11:28:51 li253-233 kernel: unevictable:0 dirty:0 writeback:143 unstable:0
Sep 24 11:28:51 li253-233 kernel: free:1202 slab_reclaimable:500 slab_unreclaimable:3076
Sep 24 11:28:51 li253-233 kernel: mapped:322 shmem:5 pagetables:1155 bounce:0
Sep 24 11:28:51 li253-233 kernel: DMA free:2092kB min:84kB low:104kB high:124kB active_anon:1952kB inactive_anon:2008kB active_file:8kB inactive_file:104kB unevictable:0kB isolated(anon):128kB isolated(file):0kB present:15808kB mlocked:0kB dirty:0kB writeback:16kB mapped:8kB shmem:0kB slab_reclaimable:20kB slab_unreclaimable:504kB kernel_stack:184kB pagetables:232kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:657 all_unreclaimable? yes
Sep 24 11:28:51 li253-233 kernel: lowmem_reserve[]: 0 500 500 500
Sep 24 11:28:51 li253-233 kernel: Normal free:2716kB min:2816kB low:3520kB high:4224kB active_anon:231160kB inactive_anon:231488kB active_file:576kB inactive_file:5640kB unevictable:0kB isolated(anon):288kB isolated(file):0kB present:512064kB mlocked:0kB dirty:0kB writeback:556kB mapped:1280kB shmem:20kB slab_reclaimable:1980kB slab_unreclaimable:11800kB kernel_stack:1256kB pagetables:4388kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:2204 all_unreclaimable? yes
Sep 24 11:28:51 li253-233 kernel: lowmem_reserve[]: 0 0 0 0
Sep 24 11:28:51 li253-233 kernel: DMA: 1*4kB 109*8kB 72*16kB 2*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2092kB
Sep 24 11:28:51 li253-233 kernel: Normal: 367*4kB 148*8kB 4*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2716kB
Sep 24 11:28:51 li253-233 kernel: 19859 total pagecache pages
Sep 24 11:28:51 li253-233 kernel: 18207 pages in swap cache
Sep 24 11:28:51 li253-233 kernel: Swap cache stats: add 7105290, delete 7087083, find 16078947/16666764
Sep 24 11:28:51 li253-233 kernel: Free swap = 48kB
Sep 24 11:28:51 li253-233 kernel: Total swap = 262140kB
Sep 24 11:28:51 li253-233 kernel: 133104 pages RAM
Sep 24 11:28:51 li253-233 kernel: 0 pages HighMem
Sep 24 11:28:51 li253-233 kernel: 5833 pages reserved
Sep 24 11:28:51 li253-233 kernel: 7342 pages shared
Sep 24 11:28:51 li253-233 kernel: 124750 pages non-shared
Sep 24 11:28:51 li253-233 kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Sep 24 11:28:51 li253-233 kernel: [ 1033] 0 1033 592 0 0 0 0 upstart-udev-br
Sep 24 11:28:51 li253-233 kernel: [ 1035] 0 1035 559 0 1 -17 -1000 udevd
Sep 24 11:28:51 li253-233 kernel: [ 1440] 0 1440 558 0 1 -17 -1000 udevd
Sep 24 11:28:51 li253-233 kernel: [ 1471] 0 1471 558 0 0 -17 -1000 udevd
Sep 24 11:28:51 li253-233 kernel: [ 1919] 0 1919 572 0 0 0 0 dhclient3
Sep 24 11:28:51 li253-233 kernel: [ 1971] 0 1971 607 17 0 0 0 cron
Sep 24 11:28:51 li253-233 kernel: [ 1973] 0 1973 900 0 2 0 0 ntpd
Sep 24 11:28:51 li253-233 kernel: [ 1974] 103 1974 874 1 0 0 0 ntpd
Sep 24 11:28:51 li253-233 kernel: [ 1977] 101 1977 7454 262 1 0 0 rsyslogd
Sep 24 11:28:51 li253-233 kernel: [ 2072] 0 2072 460 0 0 0 0 getty
Sep 24 11:28:51 li253-233 kernel: [14026] 0 14026 1400 18 0 -17 -1000 sshd
Sep 24 11:28:51 li253-233 kernel: [20122] 0 20122 2873 63 0 0 0 sendmail-mta
Sep 24 11:28:51 li253-233 kernel: [ 1367] 0 1367 7724 52 2 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [27704] 33 27704 11908 958 0 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [27707] 33 27707 11921 1300 0 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [27874] 33 27874 11912 2359 0 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [27894] 33 27894 11912 1057 3 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [27968] 104 27968 36344 1979 0 0 0 mysqld
Sep 24 11:28:51 li253-233 kernel: [27977] 33 27977 12398 1316 2 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [27995] 33 27995 12392 1504 0 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28001] 33 28001 12328 1907 0 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28002] 33 28002 12264 2563 0 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28005] 33 28005 12268 3320 0 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28008] 0 28008 749 34 0 0 0 debian-start
Sep 24 11:28:51 li253-233 kernel: [28016] 33 28016 12264 3154 1 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28019] 33 28019 11880 2658 0 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28020] 33 28020 12268 2683 1 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28021] 33 28021 11831 2994 0 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28022] 33 28022 12392 2543 0 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28030] 33 28030 11560 3178 0 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28032] 33 28032 11564 2849 0 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28033] 33 28033 12202 3191 0 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28040] 33 28040 12264 3367 0 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28042] 33 28042 12137 3135 0 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28046] 33 28046 8919 971 0 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28047] 33 28047 12044 3286 1 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28048] 33 28048 11836 3035 0 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28050] 33 28050 12237 3280 0 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28051] 33 28051 12137 3198 1 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28052] 33 28052 12264 3391 0 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28055] 33 28055 12264 3799 1 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28056] 33 28056 11836 3640 0 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28057] 33 28057 11560 2507 0 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28059] 33 28059 12237 3804 0 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28062] 33 28062 12173 3365 1 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28064] 33 28064 11496 2663 0 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28065] 33 28065 12237 3741 0 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28073] 33 28073 12173 3815 0 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28076] 33 28076 12237 3747 0 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28077] 33 28077 7834 309 0 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28078] 33 28078 7724 76 0 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28079] 33 28079 8572 1164 1 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28080] 33 28080 9181 1742 0 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28081] 33 28081 9117 1718 2 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28082] 33 28082 9181 1702 0 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28083] 33 28083 7834 315 1 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28084] 33 28084 7724 88 0 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28098] 33 28098 7724 70 0 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28100] 33 28100 7724 61 0 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28102] 0 28102 2873 116 0 0 0 sendmail-mta
Sep 24 11:28:51 li253-233 kernel: [28103] 33 28103 7724 75 0 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28104] 0 28104 749 35 0 0 0 debian-start
Sep 24 11:28:51 li253-233 kernel: [28107] 0 28107 1150 29 0 0 0 mysql
Sep 24 11:28:51 li253-233 kernel: [28109] 33 28109 7724 67 0 0 0 apache2
Sep 24 11:28:51 li253-233 kernel: [28110] 0 28110 478 21 0 0 0 poweroff</c011f160></c06c00e6></c011f160></c01b348e></c01b33ed></c01a4555></c011f261></c019e5e4></c019de80></c019c426></c0103e95></c0104c7a></c0186bba></c018bcbb></c018839a></c0187d55></c013e24c></c018806e></c0153fb1></c045c367></c06bf891></c01063fb></c0187ed6></c011f160></c06b0000></c06c00e6></c011f160></c017792e></c01063fb></c0106404></c0105c27></c0105c27></c0175771></c0477810></c0477115></c011f261></c019e5e4></c019e01c></c019dc90></c01ab511></c01ab477></c018bcbb></c018839a></c0187d55></c013e24c></c018806e></c0178ad4></c045c367></c06bf891></c01063fb></c0187ed6>
FWIW, a fixed kernel was release late last week. Reboot into "Latest 3.0" to pick it up.
-Chris