Server crash...

Something crash for some reason, that's only half the trace, check /var/log/messages for the full trace

Nothing in /var/log/messages or even the rotated messages.* have any part of the message contained within them. Could it be anywhere else?

Thanks for getting back so quickly.

That's a bit weird, ok do this

ssh to your lish terminal, press ctrl-a + d this will detach your console, type logview and press enter, that should bring up the entire message.

That worked here's the whole dump from the logview commmand in from lish.

Deep breath…

[linode42786@london85 lish]# logview

Showing last 250 lines from previous boot
-----------------------------------------
Switching to clocksource xen
NET: Registered protocol family 2
IP route cache hash table entries: 65536 (order: 7, 524288 bytes)
TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 262144 bind 65536)
TCP reno registered
NET: Registered protocol family 1
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
platform rtc_cmos: registered platform RTC device (no PNP device found)
Intel AES-NI instructions are not detected.
audit: initializing netlink socket (disabled)
type=2000 audit(1272548257.063:1): initialized
VFS: Disk quotas dquot_6.5.2
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
DLM (built Apr 28 2010 18:46:05) installed
Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
Slow work thread pool: Starting up
Slow work thread pool: Ready
ROMFS MTD (C) 2007 Red Hat, Inc.
fuse init (API version 7.13)
JFS: nTxBlock = 8192, nTxLock = 65536
SGI XFS with ACLs, security attributes, large block/inode numbers, no debug enabled
SGI XFS Quota Management subsystem
GFS2 (built Apr 28 2010 18:46:41) installed
msgmni has been set to 3507
alg: No test for cipher_null (cipher_null-generic)
alg: No test for ecb(cipher_null) (ecb-cipher_null)
alg: No test for digest_null (digest_null-generic)
alg: No test for compress_null (compress_null-generic)
alg: No test for fcrypt (fcrypt-generic)
alg: No test for stdrng (krng)
alg: No test for ghash (ghash-generic)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
Event-channel device installed.
brd: module loaded
loop: module loaded
nbd: registered device at major 43
PPP generic driver version 2.4.2
PPP Deflate Compression module registered
PPP BSD Compression module registered
PPP MPPE Compression module registered
SLIP: version 0.8.4-NET3.019-NEWTTY (dynamic channels, max=256) (6 bit encapsulation enabled).
CSLIP: code copyright 1989 Regents of the University of California.
SLIP linefill/keepalive option.
Initialising Xen virtual ethernet driver.
blkfront: xvda: barriers enabled
tun: Universal TUN/TAP device driver, 1.6
tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>xvda:
i8042.c: No controller found.
 unknown partition table
mice: PS/2 mouse device common for all mice
md: linear personality registered for level -1
md: raid0 personality registered for level 0
md: raid1 personality registered for level 1
md: raid6 personality registered for level 6
md: raid5 personality registered for level 5
md: raid4 personality registered for level 4
md: multipath personality registered for level -4
device-mapper: ioctl: 4.15.0-ioctl (2009-04-01) initialised: dm-devel@redhat.com
GACT probability NOT on
u32 classifier
    Performance counters on
    Actions configured 
Netfilter messages via NETLINK v0.30.
nf_conntrack version 0.5.0 (8192 buckets, 65536 max)
CONFIG_NF_CT_ACCT is deprecated and will be removed soon. Please use
nf_conntrack.acct=1 kernel parameter, acct=1 nf_conntrack module option or
sysctl net.netfilter.nf_conntrack_acct=1 to enable it.
xt_time: kernel timezone is -0000
IPv4 over IPv4 tunneling driver
GRE over IPv4 tunneling driver
blkfront: xvdb: barriers enabled
ip_tables: (C) 2000-2006 Netfilter Core Team
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
tunl0: Disabled Privacy Extensions
ip6_tables: (C) 2000-2006 Netfilter Core Team
IPv6 over IPv4 tunneling driver
 xvdb:
sit0: Disabled Privacy Extensions
 unknown partition table
ip6tnl0: Disabled Privacy Extensions
NET: Registered protocol family 17
NET: Registered protocol family 15
Bridge firewalling registered
Ebtables v2.0 registered
ebt_ulog: out of memory trying to call netlink_kernel_create
802.1Q VLAN Support v1.8 Ben Greear <greearb@candelatech.com>All bugs added by David S. Miller <davem@redhat.com>SCTP: Hash tables configured (established 65536 bind 65536)
registered taskstats version 1
XENBUS: Device with no driver: device/console/0
md: Waiting for all devices to be available before autodetect
md: If you don't use raid, use raid=noautodetect
md: Autodetecting RAID arrays.
md: Scanned 0 and added 0 devices.
md: autorun ...
md: ... autorun DONE.
kjournald starting.  Commit interval 5 seconds
EXT3-fs: mounted filesystem with writeback data mode.
VFS: Mounted root (ext3 filesystem) readonly on device 202:0.
Freeing unused kernel memory: 496k freed
Write protecting the kernel read-only data: 7204k
fsck from util-linux-ng 2.16
/dev/xvda: clean, 30391/1015808 files, 333063/4063232 blocks
FATAL: Module nf_conntrack_ftp not found.
FATAL: Module nf_nat_ftp not found.
FATAL: Module nf_conntrack_irc not found.
FATAL: Module nf_nat_irc not found.
 * Setting preliminary keymap...                                                 * Stopping NTP server ntpdShell server sshd       
 * Starting NTP server ntpd
 * Starting MySQL database server mysqld                                           ...done.
 * Checking for corrupt, not cleanly closed and upgrade needing tables.
 * Starting NTP server ntpd                                                     
Ubuntu 9.10 localhost hvc0

localhost login: divide error: 0000 [#1] SMP 
last sysfs file: /sys/devices/vbd-51712/block/xvda/removable
CPU 3 
Modules linked in:
Pid: 2677, comm: mysqld Not tainted 2.6.32.12-x86_64-linode12 #1 
RIP: e030:[<ffffffff8103ad36>]  [<ffffffff8103ad36>] find_busiest_group+0x5f6/0xb60
RSP: e02b:ffff88006e2359d8  EFLAGS: 00010046
RAX: 0000000000000000 RBX: ffffffffffffffff RCX: 0000000000000001
RDX: 0000000000000000 RSI: 0000000000000010 RDI: 0000000000000000
RBP: 0000000000000000 R08: 0000000000000010 R09: ffff880001d243d0
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000013940
R13: 00000000ffffffff R14: 0000000000013940 R15: 0000000000013940
FS:  00007fdd18741910(0000) GS:ffff880001d15000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f6c7d917000 CR3: 000000006d876000 CR4: 0000000000002660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process mysqld (pid: 2677, threadinfo ffff88006e234000, task ffff88006f3ff7f0)
Stack:
 0000000000000000 ffff88006e235ae8 0000000000013940 0000000000000008
<0> 0000000000013940 ffff880001d24288 ffff88006e235bec ffff88006e235bd8
<0> 0000000300000140 ffff880001d24340 0000000000000003 ffff880001d243b0
Call Trace:
 [<ffffffff815855d3>] ? thread_return+0x39a/0x667
 [<ffffffff8100922a>] ? hypercall_page+0x22a/0x1010
 [<ffffffff8100922a>] ? hypercall_page+0x22a/0x1010
 [<ffffffff8100de29>] ? xen_force_evtchn_callback+0x9/0x10
 [<ffffffff8100e552>] ? check_events+0x12/0x20
 [<ffffffff81067dcd>] ? futex_wait_queue_me+0xbd/0x100
 [<ffffffff81067f56>] ? futex_wait+0x146/0x2f0
 [<ffffffff81587515>] ? _spin_lock+0x5/0x10
 [<ffffffff810682e2>] ? futex_wake+0x112/0x130
 [<ffffffff8106a238>] ? do_futex+0x148/0xc30
 [<ffffffff81587baa>] ? error_exit+0x2a/0x60
 [<ffffffff8101191d>] ? retint_restore_args+0x5/0x6
 [<ffffffff8100922a>] ? hypercall_page+0x22a/0x1010
 [<ffffffff8100922a>] ? hypercall_page+0x22a/0x1010
 [<ffffffff8100922a>] ? hypercall_page+0x22a/0x1010
 [<ffffffff81026119>] ? pvclock_clocksource_read+0x49/0x80
 [<ffffffff81038525>] ? set_next_entity+0x35/0x80
 [<ffffffff81038688>] ? pick_next_task_fair+0xd8/0x150
 [<ffffffff815853f0>] ? thread_return+0x1b7/0x667
 [<ffffffff8100de29>] ? xen_force_evtchn_callback+0x9/0x10
 [<ffffffff8100e552>] ? check_events+0x12/0x20
 [<ffffffff8106ada9>] ? sys_futex+0x89/0x170
 [<ffffffff8106236c>] ? getnstimeofday+0x5c/0xe0
 [<ffffffff81062450>] ? do_gettimeofday+0x10/0x50
 [<ffffffff81010f42>] ? system_call_fastpath+0x16/0x1b
Code: 7c 24 50 c7 44 24 54 01 00 00 00 e9 9b fb ff ff 0f 1f 44 00 00 48 8b 54 24 70 48 8b 84 24 18 01 00 00 8b 7a 08 48 c1 e0 0a 31 d2 <48> f7 f7 48 8b bc 24 20 01 00 00 48 89 84 24 10 01 00 00 31 c0 
RIP  [<ffffffff8103ad36>] find_busiest_group+0x5f6/0xb60
 RSP <ffff88006e2359d8>---[ end trace a35477c3e4ee7a14 ]---
divide error: 0000 [#2] SMP 
last sysfs file: /sys/devices/vbd-51712/block/xvda/removable
CPU 0 
Modules linked in:
Pid: 3168, comm: ruby Tainted: G      D    2.6.32.12-x86_64-linode12 #1 
RIP: e030:[<ffffffff8103ad36>]  [<ffffffff8103ad36>] find_busiest_group+0x5f6/0xb60
RSP: e02b:ffff88006e975c58  EFLAGS: 00010046
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000010 RDI: 0000000000000000
RBP: 0000000000000000 R08: 0000000000000010 R09: ffff880001d243d0
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000013940
R13: 0000000000000000 R14: 0000000000013940 R15: 0000000000013940
FS:  00007f6c770c1910(0000) GS:ffff880001cc1000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f6c7d917000 CR3: 000000006da65000 CR4: 0000000000002660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process ruby (pid: 3168, threadinfo ffff88006e974000, task ffff88006f36a300)
Stack:
 0018744d2553f657 ffff88006e975d68 0000000000013940 0000000000000008
<0> 0000000000013940 ffff880001cd0288 ffff88006e975e6c ffff88006e975e58
<0> 0000000000000180 ffff880001cd0340 00000000ffffffff ffff880001cd03b0
Call Trace:
 [<ffffffff815855d3>] ? thread_return+0x39a/0x667
 [<ffffffff8105d4df>] ? __hrtimer_start_range_ns+0x23f/0x3f0
 [<ffffffff8105d229>] ? hrtimer_try_to_cancel+0x39/0x80
 [<ffffffff815867ed>] ? do_nanosleep+0x8d/0xc0
 [<ffffffff8105d74c>] ? hrtimer_nanosleep+0x9c/0x130
 [<ffffffff8105c7c0>] ? hrtimer_wakeup+0x0/0x30
 [<ffffffff8105d857>] ? sys_nanosleep+0x77/0x80
 [<ffffffff81010f42>] ? system_call_fastpath+0x16/0x1b
Code: 7c 24 50 c7 44 24 54 01 00 00 00 e9 9b fb ff ff 0f 1f 44 00 00 48 8b 54 24 70 48 8b 84 24 18 01 00 00 8b 7a 08 48 c1 e0 0a 31 d2 <48> f7 f7 48 8b bc 24 20 01 00 00 48 89 84 24 10 01 00 00 31 c0 
RIP  [<ffffffff8103ad36>] find_busiest_group+0x5f6/0xb60
 RSP <ffff88006e975c58>---[ end trace a35477c3e4ee7a15 ]---
divide error: 0000 [#3] SMP 
last sysfs file: /sys/devices/vbd-51712/block/xvda/removable
CPU 1 
Modules linked in:
Pid: 12, comm: events/1 Tainted: G      D    2.6.32.12-x86_64-linode12 #1 
RIP: e030:[<ffffffff8103ad36>]  [<ffffffff8103ad36>] find_busiest_group+0x5f6/0xb60
RSP: e02b:ffff8800700b1be0  EFLAGS: 00010046
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000010 RDI: 0000000000000000
RBP: 0000000000000000 R08: 0000000000000010 R09: ffff880001d243d0
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000013940
R13: 0000000000000000 R14: 0000000000013940 R15: 0000000000013940
FS:  00007f6c770c1910(0000) GS:ffff880001cdd000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fdd08097000 CR3: 000000006da65000 CR4: 0000000000002660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process events/1 (pid: 12, threadinfo ffff8800700b0000, task ffff8800700a8790)
Stack:
 0000000000000100 ffff8800700b1cf0 0000000000013940 0000000000000008
<0> 0000000000013940 ffff880001cec288 ffff8800700b1df4 ffff8800700b1de0
<0> 0000000100000100 ffff880001cec340 00000000ffffffff ffff880001cec3b0
Call Trace:
 [<ffffffff815855d3>] ? thread_return+0x39a/0x667
 [<ffffffff8100e3d0>] ? xen_vcpuop_set_next_event+0x0/0x70
 [<ffffffff810a1ca0>] ? vmstat_update+0x0/0x40
 [<ffffffff81055d65>] ? worker_thread+0x205/0x210
 [<ffffffff81059a30>] ? autoremove_wake_function+0x0/0x30
 [<ffffffff81055b60>] ? worker_thread+0x0/0x210
 [<ffffffff81059726>] ? kthread+0x96/0xb0
 [<ffffffff81011f8a>] ? child_rip+0xa/0x20
 [<ffffffff81011161>] ? int_ret_from_sys_call+0x7/0x1b
 [<ffffffff8101191d>] ? retint_restore_args+0x5/0x6
 [<ffffffff81011f80>] ? child_rip+0x0/0x20
Code: 7c 24 50 c7 44 24 54 01 00 00 00 e9 9b fb ff ff 0f 1f 44 00 00 48 8b 54 24 70 48 8b 84 24 18 01 00 00 8b 7a 08 48 c1 e0 0a 31 d2 <48> f7 f7 48 8b bc 24 20 01 00 00 48 89 84 24 10 01 00 00 31 c0 
RIP  [<ffffffff8103ad36>] find_busiest_group+0x5f6/0xb60
 RSP <ffff8800700b1be0>---[ end trace a35477c3e4ee7a16 ]---

Showing last 100 lines from current boot
-----------------------------------------
SGI XFS Quota Management subsystem
GFS2 (built Apr 28 2010 18:46:41) installed
msgmni has been set to 3507
alg: No test for cipher_null (cipher_null-generic)
alg: No test for ecb(cipher_null) (ecb-cipher_null)
alg: No test for digest_null (digest_null-generic)
alg: No test for compress_null (compress_null-generic)
alg: No test for fcrypt (fcrypt-generic)
alg: No test for stdrng (krng)
alg: No test for ghash (ghash-generic)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
Event-channel device installed.
brd: module loaded
loop: module loaded
nbd: registered device at major 43
PPP generic driver version 2.4.2
PPP Deflate Compression module registered
PPP BSD Compression module registered
PPP MPPE Compression module registered
SLIP: version 0.8.4-NET3.019-NEWTTY (dynamic channels, max=256) (6 bit encapsulation enabled).
CSLIP: code copyright 1989 Regents of the University of California.
SLIP linefill/keepalive option.
Initialising Xen virtual ethernet driver.
blkfront: xvda: barriers enabled
 xvda: unknown partition table
tun: Universal TUN/TAP device driver, 1.6
tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>i8042.c: No controller found.
mice: PS/2 mouse device common for all mice
md: linear personality registered for level -1
md: raid0 personality registered for level 0
md: raid1 personality registered for level 1
md: raid6 personality registered for level 6
md: raid5 personality registered for level 5
md: raid4 personality registered for level 4
md: multipath personality registered for level -4
device-mapper: ioctl: 4.15.0-ioctl (2009-04-01) initialised: dm-devel@redhat.com
GACT probability NOT on
u32 classifier
    Performance counters on
    Actions configured 
Netfilter messages via NETLINK v0.30.
nf_conntrack version 0.5.0 (8192 buckets, 65536 max)
CONFIG_NF_CT_ACCT is deprecated and will be removed soon. Please use
nf_conntrack.acct=1 kernel parameter, acct=1 nf_conntrack module option or
sysctl net.netfilter.nf_conntrack_acct=1 to enable it.
xt_time: kernel timezone is -0000
IPv4 over IPv4 tunneling driver
GRE over IPv4 tunneling driver
ip_tables: (C) 2000-2006 Netfilter Core Team
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
tunl0: Disabled Privacy Extensions
ip6_tables: (C) 2000-2006 Netfilter Core Team
IPv6 over IPv4 tunneling driver
sit0: Disabled Privacy Extensions
ip6tnl0: Disabled Privacy Extensions
NET: Registered protocol family 17
NET: Registered protocol family 15
Bridge firewalling registered
Ebtables v2.0 registered
ebt_ulog: out of memory trying to call netlink_kernel_create
802.1Q VLAN Support v1.8 Ben Greear <greearb@candelatech.com>All bugs added by David S. Miller <davem@redhat.com>SCTP: Hash tables configured (established 65536 bind 65536)
registered taskstats version 1
blkfront: xvdb: barriers enabled
 xvdb: unknown partition table
XENBUS: Device with no driver: device/console/0
md: Waiting for all devices to be available before autodetect
md: If you don't use raid, use raid=noautodetect
md: Autodetecting RAID arrays.
md: Scanned 0 and added 0 devices.
md: autorun ...
md: ... autorun DONE.
kjournald starting.  Commit interval 5 seconds
EXT3-fs: mounted filesystem with writeback data mode.
VFS: Mounted root (ext3 filesystem) readonly on device 202:0.
Freeing unused kernel memory: 496k freed
Write protecting the kernel read-only data: 7204k
fsck from util-linux-ng 2.16
FATAL: Module nf_conntrack_ftp not found.
/dev/xvda: clean, 31214/1015808 files, 391337/4063232 blocks
FATAL: Module nf_nat_ftp not found.
FATAL: Module nf_conntrack_irc not found.
FATAL: Module nf_nat_irc not found.
 * Setting preliminary keymap...                                                 * Restarting OpenBSD Secure Shell server sshd     
 * Stopping NTP server ntpd
   ...done.
 * Checking for corrupt, not cleanly closed and upgrade needing tables.
 * Starting NTP server ntpd                                                      * Starting NTP server ntpd

Ubuntu 9.10 localhost hvc0

localhost login: 
[linode42786@london85 lish]#</davem@redhat.com></greearb@candelatech.com></maxk@qualcomm.com></ffff8800700b1be0></ffffffff8103ad36></ffffffff81011f80></ffffffff8101191d></ffffffff81011161></ffffffff81011f8a></ffffffff81059726></ffffffff81055b60></ffffffff81059a30></ffffffff81055d65></ffffffff810a1ca0></ffffffff8100e3d0></ffffffff815855d3></ffffffff8103ad36></ffffffff8103ad36></ffff88006e975c58></ffffffff8103ad36></ffffffff81010f42></ffffffff8105d857></ffffffff8105c7c0></ffffffff8105d74c></ffffffff815867ed></ffffffff8105d229></ffffffff8105d4df></ffffffff815855d3></ffffffff8103ad36></ffffffff8103ad36></ffff88006e2359d8></ffffffff8103ad36></ffffffff81010f42></ffffffff81062450></ffffffff8106236c></ffffffff8106ada9></ffffffff8100e552></ffffffff8100de29></ffffffff815853f0></ffffffff81038688></ffffffff81038525></ffffffff81026119></ffffffff8100922a></ffffffff8100922a></ffffffff8100922a></ffffffff8101191d></ffffffff81587baa></ffffffff8106a238></ffffffff810682e2></ffffffff81587515></ffffffff81067f56></ffffffff81067dcd></ffffffff8100e552></ffffffff8100de29></ffffffff8100922a></ffffffff8100922a></ffffffff815855d3></ffffffff8103ad36></ffffffff8103ad36></davem@redhat.com></greearb@candelatech.com></maxk@qualcomm.com> 

@kenfodder:

That worked here's the whole dump from the logview commmand in from lish.

Pid: 2677, comm: mysqld Not tainted 2.6.32.12-x86_64-linode12 

Pid: 3168, comm: ruby Tainted: G      D    2.6.32.12-x86_64-linode12 #1 

Pid: 12, comm: events/1 Tainted: G      D    2.6.32.12-x86_64-linode12 #1 

"Tainted" looks ominous. Does that have any particular meaning that is actually problematic here?

James

Tainted just means there is no freely available source to troubleshoot the kernel crash - it is not an indication of a problem.

divide error: 0000 [#1] SMP
divide error: 0000 [#2] SMP
divide error: 0000 [#3] SMP

These are the errors. I'd file a support ticket and see what linode staff thinks.

Thanks guys I'll create a support ticket with them with the details and a link to this thread.