Kernel panic

Something nasty just happened on my linode. Ideas?

printk: 1 messages suppressed.
sshd: page allocation failure. order:0, mode:0x20
Call Trace: [<8004c46a>] [<8004c4f6>] [<8004f641>] [<800500fa>] [<8005033d>]
   [<800508f2>] [<800508c2>] [<8021fd6f>] [<80224a36>] [<80023f97>] [<800240f5>]
   [<80017456>] [<800175ec>] [<80017bef>] [<80023486>] [<8002342d>] [<8001e2ba>]
   [<8001e2ba>] [<802f0778>] [<802f09a1>] [<802f0a6b>] [<8001c228>] [<802f0a6b>]
   [<8001c228>] [<8021f8c0>] [<8025cba2>] [<80264c89>] [<80265284>] [<8028a120>]
   [<8024a524>] [<8024a524>] [<8001c1f8>] [<80050632>] [<80050613>] [<8022008c>]
   [<802cd9c7>] [<80224d6b>] [<80224e7a>] [<80224e6e>] [<80224f84>] [<80224f5f>]
   [<80036e16>] [<80036dc4>] [<80036e60>] [<80017648>] [<80017bef>] [<80023486>]
   [<8002342d>] [<8001e2ba>] [<802f0778>] [<80307ba0>] [<802f0a6b>] [<80272776>]
   [<8001c228>] [<8021c95f>] [<802f0a6b>] [<80062508>] [<8003e002>] [<80023688>]
   [<800625e5>] [<800627d3>] [<80022f93>] [<80022637>] [<80022412>] [<80022d2c>]
   [<802f0778>] [<80022c78>] [<803091ed>]
sshd: page allocation failure. order:0, mode:0x20
Call Trace: [<8004c46a>] [<8004c4f6>] [<8004f641>] [<800500fa>] [<8005033d>]
   [<800508f2>] [<800508c2>] [<8021fd6f>] [<80023f97>] [<800240f5>] [<8001c11a>]
   [<80017456>] [<800175ec>] [<80017bef>] [<80023486>] [<8002342d>] [<8001e2ba>]
   [<8001e2ba>] [<802f0778>] [<802f09a1>] [<802f0a6b>] [<8001c228>] [<802f0a6b>]
   [<8001c228>] [<8021f8c0>] [<8025cba2>] [<80264c89>] [<80265284>] [<8028a120>]
   [<8024a524>] [<8024a524>] [<8001c1f8>] [<80050632>] [<80050613>] [<8022008c>]
   [<802cd9c7>] [<80224d6b>] [<80224e7a>] [<80224e6e>] [<80224f84>] [<80224f5f>]
   [<80036e16>] [<80036dc4>] [<80036e60>] [<80017648>] [<80017bef>] [<80023486>]
   [<8002342d>] [<8001e2ba>] [<802f0778>] [<80307ba0>] [<802f0a6b>] [<80272776>]
   [<8001c228>] [<8021c95f>] [<802f0a6b>] [<80062508>] [<8003e002>] [<80023688>]
   [<800625e5>] [<800627d3>] [<80022f93>] [<80022637>] [<80022412>] [<80022d2c>]
   [<802f0778>] [<80022c78>] [<803091ed>]
sshd: page allocation failure. order:0, mode:0x20
Call Trace: [<8004c46a>] [<8004c4f6>] [<8004f641>] [<800500fa>] [<8005033d>]
   [<800508f2>] [<800508c2>] [<8021fd6f>] [<80023f97>] [<800240f5>] [<8001c11a>]
   [<80017456>] [<800175ec>] [<80017bef>] [<80023486>] [<8002342d>] [<8001e2ba>]
   [<8001e2ba>] [<802f0778>] [<802f09a1>] [<802f0a6b>] [<8001c228>] [<802f0a6b>]
   [<8001c228>] [<8021f8c0>] [<8025cba2>] [<80264c89>] [<80265284>] [<8028a120>]
   [<8024a524>] [<8024a524>] [<8001c1f8>] [<80050632>] [<80050613>] [<8022008c>]
   [<802cd9c7>] [<80224d6b>] [<80224e7a>] [<80224e6e>] [<80224f84>] [<80224f5f>]
   [<80036e16>] [<80036dc4>] [<80036e60>] [<80017648>] [<80017bef>] [<80023486>]
   [<8002342d>] [<8001e2ba>] [<802f0778>] [<80307ba0>] [<802f0a6b>] [<80272776>]
   [<8001c228>] [<8021c95f>] [<802f0a6b>] [<80062508>] [<8003e002>] [<80023688>]
   [<800625e5>] [<800627d3>] [<80022f93>] [<80022637>] [<80022412>] [<80022d2c>]
   [<802f0778>] [<80022c78>] [<803091ed>]
sshd: page allocation failure. order:0, mode:0x20
Call Trace: [<8004c46a>] [<8004c4f6>] [<8004f641>] [<800500fa>] [<8005033d>]
   [<800508f2>] [<800508c2>] [<8021fd6f>] [<80023f97>] [<800240f5>] [<8001c11a>]
   [<80017456>] [<800175ec>] [<80017bef>] [<80023486>] [<8002342d>] [<8001e2ba>]
   [<8001e2ba>] [<802f0778>] [<802f09a1>] [<802f0a6b>] [<8001c228>] [<802f0a6b>]
   [<8001c228>] [<8021f8c0>] [<8025cba2>] [<80264c89>] [<80265284>] [<8028a120>]
   [<8024a524>] [<8024a524>] [<8001c1f8>] [<80050632>] [<80050613>] [<8022008c>]
   [<802cd9c7>] [<80224d6b>] [<80224e7a>] [<80224e6e>] [<80224f84>] [<80224f5f>]
   [<80036e16>] [<80036dc4>] [<80036e60>] [<80017648>] [<80017bef>] [<80023486>]
   [<8002342d>] [<8001e2ba>] [<802f0778>] [<80307ba0>] [<802f0a6b>] [<80272776>]
   [<8001c228>] [<8021c95f>] [<802f0a6b>] [<80062508>] [<8003e002>] [<80023688>]
   [<800625e5>] [<800627d3>] [<80022f93>] [<80022637>] [<80022412>] [<80022d2c>]
   [<802f0778>] [<80022c78>] [<803091ed>]
sshd: page allocation failure. order:0, mode:0x20
Call Trace: [<8004c46a>] [<8004c4f6>] [<8004f641>] [<800500fa>] [<8005033d>]
   [<800508f2>] [<800508c2>] [<8021fd6f>] [<80023f97>] [<800240f5>] [<8001c11a>]
   [<80017456>] [<800175ec>] [<80017bef>] [<80023486>] [<8002342d>] [<8001e2ba>]
   [<8001e2ba>] [<802f0778>] [<802f09a1>] [<802f0a6b>] [<8001c228>] [<802f0a6b>]
   [<8001c228>] [<8021f8c0>] [<8025cba2>] [<80264c89>] [<80265284>] [<8028a120>]
   [<8024a524>] [<8024a524>] [<8001c1f8>] [<80050632>] [<80050613>] [<8022008c>]
   [<802cd9c7>] [<80224d6b>] [<80224e7a>] [<80224e6e>] [<80224f84>] [<80224f5f>]
   [<80036e16>] [<80036dc4>] [<80036e60>] [<80017648>] [<80017bef>] [<80023486>]
   [<8002342d>] [<8001e2ba>] [<802f0778>] [<80307ba0>] [<802f0a6b>] [<80272776>]
   [<8001c228>] [<8021c95f>] [<802f0a6b>] [<80062508>] [<8003e002>] [<80023688>]
   [<800625e5>] [<800627d3>] [<80022f93>] [<80022637>] [<80022412>] [<80022d2c>]
   [<802f0778>] [<80022c78>] [<803091ed>]
sshd: page allocation failure. order:0, mode:0x20
Call Trace: [<8004c46a>] [<8004c4f6>] [<8004f641>] [<800500fa>] [<8005033d>]
   [<800508f2>] [<800508c2>] [<8021fd6f>] [<80023f97>] [<800240f5>] [<8001c11a>]
   [<80017456>] [<800175ec>] [<80017bef>] [<80023486>] [<8002342d>] [<8001e2ba>]
   [<8001e2ba>] [<802f0778>] [<802f09a1>] [<802f0a6b>] [<8001c228>] [<802f0a6b>]
   [<8001c228>] [<8021f8c0>] [<8025cba2>] [<80264c89>] [<80265284>] [<8028a120>]
   [<8024a524>] [<8024a524>] [<8001c1f8>] [<80050632>] [<80050613>] [<8022008c>]
   [<802cd9c7>] [<80224d6b>] [<80224e7a>] [<80224e6e>] [<80224f84>] [<80224f5f>]
   [<80036e16>] [<80036dc4>] [<80036e60>] [<80017648>] [<80017bef>] [<80023486>]
   [<8002342d>] [<8001e2ba>] [<802f0778>] [<80307ba0>] [<802f0a6b>] [<80272776>]
   [<8001c228>] [<8021c95f>] [<802f0a6b>] [<80062508>] [<8003e002>] [<80023688>]
   [<800625e5>] [<800627d3>] [<80022f93>] [<80022637>] [<80022412>] [<80022d2c>]
   [<802f0778>] [<80022c78>] [<803091ed>]
sshd: page allocation failure. order:0, mode:0x20
Call Trace: [<8004c46a>] [<8004c4f6>] [<8004f641>] [<800500fa>] [<8005033d>]
   [<800508f2>] [<800508c2>] [<8021fd6f>] [<80023f97>] [<800240f5>] [<8001c11a>]
   [<80017456>] [<800175ec>] [<80017bef>] [<80023486>] [<8002342d>] [<8001e2ba>]
   [<8001e2ba>] [<802f0778>] [<802f09a1>] [<802f0a6b>] [<8001c228>] [<802f0a6b>]
   [<8001c228>] [<8021f8c0>] [<8025cba2>] [<80264c89>] [<80265284>] [<8028a120>]
   [<8024a524>] [<8024a524>] [<8001c1f8>] [<80050632>] [<80050613>] [<8022008c>]
   [<802cd9c7>] [<80224d6b>] [<80224e7a>] [<80224e6e>] [<80224f84>] [<80224f5f>]
   [<80036e16>] [<80036dc4>] [<80036e60>] [<80017648>] [<80017bef>] [<80023486>]
   [<8002342d>] [<8001e2ba>] [<802f0778>] [<80307ba0>] [<802f0a6b>] [<80272776>]
   [<8001c228>] [<8021c95f>] [<802f0a6b>] [<80062508>] [<8003e002>] [<80023688>]
   [<800625e5>] [<800627d3>] [<80022f93>] [<80022637>] [<80022412>] [<80022d2c>]
   [<802f0778>] [<80022c78>] [<803091ed>]
sshd: page allocation failure. order:0, mode:0x20
Call Trace: [<8004c46a>] [<8004c4f6>] [<8004f641>] [<800500fa>] [<8005033d>]
   [<800508f2>] [<800508c2>] [<8021fd6f>] [<80023f97>] [<800240f5>] [<8001c11a>]
   [<80017456>] [<800175ec>] [<80017bef>] [<80023486>] [<8002342d>] [<8001e2ba>]
   [<8001e2ba>] [<802f0778>] [<802f09a1>] [<802f0a6b>] [<8001c228>] [<802f0a6b>]
   [<8001c228>] [<8021f8c0>] [<8025cba2>] [<80264c89>] [<80265284>] [<8028a120>]
   [<8024a524>] [<8024a524>] [<8001c1f8>] [<80050632>] [<80050613>] [<8022008c>]
   [<802cd9c7>] [<80224d6b>] [<80224e7a>] [<80224e6e>] [<80224f84>] [<80224f5f>]
   [<80036e16>] [<80036dc4>] [<80036e60>] [<80017648>] [<80017bef>] [<80023486>]
   [<8002342d>] [<8001e2ba>] [<802f0778>] [<80307ba0>] [<802f0a6b>] [<80272776>]
   [<8001c228>] [<8021c95f>] [<802f0a6b>] [<80062508>] [<8003e002>] [<80023688>]
   [<800625e5>] [<800627d3>] [<80022f93>] [<80022637>] [<80022412>] [<80022d2c>]
   [<802f0778>] [<80022c78>] [<803091ed>]
sshd: page allocation failure. order:0, mode:0x20
Call Trace: [<8004c46a>] [<8004c4f6>] [<8004f641>] [<800500fa>] [<8005033d>]
   [<800508f2>] [<800508c2>] [<8021fd6f>] [<80023f97>] [<800240f5>] [<8001c11a>]
   [<80017456>] [<800175ec>] [<80017bef>] [<80023486>] [<8002342d>] [<8001e2ba>]
   [<8001e2ba>] [<802f0778>] [<802f09a1>] [<802f0a6b>] [<8001c228>] [<802f0a6b>]
   [<8001c228>] [<8021f8c0>] [<8025cba2>] [<80264c89>] [<80265284>] [<8028a120>]
   [<8024a524>] [<8024a524>] [<8001c1f8>] [<80050632>] [<80050613>] [<8022008c>]
   [<802cd9c7>] [<80224d6b>] [<80224e7a>] [<80224e6e>] [<80224f84>] [<80224f5f>]
   [<80036e16>] [<80036dc4>] [<80036e60>] [<80017648>] [<80017bef>] [<80023486>]
   [<8002342d>] [<8001e2ba>] [<802f0778>] [<80307ba0>] [<802f0a6b>] [<80272776>]
   [<8001c228>] [<8021c95f>] [<802f0a6b>] [<80062508>] [<8003e002>] [<80023688>]
   [<800625e5>] [<800627d3>] [<80022f93>] [<80022637>] [<80022412>] [<80022d2c>]
   [<802f0778>] [<80022c78>] [<803091ed>]
sshd: page allocation failure. order:0, mode:0x20
Call Trace: [<8004c46a>] [<8004c4f6>] [<8004f641>] [<800500fa>] [<8005033d>]
   [<800508f2>] [<800508c2>] [<8021fd6f>] [<80023f97>] [<800240f5>] [<8001c11a>]
   [<80017456>] [<800175ec>] [<80017bef>] [<80023486>] [<8002342d>] [<8001e2ba>]
   [<8001e2ba>] [<802f0778>] [<802f09a1>] [<802f0a6b>] [<8001c228>] [<802f0a6b>]
   [<8001c228>] [<8021f8c0>] [<8025cba2>] [<80264c89>] [<80265284>] [<8028a120>]
   [<8024a524>] [<8024a524>] [<8001c1f8>] [<80050632>] [<80050613>] [<8022008c>]
   [<802cd9c7>] [<80224d6b>] [<80224e7a>] [<80224e6e>] [<80224f84>] [<80224f5f>]
   [<80036e16>] [<80036dc4>] [<80036e60>] [<80017648>] [<80017bef>] [<80023486>]
   [<8002342d>] [<8001e2ba>] [<802f0778>] [<80307ba0>] [<802f0a6b>] [<80272776>]
   [<8001c228>] [<8021c95f>] [<802f0a6b>] [<80062508>] [<8003e002>] [<80023688>]
   [<800625e5>] [<800627d3>] [<80022f93>] [<80022637>] [<80022412>] [<80022d2c>]
   [<802f0778>] [<80022c78>] [<803091ed>]

4 Replies

And now when I try to boot:

Checking for the skas3 patch in the host...found
Checking for /proc/mm...found
Checking for /dev/anon on the host...Not available (open failed with errno 6)
Linux version 2.4.26-linode32-2um (root@host4.linode.com) (gcc version 3.2.2 20030222 (Red Hat Linux 3.2.2-5)) #1 Mon Aug 2 17:53:57 EDT 2004
On node 0 totalpages: 16384
zone(0): 16384 pages.
zone(1): 0 pages.
zone(2): 0 pages.
Kernel command line: mem=64M fake_ide fakehd con=null con0=fd:0,fd:1 devfs=nomount root=/dev/ubda ubda=/linodes/shermozle/7251.fs ubdb=/linodes/shermozle/7252.fs eth0=tuntap,shermozle_0,fe:fd:40:05:35:93 token_max=400000 token_refill=512 ramdisk_size=8192
fakehd : Changing ubd_gendisk.major_name to "hd".
Calibrating delay loop... 2942.56 BogoMIPS
Memory: 59692k available
Dentry cache hash table entries: 8192 (order: 4, 65536 bytes)
Inode cache hash table entries: 4096 (order: 3, 32768 bytes)
Mount cache hash table entries: 512 (order: 0, 4096 bytes)
Buffer cache hash table entries: 1024 (order: 0, 4096 bytes)
Page-cache hash table entries: 16384 (order: 4, 65536 bytes)
Checking for host processor cmov support...Yes
Checking for host processor xmm support...No
Checking that ptrace can change system call numbers...OK
Checking syscall emulation patch for ptrace...OK
Checking that host ptys support output SIGIO...Yes
Checking that host ptys support SIGIO on close...No, enabling workaround
POSIX conformance testing by UNIFIX
Linux NET4.0 for Linux 2.4
Based upon Swansea University Computer Society NET3.039
Initializing RT netlink socket
Starting kswapd
VFS: Disk quotas vdquot_6.5.1
Journalled Block Device driver loaded
Coda Kernel/Venus communications, v5.3.18, coda@cs.cmu.edu
devfs: v1.12c (20020818) Richard Gooch (rgooch@atnf.csiro.au)
devfs: boot_options: 0x0
Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
SGI XFS with no debug enabled
SGI XFS Quota Management subsystem
pty: 256 Unix98 ptys configured
SLIP: version 0.8.4-NET3.019-NEWTTY (dynamic channels, max=256).
CSLIP: code copyright 1989 Regents of the University of California.
RAMDISK driver initialized: 16 RAM disks of 8192K size 1024 blocksize
loop: loaded (max 8 devices)
PPP generic driver version 2.4.2
PPP Deflate Compression module registered
PPP BSD Compression module registered
Universal TUN/TAP device driver 1.5 (C)1999-2002 Maxim Krasnyansky
md: linear personality registered as nr 1
md: raid0 personality registered as nr 2
md: raid1 personality registered as nr 3
md: raid5 personality registered as nr 4
raid5: measuring checksumming speed
   8regs     :  5171.600 MB/sec
   32regs    :  1905.200 MB/sec
raid5: using function: 8regs (5171.600 MB/sec)
md: multipath personality registered as nr 7
md: md driver 0.90.0 MAX_MD_DEVS=256, MD_SB_DISKS=27
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
LVM version 1.0.8(17/11/2003)
Initializing software serial port version 1
mconsole (version 2) initialized on /linodes/shermozle/.uml/shermozle/mconsole
F_SETLK failed, file already locked by pid 9031
Failed to lock '/linodes/shermozle/7251.fs', err = 11
unable to open /linodes/shermozle/7251.fs for validation
F_SETLK failed, file already locked by pid 9031
Failed to lock '/linodes/shermozle/7252.fs', err = 11
unable to open /linodes/shermozle/7252.fs for validation
Initializing stdio console driver
Netdevice 0 (fe:fd:40:05:35:93) : TUN/TAP backend -
NET4: Linux TCP/IP 1.0 for NET4.0
IP Protocols: ICMP, UDP, TCP, IGMP
IP: routing cache hash table of 512 buckets, 4Kbytes
TCP: Hash tables configured (established 4096 bind 8192)
IPv4 over IPv4 tunneling driver
GRE over IPv4 tunneling driver
Linux IP multicast router 0.06 plus PIM-SM
ip_conntrack version 2.1 (466 buckets, 3728 max) - 288 bytes per conntrack
ip_tables: (C) 2000-2002 Netfilter core team
ipt_recent v0.3.1: Stephen Frost <sfrost@snowman.net>.  http://snowman.net/projects/ipt_recent/
arp_tables: (C) 2002 David S. Miller
NET4: Unix domain sockets 1.0/SMP for Linux NET4.0.
NET4: Ethernet Bridge 008 for NET4.0
802.1Q VLAN Support v1.8 Ben Greear <greearb@candelatech.com>All bugs added by David S. Miller <davem@redhat.com>VFS: Cannot open root device "ubda" or 62:00
Please append a correct "root=" boot option
Kernel panic: VFS: Unable to mount root fs on 62:00
 <6>SysRq : Show Regs

EIP: 0073:[<80274491>] CPU: 0 Not tainted ESP: 007b:40001fb4 EFLAGS: 00000206
    Not tainted
EAX: 00000000 EBX: 00007e8f ECX: 00000013 EDX: 00007e8f
ESI: 00000000 EDI: 00000000 EBP: 00000000 DS: 007b ES: 007b
Call Trace: [<802d6057>] [<802d848e>] [<801b0872>] [<802dd4c0>] [<8019c179>]
   [<8001dc9e>] [<8001326c>] [<800132fe>] [<800028e8>] [<802dd440>] [<8000e5c4>]
   [<8000381d>] [<802d0398>] [<8000eb3b>] [<802d0398>] [<8000eb59>] [<8000e5d2>]
   [<802740d2>] [<8000e5c4>] [<80196e39>] [<8000e5c4>] [<80196e14>] [<8000e5c4>]
   [<801a04b5>] [<8000e5c4>] [<8019fea5>] [<80274228>] [<80274491>]</davem@redhat.com></greearb@candelatech.com></sfrost@snowman.net> 

and there seems to be a dead screen:

[shermozle@host5 lish] There is a screen on:
        9030.shermozle  (Dead ???)
Remove dead screens with 'screen -wipe'.
There is no screen to be resumed matching shermozle.
Your Linode isn't running, or another console session is already active.

Two files on the host (which form the filesystems for your Linode) are still locked by one of the host processes associated with the instance of your Linode that failed. If issuing another reboot from the Linode Platform Manager doesn't bring it back up, you'll need to raise a support ticket to get caker to manually kill off anything that remains of the previous instance.

I can't kill the lock, so I've put in a ticket.

Any idea about the cause of the crash? It looks like, possibly, sshd ran away or something.

sshd was trying to allocate 4k of memory and couldn't, but that doesn't necessarily mean sshd was at fault - it could just be the hapless victim of some other program that ate all the memory. Do the logs give any indications of other trouble (other daemons dying, apache, etc. bitching about no memory) at the time of the failure?

Reply

Please enter an answer
Tips:

You can mention users to notify them: @username

You can use Markdown to format your question. For more examples see the Markdown Cheatsheet.

> I’m a blockquote.

I’m a blockquote.

[I'm a link] (https://www.google.com)

I'm a link

**I am bold** I am bold

*I am italicized* I am italicized

Community Code of Conduct