linode won't boot: helper_main... mount failed: File exists
Error: Write failed : Bad file descriptor
Uh oh. So I run fsck, and it starts complaining big time. So before I reboot to single user mode, I jump on the lish console, and it's being flooded with messages like this:
EXT3-fs error (device ubda) in ext3reserveinode_write: IO failure
EXT3-fs error (device ubda): ext3getinode_loc: unable to read inode block - inode=14499, block=33554435
EXT3-fs error (device ubda) in ext3reserveinode_write: IO failure
EXT3-fs error (device ubda): readblockbitmap: Cannot read block bitmap - blockgroup = 0, blockbitmap = 2553887680
EXT3-fs error (device ubda): readblockbitmap: Cannot read block bitmap - blockgroup = 0, blockbitmap = 2553887680
I bring my system down via a hard stop. Even /etc/shutdown won't work. It gave me a permission denied error.
When I try and boot back up the linode, I get the following error in the job queue:
helper_main(petercrocker, /dev/vg1/petercrocker-44280, 1, , 1): mount failed: File exists
I've searched the forums, and the one other hit I see doesn't seem relevant.
So I setup a finnix recovery profile, boot it, and run fsck on my linode, and it looks like it's not going to fix anything:
root@tty0:/sbin# fsck /dev/ubdb
fsck 1.39-WIP (31-Dec-2005)
e2fsck 1.39-WIP (31-Dec-2005)
Group descriptors look bad… trying backup blocks…
ext3 recovery flag is clear, but journal has data.
Recovery flag not set in backup superblock, so running journal anyway.
/dev/ubdb: recovering journal
Block bitmap for group 0 is not in group. (block 2553887680)
Relocate
Inode bitmap for group 0 is not in group. (block 16777216)
Relocate
Inode table for group 0 is not in group. (block 1949373184)
WARNING: SEVERE DATA LOSS POSSIBLE.
Relocate
Eh. Not good.
So if any of you can offer advice, I've got the following questions:
1) Know a way to repair in a case like this?
2) Could a resize operation have caused this?
3) Assuming this disk is toasted, then ideally I'd like to resize the image. It's about 4 gigs now, but I made it a 10 gig image. I'd like to be able to salvage some files off of it by creating a new linode image, and mounting the old image on some mount point. But I can't resize, because the linode resizing tool is smart enough to know there's disk corruption:
fs_resize - resize failed
resize2fs 1.38 (30-Jun-2005)
Please run 'e2fsck -f /dev/vg1/petercrocker-44280' first.
Anyone got creative suggestions on how to accomplish this, or should I just pony up and order some more drive space?
1 Reply
Raise a ticket or grab caker/mikegrb/tasaro in the IRC channel and see if they can fix it.