linode won't boot: helper_main... mount failed: File exists

Ok, so what started out as a simple problem has turned in to a catastrophic failure. I started getting emails from a cron job complaining about:

Error: Write failed : Bad file descriptor

Uh oh. So I run fsck, and it starts complaining big time. So before I reboot to single user mode, I jump on the lish console, and it's being flooded with messages like this:

EXT3-fs error (device ubda) in ext3reserveinode_write: IO failure

EXT3-fs error (device ubda): ext3getinode_loc: unable to read inode block - inode=14499, block=33554435

EXT3-fs error (device ubda) in ext3reserveinode_write: IO failure

EXT3-fs error (device ubda): readblockbitmap: Cannot read block bitmap - blockgroup = 0, blockbitmap = 2553887680

EXT3-fs error (device ubda): readblockbitmap: Cannot read block bitmap - blockgroup = 0, blockbitmap = 2553887680

I bring my system down via a hard stop. Even /etc/shutdown won't work. It gave me a permission denied error.

When I try and boot back up the linode, I get the following error in the job queue:

helper_main(petercrocker, /dev/vg1/petercrocker-44280, 1, , 1): mount failed: File exists

I've searched the forums, and the one other hit I see doesn't seem relevant.

So I setup a finnix recovery profile, boot it, and run fsck on my linode, and it looks like it's not going to fix anything:

root@tty0:/sbin# fsck /dev/ubdb

fsck 1.39-WIP (31-Dec-2005)

e2fsck 1.39-WIP (31-Dec-2005)

Group descriptors look bad… trying backup blocks…

ext3 recovery flag is clear, but journal has data.

Recovery flag not set in backup superblock, so running journal anyway.

/dev/ubdb: recovering journal

Block bitmap for group 0 is not in group. (block 2553887680)

Relocate? yes

Inode bitmap for group 0 is not in group. (block 16777216)

Relocate? yes

Inode table for group 0 is not in group. (block 1949373184)

WARNING: SEVERE DATA LOSS POSSIBLE.

Relocate? yes

Eh. Not good.

So if any of you can offer advice, I've got the following questions:

1) Know a way to repair in a case like this?

2) Could a resize operation have caused this?

3) Assuming this disk is toasted, then ideally I'd like to resize the image. It's about 4 gigs now, but I made it a 10 gig image. I'd like to be able to salvage some files off of it by creating a new linode image, and mounting the old image on some mount point. But I can't resize, because the linode resizing tool is smart enough to know there's disk corruption:

fs_resize - resize failed

resize2fs 1.38 (30-Jun-2005)

Please run 'e2fsck -f /dev/vg1/petercrocker-44280' first.

Anyone got creative suggestions on how to accomplish this, or should I just pony up and order some more drive space?

1 Reply

/dev/vg1/petercrocker-44280 is an LVM logical volume on the host.

Raise a ticket or grab caker/mikegrb/tasaro in the IRC channel and see if they can fix it.

Reply

Please enter an answer
Tips:

You can mention users to notify them: @username

You can use Markdown to format your question. For more examples see the Markdown Cheatsheet.

> I’m a blockquote.

I’m a blockquote.

[I'm a link] (https://www.google.com)

I'm a link

**I am bold** I am bold

*I am italicized* I am italicized

Community Code of Conduct