Ubuntu kernel 2.6.32-32-virtual hangs with pv-grub-x86_32

I've never got the virtual image to play well with linode, I use the EC2 kernel instead which works perfectly.

This is a snippet from my stack script which installs the ec2 kernel

#installs ubuntu ec2 kernel which works best on linode
#configures grub to load kernel after 3 seconds
#sets console to hvc0 so you can access via listh
#enables delayacct so iotop works
aptitude -y install linux-ec2 grub
mkdir -p /boot/grub/
echo "default 0" > /boot/grub/menu.lst
echo "timeout 3" >> /boot/grub/menu.lst
echo "Choose install package maintainer's version, press enter to continue"
read
update-grub
sed -i 's/indomU=detect/indomU=false/' /boot/grub/menu.lst
sed -i 's#kopt=root=.* ro#kopt=root=/dev/xvda ro#' /boot/grub/menu.lst
sed -i 's#groot=.*#groot=(hd0)#' /boot/grub/menu.lst
sed -i 's/defoptions=quiet splash/defoptions=quiet delayacct console=hvc0/' /boot/grub/menu.lst
chmod 0600 /boot/grub/menu.lst
update-grub

Yep I've had the same issue with this specific kernel. Earlier versions of 2.6.32-virtual worked fine too. I didn't have any luck with the 2.6.35 backport though.

Installing the EC2 kernel works - even if it does produce some (grub?) error messages about not being a bzImage.

The 2.6.32-33-virtual package in lucid-proposed seems to fix this for me though. I haven't yet dug out any relevant Launchpad issues yet though.

What's the grub error you get from ec2? I've never had an issue.

More info:

This is on my own Xen testing machine (custom built 4.01, with an Ubuntu 10.04 dom0 with custom built 2.6.32-40 pvops dom0 kernel and pvgrub domUs), not a production Linode instance :)

But this was the only place I found anyone having the same trouble with 2.6.32-32-virtual (actually it uses the generic-pae kernel on 32bit).

The Ubuntu EC2 kernel though still works fine, even though I get a serious sounding pvgrub error message:

root  (hd0)
 Filesystem type is ext2fs, using whole disk
kernel  /boot/vmlinuz-2.6.32-316-ec2 root=/dev/xvda1 ro xencons=hvc0 quiet splash 
initrd  /boot/initrd.img-2.6.32-316-ec2

ERROR Invalid kernel: xc_dom_probe_bzimage_kernel: kernel is not a bzImage

xc_dom_bzimageloader.c:350: panic: xc_dom_probe_bzimage_kernel: kernel is not a bzImage

The Ubuntu 2.6.32-33-virtual kernel in lucid-proposed looks like it has some buggy upstream patches that were included in 2.6.32-32 reverted. I suspect the EC2 kernel was released on a different schedule and doesn't (currently at least) have the bad code included.

I'm chalking it up to regressions that have already been fixed - just not fully released yet.

I managed to find the relevant Ubuntu bug report…

#791513

@styro:

This is on my own Xen testing machine (custom built 4.01, with an Ubuntu 10.04 dom0 with custom built 2.6.32-40 pvops dom0 kernel and pvgrub domUs), not a production Linode instance :)

Ah :P Anyway the error is because the ec2 kernels are gzipped. I'm guessing that your xen instance doesn't like gzipped kernels, I know diddly squat about xen though so can't help there.

@styro:

I managed to find the relevant Ubuntu bug report…

#791513

Thanks styro, you rock.

I just noticed that some of my linodes were running 2.6.32-32 successfully and I was just about to start banging my head against a wall trying to figure out the difference between the working and non-working ones.

It turns out, the ones that work are Linode 512s. From referenced bug:

> My testing suggests that this issue affects any guest which is assigned more than 578MB of RAM.

…

The failure mode is that the guest spins on 100% CPU on the first core, before any kernel messages are printed (even without the "quiet" option).

My linode graphs can certainly attest to the last bit. I'm glad to have this issue finally identified.

@verb:

I just noticed that some of my linodes were running 2.6.32-32 successfully and I was just about to start banging my head against a wall trying to figure out the difference between the working and non-working ones.

It turns out, the ones that work are Linode 512s. From referenced bug:

> My testing suggests that this issue affects any guest which is assigned more than 578MB of RAM.

…

The failure mode is that the guest spins on 100% CPU on the first core, before any kernel messages are printed (even without the "quiet" option).

My linode graphs can certainly attest to the last bit. I'm glad to have this issue finally identified.

Feel free to add your 2bits to the launchpad bug report. Pointing out that it affects Linode users, should help speed up 2.6.32-33 being released as an update.