I just upgraded my CentOS 8 Linode, and now it won't boot. How do I fix this problem?

There appears to be a bug in CentOS 8 which results in its boot process failing after a kernel upgrade. This bug has two major effects:

• The default GRUB boot entry was set to the last entry in the list, CentOS's "rescue kernel" boot
• This boot entry had faulty resource locations and failed to boot

I have instructions for both resolving the immediate issue, then actually fixing each of these problems.

Booting back into CentOS 8 after experiencing this bug

When this bug affects your Linode, it will be stuck in its boot process on an error screen that reads similar to this:

Booting `CentOS Linux (0-rescue-3e729c2d7c094902af0333ce40564ffe) 8 (Core)'

error: file `/vmlinuz-0-rescue-3e729c2d7c094902af0333ce40564ffe' not found.
error: you need to load the kernel first.

Press any key to continue...

To resolve this immediate problem, log into your Linode using the Lish console. You may then press any key to clear the error and present the full list of your Linode's boot options. From this menu, use the arrow keys on your keyboard to highlight the first boot entry in the list, then press Enter to boot into that entry.

Your Linode should now start booting as it usually does into its normal CentOS 8 environment, but you will need to run a few commands from Lish to permanently fix these issues.

Restoring the first boot entry as the default

In order for your Linode to successfully boot into its expected operating environment without requiring any intervention, it must default to one of the non-rescue kernel boot entries.

The first boot entry in this list is generally a good default selection for this purpose. You may designate this default by running the following commands:

sudo grub2-set-default 0
sudo grub2-mkconfig -o /boot/grub/grub.cfg

If you know what you are doing and would prefer to use a different boot entry, simply change the 0 to the number of whichever boot entry you would prefer to use. The counting starts from 0, so 0 is the first boot entry, 1 is the second boot entry, etc.

With this fix in place, it is now time to repair the "rescue kernel" boot entry in case you ever need to use it.

Repair the CentOS "rescue kernel" boot entry

CentOS generates a "rescue kernel" boot entry as a fallback boot entry in the event of any issues that may occur with the installation's standard kernel. Although it serves a similar purpose to Linode's own Rescue Mode, this is a feature specific to the RedHat-based family of Linux distributions provided by the Dracut initramfs generator.

This bug causes this "rescue kernel" boot entry to become unbootable due to incorrect file paths for the kernel and the initramfs, which can present a major problem in the event that your Linode's standard CentOS kernel is unusable for whatever reason.

To fix this boot entry, run the following commands from your Lish console:

sudo sed -i 's/^linux /linux \/boot/g' /boot/loader/entries/3e729c2d7c094902af0333ce40564ffe-0-rescue.conf
sudo sed -i 's/^initrd /initrd \/boot/g' /boot/loader/entries/3e729c2d7c094902af0333ce40564ffe-0-rescue.conf
sudo grub2-mkconfig -o /boot/grub/grub.cfg

These commands will fix the faulty file paths in this GRUB boot entry, then rebuild the Linode's GRUB configuration.

Depending on when you experience this error, you may need to use a different set of paths for the first two entries if you get an error message that looks like this:

sed: can't read /boot/loader/entries/3e729c2d7c094902af0333ce40564ffe-0-rescue.conf: No such file or directory

Perhaps the easiest way to determine the correct file paths is to run this command:

sudo ls /boot/loader/entries/

You should receive some output that looks similar to this:

8cc181c19e1a4f25fcdb5d1cd0e1f777-0-rescue.conf
8cc181c19e1a4f25fcdb5d1cd0e1f777-4.18.0-147.3.1.el8_1.x86_64.conf
8cc181c19e1a4f25fcdb5d1cd0e1f777-4.18.0-80.7.1.el8_0.x86_64.conf

The exact file paths will differ on your Linode, but it should be possible to copy the output text for the "rescue" configuration in Lish, then paste it onto the end of each of these commands:

sudo sed -i 's/^linux /linux \/boot/g' /boot/loader/entries/
sudo sed -i 's/^initrd /initrd \/boot/g' /boot/loader/entries/

This particular example would yield the following full commands to run:

sudo sed -i 's/^linux /linux \/boot/g' /boot/loader/entries/8cc181c19e1a4f25fcdb5d1cd0e1f777-0-rescue.conf
sudo sed -i 's/^initrd /initrd \/boot/g' /boot/loader/entries/8cc181c19e1a4f25fcdb5d1cd0e1f777-0-rescue.conf

Be sure to rebuild your GRUB configuration afterwards by running this command:

sudo grub2-mkconfig -o /boot/grub/grub.cfg

Rebooting and testing changes

After running the commands from the previous sections, leave your Lish console open and reboot your Linode from Cloud Manager. You will need to use the Lish console to determine whether the changes were successful.

First, be sure that the first boot entry gets highlighted by default (unless you wanted a different default boot entry):

https://i.imgur.com/VLZ75gw.png

Next, try booting into the "rescue kernel" entry to see that it is bootable:

https://i.imgur.com/TcgpwjV.png

You may notice some graphical issues on your Lish console as indicated in this screenshot. However, as long as the arrow keys let you navigate the menu and the Enter key lets you boot into the "rescue kernel" boot entry, you should be all set with both bugfixes in place!

If either one of these issues remains, review the above instructions to ensure that you had accurately deployed these fixes, then run these tests again to see whether they are being applied to your Linode. Feel free to leave a comment on this question if you need further assistance.

Should you find that both fixes are working as expected, simply reboot your Linode from Cloud Manager and enjoy its normal operations.

I just wanted to add a possible workaround in case the steps above do not work. You can switch your Linode's kernel to use our kernel instead of the upstream kernel. Here's our guide on how to switch kernels:

How to Switch your Kernel

The reason why this can work is because when Linodes use our kernel, they bypass GRUB altogether when they boot.

I’m not sure if this is the same issue, but I encountered something very similar on Fedora - it was always selecting the last-but-one kernel in the boot list by default (which is always the oldest kernel), despite it being set to boot the one at index 0.

In my case I disabled “BLSCFG” - a new type of configuration system for GRUB from what I understand.

In /etc/default/grub, set the line:

GRUB_ENABLE_BLSCFG=false

The rebuild the Grub config:

sudo grub2-mkconfig -o /boot/grub/grub.cfg

I’ve gone through several minor kernel updates in the last couple of weeks without a hitch.

And yes using a Linode kernel also works as, like the CentOS issue, it bypasses GRUB.