Fedora Linode "stuck" on older kernel...?
I have two Linodes, an 8GB and a 4GB. Both are running Fedora 36. I also have a personal Linux box here at my house that also runs Fedora 36, which for this discussion I'll mention as a "baseline".
For the last several Fedora versions, I've done an "in place" system upgrade via DNF. This has worked well for all three boxes, with no obvious errors that I can remember. That said, I can't remember what version of Fedora the Linodes were initially created with, so I don't know if there are any custom scripts added by Linode that may no longer be working as expected.
Every day, I check for updates:
# dnf check-update
If any are found, I install them:
# dnf -y update
If any critical services are updated (httpd, mariadb, php-fpm, etc.), I restart them. If the kernel is updated, I reboot the entire box.
Recently, the 4GB Linode has started exhibiting an odd behavior with standard kernel updates. The new kernel appears to install without errors. However, it seems to have trouble removing an old kernel:
Running scriptlet: kernel-core-5.19.4-200.fc36.x86_64 8/8
kdump: kernel 5.18.16-200.fc36.x86_64 doesn't exist
kdump: Couldn't find current running kernel
When I run uname -a, it tells me it's still running kernel 5.18.16-200, and it seems to be "stuck" on that version. As of this writing, I've updated the kernel several times, and no matter how many kernels I upgrade to, 5.18.16-200 never goes away and always becomes the running kernel.
$ rpm -q kernel
kernel-5.18.16-200.fc36.x86_64
kernel-5.19.4-200.fc36.x86_64
kernel-5.19.6-200.fc36.x86_64
$ uname -a
Linux [domain.name.tld] 5.18.16-200.fc36.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Aug 3 15:44:49 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Now, I'm no expert when it comes to grub and grubby, but as far as I can tell, the "latest" kernel is currently set to the default:
# grubby --info=ALL
index=0
kernel="/boot/vmlinuz-5.19.6-200.fc36.x86_64"
args="ro console=ttyS0,19200n8 net.ifnames=0 rhgb LANG=en_US.UTF-8"
root="/dev/sda"
initrd="/boot/initramfs-5.19.6-200.fc36.x86_64.img"
title="Fedora Linux (5.19.6-200.fc36.x86_64) 36 (Server Edition)"
id="082d98303aeb484d8eb41bc347a64c7c-5.19.6-200.fc36.x86_64"
index=1
kernel="/boot/vmlinuz-5.19.4-200.fc36.x86_64"
args="ro console=ttyS0,19200n8 net.ifnames=0 rhgb LANG=en_US.UTF-8"
root="/dev/sda"
initrd="/boot/initramfs-5.19.4-200.fc36.x86_64.img"
title="Fedora Linux (5.19.4-200.fc36.x86_64) 36 (Server Edition)"
id="082d98303aeb484d8eb41bc347a64c7c-5.19.4-200.fc36.x86_64"
index=2
kernel="/boot/vmlinuz-0-rescue-082d98303aeb484d8eb41bc347a64c7c"
args="ro console=ttyS0,19200n8 net.ifnames=0 rhgb LANG=en_US.UTF-8"
root="/dev/sda"
initrd="/boot/initramfs-0-rescue-082d98303aeb484d8eb41bc347a64c7c.img"
title="Fedora Linux (0-rescue-082d98303aeb484d8eb41bc347a64c7c) 36 (Server Edition)"
id="082d98303aeb484d8eb41bc347a64c7c-0-rescue"
# grubby --default-kernel
/boot/vmlinuz-5.19.6-200.fc36.x86_64
Kernel 5.18.16-200 isn't even in that list. So why is it still the running kernel?
I should point out that this is NOT a problem with the 8GB Linode, or with my personal box at home. I'm fairly certain this is a grub or grubby issue, but I can't see anything obviously wrong.
At this point, I'm not sure what to do. I've considered simply erasing the 5.18.16-200 kernel and seeing what happens; I have daily backups turned on for both Linodes, so I can always restore the most recent one. I'd rather not do that, though, because it'll be a hassle to deal with. I know that, as a last resort, I can always go back to my old "upgrade" procedure: spin up a new Linode, reinstall all software, copy over all configs and data, swap DNS entries, and spin down the old box. But I really don't want to go that route unless absolutely necessary.
7 Replies
What kernel is the Linode set to in its configuration profile?
If you mean the Configuration panel in the Linode Manager, it's set to "GRUB 2". As I understand it, that lets the distro's grub set the kernel, which is what I want.
Do you have more than one kernel installed? Is grub defaulting to the first one in the list?
I'm very rusty on this stuff but I think there's some file in /boot that you can edit to set the first choice. I forget what it's called.
I seem to remember having exactly this same problem with Ubuntu once…
-- sw
In Fedora, the grub boot order is managed by grubby. I included the output of grubby --info=ALL and grubby --default-kernel in my original post.
As of right now (and as of my original post), 5.19.6-200 is the latest kernel and is set as the default. 5.18.16-200, the "stuck" kernel, isn't even in the --info=ALL list at all. Yet that's the kernel that the system seems to be running under.
Hopefully that explains my frustration with this. From everything I can tell, it should be running the latest kernel, but it's not.
I did some Googling on "fedora grubby configuration ignored" and interestingly got several links back here to the Linode forums. It turns out Linode has had quite a few rough spots with Fedora, GRUB, and BLS.
Further imagine my surprise when I (re)discovered that I had posted to that first thread three years ago.
I compared the /etc/default/grub files on both of my Linodes. The 8GB (the one that does not have the "stuck" kernel) has GRUB_ENABLE_BLSCFG=true. The 4GB (the one with the "stuck" kernel) has GRUB_ENABLE_BLSCFG=false. These Linodes were both likely upgraded from Fedora 29 all those years ago. The 4GB is my "beta" box that only hosts a few small sites, while the 8GB is my "alpha" box that hosts my flagship site. I tend to upgrade the 4GB first as a test before performing upgrades on the 8GB. It's possible that I flipped this flag to false on the 4GB as part of the "hack" to get the upgrade to work, and that Linode tweaked their BLS configs before I upgraded the 8GB, so it defaulted to true.
When I get a chance, I'll try flipping the 4GB's BLS flag to true and see if that "unsticks" the kernel. But if anyone has more experience with this issue or has better advice, I'd love to hear it before taking that plunge.
By the way, the first link also mentions installing a grubby-deprecated package. I do believe I had that package installed, per the instructions in that thread, but that dnf forced me to uninstall it not that long ago due to a conflict with the main grubby package. I don't see how this could be an issue, though, since the deprecated package has been removed from both Linodes, and the 8GB is moving the the new kernel on each update without a problem.
Well, that was enlightening.
I tried setting GRUB_ENABLE_BLSCFG=true and ran grub2-mkconfig to rebuild /boot/grub2/grub.cfg. I then rebooted my Linode and crossed my fingers.
I was presented with a lovely grub prompt and that's about it. I tried a number of things to get the system to boot, but nothing seemed to work. I resorted to restoring the Linode from its last backup (this morning).
I am nearly 100% convinced that this is now the issue. If I look at the current contents of /boot/grub2/grub.cfg, it has the "stuck" 5.18.16-200 kernel, as well several other old kernels that have since been removed. That tells me that this is what the system is booting off of, and since 5.18.16-200 is the only kernel in that config that it can find, that's the one it's using.
But I'm not exactly sure how to "fix" the issue. I know I need to set GRUB_ENABLE_BLSCFG=true, and I probably need to run grub2-mkconfig at some point. But is there a step I'm missing somewhere in between?
Latest attempt:
- Take a fresh snapshot backup.
- Edit
/etc/default/grubto setGRUB_ENABLE_BLSCFG=true - Run
grub2-switch-to-blscfg - Run
grub2-mkconfig -o /boot/grub2/grub.cfg - Forced the latest kernel to be the default with
grubby --set-default=[kernel_path] - Reboot
And all I got was a lousy grub> prompt.
At this point, I have no idea what else to do. I will probably put in a formal support ticket (since I'm not getting any help here), but I strongly suspect I'm just going to have to build a new Linode from scratch and copy over my old data.