Runlevel 'unknown' after Ubuntu 10.04 Upgrade
I (tried) to upgrade my linode today from 9.10 to 10.04 Lucid. Unfortunately after the upgrade, most of the services aren't coming up. When I check the runlevel, it comes back with 'unknown'. I had to issue "telinit 2" to get the services to come up. However, things are still not 100%. Has anyone else seen this? A quick google search shows that a handful of people out there have experienced the same thing.
I am particularly aggravated at the fact that I can't boot into my backup linode disk image. Right before I started the upgrade process, I made a duplicate image of the root (xvda) image. When things went sideways, I changed the config profile and tried to boot into the backup image. The networking comes up (I can ping), but no SSH access. Has anyone observed that?
And my final rant/frustration is Lish. I was able to see the console via ssh and also AJAX in the linode manager. However, I can't get a login prompt. It has been a while since I used lish, but I thought it was interactive?
Any thoughts, suggestions, and/or nice words are appreciated. Thanks.
9 Replies
Did you have only a "root" and "my data" partitions, or did you have /usr and friends separately too? Because in the latter case, it's quite probably the reason why you can't boot from the backup - services' binaries got upgraded too, after all.
I deployed a new 10.04 to experiment. The console and came up and I was able to login. But once I configurated the hostname, firewall, and /etc/network/interfaces and then rebooted, again no console. So I am not sure what is going on there.
I would hate to rebuild this thing from scratch… but that may be the best way to handle it. I have done several upgrades with no problem in the past. Maybe its time to start "clean." But the console thing is really bothering me. I have no "out of band" access as it stands now.
Except that nothing seemed to work anymore, and for some reason beyond fathom they are now forcing you to use the 'service' command and denying /etc/init.d/ completely. Everytime I tried to reload, restart, start or stop a service it would just sit there. Even when using the switch to 'not wait for a response', the command just hung.
This was about the worst upgrade experience I've ever had with Ubuntu.
> This was about the worst upgrade experience I've ever had with Ubuntu.
+1. I'm not yet willing to say I didn't screw it up somehow. However, I am starting to think this is related to the "Upstart" changes. I don't have a lot of background, but I guess it is supposed to handle the "old" init.d type stuff. I found some info on ubuntuforums that seems to point to a solution. We'll see what happens.
I have decided that I will not rebuild the server though. That will take too damn long. I have to get this fixed so that when I boot the server, everything comes up and its listening.
Thanks for the help so far.
auto lo
iface lo inet loopback
# post-up iptables-restore < /etc/iptables.up.rules
# post-up echo 1 > /proc/sys/net/ipv4/ip_forward
# post-up echo 1 > /proc/sys/net/ipv4/conf/tun0/forwarding
auto eth0
iface eth0 inet static
address [REMOVED]
netmask 255.255.255.0
gateway [REMOVED]
post-up ifconfig eth0:0 [REMOVED] netmask 255.255.255.0 up
That has gotten be back into runlevel 2 upon bootup. So apparently some of that post-up code is breaking the boot (upstart?) process. I still don't have a console login prompt though. And, obviously, I still need to make those lines of code work somehow…
> post-up echo 1 > /proc/sys/net/ipv4/ip_forward
post-up echo 1 > /proc/sys/net/ipv4/conf/tun0/forwarding
Interface tun0 doesn't exist anymore. That was causing the problem. Actually, neither of those lines is needed for me anymore. They were required when I was using openvpn. But now that I am running openvpn-as, everything is automagic. So this seemed like a problem due to the upgrade, but it is actually my mistake. I just never reboot my server…. so it didn't show until the upgrade.
HOWEVER. I still don't have console!
matt@host:~$ ps aux | grep getty
matt 2864 0.0 0.1 3324 796 pts/0 S+ 14:28 0:00 grep getty
so it looks like init is once again failing to execute certain stuff. when i manually run the command in /etc/init/hvc0.conf
exec /sbin/getty 38400 hvc0
the console comes up. so something in the init process seems to not be going smoothly. any idea on where i can find some logging for this?
# hvc0 - getty for Linode console
start on (virtual-filesystems
or starting rcS
or starting mountall-shell)
stop on runlevel 0
stop on runlevel 1
stop on runlevel 6
respawn
exec /sbin/getty 38400 hvc0
Apparently the original "start on runlevel 2" idea wasn't working out. I kindly suggest that the linode distro deployment guy look at testing this out. I am pretty sure this is broken out of the box.
This is now officially the worst OS upgrade I have ever experienced. I am finding bugs constantly. Now I am spending Saturday night trying to reinstate some system logging…