Lish is unresponsive, but I can SSH in
When I load the Lish console, I see some messages from when my Linode booted, but I can't enter any input, and it doesn't show a login prompt.
However, I can login via SSH and run commands normally there. My services are otherwise working normally.
How can I troubleshoot the Lish issue? I'm running Ubuntu 16.04.
1 Reply
It sounds like the getty
for the serial console for your Linode isn't running. Here's some background on gettys and serial consoles:
About getty
getty
is short for 'get tty', and it is a process that manages connections to a terminal (also known as a tty
):
Your server can have different terminals. On Linode, Lish accesses the serial console/terminal for your server, and the serial console for KVM Linodes is labelled ttyS0
.
If you run the following command from your SSH connection, you should see the getty
process on ttyS0
:
root@localhost:~ ps aux | grep [t]tyS0
root 3325 0.0 0.2 15756 2144 ttyS0 Ss+ 06:20 0:00 /sbin/agetty --keep-baud 115200 38400 9600 ttyS0 vt220
Your serial console getty
lets you invoke the login process on that terminal. As well, if you enter your username (but not the password) at the Lish console and then run ps aux | grep [t]tyS0
on your separate SSH connection, you will see that the getty
process disappears, and in its place is the login process running on ttyS0
:
root@localhost:~ ps aux | grep [t]tyS0
root 3325 0.0 0.3 68744 3500 ttyS0 Ss+ 06:20 0:00 /bin/login --
This is because getty
exits and hands off control to the login process. After a successful login, a bash shell is run on ttyS0
. If the user exits the bash shell or the login fails, the getty
process will automatically respawn.
The init system for your deployment is responsible for setting up a getty
process. Ubuntu 16.04 uses systemd
to start and manage your services.
On systemd
, there are two ways to set up the serial console:
Pass
console=ttyS0
as an option to the kernel when booting. Linode does this for you when you use the kernel provided by our Manager. Or, you can configure your boot loader (Grub) to pass this option.systemd
should then automatically invoke/lib/systemd/system/serial-getty@.service
with this console.You can manually enable the serial console:
systemctl enable serial-getty@ttyS0.service
These subjects are also covered in the following article:
Gettys on Serial Consoles (and Elsewhere)
Troubleshooting your unresponsive Lish console
From your SSH connection, you should check to see if a getty
is running on ttyS0
:
ps aux | grep [t]tyS0
If you don't see any such processes, then this is probably why Lish is unresponsive.
You can also search your system logs and look for related messages. The following output is what you would normally see when Lish is responding as expected after booting your Linode:
root@localhost:~ journalctl -b | grep ttyS0
May 07 06:20:25 localhost kernel: Command line: root=/dev/sda console=tty1 console=ttyS0 ro devtmpfs.mount=1
May 07 06:20:25 localhost kernel: Kernel command line: root=/dev/sda console=tty1 console=ttyS0 ro devtmpfs.mount=1
May 07 06:20:25 localhost kernel: console [ttyS0] enabled
May 07 06:20:25 localhost kernel: 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
May 07 06:20:26 localhost systemd[1]: Found device /dev/ttyS0.
May 07 06:20:27 localhost systemd[1]: Started Serial Getty on ttyS0.
You can check to see if your serial-getty@.service
is enabled:
root@localhost:~ systemctl list-unit-files | grep serial-getty@.service
serial-getty@.service enabled
If this is disabled, you can try manually enabling it:
systemctl enable serial-getty@ttyS0.service
systemctl start serial-getty@ttyS0.service