Weird Behavior from Ubuntu 8.04
I set up an Ubuntu 8.04 Linode earlier tonight (ahem…yesterday evening now, I guess). Around 1 AM I got an email from the friendly monitoring program telling me something was horribly wrong and my server was performing an average of 422 disk IO operations a second.
I poked around trying to figure out what could be doing that and I noticed /var/log/syslog and /var/log/messages were getting spammed up with this:
> Apr 28 02:46:46 elsa init: tty0 main process (6163) terminated with status 1
Apr 28 02:46:46 elsa init: tty0 main process ended, respawning
Apr 28 02:46:56 elsa init: tty0 main process (6166) terminated with status 1
Apr 28 02:46:56 elsa init: tty0 main process ended, respawning
Apr 28 02:47:06 elsa init: tty0 main process (6181) terminated with status 1
Apr 28 02:47:06 elsa init: tty0 main process ended, respawning
Apr 28 02:47:16 elsa init: tty0 main process (6184) terminated with status 1
Apr 28 02:47:16 elsa init: tty0 main process ended, respawning
Apr 28 02:47:26 elsa init: tty0 main process (6212) terminated with status 1
Apr 28 02:47:26 elsa init: tty0 main process ended, respawning
Apr 28 02:47:36 elsa init: tty0 main process (6217) terminated with status 1
Apr 28 02:47:36 elsa init: tty0 main process ended, respawning
I wasn't sure if that was causing my disk IO woes, but that sure as hell looks like it merited fixing. I kept an eye on ps aux and the log to figure out what process was dying, and it turned out to be processes like this:
> root 6423 0.0 0.1 1628 580 ? Ss 02:52 0:00 /sbin/getty 38400 tty0
My friend and I did some googling, and we turned up with this forum post
Ubuntu handles starting the ttys slightly differently than described in that post: there's a file in /etc/event.d that tells it to respawn /sbin/getty for tty0 if it dies. I commented the respawn lines out and rebooted.
> nevans@elsa:~$ cat /etc/event.d/tty0
tty0 - getty
#
This service maintains a getty on tty1 from the point the system is
started until it is shut down again.
start on stopped rc2
start on stopped rc3
start on stopped rc4
start on stopped rc5
stop on runlevel 0
stop on runlevel 1
stop on runlevel 6
respawn
exec /sbin/getty 38400 tty0
Lassie was kind enough to turn my VM back on for me. It booted up without a problem, and the out-of-band console looked happy enough. It's using tty1 now, but I didn't check to see if its using tty0 before or not. I'd wager it wasn't.
I ssh'd in and found that my logs were no longer being spammed with messages about init dying. Yay!
Can anyone confirm whether or not this is a problem caused by Xen trying to kill tty0? Is this a bug? Is there a better solution to the problem…?
2 Replies
Thanks to duplicity backups I was able to tell the file hadn't been there that morning…