Upstart troubleshooting
[ 0.825574] EXT3-fs: mounted filesystem with ordered data mode. Begin: Running /scripts/local-bottom ... Done. Done. Begin: Running /scripts/init-bottom ... Done.
so it appears to be booting and dying sometime after the initrd mounts /
I add --verbose to the command line in grub, and I see
Begin: Running /scripts/init-bottom ... Done. init: Handling startup event init: mountall goal changed from stop to start init: mountall state changed from waiting to starting init: hostname goal changed from stop to start init: hostname state changed from waiting to starting init: Handling starting event init: plymouth goal changed from stop to start
...
init: hwclock state changed from stopping to killed init: hwclock state changed from killed to post-stop init: hwclock state changed from post-stop to waiting init: Handling stopped event init: plymouth main process (183) executable changed init: plymouth main process (183) became new process (186) init: plymouth state changed from spawned to post-start init: plymouth post-start process (187) init: plymouth post-start process (187) exited normally init: plymouth state changed from post-start to running init: Handling started event
so I am thinking the problem is in plymouth; a graphical boot animation and logger. sounds like it should be disabled.
I boot into the rescue image and chroot to the debian: (note, this process will be repeated)
mount /dev/xvda1 /mnt;chroot /mnt
next I remove plymouth
rm /etc/init/plymouth*
now it says:
Begin: Running /scripts/init-bottom ... Done. init: Handling startup event init: mountall goal changed from stop to start init: mountall state changed from waiting to starting init: hostname goal changed from stop to start init: hostname state changed from waiting to starting init: Handling starting event init: hwclock goal changed from stop to start init: hwclock state changed from waiting to starting init: ureadahead goal changed from stop to start init: ureadahead state changed from waiting to starting init: Handling starting event init: hostname state changed from starting to pre-start init: hostname state changed from pre-start to spawned init: hostname main process (177) init: hostname state changed from spawned to post-start init: hostname state changed from post-start to running init: Handling starting event init: hwclock state changed from starting to pre-start init: hwclock state changed from pre-start to spawned init: hwclock main process (178) init: hwclock state changed from spawned to post-start init: hwclock state changed from post-start to running init: Handling starting event init: ureadahead state changed from starting to pre-start init: ureadahead state changed from pre-start to spawned init: ureadahead main process (179) init: Handling started event init: Handling started event init: hostname main process (177) exited normally init: hostname goal changed from start to stop init: hostname state changed from running to stopping init: Handling stopping event init: hostname state changed from stopping to killed init: hostname state changed from killed to post-stop init: hostname state changed from post-stop to waiting init: Handling stopped event init: hwclock main process (178) exited normally init: hwclock goal changed from start to stop init: hwclock state changed from running to stopping init: Handling stopping event init: hwclock state changed from stopping to killed init: hwclock state changed from killed to post-stop init: hwclock state changed from post-stop to waiting init: Handling stopped event init: ureadahead main process (179) became new process (180) init: ureadahead state changed from spawned to post-start init: ureadahead state changed from post-start to running init: mountall state changed from starting to pre-start init: mountall state changed from pre-start to spawned init: mountall main process (181) init: Handling started event /etc/default/locale: 2: .LANG=en_GB.UTF-8: not found init: mountall main process (181) terminated with status 127 init: mountall goal changed from start to stop init: mountall state changed from spawned to stopping init: Handling stopping event init: mountall state changed from stopping to killed init: mountall state changed from killed to post-stop init: mountall post-stop process (182) init: mountall post-stop process (182) exited normally init: mountall state changed from post-stop to waiting init: Handling stopped event
and hangs.
As chris points out (I must be getting really tired) the problem is that mountall is dying on a locale issue:
/etc/default/locale: 2: .LANG=en_GB.UTF-8: not found init: mountall main process (181) terminated with status 127
and sure enough:
root@loadpi:/# locale -a locale: Cannot set LC_CTYPE to default locale: No such file or directory locale: Cannot set LC_MESSAGES to default locale: No such file or directory locale: Cannot set LC_COLLATE to default locale: No such file or directory C POSIX en_ZA.utf8
it looks like we need some language packs in that puppy.
I chroot in again, then
root@loadpi:/# apt-get install language-pack-en-base
and I reboot again. (note, you want to exit the chroot before rebooting.) and am rewarded with:
/etc/default/locale: 2: .LANG=en_GB.UTF-8: not found init: mountall main process (177) terminated with status 127
irritating 'cause I thought I just installed it. looking
root@loadpi:/# locale -a C en_AG en_AU.utf8 en_BW.utf8 en_CA.utf8 en_DK.utf8 en_GB.utf8 en_HK.utf8 en_IE.utf8 en_IN en_NG en_NZ.utf8 en_PH.utf8 en_SG.utf8 en_US.utf8 en_ZA.utf8 en_ZW.utf8 POSIX
could it be that it's just set incorrectly? let us re-set the locale
root@loadpi:/# cat /etc/default/locale # File generated by update-locale .LANG=en_GB.UTF-8
let us remove the preceding .
it boots. And this is how a single bad character in /etc/default/locale kept one of my users from booting.
Note, I don't normally go this deep in on a user's guest. four dollars a month of support, remember? but they didn't come back up after a reboot, and I think when I reboot guests, it's on me to go over and above what I'd normally do, simply because maybe the customer was planning on rebooting some time when they had more time or something?
the getty is still borked, but I'm tired and this is almost certainly not my fault. when you figure out the getty issue, please report back here; it'd be a good thing to know, and I'll clean up this page for public consumption
