Upstart troubleshooting

From PrgmrWiki
[    0.825574] EXT3-fs: mounted filesystem with ordered data mode.
Begin: Running /scripts/local-bottom ...
Done.
Done.
Begin: Running /scripts/init-bottom ...
Done.

so it appears to be booting and dying sometime after the initrd mounts /

I add --verbose to the command line in grub, and I see

Begin: Running /scripts/init-bottom ...
Done.
init: Handling startup event
init: mountall goal changed from stop to start
init: mountall state changed from waiting to starting
init: hostname goal changed from stop to start
init: hostname state changed from waiting to starting
init: Handling starting event
init: plymouth goal changed from stop to start

...

init: hwclock state changed from stopping to killed
init: hwclock state changed from killed to post-stop
init: hwclock state changed from post-stop to waiting
init: Handling stopped event
init: plymouth main process (183) executable changed
init: plymouth main process (183) became new process (186)
init: plymouth state changed from spawned to post-start
init: plymouth post-start process (187)
init: plymouth post-start process (187) exited normally
init: plymouth state changed from post-start to running
init: Handling started event

so I am thinking the problem is in plymouth; a graphical boot animation and logger. sounds like it should be disabled.

I boot into the rescue image and chroot to the debian: (note, this process will be repeated)

mount /dev/xvda1 /mnt;chroot /mnt

next I remove plymouth

rm /etc/init/plymouth*


now it says:

Begin: Running /scripts/init-bottom ...
Done.
init: Handling startup event
init: mountall goal changed from stop to start
init: mountall state changed from waiting to starting
init: hostname goal changed from stop to start
init: hostname state changed from waiting to starting
init: Handling starting event
init: hwclock goal changed from stop to start
init: hwclock state changed from waiting to starting
init: ureadahead goal changed from stop to start
init: ureadahead state changed from waiting to starting
init: Handling starting event
init: hostname state changed from starting to pre-start
init: hostname state changed from pre-start to spawned
init: hostname main process (177)
init: hostname state changed from spawned to post-start
init: hostname state changed from post-start to running
init: Handling starting event
init: hwclock state changed from starting to pre-start
init: hwclock state changed from pre-start to spawned
init: hwclock main process (178)
init: hwclock state changed from spawned to post-start
init: hwclock state changed from post-start to running
init: Handling starting event
init: ureadahead state changed from starting to pre-start
init: ureadahead state changed from pre-start to spawned
init: ureadahead main process (179)
init: Handling started event
init: Handling started event
init: hostname main process (177) exited normally
init: hostname goal changed from start to stop
init: hostname state changed from running to stopping
init: Handling stopping event
init: hostname state changed from stopping to killed
init: hostname state changed from killed to post-stop
init: hostname state changed from post-stop to waiting
init: Handling stopped event
init: hwclock main process (178) exited normally
init: hwclock goal changed from start to stop
init: hwclock state changed from running to stopping
init: Handling stopping event
init: hwclock state changed from stopping to killed
init: hwclock state changed from killed to post-stop
init: hwclock state changed from post-stop to waiting
init: Handling stopped event
init: ureadahead main process (179) became new process (180)
init: ureadahead state changed from spawned to post-start
init: ureadahead state changed from post-start to running
init: mountall state changed from starting to pre-start
init: mountall state changed from pre-start to spawned
init: mountall main process (181)
init: Handling started event
/etc/default/locale: 2: .LANG=en_GB.UTF-8: not found
init: mountall main process (181) terminated with status 127
init: mountall goal changed from start to stop
init: mountall state changed from spawned to stopping
init: Handling stopping event
init: mountall state changed from stopping to killed
init: mountall state changed from killed to post-stop
init: mountall post-stop process (182)
init: mountall post-stop process (182) exited normally
init: mountall state changed from post-stop to waiting
init: Handling stopped event

and hangs.

As chris points out (I must be getting really tired) the problem is that mountall is dying on a locale issue:

/etc/default/locale: 2: .LANG=en_GB.UTF-8: not found
init: mountall main process (181) terminated with status 127

and sure enough:


root@loadpi:/#  locale -a
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_COLLATE to default locale: No such file or directory
C
POSIX
en_ZA.utf8

it looks like we need some language packs in that puppy.

I chroot in again, then


root@loadpi:/# apt-get install language-pack-en-base

and I reboot again. (note, you want to exit the chroot before rebooting.) and am rewarded with:


/etc/default/locale: 2: .LANG=en_GB.UTF-8: not found
init: mountall main process (177) terminated with status 127

irritating 'cause I thought I just installed it. looking



root@loadpi:/# locale -a                                                
C
en_AG
en_AU.utf8
en_BW.utf8
en_CA.utf8
en_DK.utf8
en_GB.utf8
en_HK.utf8
en_IE.utf8
en_IN
en_NG
en_NZ.utf8
en_PH.utf8
en_SG.utf8
en_US.utf8
en_ZA.utf8
en_ZW.utf8
POSIX

could it be that it's just set incorrectly? let us re-set the locale

root@loadpi:/# cat /etc/default/locale 
#  File generated by update-locale
.LANG=en_GB.UTF-8


let us remove the preceding .


it boots. And this is how a single bad character in /etc/default/locale kept one of my users from booting.

Note, I don't normally go this deep in on a user's guest. four dollars a month of support, remember? but they didn't come back up after a reboot, and I think when I reboot guests, it's on me to go over and above what I'd normally do, simply because maybe the customer was planning on rebooting some time when they had more time or something?

the getty is still borked, but I'm tired and this is almost certainly not my fault. when you figure out the getty issue, please report back here; it'd be a good thing to know, and I'll clean up this page for public consumption