[Daisy] missing serial output on WaitForInstanceSignal about compute-image-tools

[gut]

maybe this bring some light: I was launching on instance at a time and also connecting to the serial to see what happens when that WaitForInstanceSignal fails:

Serial log snip:

Jul 20 13:56:27 inst-quick-test0-quick-test-gbqm8 startup-script: INFO Starting startup scripts.
Jul 20 13:56:27 inst-quick-test0-quick-test-gbqm8 google-clock-skew: INFO Clock drift token has changed: 0.
Jul 20 13:56:27 inst-quick-test0-quick-test-gbqm8 google-networking: INFO Starting Google Networking daemon.
Jul 20 13:56:27 inst-quick-test0-quick-test-gbqm8 startup-script: INFO Found startup-script in metadata.
Jul 20 13:56:27 inst-quick-test0-quick-test-gbqm8 network-setup: INFO Ethernet interfaces: ['eth0'].
Jul 20 13:56:27 inst-quick-test0[    4.824409] sysrq: SysRq : -quick-test-gbqmPower Off
8 root: BOOTED
[    4.825329] sd 0:0:1:0: [sda] Synchronizing SCSI cache
[    4.826513] ACPI: Preparing to enter system sleep state S5
[    4.827328] reboot: Power down
serialport: VM disconnected.
Connection to ssh-serialport.googleapis.com closed.

and the daisy job failed due to timeout, so it didn't match that BOOTED that is visible on the serial despite it looking that some buffer issue occurred ("Power Off\n" stepped in the middle of the line related to the "BOOTED" message).

[adjackura]

Relying on serial output can be really finicky, there's a number of things that can go wrong that are beyond our control, Linux is much worse here than Windows since multiple programs write to ttyS01, and the startup scripts don't even write directly, they go through syslog and we rely on that to log to the serial console. In the future we may have a better mechanism for getting data out of the instance, but for now some things I've found that help:

• Always run 'sync' at the end of your script, or after writing important content to the serial port
• Add sleeps to ensure the serial console can be read before shutdown (no serial access after instance shutdown, daisy only polls every second or even longer depending on API latency, this is why you may not see the whole log)
• Try using another serial port to avoid system contention, this obviously doesnt work with just an echo in the startup script, your have to redirect to ttyS02 or something
• If all else fails try double writing to ensure you messages don't get garbled

[gut]

very useful hints! Thanks. I'll try my best to improve them and hopefully get better results on useful workloads.