vm_minimal_sim run failure is not reported about ci-actions

[lsf37]

It looks like the simulate script should return a non-zero error code but might not do that. Would be good to confirm. If it does correctly return an error then something is wrong in the infrastructure setup, but it does report other errors, so it is likely something subtle in that case.

[wom-bat]

The simulate scriupt is doing what it's meant to; but the test needs to look at the output and check it's getting to guetst userspace

[lsf37]

Result of further investigation: the expect command we've been using for the simulate script will always return exit code 0, so will never fail. This goes back to bamboo times already, so who knows how many failures have been masked by it in the past (although hw runs will show them, so probably fine after all).

The main thing that is needed is to add a timeout pattern to expect, eg. something like

expect {
  timeout {exit 1}
  "{success}" {exit 0}
}'

[lsf37]

The reason the simulation scripts have worked in sel4test is that we also parse the output of the test itself into junit format and detect errors that way. For simulation runs that only wait for the success string, that step is missing.

[lsf37]

Reopening, because the issue still persists after #340. In fact, now sel4test-sim takes much longer, because it always is hitting the timeout, even in the success case, and camkes-vm-examples is still failing + hitting the timeout.

Need to investigate more.

[wom-bat]

I think the problem now is the lack of a success criterion in the build.yaml

[lsf37]

I think I have it now. I got the syntax slightly wrong, it should be:

expect -c "spawn ./simulate; set timeout XX; expect \"good output\" {exit 0} \"bad output\" {exit 1} timeout {exit 1}"

and I had:

expect -c "spawn ./simulate; set timeout XX; expect { \"good output\" {exit 0} \"bad output\" {exit 1} timeout {exit 1} }"

That additional pair of braces made everything an action, and the expect part was basically empty. Will put up a PR.

[lsf37]

This now appears to be working properly and also fails fast when the known error string is detected.