Proposal: make `ament test` return non-zero on test failure about ament_tools

[dirk-thomas]

For any kind of automation it is crucial to distinguish failures to run something vs. some tests returning non-zero return codes. In my opinion this it is a strict requirement for our tools to satisfy this requirement. With the proposal it is not possible anymore to identify if the invocation of test failed for any reason. Nor is there any other way to achieve that (e.g. by invoking more commands). That prevents those use cases to operate correctly (e.g. sending a notification email in case tests fail, but not when something in the infrastructure / invocation failed) which I consider an unacceptable "cost" which can't be outweigh by any benefit.

On the other hand with the current state your use case is easily possible by invoking two two commands.

There is an alternative which continues meeting the requirement stated above as well as your desire to get a non-zero return code for any kind of failure. The ament test command could offer an argument to choose the desired return code behavior.

[gerkey]

There is an alternative which continues meeting the requirement stated above as well as your desire to get a non-zero return code for any kind of failure. The ament test command could offer an argument to choose the desired return code behavior.

Ah, that sounds like the perfect compromise! How about this change instead: change the default behavior of ament test according to my earlier proposal, but simultaneously offer an option to produce the current behavior? Automated systems would supply the option to get the current behavior, but regular users who don't supply that option would get the new behavior.

[mikaelarguedas]

I am not sure if this is relevant here, but I do remember that we decided to ignore the return code in the first place because all our successful builds with test failures were marked as failed on the buildfarm (e.g. ). So https://github.com/ament/ament_tools/pull/110/files has been made so that we could make the difference between a failing job and a job with failing tests.

EDIT: duplicate of previous comment

[dirk-thomas]

👍 I don't mind what the default behavior is as long as the mandatory use cases are still possible.

[gerkey]

Fixed by #141.