Document the difference between --build-status failure vs error about support

[ivailop]

Thanks for pointing that out. Sure, we'll clarify that detail in the help.

Meanwhile, the --build-status values are chosen to follow widely accepted (by both test-frameworks and CI-builds) status terminology, where:

• success is mean to represent fully completed with no failures
• failure - fully completed but with expected failures (e.g. 1 or more tests failed)
• error - unexpected failure (e.g. abort, timeout...)

[caneridge]

1. How does testspace represent the --build-status in the dashboard? Specifically, how is data pushed to testspace marked as error represented differently from data pushed marked as failure?
2. The error status name is ambiguous. We have a) test runs that end abnormally with internal errors where some testing is completed but not all tests are run b) builds that timeout due to testing timeout c) builds that timeout due to overall build sequence timeout (but testing may have complete) d) builds that fail to build the product being tested thus no testing can be done e) builds that are user-aborted. I think that from a testspace perspective the only case I really care about is when the product being tested fails to build either do to a process error or user abort.
3. What does testspace do when --build-status success is specified but test data pushed to testspace shows one or more test failures?
4. What does testspace do when --build-status error is specified but test data pushed to testspace shows all tests passed?

[ivailop]

1. How does testspace represent the `--build-status` in the dashboard? Specifically, how is data pushed to testspace marked as `error` represented differently from data pushed marked as `failure`?

Currently there is no difference.

2. The `error` status name is ambiguous. We have a) test runs that end abnormally with internal errors where some testing is completed but not all tests are run b) builds that timeout due to testing timeout c) builds that timeout due to overall build sequence timeout (but testing may have complete) d) builds that fail to build the product being tested thus no testing can be done e) builds that are user-aborted. I think that from a testspace perspective the only case I really care about is when the product being tested fails to build either do to a process error or user abort.

You are right. It is ambiguous and ss stated above error and failure are not differently treated when it comes to processing and presentation.

3. What does testspace do when `--build-status success` is specified but test data pushed to testspace shows one or more test failures?

4. What does testspace do when `--build-status error` is specified but test data pushed to testspace shows all tests passed?

The success vs failure/error are considered when marking the whole Result as invalid (see https://help.testspace.com/docs/dashboard/space-metrics#health). More specifically, when non success is specified Testspace compares the tests count with the previously reported and if there is drop-in-count marks the Result invalid.