Skipping tests in test spec about cth_readable

[ferd]

Yikes, that's a good catch. I'm guessing that yeah, the best we can do there is what you mentioned and unset the suite in post_end_per_suite to avoid issues of misreporting. Better to lack info than flat out output wrong one.

The other thing I'd really like to be able to do is get group information, but the group mechanisms in CT are nestable (stackable?) to large extents where I just decided to omit the information for now.

[sirihansen]

A fix in common_test is planned for the next sprint and will be included in 19.3. Haven't started the job for real, but the alternative ways that we have discussed so far are

1. make sure pre_init_per_suite is always called. This might be a rather big change to core functionality in test_server, so not really risk free... There is also the question of whether it is really correct to call this hook when Suite:init_per_suite will not be called (which it won't, since the suite is skipped completely)

2. extend on_tc_skip to include the suite name as a separate argument, and fall back to the old function if the new is not exported.

3. add a new hook - on_suite_skip - and keep on_tc_skip unchanged. Might have to call both on_suite_skip and on_tc_skip (with TC=all) when suite is skipped, for backwards compatibility.

For now I'm leaning towards alternative 2, but might of course change my mind as we start the job for real. Any thoughts are welcome!

And yes, the group mechanism is a bit too flexible to be easily visualized... :(

[sirihansen]

FYI: The solution we chose was alternative 2 above - i.e. a Suite argument is added to on_tc_skip (and a lot of other hook callbacks). Fallback to old API if the new function is not exported. This is included in OTP-19.3, common_test-1.14.

[ferd]

Cool. I'll reopen this to track it and add support for 19.3