`mix coveralls.post` should exit with non-zero status on test failure about excoveralls

[joshsmith]

I'm inclined to agree with you on this. Curious what others think.

[zachdaniel]

Currently in our CI we run tests once, and then run them again to send coverage, which is a bit of a waste of time.

[zachdaniel]

Any movement on this? Our test time is doubled because of this, and we have quite a bit of tests.

[joshsmith]

@parroty I don't have time to look into this right now, unfortunately. Do you or does someone else here have a chance?

[asummers]

Just curious if there's been any movement on this. Looking through the code, it seems the mix task would have to capture the error status from https://github.com/parroty/excoveralls/blob/master/lib/mix/tasks.ex#L53 and then after the if block below, raise if failure?

[asummers]

After thinking about it for a little bit, there should be an on/off flag to disable this behavior. Something like: mix coveralls --exit-status-ignore-result would always return positive behavior. However since that's existing behavior, it could be flipped to mix coveralls --status-as-exit-code or something, in an effort to not break existing behavior in people's builds.

[parroty]

Thanks for the follow-up. I'm starting to check around this item, and having a on/off flag sounds nice (mix flag mentioned can be one option, but maybe it's good to have in coveralls.json setting file, assuming this setting is more static preference?).

However, what I'm looking/wondering is how to identify the test failure (exit code) before posting json file.

• mix test task exits with 1 return code when there's test failure and 0 when it's success. I think this behavior corresponds to the following area of elixir implementation.
  • https://github.com/elixir-lang/elixir/blob/v1.4.0/lib/mix/lib/mix/tasks/test.ex#L251.
• It seems the logic is when failures > 0 is satisfied, the Kernel.exit shutdown processes with exit code 1. But, the callback function for coverage tools (following line) does not get test information and cannot directly check similar condition.
  • https://github.com/elixir-lang/elixir/blob/v1.4.0/lib/mix/lib/mix/tasks/test.ex#L245.
• The callback function in excoveralls side is the following.
  • https://github.com/parroty/excoveralls/blob/v0.6.1/lib/excoveralls.ex#L31

I'll check some more, but it's great if anyone have any ideas/insights on this part.

• System.at_exit callback seems to have access to the exit code, but I assume it's too late timing to take json-post action.

[zachdaniel]

I'm not sure if it really is important not to push coverage if the tests fail. Would it be an easier task to add an option to just to carry the test exit code through? For us, that would be 99℅ of what we need.

[parroty]

I think the mix coveralls tasks carry the test exit code (if tests fail, it exits with code 1).
I think I'm misunderstanding some of your intention. Can we get some more description about how the tests are run? (like how CI tool is configured).

$ mix coveralls.post
....

  1) test fails (ExCoveralls.CircleTest)
     test/circle_test.exs:33
     Assertion with == failed
     code:  1 == 2
     left:  1
     right: 2
     stacktrace:
       test/circle_test.exs:34: (test)

..

Finished in 3.8 seconds
108 tests, 1 failure

...

Successfully uploaded the report to 'https://coveralls.io'.
$ echo $?
1

$ mix coveralls.post
......

Finished in 3.8 seconds
107 tests, 0 failures

...

Successfully uploaded the report to 'https://coveralls.io'.

$ echo $?
0

[zachdaniel]

Very interesting. If this is, in fact, a mistake on our end, then our builds have been twice as long for no reason lol. Thats the ideal. We'll investigate and get back to you. Thanks so much for your help!

[zachdaniel]

Looks like the issue was on our end. We seem to have been ignoring the exit status somehow. Sorry for wasting your time, but thank for all of the assistance.

[parroty]

Thanks for the update.