Coverage changes when running coveralls vs coveralls.html about excoveralls

[smeevil]

I have the same with schema which toggles between covered and not covered during consecutive runs.

[marceldegraaf]

Same here; the output of mix coveralls.html is inconsistent, especially for schema blocks.

[mikenichols]

I'm seeing exactly the same issue. Over multiple runs of mix espec --cover or mix coveralls, I'm seeing different coverage amounts because it sometimes counts the schema line in the models and sometimes does not.

[joshsmith]

I'm having the same issue here and it sometimes leads to increasingly lower coverage scores without reason.

I'm not sure the underlying cause of this, but it seems to be rather high priority. Would appreciate someone digging around into internals to find a reason behind this.

[jrissler]

+1 would be great to resolve this

[parroty]

Thanks for the follow up. I haven't been able to track down this item. If someone have insight, it would be great.

[Kjir]

I have proposed a solution to this problem. I don't have enough knowledge about how the Erlang cover tool works to determine the real cause for this, but here is what I found out.

First of all, the coverage seems to be lower most of the times when running mix coveralls.html, while it is usually higher when running mix coveralls or mix test --cover. Multiple runs will still yield the other result in some cases, so it seems like a concurrency issue is at work here.

Exploring the data returned by :cover.analyse/3 I quickly saw why the result is different (Elplan.User is my Ecto model, on line 5 there is my schema declaration):

# The data structure is: {{module, line_number}, count}
lines = [{{Elplan.User, 0}, 0}, {{Elplan.User, 0}, 0}, {{Elplan.User, 0}, 0},
 {{Elplan.User, 0}, 0}, {{Elplan.User, 0}, 0}, {{Elplan.User, 0}, 0},
 {{Elplan.User, 0}, 2}, {{Elplan.User, 0}, 0}, {{Elplan.User, 0}, 0},
 {{Elplan.User, 0}, 0}, {{Elplan.User, 0}, 0}, {{Elplan.User, 0}, 1},
 {{Elplan.User, 0}, 0}, {{Elplan.User, 0}, 1}, {{Elplan.User, 0}, 1},
 {{Elplan.User, 0}, 1}, {{Elplan.User, 0}, 1}, {{Elplan.User, 0}, 0},
 {{Elplan.User, 0}, 1}, {{Elplan.User, 0}, 0}, {{Elplan.User, 0}, 0},
 {{Elplan.User, 0}, 1}, {{Elplan.User, 0}, 0}, {{Elplan.User, 5}, 0},
 {{Elplan.User, 5}, 1}, {{Elplan.User, 5}, 1}, {{Elplan.User, 5}, 0},
 {{Elplan.User, 5}, 1}, {{Elplan.User, 15}, 0}, {{Elplan.User, 18}, 1}]

As you can see, we have 5 results for line 5, 3 of them report a coverage count of 1 and 2 of them a count of 0. In this case, the last result for line 5 has a count of 1 so my code is reported as covered. When the coverage gets lower, I have the same amount of results, but the latest one has a count of 0.

My hypothesis is that there are multiple processes running and reporting the coverage at the module levels (the only duplicate lines that I see are at module level, not inside functions). The coverage reports are then sent back to the master process and depending on the order of arrival we would get different results.

Is this truly what is happening? Why some lines are reported more than once? Why in some cases the count is 1 and in others 0? Why this seems to only affect schema declarations?
These questions remain unanswered, but maybe someone else can shed some light on this based on my discoveries.

My fix simply takes the highest coverage count for every line, which might or might not be correct depending on the answers to the questions above.