Umbrella projects do not report numbers overall about excoveralls

[xadhoom]

After a short chat with @josevalim I realized that something can be done on the coverage tool itself, without the need to touch elixir itself.

So I've setup a quick patch:
xadhoom@b7462df

which adds a coveralls.overall (maybe a better name can be choosen), that along with a couple of addition to the umbrella mix.exs, like:

defmodule MyUmbrella.Mixfile do
  use Mix.Project

  def project do
    [
      app: :my_umbrella, # app name is required to run test from the top level umbrella folder
      apps_path: "apps",
      start_permanent: Mix.env == :prod,
      deps: deps(),
      test_paths: test_paths(), # gets all apps test folders
      test_coverage: [tool: ExCoveralls],
      preferred_cli_env: [
        "coveralls": :test, "coveralls.detail": :test,
        "coveralls.post": :test, "coveralls.html": :test,
        "coveralls.overall": :test # run the task in :test env by default
       ]
    ]
  end

  defp test_paths do
    "apps/*/test" |> Path.wildcard |> Enum.sort
  end
  ....

Basically the patch added to elixir by @parroty has been moved into the lib itself.

Seems to work to me, and it does not impact current mix tasks (for example mix coveralls --umbrella still reports not merged in results).

what do you think?
there's room for improvement, like checking if the task is run in the umbrella or not and maybe make the --umbrella command do the same thing.

[parroty]

Is there a way to post a sample for this case? (I'm not so familiar with the umbrella project).
There's a little related description in the issue #14.

[ashneyderman]

Here is the sample:
https://github.com/ashneyderman/excoverall_issue23

README.md explains what I think is the problem. But the gist of it is if you have an umbrella project the coverage might be spread out and later child (dependent) app might be covering something in the parent app. In the case of the sample project subapp1 depends on subapp0 and covers some of the functionality in subapp0 but this coverage never gets accounted for.

Cover tool has export/import functionality so in theory it is possible to get an overall coverage of an umbrella project.

[parroty]

Thank you for the sample, it helps.

I understood that it's about the function Subapp0.not_covered under subapp0 is covered by the tests Subapp1Test under subapp1, but reported as not covered (though it's covered as entire umbrella project).

The current excoveralls task triggered by ExUnit, which will be invoked for individual subapp, and doesn't have coordination among them. Maybe it needs additional layer for accumulating the results from cover for umbrella project.

If the cover result is stored for each mix tasks without throwing out in each ExUnit invocation, it might be possible, but, need to investigate more.

[hunterboerner]

I may have a similar issue. Whenever I run mix test --cover I get undefined function jsx_config.extract_config/1 (module :jsx_config is not available)

Edit: I think I fixed it, no idea how.

[parroty]

@hunterboerner thanks for the report. If you could reproduce the error next time, mix.exs definition (or entire project if possible) would be helpful for identifying the issue.

[parroty]

I've tried to check on reporting the entire umbrella project, but haven't been able to find a good way yet.
I'm a little stuck at this moment (not so many documents around umbrella project to pursue further).

What I found so far is (as a remembering note),

• For umbrella project, Mix.Task runs mix task multiple-times for individual sub-projects. It seems it's not only about Mix.Tasks.Test, but also generally applies to other mix tasks.
  • It seems the tasks are recursively called at around https://github.com/elixir-lang/elixir/blob/master/lib/mix/lib/mix/task.ex#L246.
• The callback for coverage tool option (:cover) gets the compile-path argument to indicate the location of beam files. With umbrella project, the callback is invoked separately for individual sub-project with individual compile-path.
  • They're isolated, and :cover tool may not report the cross-sub-project coverage data (without some kind of tricks).
  • The callback is defined around https://github.com/elixir-lang/elixir/blob/master/lib/mix/lib/mix/tasks/test.ex#L159.

The each sub-projects are isolated at lower-level and haven't been able to find a good hook-point of coordination..

[parroty]

The v0.4.0 added mix coveralls --umbrella option which reports overall coverage for umbrella projects. It may partially address this issue item.

In the case of the sample project subapp1 depends on subapp0 and covers some of the functionality in subapp0 but this coverage never gets accounted for.

The :cover tool works on individual subapp separately, and I haven't been able to find a reasonable way to this part. So, the above option merges the "result" of each subapp.

[myronmarston]

We're running into this issue, too. I think the solution is to run all the tests for all applications as a single unit instead of running each sub-apps tests, one by one. We've achieved this by defining a test_all task:

defmodule Delorean.Mixfile do
  use Mix.Project

  def project do
    [
      # ...
      test_paths: test_paths,
      # ...
    ]
  end

  defp test_paths do
    "apps/*/test" |> Path.wildcard |> Enum.sort
  end
end

# `mix test` cd's into each app directory in sequence and runs the tests.
# `mix test_all` runs tests for all apps at the same time.
defmodule Mix.Tasks.TestAll do
  use Mix.Task
  @shortdoc "Runs all delorean tests at once"
  defdelegate run(args), to: Mix.Tasks.Test
end

(Unfortunately, setting test_paths does not effect mix test because it is declared as a recursive task).

We've been using mix test_all in our CI builds (and for local test runs) for a while since it's really nice to run all tests all at once. I noticed that excoveralls supports a test_coverage: [test_task: "test_all"] option, but when I try that, I get an error:

** (RuntimeError) Trying to access Mix.Project.app_path for an umbrella project but umbrellas have no app
    (mix) lib/mix/project.ex:316: Mix.Project.app_path/1
    (mix) lib/mix/project.ex:337: Mix.Project.compile_path/1
    (mix) lib/mix/tasks/test.ex:163: Mix.Tasks.Test.run/1
    lib/mix/tasks.ex:43: Mix.Tasks.Coveralls.do_run/2
    (mix) lib/mix/cli.ex:58: Mix.CLI.run_task/2

Any ideas how to deal with this?

[parroty]

Thanks for the information, I could simulate the indicated behavior.

It's interesting, but the same error occurs with default mix test --cover task too (without using excoveralls). It's in elixir's Mix.Tasks.Test module, and it may be difficult to fix only from excoveralls side.

About the error
Error occurs in the following location,
https://github.com/elixir-lang/elixir/tree/v1.2/lib/mix/lib/mix/tasks/test.ex#L163
which will be executed when mix test is called with cover option. Elixir seems checking/guarding against it for umbrella's root project, through the following codes.
https://github.com/elixir-lang/elixir/tree/v1.2/lib/mix/lib/mix/project.ex#L336
https://github.com/elixir-lang/elixir/tree/v1.2//lib/mix/lib/mix/project.ex#L316

• For umbrella project,
  • Mix.Project.config[:app] is nil and Mix.Project.config[:apps_path] is "apps".
• For normal project,
  • Mix.Project.config[:app] is project-name and Mix.Project.config[:apps_path] is nil.

Possible solution

    cover =
      if opts[:cover] do
        cover[:tool].start(Mix.Project.compile_path(project), cover)
      end

If this part in Mix.Tasks.Test module can be extended to handle multiple compile paths for sub-apps, this test_all approach could be used for gathering overall coverage, but not sure if it matches with elixir's concept.
It might be one option to ask elixir team (opening an issue on elixir project, or something).

[myronmarston]

Thanks for digging into this, @parroty. I think asking the Elixir team about this is the way to go. I've been meaning to bring up the idea of having elixir natively support running all tests in an umbrella app at once like my test_all task does with a simpler mechanism. I'll mention the issue we ran into here as well. I'll add a link to the discussion from here once I start it.

On a side note, your links to specific elixir line numbers use the master branch, which means they are likely to get out-of-date once changes are pushed to those files. In 6 months, if someone reads your comment and tries to follow those links they are not likely to make sense anymore. For this reason, it's better to use a permalink. You can get one by hitting "y" after selecting the line or lines -- github will update your browser URL to include the latest commit SHA so that it's a permalink. Or you can change the URL from master to a specific version tag, which has the nice side benefit of documenting what version is being referenced.

[myronmarston]

Discussion I started with the elixir team is here:

https://groups.google.com/forum/#!topic/elixir-lang-core/8wX8i5sEtFg

[parroty]

On a side note, your links to specific elixir line numbers use the master branch

Thanks for the comment. I've changed the link the ones in v1.2.0 branch.
I'll be checking the forum discussion too.

[myronmarston]

I've changed the link the ones in v1.2.0 branch.

👍

I'll be checking the forum discussion too.

José has some ideas in the thread if you want to take a look.

[myronmarston]

@parroty -- is there anything actionable we can do about this based on what was discussed on that thread?

[parroty]

For test_all task approach, one of the options mentioned José's comments would be required for before taking action from library side.

For merging the result approach, there might be a way to make special treatment for umbrella project, but I haven't been able to come up with a good one yet...

I would agree bullet point 3 can be a pain point but it can be solved at the coverage tool too. Maybe we can merge the files after they are run?

[myronmarston]

I think I'd like to get the test_all approach to work if possible. I don't yet understand what would have to change, though :(.

[parroty]

I was making some (very rough) experimentation on the following branch.

• https://github.com/parroty/elixir/tree/experiment
• https://github.com/parroty/excoveralls/tree/experiment
• https://github.com/parroty/excoveralls_umbrella/tree/experiment

It sounds like technically working and there might some way to pursue, but I'm not confident on addressing jose's concern mentioned in the mailing list. I'm running out of time, and may check a little more later.

elixir/projects/excoveralls_umbrella[experiment]% mix coveralls
==> subapp0
...

Finished in 0.04 seconds (0.04s on load, 0.00s on tests)
3 tests, 0 failures

Randomized with seed 208001
----------------
COV    FILE                                        LINES RELEVANT   MISSED
100.0% lib/subapp0.ex                                  9        2        0
[TOTAL] 100.0%
----------------
==> subapp1
.

Finished in 0.00 seconds
1 test, 0 failures

Randomized with seed 291710
----------------
COV    FILE                                        LINES RELEVANT   MISSED
 50.0% lib/subapp1.ex                                  9        2        1
[TOTAL]  50.0%
----------------

elixir/projects/excoveralls_umbrella[experiment]% MIX_ENV=test mix test_all --cover
....

Finished in 0.05 seconds (0.05s on load, 0.00s on tests)
4 tests, 0 failures

Randomized with seed 878778
----------------
COV    FILE                                        LINES RELEVANT   MISSED
100.0% apps/subapp0/lib/subapp0.ex                     9        2        0
100.0% apps/subapp1/lib/subapp1.ex                     9        2        0
[TOTAL] 100.0%
----------------

[xadhoom]

I've reported this discussion upstream in order to check if something can be done, because having an overall reporting is really useful when doing integration tests between umbrella apps.

[xadhoom]

I played also with --umbrella options, and this is the result:

xadhoom@52c6c8e

This makes possible to use coveralls.{json, html, detail, whatever} along with --umbrella and have proper results.

But still the above setup into umbrella mix.exs must be done, and is lost the message of how many tests are run per subapp when using the --umbrella flag (only total number of tests is shown)

[grievingbullfrog]

I need this capability as well, pretty badly, guess I'll monitor this issue...

[adrienmo]

@xadhoom Any plan on making a PR to integrate this improvement? Also could we make it work so it would be a flag instead of a task so we can use the integration aggregation for different type of tasks, for example:

mix coveralls.html --umbrella  #gives normal unit/module test aggregation
mix coveralls.html --umbrella-integration #gives  integration test aggregation

[xadhoom]

@adrienmo since there was no real interest from upstream, I never update the patch to latest release, and a lot of stuff is changed, so basically means redo it from scratch.

Said that, I stopped to use this feature from my branch, because after gaining better experience in testing I think that maybe is not really needed, since proper unit testing should already cover almost everything without the need to do cross application coverage.

And cross application testing is done with integration tests, where you're most interested into correct functionality and not in coverage.

jm2c

[adrienmo]

@xadhoom I am planning to expose two metrics:

• unit test coverage (classic mode)
• integration test coverage

100% coverage in unit test does not mean that the system works, it just means the code is well tested independently.
100% coverage in integration test is usually a good indicator to know modules works well together and that no junk code is staying in the code base (old functionality not needed anymore staying in the codebase)

I think having both number is helpful to maintain a healthy codebase on big projects. For junk code I could consider scanning the project for unused public functions (https://github.com/hauleth/mix_unused), but that's not as good as coverage, I also cannot know my current integration test progress (e.g.: I know I tested my N features, but I don't know if I tested all the code path/errors/exception)

I think having 2 coverage results and keeping track of both is standard in Java. Both results can also be interesected to give a total, for exemple a 80% coverage for unit test and 80% integration test can give a total of 65% because only 65% of the code is tested both by unit and integration test. (SonarQube, a multi-language code quality gate, is accepting both and doing this combination: https://docs.sonarqube.org/display/PLUG/Java+Unit+Tests+and+Coverage+Results+Import)

By the way, I tried to apply your patch to the current version and it seemed to work for me, so I don't think a lot of work would be needed :)

[pdobacz]

By the way, I tried to apply your patch to the current version and it seemed to work for me, so I don't think a lot of work would be needed :)

For the record: the above still holds true, in case anyone wonders (I used the latter one, invoked with mix coveralls.x --umbrella). Thank you for sharing!

I experimented with the approach and by using a rebased version of the patch c.f. here I managed to calculate "integration coverage" in our case.

Not sure yet if we're going to go down that path in our case, but it surely has some appeal.