Comments (25)
After a short chat with @josevalim I realized that something can be done on the coverage tool itself, without the need to touch elixir itself.
So I've setup a quick patch:
xadhoom@b7462df
which adds a coveralls.overall
(maybe a better name can be choosen), that along with a couple of addition to the umbrella mix.exs, like:
defmodule MyUmbrella.Mixfile do
use Mix.Project
def project do
[
app: :my_umbrella, # app name is required to run test from the top level umbrella folder
apps_path: "apps",
start_permanent: Mix.env == :prod,
deps: deps(),
test_paths: test_paths(), # gets all apps test folders
test_coverage: [tool: ExCoveralls],
preferred_cli_env: [
"coveralls": :test, "coveralls.detail": :test,
"coveralls.post": :test, "coveralls.html": :test,
"coveralls.overall": :test # run the task in :test env by default
]
]
end
defp test_paths do
"apps/*/test" |> Path.wildcard |> Enum.sort
end
....
Basically the patch added to elixir by @parroty has been moved into the lib itself.
Seems to work to me, and it does not impact current mix tasks (for example mix coveralls --umbrella still reports not merged in results).
what do you think?
there's room for improvement, like checking if the task is run in the umbrella or not and maybe make the --umbrella
command do the same thing.
from excoveralls.
Is there a way to post a sample for this case? (I'm not so familiar with the umbrella project).
There's a little related description in the issue #14.
from excoveralls.
Here is the sample:
https://github.com/ashneyderman/excoverall_issue23
README.md explains what I think is the problem. But the gist of it is if you have an umbrella project the coverage might be spread out and later child (dependent) app might be covering something in the parent app. In the case of the sample project subapp1 depends on subapp0 and covers some of the functionality in subapp0 but this coverage never gets accounted for.
Cover tool has export/import functionality so in theory it is possible to get an overall coverage of an umbrella project.
from excoveralls.
Thank you for the sample, it helps.
I understood that it's about the function Subapp0.not_covered
under subapp0
is covered by the tests Subapp1Test
under subapp1
, but reported as not covered
(though it's covered as entire umbrella project).
The current excoveralls
task triggered by ExUnit, which will be invoked for individual subapp, and doesn't have coordination among them. Maybe it needs additional layer for accumulating the results from cover
for umbrella project.
If the cover
result is stored for each mix tasks without throwing out in each ExUnit invocation, it might be possible, but, need to investigate more.
from excoveralls.
I may have a similar issue. Whenever I run mix test --cover
I get undefined function jsx_config.extract_config/1 (module :jsx_config is not available)
Edit: I think I fixed it, no idea how.
from excoveralls.
@hunterboerner thanks for the report. If you could reproduce the error next time, mix.exs definition (or entire project if possible) would be helpful for identifying the issue.
from excoveralls.
I've tried to check on reporting the entire umbrella project, but haven't been able to find a good way yet.
I'm a little stuck at this moment (not so many documents around umbrella project to pursue further).
What I found so far is (as a remembering note),
- For umbrella project,
Mix.Task
runs mix task multiple-times for individual sub-projects. It seems it's not only aboutMix.Tasks.Test
, but also generally applies to other mix tasks.- It seems the tasks are recursively called at around https://github.com/elixir-lang/elixir/blob/master/lib/mix/lib/mix/task.ex#L246.
- The callback for coverage tool option (
:cover
) gets thecompile-path
argument to indicate the location of beam files. With umbrella project, the callback is invoked separately for individual sub-project with individual compile-path.- They're isolated, and
:cover
tool may not report the cross-sub-project coverage data (without some kind of tricks). - The callback is defined around https://github.com/elixir-lang/elixir/blob/master/lib/mix/lib/mix/tasks/test.ex#L159.
- They're isolated, and
The each sub-projects are isolated at lower-level and haven't been able to find a good hook-point of coordination..
from excoveralls.
The v0.4.0 added mix coveralls --umbrella
option which reports overall coverage for umbrella projects. It may partially address this issue item.
In the case of the sample project subapp1 depends on subapp0 and covers some of the functionality in subapp0 but this coverage never gets accounted for.
The :cover
tool works on individual subapp separately, and I haven't been able to find a reasonable way to this part. So, the above option merges the "result" of each subapp.
from excoveralls.
We're running into this issue, too. I think the solution is to run all the tests for all applications as a single unit instead of running each sub-apps tests, one by one. We've achieved this by defining a test_all
task:
defmodule Delorean.Mixfile do
use Mix.Project
def project do
[
# ...
test_paths: test_paths,
# ...
]
end
defp test_paths do
"apps/*/test" |> Path.wildcard |> Enum.sort
end
end
# `mix test` cd's into each app directory in sequence and runs the tests.
# `mix test_all` runs tests for all apps at the same time.
defmodule Mix.Tasks.TestAll do
use Mix.Task
@shortdoc "Runs all delorean tests at once"
defdelegate run(args), to: Mix.Tasks.Test
end
(Unfortunately, setting test_paths
does not effect mix test
because it is declared as a recursive task).
We've been using mix test_all
in our CI builds (and for local test runs) for a while since it's really nice to run all tests all at once. I noticed that excoveralls supports a test_coverage: [test_task: "test_all"]
option, but when I try that, I get an error:
** (RuntimeError) Trying to access Mix.Project.app_path for an umbrella project but umbrellas have no app
(mix) lib/mix/project.ex:316: Mix.Project.app_path/1
(mix) lib/mix/project.ex:337: Mix.Project.compile_path/1
(mix) lib/mix/tasks/test.ex:163: Mix.Tasks.Test.run/1
lib/mix/tasks.ex:43: Mix.Tasks.Coveralls.do_run/2
(mix) lib/mix/cli.ex:58: Mix.CLI.run_task/2
Any ideas how to deal with this?
from excoveralls.
Thanks for the information, I could simulate the indicated behavior.
It's interesting, but the same error occurs with default mix test --cover
task too (without using excoveralls
). It's in elixir's Mix.Tasks.Test
module, and it may be difficult to fix only from excoveralls
side.
About the error
Error occurs in the following location,
https://github.com/elixir-lang/elixir/tree/v1.2/lib/mix/lib/mix/tasks/test.ex#L163
which will be executed when mix test
is called with cover
option. Elixir seems checking/guarding against it for umbrella's root project, through the following codes.
https://github.com/elixir-lang/elixir/tree/v1.2/lib/mix/lib/mix/project.ex#L336
https://github.com/elixir-lang/elixir/tree/v1.2//lib/mix/lib/mix/project.ex#L316
- For umbrella project,
Mix.Project.config[:app]
isnil
andMix.Project.config[:apps_path]
is"apps"
.
- For normal project,
Mix.Project.config[:app]
isproject-name
andMix.Project.config[:apps_path]
isnil
.
Possible solution
cover =
if opts[:cover] do
cover[:tool].start(Mix.Project.compile_path(project), cover)
end
If this part in Mix.Tasks.Test
module can be extended to handle multiple compile paths for sub-apps, this test_all
approach could be used for gathering overall coverage, but not sure if it matches with elixir's concept.
It might be one option to ask elixir team (opening an issue on elixir project, or something).
from excoveralls.
Thanks for digging into this, @parroty. I think asking the Elixir team about this is the way to go. I've been meaning to bring up the idea of having elixir natively support running all tests in an umbrella app at once like my test_all
task does with a simpler mechanism. I'll mention the issue we ran into here as well. I'll add a link to the discussion from here once I start it.
On a side note, your links to specific elixir line numbers use the master
branch, which means they are likely to get out-of-date once changes are pushed to those files. In 6 months, if someone reads your comment and tries to follow those links they are not likely to make sense anymore. For this reason, it's better to use a permalink. You can get one by hitting "y" after selecting the line or lines -- github will update your browser URL to include the latest commit SHA so that it's a permalink. Or you can change the URL from master
to a specific version tag, which has the nice side benefit of documenting what version is being referenced.
from excoveralls.
Discussion I started with the elixir team is here:
https://groups.google.com/forum/#!topic/elixir-lang-core/8wX8i5sEtFg
from excoveralls.
On a side note, your links to specific elixir line numbers use the master branch
Thanks for the comment. I've changed the link the ones in v1.2.0 branch.
I'll be checking the forum discussion too.
from excoveralls.
I've changed the link the ones in v1.2.0 branch.
👍
I'll be checking the forum discussion too.
José has some ideas in the thread if you want to take a look.
from excoveralls.
@parroty -- is there anything actionable we can do about this based on what was discussed on that thread?
from excoveralls.
For test_all
task approach, one of the options mentioned José's comments would be required for before taking action from library side.
For merging the result approach, there might be a way to make special treatment for umbrella project, but I haven't been able to come up with a good one yet...
I would agree bullet point 3 can be a pain point but it can be solved at the coverage tool too. Maybe we can merge the files after they are run?
from excoveralls.
I think I'd like to get the test_all
approach to work if possible. I don't yet understand what would have to change, though :(.
from excoveralls.
I was making some (very rough) experimentation on the following branch.
- https://github.com/parroty/elixir/tree/experiment
- https://github.com/parroty/excoveralls/tree/experiment
- https://github.com/parroty/excoveralls_umbrella/tree/experiment
It sounds like technically working and there might some way to pursue, but I'm not confident on addressing jose's concern mentioned in the mailing list. I'm running out of time, and may check a little more later.
elixir/projects/excoveralls_umbrella[experiment]% mix coveralls
==> subapp0
...
Finished in 0.04 seconds (0.04s on load, 0.00s on tests)
3 tests, 0 failures
Randomized with seed 208001
----------------
COV FILE LINES RELEVANT MISSED
100.0% lib/subapp0.ex 9 2 0
[TOTAL] 100.0%
----------------
==> subapp1
.
Finished in 0.00 seconds
1 test, 0 failures
Randomized with seed 291710
----------------
COV FILE LINES RELEVANT MISSED
50.0% lib/subapp1.ex 9 2 1
[TOTAL] 50.0%
----------------
elixir/projects/excoveralls_umbrella[experiment]% MIX_ENV=test mix test_all --cover
....
Finished in 0.05 seconds (0.05s on load, 0.00s on tests)
4 tests, 0 failures
Randomized with seed 878778
----------------
COV FILE LINES RELEVANT MISSED
100.0% apps/subapp0/lib/subapp0.ex 9 2 0
100.0% apps/subapp1/lib/subapp1.ex 9 2 0
[TOTAL] 100.0%
----------------
from excoveralls.
I've reported this discussion upstream in order to check if something can be done, because having an overall reporting is really useful when doing integration tests between umbrella apps.
from excoveralls.
I played also with --umbrella options, and this is the result:
This makes possible to use coveralls.{json, html, detail, whatever} along with --umbrella and have proper results.
But still the above setup into umbrella mix.exs must be done, and is lost the message of how many tests are run per subapp when using the --umbrella flag (only total number of tests is shown)
from excoveralls.
I need this capability as well, pretty badly, guess I'll monitor this issue...
from excoveralls.
@xadhoom Any plan on making a PR to integrate this improvement? Also could we make it work so it would be a flag instead of a task so we can use the integration aggregation for different type of tasks, for example:
mix coveralls.html --umbrella #gives normal unit/module test aggregation
mix coveralls.html --umbrella-integration #gives integration test aggregation
from excoveralls.
@adrienmo since there was no real interest from upstream, I never update the patch to latest release, and a lot of stuff is changed, so basically means redo it from scratch.
Said that, I stopped to use this feature from my branch, because after gaining better experience in testing I think that maybe is not really needed, since proper unit testing should already cover almost everything without the need to do cross application coverage.
And cross application testing is done with integration tests, where you're most interested into correct functionality and not in coverage.
jm2c
from excoveralls.
@xadhoom I am planning to expose two metrics:
- unit test coverage (classic mode)
- integration test coverage
100% coverage in unit test does not mean that the system works, it just means the code is well tested independently.
100% coverage in integration test is usually a good indicator to know modules works well together and that no junk code is staying in the code base (old functionality not needed anymore staying in the codebase)
I think having both number is helpful to maintain a healthy codebase on big projects. For junk code I could consider scanning the project for unused public functions (https://github.com/hauleth/mix_unused), but that's not as good as coverage, I also cannot know my current integration test progress (e.g.: I know I tested my N features, but I don't know if I tested all the code path/errors/exception)
I think having 2 coverage results and keeping track of both is standard in Java. Both results can also be interesected to give a total, for exemple a 80% coverage for unit test and 80% integration test can give a total of 65% because only 65% of the code is tested both by unit and integration test. (SonarQube, a multi-language code quality gate, is accepting both and doing this combination: https://docs.sonarqube.org/display/PLUG/Java+Unit+Tests+and+Coverage+Results+Import)
By the way, I tried to apply your patch to the current version and it seemed to work for me, so I don't think a lot of work would be needed :)
from excoveralls.
By the way, I tried to apply your patch to the current version and it seemed to work for me, so I don't think a lot of work would be needed :)
For the record: the above still holds true, in case anyone wonders (I used the latter one, invoked with mix coveralls.x --umbrella
). Thank you for sharing!
I experimented with the approach and by using a rebased version of the patch c.f. here I managed to calculate "integration coverage" in our case.
Not sure yet if we're going to go down that path in our case, but it surely has some appeal.
from excoveralls.
Related Issues (20)
- How to find out by which test selected line is covered by?
- Coveralls is not picking all my files HOT 1
- Coveralls errors out HOT 3
- Github actions fail on scheduled workflows
- Cannot specify subdir with circle
- Is it possible to include HEEX files in the coverage?
- Support --warnings-as-errors option HOT 3
- Reports different results at an umbrella level vs in-app level
- Is there a way to post coverage from Github Actions in parallel mode? HOT 3
- don't ignore `min_coverage` in `mix coveralls.json` HOT 4
- Coveralls reporting does not understand slots
- Story: Please add totals at the bottom of the report for LINES, RELEVANT and MISSED
- Cobertura error when protocols and implementations in codebase HOT 1
- Incompatibility between excoveralls and meck? HOT 4
- Umbrella run ignores config json in root HOT 5
- Protocols fail to consolidate using ExCoveralls HOT 1
- Excoveralls minimum_coverage conflicts with new Elixir changes. Result: Error: Process completed with exit code 3.
- "function ExCoveralls.start/2 is undefined" HOT 3
- Warning in build, CAStore
- Is it possible to export in multiple formats by running the tests once?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from excoveralls.