Comments (41)
FWIW, I'm running spork
with bundle exec spork rspec
, rake
with bundle exec rake
, and rspec
with bundle exec rspec --drb spec
.
from simplecov.
I'm not using Spork myself, but from what I understand is that it always spins up a new process when the test suite should be run.
The Ruby Coverage library SimpleCov uses works by tracking every require
d file AFTER it has been loaded. Therefore, if SimpleCov is loaded AFTER the Rails app and all of it's source files, it won't be able to see them (I think this is what you're getting with the each_run
solution).
So I imagine the solution would be to make Spork require 'simplecov'; SimpleCov.start
before loading the app's environment every time, so the actual code files become visible to SimpleCov. But as I said, I'm not using Spork and I'm not sure how exactly it does it's magic. Maybe the authors of Spork could shed some light onto this?
from simplecov.
Quick update: I pinged @timcharper about this, hopefully he can enlighten us a bit :)
from simplecov.
I too am having this issue. Seems not many people run into this problem, as I can't find anything anywhere else.
from simplecov.
@fj, I bet your problem is your test environment caches classes. See @Estheth's comment in #30.
fwiw, I had been using the hack below in each_run
and setting cache_classes
to true but that doesn't work:
# HACK so we can cache-classes in test.rb
Dir["#{Rails.root}/app/**/*.rb"].each { |f| load f }
require 'simplecov'
SimpleCov.start 'rails'
from simplecov.
I have this issue, i put the two lines for simplecov at the top of the Spork.prefork
block. When I use guard no coverage report is generated but when I run rake spec
it works. Would be nice if it worked with guard though.
from simplecov.
+1. Same applies to configuration with cucumber. I bet it's spork related
from simplecov.
+1. Same config as @Godisemo along with the same issue.
from simplecov.
+1 Same here. I swear, it used to work....
from simplecov.
+1
simplecov (0.5.4)
guard-rspec (0.5.10)
guard-spork (0.4.1)
spork (0.9.0)
from simplecov.
Guys, this obviously is related to #60 - the offender being Spork's requirement to set config.cache_classes = false
in config/environments/test.rb
. Please try to set it to config.cache_classes = true
. As I do not use Spork I don't know what the implications are for spork when you do this, but without it SimpleCov will fail to operate correctly :(
from simplecov.
@colszowka I had config.cache_classes
set to true
and false
and it still wouldn't generate the coverage directory/report after guard finishes running tests. Additionally setting config.cache_classes = false
in config/environments/test.rb
causes ActiveRecord::AssociationTypeMismatch
exceptions when using FactoryGirl: http://stackoverflow.com/questions/5954690/strange-activerecordassociationtypemismatch
from simplecov.
Something must be wrong in your setups. I successfully use both config.cache_classes=false
and FactoryGirl to get code coverage results under Tork (not Spork).
from simplecov.
You guys should try using the Coverage library directly in your setups. If it works, then SimpleCov needs fixing. Else, you know what to do. :)
from simplecov.
I'm afraid I don't have any experience with SimpleCov to be able to shed much light on this. But, I can say this:
You shouldn't need config.cache_classes = false
, although that's the easiest way to get your models and such to reload automatically. Read these wiki pages for more info:
https://github.com/sporkrb/spork/wiki/Troubleshooting (specifically: Some changes to files don’t take effect until I restart Spork.)
https://github.com/sporkrb/spork/wiki/Spork.trap_method-Jujutsu
All you should need to do is delay the initialization of SimpleCov until after the process forks, and SimpleCov should be none the wiser.
If a SimpleCov devel wants to do some remove pair programming with me to crack this together, I'd be happy to schedule some time.
from simplecov.
On Thu 09 Feb 2012 10:44:56 AM PST, Tim Harper wrote:
You shouldn't need
config.cache_classes = false
, although that's
the easiest way to get your models and such to reload automatically.All you should need to do is delay the initialization of SimpleCov
until after the process forks, and SimpleCov should be none the wiser.
That is correct. You should call Coverage.start()
after forking. The
coverage results should be collected and processed by an at_exit hook
that was registered once before forking. Here is a working example:
https://github.com/sunaku/tork/blob/coverage/lib/tork/config/coverage.rb#L6-10
from simplecov.
FWIW, this is working for me now. I initially had some issues and after a bunch of mucking about it now works (even though my config is right where it was when I started).
The one minor change I made to my code was to skip my custom initializer which bootstrapped some of my custom libs. I now require the initializer manually in the each_run block. This is the complete contents of my test_helper.rb (I'm using minitest):
https://gist.github.com/1820849
I also disabled cache_classes as noted above, though I'm not sure that's absolutely necessary.
from simplecov.
So I'm not familiar with Tork, but @sunaku is the ordering of that Coverage.start()
hook before app files are (re)loaded dependent simply on internal load order within Tork? It seems like we'd need something akin to prepend_each_run
in Spork to get things like SimpleCov to run immediately after forking and before (for instance) spork-rails
' each_run
blocks kick off any reloading, etc., does that sound right @timcharper or am I oversimplifying?
@chetan It's not clear what you're doing in the files you require from your Gist -- if you're loading your app env in each_run
every time, you aren't getting much out of using Spork, no?
from simplecov.
@ches oops, I forgot to attach test_prefork. Basically I bootstrap the whole rails and test environment w/ the exception of all app/ lib/ and test/ classes. My test runtime has gone from about 10sec (mostly in Bundle.setup) to 1sec. Updated the gist:
https://gist.github.com/1820849
from simplecov.
I'm having the same issue, following the same logic grid as in the first report. I tried toggling class caching in the test environment to no avail. The coverage report just doesn't run unless I manually run rake spec
. Everything works fine without spork.
Versions:
simplecov 0.6.1
guard 1.0.0
spork 0.9.0
rails 3.2.0
spec_helper.rb https://gist.github.com/1934043
from simplecov.
rails (3.2.2)
rspec-rails (2.8.1)
simplecov (0.6.1)
spork (0.9.0)
For me, SimpleCov works when Spork is shut down, but not when Spork is running. I tried all the suggestions above (and from other Google searches on the same problem) to no avail; in particular, setting caching to true or false makes no difference. In the end, I learned two things:
- With Spork running, "rake simplecov" will cause the Spork.each_run section to be run twice. Whether that's the cause, I don't know, but in any case, all LOC in app/*/.rb show up in "not covered" red.
- However (also with Spork running), "rspec --no-drb spec" gives me a correct SimpleCov report, and Spork.each_run is only executed once.
Don't know if this will help anyone, but at least I have a way to get a coverage report without shutting Spork down every time.
from simplecov.
This works:
Spork.prefork do
unless ENV['DRB']
require 'simplecov'
SimpleCov.start 'rails'
end
# other code ...
end
Spork.each_run do
if ENV['DRB']
require 'simplecov'
SimpleCov.start 'rails'
end
# other code ...
end
from simplecov.
@fredwu nice, it's working for me now.
from simplecov.
@fredwu Works for me too!
ruby - 1.9.3
rails - 3.2.2
spork - 1.0.0rc
from simplecov.
@fredwu Excellent! That was the piece I needed to unlock the puzzle. There were two other things I had to do:
- I had to add the ":require => false" option to "gem 'simplecov'" in my Gemfile (this means "rake simplecov" no longer works but I don't care).
- As noted by @chetan above, I had "SimpleCov.coverage_dir(...)" in a config/initializers/*.rb file, and I had to move that into spec/spec_helper.rb next to "SimpleCov.start" -- I think that was causing SimpleCov to get loaded too soon.
Now I get identical results for "rake spec" and "rspec spec", with and without Spork running. Thank you! (I still have "config.cache_classes = false" by the way.)
from simplecov.
I am getting irregular coverage results with rspec tests while running against spork, despite applying @fredwu 's fix and playing with stuff like:
- set config.cache_classes = false
- add the ":require => false" option to "gem 'simplecov'"
For now I am simply going to leave it alone since the trouble proved to be too much for any potential gain (for those interested, my setup: https://github.com/prusswan/hw3_rottenpotatoes/tree/hw4)
from simplecov.
@prusswan Your repo's Gemfile doesn't have :require => false
. Also, I just checked my test.rb environment, it's config.cache_classes = true
.
from simplecov.
@fredwu as I said I have already tried changing those parameters but they did not improve matters, the result with rspec is pretty erratic.
from simplecov.
@prusswan and others, the solutions suggested here got me part of the way. I think the problem is when some of the code you want coverage reports for is run for the first time during Rails initialization (for example, if an initializer in config/initializers does some config on models or custom code in your lib directory or whatever). SimpleCov won't pick that stuff up because it's loaded before SimpleCov is. A solution is to split loading and initialization between prefork and each_run by requiring application.rb instead of environment.rb, and then explicitly initializing your app in each_run. This solution works with config.cache_classes = true and if threadsafe mode has not been turned on. A detailed blog post is here: http://rickyrobinson.id.au/2012/07/20/when-spork-puts-a-fork-in-your-cucumber-and-a-spanner-in-your-specs.
from simplecov.
I still can't tell much about spork, but on the Ruby Toolbox I've been using spin(+guard-spin) with great success recently. It basically only preloads your app's gems (so probably there's a little time delay compared to spork), but works without any config, and it works smoothly with simplecov so far (of course, only mentioning coverage for the latetst spec that's been run - you can avoid this by expanding merge_timeout
and setting a pseudo-random command_name
)
from simplecov.
@colszowka Hey, spin looks pretty cool. Thanks for the pointer. It runs my specs just fine after I take out all the spork-related stuff, but not sure if it'll run my cukes. I note that spin takes the same approach to splitting loading and initialization that my spork hack takes:
We require config/application because that file (typically) loads Rails and any Bundler deps, as well as loading the initialization code for the app, but it doesn't actually perform the initialization. That happens in config/environment.
In my experience that's the best we can do in terms of preloading. Rails and the gem dependencies rarely change and so don't need to be reloaded. But you can't initialize the application because any non-trivial app will involve it's models/controllers, etc. in its initialization, which you definitely don't want to preload.
http://jstorimer.github.com/spin/#section-9
Update: got spin working with cucumber and made a few other changes: http://rickyrobinson.id.au/2012/07/24/spork-to-spin
from simplecov.
Having the simplecov runs fine with spork. But the report generated is missing the helper files. But once I stopped spork and run all the specs, then the report includes helper files. Any idea? Anyone have the same issue or know how to fix it?
from simplecov.
@fredwu's workaround works for me only if I don't add an add_filter
in simplecov
.
SimpleCov.start "rails" do
add_filter "vendor"
end
When I do it I need to configure config.cache_classes = false
in test.rb
, otherwise I get /coverage. 0 / 0 LOC (0.0%) covered.
.
I don't feel comfortable disabling cache_classes
, is that the correct way to fix it?
from simplecov.
An additional observation: If you're using ActiveRecord observers, and activating them from your config/application.rb
, the models they observe will still be skipped, even with @fredwu's fix.
The observed models being skipped seems to only be an issue when I run my specs through Spork.
from simplecov.
Works for me:
I followed the "this works" suggestion by fredwu. It didn't work with Spork at first. Then I additionally followed bjnord's suggestion to add :required => false
to the simplecov
line in the Gemfile. That combination seems to have solved the problem for me. Now coverage metrics are being generated BOTH with and without Spork.
I should note though, that in my test.rb, the following is still unchanged, and that seems to not cause a difference. It still works without modification:
config.cache_classes = true
Further update:
Some of my models weren't showing up when running Rspec under Spork with Guard iteratively, but their dependencies were showing up. After trying a few things, it turns out both of the following are needed for simplecov
to deterministically figure out the dependencies on every partial run:
In test.rb
do:
config.cache_classes = false
In your spec, do:
require 'spec_helper'
require '<model>'
from simplecov.
I had the same issue with some application_helper lines that were not evaluated as covered code when running guard+spork. When running rspec without guard, coverage was 100% for that helper.
I use a gem that is located under vendor/gems at the moment which is not watched by guard, I excluded it with a filter in the .simplecov file.
I removed the vendor-filter and added :spec_paths => ["spec", "vendor/gems/.../spec"] to the Guardfile, so the gem-tests are now run together with my rails tests. With this configuration, the coverage is correct with AND without guard/spork.
from simplecov.
SimpleCov will only pickup if we pass the --no-drb
flag, as in:
bundle exec rspec spec/ --no-drb
Though I'm not sure why...
from simplecov.
Here is how I solved the problem
in environment/test.rb
config.cache_classes = false
in spec_hepler.rb
Spork.each_run do
require 'simplecov'
SimpleCov.start do
add_filter '/spec/'
add_filter '/config/'
add_filter '/lib/'
add_filter '/vendor/'
add_group 'Controllers', 'app/controllers'
add_group 'Models', 'app/models'
add_group 'Helpers', 'app/helpers'
add_group 'Mailers', 'app/mailers'
add_group 'Views', 'app/views'
end
from simplecov.
I am using Sinatra, RSpec, Guard, Spork, and SimpleCov, and I had the same issue where code metrics weren't being generated properly while running with Guard.
Posting here, with a complete spec_helper.rb
file, in case other people try to do something similar with Sinatra.
This was my initial spec_helper, and the tip above where you checked for the DRB env var didn't work for me. This is what my spec_helper.rb
started with. You can see the requiring files in the prefork
block, which is what was the issue.
# Note: Ideas from http://blog.codenursery.com/2011/11/adding-rspec-to-sinatra.html
# and http://www.iamzp.com/blog/2013/04/12/test-driven-development-with-sinatra-rspec-and-guard/
require 'rubygems'
require 'spork'
ENV['RACK_ENV'] = 'test'
def require_files
ruby_files = File.join(File.dirname(__FILE__), '..', 'lib', '**', '*.rb')
Dir[ruby_files].each do |file|
require file
end
require File.join(File.dirname(__FILE__), '..', 'app.rb')
end
Spork.prefork do
require_files
require 'rubygems'
require 'sinatra'
require 'rspec'
require 'rack/test'
# Set up the test env
set :environment, :test
set :run, false
set :raise_errors, true
set :logging, false
def app
@app ||= Sinatra::Application
end
RSpec.configure do |config|
config.include Rack::Test::Methods
end
end
Spork.each_run do
end
I used some of the ideas above to change it to what follows, which generates the correct code coverage metrics:
# Note: Ideas from http://blog.codenursery.com/2011/11/adding-rspec-to-sinatra.html
# and http://www.iamzp.com/blog/2013/04/12/test-driven-development-with-sinatra-rspec-and-guard/
require 'rubygems'
require 'spork'
ENV['RACK_ENV'] = 'test'
def require_files
ruby_files = File.join(File.dirname(__FILE__), '..', 'lib', '**', '*.rb')
Dir[ruby_files].each do |file|
require file
end
require File.join(File.dirname(__FILE__), '..', 'app.rb')
end
Spork.prefork do
require 'rubygems'
require 'sinatra'
require 'rspec'
require 'rack/test'
# Set up the test env
set :environment, :test
set :run, false
set :raise_errors, true
set :logging, false
def app
@app ||= Sinatra::Application
end
end
Spork.each_run do
require 'simplecov'
SimpleCov.start do
add_filter '/spec/'
end
require_files
RSpec.configure do |config|
config.include Rack::Test::Methods
end
end
from simplecov.
ref: #340
from simplecov.
Closing as stale
from simplecov.
Related Issues (20)
- Fix code coverage (eval) for delegates
- Bug: random branch coverage data in most cases. HOT 2
- Reporting branch coverage percentage in the terminal output HOT 2
- branch coverage data missing after collating .resultset.json in parallel test HOT 2
- allow skipping (unreachable) branch coverage only HOT 10
- option to disable line warnings with `enable_coverage_for_eval` and rails views HOT 3
- Use of ractors makes coverage non-deterministic HOT 3
- Branch coverage for the files which don't have _spec files
- "Coverage report generated" is being written to the stdout stream
- Default configurations around coverage criteria and minimum coverage
- Minor documentation fix
- [Feature Request] Don't require all branches to be covered twice in ensure-blocks HOT 3
- Support for other Parallel Testing gems
- SimpleCov does not fail for low coverage for parallel_tests when using 1 process
- SimpleCov only reports the last controller test and sets to 0.0% all previous results HOT 1
- False positives for `begin` lines, a constant definition, and some heredoc content
- مشکل <3
- Rails 7.1 update => test coverage drop HOT 6
- Coverage changes after each run HOT 1
- 0.22.0 release is missing on GitHub
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from simplecov.