simplecov-ruby / simplecov

Code coverage for Ruby with a powerful configuration library and automatic merging of coverage across test suites

SimpleCov

Code coverage for Ruby

• Source Code
• API documentation
• Changelog
• Rubygem
• Continuous Integration

SimpleCov is a code coverage analysis tool for Ruby. It uses Ruby's built-in Coverage library to gather code coverage data, but makes processing its results much easier by providing a clean API to filter, group, merge, format, and display those results, giving you a complete code coverage suite that can be set up with just a couple lines of code. SimpleCov/Coverage track covered ruby code, gathering coverage for common templating solutions like erb, slim and haml is not supported.

In most cases, you'll want overall coverage results for your projects, including all types of tests, Cucumber features, etc. SimpleCov automatically takes care of this by caching and merging results when generating reports, so your report actually includes coverage across your test suites and thereby gives you a better picture of blank spots.

The official formatter of SimpleCov is packaged as a separate gem called simplecov-html, but will be installed and configured automatically when you launch SimpleCov. If you're curious, you can find it on GitHub, too.

Contact

Code and Bug Reports

• Issue Tracker
• See CONTRIBUTING for how to contribute along with some common problems to check out before creating an issue.

Questions, Problems, Suggestions, etc.

• Mailing List "Open mailing list for discussion and announcements on Google Groups"

Getting started

1. Add SimpleCov to your Gemfile and bundle install:

   gem 'simplecov', require: false, group: :test

2. Load and launch SimpleCov at the very top of your test/test_helper.rb (or spec_helper.rb, rails_helper, cucumber env.rb, or whatever your preferred test framework uses):

   require 'simplecov'
   SimpleCov.start
   
   # Previous content of test helper now starts here

   Note: If SimpleCov starts after your application code is already loaded (via require), it won't be able to track your files and their coverage! The SimpleCov.start must be issued before any of your application code is required!

   This is especially true if you use anything that keeps your tests application loaded like spring, check out the spring section.

   SimpleCov must be running in the process that you want the code coverage analysis to happen on. When testing a server process (e.g. a JSON API endpoint) via a separate test process (e.g. when using Selenium) where you want to see all code executed by the rails server, and not just code executed in your actual test files, you need to require SimpleCov in the server process. For rails for instance, you'll want to add something like this to the top of bin/rails, but below the "shebang" line (#! /usr/bin/env ruby) and after config/boot is required:

   if ENV['RAILS_ENV'] == 'test'
     require 'simplecov'
     SimpleCov.start 'rails'
     puts "required simplecov"
   end

3. Run your full test suite to see the percent coverage that your application has.

4. After running your tests, open coverage/index.html in the browser of your choice. For example, in a Mac Terminal, run the following command from your application's root directory:

   open coverage/index.html
   

   in a debian/ubuntu Terminal,

   xdg-open coverage/index.html
   

   Note: This guide can help if you're unsure which command your particular operating system requires.

5. Add the following to your .gitignore file to ensure that coverage results are not tracked by Git (optional):

   echo coverage >> .gitignore
   

   If you're making a Rails application, SimpleCov comes with built-in configurations (see below for information on profiles) that will get you started with groups for your Controllers, Models and Helpers. To use it, the first two lines of your test_helper should be like this:

   require 'simplecov'
   SimpleCov.start 'rails'

Example output

Coverage results report, fully browsable locally with sorting and much more:

Source file coverage details view:

Use it with any framework!

Similarly to the usage with Test::Unit described above, the only thing you have to do is to add the SimpleCov config to the very top of your Cucumber/RSpec/whatever setup file.

Add the setup code to the top of features/support/env.rb (for Cucumber) or spec/spec_helper.rb (for RSpec). Other test frameworks should work accordingly, whatever their setup file may be:

require 'simplecov'
SimpleCov.start 'rails'

You could even track what kind of code your UI testers are touching if you want to go overboard with things. SimpleCov does not care what kind of framework it is running in; it just looks at what code is being executed and generates a report about it.

Notes on specific frameworks and test utilities

For some frameworks and testing tools there are quirks and problems you might want to know about if you want to use SimpleCov with them. Here's an overview of the known ones:

Framework	Notes	Issue
parallel_tests	As of 0.8.0, SimpleCov should correctly recognize parallel_tests and supplement your test suite names with their corresponding test env numbers. SimpleCov locks the resultset cache while merging, ensuring no race conditions occur when results are merged.	#64 & #185
knapsack_pro	To make SimpleCov work with Knapsack Pro Queue Mode to split tests in parallel on CI jobs you need to provide CI node index number to the SimpleCov.command_name in KnapsackPro::Hooks::Queue.before_queue hook.	Tip
RubyMine	The RubyMine IDE has built-in support for SimpleCov's coverage reports, though you might need to explicitly set the output root using `SimpleCov.root('foo/bar/baz')`	#95
Spork	Because of how Spork works internally (using preforking), there used to be trouble when using SimpleCov with it, but that has apparently been resolved with a specific configuration strategy. See this comment.	#42
Spring	See section below.	#381
Test/Unit	Test Unit 2 used to mess with ARGV, leading to a failure to detect the test process name in SimpleCov. test-unit releases 2.4.3+ (Dec 11th, 2011) should have this problem resolved.	#45 & test-unit/test-unit#12

Configuring SimpleCov

Configuration settings can be applied in three formats, which are completely equivalent:

• The most common way is to configure it directly in your start block:

  SimpleCov.start do
    some_config_option 'foo'
  end

• You can also set all configuration options directly:

  SimpleCov.some_config_option 'foo'

• If you do not want to start coverage immediately after launch or want to add additional configuration later on in a concise way, use:

  SimpleCov.configure do
    some_config_option 'foo'
  end

Please check out the Configuration API documentation to find out what you can customize.

Using .simplecov for centralized config

If you use SimpleCov to merge multiple test suite results (e.g. Test/Unit and Cucumber) into a single report, you'd normally have to set up all your config options twice, once in test_helper.rb and once in env.rb.

To avoid this, you can place a file called .simplecov in your project root. You can then just leave the require 'simplecov' in each test setup helper (at the top) and move the SimpleCov.start code with all your custom config options into .simplecov:

# test/test_helper.rb
require 'simplecov'

# features/support/env.rb
require 'simplecov'

# .simplecov
SimpleCov.start 'rails' do
  # any custom configs like groups and filters can be here at a central place
end

Using .simplecov rather than separately requiring SimpleCov multiple times is recommended if you are merging multiple test frameworks like Cucumber and RSpec that rely on each other, as invoking SimpleCov multiple times can cause coverage information to be lost.

Branch coverage (ruby "~> 2.5")

Add branch coverage measurement statistics to your results. Supported in CRuby versions 2.5+.

SimpleCov.start do
  enable_coverage :branch
end

Branch coverage is a feature introduced in Ruby 2.5 concerning itself with whether a particular branch of a condition had been executed. Line coverage on the other hand is only interested in whether a line of code has been executed.

This comes in handy for instance for one line conditionals:

number.odd? ? "odd" : "even"

In line coverage this line would always be marked as executed but you'd never know if both conditions were met. Guard clauses have a similar story:

return if number.odd?

# more code

If all the code in that method was covered you'd never know if the guard clause was ever triggered! With line coverage as just evaluating the condition marks it as covered.

In the HTML report the lines of code will be annotated like branch_type: hit_count:

• then: 2 - the then branch (of an if) was executed twice
• else: 0 - the else branch (of an if or case) was never executed

Not that even if you don't declare an else branch it will still show up in the coverage reports meaning that the condition of the if was not hit or that no when of case was hit during the test runs.

Is branch coverage strictly better? No. Branch coverage really only concerns itself with conditionals - meaning coverage of sequential code is of no interest to it. A file without conditional logic will have no branch coverage data and SimpleCov will report 0 of 0 branches covered as 100% (as everything that can be covered was covered).

Hence, we recommend looking at both metrics together. Branch coverage might also be a good overall metric to look at - while you might be missing only 10% of your lines that might account for 50% of your branches for instance.

Primary Coverage

By default, the primary coverage type is line. To set the primary coverage to something else, use the following:

# or in configure SimpleCov.primary_coverage :branch
SimpleCov.start do
  enable_coverage :branch
  primary_coverage :branch
end

Primary coverage determines what will come in first all output, and the type of coverage to check if you don't specify the type of coverage when customizing exit behavior (SimpleCov.minimum_coverage 90).

Note that coverage must first be enabled for non-default coverage types.

Coverage for eval

You can measure coverage for code that is evaluated by Kernel#eval. Supported in CRuby versions 3.2+.

SimpleCov.start do
  enable_coverage_for_eval
end

This is typically useful for ERB. Set ERB#filename= to make it possible for SimpleCov to trace the original .erb source file.

Filters

Filters can be used to remove selected files from your coverage data. By default, a filter is applied that removes all files OUTSIDE of your project's root directory - otherwise you'd end up with billions of coverage reports for source files in the gems you are using.

You can define your own to remove things like configuration files, tests or whatever you don't need in your coverage report.

Defining custom filters

You can currently define a filter using either a String or Regexp (that will then be Regexp-matched against each source file's path), a block or by passing in your own Filter class.

String filter

SimpleCov.start do
  add_filter "/test/"
end

This simple string filter will remove all files that match "/test/" in their path.

Regex filter

SimpleCov.start do
  add_filter %r{^/test/}
end

This simple regex filter will remove all files that start with /test/ in their path.

Block filter

SimpleCov.start do
  add_filter do |source_file|
    source_file.lines.count < 5
  end
end

Block filters receive a SimpleCov::SourceFile instance and expect your block to return either true (if the file is to be removed from the result) or false (if the result should be kept). Please check out the RDoc for SimpleCov::SourceFile to learn about the methods available to you. In the above example, the filter will remove all files that have less than 5 lines of code.

Custom filter class

class LineFilter < SimpleCov::Filter
  def matches?(source_file)
    source_file.lines.count < filter_argument
  end
end

SimpleCov.add_filter LineFilter.new(5)

Defining your own filters is pretty easy: Just inherit from SimpleCov::Filter and define a method 'matches?(source_file)'. When running the filter, a true return value from this method will result in the removal of the given source_file. The filter_argument method is being set in the SimpleCov::Filter initialize method and thus is set to 5 in this example.

Array filter

SimpleCov.start do
  proc = Proc.new { |source_file| false }
  add_filter ["string", /regex/, proc, LineFilter.new(5)]
end

You can pass in an array containing any of the other filter types.

Ignoring/skipping code

You can exclude code from the coverage report by wrapping it in # :nocov:.

# :nocov:
def skip_this_method
  never_reached
end
# :nocov:

The name of the token can be changed to your liking. Learn more about the nocov feature.

Note: You shouldn't have to use the nocov token to skip private methods that are being included in your coverage. If you appropriately test the public interface of your classes and objects you should automatically get full coverage of your private methods.

Default root filter and coverage for things outside of it

By default, SimpleCov filters everything outside of the SimpleCov.root directory. However, sometimes you may want to include coverage reports for things you include as a gem, for example a Rails Engine.

Here's an example by @lsaffie from #221 that shows how you can achieve just that:

SimpleCov.start :rails do
  filters.clear # This will remove the :root_filter and :bundler_filter that come via simplecov's defaults
  add_filter do |src|
    !(src.filename =~ /^#{SimpleCov.root}/) unless src.filename =~ /my_engine/
  end
end

Groups

You can separate your source files into groups. For example, in a Rails app, you'll want to have separate listings for Models, Controllers, Helpers, and Libs. Group definition works similarly to Filters (and also accepts custom filter classes), but source files end up in a group when the filter passes (returns true), as opposed to filtering results, which exclude files from results when the filter results in a true value.

Add your groups with:

SimpleCov.start do
  add_group "Models", "app/models"
  add_group "Controllers", "app/controllers"
  add_group "Long files" do |src_file|
    src_file.lines.count > 100
  end
  add_group "Multiple Files", ["app/models", "app/controllers"] # You can also pass in an array
  add_group "Short files", LineFilter.new(5) # Using the LineFilter class defined in Filters section above
end

Merging results

You normally want to have your coverage analyzed across ALL of your test suites, right?

Simplecov automatically caches coverage results in your (coverage_path)/.resultset.json, and will merge or override those with subsequent runs, depending on whether simplecov considers those subsequent runs as different test suites or as the same test suite as the cached results. To make this distinction, simplecov has the concept of "test suite names".

Test suite names

SimpleCov tries to guess the name of the currently running test suite based upon the shell command the tests are running on. This should work fine for Unit Tests, RSpec, and Cucumber. If it fails, it will use the shell command that invoked the test suite as a command name.

If you have some non-standard setup and still want nicely labeled test suites, you have to give Simplecov a cue as to what the name of the currently running test suite is. You can do so by specifying SimpleCov.command_name in one test file that is part of your specific suite.

To customize the suite names on a Rails app (yeah, sorry for being Rails-biased, but everyone knows what the structure of those projects is. You can apply this accordingly to the RSpecs in your Outlook-WebDAV-Calendar-Sync gem), you could do something like this:

# test/unit/some_test.rb
SimpleCov.command_name 'test:units'

# test/functionals/some_controller_test.rb
SimpleCov.command_name "test:functionals"

# test/integration/some_integration_test.rb
SimpleCov.command_name "test:integration"

# features/support/env.rb
SimpleCov.command_name "features"

Note that this only has to be invoked ONCE PER TEST SUITE, so even if you have 200 unit test files, specifying it in some_test.rb is enough.

Last but not least if multiple suites resolve to the same command_name be aware that the coverage results will clobber each other instead of being merged. SimpleCov is smart enough to detect unique names for the most common setups, but if you have more than one test suite that doesn't follow a common pattern then you will want to manually ensure that each suite gets a unique command_name.

If you are running tests in parallel each process has the potential to clobber results from the other test processes. If you are relying on the default command_name then SimpleCov will attempt to detect and avoid parallel test suite command_name collisions based on the presence of ENV['PARALLEL_TEST_GROUPS'] and ENV['TEST_ENV_NUMBER']. If your parallel test runner does not set one or both of these then you must set a command_name and ensure that it is unique per process (eg. command_name "Unit Tests PID #{$$}").

If you are using parallel_tests, you must incorporate TEST_ENV_NUMBER into the command name yourself, in order for SimpleCov to merge the results correctly. For example:

# spec/spec_helper.rb
SimpleCov.command_name "features" + (ENV['TEST_ENV_NUMBER'] || '')

simplecov-html prints the used test suites in the footer of the generated coverage report.

Merging test runs under the same execution environment

Test results are automatically merged with previous runs in the same execution environment when generating the result, so when coverage is set up properly for Cucumber and your unit / functional / integration tests, all of those test suites will be taken into account when building the coverage report.

Timeout for merge

Of course, your cached coverage data is likely to become invalid at some point. Thus, when automatically merging subsequent test runs, result sets that are older than SimpleCov.merge_timeout will not be used any more. By default, the timeout is 600 seconds (10 minutes), and you can raise (or lower) it by specifying SimpleCov.merge_timeout 3600 (1 hour), or, inside a configure/start block, with just merge_timeout 3600.

You can deactivate this automatic merging altogether with SimpleCov.use_merging false.

Merging test runs under different execution environments

If your tests are done in parallel across multiple build machines, you can fetch them all and merge them into a single result set using the SimpleCov.collate method. This can be added to a Rakefile or script file, having downloaded a set of .resultset.json files from each parallel test run.

# lib/tasks/coverage_report.rake
namespace :coverage do
  desc "Collates all result sets generated by the different test runners"
  task :report do
    require 'simplecov'

    SimpleCov.collate Dir["simplecov-resultset-*/.resultset.json"]
  end
end

SimpleCov.collate also takes an optional simplecov profile and an optional block for configuration, just the same as SimpleCov.start or SimpleCov.configure. This means you can configure a separate formatter for the collated output. For instance, you can make the formatter in SimpleCov.start the SimpleCov::Formatter::SimpleFormatter, and only use more complex formatters in the final SimpleCov.collate run.

# spec/spec_helper.rb
require 'simplecov'

SimpleCov.start 'rails' do
  # Disambiguates individual test runs
  command_name "Job #{ENV["TEST_ENV_NUMBER"]}" if ENV["TEST_ENV_NUMBER"]

  if ENV['CI']
    formatter SimpleCov::Formatter::SimpleFormatter
  else
    formatter SimpleCov::Formatter::MultiFormatter.new([
      SimpleCov::Formatter::SimpleFormatter,
      SimpleCov::Formatter::HTMLFormatter
    ])
  end

  track_files "**/*.rb"
end

# lib/tasks/coverage_report.rake
namespace :coverage do
  task :report do
    require 'simplecov'

    SimpleCov.collate Dir["simplecov-resultset-*/.resultset.json"], 'rails' do
      formatter SimpleCov::Formatter::MultiFormatter.new([
        SimpleCov::Formatter::SimpleFormatter,
        SimpleCov::Formatter::HTMLFormatter
      ])
    end
  end
end

Running simplecov against subprocesses

SimpleCov.enable_for_subprocesses will allow SimpleCov to observe subprocesses starting using Process.fork. This modifies ruby's core Process.fork method so that SimpleCov can see into it, appending " (subprocess #{pid})" to the SimpleCov.command_name, with results that can be merged together using SimpleCov's merging feature.

To configure this, use .at_fork.

SimpleCov.enable_for_subprocesses true
SimpleCov.at_fork do |pid|
  # This needs a unique name so it won't be overwritten
  SimpleCov.command_name "#{SimpleCov.command_name} (subprocess: #{pid})"
  # be quiet, the parent process will be in charge of output and checking coverage totals
  SimpleCov.print_error_status = false
  SimpleCov.formatter SimpleCov::Formatter::SimpleFormatter
  SimpleCov.minimum_coverage 0
  # start
  SimpleCov.start
end

NOTE: SimpleCov must have already been started before Process.fork was called.

Running simplecov against spawned subprocesses

Perhaps you're testing a ruby script with PTY.spawn or Open3.popen, or Process.spawn or etc. SimpleCov can cover this too.

Add a .simplecov_spawn.rb file to your project root

# .simplecov_spawn.rb
require 'simplecov' # this will also pick up whatever config is in .simplecov
                    # so ensure it just contains configuration, and doesn't call SimpleCov.start.
SimpleCov.command_name 'spawn' # As this is not for a test runner directly, script doesn't have a pre-defined base command_name
SimpleCov.at_fork.call(Process.pid) # Use the per-process setup described previously
SimpleCov.start # only now can we start.

Then, instead of calling your script directly, like:

PTY.spawn('my_script.rb') do # ...

Use bin/ruby to require the new .simplecov_spawn file, then your script

PTY.spawn('ruby -r./.simplecov_spawn my_script.rb') do # ...

Running coverage only on demand

The Ruby STDLIB Coverage library that SimpleCov builds upon is very fast (on a ~10 min Rails test suite, the speed drop was only a couple seconds for me), and therefore it's SimpleCov's policy to just generate coverage every time you run your tests because it doesn't do your test speed any harm and you're always equipped with the latest and greatest coverage results.

Because of this, SimpleCov has no explicit built-in mechanism to run coverage only on demand.

However, you can still accomplish this very easily by introducing an ENV variable conditional into your SimpleCov setup block, like this:

SimpleCov.start if ENV["COVERAGE"]

Then, SimpleCov will only run if you execute your tests like this:

COVERAGE=true rake test

Errors and exit statuses

To aid in debugging issues, if an error is raised, SimpleCov will print a message to STDERR with the exit status of the error, like:

SimpleCov failed with exit 1

This STDERR message can be disabled with:

SimpleCov.print_error_status = false

Profiles

By default, SimpleCov's only config assumption is that you only want coverage reports for files inside your project root. To save yourself from repetitive configuration, you can use predefined blocks of configuration, called 'profiles', or define your own.

You can then pass the name of the profile to be used as the first argument to SimpleCov.start. For example, simplecov comes bundled with a 'rails' profile. It looks somewhat like this:

SimpleCov.profiles.define 'rails' do
  add_filter '/test/'
  add_filter '/config/'

  add_group 'Controllers', 'app/controllers'
  add_group 'Models', 'app/models'
  add_group 'Helpers', 'app/helpers'
  add_group 'Libraries', 'lib'
end

As you can see, it's just a SimpleCov.configure block. In your test_helper.rb, launch SimpleCov with:

SimpleCov.start 'rails'

or

SimpleCov.start 'rails' do
  # additional config here
end

Custom profiles

You can load additional profiles with the SimpleCov.load_profile('xyz') method. This allows you to build upon an existing profile and customize it so you can reuse it in unit tests and Cucumber features. For example:

# lib/simplecov_custom_profile.rb
require 'simplecov'
SimpleCov.profiles.define 'myprofile' do
  load_profile 'rails'
  add_filter 'vendor' # Don't include vendored stuff
end

# features/support/env.rb
require 'simplecov_custom_profile'
SimpleCov.start 'myprofile'

# test/test_helper.rb
require 'simplecov_custom_profile'
SimpleCov.start 'myprofile'

Customizing exit behaviour

You can define what SimpleCov should do when your test suite finishes by customizing the at_exit hook:

SimpleCov.at_exit do
  SimpleCov.result.format!
end

Above is the default behaviour. Do whatever you like instead!

Minimum coverage

You can define the minimum coverage percentage expected. SimpleCov will return non-zero if unmet.

SimpleCov.minimum_coverage 90
# same as above (the default is to check line coverage)
SimpleCov.minimum_coverage line: 90
# check for a minimum line coverage of 90% and minimum 80% branch coverage
SimpleCov.minimum_coverage line: 90, branch: 80

Minimum coverage by file

You can define the minimum coverage by file percentage expected. SimpleCov will return non-zero if unmet. This is useful to help ensure coverage is relatively consistent, rather than being skewed by particularly good or bad areas of the code.

SimpleCov.minimum_coverage_by_file 80
# same as above (the default is to check line coverage by file)
SimpleCov.minimum_coverage_by_file line: 80
# check for a minimum line coverage by file of 90% and minimum 80% branch coverage
SimpleCov.minimum_coverage_by_file line: 90, branch: 80

Maximum coverage drop

You can define the maximum coverage drop percentage at once. SimpleCov will return non-zero if exceeded.

SimpleCov.maximum_coverage_drop 5
# same as above (the default is to check line drop)
SimpleCov.maximum_coverage_drop line: 5
# check for a maximum line drop of 5% and maximum 10% branch drop
SimpleCov.maximum_coverage_drop line: 5, branch: 10

Refuse dropping coverage

You can also entirely refuse dropping coverage between test runs:

SimpleCov.refuse_coverage_drop
# same as above (the default is to only refuse line drop)
SimpleCov.refuse_coverage_drop :line
# refuse drop for line and branch
SimpleCov.refuse_coverage_drop :line, :branch

Using your own formatter

You can use your own formatter with:

SimpleCov.formatter = SimpleCov::Formatter::HTMLFormatter

Calling SimpleCov.result.format! will be invoked with SimpleCov::Formatter::YourFormatter.new.format(result), and result is an instance of SimpleCov::Result. Do whatever your wish with that!

Using multiple formatters

As of SimpleCov 0.9, you can specify multiple result formats. Formatters besides the default HTML formatter require separate gems, however.

require "simplecov-html"

SimpleCov.formatters = SimpleCov::Formatter::MultiFormatter.new([
  SimpleCov::Formatter::HTMLFormatter,
  SimpleCov::Formatter::CSVFormatter,
])

JSON formatter

SimpleCov is packaged with a separate gem called simplecov_json_formatter that provides you with a JSON formatter, this formatter could be useful for different use cases, such as for CI consumption or for reporting to external services.

In order to use it you will need to manually load the installed gem like so:

require "simplecov_json_formatter"
SimpleCov.formatter = SimpleCov::Formatter::JSONFormatter

Note: In case you plan to report your coverage results to CodeClimate services, know that SimpleCov will automatically use the JSON formatter along with the HTML formatter when the CC_TEST_REPORTER_ID variable is present in the environment.

Available formatters, editor integrations and hosted services

• Open Source formatter and integration plugins for SimpleCov
• Editor Integration
• Hosted (commercial) services

Ruby version compatibility

SimpleCov is built in Continuous Integration on Ruby 2.7+ as well as JRuby 9.3+.

Note for JRuby => You need to pass JRUBY_OPTS="--debug" or create .jrubyrc and add debug.fullTrace=true

Want to find dead code in production?

Try Coverband.

Want to use Spring with SimpleCov?

If you're using Spring to speed up test suite runs and want to run SimpleCov along with them, you'll find that it often misreports coverage with the default config due to some sort of eager loading issue. Don't despair!

One solution is to explicitly call eager load in your test_helper.rb / spec_helper.rb after calling SimpleCov.start.

require 'simplecov'
SimpleCov.start 'rails'
Rails.application.eager_load!

Alternatively, you could disable Spring while running SimpleCov:

DISABLE_SPRING=1 rake test

Or you could remove gem 'spring' from your Gemfile.

Troubleshooting

The most common problem is that simplecov isn't required and started before everything else. In order to track coverage for your whole application simplecov needs to be the first one so that it (and the underlying coverage library) can subsequently track loaded files and their usage.

If you are missing coverage for some code a simple trick is to put a puts statement in there and right after SimpleCov.start so you can see if the file really was loaded after simplecov was started.

# my_code.rb
class MyCode

  puts "MyCode is being loaded!"

  def my_method
    # ...
  end
end

# spec_helper.rb/rails_helper.rb/test_helper.rb/.simplecov whatever

SimpleCov.start
puts "SimpleCov started successfully!"

Now when you run your test suite and you see:

SimpleCov started successfully!
MyCode is being loaded!

then it's good otherwise you likely have a problem :)

Code of Conduct

Everyone participating in this project's development, issue trackers and other channels is expected to follow our Code of Conduct

Contributing

See the contributing guide.

Kudos

Thanks to Aaron Patterson for the original idea for this!

Copyright

Copyright (c) 2010-2017 Christoph Olszowka. See MIT-LICENSE for details.

simplecov's Issues

empty resultset.yml

If there is an empty resultset.yml file (maybe from an aborted run) you get the error:
lib/simplecov/result_merger.rb:56:in store_result': undefined method[]=' for false:FalseClass (NoMethodError)

I don't know if this needs a fix but it might be useful to have a troubleshooting note.

Coverage on ERB

Great work on this gem. Is it possible to check code coverage in ERB templates? In a Rails app there might be an "if" condition in a view which is not covered. It would be great if it were somehow possible to check this.

cucumber and rspec results not merged

When I run

rake spec cucumber

and check the coverage I'm finding that the cucumber results overwrite the rspec results. This is easy to confirm because I have a couple of classes with 100% rspec coverage but only 50% cucumber coverage. I've tried setting

SimpleCov.use_merging true

but no luck. My understanding is that they should be merged by default, is this correct? If not how's it done?

I'm using ruby 1.9.2p136, rails 3.0.5 and simplecov 0.4.1

Namespacing Coverage Class

Because I have a model in my rails app called Coverage when running tests I get the error:

Coverage is not a class (TypeError)

This is obviously a naming conflict. Do you think it is worth name spacing the simplecov internal classes so they do not conflict with the app being tested?

Thanks!

Update to work with 1.8 using ripper (gem)

A few months ago @lsegal released ripper, a rubygem that ports the ripper parser to 1.8:

http://gnuu.org/2011/01/12/announcing-ripper-for-ruby-1-8-x/

I was wondering if it were possible to get simplecov to work with that gem on 1.8, or are there other dependencies on 1.9 besides ripper?

I could see about testing this out but I wanted to check with you first in case there are constraints I don't know about.

1.8 Compatibility via TracePoint

I believe I can give you 1.8 compatibility, and in fact compatibility with any ruby engine that supports #set_trace_func.

A long time ago I wrote a tool called TracePoint. It should be pretty trivial to provide a result from it that exactly mimics Coverage.result. Would that do the trick? If so I will write it for you.

NaN FloatDomainError (Reopen of #15 and #23)

I just accidentally closed #23 which I opened this morning. ... I don't know how to reopen it. It's a real issue. Please read it.

https://github.com/colszowka/simplecov/issues/closed/#issue/23

empty reports under test-loop

Hello,

I'm getting empty reports (no files listed; all sections 100% coverage) when running my Rails 3.0.7 app's test suite in parallel with test-loop and SimpleCov 0.4.2. This problem was also described previously in issue #30.

I tried the following ways of invoking SimpleCov in my test suite.

Starting SimpleCov at the top of my test/test_helper.rb in the master process:

require 'simplecov'
SimpleCov.start 'rails'

ENV["RAILS_ENV"] = "test"
require File.expand_path('../../config/environment', __FILE__)
require 'rails/test_help'

Test::Loop.before_each_test.push lambda {
  |test_file, log_file, test_names|

  SimpleCov.root File.dirname(log_file)
  SimpleCov.coverage_dir File.basename(test_file) + '.cov'
}

Starting SimpleCov in a worker process just before it loads and runs a test/**_test.rb file from my Rails app test suite:

ENV["RAILS_ENV"] = "test"
require File.expand_path('../../config/environment', __FILE__)
require 'rails/test_help'

Test::Loop.before_each_test.push lambda {
  |test_file, log_file, test_names|

  require 'simplecov'
  SimpleCov.start 'rails'
  SimpleCov.root File.dirname(log_file)
  SimpleCov.coverage_dir File.basename(test_file) + '.cov'
}

Starting SimpleCov (after configuring its output locations) in a worker process just before it loads and runs a test/**_test.rb file from my Rails app test suite:

ENV["RAILS_ENV"] = "test"
require File.expand_path('../../config/environment', __FILE__)
require 'rails/test_help'

Test::Loop.before_each_test.push lambda {
  |test_file, log_file, test_names|

  require 'simplecov'
  SimpleCov.root File.dirname(log_file)
  SimpleCov.coverage_dir File.basename(test_file) + '.cov'
  SimpleCov.start 'rails'
}

None of these worked; I always get empty reports. Any ideas?

Thanks for your consideration.

exit code 0 when there are exceptions prior to tests

This problem is demonstrated by putting the following code in spec/spec_helper.rb of a rails project.

require 'simplecov'
SimpleCov.start 'rails'
raise "some exception in the class loading before the tests start"

When rspec is invoked via rake, the above exception is logged to stderr and no tests are run, but the rake process exits 0 (i.e. success), where it should exit non-zero (failure) as it does when the SimpleCov.start is omitted.

Context:
This is a simplified version of the problem we encountered. We had a CI box that didn't receive a new config file required by a codebase change. When rake (rspec) ran, our app initialization code correctly raised an exception to protest but because our test suite was running under simplecov the rake (rspec) task exited 0 (success) indicating a passing build.

Environment:
ruby 1.9.2-p180,
simplecov gem 0.4.2
rspec gem 2.5.0
rubygems 1.6.2

Extra information:
I note that if I register an at_exit handler which does NOT call SimpleCov.result.format!, the exit code is propogated correctly (though of course no report will be produced). E.g. SimpleCov.at_exit { puts "simplecov at_exit handler - doing nothing" }

Thanks for any help, and for your work on this gem!

in html report, pop-view has no maximize button

Hi,

sometimes my code is too big to fit in the default pop-up window for line coverage hit analysis.

Would it be possible to add a 'maximize' button ?

Christian

Create a command-line runner of simplecov

I haven't used this gem, so I may be mistaken, but it looks it's saying you have to add code mentioning simplecov to your code base.

Is it practical to create a simplecov executable and do

simplecov test/my_test.rb

without having to add any code?

Simplecov (0.4.2) with parallel_tests (0.4.13)

Hi,

Running simplecov along with parallel_tests. Added the following SimpleCov block in both spec/spec_helper.rb
and features/support/env.rb

SimpleCov.start 'rails' do
merge_timeout 7200
add_filter "/vendor/"
end

Coverage results are overwritten which I understand as each Rspec or cucumber test generates
a coverage report.

Is it possible to run simplecov with parallel_tests and merge the coverage of each of
them ?

File content should be escaped

The files content isn't escaped. So if there's HTML in it (like in dynamic_form for example), the html fields are rendered in the page unescaped.

Tests don't run on the first or second day of the month

This test doesn't work on the first or second day of the month, because Time.mktime is called with:

Time.mktime(Time.now.year, Time.now.month, Time.now.day-2)

This can result in the month day being zero or negative, which is obviously not valid. A better way of calculating the timestamp two days (since activesupport isn't available) is something like:

Time.now - 172800 # two days ago

How do you show test coverage of routes?

As the title says... how do I include test coverage for my routes?

cucumber error

I'm using cucumber-rails 1.0.2 and cucumber 1.0,0, simplecov 0.4.2, ruby 1.9.2 and rails 3.0.9

After running the specs I'm getting

simplecov-0.4.2/lib/simplecov/merge_helpers.rb:16:in block in merge_resultset': undefined method+' for #Syck::PrivateType:0x00000109058a48 (NoMethodError)
from /Users/felix/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov/merge_helpers.rb:10:in each' from /Users/felix/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov/merge_helpers.rb:10:ineach_with_index'
from /Users/felix/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov/merge_helpers.rb:10:in merge_resultset' from /Users/felix/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov/merge_helpers.rb:32:inblock in merge_resultset'
from /Users/felix/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov/merge_helpers.rb:31:in each' from /Users/felix/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov/merge_helpers.rb:31:inmerge_resultset'
from /Users/felix/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov/result_merger.rb:44:in block in merged_result' from /Users/felix/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov/result_merger.rb:43:ineach'
from /Users/felix/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov/result_merger.rb:43:in merged_result' from /Users/felix/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov.rb:51:inresult'
from /Users/felix/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov/configuration.rb:120:in block in at_exit' from /Users/felix/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov.rb:134:incall'
from /Users/felix/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov.rb:134:in `block in <top (required)>'

Rake Task

Is there a way to put SimpleCov in a rake task? I would love to have it only run the coverage when I do rake coverage and then open the results with Launchy. I tried this in a Rails app:

desc "Run RSpec with code coverage"
task :coverage do
  require "simplecov"
  SimpleCov.start "rails"
  Rake::Task["spec"].execute
end

But I got this exception when running it.

/Users/rbates/.rvm/gems/ruby-1.9.2-p136@railscasts/gems/simplecov-0.4.2/lib/simplecov/source_file.rb:25:in `initialize': Only String accepted for source (ArgumentError)
    from /Users/rbates/.rvm/gems/ruby-1.9.2-p136@railscasts/gems/simplecov-0.4.2/lib/simplecov/source_file.rb:66:in `new'
    from /Users/rbates/.rvm/gems/ruby-1.9.2-p136@railscasts/gems/simplecov-0.4.2/lib/simplecov/source_file.rb:66:in `block in lines'
    from /Users/rbates/.rvm/gems/ruby-1.9.2-p136@railscasts/gems/simplecov-0.4.2/lib/simplecov/source_file.rb:65:in `each'
    from /Users/rbates/.rvm/gems/ruby-1.9.2-p136@railscasts/gems/simplecov-0.4.2/lib/simplecov/source_file.rb:65:in `each_with_index'
    from /Users/rbates/.rvm/gems/ruby-1.9.2-p136@railscasts/gems/simplecov-0.4.2/lib/simplecov/source_file.rb:65:in `lines'
    from /Users/rbates/.rvm/gems/ruby-1.9.2-p136@railscasts/gems/simplecov-0.4.2/lib/simplecov/source_file.rb:91:in `missed_lines'
    from /Users/rbates/.rvm/gems/ruby-1.9.2-p136@railscasts/gems/simplecov-html-0.4.4/lib/simplecov-html.rb:82:in `block in lines_missed'
    from /Users/rbates/.rvm/gems/ruby-1.9.2-p136@railscasts/gems/simplecov-html-0.4.4/lib/simplecov-html.rb:82:in `map'
    from /Users/rbates/.rvm/gems/ruby-1.9.2-p136@railscasts/gems/simplecov-html-0.4.4/lib/simplecov-html.rb:82:in `lines_missed'
    from /Users/rbates/.rvm/gems/ruby-1.9.2-p136@railscasts/gems/simplecov-html-0.4.4/lib/simplecov-html.rb:72:in `lines_of_code'
    from /Users/rbates/.rvm/gems/ruby-1.9.2-p136@railscasts/gems/simplecov-html-0.4.4/lib/simplecov-html.rb:65:in `coverage'
    from (erb):4:in `formatted_file_list'
    from /Users/rbates/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/erb.rb:753:in `eval'
    from /Users/rbates/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/erb.rb:753:in `result'
    from /Users/rbates/.rvm/gems/ruby-1.9.2-p136@railscasts/gems/simplecov-html-0.4.4/lib/simplecov-html.rb:60:in `formatted_file_list'
    from (erb):28:in `block in format'
    from /Users/rbates/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/erb.rb:753:in `eval'
    from /Users/rbates/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/erb.rb:753:in `result'
    from /Users/rbates/.rvm/gems/ruby-1.9.2-p136@railscasts/gems/simplecov-html-0.4.4/lib/simplecov-html.rb:26:in `block in format'
    from /Users/rbates/.rvm/gems/ruby-1.9.2-p136@railscasts/gems/simplecov-html-0.4.4/lib/simplecov-html.rb:25:in `open'
    from /Users/rbates/.rvm/gems/ruby-1.9.2-p136@railscasts/gems/simplecov-html-0.4.4/lib/simplecov-html.rb:25:in `format'
    from /Users/rbates/.rvm/gems/ruby-1.9.2-p136@railscasts/gems/simplecov-0.4.2/lib/simplecov/result.rb:77:in `format!'
    from /Users/rbates/.rvm/gems/ruby-1.9.2-p136@railscasts/gems/simplecov-0.4.2/lib/simplecov/configuration.rb:120:in `block in at_exit'
    from /Users/rbates/.rvm/gems/ruby-1.9.2-p136@railscasts/gems/simplecov-0.4.2/lib/simplecov.rb:134:in `call'
    from /Users/rbates/.rvm/gems/ruby-1.9.2-p136@railscasts/gems/simplecov-0.4.2/lib/simplecov.rb:134:in `block in <top (required)>'

Is there a good way to do this?

How to exclude a subdir from groups target directory?

How does one exclude a subdir from a group?

add_group "Foo", "lib/foo" - "lib/foo/bar"

so to speak.

NaN FloatDomainError (Reopen of #15)

https://github.com/colszowka/simplecov/issues/closed#issue/15

Okay, so I was seeing this for a while. It was almost reproduce-able, in that it was (I could add a pending to a test, then uncomment the pending and it would pass and fail deterministically.) So I took my code, and prepared to present it here to help find the problem ... and while doing so ... it started passing every time. So, the error definitely appears to be non-deterministic.

Anyway, here's the code
https://github.com/softwaregravy/base_project (first checkin and first change)
And here's the change. I just kept doing and undoing this and saw the error
softwaregravy/base_project@896d527

I was also installing and uninstalling annotate-models 1.0.4 which was the last gem I had installed before this happened (but no, I have no idea how that might impact it, just a reaction like "oh no, what did I change last"). First observed while running autotest.

It does not now appear to be reproduce-able. Weird. Anything pulling timestamps in the code?

I was in the process of factoring out my simplecov config (so there's some code there, probably in the wrong place, but it was not in use).

So, in conclusion, I observed the failure, was able to reproduce for a short while, but not cannot. Not vary satisfying, I know.

Only String accepted for source (ArgumentError)

I've run into this twice now, and I'm sorry, but I don't know what's tripping it up.

I'm using guard to run my tests / simplecov.

If it runs with this code:

def following_all
  ...
  all.map! do |a|
    # Return array
    if a.class == MyClass
      ... do something
    else
      ... do something else
    end
  end
  all.flatten!
end

It will be fine. But if I take out that the line with the comment (the comment doesn't matter, the extra line does) then it will crash.

/Users/ryanjm/.rvm/gems/ruby-1.9.2-head/gems/simplecov-0.4.2/lib/simplecov/source_file.rb:25:in `initialize': Only String accepted for source (ArgumentError)
    from /Users/ryanjm/.rvm/gems/ruby-1.9.2-head/gems/simplecov-0.4.2/lib/simplecov/source_file.rb:66:in `new'
    from /Users/ryanjm/.rvm/gems/ruby-1.9.2-head/gems/simplecov-0.4.2/lib/simplecov/source_file.rb:66:in `block in lines'
    from /Users/ryanjm/.rvm/gems/ruby-1.9.2-head/gems/simplecov-0.4.2/lib/simplecov/source_file.rb:65:in `each'
    from /Users/ryanjm/.rvm/gems/ruby-1.9.2-head/gems/simplecov-0.4.2/lib/simplecov/source_file.rb:65:in `each_with_index'
    from /Users/ryanjm/.rvm/gems/ruby-1.9.2-head/gems/simplecov-0.4.2/lib/simplecov/source_file.rb:65:in `lines'
    from /Users/ryanjm/.rvm/gems/ruby-1.9.2-head/gems/simplecov-0.4.2/lib/simplecov/source_file.rb:91:in `missed_lines'
    from /Users/ryanjm/.rvm/gems/ruby-1.9.2-head/gems/simplecov-html-0.4.5/lib/simplecov-html.rb:80:in `block in lines_missed'
    from /Users/ryanjm/.rvm/gems/ruby-1.9.2-head/gems/simplecov-html-0.4.5/lib/simplecov-html.rb:80:in `map'
    from /Users/ryanjm/.rvm/gems/ruby-1.9.2-head/gems/simplecov-html-0.4.5/lib/simplecov-html.rb:80:in `lines_missed'
    from /Users/ryanjm/.rvm/gems/ruby-1.9.2-head/gems/simplecov-html-0.4.5/lib/simplecov-html.rb:70:in `lines_of_code'
    from /Users/ryanjm/.rvm/gems/ruby-1.9.2-head/gems/simplecov-html-0.4.5/lib/simplecov-html.rb:63:in `coverage'
    from (erb):4:in `formatted_file_list'
    ....

Any idea what is tripping this issue?

Problem after update to 0.4.0

Nothing work for me anymore... =/

    /Users/diegorv/.rvm/gems/ruby-1.9.2-p136@rails3/gems/simplecov-html-0.4.0/lib/simplecov-html.rb:113:in `require': no such file to load -- simplecov-html/version (LoadError)
from /Users/diegorv/.rvm/gems/ruby-1.9.2-p136@rails3/gems/simplecov-html-0.4.0/lib/simplecov-html.rb:113:in `<top (required)>'
from <internal:lib/rubygems/custom_require>:29:in `require'
from <internal:lib/rubygems/custom_require>:29:in `require'
from /Users/diegorv/.rvm/gems/ruby-1.9.2-p136@rails3/gems/simplecov-0.4.0/lib/simplecov.rb:122:in `<top (required)>'
from <internal:lib/rubygems/custom_require>:33:in `require'
from <internal:lib/rubygems/custom_require>:33:in `rescue in require'
from <internal:lib/rubygems/custom_require>:29:in `require'
from /Users/diegorv/Documents/rails/work/phenix/spec/spec_helper.rb:5:in `block in <top (required)>'
from /Users/diegorv/.rvm/gems/ruby-1.9.2-p136@rails3/gems/spork-0.8.4/lib/spork.rb:23:in `prefork'
from /Users/diegorv/Documents/rails/work/phenix/spec/spec_helper.rb:4:in `<top (required)>'
from <internal:lib/rubygems/custom_require>:29:in `require'
from <internal:lib/rubygems/custom_require>:29:in `require'
from /Users/diegorv/Documents/rails/work/phenix/spec/helpers/admin/admin_helper_spec.rb:3:in `<top (required)>'
from /Users/diegorv/.rvm/gems/ruby-1.9.2-p136@rails3/gems/rspec-core-2.5.1/lib/rspec/core/configuration.rb:386:in `load'
from /Users/diegorv/.rvm/gems/ruby-1.9.2-p136@rails3/gems/rspec-core-2.5.1/lib/rspec/core/configuration.rb:386:in `block in load_spec_files'
from /Users/diegorv/.rvm/gems/ruby-1.9.2-p136@rails3/gems/rspec-core-2.5.1/lib/rspec/core/configuration.rb:386:in `map'
from /Users/diegorv/.rvm/gems/ruby-1.9.2-p136@rails3/gems/rspec-core-2.5.1/lib/rspec/core/configuration.rb:386:in `load_spec_files'
from /Users/diegorv/.rvm/gems/ruby-1.9.2-p136@rails3/gems/rspec-core-2.5.1/lib/rspec/core/command_line.rb:18:in `run'
from /Users/diegorv/.rvm/gems/ruby-1.9.2-p136@rails3/gems/rspec-core-2.5.1/lib/rspec/core/runner.rb:55:in `run_in_process'
from /Users/diegorv/.rvm/gems/ruby-1.9.2-p136@rails3/gems/rspec-core-2.5.1/lib/rspec/core/runner.rb:46:in `run'
from /Users/diegorv/.rvm/gems/ruby-1.9.2-p136@rails3/gems/rspec-core-2.5.1/lib/rspec/core/runner.rb:10:in `block in autorun'

Files with no coverage

Files without any coverage are not shown at all. It is important for me to be able to view these files.

As this is not detected by the built-in Coverage module, perhaps an option could be added to search for files in the current folder?

I'll try to create a proof-of-concept for this.

SimpleCov.result.covered_percent not matching html report coverage

Unfortunately, I can't upload my app code since its all proprietary company code. I'm using .3.7 with a pretty standard rails structure. I'm running simplecov like so:

SimpleCov.start 'rails' do
#remove chef cookbooks from reports
add_filter "/cookbooks/"
root(File.dirname(FILE) + "/..")
coverage_dir("coverage")
end

SimpleCov.at_exit do
puts "Coverage is #{SimpleCov.result.covered_percent}"
puts "Your coverage has dipped bellow 90%" if SimpleCov.result.covered_percent < 90
SimpleCov.result.format!
end

Both the html report and covered_percent are finding 99 files. The html report is showing 88.33 with no one module under 80.88. The command line is showing 80.44. I did find it a little odd that plugins in the html report is showing up at 100% with no files but I don't think that would affect the numbers like I'm seeing. Could this have something to do with lines vs relevant lines?

"Bonus" Score

I was thinking it might be cool to have a "bonus" score that goes over and above the percentage coverage. Basically it would just be the average multiple of times the lines were covered.

For simplicity sake lets say a program has only one relevant line of code. And lets say is was covered. Great so that's 100% coverage. But say that line was run not just once, but 3 times. So it would have a Bonus Multiple of 3.

I think that would add some extra "strength" to the coverage scores. What do you think?

SimpleCov causes specs error in Ruby 1.8.6

Trace: https://gist.github.com/dda22fa6e6c8c9b3974a

SimpleCov should throw a warning and skip coverage analysis in Ruby 1.8.6 (as in 1.8.7).

MRI 1.8.6 is still widely used in development since it shipped with Mac OS X 10.5.

ugly path in report

Hi,
I don't think this is technically a bug but I have no idea how to change the behavior, I organize my files like this:

project_root
  specs
    spec_helper.rb
      sevices
        m1_spec.rb
        m2_spec.rb

I execute my test with a simple:

bacon specs/**/*_spec.rb

and common.rb contains the simplecov start code, my problem is that like that simplecov (or the coverage module, no idea) will consider the specs folder as the root folder and display every path relative to it.

Is there a way to tells it what is the real root folder ?
I already have a root configuration directive in the simplecov block:

root( File.expand_path('../../', __FILE__) )

which sets the root to the top folder (all the requires in my code uses absolute expanded path).

All my path are shown like that in the report:

./specs/../lib/migration_helpers.rb
./specs/../services/dhcp_link_svc.rb

simplecov BREAKS rspec

require 'simplecov'
SimpleCov.start
describe 'exit status' do
  it "should exit with a non-zero exit status when assertion fails" do
    1.should == 2
  end
end

Put the above code in a file, run the file.

echo $?
=> 0

it should return 1

require 'simplecov' break require

irb(main):001:0> require 'git'
=> true
irb(main):002:0> require 'simplecov'
=> true
irb(main):003:0> require 'thin'
LoadError: no such file to load -- thin
from (irb):3:in require' from (irb):3 from /usr/bin/irb:12:in

'
irb(main):004:0> i

Coverage % doesn't change

Hi,

maybe this is more a help than a issue, but I'm blind at this moment to got what is going on.

Everything works fine in a Rails project, but after some test rans I saw that nothing is changing reports. I can comments all my tests and the coverage still the same.

Is anyone suffering from that?
Any clue?

custom adapter throws NaN FloatDomainError in formatted_file_list

Hi there,

the following setup worked fine until Rails 3.0.1.

I got the following custom adapter:

SimpleCov.adapters.define 'rspec' do
  add_filter 'test/'
  add_filter 'config/'
  add_filter 'spec/'
  add_filter 'features/'
  add_filter 'vendor/'
  add_filter 'lib/'

  add_filter 'app/uploaders'
  add_filter 'app/controllers'
  add_filter 'app/behaviours'

  add_group 'Models', 'app/models'
  add_group 'Modules', 'app/modules'
  add_group 'Helpers', 'app/helpers'
  add_group 'Mailers', 'app/mailers'
end

SimpleCov.adapters.define 'cucumber' do
  add_filter 'test/'
  add_filter 'config/'
  add_filter 'spec/'
  add_filter 'features/'
  add_filter 'vendor/'
  add_filter 'lib/'

  add_group 'Models', 'app/models'
  add_group 'Modules', 'app/modules'
  add_group 'Mailers', 'app/mailers'
  add_group 'Helpers', 'app/helpers'
  add_group 'Uploaders', 'app/uploaders'
  add_group 'Controllers', 'app/controllers'
  add_group 'Behaviours', 'app/behaviours'
end

Added the following for Rspec to spec/spec_helper.rb:

require 'simplecov'
require File.dirname(__FILE__) + '/../lib/simplecov_custom_adapters'
SimpleCov.start 'rspec'

And the following for Cucumber to the beginning of features/support/env.rb:

require 'simplecov'
require File.dirname(__FILE__) + '/../../lib/simplecov_custom_adapters'
SimpleCov.start 'cucumber'

This throws (at least with Rails 3.0.1 and Ruby 1.9.2) the following error:

/path_to/gems/activesupport-3.0.1/lib/active_support/core_ext/float/rounding.rb:16:in 'round': NaN (FloatDomainError)
from /path_to/gems/activesupport-3.0.1/lib/active_support/core_ext/float/rounding.rb:16:in 'round'
from /path_to/gems/activesupport-3.0.1/lib/active_support/core_ext/float/rounding.rb:14:in 'round'
from (erb):4:in 'formatted_file_list'
from /path_to_ruby/lib/ruby/1.9.1/erb.rb:753:in 'eval'
from /path_to_ruby/lib/ruby/1.9.1/erb.rb:753:in 'result'
from /path_to/gems/simplecov-html-0.3.8/lib/simplecov-html.rb:58:in 'formatted_file_list'
from (erb):31:in 'block (2 levels) in format'
from (erb):30:in 'each'
from (erb):30:in 'block in format'
from /path_to_ruby/lib/ruby/1.9.1/erb.rb:753:in 'eval'
from /path_to_ruby/lib/ruby/1.9.1/erb.rb:753:in 'result'
from /path_to/gems/simplecov-html-0.3.8/lib/simplecov-html.rb:24:in 'block in format'
from /path_to/gems/simplecov-html-0.3.8/lib/simplecov-html.rb:23:in 'open'
from /path_to/gems/simplecov-html-0.3.8/lib/simplecov-html.rb:23:in 'format'
from /path_to/gems/simplecov-0.3.6/lib/simplecov/result.rb:47:in 'format!'
from /path_to/gems/simplecov-0.3.6/lib/simplecov/configuration.rb:119:in 'block in at_exit'
from /path_to/gems/simplecov-0.3.6/lib/simplecov.rb:122:in 'call'
from /path_to/gems/simplecov-0.3.6/lib/simplecov.rb:122:in 'block in '

Using the build-in Adapter 'rails' works fine. So I guess something changed in the output formatter for custom Adapters or I missed some syntax change in SimpleCov.adapters.define.

Thanks for any hint & congrats to a very good looking and useful gem,
Fred

simplecov and textmate

Hi,

I can't seem to get coverage results when running tests through textmate but when running rake spec on the command line it works just fine. I'd like to have coverage generated when I do a full run of the tests but can't seem to see what the problem is. The coverage is running I just don't get any results. This is what I have in my spec_helper.sh

require 'simplecov'
SimpleCov.start

resultset.yml not generated?

Hi,

after installing simplecov and adding the two lines to the top of spec_helper.rb, I get this message after running rake spec:

/Users/sebastian/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/psych/tree_builder.rb:24: stack level too deep (SystemStackError)

rake aborted!

The resultset.yml is empty after rake spec finishes...

Regards,
sewid

no items in the coverage report

After config simplecov in my Gemfile and spec_helper.rb, i ran my specs with rake spec. i got the coverage report in my project, but there is no items in it.

Env: rails3 + ruby 1.9.2-p0

Merging doesn't work with test-unit

Rails 3 application. Simple tests (no rspec, no cucumber).

If I add this to my gemfile :

require 'test-unit', '2.3.0'

then merging doesn't work as unit, functional, and integration test results overwrite each-other.

Thing is, I need this for the performance tests to work.

Two reports?

Hi,

I have seen that when you run rake test with SimpleCov activated there are two SimpleCov messages:

Coverage report generated for Unit Tests to /Users/fguillen/Develop/Rails/SponsorPay/OfferWall/coverage

and

Coverage report generated for Unit Tests, Functional Tests to /Users/fguillen/Develop/Rails/SponsorPay/OfferWall/coverage

Is one of them overwriting the other one? or what is happening?

Thanks

f.

Optional exit code > 0 when coverage percentage goes down

See http://twitter.com/#!/hayafirst/status/28368786944

And my responses:
Check out custom exit behaviour: http://bit.ly/953iOR - could raise an exception after result change. See http://bit.ly/9PSaOt
Basically, cache the result.covered_percent somewhere, then later compare it with previous and cause an exit with status > 0

---

This could be an (optional) built-in functionality.
The SimpleCov.result.covered_percent should be stored in coverage_path/last_coverage_percent, then compared with the current run.

Configure with:

SimpleCov.start(:rails) do
  check_coverage_change 
end

Simplecov with Spork?

Is there anyway to make simplecov work with spork? I can only seem to get test coverage results when I don't have spork running...

https://github.com/timcharper/spork

Coverage report generated every time rake invoked

We've found that running even non-test related rake tasks seem to generate/modify code coverage. Is there any way to combat this?

Example:

task :foo do
  p "foo"
end

! rake foo --trace
(in /Users/ip/Dev/health-teams)
** Invoke foo (first_time)
** Execute foo
"foo"
Coverage report generated for ...

Can't get simplecov working when configured in Rakefile

Hi,

I'm getting 0 coverage when I try to configure simplecov in the Rakefile using rspec 2.6.0 and ruby 1.9.2. It only seems to work in the spec_helper.rb. The reason I'd like it to work in the Rakefile so I can have the option of just running tests without coverage, for example "rake spec" would run just the tests and "rake coverage" would run the tests as well as coverage.

Thanks,
Jay

Simplecov 0.4.2 fails in Ruby 1.9.2-p180

Run via autotest -> rspec, with the following in my spec_helper.rb:

require 'simplecov'

SimpleCov.start do
add_filter "spec"
add_filter "binstubs" #Prevents binstubs folder from appearing in code coverage
merge_timeout 3600
end

root = File.expand_path(File.dirname(File.dirname(FILE)))
$LOAD_PATH.unshift(
File.join(root,'lib'),
File.join(root,'config'),
File.join(root,'lib','models')
)

/Users/aaronstewart/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov/merge_helpers.rb:16:in `+': Hash can't be coerced into Fixnum (TypeError)
from /Users/aaronstewart/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov/merge_helpers.rb:16:in `block in merge_resultset'
from /Users/aaronstewart/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov/merge_helpers.rb:10:in `each'
from /Users/aaronstewart/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov/merge_helpers.rb:10:in `each_with_index'
from /Users/aaronstewart/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov/merge_helpers.rb:10:in `merge_resultset'
from /Users/aaronstewart/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov/merge_helpers.rb:32:in `block in merge_resultset'
from /Users/aaronstewart/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov/merge_helpers.rb:31:in `each'
from /Users/aaronstewart/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov/merge_helpers.rb:31:in `merge_resultset'
from /Users/aaronstewart/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov/result_merger.rb:44:in `block in merged_result'
from /Users/aaronstewart/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov/result_merger.rb:43:in `each'
from /Users/aaronstewart/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov/result_merger.rb:43:in `merged_result'
from /Users/aaronstewart/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov.rb:51:in `result'
from /Users/aaronstewart/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov/configuration.rb:120:in `block in at_exit'
from /Users/aaronstewart/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov.rb:134:in `call'
from /Users/aaronstewart/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov.rb:134:in `block in <top (required)>'

If I run rspec directly (no autotest) the error message changes slightly:

/Users/aaronstewart/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov/merge_helpers.rb:16:in `block in merge_resultset': undefined method `+' for {"str"=>"", :@taguri=>"x-private:null"}:Hash (NoMethodError)
from /Users/aaronstewart/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov/merge_helpers.rb:10:in `each'
from /Users/aaronstewart/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov/merge_helpers.rb:10:in `each_with_index'
from /Users/aaronstewart/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov/merge_helpers.rb:10:in `merge_resultset'
from /Users/aaronstewart/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov/merge_helpers.rb:32:in `block in merge_resultset'
from /Users/aaronstewart/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov/merge_helpers.rb:31:in `each'
from /Users/aaronstewart/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov/merge_helpers.rb:31:in `merge_resultset'
from /Users/aaronstewart/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov/result_merger.rb:44:in `block in merged_result'
from /Users/aaronstewart/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov/result_merger.rb:43:in `each'
from /Users/aaronstewart/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov/result_merger.rb:43:in `merged_result'
from /Users/aaronstewart/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov.rb:51:in `result'
from /Users/aaronstewart/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov/configuration.rb:120:in `block in at_exit'
from /Users/aaronstewart/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov.rb:134:in `call'
from /Users/aaronstewart/.rvm/gems/ruby-1.9.2-p180/gems/simplecov-0.4.2/lib/simplecov.rb:134:in `block in <top (required)>'

Autotest messing with the results

I notice that when I have autotest running, my coverage results fluctuate as autotest runs.

For example, I run autotest for the first time and it runs my full test suite. I view the coverage results, and say the total is 80%. Then I add some new tests for some model, and autotest runs only the test file for that model, and now when I view the coverage results, the percent covered is lower, say 70%, because not all of the tests were run (I'm guessing). If I make autotest run the entire test suite again, now I see the coverage go up, to reflect the new tests I've written.

Is this normal behavior? Is there any way to configure simplecov so that the coverage doesn't go down like that?

Not covering functional tests with shoulda correctly

Using the following code setup:

http://pastie.org/private/dj86cs3ynpdkk6gjj2jo3g

And it would appear that no matter what I do, simplecov just doesn't want to mark that line as covered. I'm not exactly sure why, because I don't know how to cover it any more in my current test, so I thought perhaps simplecov isn't playing well with shoulda (or the other way around if you prefer :) ).

Covered code shown as skipped

I'm using ruby 1.9.2-p180, simplecov gem 0.4.2.

Though, i've written test for my model methods, simple coverage html file shows all of them as missed lines. This has happened only in few of my models and not all.

Removing a source file leads to errors when loading resultset cache

When a source file is deleted from the repo, simplecov will still try to read it's source lines when importing the resultset.yml cache, which obviously will fail.

'coverage' subdir needs to created first before calling any test

This is the error message I get on winXP sp2:

• ruby 1.9.1 (mingw32)
• simplecov (0.3.2)
• simplecov-html (0.3.8)
• cucumber (0.8.5)

c:/ruby/lib/ruby/gems/1.9.1/gems/simplecov-0.3.2/lib/simplecov/result_merger.rb:
57:in initialize': No such file or directory - c:/repositories/calculator/coverage/resultset.yml (Errno::ENOENT) from c:/ruby/lib/ruby/gems/1.9.1/gems/simplecov-0.3.2/lib/simplecov/result_merger.rb:57:inopen'
from c:/ruby/lib/ruby/gems/1.9.1/gems/simplecov-0.3.2/lib/simplecov/result_merger.rb:57:in store_result' from c:/ruby/lib/ruby/gems/1.9.1/gems/simplecov-0.3.2/lib/simplecov.rb:53:inresult'
from c:/ruby/lib/ruby/gems/1.9.1/gems/simplecov-0.3.2/lib/simplecov/configuration.rb:118:in block in at_exit' from c:/ruby/lib/ruby/gems/1.9.1/gems/simplecov-0.3.2/lib/simplecov.rb:119:incall'
from c:/ruby/lib/ruby/gems/1.9.1/gems/simplecov-0.3.2/lib/simplecov.rb:119:in `block in <top (required)>'

Once the folder is created, simplecov works fine.

Christian

Spork doesn't seem to run SimpleCov correctly with RSpec and Rails

tl;dr:

Summary of results:

summary of issues		Spork
		not running	running
SimpleCov	top of Spork.prefork	works as expected	SimpleCov doesn't run
	top of Spork.each_run	always 0 LOC, 0% coverage	SimpleCov doesn't run

---

I'm on an existing Rails project using Mongoid, Devise, RSpec, and Spork (0.9.0.rc5). Our spec/spec_helper.rb looks like this:

require 'spork'

ENV["RAILS_ENV"] ||= 'test'

Spork.prefork do
  require 'simplecov'
  SimpleCov.start 'rails'

  # trap mongoid
  require 'rails/mongoid'
  Spork.trap_class_method(Rails::Mongoid, :load_models)

  # trap devise
  require 'rails/application'
  Spork.trap_method(Rails::Application, :reload_routes!)

  require File.expand_path("../../config/environment", __FILE__)
  require 'rspec/rails'
  Dir[Rails.root.join("spec/support/**/*.rb")].each {|f| require f}

  DatabaseCleaner.strategy = :truncation

  RSpec.configure do |config|
    config.mock_with :rspec

    config.before(:each) do
      DatabaseCleaner.clean
    end
  end
end

Spork.each_run do
  # reload factories
  Factory.definition_file_paths = [
    File.join(Rails.root, 'spec', 'factories')
  ]
  Factory.find_definitions
end

This configuration will run SimpleCov when Spork is not running, and it produces the expected result. When Spork is running, no coverage message is shown.

If I move the require 'simplecov'; SimpleCov.start 'rails' lines to the beginning of the Spork.each_run block, as SimpleCov's readme suggests, then when Spork is not running, I get this:

Coverage report generated for RSpec to /home/johnf/projects/<elided>/coverage. 0 / 0 LOC (0.0%) covered.

If Spork is running, then instead I get no coverage message, as before.

doesn't count files under lib

I have this rails project in which I try simplecov. I had 100% coverage. then I created a dummy class in lib, and put a dummy method into it. ran spec again, the coverage was still 100%. Now I moved the same file into app/model. the coverage changed to 98%. Why didn't the files under lib get counted?

Is it possible to add more than one formatter?

In case you want to use the: SimpleCov::Formatter::HTMLFormatter for humans and another formatter for machines.. SimpleCov::Formatter::CSVFormatter for example.

So the SimpleCov process will generate both formats.

Coverage reports are incomplete

Just started using simplecov and as rcov replacement after upgrading our Rails 3 project to Ruby 1.9.2. A coverage report is generated after rake test:units but it only covers several (not all) classes under /lib and none under /models. Running a full rake added one lib class to the report and that's it, still no model or controller coverage.

Coverage percentages are different

When I run my test suite (with rake which runs rspec, then cucumber, then stats). I added
require 'simplecov'
SimpleCov.start 'rails'
to the very top of the spec_helper file. The output after rspecs are run show:
75.92% covered.
Then after cucumbers and stats run I see:
75.92% covered.
But in the HTML index page, it says
All Files (85.42%)
If I open console right after running the test suite, I try:
>> require 'simplecov'
=> ["SimpleCov"]
>> SimpleCov.result.covered_percent
=> 75.91836734693878

Earlier I tried adding the require 'simplecov'; SimpleCov.start 'rails' to the top of env.rb (for cucumber), but then I got a third, different number AND the coverage percentage in the HTML index file went down after I did that (which would seem incorrect, because merging your code coverage %'s should actually only cause them to go UP, rather than be a simple average).

So what gives? I assume the higher number is correct. I added a final task to my rake task that looks like this:
desc 'check coverage'
task :coverage do
raise "Code coverage too low!" if SimpleCov.result.covered_percent < 85
end
This seems to exit with status 0, so it seems that in that context, the covered_percent is 85. I just wish I knew what exactly was going on, and all the variance in the percentage definitely shakes my confidence in the accuracy of the tool.