toptal / crystalball

Regression Test Selection library for your RSpec test suite

Crystalball

Crystalball is a Ruby library which implements Regression Test Selection mechanism originally published by Aaron Patterson. Its main purpose is to select a minimal subset of your test suite which should be run to ensure your changes didn't break anything.

Installation

Add this line to your application's Gemfile:

group :test do
  gem 'crystalball'
end

And then execute:

$ bundle

Or install it yourself as:

$ gem install crystalball

Usage

Please see our official documentation.

Versioning

We use semantic versioning for our releases.

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/toptal/crystalball. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct.

License

Crystalball is released under the MIT License.

crystalball's Issues

Better datasource

Redis?

Neo4j (or another graph database)?
Pros:

• Would allow things like levels of proximity between files and affected specs, as well as reduce redundancy of spec file data
  Cons:
• harder to set up and to implement the changes

Syntax error in setup docs (fixed, but not on website)

It looks like a fix from 8 months ago never made it into the built site:
093294c#diff-1a523bd9fa0dbf998008b37579210e12L14

Is there a way to look at diff a different way? Like passing in files? SHA's?

Right now you support getting diff from unstaged changes, what if we want to run all the specs in our ci? We'd need to commit our changes first. So could you provide guidance on where I would change to be able to look at diff by providing a list of files? or maybe by providing SHA's?

[Question] Can I generate a list of tests that need to be performed?

I don't want to run the actual tests using bundle exec crystalball
And where the tests are running, there is not git sandbox
Instead I'd like to run a command like bundle exec crystalball predict and then get a list of source files or rspec tests that need can be used to run bundle exec rspec ...

Is this possible currently?

New Release

Hi I was wondering if there could be a release published that includes #126

The current version 0.7.0 doesn't have that latest commit.

Thank you!

Getting an exception when there's an execution map without any example groups

We're parallelizing our tests on CI and generating multiple execution maps.

Sometimes we don't actually run any tests on a specific worker, but an execution map is still generated without any example groups.

We are running into no implicit conversion of nil into Hash.

To reproduce this, in our execution maps folder, one of the yml file has just the metadata on top, but no example groups. For example, just having this in the file:

---
:type: Crystalball::ExecutionMap
:commit: <some SHA>
:timestamp: 1544572800
:version:
---

And then running CrystalBall.

Exception:

TypeError: no implicit conversion of nil into Hash
/vendor/cache/crystalball-bb6941984dbd/lib/crystalball/map_storage/yaml_storage.rb:24:in `merge!'

Parallel_tests

Is crystalball compatible with the parallel_tests gem?

[Documentation] inconsistence with released version

Hi,

While trying your gem, I noticed 2 inconsistences between the documentation and the version of the gem installed by adding gem 'crystalball' in the gemfile :

• The default path is tmp/execution_maps.yml, the s is missing in the doc;
• The map path in the config file is just map_path.

I do know these can be found when looking at fixtures in version 0.5, the only issue for me is that the doc here references the config for master.

Crashes with Crystalball::MapStorage::NoFilesFoundError when following instructions

rspec generates a file at ./execution_map.yml, but the crystalball command looks for tmp/execution_maps by default.

How can I keep these in sync?

Add license to gemspec

As the title says, add license to gemspec and at the next release, it'll show at rubygems page

Collate data from parallel runs

When running crystalball in a parallel test setup, we will have multiple crystalball_data.yml files generated.
Currently we need to write a script to merge all the different crystalball_data_*.yml files from the different runs into one single file.

It will be really helpful if the library itself supports that. For example something like SimpleCov.collate

Build not sync

It fail to parse the execute_map file correctly. The example group separators in file is "--- \n". However the yaml_storage.rb handles it as "---\n".

is it possible to pass something like --exclude-pattern "**/models/*_spec.rb"?

Is it possible to run bundle exec crystalball --exclude-pattern "**/models/*_spec.rb" and exclude running of specified specs? If not, can you advise how to exclude running of certain specs?

Thanks!

NoMethodError: undefined method `merge_base' for nil:NilClass

Trying Crystalball for the very first time.
I executed this command first:
CRYSTALBALL=true bundle exec rspec ./components/risk/spec/some_spec.rb

I got a seemingly-ok ./tmp/crystalball_data.yml

I made a trivial change to one of the ruby files mentioned in the above yml.

Now I'm running this: bundle exec crystalball
And I get:

:~/app# bundle exec crystalball
I, [2023-03-25T23:49:12.315764 #3017]  INFO -- : Crystalball starts to glow...
W, [2023-03-25T23:49:12.325606 #3017]  WARN -- : Maps are outdated!
bundler: failed to load command: crystalball (/usr/local/bundle/bin/crystalball)
NoMethodError: undefined method `merge_base' for nil:NilClass
  /usr/local/bundle/gems/crystalball-0.7.0/lib/crystalball/predictor.rb:35:in `diff'
  /usr/local/bundle/gems/crystalball-0.7.0/lib/crystalball/predictor.rb:30:in `prediction'
  /usr/local/bundle/gems/crystalball-0.7.0/lib/crystalball/rspec/prediction_builder.rb:17:in `prediction'
  /usr/local/bundle/gems/crystalball-0.7.0/lib/crystalball/rspec/runner.rb:72:in `build_prediction'
  /usr/local/bundle/gems/crystalball-0.7.0/lib/crystalball/rspec/runner.rb:19:in `run'
  /usr/local/bundle/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:45:in `invoke'
  /usr/local/bundle/gems/crystalball-0.7.0/bin/crystalball:5:in `<top (required)>'
  /usr/local/bundle/bin/crystalball:23:in `load'
  /usr/local/bundle/bin/crystalball:23:in `<top (required)>'

Additional Info:

 bundle info crystalball
  * crystalball (0.7.0)
        Summary: A library for RSpec regression test selection
        Homepage: https://github.com/toptal/crystalball
        Path: /usr/local/bundle/gems/crystalball-0.7.0

Bug: Psych::DisallowedClass: Tried to load unspecified class: Symbol

Hi there!

Thanks so much for this gem! Very excited to start using it. During setup, I ran into this error after creating the execution map and configuring crystalball to run on Circle CI. I see the error running in a local Docker container as well. Happy to provide any supplemental information you might find useful.

• ruby: 2.6.3
• os: alpine 3.10.1
• gem version: 0.7.0

Stack trace

+ CRYSTALBALL_CONFIG=./config/crystalball_data.yml
+ bundle exec crystalball
Psych::DisallowedClass: Tried to load unspecified class: Symbol
  /usr/local/lib/ruby/2.6.0/psych/class_loader.rb:97:in `find'
  /usr/local/lib/ruby/2.6.0/psych/class_loader.rb:28:in `load'
  /usr/local/lib/ruby/2.6.0/psych/class_loader.rb:39:in `block (2 levels) in <class:ClassLoader>'
  /usr/local/lib/ruby/2.6.0/psych/class_loader.rb:32:in `symbolize'
  /usr/local/lib/ruby/2.6.0/psych/class_loader.rb:82:in `symbolize'
  /usr/local/lib/ruby/2.6.0/psych/scalar_scanner.rb:83:in `tokenize'
  /usr/local/lib/ruby/2.6.0/psych/visitors/to_ruby.rb:60:in `deserialize'
  /usr/local/lib/ruby/2.6.0/psych/visitors/to_ruby.rb:123:in `visit_Psych_Nodes_Scalar'
  /usr/local/lib/ruby/2.6.0/psych/visitors/visitor.rb:16:in `visit'
  /usr/local/lib/ruby/2.6.0/psych/visitors/visitor.rb:6:in `accept'
  /usr/local/lib/ruby/2.6.0/psych/visitors/to_ruby.rb:32:in `accept'
  /usr/local/lib/ruby/2.6.0/psych/visitors/to_ruby.rb:337:in `block in revive_hash'
  /usr/local/lib/ruby/2.6.0/psych/visitors/to_ruby.rb:336:in `each'
  /usr/local/lib/ruby/2.6.0/psych/visitors/to_ruby.rb:336:in `each_slice'
  /usr/local/lib/ruby/2.6.0/psych/visitors/to_ruby.rb:336:in `revive_hash'
  /usr/local/lib/ruby/2.6.0/psych/visitors/to_ruby.rb:162:in `visit_Psych_Nodes_Mapping'
  /usr/local/lib/ruby/2.6.0/psych/visitors/visitor.rb:16:in `visit'
  /usr/local/lib/ruby/2.6.0/psych/visitors/visitor.rb:6:in `accept'
  /usr/local/lib/ruby/2.6.0/psych/visitors/to_ruby.rb:32:in `accept'
  /usr/local/lib/ruby/2.6.0/psych/visitors/to_ruby.rb:311:in `visit_Psych_Nodes_Document'
  /usr/local/lib/ruby/2.6.0/psych/visitors/visitor.rb:16:in `visit'
  /usr/local/lib/ruby/2.6.0/psych/visitors/visitor.rb:6:in `accept'
  /usr/local/lib/ruby/2.6.0/psych/visitors/to_ruby.rb:32:in `accept'
  /usr/local/lib/ruby/2.6.0/psych.rb:360:in `safe_load'
  /usr/local/bundle/gems/crystalball-0.7.0/lib/crystalball/rspec/runner.rb:44:in `config'
  /usr/local/bundle/gems/crystalball-0.7.0/lib/crystalball/rspec/runner.rb:16:in `run'
  /usr/local/bundle/gems/rspec-core-3.8.2/lib/rspec/core/runner.rb:45:in `invoke'
  /usr/local/bundle/gems/crystalball-0.7.0/bin/crystalball:5:in `<top (required)>'
  /usr/local/bundle/bin/crystalball:23:in `load'
  /usr/local/bundle/bin/crystalball:23:in `<top (required)>'
bundler: failed to load command: crystalball (/usr/local/bundle/bin/crystalball)

cc/ @erikdstock

Minitest Runner

I see that it has built in support for RSpec, it would be nice if we could use this with minitest as well.

New release?

Hello guys,

Thanks for your work.

Any plan to release a new version on RubyGem? The last few commits has some fixes that can be useful.

Get prediction as a list in STDOUT

Crystalball needs to be able to list predicted examples without actually running them. It's useful for further load balancing with Knapsack\whatever.

Proposed interface:
crystalball --dry-prediction which outputs a list of predicted examples one per line.

Related to #112

Improve lookup algorithm efficiency

The way lookup works now:

examples_to_run = [ ]

changed_files.each do |file| # n
  examples.each do |ex| # n * n
    examples_to_run << ex if ex.related_files.include?(file) # n * n * n
  end
end

Invert the way data is stored

examples_to_run = changed_files.flat_map(&:related_examples) # O(n)

# For a single file (as in the file that was just
# saved when using --watch)
examples_to_run = changed_file.related_examples # O(1)

Pros:

• Faster lookup
  Cons:
• Uses more memory
• Have larger dump files
• Significant effort to perform this inversion
• Backwards incompatibility

Implement weights "pruning" logic

By this point, we should have an extension point for custom pruning logic.

We need another class that implements weights based on pruning logic.

E.g. with:

# crystalball.yml
weights:
 - ["/app/models/*", 10]
 - ["/app/controllers/*", 5]
 - ["/app/mailers/*", 1]

Crystalball should fill examples limit to closest possible ratio as in weights.

Create flexible map generation filtering logic

Currently we have filtering logic literally hardcoded here: lib/crystalball/map_generator/helpers/path_filter.rb.

We need it to be a bit more flexible so we can specify a list of directories\files to filter out of profiling data.

Common use case: bundle install --standalone installs everything to vendor => falls into the data.

Expected configuration:

Crystalball::MapGenerator.start!(exclude_sources: [/\.gems/, /vendor/], exclude_specs: [/features/]) do
  # ...
end

it should exclude all sources containing .gems or vendor in paths and it should exclude the data for all spec files including features in paths

Watch mode

Add a --watch that would run tests as Crystalball finds the changes in the source code.

• Use guard (or similar)
• Lookup needs to be faster
• Only run fast specs
• Support --fail-fast
• Have a predictor for a single file
• • Right now it only works with the git diff

Is there a way to get the prediction list only? and not run the specs?

Can I do something like bundle exec crystalball --list-only and have the prediction list returned?

is there a way to exclude generating the maps for certain specs?

I'm wondering if it's possible to exclude map generation for spec/features/?

W, [2019-06-13T18:17:54.922276 #30849] WARN -- : Maps are outdated!

What does this mean? I generated the map just before using it... but noted that the version field in my .yml file is blank...maybe that?

Also - on your documentation page, you might change
bundle exec crystalball
to
bundle exec crystalball spec

NoMethodError with empty examples

❯ crystalball            
I, [2020-06-25T13:58:05.413206 #24937]  INFO -- : Crystalball starts to glow...
Traceback (most recent call last):
	15: from /home/alexeymatskevich/.rbenv/versions/2.7.1/bin/crystalball:23:in `<main>'
	14: from /home/alexeymatskevich/.rbenv/versions/2.7.1/bin/crystalball:23:in `load'
	13: from /home/alexeymatskevich/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/crystalball-0.7.0/bin/crystalball:5:in `<top (required)>'
	12: from /home/alexeymatskevich/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rspec-core-3.9.2/lib/rspec/core/runner.rb:45:in `invoke'
	11: from /home/alexeymatskevich/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/crystalball-0.7.0/lib/crystalball/rspec/runner.rb:19:in `run'
	10: from /home/alexeymatskevich/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/crystalball-0.7.0/lib/crystalball/rspec/runner.rb:72:in `build_prediction'
	9: from /home/alexeymatskevich/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/crystalball-0.7.0/lib/crystalball/rspec/prediction_builder.rb:17:in `prediction'
	8: from /home/alexeymatskevich/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/crystalball-0.7.0/lib/crystalball/predictor.rb:30:in `prediction'
	7: from /home/alexeymatskevich/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/crystalball-0.7.0/lib/crystalball/predictor.rb:44:in `predict!'
	6: from /home/alexeymatskevich/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/crystalball-0.7.0/lib/crystalball/predictor.rb:44:in `flat_map'
	5: from /home/alexeymatskevich/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/crystalball-0.7.0/lib/crystalball/predictor.rb:44:in `each'
	4: from /home/alexeymatskevich/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/crystalball-0.7.0/lib/crystalball/predictor.rb:44:in `block in predict!'
	3: from /home/alexeymatskevich/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/crystalball-0.7.0/lib/crystalball/predictor/modified_execution_paths.rb:21:in `call'
	2: from /home/alexeymatskevich/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/crystalball-0.7.0/lib/crystalball/predictor/strategy.rb:12:in `call'
	1: from /home/alexeymatskevich/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/crystalball-0.7.0/lib/crystalball/predictor/modified_execution_paths.rb:22:in `block in call'
/home/alexeymatskevich/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/crystalball-0.7.0/lib/crystalball/predictor/helpers/affected_example_groups_detector.rb:13:in `detect_examples': undefined method `map' for nil:NilClass (NoMethodError)

There are no tests in my application
crystalball_data.yml:

---
:type: Crystalball::ExecutionMap
:commit: 50c5c73bd645e7f7893a8d50235834331874e233
:timestamp: 1593081312
:version: 

Provide an extension point for prediction "pruning" logic

Currently, we have no weights logic at all.
We pick the first X examples to run in case prediction size is bigger than the specified limit.

Sometimes it leads to predictions with critical tests ignored. We need to extract prediction "pruning" logic to separate class so we alter behaviour with different strategies.

Future expected strategies:

• First tests
• Random tests
• Tests with weights

Expected outcome

A class that encapsulates current logic ('first tests') that can be easily substituted with a different class with different logic.

Rework map storage format

Currently large profiling data sets consume a lot of RAM when loaded.

We should minimize memory consumption for our execution maps. So we should group shared "affected files" by context where possible.
E.g. for specs structured like this

RSpec.describe 'test' do
  describe '#one' do
    context "when two" do
      it { is_expected.to eq 'a'  }
      ....
    end
  end
end

We should have a map:

"test_spec.rb[1]": [files affected by all examples in describe "test"],
"test_spec.rb[1:1]": [files affected by all examples in describe "one" without ones listed in "test_spec.rb[1]"],
"test_spec.rb[1:1:1]": [files affected by all examples in context "when two" without ones listed in "test_spec.rb[1]" and "test_spec.rb[1:1]"],
"test_spec.rb[1:1:1:1]": [files affected by example "a" without ones listed in "test_spec.rb[1]", "test_spec.rb[1:1]" and "test_spec.rb[1:1:1]",
...

According to a modelling code done on a large codebase, it should make Crystalball data files ~4 times smaller.

class InvestigateNewApproach
  def initialize(map_path = Pathname('../large-project/tmp/crystalball_data/execution_maps/'))
    @map_path = map_path
  end

  def map
    @map ||= Crystalball::MapStorage::YAMLStorage.load(@map_path)
  end

  def investigate
    result = {}

    file_based_cases.each do |filename, group|
      result[filename] = calc_group_stats(filename, group.to_h)
      unless result[filename][:check]
        puts "ERROR!"
        break
      end
    end

    puts '================================TOTAL=============='
    puts "Now #{result.values.map { |v| v[:old_approach] }.sum}, NEW: #{result.values.map { |v| v[:new_approach] }.sum}"

    result
  end

  def convert!(resulting_map = 'converted_map.yml')
    storage = Crystalball::MapStorage::YAMLStorage.new(Pathname(resulting_map))

    map = Crystalball::ExecutionMap.new(metadata: {commit: nil, version: 10, timestamp: Time.now.to_i})

    require 'ostruct'

    file_based_cases.each do |filename, group|
      files_grouped_by_contexts(group.to_h).each do |context, info|
        e = OpenStruct.new(id: "#{filename}[#{context}]", file_path: filename)
        map << Crystalball::ExampleGroupMap.new(e, info[:files]) if info[:files].size > 0
      end
    end

    storage.dump(map.metadata.to_h)
    storage.dump(map.cases.to_h)
  end

  def file_based_cases
    @file_based_cases ||= map.cases.group_by { |k, _v| example_filename(k) }
  end

  def example_filename(e)
    e.split('[').first
  end

  def get_all_contexts(example_names)
    get_contexts = lambda { |e|
      numbers = e.split('[').last.split(':')[0..-2]
      result = []
      until numbers.empty?
        result << numbers.join(':')
        numbers.pop
      end
      result
    }

    example_names.map { |e| get_contexts.call(e) }.compact.flatten.sort_by { |v| v.split(':').size }
  end

  def parent_context(context)
    context.split(':')[0..-2].join(':')
  end

  def calc_group_stats(filename, group)
    context_files = files_grouped_by_contexts(group)

    example_uniq = {}

    group.each do |key, value|
      example_uniq[key] = value
      get_all_contexts([key]).each do |c|
        example_uniq[key] -= context_files[c][:files]
      end
    end

    values = group.values
    file_result = {
      old_approach: values.map(&:size).sum,
      example_count: values.size,
      contexts: context_files,
      example_uniq: example_uniq
    }
    file_result[:new_approach] = file_result[:old_approach]
    file_result[:contexts].each do |_, value|
      file_result[:new_approach] -= (value[:examples_count] - 1) * value[:files].size
    end

    puts "#{filename}. #{(file_result[:new_approach].to_f * 100 / file_result[:old_approach]).round(2)}           Now: #{file_result[:old_approach]}, Examples: #{file_result[:example_count]}, Example_Uniq: #{file_result[:example_uniq].values.inject(0) { |sum, v| sum + v.size }}, New: #{file_result[:new_approach]}"

    # Check

    check = file_result[:contexts].values.inject(0) do |sum, value|
      sum + value[:files].size * value[:examples_count]
    end + example_uniq.values.inject(0) do |sum, value|
      sum + value.size
    end
    file_result[:check] = extended_check(file_result, group)
    unless file_result[:check]
      puts "Check did not pass!"
      puts "Contexts #{file_result[:contexts].map { |k, v| [k, "#{v[:examples_count]}*#{v[:files].size}"] }.to_h}"
    end

    file_result
  end

  def files_grouped_by_contexts(group)
    contexts = get_all_contexts(group.keys)

    context_files = {}

    # puts "============RAW MAP======="
    # puts group.to_yaml
    # puts "============RAW MAP======="

    all_parent_files = lambda do |context|
      parent_c = parent_context(context)
      if parent_c.empty?
        return context_files[context][:files]
      else
        return context_files[context][:files] + all_parent_files.call(parent_c)
      end
    end

    contexts.each do |c|
      current_context_results = {examples_count: 0}

      group.each do |key, value|
        next unless key.include?("[#{c}")

        current_context_results[:examples_count] += 1

        if current_context_results[:files].nil?
          current_context_results[:files] = value

          parent_c = parent_context(c)
          current_context_results[:files] -= all_parent_files.call(parent_c) unless parent_c.empty?
        else
          current_context_results[:files] = current_context_results[:files] & value
        end
      end

      context_files[c] = current_context_results
    end
    context_files
  end

  def extended_check(file_result, group)
    result = true
    group.each do |e, values|
      sum = file_result[:example_uniq][e].size
      get_all_contexts([e]).each do |con|
        sum += file_result[:contexts][con][:files].size
      end

      if sum != values.size
        puts "#{e} #{sum} != #{values.size}"
        result = false
      end
    end
    result
  end
end

Using Crystalball with Github Actions

I stumbled upon this project while searching a way to speed up the running of the test suites we do as part of our Github Actions workflow.

I played a bit with it locally and it seems to work just fine, and as much as I understand this is not an issue per say, I couldn't find any documentation about how and if it was possible to use Crystalball from inside a Github Actions workflow.

If anyone around here could give me some insight about this, it would be much appreciated.

Proper configuration

First, this project is brilliant, thank you. Our CI will hopefully benefit.

If I wanted to modify/augment the prediction approach, should I just tweak standard_prediction_builder.rb in lib/crystalball/rspec?

I'm not entirely clear how to point crystalball to a custom prediction builder class.

Incremental mode

On a large test suite that takes a long time to run, adding new files especially by refactoring existing functionality but also just new models and tests thwarts the benefits of crystalball, but there is no information linking the new files/tests to the rest of the system. An incremental mode for crystalball would allow us to exercise the new tests and new files and add that information to the map file without taking much more time and resources to do approximately the same thing. Or is there a way to make crystalball do this already?