GithubHelp home page GithubHelp logo

minitest-handler-cookbook's Introduction

minitest-handler cookbook

Build Status

License: Apache 2.0
Copyright: 2012 Opscode, Inc.
Author: Bryan McLellan [email protected]
Author: Bryan W. Berry [email protected]

Description

This cookbook utilizes the minitest-chef-handler project to facilitate cookbook testing. By default, minitest-handler will collect all the tests in your cookbook_path and run them.

minitest-chef-handler project: https://github.com/calavera/minitest-chef-handler
stable minitest-handler cookbook: http://community.opscode.com/cookbooks/minitest-handler
minitest-handler cookbook development: https://github.com/btm/minitest-handler-cookbook

Note: Version 0.1.8 deprecated use of files/default/tests/minitest/*_test.rb and the location of support files. Test files should now be located in files/default/test/*_test.rb

Note: Version 0.1.0 added a change that breaks backward compatibility. The minitest-handler now only loads
test files named "_test.rb" rather than all test files in the path files/default/test/*_test.rb

If you have any helper libraries, they should match files/default/test/*helper*.rb

Attributes

  • node[:minitest][:gem_version] - The version of the minitest gem to install and use.
    • Default: 3.0.1
  • node[:minitest][:chef_handler_gem_version] - The version of the minitest-chef-handler gem to install and use.
    • Default: 1.0.3
  • node[:minitest][:chef_handler_gem_source] - Alternate remote source for the minitest-chef-handler gem if not using RubyGems.org.(Ex http://mysource.com/gems')
    • Default: RubyGems.org
  • node[:minitest][:ci_reporter_gem_version] - The version of the ci_reporter gem to install and use.
    • Default: 1.9.2
  • node[:minitest][:path] - Location to store and find test files.
    • Default: /var/chef/minitest
  • node[:minitest][:recipes] - The names of all recipes included during the chef run, whether by insertion to the run_list, inclusion through a role, or added with include_recipe. If you only want tests for select recipes to run, override this value with the names of the recipes that you want tested.
    • Default: []
  • node[:minitest][:filter] - Filter test names on a pattern (regex)
    • Default: nil
    • Example: /apache2/ could be used to only run tests for recipes starting with apache2
  • node[:minitest][:seed] - Set random seed
    • Default: nil
  • node[:minitest][:ci_reports] - Path to write out the result of each test in a JUnit-compatible XML file, parseable by many CI platforms
    • Default: nil
  • node[:minitest][:tests] - Test files to run.
    • Default to **/*_test.rb
  • node[:minitest][:verbose] - Display verbose output
    • Default: true

Usage

  • Add recipe[minitest-handler] somewhere on your run_list, preferably last
  • Place tests in files/default/test/ with the name your-recipe-name_test.rb (default recipe is named default_test.rb)
  • Put any helper functions you have in files/default/test/spec_helper.rb but minitest-handler will ensure that you have access to any file that matches the glob files/test/*helper.rb

Minitest

Examples

Traditional minitest

class TestApache2 < MiniTest::Chef::TestCase
  def test_that_the_package_installed
    case node[:platform]
    when "ubuntu","debian"
      assert system('apt-cache policy apache2 | grep Installed | grep -v none')
    end
  end

  def test_that_the_service_is_running
    assert system('/etc/init.d/apache2 status')
  end

  def test_that_the_service_is_enabled
    assert File.exists?(Dir.glob("/etc/rc5.d/S*apache2").first)
  end
end

Using minitest/spec

require 'minitest/spec'

describe_recipe 'ark::test' do

  it "installed the unzip package" do
    package("unzip").must_be_installed
  end

  it "dumps the correct files into place with correct owner and group" do
    file("/usr/local/foo_dump/foo1.txt").must_have(:owner, "foobarbaz").and(:group, "foobarbaz")
  end

 end

For more detailed examples, see here

Testing this cookbook

This cookbook currently uses test-kitchen along with the kitchen-vagrant.

All test are currently written using BATS, which is essentially bash. Using BATS was done so that minitest-handler nor minitest-chef-handler were used to test itself. For more examples of bats than are in this cookbook, see the chef-rvm, chef-ruby_build, and chef-rbenv cookbooks.

NOTE A known limitation of using BATS is that the cookbook is not currently tested on Windows machines. See #45 for more of the background on this.

Releasing

This cookbook uses an 'even number' release strategy. The version number in master will always be an odd number indicating development, and an even number will be used when an official build is released.

Come release time here is the checklist:

  • Ensure the metadata.rb reflects the proper even numbered release
  • Ensure there is a dated change log entry in CHANGELOG.md
  • Commit all the changes
  • Use stove to release (bundle exec stove)
  • Bump the version in metadata.rb to the next patch level odd number

Contributors

minitest-handler-cookbook's People

Contributors

b-dean avatar bryanwb avatar btm avatar cgriego avatar dafyddcrosby avatar dpetzel avatar grosser avatar hectcastro avatar jeanmertz avatar jimhopp-lookout avatar jonathantron avatar josegonzalez avatar juanje avatar matt-richardson avatar mfischer-zd avatar nathenharvey avatar reijop avatar stevendanna avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

minitest-handler-cookbook's Issues

Graphing Output

I wish to graph the output of minitest handler in graphite (also using a handler). I've scanned the code but can't seem to get my head around how i would obtain the test summary data from within another handler?

One option i'm looking at is using the ci_reports to output to the files and then have them parsed by my graphite plugin but it feels convoluted...

test-kitchen run fails with minitest-handler-cookbook v1.0.1 but not 0.2.1

Honestly I haven't dug deep enough to determine if this is a minitest-handler-cookbook bug or a test-kitchen bug, so apologies in advance if this is the wrong place to submit this.

A test-kitchen run that uses minitest-handler-cookbook v0.2.1 works; v1.0.1 ends as such:

================================================================================       
Error executing action `run` on resource 'ruby_block[load tests]'       
================================================================================       

NoMethodError       
-------------       
undefined method `scratch_dir' for Chef::Resource::RubyBlock       


Cookbook Trace:       
---------------       
/tmp/kitchen-chef-zero/cache/cookbooks/minitest-handler/libraries/test_loader.rb:64:in `block (2 levels) in load_tests'       
/tmp/kitchen-chef-zero/cache/cookbooks/minitest-handler/libraries/test_loader.rb:55:in `each'       
/tmp/kitchen-chef-zero/cache/cookbooks/minitest-handler/libraries/test_loader.rb:55:in `block in load_tests'       
/tmp/kitchen-chef-zero/cache/cookbooks/minitest-handler/libraries/test_loader.rb:24:in `each'       
/tmp/kitchen-chef-zero/cache/cookbooks/minitest-handler/libraries/test_loader.rb:24:in `load_tests'       
/tmp/kitchen-chef-zero/cache/cookbooks/minitest-handler/recipes/default.rb:48:in `block (2 levels) in from_file'       


Resource Declaration:       
---------------------       
# In /tmp/kitchen-chef-zero/cache/cookbooks/minitest-handler/recipes/default.rb       

 45: ruby_block "load tests" do       
 46:   block do       
 47:     # Leverage the library code to load the test files       
 48:     load_tests()       
 49:   end       
 50: end       

Compiled Resource:       
------------------       
# Declared in /tmp/kitchen-chef-zero/cache/cookbooks/minitest-handler/recipes/default.rb:45:in `from_file'       

ruby_block("load tests") do       
  action "run"       
  retries 0       
  retry_delay 2       
  block_name "load tests"       
  cookbook_name "minitest-handler"       
  recipe_name "default"       
  block #<Proc:0x00000002531680@/tmp/kitchen-chef-zero/cache/cookbooks/minitest-handler/recipes/default.rb:46>       
end

[2013-10-27T21:50:24+00:00] INFO: Running queued delayed notifications before re-raising exception       
[2013-10-27T21:50:24+00:00] ERROR: Running exception handlers       
[2013-10-27T21:50:24+00:00] ERROR: Exception handlers complete       
[2013-10-27T21:50:24+00:00] FATAL: Stacktrace dumped to /tmp/kitchen-chef-zero/cache/chef-stacktrace.out       
Chef Client failed. 5 resources updated       
[2013-10-27T21:50:24+00:00] INFO: Shutting down Chef Zero server       
[2013-10-27T21:50:24+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)       
[2013-10-27T21:50:24+00:00] INFO: Shutting down Chef Zero server       
[2013-10-27T21:50:24+00:00] ERROR: Server did not stop within 5 seconds. Killing...       
>>>>>> Converge failed on instance <default-ubuntu-1204>.
>>>>>> Please see .kitchen/logs/default-ubuntu-1204.log for more details
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: SSH exited (1) for command: [sudo -E /opt/chef/embedded/bin/ruby /tmp/kitchen-chef-zero/chef-client-zero.rb --config /tmp/kitchen-chef-zero/client.rb --json-attributes /tmp/kitchen-chef-zero/dna.json --log_level info]
>>>>>> ----------------------

No longer works with chef 12

It looks like chef client has removed the rest-client gem by default. So when minitest runs the below error is seen:

Running handlers:       
[2015-05-06T14:15:31+00:00] INFO: Running report handlers       
[2015-05-06T14:15:31+00:00] ERROR: Report handler MiniTest::Chef::Handler raised #<LoadError: cannot load such file -- rest_client>       

FATAL: LoadError: cannot load such file -- minitest-chef-handler

I get the following error when running the chef-logstash cookbook in Vagrant:

Cross-reference: lusis/chef-logstash#133

2013-06-29T14:43:06+02:00] INFO: Processing gem_package[minitest-chef-handler] action install (minitest-handler::default line 7)
[2013-06-29T14:43:06+02:00] DEBUG: gem_package[minitest-chef-handler] detected omnibus installation in /opt/chef/embedded/bin
[2013-06-29T14:43:06+02:00] DEBUG: gem_package[minitest-chef-handler] searching for 'gem' binary in path: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/opt/vagrant_ruby/bin:/opt/chef/embedded/bin:/opt/chef/embedded/bin
[2013-06-29T14:43:06+02:00] DEBUG: gem_package[minitest-chef-handler] using gem '/opt/vagrant_ruby/bin/gem'
[2013-06-29T14:43:06+02:00] DEBUG: gem_package[minitest-chef-handler] found installed gem minitest-chef-handler version 1.0.1 matching minitest-chef-handler (>= 0)
[2013-06-29T14:43:06+02:00] DEBUG: gem_package[minitest-chef-handler] is already installed - nothing to do
[2013-06-29T14:43:06+02:00] DEBUG: gem_package[minitest-chef-handler] resetting gem environment to default
[2013-06-29T14:43:06+02:00] DEBUG: filtered backtrace of compile error: /etc/vagrant-chef/chef-solo-1/cookbooks/minitest-handler/recipes/default.rb:14:in from_file' [2013-06-29T14:43:06+02:00] DEBUG: filtered backtrace of compile error: /etc/vagrant-chef/chef-solo-1/cookbooks/minitest-handler/recipes/default.rb:14:infrom_file'
[2013-06-29T14:43:06+02:00] DEBUG: backtrace entry for compile error: '/etc/vagrant-chef/chef-solo-1/cookbooks/minitest-handler/recipes/default.rb:14:in `from_file''
[2013-06-29T14:43:06+02:00] DEBUG: Line number of compile error: '14'

Recipe Compile Error in /etc/vagrant-chef/chef-solo-1/cookbooks/minitest-handler/recipes/default.rb

LoadError


cannot load such file -- minitest-chef-handler

Cookbook Trace:

/etc/vagrant-chef/chef-solo-1/cookbooks/minitest-handler/recipes/default.rb:14:in `from_file'

Relevant File Content:

/etc/vagrant-chef/chef-solo-1/cookbooks/minitest-handler/recipes/default.rb:

  7:  gem_package "minitest-chef-handler" do
  8:    action :nothing
  9:  end.run_action(:install)
 10:
 11:  Gem.clear_paths
 12:  # Ensure minitest gem is utilized
 13:  gem "minitest"
 14>> require "minitest-chef-handler"
 15:
 16:  recipes = node['recipes']
 17:  if recipes.empty? and Chef::Config[:solo]
 18:    #If you have roles listed in your run list they are NOT expanded
 19:    recipes = node.run_list.map {|item| item.name if item.type == :recipe }
 20:  end
 21:
 22:  # Directory to store cookbook tests
 23:  directory "minitest test location" do

[2013-06-29T14:43:06+02:00] DEBUG: Re-raising exception: LoadError - cannot load such file -- minitest-chef-handler
/opt/chef/embedded/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in require' /opt/chef/embedded/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:inrequire'
/etc/vagrant-chef/chef-solo-1/cookbooks/minitest-handler/recipes/default.rb:14:in from_file' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/mixin/from_file.rb:30:ininstance_eval'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/mixin/from_file.rb:30:in from_file' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/cookbook_version.rb:346:inload_recipe'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/run_context.rb:151:in load_recipe' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/run_context/cookbook_compiler.rb:139:inblock in compile_recipes'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/run_context/cookbook_compiler.rb:137:in each' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/run_context/cookbook_compiler.rb:137:incompile_recipes'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/run_context/cookbook_compiler.rb:74:in compile' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/run_context.rb:86:inload'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/client.rb:224:in setup_run_context' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/client.rb:467:indo_run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/client.rb:200:in run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application.rb:190:inrun_chef_client'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application/solo.rb:239:in block in run_application' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application/solo.rb:231:inloop'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application/solo.rb:231:in run_application' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application.rb:73:inrun'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/bin/chef-solo:25:in <top (required)>' /usr/bin/chef-solo:23:inload'
/usr/bin/chef-solo:23:in <main>' [2013-06-29T14:43:06+02:00] ERROR: Running exception handlers [2013-06-29T14:43:06+02:00] ERROR: Exception handlers complete [2013-06-29T14:43:06+02:00] FATAL: Stacktrace dumped to /etc/vagrant-chef/chef-stacktrace.out [2013-06-29T14:43:06+02:00] DEBUG: LoadError: cannot load such file -- minitest-chef-handler /opt/chef/embedded/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:inrequire'
/opt/chef/embedded/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in require' /etc/vagrant-chef/chef-solo-1/cookbooks/minitest-handler/recipes/default.rb:14:infrom_file'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/mixin/from_file.rb:30:in instance_eval' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/mixin/from_file.rb:30:infrom_file'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/cookbook_version.rb:346:in load_recipe' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/run_context.rb:151:inload_recipe'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/run_context/cookbook_compiler.rb:139:in block in compile_recipes' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/run_context/cookbook_compiler.rb:137:ineach'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/run_context/cookbook_compiler.rb:137:in compile_recipes' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/run_context/cookbook_compiler.rb:74:incompile'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/run_context.rb:86:in load' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/client.rb:224:insetup_run_context'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/client.rb:467:in do_run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/client.rb:200:inrun'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application.rb:190:in run_chef_client' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application/solo.rb:239:inblock in run_application'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application/solo.rb:231:in loop' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application/solo.rb:231:inrun_application'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application.rb:73:in run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/bin/chef-solo:25:in<top (required)>'
/usr/bin/chef-solo:23:in load' /usr/bin/chef-solo:23:in

'
[2013-06-29T14:43:06+02:00] FATAL: LoadError: cannot load such file -- minitest-chef-handler

Tests don't run from chef-client

I cant seem to get minitest to run tests when i run it from chef-client. I have placed all the tests in files/default/test in my cookbook. When I run through vagrant and chef-solo, the tests run as I would expect. When running from chef-client though, the tests don't run (0 tests, 0 assertions, 0 failures, 0 errors, 0 skips) .

I receive warnings like the following:

WARN: A test file was detected at map_test.rb in the files directory of the windows_base cookbook. As of version 1.2.0 of the minitest-handler-cookbook storing test files in this location is not supported. If you would like these tests executed you should move them into files/default/test/
WARN: A test file was detected at assetmanager_test.rb in the files directory of the windows_base cookbook. As of version 1.2.0 of the minitest-handler-cookbook storing test files in this location is not supported. If you would like these tests executed you should move them into files/default/test/

FYI, these are windows 2008r2 nodes, using minitest-handler cookbook 1.3.0, chef-client 11.12.8.

js

Minitest performance - Gem files

Minitest gem files are downloaded and deployed upon each chef-client run. This adds a lot of time to our runs. Would it be possible to make it cache the gems?

Support specifying minitest-chef-handler gem version

If you've been using this cookbook for a period of time you might end up with nodes that have different versions of the minitest-chef-handler gem installed on nodes.

You might then write a minitest using a newly introduced assertion, which will work on some nodes, but fail on others. It would be nice if you could set a node attribute that would be used to determine (and enforce) a standardized version of the gem across all nodes.

TypeError: no implicit conversion from nil to integer

When running with Chef 0.10.12, and minitest-handler version 0.0.10, with a very simple test via a private chef server, I get this on a CentOS 6.2 server:

Run options: -v --seed 49948

# Running tests:

recipe::system_test::default::system files ok#test_0001_passwd_file_has_exists_with_correct_perms = 
0.00 s = E


Finished tests in 0.003376s, 296.2085 tests/s, 296.2085 assertions/s.

  1) Error:
test_0001_passwd_file_has_exists_with_correct_perms(recipe::system_test::default::system files ok):
TypeError: no implicit conversion from nil to integer
    /var/chef/minitest/autotesting/system_test.rb:14:in `test_0001_passwd_file_has_exists_with_correct_perms'
    /usr/lib64/ruby/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/handler.rb:218:in `run_report_unsafe'
    /usr/lib64/ruby/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/handler.rb:206:in `run_report_safely'
    /usr/lib64/ruby/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/handler.rb:91:in `run_report_handlers'
    /usr/lib64/ruby/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/handler.rb:90:in `each'
    /usr/lib64/ruby/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/handler.rb:90:in `run_report_handlers'
    /usr/lib64/ruby/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/handler.rb:99
    /usr/lib64/ruby/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/client.rb:104:in `call'
    /usr/lib64/ruby/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/client.rb:104:in `run_completed_successfully'
    /usr/lib64/ruby/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/client.rb:103:in `each'
    /usr/lib64/ruby/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/client.rb:103:in `run_completed_successfully'
    /usr/lib64/ruby/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/client.rb:168:in `run'
    /usr/lib64/ruby/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/application/client.rb:254:in `run_application'
    /usr/lib64/ruby/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/application/client.rb:241:in `loop'
    /usr/lib64/ruby/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/application/client.rb:241:in `run_application'
    /usr/lib64/ruby/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/application.rb:70:in `run'
    /usr/lib64/ruby/gems/1.8/gems/chef-10.12.0/bin/chef-client:26
    /usr/bin/chef-client:23:in `load'
    /usr/bin/chef-client:23

1 tests, 1 assertions, 0 failures, 1 errors, 0 skips

The run_list looks like:

  "run_list": [
    "recipe[minitest-handler]",
    "recipe[autotesting::default]"
  ]

And the autotesting cookbook was crated with 'knife cookbook create autotesting', followed by a 'mkdir -p site-cookbooks/autotesting/files/default/tests/minitest/'

I then added a new 'system_test.rb' file to the minitest directory with the following contents:

require "rubygems"
require 'minitest/spec'
require 'minitest/autorun'

describe_recipe 'system_test::default' do

    include MiniTest::Chef::Assertions
    include MiniTest::Chef::Context
    include MiniTest::Chef::Resources

    describe "system files ok" do
        it "passwd file has exists with correct perms" do
            file("/etc/passwd").must_exist.with(:owner, "root")
        end
    end
end

However the test will pass if I strip off the .with(:owner, "root") from the file resource.

Tests not running, and what looks like won't get copied into the correct location.

As i wrote on 8fa681a comment, it seems that the merge broke the cookbook. I down graded to 0.1.7 and it works great.

[2013-05-07T22:54:01-05:00] INFO: Processing cookbook_file[/var/chef/minitest/minitest-handler-cookbook/support/helpers.rb] action create (dynamically defined)
[2013-05-07T22:54:01-05:00] INFO: cookbook_file[/var/chef/minitest/minitest-handler-cookbook/support/helpers.rb] mode changed to 644
[2013-05-07T22:54:01-05:00] INFO: cookbook_file[/var/chef/minitest/minitest-handler-cookbook/support/helpers.rb] created file /var/chef/minitest/minitest-handler-cookbook/support/helpers.rb
[2013-05-07T22:54:01-05:00] INFO: Enabling minitest-chef-handler as a report handler
[2013-05-07T22:54:01-05:00] INFO: ruby_block[load tests] called
[2013-05-07T22:54:01-05:00] INFO: template[nginx.conf] sending reload action to service[nginx] (delayed)
[2013-05-07T22:54:01-05:00] INFO: Processing service[nginx] action reload (nginx::default line 43)
[2013-05-07T22:54:02-05:00] INFO: service[nginx] reloaded
[2013-05-07T22:54:02-05:00] INFO: Chef Run complete in 29.168417429 seconds
[2013-05-07T22:54:02-05:00] INFO: Running report handlers
Run options: -v --seed 26761


# Running tests:


minitesthandler::default#test_0001_runs no tests =
0.00 s =
.

nginx::default#test_0001_installs nginx =
0.27 s =
.

nginx::default#test_0002_runs a service named nginx =
0.13 s =
.



Finished tests in 0.406986s, 7.3713 tests/s, 4.9142 assertions/s.


3 tests, 2 assertions, 0 failures, 0 errors, 0 skips
[2013-05-07T22:54:02-05:00] INFO: Report handlers complete

no tests are being detected

I am using this cookbook with test-kitchen v1 with Vagrant and when it checks to see if any tests exist it cannot find them.

The issue is caused by Chef::Config[:cookbook_path] returning an array instead of a string which creates an invalid path. I have forked and added a potential fix but I wanted to know if anyone else has been experiencing this or if I am just not using this cookbook correctly.

minitest-handler deleting its own tests

After upgrading to 0.1.6, my tests are no longer being run, presumably because they're being deleted!

[2013-03-27T11:39:40+00:00] INFO: *** Chef 11.2.0 ***
[2013-03-27T11:39:40+00:00] INFO: Setting the run_list to ["minitest-handler", ...] from JSON
[2013-03-27T11:39:41+00:00] INFO: Run List is [recipe[minitest-handler], ...]
[2013-03-27T11:39:41+00:00] INFO: Run List expands to [minitest-handler, ...]

[2013-03-27T11:39:42+00:00] INFO: Processing ruby_block[delete tests from old cookbooks] action run (minitest-handler::default line 36)
[2013-03-27T11:39:42+00:00] INFO: Cookbook minitest-handler no longer in run list, remove minitest tests
[2013-03-27T11:39:42+00:00] INFO: ruby_block[delete tests from old cookbooks] called

[Possibly PEBCAK] Minitest-handler not copying all tests to local filesystem

I have downloaded the OpsCode community cookbook to manage Chef Client. In that cookbook are some minitest tests, in the following hierarchy:

[bhatfield@bhatfield-mac chef-client]$ tree files/
files/
└── [Oct  3 12:09]  default/
    └── [Oct  3 12:09]  tests/
        └── [Oct  3 12:09]  minitest/
            ├── [Oct  3 12:09]  config_test.rb
            ├── [Oct  3 12:09]  cron_test.rb
            ├── [Oct  3 12:09]  delete_validation_test.rb
            ├── [Oct  3 12:09]  helpers.rb
            └── [Oct  3 12:09]  service_test.rb

Within the given tests, there is a cross dependency on helpers.rb, like so:

[bhatfield@bhatfield-mac chef-client]$ cat files/default/tests/minitest/service_test.rb 
require File.expand_path('../helpers', __FILE__)

describe 'chef-client::service' do
  include Helpers::ChefClient
  it "starts the chef-client service" do
    service("chef-client").must_be_running
  end
end

However, when running chef-client with minitest handler enabled, I get the following error:

ERROR: Report handler MiniTest::Chef::Handler raised #<LoadError: cannot load such file -- /var/chef/minitest/chef-client/helpers>

This was confusing to me, so I troubleshooted the 'require' code above, and determined that the file was not actually there, which is confirmed on the host:

 vagrant@vagrant-ubuntu-oneric:/var/chef/minitest$ tree chef-client/
 chef-client/
 ├── config_test.rb
 ├── delete_validation_test.rb
 └── service_test.rb

I am having a hard time understanding how I am getting into this state. What can I do to provide more troubleshooting data? Is there something I am not understanding about minitest-handler?

Only run tests for recipes in my run_list

Hi,

It seems mintest-handler-cookbook doesn't clean up test files on the test machine each time meaning if I changed my run_list to not include a recipe that had a test and run chef-client again it will still run those tests. The only way to start clean is to build a new box or go delete the test files by hand.

Just wanted to add this as an issue, it's not really bad but something to be aware of. I may fork it and put that change in if I have time.

Thanks,
Jay

Error executing action `run` on resource 'ruby_block[load_tests_and_register_handler]'

Chef Version 13.0.113

    Error executing action `run` on resource 'ruby_block[load_tests_and_register_handler]'
    ================================================================================

    NoMethodError
    -------------
    undefined method `each' for nil:NilClass

    Cookbook Trace:
    ---------------
    /tmp/kitchen/cache/cookbooks/minitest-handler/libraries/test_loader.rb:235:in `cookbook_file_paths'
    /tmp/kitchen/cache/cookbooks/minitest-handler/libraries/test_loader.rb:247:in `cookbook_file_names'
    /tmp/kitchen/cache/cookbooks/minitest-handler/libraries/test_loader.rb:220:in `all_test_files'
    /tmp/kitchen/cache/cookbooks/minitest-handler/libraries/test_loader.rb:206:in `test_files'
    /tmp/kitchen/cache/cookbooks/minitest-handler/libraries/test_loader.rb:35:in `block in load_tests'
    /tmp/kitchen/cache/cookbooks/minitest-handler/libraries/test_loader.rb:10:in `each'
    /tmp/kitchen/cache/cookbooks/minitest-handler/libraries/test_loader.rb:10:in `load_tests'
    /tmp/kitchen/cache/cookbooks/minitest-handler/recipes/default.rb:52:in `block (2 levels) in from_file'

    Resource Declaration:
    ---------------------
    # In /tmp/kitchen/cache/cookbooks/minitest-handler/recipes/default.rb

     50: ruby_block 'load_tests_and_register_handler' do
     51:   block do
     52:     load_tests
     53:     register_handler
     54:   end
     55: end

    Compiled Resource:
    ------------------
    # Declared in /tmp/kitchen/cache/cookbooks/minitest-handler/recipes/default.rb:50:in `from_file'

    ruby_block("load_tests_and_register_handler") do
      action [:run]
      default_guard_interpreter :default
      block_name "load_tests_and_register_handler"
      declared_type :ruby_block
      cookbook_name "minitest-handler"
      recipe_name "default"
      block #<Proc:0x000000041ca968@/tmp/kitchen/cache/cookbooks/minitest-handler/recipes/default.rb:51>
    end

    System Info:
    ------------
    chef_version=13.0.113
    platform=ubuntu
    platform_version=16.04
    ruby=ruby 2.4.1p111 (2017-03-22 revision 58053) [x86_64-linux]
    program_name=chef-client worker: ppid=3195;start=23:06:03;
    executable=/opt/chef/bin/chef-client

[2017-04-10T23:06:07+00:00] INFO: Running queued delayed notifications before re-raising exception
[2017-04-10T23:06:07+00:00] INFO: Running queued delayed notifications before re-raising exception
[2017-04-10T23:06:07+00:00] DEBUG: Re-raising exception: NoMethodError - ruby_block[load_tests_and_register_handler] (minitest-handler::default line 50) had an error: NoMethodError: undefined method `each' for nil:NilClass
/tmp/kitchen/cache/cookbooks/minitest-handler/libraries/test_loader.rb:235:in `cookbook_file_paths'
  /tmp/kitchen/cache/cookbooks/minitest-handler/libraries/test_loader.rb:247:in `cookbook_file_names'
  /tmp/kitchen/cache/cookbooks/minitest-handler/libraries/test_loader.rb:220:in `all_test_files'
  /tmp/kitchen/cache/cookbooks/minitest-handler/libraries/test_loader.rb:206:in `test_files'
  /tmp/kitchen/cache/cookbooks/minitest-handler/libraries/test_loader.rb:35:in `block in load_tests'
  /tmp/kitchen/cache/cookbooks/minitest-handler/libraries/test_loader.rb:10:in `each'
  /tmp/kitchen/cache/cookbooks/minitest-handler/libraries/test_loader.rb:10:in `load_tests'

Need to run the minitest-handler cookbook twice in a row for tests to run

Hey Bryan,

I am now using the latest version of minitest-handler-cookbook 0.0.5 and for some reason I now need to run the chef-client twice in a row in order for tests to report. Not sure what is going on here but was wondering if this is a known issue. I'll try reverting back to 0.0.4 just to be sure it's in 0.0.5. I'll post with my findings.

Thanks,
Jay

workflow

Hey Bryan,

I'm trying to layout a practical workflow that works with mintest-handler-cookbook. I'm thinking of using knife-flow in combination so developers can test their changes locally with vagrant. My only concern is that if they take a TDD approach they will have to constantly push up test files to the development organization and when using knife-flow this would require bumping the patch level version which I guess is okay considering it is our development organization. So my question is how do you feel about this approach with minitest-handler-cookbook. And also is there a way to run the minitests locally without having to push them to the server and using chef-client? It's probably not designed for this but wanted to ask.

Thanks,
Jay

Minitest output is still semi-verbose even if verbose = false

with defaults:

redis#test_0003_creates_bgrewriteaof_cron = 
0.00 s = .
redis#test_0002_starts_redis = 
0.01 s = .
redis#test_0005_creates_redis_user = 
0.01 s = .
redis#test_0001_installs_redis = 
0.02 s = .
redis#test_0004_starts_redis_service = 
0.03 s = .
redis#test_0006_creates_db_dir = 
0.00 s = .

with chef.json = { "minitest" => {"verbose" => false} }


# Running tests:

.
.
.
.
.
.


Finished tests in 0.071687s, 83.6973 tests/s, 153.4450 assertions/s.

undefined method '_run'

Hi,

I'm running the cookbook per instructions and I'm getting "Report Handler MiniTest::Chef::Handler raised NoMethodError: undefined method '_run' when I run chef-client on a node with the run_list suggested in the readme. Any ideas?

Thanks.

Why do this cookbook forces to have a test/support dir for every cookbook?

Hi,
I've added minitest-handler and added a test for one my cookbook.

Now when it runs, it checks for support directory for everyother cookbooks I've.

Doesn't matter how many cookbooks I've, I just want to add test for my cookbook only.

So, how to tell minitest-handler to just run the specs of my cookbook and don't care other cookbooks have tests or not?

.........
Compiled Resource:
------------------
# Declared in /var/cache/chef/cookbooks/minitest-handler/recipes/default.rb:53:in `block in from_file'

cookbook_file("tests-phpmyadmin-privileges") do
  provider Chef::Provider::CookbookFile
  action "create"
  ignore_failure true
  retries 0
  retry_delay 2
  path "/var/chef/minitest/phpmyadmin/privileges_test.rb"
  backup 5
  source "tests/minitest/privileges_test.rb"
  cookbook "phpmyadmin"
  cookbook_name "minitest-handler"
  recipe_name "default"
end

[2012-10-08T09:01:34+00:00] INFO: Processing remote_directory[tests-support-phpmyadmin-privileges] action create (minitest-handler::default line 60)
[2012-10-08T09:01:34+00:00] ERROR: remote_directory[tests-support-phpmyadmin-privileges] (minitest-handler::default line 60) had an error: cookbook phpmyadmin has no directory files/default/tests/minitest/support; ignore_failure is set, continuing

================================================================================
Error executing action `create` on resource 'remote_directory[tests-support-phpmyadmin-privileges]'
================================================================================

Chef::Exceptions::FileNotFound
------------------------------
cookbook phpmyadmin has no directory files/default/tests/minitest/support

Resource Declaration:
---------------------
# In /var/cache/chef/cookbooks/minitest-handler/recipes/default.rb

 60:     remote_directory "tests-support-#{cookbook_name}-#{recipe_name}" do
 61:       source "tests/minitest/support"
 62:       cookbook cookbook_name
 63:       path "#{node['minitest']['path']}/#{cookbook_name}/support"
 64:       recursive true
 65:       ignore_failure true
 66:     end
 67:   end
................

default version of chef handler gem

When running a test with the minitest assertion 'refute_sh' and this cookbook the command fails with this error:

[2014-05-30T20:39:06+00:00] FATAL: RuntimeError: MiniTest failed with 0 failure(s) and 1 error(s).
Error:
recipe::percona-utilities-hp::zabbix_mysql#test_0009_should have a non-empty password for zabbix@localhost MySQL user:
NoMethodError: undefined method refute_sh' for #<#<Class:0x0000000630e098>:0x000000042197e8> /var/chef/minitest/percona-utilities-hp/zabbix_test.rb:48:inblock (2 levels) in <top (required)>'

It seems that refute_sh is an assertion that was added in a later version of the minitest-chef-handler. Changing the version of the gem 'minitest-chef-handler' to 1.0.2 fixes the problem for me. Would it be possible to change the default attribute (default[:minitest][:chef_handler_gem_version]) in this cookbook to be 1.0.2 instead of 1.0.1?

/var/chef/minitest doesn't always update with newest tests

This has bitten me several times. /var/chef/minitest doesn't always load the newests tests. I have found that even if my tests have updated, the actual tests stored in /var/chef/minitest/recipe_name are older. /var/chef/cache/cookbooks/cook_book_name has the newer tests.

Only load filtered tests

The filter for tests is really useful, but it seems that the handler still loads all tests it finds which feels unnecessary. Is there some way to only load tests that pass the filter?

I would submit a PR, but I have no idea what would need changing :P

too short control escape

I had Chef 12 working with Windows 2008 R2 the other week, but when I tried it again today, I receive this error while using test kitchen on two separate cookbooks:

       Recipe: minitest-handler::default
         * chef_gem[ci_reporter] action nothing (skipped due to action :nothing)
         * chef_gem[minitest] action nothing (skipped due to action :nothing)
         * chef_gem[minitest-chef-handler] action nothing (skipped due to action :nothing)
         * directory[delete minitest test location] action delete (up to date)

           - create new directory /var/chef/minitest


           ================================================================================
           Error executing action `run` on resource 'ruby_block[load_tests_and_register_handler]'
           ================================================================================

           RegexpError
           -----------
           too short control escape: /[A-Za-z]?:?C:\Users\CLOUD-~1\AppData\Local\Temp\kitchen\cache\/cookbooks\/icberg7_patch\//

           Resource Declaration:
           ---------------------
           # In C:/Users/CLOUD-~1/AppData/Local/Temp/kitchen/cache/cookbooks/minitest-handler/recipes/default.rb

            47: ruby_block 'load_tests_and_register_handler' do
            48:   block do
            49:     load_tests
            50:     register_handler
            51:   end
            52: end

           Compiled Resource:
           ------------------
           # Declared in C:/Users/CLOUD-~1/AppData/Local/Temp/kitchen/cache/cookbooks/minitest-handler/recipes/default.rb:47:in `from_file'

           ruby_block("load_tests_and_register_handler") do
             action [:run]
             retries 0
             retry_delay 2
             default_guard_interpreter :default
             block_name "load_tests_and_register_handler"
             declared_type :ruby_block
             cookbook_name "minitest-handler"
             recipe_name "default"
             block #<Proc:0x1ef9ae0@C:/Users/CLOUD-~1/AppData/Local/Temp/kitchen/cache/cookbooks/minitest-handler/recipes/default.rb:48>
           end

It looks like only part of the path is being escaped. This is using both chef 12.5.1 and 12.4.3.

Test files not copied from a Private Chef cookbook to chef-client host

We are attempting to upgrade from v0.1.3 to 0.2.1. After the upgrade none of our tests are being executed by minitest-handler. It appears that the issue may have been introduced with [65a3e09] on April 10th.

The previous version used the cookbook_file resource which transfers the test file from a Private Chef cookbook to the minitest directory on the host running chef-client.

      ckbk_f = Chef::Resource::CookbookFile.new("tests-#{cookbook_name}-#{recipe_name}", run_context)
      ckbk_f.source "tests/minitest/#{recipe_name}_test.rb"
      ckbk_f.cookbook cookbook_name
      ckbk_f.path "#{node['minitest']['path']}/#{cookbook_name}/#{recipe_name}_test.rb"

This code was removed and replaced with logic that copies the test file(s) from the Chef cache (on the host running-chef client) to the minitest directory.

      tests = ::Dir["#{Chef::Config[:cookbook_path]}/#{cookbook_name}/files/default/**/_test.rb"]
      unless tests.empty?
        FileUtils.cp tests, "#{node[:minitest][:path]}/#{cookbook_name}/"
      end

We do not see any logic that would transfer the test file from a Private Chef cookbook to the chef-client host. Therefore, no tests are copied to the minitest directory and no tests are executed.

We can provide additonal information as needed to investigate this issue.

FEATURE: Disable flag

would it be possible to add a simple node['minitest']['disabled'] attribute for when I don't want tests to be run?

Thanks!

How to access data_bag items in the test file?

In my test file, I want to access the data_bag_item from the chef-server.

When I put

require 'minitest/spec'
require 'chef'
#
# Cookbook Name:: task_server
# Spec:: default
#
#

describe_recipe 'task_server::default' do

  let(:app) { data_bag_item("apps", "task_server") }
  ..............
end

And running the chef-client, I get the following error:

NoMethodError: undefined methoddata_bag_item' for #<#Class:0xad68f44:0xa032bcc>`

How to acces the data_bag items in the test file?

Problem with over-keen test file detection in libraries/test_loader.rb

Line 188
Recommend

  •    files << test_file if test_file.end_with?("#{recipe_name}_test.rb")
    
  •    files << test_file if test_file.end_with?("/#{recipe_name}_test.rb")
    

Consider two recipes: cookbook::users and cookbook::extra_users
The test files for extra_users match the end_with? test when you are searching for test files for users. This means running tests for recipes that have not been run; errors abound.

The filenames should all be either 'tests/filename.rb' or 'test/minitest/filename.rb' those being the search locations, so the prepended / should never exclude legitimate matches.

automatic test detection

Does the minitest-handler use the automatic test detection?

I'm putting my files under a specs directory but the tests don't seem to be picked up. Do I need to set something so they are picked up?

Thanks,
Jay

error codes

Hi,

I am running minitest-handler and was wondering if tests fail should the report handlers be reporting an error code? I ask because I am using vagrant to provision a machine and I have tests fail but vagrant doesn't report back a failure. Should this happen? Are the test results put anywhere? I need this for a continuous integration environment where the return codes are important.

Thanks,
Jay

unneeded minitest tests get ran on nodes

Hello,

I've noticed that minitest handler copies all tests belonging to a cookbook into the minitests path, this causes a test failure when a minitest is ran for a recipe that is not in the node's runlist.

Any idea around this?

Remove recipe name matching requirement

In def test_files..., you have:

files << test_file if test_file_name == "#{recipe_name}_test.rb"

I have logical test grouping by file, right now, and I don't want to put it all into one giant file for the recipe being tested. Right now, I have tests for a cluster node, tests for a single node... all different ways of converging a single recipe.

Could you at least do a substring match here? Or just don't filter at all? It doesn't hurt anything to copy all the test files, and filter them on runtime, right? Thanks.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.