yanick / test-some

NAME

Test::Some - test a subset of tests

VERSION

version 0.2.1

SYNOPSIS

use Test::More;

use Test::Some 'foo';

plant tests => 3;

subtest foo => sub { pass };

# will be skipped
subtest bar => sub { fail };

DESCRIPTION

This module allows to run a subset of the 'subtest' tests given in a test file.

The module declaration takes a whitelist of the subtests we want to run. Any subtest that doesn't match any of the whitelist items will be skipped (or potentially bypassed).

The test files don't even need to be modified, as the module can also be invoked from the command-line. E.g.,

$ perl -MTest::Some=foo t/tests.t

If no argument is given to the module declaration, the environment variable TEST_SOME will be used as the defaults. For example, this is equivalent to the example above:

$ export TEST_SOME=foo
$ perl -MTest::Some t/tests.t

Whitelist items

'~'

Tells Test::Some to bypass the non-whitelisted tests instead of skipping them. That makes for a smaller output, but the test file would now fail if it has a plan tests = $n> line (as we'll only report on $n - bypassed tests).

Subtest name

At its most simple, the names of the subtests we want to run can be passed.

# run subtests 'foo' and 'bar'
use Test::Some 'foo', 'bar';

Negation

An item prefixed with a bang (!) is negated.

use Test::Some '!foo';  # run all tests but 'foo'

Note that a subtest is run if it matches any item in the whitelist, so

use Test::Some '!foo', '!bar';

will run all tests as `foo` is not `bar` and vice versa.

Regular expression

A string beginning with a slash (/), or a regular expression object will be considered to be a regular expression to be compared against the subtest name

use Test::Some '/foo';  # only tests with 'foo' in their name

# equivalent to 
use Test::Some qr/foo/;

Tags

Strings prefixed with a colon (:) are considered to be tags.

# run all tests with the 'basic' tag
use Test::Some ':basic';

Tags can be assigned to a subtest by putting them after the coderef. E.g.,

subtest foo, sub { 
    ...     
}, 'tag1', 'tag2';

Test::More's subtest ignore those trailing arguments, so they be put there without breaking backward compatibility. If you want to give more visibility to those tags, you can also do

subtest foo => $_, 'tag1', 'tag2', for sub {
    ...;
};

(that neat trick, incidentally, was pointed out by aristotle)

Code

A coderef can be passed. It'll have the subtest name and its tags passed in as $_ and %_, respectively.

# run tests with tags 'important' *and* 'auth'
use Test::Some sub { 
    $_{important} and $_{auth} 
};

SEE ALSO

* http://techblog.babyl.ca/entry/test-some - introduction blog entry

AUTHOR

Yanick Champoux [email protected]

COPYRIGHT AND LICENSE

This software is copyright (c) 2015 by Yanick Champoux.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.

test-some's Issues

Exodist recommendations

Couple concerns:

1. Loading Test::More for them seems like overkill, maybe just die if it looks like subtest() is not in the callers namespace, with a helpful message saying Test::More needs to be loaded first?

2. Your string eval pretends to be in the callers package, but does not alter the line number or file. If someone used this and Class::Method::Modifiers was not installed then the require error will report inside an eval and be very confusing.

This would be a good way to make the errors in your string eval sane, the comment is not a true comment, it makes perl think it is in that line and file+extra little note

eval <<EOT

line 55 "filename (inside Test::Some)"

use Test::More; use Class::Method::Modifiers; around ...;
EOT

3. Loading Class::Method::Modifier into their namespace for them could also potentially cause problems.

I would suggest simplifying this:

sub import {
my $class = shift;
my $caller = caller;
my $subtest = $caller->can('subtest') or croak "Test::More must be loaded before Test::Some";
no warnings 'redefine';
no strict 'refs';
*{"$caller::subtest"} = sub { Test::Some::subtest( @, $subtest ) };
}

This is shorter, and does nto contaminate their namespace. It also passes in their current subtest subref so that your subtest code can defer to it instead of assuming they just have Test::More's, which means you can wrap subtest from other tools, and support other tools that also wrap subtest.

Run test by line number?

Fennec and Test::Stream implement this functionality, though with less features. In order to make it easier to run specific test blocks in isolation from an editor keybinding we added the ability to run a test by specifying any line number within the subtest codeblock.

There are 2 ways to get the line number data:

1. Use caller to get the line where 'subtest' was called, and use the B module to get the start line, (caller returns the last line) which gives you a rough start and end. You can see this in Fennec here: https://github.com/exodist/Fennec/blob/master/lib/Test/Workflow/Block.pm#L54

2. Use B to iterate all ops in the coderef and get every line number keeping only the first and the last.
   https://github.com/Test-More/Test-Stream/blob/master/lib/Test/Stream/Util.pm#L323

Both work fairly well, but usually need the start and end lines to be adjusted since they do not cover the declaration or opening and closing braces. If you are interested in using the logic I would be happy to extract it for you so that you don't need to duplicate it.

Monkeypatching may pose a couple problems

The module modifies &Test::More::subtest in-place. It does so in what is probably the safest way possible, but it still may be undesired. This will effect scopes that use Test::More without importing Test::Some. This may be desired in many cases, but should probably not be a default.

The module could instead wrap the sub in the importing namespace instead of wrapping the one in Test::More. This would prevent it from effecting unexpected places. It would also play nicer with other modules that do similar things. As it stands this also would cause a problem if I or other Test::More maintainers decided to change things under the hood and not have a 'subtest' function directly, but maybe generate importer-specific subtest functions. (I do not intend to do this, I am just stating possibilities)

Run nth test(s)?

People like to name subtests weird things, but I know I want to run the first one. Perhaps we can use = to specify the nth test. In combination with other filters, could presumably run only the nth matching one(s).

use Test::Some '=1';

use Test::Some '=1,2,6';

use Test::Some '/foo', '=1';

Environment variables for control?

There are 2 other modules which implement similar behavior, Fennec and Test::Stream::Plugin::Spec. These are not as nice as yours (good work!) in that they rely only on test name, or a line number from within the subtest, your tag system blows that away. However needing to specify the parameters as arguments to the module at import is cumbersome.

Fennec uses the FENNEC_TEST environment variable. Test::Stream uses the TS_WORKFLOW variable (cause it effects workflows, not subtests). These have proven to be very useful as they make it easy to bind keys in vim to running the specified test block.