raku / tap-harness6

A TAP harness for Raku

NAME

TAP

DESCRIPTION

An asynchronous TAP framework written in Raku.

SYNOPSIS

use TAP;
my $harness = TAP::Harness.new(|%args);
$harness.run(@tests);

METHODS

Class Methods

new

my %args = jobs => 1, err  => 'ignore';
my $harness = TAP::Harness.new( |%args );

The constructor returns a new TAP::Harness object. It accepts an optional hash whose allowed keys are:

• volume

  Default value: Normal

  Possible values: Silent ReallyQuiet Quiet Normal Verbose

• jobs

  The maximum number of parallel tests to run.

  Default value: 1

  Possible values: An Int

• timer

  Append run time for each test to output.

  Default value: False

  Possible values: True False

• err

  Error reporting configuration.

  Default value: stderr

  Possible values: stderr ignore merge Supply IO::Handle

  Value	Definition
  stderr	Direct the test's $*ERR to the harness' $*ERR
  ignore	Ignore the test scripts' $*ERR
  merge	Merge the test scripts' $*ERR into their $*OUT
  Supply	Direct the test's $*ERR to a Supply
  IO::Handle	Direct the test's $*ERR to an IO::Handle

• ignore-exit

  If set to True will instruct TAP::Parser to ignore exit and wait for status from test scripts.

  Default value: False

  Possible values: True False

• trap

  Attempt to print summary information if run is interrupted by SIGINT (Ctrl-C).

  Default value: False

  Possible values: True False

• handlers

  Default value: TAP::Harness::SourceHandler::Raku

  Possible values: TAP::Harness::SourceHandler::Raku TAP::Harness::SourceHandler::Exec

  Language	Handler
  Raku	TAP::Harness::SourceHandler::Raku.new
  Perl 5	TAP::Harness::SourceHandler::Exec.new('perl')
  Ruby	TAP::Harness::SourceHandler::Exec.new('ruby')
  Python	TAP::Harness::SourceHandler::Exec.new('python')

Instance Methods

run

$harness.run(@tests);

Accepts an array of @tests to be run. This should generally be the names of test files.

TODO

These features are currently not implemented but are considered desirable:

• Rule based parallel scheduling
• Source Handlers other than ::Raku
• Better documentation

LICENSE

You can use and distribute this module under the terms of the The Artistic License 2.0. See the LICENSE file included in this distribution for complete details.

tap-harness6's Issues

All tests successful but Result: FAILED

Hey :-)
We noticed that prove6 reports a failure even though all tests did pass:

Bailador/Bailador#284
https://travis-ci.org/Bailador/Bailador/builds/314951529

Thank you very much.

Fails with some output

Check out Raku/doc#2850

version 0.3.7 seems broken

Steps to reproduce:

❯ raku -v
Welcome to Rakudo™ v2022.07.
Implementing the Raku® Programming Language v6.d.
Built on MoarVM version 2022.07.

❯ git clone https://github.com/Raku/tap-harness6

❯ cd tap-harness6

❯ git describe --tags --dirty
0.3.7

❯ echo 'use Test; plan 1; pass "ok";' > test.t

❯ raku -I. -MTAP -e 'TAP::Harness.new.run("test.t").result'
test.t ..1/1Tried to get the result of a broken Promise
  in block <unit> at -e line 1

Original exception:
    No such method 'exit' for invocant of type 'Proc'.  Did you mean
    'emit'?
      in method exit-status at /Users/skaji/tap-harness6/lib/TAP.pm (TAP) line 800
      in method result at /Users/skaji/tap-harness6/lib/TAP.pm (TAP) line 892
      in sub reap-finished at /Users/skaji/tap-harness6/lib/TAP.pm (TAP) line 1060
      in block  at /Users/skaji/tap-harness6/lib/TAP.pm (TAP) line 1039
      in block  at /Users/skaji/tap-harness6/lib/TAP.pm (TAP) line 1028

Note that the previous version 0.3.6 works fine:

❯ git checkout 0.3.6

❯ raku -I. -MTAP -e 'TAP::Harness.new.run("test.t").result'
test.t .. ok
All tests successful.
Files=1, Tests=1,  0 wallclock secs
Result: PASS

Spin off grammar to its own module

Which will help leverage it for other harnesses

Complaints about deprecated Proc code

Message received when executing zef:

$ zef install --force-install .
===> Testing [OK] for Raku::Lint:ver<0.1.0>
===> Installing: Raku::Lint:ver<0.1.0>

1 bin/ script [raku-lint] installed to:
/home/tbrowde/.raku/bin
Saw 1 occurrence of deprecated code.
================================================================================
Method status (from Proc) seen at:
  /home/tbrowde/.raku/sources/6FEC28FA0762AC132736C81C67CCDB823E5E83FA (TAP), lines 104,111
Please use exitcode and/or signal methods (status is to be removed in 2022.06) instead.
--------------------------------------------------------------------------------
Please contact the author to have these occurrences of deprecated code
adapted, so that this message will disappear!

The offending lines :

    method wait() {
        $!exit-status.defined ?? $!exit-status.status !! Int;         # <== line 104
    }

    method exit-failed() {
        $!exit-status.defined && $!exit-status.status;                 # <== line 111
    }

How to adjust verbosity?

@Leont would you know how to adjust verbosity of the output?

I see there's this enum, which suggests it's possible, but I'm having trouble finding the user-facing knob to turn to achive these volumes:

enum Formatter::Volume <Silent ReallyQuiet Quiet Normal Verbose>;

Trying to make output Verbose, like prove -v so all the test details get printed.

Add proper job control

Currently, the number of jobs applies to every run individually. This is generally not desirable: one wants to limit how many CPUs are used. Therefor, this limit should be shared between different instances of run.

prove6 --version reports 0.0.1 while this is already 0.0.2

$ prove6 --version
prove6 0.0.1 with TAP::Harness 0.001 on Perl 6

$ zef info TAP::Harness
- Info for: TAP::Harness
- Identity: TAP::Harness:ver('0.0.2'):auth('Leon Timmermans')
...

prove6 cant handle multiple -I

I would be happy if prove6 could actually handle multiple -I to include several libs. I think perl5 prove is able to do so.

diagnostics/comments only accepted in subtests if indented

prove6 only accepts comments/diagnostics in subtests if they are indented correctly, as shown by this small examples:

use Test;

ok(True, "A");
say "# test A";
subtest {
    plan 2;
    is(1, 1, "1");
    say "# test B";
    is(2, 2, "2"); 
}
ok(!False, "B");
done-testing;

My reading of the TAP "spec" is that comments should be ignored independently of their indentation, that is also the behavior of perl5 prove. there is also a practical reason to allow this: it allows diagnostics from somewhere deeper in the code being tested, that should not rely on Test.pm6 itself, and should certainly not need to be aware of the level of subtest it is being called from.

perl5 sample:

use Test::More;

ok(1);
print "# A\n";
subtest try => sub {
    plan tests => 1;
    print "# A\n";
    ok(1);
};
done_testing;

I am happy top provide a patch, but need some steer on the direction to go in. My instinctive reaction would be to remove the comment from the grammar, and just have a simple match in the loop that goes over the lines in the input...

``zef install readline`` fails with --tap-harness

Issue

zef install Readline  
# fails - --verbose shows
# [Readline] t/02-base.t (Wstat: 0 Tests: 17 Failed: 0)
# [Readline]   Parse errors: Subtest 5 expected 3 but contains 2 tests

zef install Readline --/tap-harness # works

If I download the source, and comment out this line: https://github.com/lathropd/perl6-readline/blob/1e2f1f00c56c8cc1762b997e083ea460ea7ee1ae/t/02-base.t#L58 the tests pass.

The -verbose output from the test shows that terminal actually runs three tests, but then reports expected 3 but contains 2 tests

Environment:

• Cygwin MINTTY 3.6.1-1 terminal 'xterm' connecting via ssh to:
• Linux 5.10.0-20-amd64 #1 SMP Debian 5.10.158-2 (2022-12-13) x86_64 GNU/Linux
• TERM=xterm

Please let me know if you need any further information. I can also test if you don't have a similar environment.

:)

Colors on Windows with bash and Microsoft Visual Code

I am using the Microsoft Visual Code IDE on Windows 10. In Visual Code i've got a terminal which uses the git shell, so it starts a bash. unfortunatally the colors of prove6 are not displayed correctly. This is what I get:

?[32mAll tests successful.
?[0mFiles=5, Tests=46,  115 wallclock secs
Result: PASS

The Colors of the prompt or the colors of ls --color are displayed correctly.

Not sure if this is TAP::Harness fault but maybe you could provide a command line switch to disable colored output?

Restore subtest and yaml parsing

Previously these were parsed, but due to naivite in the parser this caused "test out of order" issues. Hence they are suppressed now (they are ignored instead).

The challenge is fundamentally that these entries are multi-line yet all unparsable input is still considered valid. This makes it complicated to distinguish an incomplete parse from an incorrect one.

Method 'failed' not found for invocant of class 'Slip'

Running a benchmark for this vs prove vs flow - I'm receiving this error while running prove6 - any ideas?

tonyo@mbp:~/projects/benchmark/Bailador$ /Users/tonyo/.rakudobrew/moar-nom/install/share/perl6/site/bin/prove6 -I=lib t/
t/01-route-existance.t ..... ok
Method 'failed' not found for invocant of class 'Slip'
  in sub MAIN at /Users/tonyo/.rakudobrew/moar-nom/install/share/perl6/site/resources/05C7ED85897FC8EEDD8A30A4B34FD1DFD2E691D0 line 48
  in block <unit> at /Users/tonyo/.rakudobrew/moar-nom/install/share/perl6/site/resources/05C7ED85897FC8EEDD8A30A4B34FD1DFD2E691D0 line 63

A couple of unfixed bugs

FWIW there are several bugs in this module that causes "test out of order" issues when people install this module and use zef.

Those bugs are fixed in the TAP::Harness that comes with rakudo.

It was probably not the original intent to have it included with rakudo, but now we have a situation that we have two TAP::Harness'es in the ecosystem; one's broken, another one isn't.

How can we end up with all working; or just one harness in the ecosystem?

prove6 does not seem to be installed

Or actually present in this repo...

CI is not operational

It seems the CI hasn't updated in 2 years, and no longer compiles TAP. It should probably be rearchitected.

Getting an error in Windows during CI tests on Github that Tap won't install

Please see: https://github.com/sdondley/Directory/actions/runs/3972369620/jobs/6810208097

Error:

Run zef install --/test App::Prove6
[4](https://github.com/sdondley/Directory/actions/runs/3972369620/jobs/6810208097#step:5:5)
===> Searching for: App::Prove6
[5](https://github.com/sdondley/Directory/actions/runs/3972369620/jobs/6810208097#step:5:6)

[6](https://github.com/sdondley/Directory/actions/runs/3972369620/jobs/6810208097#step:5:7)
===> Searching for missing dependencies: Getopt::Long:ver<0.3.0+>, Path::Finder:ver<0.4.4+>, Pod::Usage, TAP:ver<0.3.4+>, sigpipe
[7](https://github.com/sdondley/Directory/actions/runs/3972369620/jobs/6810208097#step:5:8)
===> Installing: Getopt::Long:ver<0.4.1>
[8](https://github.com/sdondley/Directory/actions/runs/3972369620/jobs/6810208097#step:5:9)
===> Installing: Path::Finder:ver<0.4.5>
[9](https://github.com/sdondley/Directory/actions/runs/3972369620/jobs/6810208097#step:5:10)
===> Installing: Pod::Usage:ver<0.0.1>:auth<zef:leont>
[10](https://github.com/sdondley/Directory/actions/runs/3972369620/jobs/6810208097#step:5:11)
===> Installing: TAP:ver<0.3.13>
[11](https://github.com/sdondley/Directory/actions/runs/3972369620/jobs/6810208097#step:5:12)
===> Install [FAIL] for TAP:ver<0.3.13>: 
[12](https://github.com/sdondley/Directory/actions/runs/3972369620/jobs/6810208097#step:5:13)
Error: Process completed with exit code 1.

option “--loose” not shown in the option list for prove6

Feature request: report failed test comments in summary

I'm currently trying to catch a flip-flop failure of a roast test which only takes place under harness and only when tests are executed as a batch, i.e. with make spectest, or make m-spectest6. Unfortunately, I have no other information but that test somehow fails. I can't run it in verbose mode a pass to tee or less:

Unhandled exception in code scheduled on thread 4
Cannot flush handle: Failed to flush filehandle: Operation not supported
  in block  at /Users/vrurg/src/Perl6/rakudo/t/packages/tap-harness6/lib/TAP.pm (TAP) line 927

It'd be great if comments following failed test are collected and reported at summary section when requested. Say, with an environment variable set or with volume set to Normal.

Fails on Rakudo Perl6 2017.10

Cannot assign to a readonly variable ($/) or a value

Can't run subst on $/, like this line:

https://github.com/perl6/tap-harness6/blob/master/lib/TAP.pm#L233

        make ~$/.subst(/\\('#'|'\\')/, { $_[0] }, :g)

Make `ok`s green

Is it possible to make the ok words for each successful test green?

Use RAKULIB instead of -I for directory paths

Currently TAP::Harness adds include paths using -I, which works fine in most cases but fails if any tests start new raku processes that depend on those paths. Using RAKULIB instead should allow those programs to function as expected.

deprecation message

runnings raku version

Welcome to 𝐑𝐚𝐤𝐮𝐝𝐨™ v2021.06-78-gd668d9929.
Implementing the 𝐑𝐚𝐤𝐮™ programming language v6.d.
Built on MoarVM version 2021.06-16-g74e828f0a.

I get the following message when testing

Saw 1 occurrence of deprecated code.
================================================================================
Method wait (from TAP::Result) seen at:
  /home/Data/root-opt/Raku/rakudo/install/share/perl6/site/sources/6FEC28FA0762AC132736C81C67CCDB823E5E83FA (TAP), line 148
Please use exitcode and/or signal methods (status is to be removed in 2022.06) instead.
--------------------------------------------------------------------------------

prove6 -v exception thrown on "Bash on Ubuntu on Windows"

Running prove6 -v on "Bash on Ubuntu on Windows" throws the following exception:

Unhandled exception in code scheduled on thread 6
Cannot flush handle: Failed to flush filehandle: Input/output error
  in block  at /home/kmel/.perl6/sources/850D4CD3715B1F98EE30D84A228A851CCD444986 (TAP) line 914

TAP 0.3.1 breaks zef install

If I have the latest TAP installed (0.3.1 as of this writing), then zef install fails because it cannot find test files. With TAP:ver<0.3.0> installed, everything is ok. Sample output:

$ zef --verbose install eigenstates
===> Searching for: eigenstates
===> Found: eigenstates:ver<0.0.9>:auth<zef:lizmat> [via Zef::Repository::Ecosystems<fez>]
===> Fetching [OK]: eigenstates:ver<0.0.9>:auth<zef:lizmat> to /Users/liz/.zef/tmp/1642188589.59951.8849.479889991366/46cb93d3980b5dcd88d190e9d956659936b0f0cc.tar.gz
===> Extraction [OK]: eigenstates to /Users/liz/.zef/tmp/46cb93d3980b5dcd88d190e9d956659936b0f0cc.tar.gz
===> Testing: eigenstates:ver<0.0.9>:auth<zef:lizmat>
[eigenstates] Could not open t/01-basic.t. Failed to stat file: no such file or directory
[eigenstates] t/01-basic.t .. Dubious, test returned 1
[eigenstates] No subtests run
[eigenstates] All tests successful.
[eigenstates] 
[eigenstates] Test Summary Report
[eigenstates] -------------------
[eigenstates] t/01-basic.t (Wstat: 256 Tests: 0 Failed: 0)
[eigenstates] Non-zero exit status: 1
[eigenstates]   Parse errors: No plan found in TAP output
[eigenstates] Files=1, Tests=0,  0 wallclock secs
[eigenstates] Result: FAILED
===> Testing [FAIL]: eigenstates:ver<0.0.9>:auth<zef:lizmat>
Aborting due to test failure: eigenstates:ver<0.0.9>:auth<zef:lizmat> (use --force-test to override)

Rename the repo to better reflect the module’s name

If the module is used in Github Workflows testing is named ‘TAP’, its repo here should be renamed to ‘TAP’. Otherwise, the open issues and comments are very confusing to any stranger trying to follow them and offer constructive suggestions.

Move some TAP::Harness arguments to run for consistency.

It was my design intention to make policy parameters (ones that are typically the same for different run calls such as volume, reporter-class or color) part of the constructor and arguments that change with every run invocation (such as filenames). Generally speaking, TAP::Harness does a decent job of detecting the former. However this this distinction has eroded; I think it should be restored. In particular:

• cwd
  This is clearly tied with the files that are run more than anything else, so really should be a run argument instead of a constructor argument
• out/err
  These are a bit in-between, in the sense that they can be either. So probably both will be supported.
• handlers
  This is a trickier case. Overriding it was intended to be unusual but turns out to be common because of additional include paths. This strongly suggests those include paths should be a parameter to run instead, which requires run to forward that to the SourceHandler.

The end result would be a design where most applications only need to ever construct a single TAP::Harness object, that they would repeatedly call run on that. This would also facilitate job control.

Need to update README.md

Currently SYNOPSIS in README.md says

use TAP::Harness;
my $harness = TAP::Harness.new(|%args);
$harness.run(@tests);

But, we cannot use TAP::Harness actually:

❯ raku -MTAP::Harness -e 'my $harness = TAP::Harness.new(|%args); $harness.run(@tests);'
===SORRY!=== Error while compiling -e
Could not find TAP::Harness in:
    inst#/Users/skaji/.raku
    inst#/Users/skaji/env/rakuenv/versions/2021.12-01/share/perl6/site
    inst#/Users/skaji/env/rakuenv/versions/2021.12-01/share/perl6/vendor
    inst#/Users/skaji/env/rakuenv/versions/2021.12-01/share/perl6/core
    ap#
    nqp#
    perl5#
at -e:1

It would be nicer if prove6 hid output of todo failures

We have a test marked todo in Bailador. prove6 displays the failure as follows:

$ prove6 -l t/30-examples-app.t
t/30-examples-app.t ..1/15
    # Failed test 'route GET /hello/Foo.html'
    # at t/30-examples-app.t line 68
    # expected: $[200, [:Content-Type("text/html")], "Hello Foo.html!"]
    #      got: $[404, [:Content-Type("text/html;charset=UTF-8")], "<html>\n    <head>\n        <title>Custom 404 page</title>\n        <meta charset=\"UTF-8\">\n    </head>\n    <body>\n        <h1>Hello, this is 404 for you.</h1>\n    </body>\n</html>\n"]
    # Looks like you failed 1 test of 2
t/30-examples-app.t .. ok
All tests successful.
Files=1, Tests=15,  4 wallclock secs
Result: PASS

I think this makes people, especially people new to perl* uneasy as the failure takes up a lot of the reporting and the final success is less obvious.
I just checked and prove of Perl 5 does the same, but I wonder if it wouldn't be more user-friendly if the default was to hide the error from todo tests and only show them in verbose mode.

In addition IMHO the final report should also say "1 todo"

Our repo is https://github.com/Bailador/Bailador

Formatter is not concurrency-safe

Due to this, output may end up completely garbled. Either synchronisation or a complete rewrite is needed (probably both).

needed “--loose” for perl6 version 2017.11

Passing a Supply :$err to TAP::Harness results in "No such method 'done' for invocant of type 'Supply'"

The Supply handling of :$err appears to be broken since there is no .done method on a supply as used here:

tap-harness6/lib/TAP.pm

Lines 828 to 830 in d0f95e0

	when Supply:D {
	$async.stderr.act({ $err.emit($_) }, :done({ $err.done }), :quit({ $err.quit($^reason) }));
	}

# ran from root of zef repository

raku -e 'use TAP; my $supplier = Supplier.new; my $supply = $supplier.Supply; my @handlers = TAP::Harness::SourceHandler::Raku.new(:incdirs(["$*CWD/lib"])); my $parser = TAP::Harness.new(:@handlers, :err($supply)); my $p = $parser.run("t/00-load.t"); say $p.result'

t/00-load.t ..2/2Unhandled exception in code scheduled on thread 6
No such method 'done' for invocant of type 'Supply'.  Did you mean any
of these: 'none', 'one', 'clone', 'do', 'does', 'note'?
  in block  at /Users/ugexe/.rakubrew/versions/moar-2021.12/install/share/perl6/site/sources/6FEC28FA0762AC132736C81C67CCDB823E5E83FA (TAP) line 829

Check out what's wrong with tests

Tests simply are not running. May be a problem with the Docker container, or something totally different. Obsrved in #35

Please tag your releases

Hello

I'm part of the Debian team that package Tap::Harness6 (and other Raku modules) for Debian.

I've seen that you have bumped Tap::Harness to version 0.1.0 but there's no matching tag in your repo. This is a problem for us because some of our tools rely on git tag to identify a new release. Without tag, Tap::Harness6 on Debian might get stale because we won't notice that a new version is available.

So could you please tag version 0.1.0 ?

All the best

Dod (for Debian)

Fails when there's some additional string in output

When there's something like this:

    ok 4 - routine name input
stuff
    ok 5 - identifier (sub) input

it fails as indicated in #32
prove didn't fail.

Deprecated warn message

Unfortunately, there is this message shown in each test run:

Saw 1 occurrence of deprecated code.
================================================================================
Method wait (from TAP::Result) seen at:
  /usr/share/perl6/vendor/sources/9E59A66A5978067EEE04BC039416C2CBDD2F5F92 (TAP), line 148
Please use exitcode and/or signal methods (status is to be removed in 2022.06) instead.
--------------------------------------------------------------------------------
Please contact the author to have these occurrences of deprecated code
adapted, so that this message will disappear!

The code line in question is
https://github.com/Raku/tap-harness6/blob/master/lib/TAP.pm#L148

Please, would someone be so nice and take a look?
Thank you!

Docker image isn't working for poweron -ppc64le

resolvconf
services
3.02s$ sudo systemctl start docker
git.checkout
0.54s$ git clone --depth=50 --branch=master https://github.com/asellappen/tap-harness6.git asellappen/tap-harness6
0.01s$ bash -c 'echo $BASH_VERSION'
4.3.48(1)-release
install
2.89s$ docker pull jjmerelo/test-perl6
0.79s$ docker run -t -v $TRAVIS_BUILD_DIR:/test jjmerelo/test-perl6
standard_init_linux.go:211: exec user process caused "exec format error"
The command "docker run -t -v $TRAVIS_BUILD_DIR:/test jjmerelo/test-perl6" exited with 1.
Done. Your build exited with 1.

starting dokcr image is failing on power on , Its probably due to the fact that the image isn't compiled for poweron -ppc64le,Can you check ?