tapjs / tapjs

Test Anything Protocol tools for node

@tapjs

Workspace for node-tap development.

Dev Commands

Do this at least once to get everything set up and ready to go:

npm run bootstrap

(Note: npm install will not work until you do this, because the generated TypeScript eats its own tail.)

---

Build the test class (required after any plugin or core changes):

npm run build

---

Any other builds:

npm run prepare -w src/{whatever}

---

After adding or removing workspaces:

npm i

---

Run all tests in all workspaces:

npm test

---

Run all tests, saving snapshots:

npm run snap

---

Build and serve docs:

npm start

Contents

• tap The main entry point module, which sets up the root test runner and exposes an alias to the cli runner.
• tap-parser The module that parses TAP
• @tapjs/core Most of the basic moving parts of tap
• tap-yaml Thin wrapper around YAML and yaml-types for consistent handling of JavaScript values in YAML diagnostics.
• @tapjs/test The plugin-ified Test class.
• @tapjs/config Handling config files, command line interface parsing, environment variables, and validation
• @tapjs/run The command line runner
• tcompare The library that does comparison and object formatting (use heavily by @tapjs/asserts methods).
• @tapjs/stack Library for capturing stack frames, the descendant of stack-utils.
• @tapjs/processinfo The library that tracks process information and code coverage (hosted outside the monorepo, because it can't be tested by a version of tap that uses itself without bootstrap paradoxes)
• default plugins:
  • @tapjs/typescript Adds typescript support, and the --typecheck config option
  • @tapjs/before Adds t.before()
  • @tapjs/before-each Adds t.beforeEach()
  • @tapjs/after Adds t.after() and t.teardown() (which are the same thing now)
  • @tapjs/after-each Adds t.afterEach()
  • @tapjs/spawn Adds t.spawn()
  • @tapjs/stdin Adds t.stdin()
  • @tapjs/asserts Adds all the various assertion methods, like t.equal(), t.match(), etc.
  • @tapjs/snapshot Adds t.matchSnapshot()
  • @tapjs/fixture Adds t.testdir() features
  • @tapjs/mock Adds t.mockRequire() and t.mockImport()
  • @tapjs/intercept Adds t.intercept() and t.capture() functionality (sort of like a very scaled-down minimal form of Sinon. If you want more, consider using the @tapjs/sinon plugin.)
  • @tapjs/filter Adds t.only() and support for the --grep and --only cli options.
• optional plugins:
  • @tapjs/nock Optional plugin providing t.nock() method (descendent of @npmjs/tap-nock, and a significant inspiration for tap's plugin architecture)
  • @tapjs/clock Optional plugin providing a clock-mock object on t.clock
  • @tapjs/sinon Optional plugin providing a Sinon sandbox at t.sinon that automatically restores at the end of the test.
  • @tapjs/esbuild-kit Optional plugin that loads TypeScript using @esbuild-kit/cjs-loader and @esbuild-kit/esm-loader instead of ts-node.
• other stuff:
  • npm-init-template A library for more easily creating npm init packages. This will move out as soon as this version of tap is published.
  • @tapjs/create-plugin An npm init library facilitating npm init @tapjs/plugin to create new plugins.

Bootstrap and skipLibCheck

Run npm run bootstrap to build the @tapjs/test module with the default set of plugins, so that the other libraries can build properly. (This only has to be done once, unless the build script or set of default plugins are changed, of course.)

Because there's a bootstrapping cycle between @tapjs/core, @tapjs/test, and all of the plugins, they MUST use skipLibCheck: true in their tsconfigs. It should not be used in other packages.

tapjs's Issues

CoffeeScript

can you please add a command line option to support CoffeeScript?

For example Forever and Vows have supported CoffeeScript with command line options.

ignore non executable scripts

tap attempts to run any thing in the test dir that is not .js as a script.

should ignore files that are not executable.

show me the stringified exception

node-tap shows

thrown: true

But if it shows stringified exception value, it's more useful.

Using t.equal on two similar objects produces confusing output

I mistakenly used t.equal( output, json ) and the output it produced was unhelpful in working out what I did wrong

not ok 1
  ---
    file:   /Users/ash/code/js/markdown-js/test/features.t.js
    line:   53
    column: 17
    stack:  
      - getCaller (/Users/ash/code/js/markdown-js/node_modules/tap/node_modules/tap-assert/assert.js:369:17)
      - assert (/Users/ash/code/js/markdown-js/node_modules/tap/node_modules/tap-assert/assert.js:17:16)
      - Function.equal (/Users/ash/code/js/markdown-js/node_modules/tap/node_modules/tap-assert/assert.js:158:10)
      - Test._testAssert [as equal] (/Users/ash/code/js/markdown-js/node_modules/tap/node_modules/tap-test/test.js:86:16)
      - Test.<anonymous> (/Users/ash/code/js/markdown-js/test/features.t.js:53:17)
      - Test.<anonymous> (native)
      - Test.<anonymous> (events.js:61:17)
      - Test.emit (/Users/ash/code/js/markdown-js/node_modules/tap/node_modules/tap-test/test.js:102:8)
      - GlobalHarness.<anonymous> (/Users/ash/code/js/markdown-js/node_modules/tap/node_modules/tap-harness/harness.js:86:13)
      - Array.0 (native)
    found:  
      - html
      - 
        - blockquote
        - 
          - p
          - There's a code block in here:
        - 
          - pre
          - 
            - code
            - |
              SET foo = TRUE

              IF foo GOTO 10
        - 
          - p
          - Wasn't that nice?
    wanted: 
      - html
      - 
        - blockquote
        - 
          - p
          - There's a code block in here:
        - 
          - pre
          - 
            - code
            - |
              SET foo = TRUE

              IF foo GOTO 10
        - 
          - p
          - Wasn't that nice?
    diff:   ~
    unique: 0

The main issue here is diff: ~. Having some message saying that the objects are similar but different references/ids might hlep?

segv test fails

 ∴  node-tap (master) : npm test

> [email protected] test /Users/julian/pro/node-tap
> bin/tap.js test/*.js

ok test/buffer_compare.js ............................... 3/3
ok test/common.js ....................................... 2/2
ok test/deep.js ......................................... 4/4
ok test/expose-gc-test.js ............................... 4/4
ok test/independent-timeouts.js ......................... 1/1
ok test/isolated-conf-test.js ........................... 5/5
ok test/meta-test.js .................................. 11/11
ok test/nested-test.js .................................. 5/5
ok test/non-tap-output.js ............................... 4/4
ok test/output-childtest-description.js ................. 6/6
ok test/result-trap.js .................................. 1/2
not ok test/segv.js ..................................... 8/9
    Command: "node" "segv.js"
    TAP version 13
    ok 1 compiled seg faulter
    ok 2 should be equivalent
    ok 3 should be equivalent
    not ok 4 should be equivalent
      ---
        file:   /Users/julian/pro/node-tap/test/segv.js
        line:   47
        column: 7
        stack:
          - getCaller (/Users/julian/pro/node-tap/lib/tap-assert.js:418:17)
          - assert (/Users/julian/pro/node-tap/lib/tap-assert.js:21:16)
          - Function.equivalent (/Users/julian/pro/node-tap/lib/tap-assert.js:182:12)
          - Test._testAssert (/Users/julian/pro/node-tap/lib/tap-test.js:87:16)
          - TapConsumer.<anonymous> (/Users/julian/pro/node-tap/test/segv.js:47:7)
          - TapConsumer.EventEmitter.emit (events.js:117:20)
          - TapConsumer._parseLine (/Users/julian/pro/node-tap/lib/tap-consumer.js:115:10)
          - TapConsumer.write (/Users/julian/pro/node-tap/lib/tap-consumer.js:77:10)
          - Runner.ondata (stream.js:51:26)
          - Runner.EventEmitter.emit (events.js:95:17)
        found:
          id:       1
          ok:       false
          name:     ././segv
          exit:     ~
          timedOut: true
          signal:   SIGTERM
          command:  "./segv"
        wanted:
          id:       1
          ok:       false
          name:     ././segv
          exit:     ~
          timedOut: true
          signal:   SIGBUS
          command:  "./segv"
        diff:   |
          {
            "id" : 1,
            "ok" : false,
            "name" : " ././segv",
            "exit" : null,
            "timedOut" : true,
            "signal" : "SIGTERM", // != "SIGBUS"
            "command" : "\"./segv\""
          }
      ...
    ok 5 should be equivalent
    ok 6 should be equivalent
    ok 7 should be equal
    ok 8 cleaned up
    ok 9 test/segv.js

    1..9
    # tests 9
    # pass  8
    # fail  1

ok test/simple-harness-test-with-plan.js ................ 4/4
ok test/simple-harness-test.js .......................... 4/4
ok test/test-test.js ................................ 148/148
ok test/timeout.js ...................................... 5/5
ok test/trivial-success.js .............................. 1/1
ok test/undefined_indented.js ........................... 2/2
total ............................................... 218/220

not ok
npm ERR! weird error 1
npm ERR! not ok code 0
 ∴  node-tap (master) : node -v
v0.10.15

Please add a --stdout and --stderr to the tap runner

It's really useful to be able to get my console.log()'s to show up when writing tests, until stuff is in a "steady state".

output doesn't show full relative path to test files

$ find tst
tst
tst/agent.test.js
tst/master
tst/master/get.test.js
tst/relay.test.js
$ bin/tap tst
ok agent.test.js .................... 1/1
ok get.test.js ...................... 1/1
ok relay.test.js .................... 1/1
total ............................... 6/6

ok

That should show:

...
ok master/get.test.js ...................... 1/1
...

right? Or perhaps, it would be nice for it to show that. My naive studying of TAP suggests that TAP doesn't require this.

Opinion one way or the other? If it sounds good, i can look at working up a patch for this.

'bin' is not recognized as an internal or external command on Windows 7 box

I just checked out the code on a Windows 7 box and installed dependent modules. Running the 'npm test' produced the following output:

D:\2013\github\node-tap>npm test

[email protected] test D:\2013\github\node-tap
bin/tap.js test/*.js

'bin' is not recognized as an internal or external command,
operable program or batch file.
npm ERR! Test failed. See above for more details.
npm ERR! not ok code 0

What am I missing?

Deal with high amount of tests

I'm doing one test.ok() call for every node in an autogenerated AST - maybe this is a bad idea? Anyway, tap dies:

[jann@Jann-PC node-astjourney locations]$ npm test

> [email protected] test /home/jann/gitty/node-astjourney
> tap test/*.js

ok find-requires.js ............... 12/12
ok scope.js ......................... 8/8
ok stringify.js ..................... 2/2
ok token-locations.js ............... 2/2
total ............................. 27/28

not ok
npm ERR! [email protected] test: `tap test/*.js`
npm ERR! `sh "-c" "tap test/*.js"` failed with 1
[...]

Could this be because the RAM fills up? The test is at https://gist.github.com/72f7acd0bbfb07a908e3 - I'm doing util.inspect on a lot of nodes.

`t.bailout` results in a hang

The following hangs:

$ ../bin/node --version
v0.4.12
$ cat foo.test.js
var test = require('tap').test;
test("first", function(t) {
    t.end();
});
test("bail", function(t) {
    t.bailout("boom");
    t.end();
});
test("last", function(t) {
    t.end();
});
$ ../bin/node foo.test.js
# first
^C

I see this with both tap 0.0.14 and 0.0.1. My understanding from the example test file is that this should abort all subsequent tests in the test file.

Subtests don't Run

Best illustrated by an example:

var test = require("tap").test;
test("parent", function (t) {
  t.plan(1)
  t.test("subtest", function (t) {
    t.ok()
    t.end()
  })
  t.end()
})

The output of running this via node foo.t.js is:

# parent

1..0
# tests 0

ok

(This is basically a cut down version of the example from the README.md)

exceptions in t.on('end', fn) cause infinite loops

Given this test:

var test = require('tap').test;

test(function (t) {
    t.plan(1);

    t.on('end', function () {
        console.log('end()');
        throw new Error('beep');
    });

    t.equal(3, 3);
});

the program produces this output:

$ node t.js 
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
^C

Probably it should just display the error message instead.

tap gets confused with multiple tests providing timeout

First case: one test of four with a 'timeout' setting. Another test has a bogus config object ('xxx' key). This suite as expected.

var test = require('tap').test;
console.warn("[%s] Start.", new Date())
test("one", function(t) {
  t.ok("one", "one potato");
  t.end();
});
test("two", {timeout: 3000}, function(t) {
  setTimeout(function() {
    t.ok("two", "two potato");
    t.end()
    console.warn("[%s] Finished executing test two.", new Date())
  }, 7000);
});
test("three", {xxx: 1000}, function(t) {
  t.ok("three", "three potato");
  t.end();
});
test("four", function(t) {
  t.ok("four", "four");
  t.end();
});

output:

$ node foo.js 
[Mon Dec 05 2011 11:24:11 GMT-0800 (PST)] Start.
# one
ok 1 one potato
# two
not ok 2 Timeout!
  ---
    file:   timers.js
    line:   83
    column: 39
    stack:  
      - getCaller (/Users/trentm/src/node-tap/lib/tap-assert.js:376:17)
      - assert (/Users/trentm/src/node-tap/lib/tap-assert.js:17:16)
      - Function.fail (/Users/trentm/src/node-tap/lib/tap-assert.js:91:10)
      - Test._testAssert [as fail] (/Users/trentm/src/node-tap/lib/tap-test.js:86:16)
      - Test.timeout (/Users/trentm/src/node-tap/lib/tap-test.js:30:8)
      - Object._onTimeout (native)
      - Timer.callback (timers.js:83:39)
  ...
# three
ok 3 three potato
# four
ok 4 four

1..4
# tests 4
# pass  3
# fail  1
[Mon Dec 05 2011 11:24:18 GMT-0800 (PST)] Finished executing test two.

Second case: Both tests "two" and "three" have a timeout. We expect test two to timeout (as above) and test "three" to complete within its timeout:

var test = require('tap').test;
console.warn("[%s] Start.", new Date())
test("one", function(t) {
  t.ok("one", "one potato");
  t.end();
});
test("two", {timeout: 3000}, function(t) {
  setTimeout(function() {
    t.ok("two", "two potato");
    t.end()
    console.warn("[%s] Finished executing test two.", new Date())
  }, 7000);
});
test("three", {timeout: 1000}, function(t) {
  t.ok("three", "three potato");
  t.end();
});
test("four", function(t) {
  t.ok("four", "four");
  t.end();
});

output:

$ node foo.js 
[Mon Dec 05 2011 11:24:58 GMT-0800 (PST)] Start.
# one
ok 1 one potato
# two
not ok 2 Timeout!
  ---
    file:   timers.js
    line:   83
    column: 39
    stack:  
      - getCaller (/Users/trentm/src/node-tap/lib/tap-assert.js:376:17)
      - assert (/Users/trentm/src/node-tap/lib/tap-assert.js:17:16)
      - Function.fail (/Users/trentm/src/node-tap/lib/tap-assert.js:91:10)
      - Test._testAssert [as fail] (/Users/trentm/src/node-tap/lib/tap-test.js:86:16)
      - Test.timeout (/Users/trentm/src/node-tap/lib/tap-test.js:30:8)
      - Object._onTimeout (native)
      - Timer.callback (timers.js:83:39)
  ...
# three
not ok 3 Timeout!
  ---
    file:   timers.js
    line:   83
    column: 39
    stack:  
      - getCaller (/Users/trentm/src/node-tap/lib/tap-assert.js:376:17)
      - assert (/Users/trentm/src/node-tap/lib/tap-assert.js:17:16)
      - Function.fail (/Users/trentm/src/node-tap/lib/tap-assert.js:91:10)
      - Test._testAssert [as fail] (/Users/trentm/src/node-tap/lib/tap-test.js:86:16)
      - Test.timeout (/Users/trentm/src/node-tap/lib/tap-test.js:30:8)
      - Object._onTimeout (native)
      - Timer.callback (timers.js:83:39)
  ...
ok 4 three potato
not ok 5 end called more than once
  ---
    file:   /Users/trentm/src/node-tap/tmp/foo.js
    line:   33
    column: 5
    stack:  
      - getCaller (/Users/trentm/src/node-tap/lib/tap-assert.js:376:17)
      - assert (/Users/trentm/src/node-tap/lib/tap-assert.js:17:16)
      - Function.fail (/Users/trentm/src/node-tap/lib/tap-assert.js:91:10)
      - Test.end (/Users/trentm/src/node-tap/lib/tap-harness.js:102:29)
      - Test.<anonymous> (/Users/trentm/src/node-tap/tmp/foo.js:33:5)
      - Test.<anonymous> (native)
      - Test.<anonymous> (events.js:61:17)
      - Test.emit (/Users/trentm/src/node-tap/lib/tap-test.js:102:8)
      - GlobalHarness.<anonymous> (/Users/trentm/src/node-tap/lib/tap-harness.js:86:13)
      - Array.0 (native)
  ...
# four
ok 6 four

1..6
# tests 6
# pass  3
# fail  3
[Mon Dec 05 2011 11:25:05 GMT-0800 (PST)] Finished executing test two.

IOW, this does NOT work as expected.

On a possibly related note, you can see from the printed timestamps that the test run does not terminate until the 7s "setTimeout" still runs to completion. Say this is a test case that takes a crazy long time, or forever to complete. Ideally I'd want my run to be aborted after the timeout, but I don't know how I'd implement that in this case.

Third case: Extend the second timeout (for test "three") to 13s so that there is sufficient time after test "two" has timed out and completed as well. Code:

var test = require('tap').test;
console.warn("[%s] Start.", new Date())
test("one", function(t) {
  t.ok("one", "one potato");
  t.end();
});
test("two", {timeout: 3000}, function(t) {
  setTimeout(function() {
    t.ok("two", "two potato");
    t.end()
    console.warn("[%s] Finished executing test two.", new Date())
  }, 7000);
});
test("three", {timeout: 13000}, function(t) {
  t.ok("three", "three potato");
  t.end();
});
test("four", function(t) {
  t.ok("four", "four");
  t.end();
});

Result:

$ TAP=1 ./node_modules/.bin/tap foo.js
# one
ok 1 one potato
# two
not ok 2 Timeout!
# three
ok 3 three potato
# four
ok 4 four
# tests 4
# pass  3
# fail  1
ok 5 foo.js


1..5
# tests 5
# pass  4
# fail  1

I.e., this is as expected.

Theory: The timeout handling for each test case isn't correctly starting the count from when that test case itself is being run... but is starting from the start of the full test suite run (or rather the start of when this test file is being run if there are multiple test files being run).

Running files in the same process

Is there a flag or an option to not spin up a new process per test file.

I'd be nice to do an integration test without resetting the environment each time.

assert.throws when throwing an object (not an Error)

Hi,

I want to test that my couchDB's validate_doc_update function rejects unidentifed users:

module.exports = function (newDoc, oldDoc, userCtx, dbCtx) {
  if (!userCtx || !userCtx.name) {
    throw {unauthorized: "Fuck off!"};
  }
}

I don't know tap very well, but I'm pretty sure I should use assert.throws:

tap = require('tap')

validate_doc_update = require('./validate_doc_update')

icognitoCtx = {
  db: 'foo'
  name: null,
  roles: []
}

tap.test('Reject unidentified users', function (t) {
  t.throws(
    function () {
      validate_doc_update({}, {}, icognitoCtx);
    },
    {unauthorized: "Go away!"},
    'should stay polite and tell you to go away (not to fuck off)'
  )
});

My problem is this test passes despite {unauthorized: "Fuck off!"} does not equal {unauthorized: "Go away!"}...

Looking at the code, it seems tap deals with Errors only, not objects like the ones couchDB likes, eg: {forbidden/unauthorized: 'message'}

Do you see any work around for this?

thank you

Please move required tap-* modules into this repo

To use node-tap at all you need several other repos including:

https://github.com/isaacs/tap-runner.git
https://github.com/isaacs/tap-producer.git
https://github.com/isaacs/tap-results.git
https://github.com/isaacs/tap-consumer.git
https://github.com/isaacs/tap-global-harness.git
https://github.com/isaacs/tap-assert.git

which seem like they generally belong in this repo. My suggestion is to move these into this node-tap repo to decrease the number of dependencies.

node-tap hangs on certain functions

Running the following test:

var test = require("tap").test,                                                                                                                                                    
    http = require("http");                                                     

test("basic", function(t){                                                      
    var server = http.createServer().listen(8001);                              
    t.end();                                                                    
});

As such:

$ tap test/basic.js

Results in a timeout error:

not ok test/basic.js .................................... 0/1
    Command: "node" "basic.js"
    TAP version 13
    not ok 1 test/basic.js
      ---
        timedOut: true
        exit:     1
        command:  "node" "basic.js"
      ...

    1..1
    # tests 1
    # fail  1

total ................................................... 0/1

not ok

Whereas, I had expected the test to run and exit without errors.

t.throws seems to always pass

Why does the following test pass?

var test = require('tap').test;

function foo() {
  throw new Error('one');
}

test('demonstrate bug in t.throws', function (t) {
  t.throws(
    function () {
      foo();
    },
    'two'); // not 'one'!
  t.end();
});

tap callbacks and errors printing wrong stack

The tap runner will print the stack from the error, not the line in the test where the assertion failed:

var fs = require('fs');
var test = require('tap').test;


function ls(dir, callback) {
  fs.readdir(dir, function(err, files) {
    if (err)
      return callback(new Error(err.message));

    return callback(null, files);
  });
}


test('assertion stack wrong', function(t) {
  ls('/_tmp', function(err, files) {
    t.ifError(err); // prints the error stack, not where this failed
    t.end();
  });
});

bluesnoop:node-ldap mark$ node tst/fail.test.js 
# assertion stack wrong
not ok 1 ENOENT, No such file or directory
  ---
    type:    Error
    message: ENOENT, No such file or directory
    code:    ~
    errno:   ~
    stack:   
      - /Users/mark/work/node-ldap/tst/fail.test.js:8:23
    unique:  0
  ...

1..1
# tests 1
# fail  1
bluesnoop:node-ldap mark$ 

split tap into two packages (not 10)

tap-core: Basically everything in lib/ except coverage and runner. Depends on yamlish and inherits, and that's it.

tap: Depends on tap-core, runforcover, slide. Adds runner and coverage, and the stuff in bin/

Reasoning: it's very useful for other test runners and harnesses to be able to produce and consume tap formatted data, but it's silly for them to install the node-tap runner in their test runner.

NaN as a special case

It would be nice if NaN was treaded as a special case, so the following would be OK:

t.deepEqual({ prop: NaN }, { prop: NaN });

Right now the diff says:

diff:    |
  {
    "prop" : NaN, // != NaN
  }

and that is just a bit odd.

Tap bundles a version of yamlish that's out of sync with yamlish repo

I found that the tap module on npm bundles with it a yamlish dependency that advertises version 0.0.4 which is out of sync with the 0.0.4 tag on the yamlish repo https://github.com/isaacs/yamlish.git. See isaacs/yamlish#2 for more details on the difference.

exceptions in t.on('end', fn) cause infinite loops

Given this test:

var test = require('tap').test;

test(function (t) {
    t.plan(1);

    t.on('end', function () {
        console.log('end()');
        throw new Error('beep');
    });

    t.equal(3, 3);
});

the program produces this output:

$ node t.js 
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
end()
^C

Probably it should just display the error message instead.

Which test timed out?

It would be very helpful if when receiving a timeout from a test, the test that timed out was reported. Right now it is hard to determine which tests are timing out.

Output like this is common:

killing: test/router.test.js (timed out)
ok test/router.test.js .................................. 2/2
total ................................................... 2/3

Detailed output should include child test descriptions

When running the below with node tapme.js

// tapme.js
var test = require('tap').test;

test('parent test description', function (t) {
  t.plan(2)
  t.ok(true, 'test in parent')
  t.test('child test description', function (t) {
    t.plan(1)
    t.ok(true, 'test in child')  
  })
})

it outputs:

TAP version 13
# parent test description
ok 1 test in parent
ok 2 test in child

1..2
# tests 2
# pass  2

# ok

It would be nice if it could include the child test description before test 2, e.g., :

TAP version 13
# parent test description
ok 1 test in parent
# child test description  
ok 2 test in child

[...]

If you agree that this is a valid request I'd be willing to fix it myself and submit a PR, just wanted to get your opinion on this first.

coverage does too much file reading

The coverage thing does this right now:

1. Read all files in the --cover dir
2. attach the coverage hook to apply to /.*/

This is unnecessarily costly. Also, it's using a bunch of sync io, which needs to be factored out.

It'd be better to just apply the coverage hook to the --cover dir, so it works without the second read.

Syntax errors result in tap being "mysterious"

A syntax error at the top of my file resulted in output like this (basically it tells you nothing):

bluesnoop:node-asn1 mark$ ./node_modules/.bin/tap ./tst
ok ber_decoder.test.js .............. 0/0
total ............................... 0/1

not ok
bluesnoop:node-asn1 mark$ TAP=1 ./node_modules/.bin/tap ./tst

ber_decoder.test.js

not ok 1 ber_decoder.test.js

1..1

tests 1

fail 1

bluesnoop:node-asn1 mark$

tap not catching errors in async code

The code below will dump:

bluesnoop:node-ldap mark$ node tst/fail.test.js 

/Users/mark/work/node-ldap/tst/fail.test.js:25
    t.ok(files[0]);
              ^
TypeError: Cannot read property '0' of undefined
    at /Users/mark/work/node-ldap/tst/fail.test.js:25:15
    at /Users/mark/work/node-ldap/tst/fail.test.js:8:14
bluesnoop:node-ldap mark$ 

var fs = require('fs');
var test = require('tap').test;


function ls(dir, callback) {
  fs.readdir(dir, function(err, files) {
    if (err)
      return callback(new Error(err.message));

    return callback(null, files);
  });
}


test('tap not catching error', function(t) {
  ls('/_tmp', function(err, files) {
    t.ok(files[0]);
    t.end();
  });
});

tap command not working unless tap.js is accessed directly

tap ./test =>
bash: /usr/local/bin/tap: No such file or directory

I had to do this:
./node_modules/tap/bin/tap.js ./test

link in readme dead?

http://testanything.org/ seems to have been down since November 2nd (of last year?).

a `tap --version` would be very useful

Right now:

$ tap --version
total ............................... 0/1

not ok

Run only tests matching a pattern

If you have many tests and you only want to run some of them, you want to be able to tell tap to only run tests matching a certain pattern (glob, regex, etc.).

And how could I use this now to call it from npm test where "test" script is in my package.json something like:

node node_modules/tap/bin/tap.js ./test

tap command should look a exit status from test case.

tap command returns success when node.js segmentation fault-ed.

tap doesn't handle a throw being a non-Error.

I came across this by accident (forgot to set the prototype on a custom error), but my guess is somebody did something like throw 'foo' you'd hit this. Sample test case below, runner output farther down.

var test = require('tap').test;

function Foo() {
  function MyError(msg) {
    this.name = 'MyError';
    this.message = msg || '';
  }
  // Explicitly missing MyError.prototype = new Error();
  this.__defineGetter__('throwy', function() {
    throw new MyError('fubar!');
  });
}

test('bad throw', function(t) {
  var f = new Foo();
  t.ok(f);
  f.throwy;
  t.end();
});

bluesnoop:node-asn1 mark$ node foo.js 

node.js:134
        throw e; // process.nextTick error, or 'error' event on first tick
        ^
TypeError: Cannot call method 'split' of undefined
    at getCaller (/Users/mark/work/node-asn1/node_modules/tap/node_modules/tap-assert/assert.js:352:17)
    at assert (/Users/mark/work/node-asn1/node_modules/tap/node_modules/tap-assert/assert.js:14:16)
    at Function.fail (/Users/mark/work/node-asn1/node_modules/tap/node_modules/tap-assert/assert.js:73:10)
    at Test._testAssert [as fail] (/Users/mark/work/node-asn1/node_modules/tap/node_modules/tap-test/test.js:86:16)
    at Test.threw (/Users/mark/work/node-asn1/node_modules/tap/node_modules/tap-test/test.js:46:8)
    at Test.emit (/Users/mark/work/node-asn1/node_modules/tap/node_modules/tap-test/test.js:106:10)
    at GlobalHarness.<anonymous> (/Users/mark/work/node-asn1/node_modules/tap/node_modules/tap-harness/harness.js:77:13)
    at Array.<anonymous> (native)
    at EventEmitter._tickCallback (node.js:126:26)

Merge in thejh's shebang test thing for runner

Fell out of sync with all the coverage stuff.

Parse globs

On Windows, globs don't auto-expand, so tap test/*.js is no good. This is sad; using node-glob would make things less sad.

(I tried tap test/ to use the useful directory feature, but it crawls recursively, picking up random stuff I've put in subdirectories.)

Fails to find .js files when given a directory on Windows

When trying to run tap (current version on NPM) on Windows using, I just get the following result:

> tap ./tests
total ................................................... 0/0

ok

If I specify a specific .js file in the tests folder, it runs just fine.

I dug into this and found that on line 162 of tap-runner.js is the following check:
} else if ((st.mode & 0001) == 0) {
return cb()
}

I debugged this and when provided just the directory, the st.mode was 0x41B6. In the Node source it looks like this would be a check against UV_FS_SYMLINK_DIR? But since this is a real directory and not a symlink, this is true, and the directory is not processed further. Making this check false, allows the directory to drop down to line 167 where it runs as expected.

I'd submit a pull request, but I'm new to the internals of Node and I have no idea what exactly this is a check for.

test/expose-gc-test.js doesn't actually test the child's output

t.ok("false", stdo)
// ... and ...
t.ok("true", stdo)

Those tests will always pass as they're just testing whether the strings "true" and "false" are truthy. I tried to fix the test using t.equal() and splitting stdo into separate lines but for some reason the child process didn't really produce the desired output.

Not sure what's going on and why I couldn't fix the test but I thought I'll file an issue so someone more familiar with the code can take a look at it.

keeps running test function after failed assert

tap continues executing the code in a test method after an assert fails. This allows other errors to mask the failed assertion. In the example below, getObj fails to return an object. The assertion t.ok(obj) fails as expected, but execution of the test function continues and we get the following error:
"not ok 3 TypeError: Cannot call method 'someMethod' of undefined".

I think it should stop on the first failed assertion in each test function and report that error. Most other test frameworks work that way and I think that is less confusing.

var test = require('tap').test;

function getObj(cb) {
  cb(null);
}

test('keeps going', function (t) {
  getObj(function (err, obj) {
    t.notOk(err);
    t.ok(obj);
    console.log('obj =', obj);
    obj.someMethod();
    t.end();
  });
});

order keys should not be significant to deep equals.

this test should pass:

test('order of keys should not be significant', function (t) {
  t.deepEqual({a: 0, b: 0}, {b: 0, a: 0})
  t.end()
})

instead, output is:

    found:  
      a: 0
      b: 0
    wanted: 
      b: 0
      a: 0
    diff:   |
      FOUND:  {"a":0,"b":0}
      WANTED: {"b":0,"a":0}
                ^ (at position = 2)
  ...

stringify(a) == stringify(b) is not generally they right way to test if two objects have the same structure.

Please add a --timeout option to the tap runner

This is useful to kill off ill-behaved tests if tap runs as part of a script, etc.

tap harness caches the test name on a possibly shared 'conf' object

Code ("foo.js"):

var test = require('tap').test;
var config = {timeout: 8000}
test("one", config, function(t) {
  t.ok("one");
  t.end()
});
test("two", config, function(t) {
  t.ok("two");
  t.end();
});
test("three", config, function(t) {
  t.ok("three");
  t.end();
});

Output:

$ TAP=1 ./node_modules/.bin/tap foo.js
# three
ok 1 (unnamed assert)
# three
ok 2 (unnamed assert)
# three
ok 3 (unnamed assert)
# tests 3
# pass  3
# ok
ok 4 foo.js


1..4
# tests 4
# pass  4

# ok

Notice "# three" for each test.

One fix is to create a copy of the 'conf' object: https://gist.github.com/1435063

Alternatively, if only the 'name' is put on this.conf, then could just have a separate this.name.

broken in v0.10.0

path.js:313
        throw new TypeError('Arguments to path.resolve must be strings');
              ^
TypeError: Arguments to path.resolve must be strings
    at Object.exports.resolve (path.js:313:15)
    at Array.map (native)
    at /home/raynos/Documents/colingo-api/node_modules/tap/lib/tap-runner.js:119:19

:(

tap and custom errors = tap implosion on .trim()

node.js:134
throw e; // process.nextTick error, or 'error' event on first tick
^
TypeError: Object NoSuchObjectError: No Such Object has no method 'trim'
at encodeResult (/Users/mark/work/node-ldapjs-riak/node_modules/tap/node_modules/tap-producer/tap-producer.js:92:35)
at TapProducer.write (/Users/mark/work/node-ldapjs-riak/node_modules/tap/node_modules/tap-producer/tap-producer.js:37:21)
at /Users/mark/work/node-ldapjs-riak/node_modules/tap/node_modules/tap-global-harness/global-harness.js:43:14
at Array.forEach (native)
at GlobalHarness. (/Users/mark/work/node-ldapjs-riak/node_modules/tap/node_modules/tap-global-harness/global-harness.js:40:23)
at GlobalHarness.emit (events.js:64:17)
at GlobalHarness.childEnd (/Users/mark/work/node-ldapjs-riak/node_modules/tap/node_modules/tap-harness/harness.js:137:8)
at GlobalHarness. (/Users/mark/work/node-ldapjs-riak/node_modules/tap/node_modules/tap-harness/harness.js:62:10)
at Array. (native)
at EventEmitter._tickCallback (node.js:126:26)

Include `tap-runner`

Been seeing this pop up across my installation of node/npm:

  npm WARN tap-runner 0.x Unmet dependency in /usr/local/lib/node_modules/sudojitsu/node_modules/optimist/node_modules/wordwrap/node_modules/tap

Looks like it exists in the package.json file, using the 0.x syntax. npm view tap-runner does in-fact yield a package, so it's even stranger to see these warnings. No clue really where the bug could be, but worth mentioning.

node-tap shouldn't use graceful-fs by default

Because tap uses glob, as a byproduct of loading the glob module, graceful-fs's polyfills are patched onto fs globally. Among other things, this means that fs.chown() starts swallowing EPERM. Reasonable people may disagree about the usefulness of this error when running as a non-privileged user (when chown is never going to work, at least on OS X), but the silent failure was surprising and cost me an hour or two of debugging and led me to believe the problem was in Node itself. Is there a way to use graceful-fs in glob without poisoning the fs module for tap users?

node-tap needs to require 'inherits-1.x'

Since inherits-2.x has been released a couple of days ago Class.super isn't defined anymore. node-tap should require inherits-1.x or use Class.super_.

Please add a --help

./node_modules/.bin/tap --help
total ................................................... 0/1

not ok
bluesnoop:node-kahu mark$

tap runs very slowly

I couldn't find a mailing list so I will post here. Apologies if I didn't look hard enough :(

When I create a simple test setup and run tap on it, it seems to take a long time to run (several seconds). The same test setup with nodeunit takes fractions of a second.

I am not running test coverage or doing anything in the test that would be causing such delays. Maybe I setup something else wrong? Is this expected to be slower than other testing options?

There was nothing complex about the testcase but I can provide the example if desired.

node: 0.6.11-pre
tap: 0.2.0

I get it, but are there any useful consumers for it already?

I read through a few pages on http://testanything.org/, and it's a great concept to standardise test results, but honestly thought that there doesn't seem like much going for it at the moment.

So my question is, is there anything really useful you can already do with it? Or at least some ideas for node programs?

Chris