Web Test Runner and QUnit reporting problems about qunit

[Krinkle]

@brandonaaron If that works, I suppose you could use that for now. If I read correctly, this code now makes the assumption that the two events will re-expose the same object by-reference in both situations and allows attachment and preservation an extra property to a foreign object. This is likely to work for a good number of years, but it goes without saying it's not officially supported and might break in a minor release without changelog as this isn't something we document or test for.

The approach I was thinking about, should work fine for Web Test Runner. WTR doesn't have to support this natively. I was thinking something like this, where you add the results to an array, and then send it at once.

const errors = [];
const testResults = [];

QUnit.on('testEnd', function (test) {
  const err = test.errors && test.errors[0];
  const errCopy = err ? {
      name: err.name,
      message: err.message,
      expected: err.expected,
      actual: err.actual,
      stack: err.stack,
    } : undefined;
  testResults.push({
    name: test.fullName.join(' > '),
    passed: test.status === 'passed' || test.status === 'todo',
    skipped: test.status === 'skipped',
    duration: test.runtime,
    error: errCopy,
  });
  if (errCopy) errors.push(errCopy);
});

QUnit.on('error', function (globalError) {
  errors.push(globalError);
});

QUnit.on('runEnd', function (runEnd) {
  sessionFinished({
    passed: runEnd.status === 'passed',
    testResults,
    errors,
  });
});

Note that the above is untested, but I hope it demonstrates the general idea.

I also noticed in your code a fair amount of handling for todo test that I believe should not be needed. We already report todo tests as failing if they have no expected failures anymore (per https://api.qunitjs.com/callbacks/QUnit.on/).

The above code sample is largely derived from the built-in TapReporter which might be a good reference example of how to consume the API in a way that collects and summarises everything correctly: https://github.com/qunitjs/qunit/blob/2.20.0/src/reporters/TapReporter.js#L172.

[brandonaaron]

Just signaling I'm around but time constrained. Happy to get improvements into web-test-runner-qunit and will make some time soon to review the associated issue @rnixx logged on web-test-runner-qunit.

[brandonaaron]

Thanks again @Krinkle for the insightful response. I threw together another PR (brandonaaron/web-test-runner-qunit#5) that builds up the data structure that @web/test-runner is expecting as we get the testEnd event from QUnit. This, as you suggested, is a more maintainable approach.

I still have some special handling for todo tests. I do this in order to get what I think is better reporting output within @web/test-runner. I might be wrong about this handling though and certainly open to improvements.

[Krinkle]

The actual/expected are included during the public testEnd event and are only deleted after that event has completed. The reason is to avoid large memory leaks.

Which API are you to to access the object? Can you perform the serialisation or formatting earlier so that you retain your copy earlier?

[rnixx]

Thank you for the quick reply.

The current implementation of web-test-runner-qunit uses testEnd for collecting the errors. However, the web test runner's default summary reporter expects the actual test failures in a TestSuiteResult object containing a summary and all test failures, while QUnit reports the test suite summary on runEnd and suiteEnd. In the context of web test runner it feels wrong to collect the actual test failures on testEnd and re-map them to the corresponding test suite summary errors afterwards (if this is even possible in a reliable way with the contained information).

I experimented a little bit with web-test-runner-qunit, and the easiest way to get sane test failure reporting was to comment out slimAssertions() in test.js and among some tweaks in web-test-runner-qunit.

The reason is to avoid large memory leaks.

Memory leak or memory usage? I mean even 1000 test failures with a payload of 3000 characters would mean a total memory usage below 3MB, which are in memory anyways in the first place. Please correct me if i misunderstand something.

About the possible solutions for the problem, there are several options:

• Make slimAssertions() call optional (feels the lowest hanging fruit right now, depends if there is a real memory leak or just some MB of RAM usage)
• Re-map errors reported on testEnd to the data received on runEnd (feels clumsy, and depends whether data even can be remapped reliably)
• Write a dedicated web test runner reporter for QUnit (would require the most effort and LOC)

I also got no answer yet from the author of web-test-runner-qunit. Maybe @brandonaaron has some additional input, complaints, ideas

[Krinkle]

@brandonaaron @rnixx Your code looks to me like it should work as-is, so in terms of what QUnit supports and what I would recommend, I would not want you to have to do much different than what you're doing today. I hope that gives you some confidence in that I agree with your use case and that this isn't a controversial issue.

The reason is to avoid large memory leaks.

Memory leak or memory usage?

Usually, memory that is held practically indefinitely is treated equivalent to a leak.

My expectation (on behalf of QUnit users) is generally that by the end of a test case, after the testEnd event, any references we received to live objects from user code, should be released. Projects can have very large documents and data fixtures, up to several dozens or hundreds of megabytes large that may be held via properties and closures inside classes via actual/expected assertions. Most of these will not become part of a JSON serialization or diff in CI, of course. If we held these indefinitely (note we expose both failed and passed assertions), that would significantly change what you can test with QUnit out-of-the-box.

However, it's entirely reasonable for specific QUnit plugins and QUnit reporters to say they don't (yet) care about such large test suites. All you have to do is, from the testEnd event, instead of storing the event in its entirety for later use, copy what you need at that time, possibly as simple as myError = { ...error };.

I would generally recommend to go one step further, and also reduce/clone/stringify at this time, and print progress to the developer running their tests. But, that's your call to make, and doesn't make sense for reporters (e.g. JUnit export).

Getting back to my opening sentence, it appears that you already do this? In https://github.com/brandonaaron/web-test-runner-qunit/blob/2e81cba5b5ab335a6ca16c0513ce314893a0722b/src/autorun.ts#L160-L166, I see QUnit.on('testEnd') calling a collectErrors function, which creates its own error object, with own reference to error.expected and error.actual. I would hope QUnit is not deleting those properties. Or am I looking in a different place from where you are observing this issue?

Re-map errors reported on testEnd to the data received on runEnd (feels clumsy, […])

Ah, I understand. Indeed, that would be clumpsy and I wouldn't want you to have to do that.

Generally the idea is that if you only want an aggregate result, use runEnd. If you want to provide detailed progress (either in real-time, or aggregate yourself) then use testEnd. I'm not sure why you'd want to consume both, other than for runEnd to let you know that the run has finished and maybe for some meta data (i.e. run duration, and overall boolean status).

If you really like the format of runEnd (perhaps it is very close to your final output?), you could build up your desired and typed structure directly during the testEnd events (with the tree implied by testEnd.fullName, or suiteEnd events if prefer a buffer+flush approach). That might be easier than having to modify and/or reconcile with the tree from runEnd after the fact.

Looking at web-test-runner as a whole very briefly, it seems the desired "final" output is typed by @web/test-runner:TestSuiteResult and that almost exactly resembles QUnit's tree structure, plus inlined assertion/error objects. I can see why you went for this approach. (Perhaps there's some past inspiration there from QUnit?)

[rnixx]

Your code looks to me like it should work as-is

The point here is that the errors collected in testResultErrors on testEnd are printed without actual/expected in web test runner (https://github.com/modernweb-dev/web/blob/master/packages/test-runner/src/reporter/reportTestFileErrors.ts#L49-L62) while the actual/expected information contained in testResults gets printed here https://github.com/modernweb-dev/web/blob/master/packages/test-runner/src/reporter/reportTestsErrors.ts#L94-L107 including the diffs.

The proof of concept for a fix - which requires skipping the slimAssertions() call - can be found here https://github.com/rnixx/web-test-runner-qunit/blob/rnixx-poc-qunit-patch/src/autorun.ts#L119-L172

[brandonaaron]

Thanks for the in-depth response @Krinkle. I made some time today to dig into this and came up with a PR that I hope addresses the issue.

I did end up having to re-map errors collected during the testEnd event in order to re-populate the expected and actual properties at the runEnd event. It appears that @web/test-runner expects to have these values as strings and for now I just stuck them in a template string... but I wonder if maybe I should call out to JSON.stringify or some other alternative.

I originally based the web test runner for qunit on their documentation and their mocha test runner. It doesn't appear that they currently support a way to build up a set of results in real-time using something like testEnd. I admit though that I haven't explored that further than the linked documentation and existing runner.

I'm not sure if QUnit needs/wants to make any changes regarding this issue.

[Krinkle]

@brandonaaron You're welcome. I see what you mean regarding todo. That seems like a good trade-off. I'll close this issue now as it seems this is resolved, but if you feel things could be easier still, or have any other question, feel free to continue below (I'm happy to re-open this, or to continue on a new issue as you see fit).