Planning Japa version 3 about runner

[McSneaky]

@jlenon7 I don't know how much time you have, but would be cool if you can start community package for it. It's a thing that has been asked / discussed several times in Discord too, so there are some more people interested in it

[thetutlage]

@jlenon7 To be honest, I will not be interested in maintaining another API myself. Also, I think having it as a community package is a valuable as having it in the core. Developers who like that API will use the package and get benefited from it.

[davidharting]

I am curious what the Japa core team thinks of the new native node test runner.

Do you imagine a future in Japa v3 or later where Japa integrates with Node.js test runner instead of a custom runner?

I love that the Japa team is so committed to making Node.js more capable, so I wonder if leaning into and enhancing native node functionality could make sense for this project.

[ndianabasi]

This concerns the integration of Japa with AdonisJS. I don't know if this is possible for it to be fixed in the new release but it is worth considering.

I recently noticed that when one run tests with dataset and calls an endpoint with a database validation rule like "exists", the test doesn't work as expected. I guessed it could be because the "exists" rule is not running within the same transaction block as the tests.

To expantiate more. I have a model with a "name" column. The "name" column has a unique DB level constraints. So I'm trying to avoid errors when duplicate names are created. I can't use "Model.firstOrCreate" here as there is no other column to tie the uniqueness to.

So when I check for uniqueness with the "exists" validation rule, the rule doesn't catch the duplicate insertion.

[jlenon7]

AdonisJS is a TypeScript framework, it comes with object-oriented programming style by default. So why do we need to write tests in functional style?

I work a lot with Java too, and whenever I come to Node.js ecosystem I miss JUnit so much 😫.

Japa is the best testing framework in the backend ecosystem for Node.js nowadays, and I wonder why we couldn't create TypeScript decorators for Japa to create tests using classes?

The test class would be the test.group('${className}') fn and the @Test decorator would be the test('${methodNameOrTitle}') fn:

export default class MyTest {
  @Test('my test title')
  public async myTest({ assert }: TestContext) {
    assert.equal(1, 1)
  }
}

[Julien-R44]

@jlenon7 my two cents on that: i think it's primarily a matter of language convention. in javascript, such test writing style isn't common. as far as I'm aware, none of the existing test framework allow this

and, tell me if i'm wrong, but i can't see any real advantage or disadvantage in using either one or the other. well, maybe the class-based approach is more verbose, because you need decorators for absolutely everything (tags, skip, timeout, retry etc.). so i am not sure why you are missing junit so much, i would appreciate to have your thoughts on that

otherwise i think all the needed APIs are open in japa to allow the emergence of a third-party lib that will use a class-based approach

[thetutlage]

@jlenon7 As you already mentioned that it is matter a taste, so it is not possible to have something that appeals everyone.

Also, I think changing the syntax of the framework is out of the question at this stage, regardless of the merits and demerits.

If you fancy creating a class based tests runner on top of Japa, then you need to dig into the source code of @japa/runner and see how it using the @japa/core. All the heavy lifting is done by @japa/core, which does not enforce any high-level API choices.

[thetutlage]

@Julien-R44

Have not added any colors. But here is a proof of concept. The plugin will receive an instance of the SummaryBuilder and it can register a summary reporter with it.

Rough code.

plugins: [({ runner }) => {
  runner.summaryBuilder.use(() => {
    return [{
      key: 'Snapshots',
      value: [
        '1 Failed',
        '3 obsolete',
        'tests/suite.test.ts',
        '  * suite name > snapshot 1',
        '  * suite name > snapshot 2',
        '  * suite name > snapshot 3'
      ]
    }]
  })
}]

• You can give any color to the strings
• Each array item from the value array will be printed in its own line, respecting the indentation.

Related commit japa/core@1144da5

[thetutlage]

Alright, so Japa v3 is out as a stable release.

Important

If you are using AdonisJS, you must stick with Japa v2 for a while, because v3 is ESM only and AdonisJS v5 apps uses CommonJS.
This is going to be a theme for the coming month as we start releasing everything approaching towards AdonisJS v6 release.

Sticking with older version?

• You can access the docs for Japa v2 here. http://v2.japa.dev/
• New AdonisJS projects will scaffold the application with Japa v2.

That's pretty much all!

Ready to migrate to v3?

• The new docs are available at japa.dev
• Here's the link to upgrade guide.

Facing issues?

• Just report them in the relevant repos and we will try to get back to them ASAP.

Enjoy your weekend 🏝️

[jlenon7]

in javascript, such test writing style isn't common. as far as I'm aware, none of the existing test framework allow this

Yea, I can't find anyone too. I think writing functional tests is the common because of JavaScript. In the beginning, people didn't have TypeScript, so this pattern has been established.

and, tell me if i'm wrong, but i can't see any real advantage or disadvantage in using either one or the other

The advantage I see when using class based tests is creating base test classes (BaseTest) that could have a lot of base helpers defined for each case. Like creating fake data, mocking some dependency of the IoC, generating JWT's. Other nice example would be to create a BaseCommandTest that would have common helpers to test commands, like a method to execute an ace command in a child process and return the stdout, stderr.

In the end of the day, all of these things could be done in functional style also. I think it's much more a matter of taste. I've had a lot of experience creating tests in both Java and Node.js (even more so in Node.js) and yet it feels much more natural for me to write tests in classes. The point I want to make here is the following; Since it's a matter of taste, and we already write object-oriented code, why not creating these decorators internally? I mean, official decorators are coming, and they are here to help, we should try to use them, not avoid it.

well, maybe the class-based approach is more verbose

This depends on each person, of course, but I prefer a verbose test written with classes than one written in a functional way (I find it much easier to read and maintain).

otherwise i think all the needed APIs are open in japa to allow the emergence of a third-party lib that will use a class-based approach

Is it possible to make Japa understand what is a class without changing the importer in configure() fn?

[thetutlage]

@ndianabasi Can you please create an issue for this with the reproduction code?

[jlenon7]

I was not talking about changing the syntax of the framework, but adding a new choice, decorators would be a plus. The user would be asked when running npm init japa command if he wants functional or class based tests.

The idea of creating a separate package for this is good, but I think it would make more sense to create something built-in, something like @japa/annotations or @japa/decorators or even exporting the decorators inside @japa/runner.

Perhaps I could create a POC and you guys can check them if it makes sense bringing it internally or not, what do you think?

[ndianabasi]

@thetutlage, While working on a reproduction repo, I realised that I used the wrong rule. I should have used unique instead of exists. 🤦‍♂️💔

Apologies for the confusion. 😄

[thetutlage]

@ndianabasi No worries, glad it worked

[Julien-R44]

I just thought of this: it would be nice to have a system so that plugins could inject what they want into the final summary printed by the reporters.

concrete use case: in @japa/snapshot, it would be cool to display, at the end of test execution, the number of snapshots that have been updated. For example Vitest print that :

we know that a snapshot comparison failed and that three snapshots are obsolete (i.e. they are present in the .ts.snap files but are not used in tests. So they can be removed )

[Julien-R44]

looks perfect. thanks dude !

[jlenon7]

One thing that is very annoying when working with Japa is that we need to handle every minimal change in your hooks (setup, teardown, etc).

Is there any way to run each test file in a worker thread? I've tried to instantiate the Worker in importer but it throws that I can't run tests without configuring it.

It would be nice to have a plugin to avoid this behavior is certain scenarios. Do you guys have any idea on how this could be implemented?

The only solution that I could think is that would be necessary to implemented a different runner, but I prefer the idea of this being a plugin instead.

[thetutlage]

@jlenon7 Not sure what you mean by "handle every minimal change in your hooks" and how running in worker threads is related to that.

Can you share some examples of the same?

[jlenon7]

Let's suppose I'm working with environment variables and configuration files. In setup() I would need to load the envs and configuration files, while in teardown() I would need to unload everything. If I don't do that, the behavior of the test will leak to the other one.

Another example is if I'm using decorators, I should always delete the metadata of decorated classes, which is very difficult to do as the application grows.

As mentioned before, everything could be handled in test hooks, but with worker threads we would have a clean environment for each test to run, leaving our test hooks cleaner with what is really worth running. I can't even count on my fingers how much time I've already spent debugging my tests and test groups to find which one was making another one not work.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[MANTENN]

@thetutlage @Julien-R44 are there any plans to change the return type of pluginfn from void to the type the plugin passes as a generic. I don't mind creating a RFC and doing the work—github link keeps on looping on the same page.

For ex:

export type PluginFn = (
  config: Required<Config>,
  runner: Runner,
  classes: {
    Test: typeof Test
    TestContext: typeof TestContext
    Group: typeof Group
  }
) => void | Promise<void>

Would be:

export type PluginFn = <T>(
  config: Required<Config>,
  runner: Runner,
  classes: {
    Test: typeof Test
    TestContext: typeof TestContext
    Group: typeof Group
  }
) => T | Promise<T>

This way we can easily extend chai and expect with custom modules such as jest-json-schema.

Currently, I get the prototype of expect to add jest-json-schema and add a type declaration provided by jest-json-schema to extend:

export const extendExpectWithJsonSchemaMatch = () => (_config: Required<Config>, _runner: Runner, classes: Classes) => {
  const expect = classes.TestContext.prototype.expect

  ApiResponse.macro('expect', function (expectedValue) {
    return expect(expectedValue);
  })
  ApiResponse.macro('toMatchSchema', function (matchSchema) {
    return expect(this.body()).toMatchSchema(matchSchema)
  })
  // TestContext created with https://www.npmjs.com/package/macroable
  // so we can tap into prototype after expect plugin has been loaded to add any plugin
  expect.extend(matchers)
  // have to export expect here
  return expect;
}

import { Expect } from "@japa/expect";

declare module 'expect' {
  interface AsymmetricMatchers {
    toMatchSchema(schema: object): void;
  }
  interface Matchers<R> {
    toMatchSchema(schema: object): R;
  }
}

declare module '@japa/api-client' {

  interface ApiResponse {
    toMatchSchema(schema: object): this
    expect<T = unknown>(actual: T): ReturnType<Expect>
  }
}

export { };

bootstrap.js

/**
 * File source: https://bit.ly/3ukaHTz
 *
 * Feel free to let us know via PR, if you find something broken in this contract
 * file.
 */

import type { Config } from '@japa/runner'
import TestUtils from '@ioc:Adonis/Core/TestUtils'
import { assert, runFailedTests, specReporter, apiClient } from '@japa/preset-adonis'
import { expect } from '@japa/expect'


import { extendExpectWithJsonSchemaMatch } from './extend-jest-and-api-client'
/*
|--------------------------------------------------------------------------
| Japa Plugins
|--------------------------------------------------------------------------
|
| Japa plugins allows you to add additional features to Japa. By default
| we register the assertion plugin.
|
| Feel free to remove existing plugins or add more.
|
*/


export const plugins: Required<Config>['plugins'] = [assert(), runFailedTests(), apiClient(), expect(), extendExpectWithJsonSchemaMatch()]

/*
|--------------------------------------------------------------------------
| Japa Reporters
|--------------------------------------------------------------------------
|
| Japa reporters displays/saves the progress of tests as they are executed.
| By default, we register the spec reporter to show a detailed report
| of tests on the terminal.
|
*/
export const reporters: Required<Config>['reporters'] = [specReporter()]

/*
|--------------------------------------------------------------------------
| Runner hooks
|--------------------------------------------------------------------------
|
| Runner hooks are executed after booting the AdonisJS app and
| before the test files are imported.
|
| You can perform actions like starting the HTTP server or running migrations
| within the runner hooks
|
*/
export const runnerHooks: Pick<Required<Config>, 'setup' | 'teardown'> = {
  setup: [
    () => TestUtils.ace().loadCommands(),
    // setup db and then rollback changes
    // https://youtu.be/PE0Jmu8Qqjo?t=2192
    // () => TestUtils.db().migrate(),
    // () => TestUtils.db().seed(),
  ],
  teardown: [],
}

/*
|--------------------------------------------------------------------------
| Configure individual suites
|--------------------------------------------------------------------------
|
| The configureSuite method gets called for every test suite registered
| within ".adonisrc.json" file.
|
| You can use this method to configure suites. For example: Only start
| the HTTP server when it is a functional suite.
*/
export const configureSuite: Required<Config>['configureSuite'] = (suite) => {
  if (suite.name === 'functional') {
    suite.setup(() => TestUtils.httpServer().start())
  }
}