storybookjs / addon-coverage Goto Github PK

Describe the bug

Describe the bug

nycrcPath does not seem to be working.

A clear and concise description of what the bug is.
I am trying to use nyc.config.js but setting this in nycrcPath doesnt do anything.

Steps to reproduce the behavior

1. add addons to /.storybook/main.ts
   {
   name: "@storybook/addon-coverage",
   options: {
   istanbul: {
   nycrcPath: './../nyc.config.sb.js',
   },
   },
   },

2. create nyc.config.sb.js at /nyc.config.sb.js

3. add lines to nyc.config.sb.js
   'use strict';
   module.exports = {
   "tempDirectory": './.nyc_customout'
   };

4. Run storybook test runner

Expected behavior

expected the temporary directory to use .nyc_customout temp directory during test runner but .nyc_out is being used which is the default temporary directory

Screenshots and/or logs

If applicable, add screenshots and/or logs to help explain your problem.

Environment

• OS: iOS
• Node.js version: v18.16.1
• NPM version: 9.5.1
• Browser (if applicable): [e.g. chrome, safari]
• Browser version (if applicable): [e.g. 22]
• Device (if applicable): [e.g. iPhone6]

Additional context

Add any other context about the problem here.

Describe the bug

I have some stories I want to exclude from coverage.
So I have this config:

const config = {
  addons: [{
    name: '@storybook/addon-coverage',
    options: {
      istanbul: {
        exclude: ["**/Button*"]  // <-- here
      }
    }
  }]
}

When running test-storybook --coverage,
those stories failed with:

    [Test runner] An error occurred when evaluating code coverage:
      The code in this story is not instrumented, which means the coverage setup is likely not correct.
      More info: https://github.com/storybookjs/test-runner#setting-up-code-coverage

Steps to reproduce the behavior

https://github.com/cyberuni/storybook-coverage-instrument-issue

pnpm i
pnpm storybook
pnpm test-storybook --coverage # another terminal

Expected behavior

Those stories should be excluded from the coverage and should not fail.

Screenshots and/or logs

Environment

• OS: Windows 11
• Node.js version: v18.12.1
• PNPM version: 7.18.0

Additional context

Describe the bug
I'm having a storybook with vue3 and vite. I want to measure my code coverage via istanbul when I run playwright tests.
Therefore I configured my storybook vite under .storybook/main.ts as follows:

const config: StorybookViteConfig = {
 ....
  typescript: {
    check: false,
    checkOptions: {},
  },
  framework: '@storybook/vue3',
  core: {
    builder: '@storybook/builder-vite',
  },
 ...
  async viteFinal(config, { configType }) {
    return mergeConfig(config, {
      plugins: [
        istanbul({
          include: 'src/*',
          exclude: ['node_modules', 'test/'],
          extension: ['.js', '.ts', '.vue'],
        }),
      ],
      ....
    })
  },
}
export default config

When I run storybook in dev mode with start-storybook -p 6006 and execute my playwright tests afterwards, the code is instrumented (coverage is not null) and a code coverage is measured.

However, when I build storybook and start the static build afterwards with these commands: build-storybook && http-server storybook-static --port 6006, the website works fine, but the coverage variable doesn't exist and no code coverage is measured

System
System:
OS: macOS 12.6
CPU: (10) arm64 Apple M1 Pro
Binaries:
Node: 16.16.0 - ~/.nvm/versions/node/v16.16.0/bin/node
npm: 8.11.0 - ~/.nvm/versions/node/v16.16.0/bin/npm
Browsers:
Chrome: 105.0.5195.102
Firefox: 104.0.2
Safari: 16.0
npmPackages:
@storybook/addon-a11y: 6.5.12 => 6.5.12
@storybook/addon-actions: 6.5.12 => 6.5.12
@storybook/addon-essentials: 6.5.12 => 6.5.12
@storybook/addon-interactions: 6.5.12 => 6.5.12
@storybook/addon-links: 6.5.12 => 6.5.12
@storybook/builder-vite: 0.2.2 => 0.2.2
@storybook/testing-library: 0.0.13 => 0.0.13
@storybook/vue3: 6.5.12 => 6.5.12

Additional context
Add any other context about the problem here.

Describe the bug

I'm using a pretty much vanilla create-react-app with Storybook v7 initialised and I can't get code coverage to record anything at all.

Steps to reproduce the behavior

To make this easy, I've shared a small repository all stood up ready to go:

1. git clone [email protected]:aquicore/chromatic-demo.git
2. Open the terminal and cd chromatic-demo
3. npm install
4. npm run storybook
5. (You may need to npx install playwright
6. (Separate terminal) npm run storybook-coverage

Expected behavior

I would expect for the Button.jsx file to have a fairly high coverage reported in the terminal in in the output file.

Screenshots and/or logs

Environment

• OS: Ubuntu 22.04.3
• Node.js version: v18.13.0
• NPM version: 8.19.3

Additional context

I've tried tweaking various things all over the place. The culprit I thought most likely was either something in react-app (I tried ejecting but it didn't do much with babel) or the fact I'm using @storybook/react-vite as the framework, but @storybook/react-webpack5 didn't appear to do anything either.

Thank you for the addon, it came to me just at the right time when I configured coverage reports for my recent codebase.

I spend the last few hours figuring out how to merge the coverage report from Storybook with the coverage report from Jest. I find the Storybook interaction testing convenient for React components and I don't want to duplicate the interaction tests in Jest, so when I run coverage for the Jest and then a Storybook, I obtain two coverages for the same codebase and I needed to merge them somehow to measure, how much is my codebase covered in total.

After fiddling with instanbul/nyc, which leads nowhere, I find out istanbul-merge package. When you have two or more JSON reports, it can merge them to give you a holistic overview.

The Storybook already produces JSON reports, so I only configured the Jest to use json reporter and then I merged the two reports together with

npx istanbul-merge --out coverage/coverage-merged.json coverage/storybook/coverage-storybook.json coverage/coverage-final.json

which gives you a merged json report. You can then covert it to the LCOV report with

npx nyc report -t coverage --reporter=lcov --reporter=text

Describe the bug

I upgrade Storybook to "7.6.4 ", then "DevTools"（Chrome）can not load source maps correctly. I created a new project try to reappear.

1. The new project works correctly.

2. Then I add '@storybook/addon-coverage' addon to the project. The "DevTools"（Chrome）can not load source maps again.

This is “man.ts”

3. When I delete "@storybook/addon-coverage" from "man.ts". The "DevTools" works correctly again.

`
import type { StorybookConfig } from "@storybook/react-webpack5";

const config: StorybookConfig = {
stories: ["../src//*.mdx", "../src//*.stories.@(js|jsx|mjs|ts|tsx)"],
addons: [
"@storybook/addon-links",
"@storybook/addon-essentials",
"@storybook/preset-create-react-app",
"@storybook/addon-onboarding",
"@storybook/addon-interactions",
// '@storybook/addon-coverage'
],
framework: {
name: "@storybook/react-webpack5",
options: {
builder: {
useSWC: true,
},
},
},
docs: {
autodocs: "tag",
},
staticDirs: ["../public"],
};
export default config;

`

The “name” component is used by Button component.

To Reproduce

No response

System

Storybook Environment Info:

  System:
    OS: macOS 13.6.1
    CPU: (16) x64 Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 18.17.1 - /usr/local/bin/node
    Yarn: 1.22.19 - /usr/local/bin/yarn
    npm: 9.6.7 - /usr/local/bin/npm <----- active
  Browsers:
    Safari: 17.1
  npmPackages:
    @storybook/addon-coverage: ^1.0.0 => 1.0.0 
    @storybook/addon-essentials: ^7.6.4 => 7.6.4 
    @storybook/addon-interactions: ^7.6.4 => 7.6.4 
    @storybook/addon-links: ^7.6.4 => 7.6.4 
    @storybook/addon-onboarding: ^1.0.9 => 1.0.9 
    @storybook/blocks: ^7.6.4 => 7.6.4 
    @storybook/preset-create-react-app: ^7.6.4 => 7.6.4 
    @storybook/react: ^7.6.4 => 7.6.4 
    @storybook/react-webpack5: ^7.6.4 => 7.6.4 
    @storybook/test: ^7.6.4 => 7.6.4 
    @storybook/test-runner: ^0.16.0 => 0.16.0 
    eslint-plugin-storybook: ^0.6.15 => 0.6.15 
    storybook: ^7.6.4 => 7.6.4

Additional context

No response

Describe the bug

Both examples are failing when test-storybook --coverage is run.

Steps to reproduce the behavior

1. Fresh repo clone and install all dependencies
2. Go to each example root
3. yarn storybook to start app
4. Run yarn test-storybook --coverage on separate terminal
5. See errors

Expected behavior

Successful test coverage report should appear on terminal.

Screenshots and/or logs

Environment

• OS: macOS Ventura 13.5.2
• Node.js version: v18.17.0
• NPM version: v9.6.7

Additional context

Add any other context about the problem here.

Describe the bug

When using storybook-addon-swc, this addons complains about the code not being implemented.

Steps to reproduce the behavior

1. Add https://storybook.js.org/addons/storybook-addon-swc to a project
2. Use this addon

Expected behavior

Coverage works

Screenshots and/or logs

All tests return:

   [Test runner] An error occurred when evaluating code coverage:
             The code in this story is not instrumented, which means the coverage setup is likely not correct.
             More info: https://github.com/storybookjs/test-runner#setting-up-code-coverage

Environment

• OS: MacOS and Linux
• Node.js version: 18.X
• NPM version: yarn 1.22.19

Describe the bug

Tests are passed by default, but after runing the --coverage command, it throw error:

Steps to reproduce the behavior

1. run yarn test-storybook --coverage

Expected behavior

A clear and concise description of what you expected to happen.

Screenshots and/or logs

Here are the settings in main.js

The @storybook/addon-coverage plugin only works for Webpack5 projects if Babel is used as a compiler. In the future, we want to support other compilers as well.

Goal

@storybook/addon-coverage should work for all Webpack5 projects, regardless of whether Babel or e.g. SWC is used.

Describe the bug

--coverage flag makes all test fail

Steps to reproduce the behavior

1. Create a simple {Component}.stories.ts file
2. Run storybook dev -p 3001, followed by test-storybook --url http://127.0.0.1:3001 --browsers firefox chromium webkit
3. All the tests passed
4. Add the flag --coverage (test-storybook --url http://127.0.0.1:3001 --browsers firefox chromium webkit --coverage)
5. All the tests failed

Expected behavior

adding the flag --coverage shouldn't make all tests faIil

Screenshots and/or logs

Results without --coverage

Results with --coverage

The error log:

Environment

• OS: Windows 11
• Node.js version: v20.11.1
• NPM version: 10.2.4 (pnpm version: 8.15.4)
• @storybook/addon-coverage: v1.0.1
• @storybook/test-runner: v0.17.0

Additional context

.storybook/main.ts

import type { StorybookConfig } from '@storybook/react-vite';

const config: StorybookConfig = {
  stories: ['../src/**/*.mdx', '../src/**/*.stories.@(js|jsx|mjs|ts|tsx)'],
  addons: [
    '@storybook/addon-links',
    '@storybook/addon-essentials',
    '@storybook/addon-a11y',
    '@storybook/addon-interactions',
    '@storybook/addon-coverage',
  ],
  framework: {
    name: '@storybook/react-vite',
    options: {},
  },
  docs: {
    autodocs: 'tag',
  },
};
export default config;

.storybook/test-runner.ts

import type { TestRunnerConfig } from '@storybook/test-runner';
import { getStoryContext } from '@storybook/test-runner';

import { injectAxe, checkA11y, configureAxe } from 'axe-playwright';

/*
 * See https://storybook.js.org/docs/writing-tests/test-runner#test-hook-api
 * to learn more about the test-runner hooks API.
 */
const config: TestRunnerConfig = {
  async preVisit(page) {
    await injectAxe(page);
  },
  async postVisit(page, context) {
    // Get the entire context of a story, including parameters, args, argTypes, etc.
    const storyContext = await getStoryContext(page, context);

    // Apply story-level a11y rules
    await configureAxe(page, {
      rules: storyContext.parameters?.a11y?.config?.rules,
    });

    await checkA11y(page, '#storybook-root', {
      detailedReport: true,
      detailedReportOptions: {
        html: true,
      },
    });
  },
};

export default config;

i'm assuming this has something to do with the a11y config in the test-runner, but i would like to know what's wrong.
Edit: just removed @storybook/addon-a11y from the equation, still the same issue

Hello, I've recently started using Storybook built in test-runner in combo with addon-coverage.

I was wondering if there's a way to define some threshold percentage and eventually make tests fail if these thresholds are not met. In Jest this can be achieved defining coverageThreshold as follows:

  coverageThreshold: {
    global: {
      branches: 80,
      functions: 80,
      lines: 80,
      statements: 80,
    },
  },

I've tried using the options.istanbul object of addon-coverage in main.js without success (https://storybook.js.org/docs/react/writing-tests/test-coverage#configure)

How can I achieve something similar with test-runner and addon-coverage?
Thank you

Describe the bug

I have 2 yarn workspaces, one containing my app and another containing storybook.

Steps to reproduce the behavior

1. Setup react in one yarn workspace, create react components and stories here
2. Setup a separate yarn workspace and install storybook there
3. Set storybook main.js stories to be found in the react workspace
4. Start storybook
5. Run tests with coverage -> Error: The code in this story is not instrumented, which means the coverage setup is likely not correct.

// main.js
module.exports = {
  stories: [
    "../../components/src/**/*.stories.mdx",
    "../../components/src/**/*.stories.@(js|jsx|ts|tsx)",
  ],
  addons: [
    "@storybook/addon-links",
    "@storybook/addon-essentials",
    "@storybook/addon-interactions",
    '@storybook/addon-coverage',
  ],
  framework: "@storybook/react",
  features: {
    interactionsDebugger: true,
  },
}

I have also ejected storybook to get the tests to run:

// test-runner-jest.config.js
const {getJestConfig} = require('@storybook/test-runner');

module.exports = {
  // The default configuration comes from @storybook/test-runner
  ...getJestConfig(),
  /** Add your own overrides below
   * @see https://jestjs.io/docs/configuration
   */
  rootDir: '../components/',
  testEnvironmentOptions: {
    'jest-playwright': {
      browsers: ['chromium'],
    },
  },
};

Expected behavior

Tests with coverage work.

Screenshots and/or logs

Error:

 [Test runner] An error occurred when evaluating code coverage:
      The code in this story is not instrumented, which means the coverage setup is likely not correct.
      More info: https://github.com/storybookjs/test-runner#setting-up-code-coverage

Environment

• OS: Windows
• Node.js version: v14.21.1
• NPM version: 6.14.17
• Browser (if applicable): chrome (although storybook app isnt open)

Describe the bug

Storybook 7.6 introduced a new --test flag for building Storybook with less of the fluff that is not needed for testing environments.
When you try to run test-storybook with --coverage on a storybook that was built with --test, I get the following error:

The code in this story is not instrumented, which means the coverage setup is likely not correct.

Steps to reproduce the behavior

1. Build a Storybook with --test
2. Run test-storybook on that with --coverage
3. Rebuild the same Storybook without --test
4. Test storybook passes

Expected behavior

I believe that the istanbul plugin is not being properly injected when --test is enabled. I don't get the [addon-coverage] Adding istanbul plugin to Vite config message when I use it. The coverage should work when --test is enabled.

Describe the bug

Everything runs smoothly with pnpm test-storybook. On the other hand, pnpm test-storybook --coverage returns an empty coverage report.

Steps to reproduce the behavior

1. Run pnpm storybook
2. Run pnpm test-storybook --coverage
3. Console Coverage Summary will be empty (no files found)
4. coverage/storybook/coverage-storybook.json only contains {}

Expected behavior

Show **/*.tsx files on the report.

Screenshots and/or logs

.storybook/main.ts

package.json

EMPTY Coverage (Console) Report

EMPTY Coverage (File) Report

Environment

• OS: Ubuntu 22.04.2 (WSL)
• Node.js version: v20.9.0
• PNPM version: 8.10.2 -> Also happens with NPM version: 10.1.0
• Browser (if applicable): chrome
• Browser version (if applicable): Not Sure (installed through playwright pnpm exec playwright install + pnpm exec playwright install-deps)
• Device (if applicable): Not Applicable

Describe the bug

project use Vue3(^3.3.4) + vite(^5.0.9);

strorybook use VueJsx (https://github.com/vitejs/vite-plugin-vue)

execute ccmmand "test-storybook -- --coverage", coverage does not include code for vue files.

Steps to reproduce the behavior

1.storybok main config:

2.packages.json:

3. execute result:
   

Expected behavior

coverage include code for vue files.

Screenshots and/or logs

see above

Environment

• OS: Windows 11
• Node.js version: v16.14.2np
• NPM version: 8.5.0
• Browser (if applicable): chrome
• Browser version (if applicable): 121.0.6167.85

Additional context

Add any other context about the problem here.

When the addon is included the story book shows an error localFont is not defined

Steps to reproduce the behavior

1. Configure next/font/local in any of the react component
2. Include the addon '@storybook/addon-coverage',
3. Check the story which includes localFonts
4. See error (Refer to image below)
5. As soon as I remove '@storybook/addon-coverage', storybook works as expected, but coverage is broken!

Expected behavior

It should include the local fonts

Screenshots and/or logs

Environment

• OS: mac
• Node.js version: [e.g. v12.17.0]
• NPM version: v18.15.0
• Browser (if applicable): chrome
• Browser version (if applicable): 113.0.5672.92
• Device (if applicable): macbook

Additional context

package versions
"next": "^13.4.2",
"@storybook/addon-coverage": "^0.0.8",
"@storybook/nextjs": "^7.0.12",

Describe the bug

Using examples and running yarn test-storybook --coverage on vite example all tests gives error:
The code in this story is not instrumented

Steps to reproduce the behavior

1. Pull repo
2. Install node_modules
3. Execute yarn storybook
4. Execute yarn test-storybook --coverage
5. See error

Expected behavior

Working like a charm

Screenshots and/or logs

Environment

• OS: macOS 13.1
• Node.js version: 16.13.0
• NPM version: 8.12.2

Causes downstream code to report errors

Describe the Issue

I want to merge cumulatively storybook coder coverage reports with reports from vitest.

This one was generated using vitest coverage-final.json
This one was generated using the storybook addon coverage-storybook.json

Here are my configs for vitest

    coverage: {
      all: true,
      clean: false,
      exclude: [
      ...
      ],
      provider: 'istanbul',
      reporter: ['json', 'html']
    }
  },
// my command 
"test:coverage": "vitest --run --coverage",

For storybook I used:

addons: [
 {
      name: '@storybook/addon-coverage',
      options: {
        istanbul: {
          exclude: [
....
          ]
        }
      }
    },
    ]
    "test-storybook:coverage": "test-storybook --coverage --json --coverageDirectory=./coverage",

At the end I use nyc to merge the output files:

"test-all:coverage": "nyc merge coverage coverage/merged/coverage.json && nyc report -t coverage/merged --report-dir coverage/merged --reporter=html --reporter=cobertura",

In my merged coverage report I can then see:

However, when looking at only the unit test report, I can see that these lines are being hitted.

It seems that the storybook addon and vitest generate slightly different line and column for the statements.
For example in one component vitest said that there is a stmt on line 38 start at col 18 and ends on the same line.

            "2": {
                "start": {
                    "line": 38,
                    "column": 18
                },
                "end": {
                    "line": 38,
                    "column": null
                }
            },

However, storybook coverage report says the statement is on line 38 in column 2:

      "3": {
        "start": {
          "line": 38,
          "column": 2
        },
        "end": {
          "line": 38,
          "column": null
        }
      },

Steps to reproduce the behavior

1. Create a vite react project
2. Setup storybook and react testing library
3. create code coverage reports with istanbul
4. create code coverage report with this addon
5. use nyc to merge them

Expected behavior

That I can somehow define how code is instrumented or which version of Istanbul is used. So I can merge reports from multiple testing tools.

Environment

• OS: Ubuntu
• Node.js version: v16.20.0
• NPM version: ^9

Additional context

Here are the current versions of my package.json

    "vitest": "^0.29.8"
    "@vitest/coverage-istanbul": "^0.34.5",
    // looking at node_modules I can see that coverage-istanbul requires the following:
    "istanbul-lib-coverage": "^3.2.0",
    "istanbul-lib-instrument": "^6.0.0",
    "istanbul-lib-report": "^3.0.1",
    "istanbul-lib-source-maps": "^4.0.1",
    "istanbul-reports": "^3.1.5",


    
    "@storybook/addon-coverage": "^1.0.0",
    // looking at node_modules I can see:
    "istanbul-lib-instrument": "^6.0.1",
    "vite-plugin-istanbul": "^3.0.1" 
    // following vite-plugin-istanbul I can see that it uses:
    "istanbul-lib-instrument": "^5.1.0",

So now I am considering if the issue comes from different versions?

I also posted here about it:
storybookjs/storybook#25667
and commented here:
istanbuljs/nyc#1302

Describe the bug

Coverage is not reported for files where JSX is used.

I've set up a minimal repository that recreates the issue with the same config - https://github.com/jMalojlo/sb-repro.
It's a basic button component with a story containing a play function to 'test' a click handler.
The component itself contains some JSX and imports some jsx and non-jsx module. What we see in the coverage report is that only the files without JSX are properly counted.
We're running an ejected CRA and I assume it's a Babel config issue, but can't figure this out myself.

Steps to reproduce the behavior

1. Clone https://github.com/jMalojlo/sb-repro
2. npm install && npx playwright install chrome
3. npm run storybook
4. npm run storybook-test runs tests with --coverage
5. npm run generate-storybook-coverage nyc report with lcov
6. open ./coverage/storybook/lcov-report/index.html to inspect the coverage HTML file
7. 

Environment

• Node.js version: 20.9.0
• NPM version: 10.1.0

Describe the bug

code comments to hide lines from being included in coverage are not working.

Steps to reproduce the behavior

can be seen from a clean install (storybook 7.0.12)

(package release below for reference)

pnpm create vite

• react
• typescript

pnpm install

pnpm dlx storybook@latest init

• eslint Y

pnpm add --save-dev @storybook/test-runner

pnpm add --save-dev @storybook/addon-coverage

package.json

"scripts": {
    "test-storybook": "test-storybook --coverage"
  }

.storybook/main.ts

addons: [
    '@storybook/addon-coverage',
  ],

terminal1: pnpm storybook

terminal2: pnpm test-storybook

nyc not installed!

pnpm add --save-dev nyc

update to latest testing library

pnpm add --save-dev @storybook/[email protected]

rerun test

coverage shows Page.tsx uncovered line 17

add parsing hint at line 17 (pushes code to line 18)

/* istanbul ignore next */

rerun test

coverage shows Page.tsx uncovered line 18-19

move comment so line is now:

onLogin={
          /* istanbul ignore next */
          () => setUser({ name: "Jane Doe" })
        }

rerun test

coverage shows Page.tsx uncovered line 19

Expected behavior

It should remove/ hide the line of code from coverage report

Screenshots and/or logs

Environment

• OS: mac 13.3.1
• Node.js version: 18.16.0
• NPM version: 9.5.1 (pnpm 8.5.1)
• Browser (if applicable): [e.g. chrome, safari]
• Browser version (if applicable): [e.g. 22]
• Device (if applicable): [e.g. iPhone6]

Additional context

  "devDependencies": {
    "@storybook/addon-coverage": "^0.0.8",
    "@storybook/addon-essentials": "^7.0.12",
    "@storybook/addon-interactions": "^7.0.12",
    "@storybook/addon-links": "^7.0.12",
    "@storybook/blocks": "^7.0.12",
    "@storybook/react": "^7.0.12",
    "@storybook/react-vite": "^7.0.12",
    "@storybook/test-runner": "^0.10.0",
    "@storybook/testing-library": "^0.1.0",
    "@types/react": "^18.0.28",
    "@types/react-dom": "^18.0.11",
    "@typescript-eslint/eslint-plugin": "^5.57.1",
    "@typescript-eslint/parser": "^5.57.1",
    "@vitejs/plugin-react": "^4.0.0",
    "eslint": "^8.38.0",
    "eslint-plugin-react-hooks": "^4.6.0",
    "eslint-plugin-react-refresh": "^0.3.4",
    "eslint-plugin-storybook": "^0.6.12",
    "nyc": "^15.1.0",
    "prop-types": "^15.8.1",
    "storybook": "^7.0.12",
    "typescript": "^5.0.2",
    "vite": "^4.3.2"
  }