[Suggestion] Add a CLI option to only test stories with a play function

[jeremytenjo]

I would like to propose the addition of a new CLI option that would allow users to test only stories that contain a "play" function. This would be particularly helpful for those of us who are working with large Storybook projects and want to quickly test those stories.

Thank you

[yannbf]

Hey there! This sounds like a good idea, in fact it should already be possible with the following ejected test-runner config:

// test-runner-jest.config.js
const { getJestConfig } = require('@storybook/test-runner')
/**
 * @type {import('@jest/types').Config.InitialOptions}
 */
module.exports = {
  // The default configuration comes from @storybook/test-runner
  ...getJestConfig(),
  testNamePattern: 'play-test', // only runs tests with play function
}

However...

This seems to break the test-runner, and I believe it's a bug in jest-playwright as I noticed here. If anyone would like to take a look into why the bug happens and is willing to open a PR, that is very welcome!

[jrencz]

Just for the record: I'm trying to add test-runner to a Storybook I maintain and for several days I'm struggling with the problem of unstable tests

For some of them I narrowed the problem down to "They touch document so without jsdom it may be a no-go" (and test-runner doesn't allow to switch to jest-environment-jsdom - I hope to have time later today to write down my findings in a distinct issue which I will link here) so I went for "let's have something than nothing" and started looking for options to skip tests by name pattern. I ended up finding this issue

Other options I considered to detect it's the runner and render fake instead of problematic story (if smoke-test fails) or a blank play if it's play-test, but it seems like not that good idea as hinting Jest to skip directly (which I imagine will also produce better report: not "all ok, won't tell which ones were actual and which ones were faked" but "some OK, other skipped")

[panvourtsis]

hi @jrencz, i made a PR to the emotion about the issue. Also there is a suggested way for test-runner config emotion-js/emotion#3122

[jrencz]

@panvourtsis great news!

I see that package @storybook/theming uses emotion (@emotion/use-insertion-effect-with-fallbacks), but I don't quite get how is that related to the test runner?

Do I get it right that after your PR gets merged, then released, new version of @emotion/use-insertion-effect-with-fallbacks needs to be adopted in Storybook and only then I, as end user of test-runner may benefit from it, right?

[kjanturaWongaPL]

@JReinhold What is happening about planned changes in Storybook's test-runner (testing stories with Vitest): https://youtu.be/veCxKeLl35A?feature=shared&t=9209, do we have to wait for Storybook 8.0?

[yannbf]

Hey everyone! We just released a test filtering feature in the test-runner v0.15.0, which you can read here.

Stories with a play function automatically have a play-fn tag, so for running tests against only those stories, you can do like this:

yarn test-storybook --includeTags="play-fn"

And if you only want to run smoke tests instead, you can do the inverse:

yarn test-storybook --excludeTags="play-fn"

# or this

yarn test-storybook --skipTags="play-fn"

Please try it out and provide feedback! Thank you!

[jeremytenjo]

Thank you Storybook team ❤️