How to dynamically skip tests with a custom runner

[nweinbach-binagora]

Hi everyone! After exploring some options without success, I'm looking for ideas.

The thing is that I was able to implement a custom runner to skip some tests given by a JSON file dynamically (on my GH job I first hit an API to get that file - if a test exhibits flaky behavior it will be there).

function getFullTestName(test: RunnerTask): string {
	if (test.suite === undefined || test.suite === null) {
		return `${test.name}`;
	} else {

		return `${getFullTestName(test.suite)} > ${test.name}`;
	}
}

function isQuarantined(test: RunnerTask) {
	const testName = getFullTestName(test);
	const quarantinedTest = quarantineFile?.find((test: { name: any; }) => test.name === testName);

	return Boolean(quarantinedTest);
}

export default class QuarantineVitestRunner extends VitestTestRunner {
    constructor(public override config: SerializedConfig) {
      super(config);
    }

    override async onBeforeRunTask(test: RunnerTask) {
        if (isQuarantined(test)) {
            console.log('[QUARANTINED]', getFullTestName(test));
            test.mode = 'skip';            
        }

        super.onBeforeRunTask(test);
    }
}

There are a couple things I observe:

1. It would be nice to be able to call the built in getTestName instead of having to redefine the method. A possible change in the config about naming will impact this.
2. I'm able to skip, but the reporters doesn't pick up the "mode" or result correctly, as it's not reported the same way as the tests statically skipped (see the test with an arrow down and with a dot below in the output). I guess this is because I can't call the updateTask to reflect the changes in the test mode. This way, I'm forced to implement a reporter overriding reportTestSummary to change the implementation of getTests and apply a conditional checking again to see if the test is quarantined.

Current output:

stdout | test/sample.spec.ts > fastDoesIntersect
[QUARANTINED] first group > returns false if the input arrays contain `NaN` values

 ✓ test/sample.spec.ts (8)
   ✓ first group (5)
     ✓ returns false if the input arrays do not intersect
     ✓ returns false if one input array is empty
     ✓ returns false if both the input arrays are empty
     ✓ returns true if the input arrays contain `undefined` values
     · returns false if the input arrays contain `NaN` values
   ✓ second group (3)
     ✓ extra (3)
       ✓ works with number
       ↓ works with boolean [skipped]
       ✓ works with string

 Test Files  1 passed (1)
      Tests  6 passed | 1 skipped (8)
   Start at  10:57:40
   Duration  6ms

Any ideas to do this better? I'm not a TypeScript expert, so maybe I'm missing something...

[sheremet-va]

It would be nice to be able to call the built in getTestName instead of having to redefine the method. A possible change in the config about naming will impact this.

I think you can import this function from @vitest/runner/utils. Generally speaking, the runner API is very low level and we didn't have a lot of feedback to refine it.

I'm able to skip, but the reporters doesn't pick up the "mode" or result correctly, as it's not reported the same way as the tests statically skipped

You are right, updateTask is not called if you skip the test in onBeforeRunTask.

Any ideas to do this better? I'm not a TypeScript expert, so maybe I'm missing something...

What you can do is call skip inside beforeEach:

beforeEach(t => {
  if(t.task.name === 'skipped name') {
    t.skip()
  }
})

[nweinbach-binagora]

Hi Vladimir! Thank you for your answer.

I can import getTestName but I can't override it to include the conditional skipping logic.
Also, I thought about the beforeEach but I'd like to do a global thing for my test files, without the need to edit all the test files.

I'm now implemeting a custom reporter to see if I can do the same check and print it out differently.

[sheremet-va]

beforeEach can be called in a setupFile, you don’t have to write it in every file

[nweinbach-binagora]

Thank you! Turned out to be that simple