Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(vitest): support hooks in bench runner #5076

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

feat(vitest): support hooks in bench runner #5076

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion packages/runner/src/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
export { startTests, updateTask } from './run'
export { startTests, updateTask, callSuiteHook, callCleanupHooks } from './run'
export { test, it, describe, suite, getCurrentSuite, createTaskCollector } from './suite'
export { beforeAll, beforeEach, afterAll, afterEach, onTestFailed } from './hooks'
export { setFn, getFn, getHooks, setHooks } from './map'
Expand Down
2 changes: 1 addition & 1 deletion packages/runner/src/run.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ async function sendTasksUpdate(runner: VitestRunner) {
}
}

async function callCleanupHooks(cleanups: HookCleanupCallback[]) {
export async function callCleanupHooks(cleanups: HookCleanupCallback[]) {
await Promise.all(cleanups.map(async (fn) => {
if (typeof fn !== 'function')
return
Expand Down
18 changes: 15 additions & 3 deletions packages/vitest/src/runtime/runners/benchmark.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import type { Suite, Task, VitestRunner, VitestRunnerImportSource } from '@vitest/runner'
import { updateTask as updateRunnerTask } from '@vitest/runner'
import type { HookCleanupCallback, Suite, Task, VitestRunner, VitestRunnerImportSource } from '@vitest/runner'
import { callCleanupHooks, callSuiteHook, updateTask as updateRunnerTask } from '@vitest/runner'
import { createDefer, getSafeTimers } from '@vitest/utils'
import { getBenchFn, getBenchOptions } from '../benchmark'
import { getWorkerState } from '../../utils'
Expand Down Expand Up @@ -35,6 +35,10 @@ async function runBenchmarkSuite(suite: Suite, runner: NodeBenchmarkRunner) {
benchmarkSuiteGroup.push(task)
}

let beforeAllCleanups: HookCleanupCallback[] = []

beforeAllCleanups = await callSuiteHook(suite, suite, 'beforeAll', runner, [suite])

if (benchmarkSuiteGroup.length)
await Promise.all(benchmarkSuiteGroup.map(subSuite => runBenchmarkSuite(subSuite, runner)))

Expand Down Expand Up @@ -90,7 +94,12 @@ async function runBenchmarkSuite(suite: Suite, runner: NodeBenchmarkRunner) {
await task.warmup()
tasks.push([
await new Promise<BenchTask>(resolve => setTimeout(async () => {
resolve(await task.run())
let beforeEachCleanups: HookCleanupCallback[] = []
beforeEachCleanups = await callSuiteHook(benchmark.suite, benchmark, 'beforeEach', runner, [benchmark.context, benchmark.suite])
const result = await task.run()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use tinybench's hooks? I don't have any issues with this, but just asking.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting idea!
I think we might need to refine what each hook does, since in benchmarking there is one more level of hooks.

At the moment, with my implementation, this is what would happen (I added some comment at the end to see the execution order):

import {
  afterAll,
  afterEach,
  beforeAll,
  beforeEach,
  bench,
  describe,
} from 'vitest'

describe('hooks', () => {
  beforeAll(() => {
    console.log('beforeAll')
  })
  beforeEach(() => {
    console.log('beforeEach')
  })
  afterEach(() => {
    console.log('afterEach')
  })
  afterAll(() => {
    console.log('afterAll')
  })

  bench('one', () => {
    console.log('bench one')
  }, {
    setup: () => {
      console.log('setup')
    },
    teardown: () => {
      console.log('tearDown')
    },
    iterations: 10,
    time: 1,
  })
})

// when the bench is executed, the following should be logged

// beforeAll
// beforeEach
// setup     x10
// bench one x10
// tearDown  x10
// afterEach
// beforeAll

Do you think it is what should be expected?

As far as using the hooks from tinybench, I can map the *Each hooks from vitest to the Task *All hooks from tinybench.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh interesting edge case, nice, I think let's keep it as is and this should be documented somewhere.

After all, @sheremet-va might have some interesting ideas and insights around this.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it makes more sense to use Vitest's hooks. They are shown in the reporter for example if they take too long.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, I will go and try to add a test that verify the order of execution of the hooks

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will switch the PR to draft for the moment, and mark it as ready when I will have added the tests

await callSuiteHook(benchmark.suite, benchmark, 'afterEach', runner, [benchmark.context, benchmark.suite])
await callCleanupHooks(beforeEachCleanups)
resolve(result)
})),
benchmark,
])
Expand All @@ -115,6 +124,9 @@ async function runBenchmarkSuite(suite: Suite, runner: NodeBenchmarkRunner) {
await defer
}

await callSuiteHook(suite, suite, 'afterAll', runner, [suite])
await callCleanupHooks(beforeAllCleanups)

function updateTask(task: Task) {
updateRunnerTask(task, runner)
}
Expand Down
4 changes: 3 additions & 1 deletion test/benchmark/specs/runner.test.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ if (existsSync('./bench.json'))
rmSync('./bench.json')

try {
await startVitest('benchmark', ['base.bench', 'mode.bench', 'only.bench'], {
await startVitest('benchmark', ['base.bench', 'mode.bench', 'only.bench', 'hooks.bench'], {
watch: false,
})
}
Expand All @@ -28,6 +28,8 @@ await test('benchmarks are actually running', async () => {
assert.ok(resultJson.testResults.a2, 'a2 is in results')
assert.ok(resultJson.testResults.b3, 'b3 is in results')
assert.ok(resultJson.testResults.b4, 'b4 is in results')
assert.ok(resultJson.testResults.hooks, 'hooks is in results')
assert.ok(resultJson.testResults['hooks-cleanup'], 'hooks-cleanup is in results')
})

await test('doesn\'t have skipped tests', () => {
Expand Down
60 changes: 60 additions & 0 deletions test/benchmark/test/hooks.bench.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
import {
afterAll,
afterEach,
beforeAll,
beforeEach,
bench,
describe,
expect,
} from 'vitest'

describe('hooks', () => {
let cleanUpCount = 0
describe('run', () => {
beforeAll(() => {
cleanUpCount += 10
})
beforeEach(() => {
cleanUpCount += 1
})
afterEach(() => {
cleanUpCount -= 1
})
afterAll(() => {
cleanUpCount -= 10
})

bench('one', () => {
expect(cleanUpCount).toBe(11)
})
bench('two', () => {
expect(cleanUpCount).toBe(11)
})
})
bench('end', () => {
expect(cleanUpCount).toBe(0)
})
})

describe('hooks-cleanup', () => {
let cleanUpCount = 0
beforeAll(() => {
cleanUpCount += 10
return () => {
cleanUpCount -= 10
}
})
beforeEach(() => {
cleanUpCount += 1
return () => {
cleanUpCount -= 1
}
})

bench('one', () => {
expect(cleanUpCount).toBe(11)
})
bench('two', () => {
expect(cleanUpCount).toBe(11)
})
})