Coverage bugs when re-running same function

[silverbackdan]

Describe the bug

It appears when re-running tests on the same function, only the last run test result is kept on conditionals.

It seems to be the same with Istanbul and C8 so I'm not sure where the issue lies.

Reproduction

My middleware:

import { defineNuxtRouteMiddleware, useNuxtApp } from '#app'

export default defineNuxtRouteMiddleware(async (to) => {
  const { $cwa } = useNuxtApp()

  if (to.meta.cwa === false) {
    return
  }

  await $cwa.fetcher.fetchRoute(to.path)
})

My tests:

import { afterEach, describe, expect, test, vi, beforeEach } from 'vitest'
import { RouteLocationNormalizedLoaded } from 'vue-router'
import * as nuxt from '#app'
import routeMiddleware from './route-middleware'

function createToRoute (cwa?: boolean|undefined): RouteLocationNormalizedLoaded {
  return {
    name: '',
    path: '/',
    fullPath: '/',
    query: {},
    hash: '',
    matched: [],
    params: {},
    meta: {
      cwa
    },
    redirectedFrom: undefined
  }
}

describe('Test route middleware', () => {
  const fetchRouteFn = vi.fn()
  // @ts-ignore
  vi.spyOn(nuxt, 'useNuxtApp').mockImplementation(() => {
    return {
      $cwa: { fetcher: { fetchRoute: fetchRouteFn } }
    }
  })
  // @ts-ignore
  vi.spyOn(nuxt, 'defineNuxtRouteMiddleware').mockImplementation(() => {
    return {}
  })

  afterEach(() => {
    fetchRouteFn.mockReset()
  })

  test('Test route middleware is enabled by default', () => {
    const toRoute = createToRoute()
    routeMiddleware(toRoute, toRoute)
    expect(fetchRouteFn).toHaveBeenCalledTimes(1)
    expect(fetchRouteFn).toHaveBeenCalledWith(toRoute.path)
  })
  test('Test route middleware can be set to true', () => {
    const toRoute = createToRoute(true)
    routeMiddleware(toRoute, toRoute)
    expect(fetchRouteFn).toHaveBeenCalledTimes(1)
    expect(fetchRouteFn).toHaveBeenCalledWith(toRoute.path)
  })
  test('Test route middleware can be disabled', () => {
    const toRoute = createToRoute(false)
    routeMiddleware(toRoute, toRoute)
    expect(fetchRouteFn).not.toHaveBeenCalled()
  })
})

Clover for the line:

if (to.meta.cwa === false) {

comes out as:

<line num="6" count="2" type="cond" truecount="1" falsecount="0"/>

If I run the test where the condition is false as the last test, then it shows as falsecount="1"
But as you can see, it was run multiple times so really the truecount and falsecount should equate to count

Any thoughts if this is a vitest bug or coverage providers?

System Info

System:
    OS: macOS 12.6
    CPU: (10) arm64 Apple M1 Max
    Memory: 656.52 MB / 64.00 GB
    Shell: 5.8.1 - /bin/zsh
  Binaries:
    Node: 16.16.0 - /usr/local/bin/node
    Yarn: 1.22.19 - ~/.npm-packages/bin/yarn
    npm: 8.11.0 - /usr/local/bin/npm
  Browsers:
    Chrome: 108.0.5359.94
    Safari: 16.0
  npmPackages:
    vitest: ^0.25.4 => 0.25.5

Used Package Manager

npm

Validations

• Follow our Code of Conduct
• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
• The provided reproduction is a minimal reproducible example of the bug.

[AriPerkkio]

Without reproduction setup it is hard to help, but could you try if setting test.coverage.cleanOnRerun as true fixes the issue? Maybe the results from earlier run are merged into the new ones. By "re-running" you mean watch-mode, right?

[silverbackdan]

Hi, I'm sorry I wasn't clear. Hard to get this info correct when trying to describe to someone coming into it cold.

The reproduction above should be all that's needed I think. What I meant by re-run is that I have 3 tests there running routeMiddleware to test for different results. The clover is showing that the conditional has been called twice, but only shows truecount as 1 and falsecount as 0 - the truecount and falsecount should be equal to count.

Basically it ends up showing that I've only got partial coverage on that line.

I've actually ended up noticing many issues with the partial coverage with vitest and c8/instanbul.

Sometimes c8 is detecting a function returned as a conditional as well.

I'm not sure if it'd help, but I can share my codecov URL and you'd see in many instances, the code within a simple if block is run, and also the code which would run if the if is false, but the if statement itself has partial coverage.

https://app.codecov.io/github/components-web-app/cwa-nuxt-module/tree/feat%2Fmercure-and-api-docs/

This is the specific file results as you can see visually:
https://app.codecov.io/github/components-web-app/cwa-nuxt-module/blob/feat%2Fmercure-and-api-docs/src/runtime/route-middleware.ts

This URL will change at some point as I'm implementing the tests on this feature branch which will be merged when work is complete on it.

[silverbackdan]

Here is a minimal reproduction:
https://github.com/components-web-app/cwa-nuxt-module/tree/repro/coverage

You'll see clover is produced with

<line num="2" count="3" type="cond" truecount="1" falsecount="0"/>

So the truecount and falscount are being replaced and not increased each time the line is called.

This results in tools such as codecov to deem the line as partially tested

Additionally, using c8 the export line comes out as a conditional statement that is only partially covered as well.

Edit:
I tried doing a little digging and found in the module instanbul-resports the truecount and falsecount being populated by this:

 if (branchDetail) {
    attrs.type = 'cond';
    attrs.truecount = branchDetail.covered;
    attrs.falsecount = branchDetail.total - branchDetail.covered;
}

I'm just not sure what the whole process is for the coverage functionality to try and trace these values and where it's all being set etc. I've never looked into the coverage side or this functionality yet and a bit snowed under just trying to write the tests and build a big application atm. So I'm unsure whether the issue is in vitest or the coverage providers.

[AriPerkkio]

@silverbackdan I haven't yet debugged this, but does this istanbul-reporter issue describe same problem? istanbuljs/istanbuljs#695

[silverbackdan]

Thanks so much, yes, I think this will be an example of the same issue so I'll reference my reproduction there.
Thanks for finding this, much appreciated.