Branch coverage false positive for exit context

[jaraco]

Describe the bug

In twisted/towncrier#591 (comment), I encountered what appears to be a spurious false positive. The coverage report is indicating that 20->exit is not covered. even though that line seems to be unequivocally covered (how could that context not exit?).

To Reproduce
How can we reproduce the problem? Please be specific. Don't link to a failing CI job. Answer the questions below:

1. What version of Python are you using? All (3.8->3.12)
2. What version of coverage.py shows the problem?

 towncrier debt/pathlib-read @ .nox/coverage_report/bin/coverage debug sys
-- sys -------------------------------------------------------
               coverage_version: 7.4.1
                coverage_module: /Users/jaraco/code/twisted/towncrier/.nox/coverage_report/lib/python3.12/site-packages/coverage/__init__.py
                           core: -none-
                        CTracer: available
           plugins.file_tracers: -none-
            plugins.configurers: -none-
      plugins.context_switchers: -none-
              configs_attempted: .coveragerc
                                 setup.cfg
                                 tox.ini
                                 pyproject.toml
                   configs_read: /Users/jaraco/code/twisted/towncrier/pyproject.toml
                    config_file: /Users/jaraco/code/twisted/towncrier/pyproject.toml
                config_contents: b'[build-system]\nrequires = [\n    "hatchling",\n    "incremental == 22.10.0",\n]\nbuild-backend = "hatchling.build"\n\n\n[project]\ndynamic = ["version"]\nname = "towncrier"\ndescription = "Building newsfiles for your project."\nreadme = "README.rst"\nlicense = "MIT"\n# Keep version list in-sync with noxfile/tests & ci.yml/test-linux.\nclassifiers = [\n    "Intended Audience :: Developers",\n    "License :: OSI Approved :: MIT License",\n    "Operating System :: POSIX :: Linux",\n    "Operating System :: MacOS :: MacOS X",\n    "Operating System :: Microsoft :: Windows",\n    "Programming Language :: Python :: 3.8",\n    "Programming Language :: Python :: 3.9",\n    "Programming Language :: Python :: 3.10",\n    "Programming Language :: Python :: 3.11",\n    "Programming Language :: Python :: 3.12",\n    "Programming Language :: Python :: Implementation :: CPython",\n    "Programming Language :: Python :: Implementation :: PyPy",\n]\nrequires-python = ">=3.8"\ndependencies = [\n    "click",\n    "importlib-resources>=5; python_version<\'3.10\'",\n    "incremental",\n    "jinja2",\n    "tomli; python_version<\'3.11\'",\n]\n\n[project.optional-dependencies]\ndev = [\n    "packaging",\n    "sphinx >= 5",\n    "furo",\n    "twisted",\n    "nox",\n]\n\n[project.scripts]\ntowncrier = "towncrier._shell:cli"\n\n[project.urls]\nDocumentation = "https://towncrier.readthedocs.io/"\nChat = "https://web.libera.chat/?channels=%23twisted"\n"Mailing list" = "https://mail.python.org/mailman3/lists/twisted.python.org/"\nIssues = "https://github.com/twisted/towncrier/issues"\nRepository = "https://github.com/twisted/towncrier"\nTests = "https://github.com/twisted/towncrier/actions?query=branch%3Atrunk"\nCoverage = "https://codecov.io/gh/twisted/towncrier"\nDistribution = "https://pypi.org/project/towncrier"\n\n\n[tool.hatch.version]\nsource = "code"\npath = "src/towncrier/_version.py"\nexpression = "_hatchling_version"\n\n[tool.hatch.build]\nexclude = [\n    "admin",\n    "bin",\n    "docs",\n    ".readthedocs.yaml",\n    "src/towncrier/newsfragments",\n]\n\n\n[tool.towncrier]\n    package = "towncrier"\n    package_dir = "src"\n    filename = "NEWS.rst"\n    issue_format = "`#{issue} <https://github.com/twisted/towncrier/issues/{issue}>`_"\n\n    [[tool.towncrier.section]]\n        path = ""\n\n    [[tool.towncrier.type]]\n        directory = "feature"\n        name = "Features"\n        showcontent = true\n\n    [[tool.towncrier.type]]\n        directory = "bugfix"\n        name = "Bugfixes"\n        showcontent = true\n\n    [[tool.towncrier.type]]\n        directory = "doc"\n        name = "Improved Documentation"\n        showcontent = true\n\n    [[tool.towncrier.type]]\n        directory = "removal"\n        name = "Deprecations and Removals"\n        showcontent = true\n\n    [[tool.towncrier.type]]\n        directory = "misc"\n        name = "Misc"\n        showcontent = false\n\n\n[tool.black]\nexclude = \'\'\'\n\n(\n  /(\n      \\.eggs         # exclude a few common directories in the\n    | \\.git          # root of the project\n    | \\.nox\n    | \\.venv\n    | \\.env\n    | env\n    | _build\n    | _trial_temp.*\n    | build\n    | dist\n    | debian\n  )/\n)\n\'\'\'\n\n\n[tool.isort]\nprofile = "attrs"\nline_length = 88\n\n\n[tool.ruff.isort]\n# Match isort\'s "attrs" profile\nlines-after-imports = 2\nlines-between-types = 1\n\n\n[tool.mypy]\nstrict = true\n# 2022-09-04: Trial\'s API isn\'t annotated yet, which limits the usefulness of type-checking\n#             the unit tests. Therefore they have not been annotated yet.\nexclude = \'^src/towncrier/test/test_.*\\.py$\'\n\n[[tool.mypy.overrides]]\nmodule = \'towncrier.click_default_group\'\n# Vendored module without type annotations.\nignore_errors = true\n\n[[tool.mypy.overrides]]\nmodule = \'incremental\'\n# No released version with type hints.\nignore_missing_imports = true\n\n\n[tool.coverage.run]\nparallel = true\nbranch = true\nsource = ["towncrier"]\n\n[tool.coverage.paths]\nsource = ["src", ".nox/tests-*/**/site-packages"]\n\n[tool.coverage.report]\nshow_missing = true\nskip_covered = true\nexclude_lines = [\n    "pragma: no cover",\n    "if TYPE_CHECKING:",\n]\nomit = [\n    "src/towncrier/__main__.py",\n    "src/towncrier/test/*",\n    "src/towncrier/click_default_group.py",\n]\n'
                      data_file: -none-
                         python: 3.12.3 (main, Apr  9 2024, 08:09:14) [Clang 15.0.0 (clang-1500.3.9.4)]
                       platform: macOS-14.4.1-arm64-arm-64bit
                 implementation: CPython
                     executable: /Users/jaraco/code/twisted/towncrier/.nox/coverage_report/bin/python
                   def_encoding: utf-8
                    fs_encoding: utf-8
                            pid: 38077
                            cwd: /Users/jaraco/code/twisted/towncrier
                           path: /Users/jaraco/code/twisted/towncrier/.nox/coverage_report/bin
                                 /opt/homebrew/Cellar/[email protected]/3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python312.zip
                                 /opt/homebrew/Cellar/[email protected]/3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python3.12
                                 /opt/homebrew/Cellar/[email protected]/3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python3.12/lib-dynload
                                 /Users/jaraco/code/twisted/towncrier/.nox/coverage_report/lib/python3.12/site-packages
                    environment: HOME = /Users/jaraco
                                 PIPX_DEFAULT_PYTHON = /opt/homebrew/bin/python3
                                 PIP_NO_PYTHON_VERSION_WARNING = 1
                   command_line: .nox/coverage_report/bin/coverage debug sys
         sqlite3_sqlite_version: 3.45.3
             sqlite3_temp_store: 0
        sqlite3_compile_options: ATOMIC_INTRINSICS=1, COMPILER=clang-15.0.0, DEFAULT_AUTOVACUUM,
                                 DEFAULT_CACHE_SIZE=-2000, DEFAULT_FILE_FORMAT=4,
                                 DEFAULT_JOURNAL_SIZE_LIMIT=-1, DEFAULT_MMAP_SIZE=0, DEFAULT_PAGE_SIZE=4096,
                                 DEFAULT_PCACHE_INITSZ=20, DEFAULT_RECURSIVE_TRIGGERS,
                                 DEFAULT_SECTOR_SIZE=4096, DEFAULT_SYNCHRONOUS=2,
                                 DEFAULT_WAL_AUTOCHECKPOINT=1000, DEFAULT_WAL_SYNCHRONOUS=2,
                                 DEFAULT_WORKER_THREADS=0, DIRECT_OVERFLOW_READ, ENABLE_API_ARMOR,
                                 ENABLE_COLUMN_METADATA, ENABLE_DBSTAT_VTAB, ENABLE_FTS3,
                                 ENABLE_FTS3_PARENTHESIS, ENABLE_FTS4, ENABLE_FTS5, ENABLE_GEOPOLY,
                                 ENABLE_MATH_FUNCTIONS, ENABLE_MEMORY_MANAGEMENT, ENABLE_PREUPDATE_HOOK,
                                 ENABLE_RTREE, ENABLE_SESSION, ENABLE_STAT4, ENABLE_UNLOCK_NOTIFY,
                                 MALLOC_SOFT_LIMIT=1024, MAX_ATTACHED=10, MAX_COLUMN=2000,
                                 MAX_COMPOUND_SELECT=500, MAX_DEFAULT_PAGE_SIZE=8192, MAX_EXPR_DEPTH=1000,
                                 MAX_FUNCTION_ARG=127, MAX_LENGTH=1000000000, MAX_LIKE_PATTERN_LENGTH=50000,
                                 MAX_MMAP_SIZE=0x7fff0000, MAX_PAGE_COUNT=0xfffffffe, MAX_PAGE_SIZE=65536,
                                 MAX_SQL_LENGTH=1000000000, MAX_TRIGGER_DEPTH=1000,
                                 MAX_VARIABLE_NUMBER=250000, MAX_VDBE_OP=250000000, MAX_WORKER_THREADS=8,
                                 MUTEX_PTHREADS, SYSTEM_MALLOC, TEMP_STORE=1, THREADSAFE=1, USE_URI

1. What versions of what packages do you have installed?

 towncrier debt/pathlib-read @ .nox/coverage_report/bin/pip freeze
coverage==7.4.1

1. What code shows the problem? https://github.com/jaraco/towncrier/blob/e5c6df77d40c8a5ae7aee7a5ac4fbb74c25a3bcd/src/towncrier/_writer.py#L20-L21
2. What commands should we run to reproduce the problem?
   1. install nox
   2. git clone https://github.com/jaraco/towncrier
   3. cd towncrier
   4. git checkout e5c6df77d
   5. nox

Expected behavior
That line is covered by the code, so there should be no coverage concerns. Moreover, an earlier version of the code does not exhibit the issue (even though it also opens the file in a context).

Am I missing some crucial detail here?

[nedbat]

git checkout e5c6df77d

This commit doesn't seem to be in the repo anymore, and when I ran nox on trunk, I got:

nox > Running session coverage_report
nox > Creating virtual environment (virtualenv) using python in .nox/coverage_report
nox > python -m pip install 'coverage[toml]'
nox > coverage combine
Combined data file .coverage.geometer.lan.5627.XAuKZYsx
Combined data file .coverage.geometer.lan.5871.XazxqWlx
Combined data file .coverage.geometer.lan.6111.XTCIvhpx
Combined data file .coverage.geometer.lan.6360.XbLLYRNx
Combined data file .coverage.geometer.lan.6611.XtIfvEOx
Combined data file .coverage.geometer.lan.6851.XzORrlFx
nox > coverage report
Name    Stmts   Miss Branch BrPart  Cover   Missing
---------------------------------------------------
TOTAL     674      0    265      0   100%

13 files skipped due to complete coverage.

Is this still happening somewhere I can look at?

[jaraco]

I pushed a tag for the commit. git checkout coverage-1770 should get the commit. It's definitely there, as the github link to the code above is pinned to that commit. You may need to fetch it explicitly, something like git fetch https://github.com/jaraco/towncrier e5c6df77d.

I just double-checked and ran the repro steps in a clean docker image and they do work for me. So maybe pushing that tag helped keep that commit active.

[nedbat]

OK, I got the commit, but I still have 100% coverage: https://gist.github.com/nedbat/3ad003d2c6428282e3249fd33a5f8e2d

[jaraco]

So hard to reproduce the issue reliably :(

I see in your run that the typecheck job failed, but the others passed. In my run, the docs job fails, but the others pass. When I run it locally on my mac, all the jobs pass (but tests-pypy3.8 is skipped). In both of my cases, the coverage reports show the missing branch.

Since I supplied the Docker repro instructions, are you able to use those to replicate the missed expectation?

Since you're not able to reproduce the issue, that suggests the issue may be sensitive to the environment, and since the issue occurs on my mac and in a Linux docker image I created, it makes me think there may be something about my environment that's triggering the behavior. I'll try paring the Docker image down to its minimal requirements.

[jaraco]

Here's a more minimal docker image that replicates the missed branch:

FROM ubuntu:noble

ENV TZ=UTC

RUN apt update
RUN apt install -y software-properties-common
RUN apt-add-repository -y ppa:deadsnakes
RUN apt update

RUN DEBIAN_FRONTEND=noninteractive apt install -y git python3 pipx
RUN apt install -y python3.9-dev python3.9-venv
RUN git clone https://github.com/jaraco/towncrier
WORKDIR towncrier
RUN git checkout e5c6df77d
RUN pipx install nox
ENV PATH=/root/.local/bin:$PATH
CMD nox