Bisect_ppx is a code coverage tool for OCaml and Reason. It helps you test thoroughly by showing what's not tested.
You can browse the report seen above online here.
- Usage
- Sending to Coveralls and Codecov
- Controlling coverage with
[@coverage off]
- Other topics
- Bisect_ppx users
- Contributing
Refer to aantron/bisect-starter-dune, which produces this report.
-
Depend on Bisect_ppx in your
opam
file:depends: [ "bisect_ppx" {dev & >= "2.5.0"} "dune" {>= "2.7.0"} ]
-
Mark the code under test for instrumentation by
bisect_ppx
in yourdune
file:(library (public_name my_lib) (instrumentation (backend bisect_ppx)))
-
Build and run your test binary. In addition to testing your code, when exiting, it will write one or more files with names like
bisect0123456789.coverage
:find . -name '*.coverage' | xargs rm -f dune runtest --instrument-with bisect_ppx --force
The
--force
flag forces all your tests to run, which is needed for an accurate coverage report.To run tests without coverage, do
dune runtest
-
Generate the coverage report in
_coverage/index.html
:bisect-ppx-report html
You can also generate a short summary in the terminal:
bisect-ppx-report summary
Refer to aantron/bisect-starter-esy, which produces this report.
The instructions are the same as for regular Dune usage, but...
-
Depend on Bisect_ppx in
package.json
, instead of in anopam
file:"devDependencies": { "@opam/bisect_ppx": "^2.5.0" }, "dependencies": { "@opam/dune": "^2.7.0" }
-
Use the
esy
command for the build and for running binaries:esy install esy dune runtest --instrument-with bisect_ppx --force esy bisect-ppx-report html
Refer to aantron/bisect-starter-rescript, which produces this report.
-
Depend on Bisect_ppx in
package.json
, and install it:"devDependencies": { "bisect_ppx": "^2.0.0" }, "dependencies": { "rescript": "*" }
npm install
If pre-built binaries aren't available for your system, the build will automatically fall back to building Bisect_ppx from source using esy, which will take a few minutes the first time. If this happens, you may need to install esy, if it is not already installed:
npm install -g esy npm install
-
Add Bisect_ppx to your
bsconfig.json
:"bs-dependencies": [ "bisect_ppx" ], "ppx-flags": [ "bisect_ppx/ppx" ]
-
If you are using Jest, add this to your
package.json
:"jest": { "setupFilesAfterEnv": [ "bisect_ppx/lib/js/src/runtime/js/jest.js" ] }
Or, if you have enabled the
package-specs.in-source
flag inbsconfig.json
, replace the path by"bisect_ppx/src/runtime/js/jest.js"
You can exclude your test cases from the coverage report by adding this to
bsconfig.json
:"ppx-flags": [ ["bisect_ppx/ppx", "--exclude-files", ".*_test\\.res$$"] ]
Usage with Jest requires Bisect_ppx version 2.4.0 or higher. See the aantron/bisect-starter-jest for a complete minimal example project. That repo produces this report.
If the tests will be running in the browser, at the end of testing, call
Bisect.Runtime.get_coverage_data();
This returns binary coverage data in a
string option
, which you should upload or otherwise get out of the browser, and write into a.coverage
file. -
Build in development with
BISECT_ENABLE=yes
, run tests, and generate the coverage report in_coverage/index.html
:BISECT_ENABLE=yes npm run build npm run test npx bisect-ppx-report html
To exclude your test files from the report, change your PPX flags like so:
"ppx-flags": [ ["bisect_ppx/ppx", "--exclude-files", ".*test\\.re"] ]
The last argument is a regular expression in the syntax of OCaml's
Str
module. Note that backslashes need to be escaped both inside the regular expression, and again because they are inside a JSON string.Multiple
--exclude-files
option can be specified if you want to provide multiple patterns. -
If your project uses both ReScript and native Dune, native Dune will start picking up OCaml files that are part of the ReScript
bisect_ppx
package. To prevent this, add adune
file with the following contents to the root of your project:(data_only_dirs node_modules)
Refer to aantron/bisect-starter-jsoo, which produces this report.
-
Follow the Dune instructions above, except that the final test script must be linked with
bisect_ppx.runtime
(but not instrumented):(executable (name my_tester) (modes js) (libraries bisect_ppx.runtime))
-
If the tests will run on Node, call this function at the end of testing to write
bisect0123456789.coverage
:Bisect.Runtime.write_coverage_data ()
If the tests will run in the browser, call
Bisect.Runtime.get_coverage_data ()
to get binary coverage data in a string option. Upload this string or otherwise extract it from the browser to create a
.coverage
file. -
Build the usual Js_of_ocaml target, including the instrumented code under test, then run the reporter to generate the coverage report in
_coverage/index.html
:dune build my_tester.bc.js --instrument-with bisect_ppx node _build/default/my_tester.bc.js # or in the browser bisect-ppx-report html
-
Ocamlbuild and OASIS instructions can be found at aantron/bisect_ppx-ocamlbuild.
-
With Ocamlfind, you must have your build script issue the right commands, to instrument the code under test, but not the tester:
ocamlfind opt -package bisect_ppx -c src/source.ml ocamlfind opt -c test/test.ml ocamlfind opt -linkpkg -package bisect_ppx src/source.cmx test/test.cmx
Running the tester will then produce
bisect0123456789.coverage
files, which you can process withbisect-ppx-report
.
bisect-ppx-report
can send reports to Coveralls
and Codecov directly from Travis, CircleCI,
and GitHub Actions. To do this, run
bisect-ppx-report send-to Coveralls
or
bisect-ppx-report send-to Codecov
When sending specifically from GitHub Actions to Coveralls, use
- run: bisect-ppx-report send-to Coveralls
env:
COVERALLS_REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }}
PULL_REQUEST_NUMBER: ${{ github.event.number }}
Put these commands in your CI script in the same place you would run
bisect-ppx-report html
locally. See
bisect-ci-integration-megatest
for example CI scripts and current status of these integrations.
If you'd like Bisect_ppx to support other CI and/or coverage services, please send a pull request!
As a workaround for missing CI/coverage integrations, and for development,
bisect-ppx-report
can also generate a JSON report in Coveralls format, which
can be uploaded to a service of your choice using a separate command. For
example, to send manually from Travis to Coveralls:
bisect-ppx-report \
coveralls coverage.json \
--service-name travis-ci \
--service-job-id $TRAVIS_JOB_ID
curl -L -F json_file=@./coverage.json https://coveralls.io/api/v1/jobs
For other CI services, replace --service-name
and --service-job-id
as
follows:
CI service | --service-name |
--service-job-id |
---|---|---|
Travis | travis-ci |
$TRAVIS_JOB_ID |
CircleCI | circleci |
$CIRCLE_BUILD_NUM |
Semaphore | semaphore |
$REVISION |
Jenkins | jenkins |
$BUILD_ID |
Codeship | codeship |
$CI_BUILD_NUMBER |
GitHub Actions | github |
$GITHUB_RUN_NUMBER |
Note that Coveralls-style reports are less precise than the HTML reports generated by Bisect_ppx, because Coveralls considers entire lines as visited or not visited. There can be many expressions on a single line, and the HTML report separately considers each expression as visited or not visited.
You can tag expressions with [@coverage off]
, and neither they, nor their
subexpressions, will be instrumented by Bisect_ppx.
Likewise, you can tag module-level let
-declarations with [@@coverage off]
,
and they won't be instrumented.
You can also turn off instrumentation for blocks of declarations inside a
module with [@@@coverage off]
and [@@@coverage on]
.
Finally, you can exclude an entire file by putting [@@@coverage exclude_file]
into its top-level module. However, whenever possible, it is recommended to
exclude files by not instrumenting with Bisect_ppx to begin with.
See advanced usage for:
- Exhaustiveness checking.
- Excluding generated files from coverage.
- SIGTERM handling.
- Environment variables.
Cornell CS3110 offers a Bisect_ppx tutorial, featuring a video.
A small sample of projects using Bisect_ppx:
-
Core tools
-
Libraries
-
Applications
Bug reports and pull requests are warmly welcome. Bisect_ppx is developed on GitHub, so please open an issue.
After cloning the repo, try these Makefile
targets:
make test
for unit tests.make usage
for build system integration tests, except ReScript.make -C test/js full-test
for ReScript. This requires npm and esy.
If you'd like to build an npm package, run npm pack
. You can install the
resulting .tgz
file in another project with npm install
. This requires esy,
as the Bisect binaries will not be pre-built. The npm package will use esy to
build them automatically.