Visual Tests (VTests)

MuseScore runs a suite of Visual Tests (VTests) to detect regressions in the score layout/drawing engine. The input data for this test suite is the collection of scores found in the vtest/scores folder. These scores are converted to PNG images, once using a "reference build" (usually from the master branch) and once using the build to be tested (often referred to as "current build"). The resulting PNG images are then compared using the tools from ImageMagick, and a HTML report is generated.

Running VTests locally

The VTests are always run on CI whenever you create or edit a pull request, so normally you don't need to worry about running them. However, when experimenting during development, it might be convenient to be able to run them locally. Here are instructions for how to do so.

Preparation

1. Make sure you have a build from the latest master branch somewhere on your computer. To create one, simply checkout the master branch, build & install MuseScore like you would do normally, and copy the build from the install folder to a different folder. On macOS, that means simply copying the .app bundle; on Windows and Linux that means copying the executable (MuseScore4.exe or mscore plus the sibling folders containing data).
2. Generate the reference PNGs by running the following script:

   $ vtest/vtest-generate-pngs.sh --output-dir <reference PNGs dir> --mscore <path to reference MuseScore executable>

   • Replace <reference PNGs dir> with the folder where you want to store the reference PNGs. This can be any folder of your choice.
   • Replace <path to reference MuseScore executable> with the path to the reference MuseScore executable that you just copied somewhere.
     • On macOS, this will end in mscore.app/Contents/MacOS/mscore
     • On Windows, this will end in MuseScore4.exe
     • On Linux, this will end in mscore, (perhaps with some suffix if you specified one during the build process)

Running the tests

1. Build MuseScore on the branch that you want to test.
2. Generate the current PNGs by running the same script again:

   $ vtest/vtest-generate-pngs.sh --output-dir <current PNGs dir> --mscore <path to current MuseScore executable>

   • Replace <current PNGs dir> with the folder where you want to store the current PNGs. This can again be any folder of your choice, but must be a different one than the reference output dir.
   • Replace <path to current MuseScore executable> with the path to the just-built MuseScore executable (found in the install folder).
3. Compare the reference and current PNGs by running the following script:

   $ vtest/vtest-compare-pngs.sh --reference-dir <reference PNGs dir> --current-dir <current PNGs dir> --output-dir <comparison dir>

   • Replace <reference PNGs dir> and <current PNGs dir> with the same values as before.
   • Replace <comparison dir> with the folder where you want to store the comparison result.
4. View the results by opening <comparison dir>/vtest_compare.html in your browser.

Running VTests in Qt Creator

For convenience, we've created a way to run the VTests in Qt Creator. It's a bit of a workaround but it should work. The workflow is based on the workflow above.

Preparation

1. As before, make sure you have a master build somewhere on your computer.
2. Specify the path to this build in the MUE_VTEST_MSCORE_REF_BIN CMake option.
3. In the Tests panel of Qt Creator, found in the left or right sidebar, look for Google Test > Engraving_VTest > 1_GenerateRef. Right-click it and select "Run this test".

Running the tests

1. Build MuseScore.
2. To generate the current PNGs and compare them to the reference PNGs: In the Tests panel of Qt Creator, found in the left or right sidebar, look for Google Test > Engraving_VTest > 2_GenerateCurrentAndCompare. Right-click it and select "Run this test".
3. View the results by opening comparison/vtest_compare.html in your browser. (The location of this comparison dir depends on the working directory where Qt Creator runs the tests; you'll probably find it either in vtest or somewhere in the build directory.)

The VTests failed on GitHub on my Pull Request, but the differences seem unrelated to my changes

In that case, make sure you have rebased your PR on top of the latest master branch. GitHub determines the reference and current commit as follows:

• The reference commit is the commit from the master branch onto which you rebased your PR the last time
• The current commit is what you would get if your PR would be merged into the latest master

So if your branch is behind the master branch, your changes merged with the latest master branch might be compared to an old version of the master branch, and thus the vtests will detect changes that were not made by your PR.

Again, to solve this, just rebase your PR. (git pull --rebase upstream master; git push -f)