Export the obtained file paths

[azmeuk]

I am using pytest-regressions in CI workflows like GHA, mostly with pictures.
Often times, tests run well locally but fail in the CI environment, however GHA does not allow to display pictures to see what is wrong.
A solution is to use the upload-artifact action to make the faulty pictures downloadable.

Achieving this is not straightforward: the pytest output should be exported, then parsed in search of the .obtained pattern, then passed to upload-artifact.

It would be awesome if pytest-regression would implement an option to automate this in pytest.
For example, a --export-regressed-file-paths=/path/to/something.txt and write the paths inside.

What do you think?

[azmeuk]

For the record, is here how I currently achieve this:

      - name: Run unittest
        id: pytest
        shell: python
        run: |
          import os
          import subprocess
          import sys

          result = subprocess.run(['pytest'], capture_output=True, text=True)
          output = result.stdout
          print(output)
          if result.returncode > 0:
            files = [line.strip() for line in output.splitlines() if "obtained" in line and os.path.exists(line.strip())]
            if files:
              with open(os.getenv("GITHUB_OUTPUT"), "a") as fd:
                fd.write("pytest=" + "\n".join(files))

          sys.exit(result.returncode)
      - name: Upload failed image artifact
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: failed-images
          path: |
            ${{ steps.pytest.outputs.pytest }}

[nicoddemus]

Hi @azmeuk,

Indeed we have identified the same need as you, we just did not get around the time to do it.

One idea is to use a command-line flag as you did, but provide a path instead, where the plugin would generate the .obtained and .expected files for failed tests, along with some metadata to help map tests to files. Having a separate directory would be useful to easily create an artifact only with the failed snapshots.

We plan to work this eventually, but might take a awhile.

If you would like to contribute this, we would be happy to review it!

[azmeuk]

One idea is to use a command-line flag as you did, but provide a path instead, where the plugin would generate the .obtained and .expected files for failed tests, along with some metadata to help map tests to files.

Even better!

If you would like to contribute this, we would be happy to review it!

Unfortunately it is unlikely that I will have time to spare for this soon.