Visual regression testing: handling cross OS differences

[LarsDenBakker]

See #427 for the main discussion on visual regression testing.

Screenshots of the same element render differently in different operating systems. JS diffing libraries likely can't handle these correctly, but more advanced tools like https://applitools.com/ might be able to handle these.

We should investigate different workflows we can suggest to people who want to do visual regression testing. Some options (can have multiple suggestions):

• configuration of JS diffing libraries which handle these differences (may not be possible)
• run visual regression tests only in the CI (@Westbrook shared this approach)
• run browser in a docker container, either the whole test runner or just he browser
• run tests on an external service provider, like saucelabs or browserstack
• use service providers which can handle this for you https://applitools.com/

[web-padawan]

In our team, we only run visual tests in Sauce Labs. I also tried using vrtest (used by Material UI) in Docker.

[LarsDenBakker]

Running them in sauce or browserstack is definitely a good solution as well, adding it to the list

[Westbrook]

The idea brought by @castastrophe and team that instead of "running cross-os" you actually run "both baseline and test at the same time" is quite interesting. #427 (comment) An API that supported this sort of work flow would be great.

[castastrophe]

A nice approach for feature branches though I think having stored baselines to run your main branch against is a good sanity check.