Support for pixel-by-pixel comparison of screenshots

[AzamatCherchesov]

No description provided.

[sergio-sastre]

Once this is groomed, I'd like to review the approach taken.
I believe it will be super easy to oversee something and I have pretty good experience in the area

[AzamatCherchesov]

@sergio-sastre Hello.
Sounds very good.
Do you have elements of an implementation available on github?

We're not sure if it's in the Kaspresso area

We can implement a method for tests that will screenshot the screen during the test run, compare it with the reference value and show the difference

But we see a lot of edge cases, such as runs on different devices. Do you think it makes sense to do this within the Kaspresso, or should it be done outside?
On the infrastructure side and CI/CD for example?

[Areyana]

I reviewed several approaches for pixel-by-pixel comparison on github. There are a lot of simple solutions, with just pixel diff (and recolored pixel diff) and more complex solutions with side image comparison tools on lua/python. The main questions are:

1. Where to store reference images, how to update reference images ?

We can write some gradle plugin that can integrate our reference images by command in AS project (for example, src/main/references). But is it really in the Kaspresso area ? It's more like an infrastructure tool, that is not related for Kaspresso capabilities as a Test framework. Also, i think that most of production teams will create some kind of CI/CD tools to handle reference images on their pipelines.

2. Edge cases on different devices

As a UI test frame we need to provide basic solution out of the box, so again we need to write some sh scripts to create one ideal "emulator" but the UI view is not deterministic even around one emulator (also Kaspresso in this cases needs to manage folders for different resolutions like android resources for example). Again seems to be like CI/CD pipelines work.

3. Image comparison mechanic + reports

The most simple solution as i mentioned was pixel by pixel diff with % threshold. As a output we'll have the screenshoted image and reference image with recolored pixel diff, but i think it's too simple solution for Kaspresso. As for more complex solutions i think again it will be better to run on CI/CD pipelines cause of running complicated 3rd party tools is a bit too much for an on device ui test framework.

4. Maintenance
   As for complex feature we need to maintain it, but popularity of that feature under question.

In conclusion, as we discussed with @AzamatCherchesov and team, now we won't do anything related for pixel-by-pixel comparison cause it's not Kaspresso related feature from our point of view. Maybe we can create another one 3rd party tool for that problem, but not in Kaspresso for now.

[sergio-sastre]

I reviewed several approaches for pixel-by-pixel comparison on github. There are a lot of simple solutions, with just pixel diff (and recolored pixel diff) and more complex solutions with side image comparison tools on lua/python. The main questions are:

1. Where to store reference images, how to update reference images ?

We can write some gradle plugin that can integrate our reference images by command in AS project (for example, src/main/references). But is it really in the Kaspresso area ? It's more like an infrastructure tool, that is not related for Kaspresso capabilities as a Test framework. Also, i think that most of production teams will create some kind of CI/CD tools to handle reference images on their pipelines.

2. Edge cases on different devices

As a UI test frame we need to provide basic solution out of the box, so again we need to write some sh scripts to create one ideal "emulator" but the UI view is not deterministic even around one emulator (also Kaspresso in this cases needs to manage folders for different resolutions like android resources for example). Again seems to be like CI/CD pipelines work.

3. Image comparison mechanic + reports

The most simple solution as i mentioned was pixel by pixel diff with % threshold. As a output we'll have the screenshoted image and reference image with recolored pixel diff, but i think it's too simple solution for Kaspresso. As for more complex solutions i think again it will be better to run on CI/CD pipelines cause of running complicated 3rd party tools is a bit too much for an on device ui test framework.

4. Maintenance
   As for complex feature we need to maintain it, but popularity of that feature under question.

In conclusion, as we discussed with @AzamatCherchesov and team, now we won't do anything related for pixel-by-pixel comparison cause it's not Kaspresso related feature from our point of view. Maybe we can create another one 3rd party tool for that problem, but not in Kaspresso for now.

@AzamatCherchesov @Areyana
Wow, sorry for not replying before, I completely forgot about this!

Completely agree on that.
One can always use Kaspresso for "waiting for views" to be ready before taking any screenshot. Moreover, since Kaspresso can also run with Robolectric, any current framework, including Paparazzi which runs on the JVM as well, could be use together with Kaspresso for screenshot testing.

What could be interesting is to add some samples in Kaspresso on how you can use it together with e.g Shot and Paparazzi