Projective transform for equivariant imaging

[Andrewwango]

This PR introduces the projective transformation, or homography, an image transformation available for self-supervised learning using equivariant imaging. The transformation is implemented at deepinv.transform.Homography.

I've also implemented a few subgroups of transformations, including deepinv.transform.projective.Affine, deepinv.transform.projective.Similarity and deepinv.transform.projective.Euclidean transformations, which may also be useful in some use-cases.

This was introduced in our recent paper Perspective-Equivariant Imaging: an Unsupervised Framework for Multispectral Pansharpening, as the homography provides a richer group of transformations and so provides better performance for equivariant imaging in camera-based imaging scenarios such as satellite imaging.

See further demos of this functionality at this webpage notebook.

Note this adds kornia as a dependency, but happy to make it optional dependency instead if needed.

Checks to be done before submitting your PR

• python3 -m pytest tests/ runs successfully.
• black . runs successfully.
• make html runs successfully (in the docs/ directory).
• Updated docstrings related to the changes (as applicable).
• Added an entry to the CHANGELOG.rst.

[tachella]

Hey Andrew,

This would be a nice contribution! I think it would be good to document all of the transforms, and add them in the documentation.

I can have a look once the tests are passing!

[samuro95]

I am in favor of making the kornia dependency optional, especially because it is used only once, and only in case the transform is applied to tensors. @tachella what do you think ?

[codecov]

Codecov Report

Attention: Patch coverage is 84.21053% with 18 lines in your changes missing coverage. Please review.

Project coverage is 66.70%. Comparing base (2e138eb) to head (709c629).
Report is 31 commits behind head on main.

❗ Current head 709c629 differs from pull request most recent head 7a67432

Please upload reports for the commit 7a67432 to get more accurate results.

Files	Patch %	Lines
deepinv/transform/projective.py	82.85%	10 Missing and 2 partials ⚠️
deepinv/tests/test_transform.py	93.33%	1 Missing and 1 partial ⚠️
deepinv/training/testing.py	33.33%	2 Missing ⚠️
deepinv/training/trainer.py	66.66%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #173      +/-   ##
==========================================
- Coverage   76.01%   66.70%   -9.32%     
==========================================
  Files         101      124      +23     
  Lines        7481     8782    +1301     
  Branches     1013     1264     +251     
==========================================
+ Hits         5687     5858     +171     
- Misses       1474     2536    +1062     
- Partials      320      388      +68     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Andrewwango]

I am in favor of making the kornia dependency optional, especially because it is used only once, and only in case the transform is applied to tensors. @tachella what do you think ?

How do I add it as an optional dependency?

[tachella]

I am in favor of making the kornia dependency optional, especially because it is used only once, and only in case the transform is applied to tensors. @tachella what do you think ?

How do I add it as an optional dependency?

You can do it in the same way as bm3d for example, see

deepinv/deepinv/models/bm3d.py

Line 6 in 311f98e

try:

[tachella]

The PR looks great, I just found a few details in the docstrings to correct: at the moment the example snippets will not run since the :: indicator for code is not recognized by doctest.

[tachella]

why did you apply these changes?

[Andrewwango]

I believe that was @samuro95 ?

[tachella]

can you please update to the Urban100HR class added in a recent PR?

[Andrewwango]

This increases demo runtime by ~1min as the current deepinv Urban100HR is much bigger than the HF Urban100 I'm using. I could create an issue?

[tachella]

many thanks @Andrewwango for this PR - everything looks good to me