Image stacking issue with data augmentation (Docker 0.20, Object detection)

[laetitialalla]

Hello (again) !

I am updating my code to make it compatible with version 0.20, but I'm facing an issue when data augmentation is applied for Object Detection.

It's as if some Augmentors are slightly changing the size of the image (I couldn't pinpoint which ones), causing an issue when images are stacked together later.

I am using a dataset of images size 256x256 pixels. Here is the error I get after the training began :

2023-02-22 10:08:21:rastervision.pytorch_learner.learner: INFO - epoch: 0
Training:   3%|████▋ 
[...]
  File "/opt/conda/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch
    return self.collate_fn(data)
  File "/opt/src/rastervision_pytorch_learner/rastervision/pytorch_learner/object_detection_utils.py", line 248, in collate_fn
    x = torch.stack(imgs)
RuntimeError: stack expects each tensor to be equal size, but got [3, 264, 264] at entry 0 and [3, 276, 276] at entry 1

The dimensions can be slighty different from trial to trial. I also had

stack expects each tensor to be equal size, but got [3, 241, 241] at entry 0 and [3, 256, 256] at entry 1

The error disappears if I remove data augmentation completely.

Since, I am using the Docker 0.20, maybe I need to make changes in the config file ?
This is the config I currenlty use :

    # define parameters for window selection (sliding or random)
    window_opts = ObjectDetectionGeoDataWindowConfig(
        method=GeoDataWindowMethod.sliding,
        size=chip_sz,
        stride=chip_sz)

    # define pipeline of transforms for Data Augmentation
    aug_transform = A.Compose([
        A.OneOf([
            A.RGBShift(),
            A.ToGray(),
            A.ToSepia(),
            ]),
        A.OneOf([
            A.RandomBrightnessContrast(),
            A.RandomGamma(),
            A.HueSaturationValue()
            ]),
        A.OneOf([
            A.Blur(),
            A.GaussNoise()
            ]), 
        A.OneOf([
            A.Flip(), 
            A.HorizontalFlip(), 
            A.VerticalFlip(), 
            A.Transpose(),
            ]), 
        A.OneOf([
            A.RandomScale(), 
            A.RandomRotate90()
            ])
        ])
    aug_transform = A.to_dict(aug_transform)

    data = ObjectDetectionGeoDataConfig(
        scene_dataset=scene_dataset,
        window_opts=window_opts,
        img_sz=chip_sz,
        aug_transform=aug_transform
    )

I confess I am a bit lost between all the GeoDataSet classes. I am using ObjectDetectionGeoDataWindowConfig to define the windows parameters, which I put in a ObjectDetectionGeoDataConfig. Maybe I should use a SlidingWindowGeoDataset and pass the augmentors there ?

I kept this config based on an answer @AdeelH gave me on the old gitter here, but it may need to be modified with 0.20 ?

Thanks a lot ;)

Keep up the great work \o/

Laetitia

[AdeelH]

The size error is due to A.RandomScale which resizes the images. If you want to retain the image size after scaling (i.e. by cropping after upscaling and padding after downscaling), you can try A.ShiftScaleRotate. If you don't want the shifting and rotation, you can set shift_limit and rotate_limit to zero.

I would recommend spending some time trying out these augmentation transforms on sample images from your data in a notebook to get a feel for how they affect your data and then choosing the ones that seem most useful. The full list of available transforms can be found here.

I confess I am a bit lost between all the GeoDataSet classes. I am using ObjectDetectionGeoDataWindowConfig to define the windows parameters, which I put in a ObjectDetectionGeoDataConfig. Maybe I should use a SlidingWindowGeoDataset and pass the augmentors there ?

No, you're doing it right. As the new docs emphasize, RV can be used as either a framework or a library. When using it as a framework (i.e. what you are currently doing) you specify settings in Configs instead of instantiating the classes directly; RV then instantiates the appropriate classes internally based on those Configs. The use-as-library thing is new in v0.20; if you want to learn more, check out the tutorials.

I kept this config based on an answer @AdeelH gave me on the old gitter here, but it may need to be modified with 0.20 ?

Take a look at #1666 to see if any of those apply to your config.

Keep up the great work \o/

o7

[laetitialalla]

Thank you for your reply @AdeelH =)

I did not have any issue with A.RandomScale with v0.13. Was the Albumentations' version upgraded in RV 0.20 maybe ?
(Also, I did have issue with A.ShiftScaleRotate in v0.13 when I tested it before ^^).
For 0.20, I changed to A.ShiftScaleRotate(shift_limit=0, rotate_limit=0) and it works perfectly, thanks !

I had to make some other minor changes to the config for 0.20, I list them here in case this is useful for someone else :

• I made a change when inferring classes with GeoJSONVectorSourceConfig (as explained in v0.13 to v0.20 migration guide #1666)
• I added import albumentations as A in the config file
• I changed A.RandomScale() to A.ShiftScaleRotate(shift_limit=0, rotate_limit=0)
• I downloaded the resnet50-0676ba61.pth (I downloaded it once and for all, and put it "manually" in the docker, because of proxy issue)

And it works perfectly \o/

However, I still have recurring warnings, and I don't really know how to fix this :

rastervision.core.data.raster_source.rasterio_source: 

WARNING - Raster block size (8, 256) is too non-square. This can slow down reading. Consider re-tiling using GDAL.

Indeed, when I look at my images' metadata with rasterio, I do have block_shapes = [(8, 256), (8, 256), (8, 256), (8, 256)] (4 channels). However, I don't really know what to do with it.
For example, with the public xView dataset, I have one image with block_shapes = [(2, 3378), (2, 3378), (2, 3378)] (RGB image). But it came like this.
Is this something I should investigate more and consider re-tilling the images with GDAL ? (It seems tedious...)

Thank you for your help !

[AdeelH]

However, I still have recurring warnings, and I don't really know how to fix this

Yeah, I added these warnings in v0.20 and I agree they can get pretty annoying and clutter the output. These are just suggestions so you can feel free to ignore them.

I would like to do the following in the near future:

1. More thoroughly investigate the effect of block size on read performance and remove the warning if it is not significant.
2. Change the log level of this warning, so that it is only displayed when the user asks for more verbose output.

[AdeelH]

Was the Albumentations' version upgraded in RV 0.20 maybe ?

Looking at the requirements file, it was updated from 0.5.0 to 1.3.0 between the two RV versions, so I guess major changes are not unexpected.