[Feature requests] Support cloud storage URL (AWS, Google ...)

[vincentsarago]

It will be really useful to support cloud storage url as s3://

I understand this is not an easy task, I could see if I can submit a PR (at least for AWS support)

cc @constantinius

[constantinius]

Hi @vincentsarago,

Thank you for for issuing this feature request. It definitely sounds like a really useful enhancement. Unfortunately I'm really not an expert for Amazon services. Looking into the matter, I'm a little confused about this s3:// protocol URL prefix. As far as I can tell, this is just a way to shorten the URL. I think I can remember that it also includes authorization management of some kind.

What would you say would be required to have a useful support of s3://?

On the other hand, this sounds like a good place for a customized Source. Such a Source must only have an async fetch(offset, length) method that actually makes the request. For HTTP based sources, I simply set the Range header to get slices of the file. It is then possible to use the fromSource() factory with such a source to instantiate a GeoTIFF object.

It seems like using the aws-sdk with the mentioned methods might yield the desired results.

Is this going into the desired direction?

[vincentsarago]

I need to dig more too.

On the other hand, this sounds like a good place for a customized Source. Such a Source must only have an async fetch(offset, length) method that actually makes the request. For HTTP based sources, I simply set the Range header to get slices of the file. It is then possible to use the fromSource() factory with such a source to instantiate a GeoTIFF object.

This is how we do it in COGDumper (python lib to read COG files)
https://github.com/mapbox/COGDumper/blob/master/cogdumper/s3dumper.py#L20-L25