Google Maps example needed

[philip-firstorder]

Problem:
I want to test the performance of geotiff.js on google maps vs reading pre-rendered tiles directly from s3.

I loaded my tiff in COG Explorer, but it‘s 3x slower than loading pre-rendered png tiles from s3 in Google maps.

What I expect:
To have similar performance reading from a single tiff with geotiff.js as opposed to load pre-rendered png tiles.

Reasons for delay:

• Preflight requests. Because geotiff.js uses the Range header the http requests are not simple GET requests and so an OPTION preflight needs to be done for each range, this adds a 200ms delay for each tile. How can I solve this?
• Geotiff compression. In my experience with streaming big zip files I noticed that using STORE (no compression) results in 5x faster downloads than using DEFLATE. Is this true also for Geotiff files?
• It would have been nice if the COG explorer would give the examples depending on the compression used, so we can test the speed without having the preflight delays if they would host those files on the same origin server. Can you maybe provide a link to an optimal COG file for us to test and compare with?
• I noticed that getting a range of 40KB bytes takes 200ms, but getting a 40KB png file takes only about 50ms? Are range requests slower than full file requests on s3?

[constantinius]

Hi @philip-firstorder

Thanks for submitting this issue. I'll try to answer in my best knowledge.

I loaded my tiff in COG Explorer, but it‘s 3x slower than loading pre-rendered png tiles from s3 in Google maps.

First off, I have to remark that TIFFs are a generic file format, providing options for a very large field of use cases. This is great for interoperability but comes at a cost of an overhead. The simplest example in my opinion is an RGB JPEG encoded TIFF. It has the almost exact same file as a simple JPEG of the same data, just packed in the TIFF headers/IFDs. For other cases the overhead to natively supported file types like PNG/JPEG can be even bigger.

What I want to point out is that TIFFs might not be the best option for every use case, especially if you consider the extra cost you have to pay when you want to use them.

Preflight requests. Because geotiff.js uses the Range header the http requests are not simple GET requests and so an OPTION preflight needs to be done for each range, this adds a 200ms delay for each tile. How can I solve this?

Unfortunately, preflight requests are an essential part of the CORS mechanism built into all modern browsers. They always sent when the origin of the website is not the same as the resource and the request is not "Simple", i.e when a specific header like Range is sent. This cannot be deactivated from the client side.

There are some ways to mitigate that problem a little. One way is to ensure that the resource is accessible via a URL that is of the same origin as the main HTML page. I'm not sure if this is easily configured in S3, my guess is that there is some smart rerouting capability to be configured somewhere. As a last resort, a proxy can be used, that runs under the same origin.

Geotiff compression. In my experience with streaming big zip files I noticed that using STORE (no compression) results in 5x faster downloads than using DEFLATE. Is this true also for Geotiff files?

I'm a little confused here, as this does not seem to make sense. Using compression results in smaller file sizes, thus should increase the download performance, not the other way around. Maybe there is an additional component here that I don't see.

For TIFFs: Using DEFLATE should in most case result in smaller tile/row sizes, thus reducing the amount of raw bytes that need to be transmitted. Compression ratio heavily depends on the actual content of the file, used predictor and tiling scheme.
Compression comes with an additional overhead on the client side, where the tiles/rows need to be decompressed. This decompression is written in JavaScript and is not hardware supported, so may not be as fast as its C counterpart. You can use decoder pools to parallelize the decompression of multiple tiles at once to somewhat mitigate for this.

You have to benchmark for your use-case whether there is a net gain in performance.

It would have been nice if the COG explorer would give the examples depending on the compression used, so we can test the speed without having the preflight delays if they would host those files on the same origin server. Can you maybe provide a link to an optimal COG file for us to test and compare with?

I'm sorry, but that is not possible. COG explorer is hosted on Github itself and has no associated data next to it (So always uses CORS requests to access remote files). You can however, as is the nature of the open source software, put the bundled application and place it next to your data files to compare the different options. In that case no OPTIONS requests should be sent and you can benchmark the raw data access.

Additionally: as already mentioned, there is no single "optimal" TIFF configuration, suitable for all needs, as several factors need to be weighed.

I noticed that getting a range of 40KB bytes takes 200ms, but getting a 40KB png file takes only about 50ms? Are range requests slower than full file requests on s3?

I did not benchmark the cloud storage providers and myself do not have any experience with performance tuning the access. I think I can somewhat understand the additional overhead of the Range access: additional server logic that must read the parts whereas a whole file can use sendfile for the best performance. So it seems, that there is a considerable bill to be paid when using Range requests.

---

I hope I could answer your questions in a satisfactory manner. Thank you for your interest in this project!

[philip-firstorder]

Thank you Fabian for your detailed answer.

I will start now finding solutions for each problem to improve speed. I think the approach of range reading on the client-side is the future, and NASA and ESA are, as you know, also pursuing this because it's so much easier than setting up GDAL servers for tiling millions of files for each zoom level.

If I can bring the responsiveness to parity with pre-rendered tiles, then your library is the future, I'll keep you updated on this issue.

[constantinius]

You're welcome and I wish you best luck for your ventures.

I've just returned from this years foss4g and this talk by Ivàn Sanchez Ortega was eye opening: https://media.ccc.de/v/bucharest-348-breaking-the-curse-of-raster-processing-software-as-a-service

There are hurdles fo using such a technology, unfortunately, but I think too that this is the way to go.

[philip-firstorder]

I saw your foss4g presentation (very nice) and as a coincidence I am also from Bucharest, but now I live in Munich.

Thanks for the extra link, I am watching it now.

I will post my findings on this thread, as I predict it to be a big breakthrough that absolutely needs open sourcing.

[philip-firstorder]

Here is what solutions I found:

1. Preflight requests

• I will implement CloudFront and forward all the www.mydomain/storage/* requests to my s3.bucket, this should get rid of the OPTIONS requests.
• For security I will add CloudFront signed URLs, which is similar the S3 pre-signed URLs

2. Geotiff compression

• DEFLATE is the best compression for COGs, 500KB per tile and 500ms parsing time
• JPEG@75% makes the smallest COGs, 64KB per tile, but you get a lot of artefacts, very visible at the margins where you get some black particles, parsing time is 500ms
• JPEG@100% makes 2x bigger COGs than DEFLATE, 1MB per tile, you still get black particles at the margins, and parsing time increases from 500ms to => 2000ms
• UNCOMPRESSED creates 4x bigger COGs than DEFLATE, 2MB per tile, quality is same as DEFLATE, parsing time is still 500ms

3. Load tiles while animating

• I added these 2 properties and increased responsiveness from 2-3 seconds to about 600ms.
• https://github.com/geotiffjs/cog-explorer/src/components/mapview.jsx#L60

    this.map = new Map({
      layers: [
      ...
      ],
      // Improve user experience by loading tiles while animating. Will make
      // animations stutter on mobile or slow devices.
      loadTilesWhileAnimating: true,
      loadTilesWhileInteracting: true,
      view: new View({
      ...
      }),
    });

4. Can't resolve 'fs' in dist Can't resolve 'fs' in dist #98
   SOLVED by using:

import { fromUrl } from 'geotiff/dist/geotiff.bundle.min.js';

5. Read the COG into google.maps.MapType. FINISHED and working in production since today.

[neelduttahere]

Hey all, I am very new to this topic, and currently I am trying to host a geotiff on my website with google maps javascript API, using geotiff.js. @philip-firstorder Has mentioned:
Read the COG into google.maps.MapType. FINISHED and working in production since today.

Can you elaborate? I am currently confused regarding how to use geotiff.js with google maps. I am able to read the geotiff and fetch the attributes of my geotiff using geotiff.js. And since google maps only supports raster tilling natively, how do I stream the cog data to google maps?

[philip-firstorder]

Hey all, I am very new to this topic, and currently I am trying to host a geotiff on my website with google maps javascript API, using geotiff.js. @philip-firstorder Has mentioned:
Read the COG into google.maps.MapType. FINISHED and working in production since today.

Can you elaborate? I am currently confused regarding how to use geotiff.js with google maps. I am able to read the geotiff and fetch the attributes of my geotiff using geotiff.js. And since google maps only supports raster tilling natively, how do I stream the cog data to google maps?

I implemented GeoTiff with GoogleMaps, works in production for over a year now. However the code is copyrighted and I cannot opensource it.

[abdul-imran]

Hey @philip-firstorder ,
Can we use COG's stored in AWS to display in google Maps using google maps JavaScript API?

[abdul-imran]

Hey @philip-firstorder , @neelduttahere
Could you please share the snippet as to how we can load COGs onto google maps without using Leaflet?
Been struggling for a while now. Thanks.