Pre-processing of ICESat-2 ATL11 data to cloud-optimized format

[weiji14]

To enable fast reads of ice surface elevation time-series data for analytical and/or visualization purposes!

Current state

ICESat-2 ATL11 data is stored in an HDF5 format (see https://nsidc.org/data/atl11/versions/5), in a cumbersome nested hierarchical format (one 'dataset' per laser pair track). While the ICESat-2 HDF5 files are now on AWS S3 object storage (https://nsidc.org/data/user-resources/data-announcements/data-set-updates-new-earthdata-cloud-access-option-icesat-2-and-icesat-data-sets) as of 29 Sep 2022, which streamlines data read speeds (as long as compute is next to the data on AWS us-west2), the nested HDF5 structure of ATL11 can still be a pain to handle.

Desired state

Remove the nested pair-track data structure (pt1, pt2, pt3), i.e. flatten it into a single non-nested structure. There has been some work on this before to convert the ICESat-2 HDF5 format to:

• Zarr, via xarray, optimized for analytics, less so for visualization
  • https://github.com/weiji14/deepicedrain/blob/v0.4.2/atl06_to_atl11.ipynb
  • Reading multiple ICESat-2 ATL11 point cloud data nicely via Zarr weiji14/deepicedrain#100
• Cloud-optimized point cloud (COPC)/Entwine Point Tiles (EPT), optimized for visualization, less so for analytics
  • https://devseed.com/blog/2020-12-15-analysis-optimized-data-biomass by @abarciauskas-bgse, though it's for ATL08 data
  • https://github.com/developmentseed/example-jupyter-notebooks/blob/main/notebooks/ATL08-EPT-Demo/ATL08-EPT-Demo.ipynb

There's a good blog post about cloud-native vector formats at https://cholmes.medium.com/an-overview-of-cloud-native-vector-c223845638e0, not point-cloud specific, but it discusses about the analytics/visualization trade-offs. Also check out new developments on cloud-optimized ICESat-2 data, for example, see what @tsnow03 is doing at https://github.com/CryoInTheCloud/IS2CloudOptimizedData

Action points

• Chat with people from NSIDC/NASA who are currently thinking about cloud-optimized formats, see if there's anyone keen on getting a pangeo-forge recipe set-up - and use it!
• Focus on a small geographic region of Antarctica as a test case - e.g. Siple Coast
• Get icepyx to read ATL11 properly, both for local and cloud-hosted files (update read-in module for ATL11 icesat2py/icepyx#398, and for cloud?)

References

• https://icepyx.readthedocs.io/en/latest/example_notebooks/IS2_cloud_data_access.html
• https://nasa-openscapes.github.io/earthdata-cloud-cookbook/examples/NSIDC/ICESat2-CMR-OnPrem-vs-Cloud.html
• updates now that cloud data access is public icesat2py/icepyx#379