nsidc / cloud-optimized-icesat2 Goto Github PK

Include the following components:

• Summary of previous work
• • Why working with HDF5 in the cloud is complicated
    • Brief History of HDF
    • Latency in the cloud
    • Python drivers and IO bounded tasks

• Working with HDF5 in the cloud using open source tools
  • GDAL
  • H5Py
  • H5Coro
  • Kerchunk
• Performance considerations for HDF5 in the cloud and paged aggregation of nested metadata
  • HDF5 and nested metadata
  • H5Repack and chunk sizes
  • Simple benckmarking
• Potential Cloud Optimized Formats for HDF5 datasets, the ATL03 case.
  • Raster data and N-dimensional: clear path forward
  • Point Cloud and hybrid datasets: ?
    • Zarr
    • Columnar formats for point cloud data: GeoParquet/Arrow
    • Cloud Native Data formats for PCD: https://copc.io/
  • lessons learned

• Benchmarking results
• Downstream processing pipeline considerations
• Target audience:
  • ESDIS, DAAC management
  • ICESat-2 Science Team
  • Consider providing to NISAR community
• ATL14/15 (COG)? Provide recommendations to help conversion in future release?

Work already underway by @betolink - may just need to add to this repo from local

Candidate criteria:

• Formats / chunking schemes to compare
  • Re-chunked HDF5
  • Cloud-optimized HDF5
  • Geoparquet
  • Zarr
  • Kerchunk json
  • h5coro
• Environment
  • CryoCloud - Small instance
  • Assume we'll store all example files in CryoCloud (i.e. Sync or shared_public)
• Libraries or clients used to open/read data
• For each format option:
  • Dataset(s)
    • Based on community feedback/discussion, initial focus on ATL03
  • Files
    • Single and multiple? Files can vary by several GBs ; optimally produce and test 10 files
  • Variable(s)
  • Spatial subset(s)
  • Temporal subset(s)
  • Aggregation
  • End-to-end wall clock time
    • Time to re-chunk or reformat
    • Time to open/read file
      • Multiple tools/libraries/clients to compare per format option?
        • Geopandas, xarray
        • Should we consider dask data frame
  • Compute cost
  • Do we include a real-world example?
    • Time series of 60 day repeat cycle
    • Real world example tie in: Jacobshavn surface height

Consider following challenges:

• ATL03 has sim-links, etc.
• 1 ground track - can still use kerchunk

Beyond increasing chunk size:

• Minimal use of variable-length datatypes
• Consolidated internal file metadata
• HDF5 paged aggregation (recommended if HDF5 library will also be used for cloud access)

See https://github.com/HDFGroup/nasa_cloud/blob/main/benchmarks/python/icesat2_selection.py - we may be able to leverage this work.