:)

Please let me know how testing has gone for you.

Note that there is more care taken for CF times in both the single-HDF and dataset combine steps now.

https://github.com/jonblower/jon-kerchunk-test

Proposed method of demonstrating kerchunk usage with Quobyte on JASMIN

Exposing Quobyte mounts as S3

We have a demonstration Quobyte volume (the acclim Group Workspace - /gws/nopw/j04/acclim/outputs) mounted on a JASMIN server. We are able to interact with it using existing S3-compliant libraries, including s3fs and xarray.

Creating kerchunk "index" files for NetCDF

The README in this repository demonstrates how to generate a kerchunk index file for (1) a single file, and for (2) multiple files.

NOTE: There is an update to kerchunk since the scripts were created - so that you can create the aggregated files directly (rather than in a 2-stage process.

That example reads the files from object store, rather than as POSIX files.

Creating kerchunk files, on POSIX, but for object store

During early experimentation with the Quobyte S3 interface we have not managed to work out how to configure the authentication components of the s3fs and kerchunk libraries to be able to run the kerchunk scripts.

Our recommended solution to overcome these problems is:

1. Create the aggregated kerchunk index file on POSIX (both NC files and output JSON file)
2. Post-process the index file to convert the local POSIX paths to object store URLs
3. Write the JSON file to the GWS
4. Test reading the JSON file via the object store interface

Need to think about:

• Check cf-decoding of times works - probably just involves passing through kwargs.
• Comparison of metadata, bounds, coordinates and values between original and new data.