leap-stc / leap-data-management-utils

LEAP Data Management Utils

This Repo is intended to be installed via the requirements.txt during recipe execution in pangeo-forge-recipes. It currently contains beam transforms to log information into BigQuery tables. Additional shared utilities and checks that are commonly used between LEAP pangeo-forge-feedstocks can live here.

leap-data-management-utils's Issues

Remove BQ logging

If we are not planning on using BQ for the current cataloging schema. Should we remove any BQ related table building and cataloging transforms from this module?

cc @jbusecke @andersy005

Issue new release under LEAP Pypi org

I just applied for a Pypi org for leap. Once that is approved we should

• Increase the bus factor and sign members of the @leap-stc/data-and-compute team up for that account
• Get a new pypi key as an org secret
• Publish a new version of this package there.

Why is tornado a dependency?

@norlandrhagen out of curiosity why is tornado a dependency here?

maybe rename to `leap-data-management-utils`?

I just realized that this will be published to pypi with the quite generic name data-management-tools. Should we maybe prepend leap? @norlandrhagen

Need tests + and CI

I need to add some more testing to the code added with #9, e.g. the cataloging tools in https://github.com/leap-stc/leap-data-management-utils/blob/factor-out-cmip/leap_data_management_utils/cmip_catalog.py.

Tracking this here to make sure I do not block progress on other things here with #9

Better module organization

@andersy005 @norlandrhagen since I expect that we will add a bunch more stuff here as part of the data-management/catalog refactor I would like to organize things a bit better.

As far as I can see we will have the following categories of code (each of which should probably have its own module?):

• Custom Beam Stages for CMIP 'cmip-transforms.py' #9
• Custom Beam Stages for data-management 'data-management-transforms.py'. leap-stc/LEAP_template_feedstock#31 is related.
• Custom Code for the Catalog (are there several categories here?) 'catalog.py'
• Python code for data-management (e.g. this helper function in leap-stc/LEAP_template_feedstock#27) -> Move to 'data-management-stages'

Can you add to the list of things here, and also naming suggestions are very welcome ☺️

CI broken due to missing pip?

The error tracer indicates pkg_resources is missing. I forget where I encountered this recently but adding setuptools to the dependencies fixed it there.

Implement automatic batching for `CMIPBQInterface.iid_list_exists`

I believe I hit the 10k limitation (currently coded as a naked assert 😩) with the CMIP feedstock yesterday.

I seem to have written the solution in the docstring, so this should be pretty easy 😆.

Pull CMIP6 naming schema from upstream

Once I get a response in WCRP-CMIP/CMIP6_CVs#1229 we should refactor

leap-data-management-utils/leap_data_management_utils/cmip_catalog.py

Line 7 in 8c07e4b

CMIP6_naming_schema_official = 'mip_era.activity_id.institution_id.source_id.experiment_id.member_id.table_id.variable_id.grid_label.version'

to pull directly from an upstream repo.

Release not working

I really don't know what I am missing here... Ive had this damn release workflow soo many times now...

Ok so I initially got an error related to authentication (which made sense since there was no token set up as secret)
However #16 did not fix this.

There is an issue with writing out the version file here but I really do not get why. I am using pretty much an identical setup in e.g. https://github.com/jbusecke/pangeo-forge-esgf, but it works without any issue. What is going on here?

Maybe @norlandrhagen @andersy005 have an idea?