To get YAML checks to work they need version numbers

To generate the training data and run the checker on it all (in a re-runnable way):

https://github.com/barbarabrooks/NCAS-Data-Project-Training-Data

Download, run it locally and it should generate the training data in a local data/ directory.

@gapintheclouds it would be great to do more extensive testing of Barb's test data repository - against the current state of the checker. It is bound to show up some issues in the integrity of the checker.

Update documentation within python scripts with more details on how they work.

Re-run compliance-check-lib tests with newly generated version of PYESSV vocabs. To check nothing big has changed.

This code shows how to create YAML with nice line breaks and it preserves the order of components based on an OrderedDict in the order they were added:

https://github.com/roocs/roocs-utils/blob/master/roocs_utils/inventory/inventory.py#L21-L44

Because of the way I've implemented specific global attributes for the YAML check creator, the controlled variable creator doesn't work for specific global attributes.

download-from-drive:

• check correct spreadsheets produced
• check correct worksheets produced

create-yaml-checks:

• check correct yaml checks produced

create-cvs:

• check correct JSON CVs produced
• check correct pyessv CVs produced

E.g.: aerosol-size-distribution/aerosol-size-distribution.xlsx/Global Attributes - Specific

Install:

https://github.com/cedadev/compliance-check-lib

Initially, we just want the unit tests to run, as outlined here:

https://github.com/cedadev/compliance-check-lib#testing

The tests use lots of test data inside the repository.

See:

$ diff /vagrant-share/amf-compliance-checker-work-2021/check-data-2021-08-27/yaml_checks/v2.0/AMF_global_attrs.yml  /vagrant-share/amf-compliance-checker-work-2021/check-data-2021-08-27/yaml_checks/v2.0/AMF_product_common_global-attributes_land.yml
1c1
< suite_name: global_attrs_checks:2.0
---
> suite_name: product_common_global-attributes_land_checks:2.0
3c3
< description: Check 'global_attrs' in AMF files
---
> description: Check 'product common global-attributes land' in AMF files

They are the same, but both are referenced in the main check files (land, sea, air and trajectory):

$ grep attr yaml_checks/v2.0/AMF_product_aerosol-backscatter-radial-winds_{land,sea,air,trajectory}.yml
yaml_checks/v2.0/AMF_product_aerosol-backscatter-radial-winds_land.yml:- __INCLUDE__: AMF_global_attrs.yml
yaml_checks/v2.0/AMF_product_aerosol-backscatter-radial-winds_land.yml:- __INCLUDE__: AMF_product_aerosol-backscatter-radial-winds_global-attributes.yml
yaml_checks/v2.0/AMF_product_aerosol-backscatter-radial-winds_land.yml:- __INCLUDE__: AMF_product_common_global-attributes_land.yml
yaml_checks/v2.0/AMF_product_aerosol-backscatter-radial-winds_sea.yml:- __INCLUDE__: AMF_global_attrs.yml
yaml_checks/v2.0/AMF_product_aerosol-backscatter-radial-winds_sea.yml:- __INCLUDE__: AMF_product_aerosol-backscatter-radial-winds_global-attributes.yml
yaml_checks/v2.0/AMF_product_aerosol-backscatter-radial-winds_sea.yml:- __INCLUDE__: AMF_product_common_global-attributes_sea.yml
yaml_checks/v2.0/AMF_product_aerosol-backscatter-radial-winds_air.yml:- __INCLUDE__: AMF_global_attrs.yml
yaml_checks/v2.0/AMF_product_aerosol-backscatter-radial-winds_air.yml:- __INCLUDE__: AMF_product_aerosol-backscatter-radial-winds_global-attributes.yml
yaml_checks/v2.0/AMF_product_aerosol-backscatter-radial-winds_air.yml:- __INCLUDE__: AMF_product_common_global-attributes_air.yml
yaml_checks/v2.0/AMF_product_aerosol-backscatter-radial-winds_trajectory.yml:- __INCLUDE__: AMF_global_attrs.yml
yaml_checks/v2.0/AMF_product_aerosol-backscatter-radial-winds_trajectory.yml:- __INCLUDE__: AMF_product_aerosol-backscatter-radial-winds_global-att

We need to work out which one should be favoured.

Mirror other classes, DimensionsCV is probably most useful.

We might be able to completely re-use this class:

Also need to make sure that global attributes worksheets are saved as TSV files in the same way that dimensions-specific and variables-specific are saved, e.g.:

https://github.com/ncasuk/AMF_CVs/tree/master/product-definitions/tsv/moisture-profiles

meaningful stuff

The spreadsheets define a possible type as: "Integer"

However, the regex checks expect everything to be a string. This causes a warning when running the test because the check tries to perform a regular expression match against a non-string type.

The simplest fix is just to patch compliance-check-lib so that it will convert all global attrs to strings if doing a regex check:

(amf)$ cd compliance-check-lib/
(amf)$ git diff
diff --git a/checklib/code/nc_util.py b/checklib/code/nc_util.py
index 66553b2..41672e6 100644
--- a/checklib/code/nc_util.py
+++ b/checklib/code/nc_util.py
@@ -77,7 +77,9 @@ def check_global_attr_against_regex(ds, attr, regex):
     """
     if attr not in ds.ncattrs():
         return 0
-    if not re.match("^{}$".format(regex), getattr(ds, attr), re.DOTALL):
+
+    # Always coerce the attribute to a string to do the regex check
+    if not re.match("^{}$".format(regex), str(getattr(ds, attr)), re.DOTALL):
         return 1
     # Success
     return 2

NOTE: The test code for seeing if you get a warning is:

export DATA_DIR=check-data-2021-09-15
VERSION=v2.0
export PYESSV_ARCHIVE_HOME=$DATA_DIR/$VERSION/pyessv-vocabs
TEST_FILE=../NCAS-Data-Project-Training-Data/Data/ncas-anemometer-1_ral_29001225_mean-winds_v0.1.nc
amf-checker --yaml-dir $DATA_DIR/$VERSION/checks $TEST_FILE --version $VERSION

A new compliance checking rule for global attributes called "Exact match in vocabulary" needs to check whether the corresponding issue exists in the corresponding volcabulary.

Eg. In the 'global_attributes' sheet in the file '_common.xlsx', the check called 'source' needs an "Exact match in vocabulary" from the list contained in the 'Descriptor' column of the 'ncas_instrument_name_and_descriptors' (or in this case possibly also 'community-instrument-name-and-descriptors') sheet in the file '_vocabularies.xlsx'.

The file I've been testing with is the training file 'ncas-anemometer-1_ral_29001225_mean-winds_v0.1.nc'. For this file the source global attribute needs to equal "NCAS Mechanicle Anemometer unit 1" as given in cell L37.

Old

Bring up to date with current state.

Currently a warning is produced when the spreadsheet_handler is looking for vocabularies sheets. Need to adjust for this.

• Decide on a list of deployment locations
• Build common deployment method / playbook
• Deploy on JASMIN
• Deploy at Leeds
• Deploy at others...

Suggested a meeting on Wed 15th Sept (2021!)

Once published:

• Publish amf-compliance-checks to Zenodo
• Publish AMF_CVs to Zenodo

The initial workflow is as follows:

1. Download the Data Product and related spreadsheets from Google Drive
2. Convert the cached versions into TSV
3. Regenerate the CVs
4. Regenerate the checks

Something is going wrong when using a relative path for spreadsheets dir with create-cvs:

(venv) vagrant@localhost:/vagrant/AMF_CVs$ create-cvs spreadsheets ./AMF_CVs
WARNING: Expected to find file at 'spreadsheets/spreadsheets/Common.xlsx/Variables - Air.tsv'
WARNING: Expected to find file at 'spreadsheets/spreadsheets/Common.xlsx/Variables - Land.tsv'
WARNING: Expected to find file at 'spreadsheets/spreadsheets/Common.xlsx/Variables - Sea.tsv'
WARNING: Expected to find file at 'spreadsheets/spreadsheets/Common.xlsx/Dimensions - Air.tsv'
WARNING: Expected to find file at 'spreadsheets/spreadsheets/Common.xlsx/Dimensions - Land.tsv'
WARNING: Expected to find file at 'spreadsheets/spreadsheets/Common.xlsx/Dimensions - Sea.tsv'

It is looking under spreadsheets/spreadsheets instead of just spreadsheets. Using absolute path works as expected.

This may also affect create-yaml-checks.

AMF_instrument.json
AMF_platform.json
AMF_product.json
AMF_scientist.json

Get amf-checker working. Stages are:

• Get Python 3.7 running on Linux
• Set up virtual environment for project
• Fork amf-checker to your github
• Clone your amf-checker
• Modify requirements.txt to non-specific versions
• pip install -r requirements.txt - to install all dependencies.
• pip install --editable . --no-deps - to install local package as a link so that any local edits are reflected when you run.

Can you now run command-line script: download-from-drive?

Once we have a reproducible installation for the compliance-checker, we want to check that our plugin works with it.

The repository is here:

https://github.com/cedadev/cc-yaml

See README for instructions to getting it started.

The measure of success is whether the command-line setup works okay - i.e. the cchecker.py script is accepting our new command-line option:

--yaml <path-to-YAML-file>

@gapintheclouds Assuming you have your base environment activated. Try:

git clone https://github.com/ES-DOC/pyessv

export PYESSV_ARCHIVE_HOME=<YOUR_WORKING_DIR>/compliance-check-lib/cc-vocab-cache/pyessv-archive-eg-cvs

Then save this in a script (test-pyessv.py) and see if it runs okay with python test-pyessv.py:

import pyessv

# Authority: ncas.
ncas = pyessv.NCAS
assert isinstance(ncas, pyessv.Authority)

print('ncas/amf/flux-components-variable/air-pressure')

amf = ncas.amf
assert isinstance(amf, pyessv.Scope)

fcv = amf.flux_components_variable
assert isinstance(fcv, pyessv.Collection)

air_pressure = fcv.air_pressure
assert isinstance(air_pressure, pyessv.Term)

assert pyessv.load('ncas:amf:flux-components-variable') == fcv

assert pyessv.load('ncas:amf:flux-components-variable:air-pressure') == air_pressure

It seems to work fine for me.

With a change made to the compliance-checker-lib this now works. The change was:

cedadev/compliance-check-lib#6

Look at the overall workflow and document all the components and their interactions.

Write a basic document that outlines the entire system with diagram(s).

The tests need checking and updating.

• In particular we need to get this working...

cd compliance-check-lib/
python -m pytest tests

• Update the test data to test the source attribute when comparing to multiple vocab lists (NCAS and community instruments - accessed via data:description properties.
• Add more tests as examples of specific kinds of checks that we have been working on. The tests should have enough annotation (i.e. comments) to guide somebody creating YAML checks in how to write them for a specific check class. So each possible option should be included as an example in the tests.

Tagging: @gapintheclouds

See example repo here, with links to binder for running notebooks in the cloud.

https://github.com/roocs/rooki/

I have added a warning line in the output (to stderr) when the parser cannot understand a row (normally because the compliance checking rule is not defined yet). We can use these to retrospectively fix the spreadsheets on the google drive.

I will push to master.

We are dependent on the IOOS compliance checker. It has a number of its own dependencies. We need a standard python environment to install it into.

Tasks:

1. Start at current version: 4.3.3
2. Try: Python3.8 (we might have to go back to Python3.7 if there are problems).
3. Let's go for installing with Conda following recipes below.

Installation recipes:

• https://github.com/ioos/compliance-checker/#installation
• https://github.com/ioos/compliance-checker/wiki/Installation#using-conda
• http://ioos.github.io/notebooks_demos/other_resources/

Discuss appropriate format.
Should it be converted to readthedocs?
Can it be used for a paper?

@gapintheclouds this may overlap with an existing github Issue but just creating it as a reminder to pick up.

Tasks:

1. Check we can install: https://github.com/cedadev/cc-yaml

2. Update dependencies

3. Update to Python 3

See if the workflow works okay retrospectively with v1.1 of the vocabs and checks.

Updates to spreadsheets (progress):

• aerosol-backscatter-radial-winds/global-attributes-specific.tsv
• aerosol-size-distribution/global-attributes-specific.tsv (https://docs.google.com/spreadsheets/d/1vB1Pig6fRRT2LfsZXeuPP8gqoLeeHjFoqha6B3IlwsI/edit#gid=1572749845)
• boundary-layer-thickness-ceilometer/global-attributes-specific.tsv
• cloud-base/global-attributes-specific.tsv
• cloud-coverage/global-attributes-specific.tsv
• _common/global-attributes.tsv
• depolarisation-ratio/global-attributes-specific.tsv
• particle-size-distribution/global-attributes-specific.tsv (https://docs.google.com/spreadsheets/d/1EJvC9wEryV7Xub5lr6KFsMZtMd2zN-FgRRdDlsXWUwE/edit#gid=1572749845)
• precipitation/global-attributes-specific.tsv (https://docs.google.com/spreadsheets/d/1cKF9fZ5nnYt3y4svGa29enZa6bqcDPiq0Bj0Ok28bJY/edit#gid=232604322)

Three of them still need attention. @barbarabrooks - please can you take a look at those 3.

1. Vocabularies_v2.xlsx is now Vocabularies.xlsx
2. It now contains content that was also in: Instrument Names & Data Products.xlsx

• need to refactor parser to pick that info up and use it appropriately.

https://github.com/AtMoDat/atmodat_standard_checker/

Put in WARNING for now in code that they are not working yet.

Request BB adds in compliance checking rules in spreadsheets.

These 2 packages can be installed from external releases (PyPI or conda-forge):

• IOOS Compliance-checker
• Pyessv

We should package, release and publish our own packages to PyPI - and maybe conda-forge.

Currently the voc-concentration.xlsx spreadsheet contains the variable name mole_fraction_of_<voc_species_name>_in_air. This needs to be addressed by adding a system that can cope with this syntax.

• how to put into ingest pipeline?
• we need to store a report
• we need to respond to the results appropriately

Tagging @gap736uk

I've currently commented out the raw spreadsheet downloader as I am having difficulty authorising the downloader with the correct credentials file.

https://github.com/ncasuk/amf-compliance-checks - v2.0.0rc - ncasuk/amf-compliance-checks#1
https://github.com/ncasuk/AMF_CVs/ - v2.0.0rc - ncasuk/AMF_CVs#7

VERSION 3.0.0 is different!!!

but this is only needs looking at when it is officially released, currently labelled as "WORKING"

https://drive.google.com/drive/u/1/folders/1GNfifCvctYJgTjUoBjfkFKx9yR_ddMjv

Data Products / v3.0.0 /
 comm & vocab/
   _common
   _vocabularies
 timeSeries/
   <instr sub-type>/
     <product>
 timeSeriesProfile/
   <instr sub-type>/
     <product>
 trajectory/
   <instr sub-type>/
     <product>

** We hope that we can just identify the product spreadsheets and
write them to the same flattened structure used in v1 and v2! **

We need to make sure that the spreadsheet scanner can walk the directory structure and download all the spreadsheets that it finds.

Git submodule does behave exactly as we would expect. In particular, the main github repo that holds the submodule, keeps track of the specific commit point on the submodule timeline and binds permanently to that unless you manage it. In order to manage it you might need to:

cd submod/
git checkout master # or other point
cd ../  # back to main repo
git add submod  # to tell it you want to update commit that is used in submodule
git commit -m 'Updated submodule commit point'
git push

See details here:

https://intellipaat.com/community/9971/git-update-submodule-to-latest-commit-on-origin#:~:text=The%20git%20submodule%20update%20command,this%20directly%20within%20the%20submodules.

Maybe it would be easier for us to avoid using submodules. Not sure at the moment.