bd2kgenomics / dcc-spinnaker Goto Github PK

Updates to the validation framework:

• Kick off a validation automatically when the submission is fully uploaded

• turns out the after_request decorator or teardown_request are inappropriate as they still call before response. Will need a task queue. stackoverflow with helpful explanation

• Additional test cases!

• Turn validations into functions, not classes

• use a standardized signature & name prefix, eg validate_

• set up the validator module to introspect for all found validation functions and run in a queue

  • this makes it very easy to add additional validations upon request

Not a big deal in real world usage, but I'm seeing a pattern in testing:

• sub 15 is updated with a receipt and a validation is queued.
• sub 15 is deleted.
• a new sub 15 is created
• the validation runs and tries to validate the new sub 15, not knowing it's a different submission.

A couple of potential fixes:

• switch submission IDs to guids
• the 'delete' endpoint also seeks out and destroys queued tasks for that submission

Something I haven't seen -- but would be a BIG problem -- would be:

• sub 15 is updated with a receipt and a validation is queued.
• validation begins running
• sub 15 is deleted.
• a new sub 15 is created
• validation of original sub 15 completes and new sub15 is updated with the results

GUIDs would fix this, but not destroying queued tasks.

Should verify specimen type values with accepted codelist values.

Should verify that submitter specimen id and specimen sample id values are all distinct (never the same value in both columns).

Should warn if required jars not found

There's a related ticket here:

BD2KGenomics/dcc-storage#21

This ticket is to accumulate validations to be added to the validation engine, & note if further info needs to be gathered before validation can be implemented.

"Lite" validations -- can be done quickly, does not require lots of IO

Validate metadata.json for each data file

• Metadata.json should be a valid non-empty json file.
• what fields and values are mandatory for a metadata.json file?

Catch empty files

• All data files should be verified non-empty.

Catch incomplete files

• All data files should be verified that upload has finalized. Per Brian, email [Treehouse 2 fastq.tar partial uploads]: " I believe there’s an endpoint you can hit to check that a file has been finalized"

"Full" validations -- requires access to entire contents of data files (possibly several GB)

Catch improperly paired FASTQ

• Given two FASTQ files that are intended to be paired, how does one determine whether they are properly paired or not?
• How will the validator determine that two fastq files are intended to be paired with each other?

The client assumes all data listed in the input metadata.tsv is stored locally on disk and processes it in one big pass; for large dataset uploads the uploader currently has to split the master metadata.tsv into pieces so that they can download one batch of data to the spinnaker client machine's disk, run the client, remove the data, then repeat.

I haven't thought a lot about the best way to fix this, but one good solution would be to add a -r/--rows program option that accepts a list or range of row numbers (e.g. -r 1,2,3 or -r 1-3) and only upload the specified files. Then metadata.tsv won't need to be split up. Discussion welcome.