move test data out into separate library about pbcore

[mpkocher]

Here's some thoughts on a potential model. I believe it addresses the items you've outlined.

https://gist.github.com/mpkocher/79333f8e1f9059914b8bdcdee3f17cea

Let me know if you have any feedback.

[natechols]

I agree that we should have a JSON file cataloging everything. The rest looks reasonable to me as well.

[mpkocher]

@mdsmith and @pezmaster31 Any comments or feedback?

[mdsmith]

Sounds good to me, especially with API and CLI accessors. I support using json, but we may still get an expansion of non-standardized naming to convey characteristics that lend certain datasets to boundary case testing, e.g. 'lambda', 'lambda-two-movie', 'aligned-lambda-one-zmw', 'empty-bam', 'empty-aligned-bam', 'bam-missing-pbi', 'aligned-bam-missing-pbi', 'rs-cmph5', 'rs-bam', etc. We can either:

1. accept that as an inevitability, because it isn't too bad now
2. have a separate "internal-test" file repo with these arbitrary combinations
3. disallow boundary case test files entirely or within reason, and force users to develop their own or generate them at test-time or
4. develop a set of characteristics and allow for more advanced indexing (something like the pbi recarray).

We'll probably go with option 1, as it is what we have already and seems to work, but it is something to think about.

[natechols]

I mostly agree, with the caveat that we should restrict this repo to files that adhere to our specs (which includes the empty BAM, I think - anyway, that's used in a lot of different tests - but not, for example, an AlignmentSet with missing pbi, although a BAM missing a pbi is potentially valid). We might want to clearly distinguish the corner cases from canonical examples, for example by dumping them in subdirectories (so AlignmentSet/other/empty_aligned.subreads.bam, etc.).

[mpkocher]

Would adding a "description" field help address this issue?

[natechols]

Yes, we probably need that anyway.

[mpkocher]

In version 1, the general idea I was going for would be to group files by whatever metric or grouping that make sense for your application or usecase within a bundle JSON file.

Such as grouping by size, or "bad files" using huge.json, and bad-alignmentsets.json.

However, I am open to other approaches to group within a bundle.

[mpkocher]

Updated with description https://gist.github.com/mpkocher/79333f8e1f9059914b8bdcdee3f17cea