Comments (9)
Here's some thoughts on a potential model. I believe it addresses the items you've outlined.
https://gist.github.com/mpkocher/79333f8e1f9059914b8bdcdee3f17cea
Let me know if you have any feedback.
from pbcore.
I agree that we should have a JSON file cataloging everything. The rest looks reasonable to me as well.
from pbcore.
@mdsmith and @pezmaster31 Any comments or feedback?
from pbcore.
Sounds good to me, especially with API and CLI accessors. I support using json, but we may still get an expansion of non-standardized naming to convey characteristics that lend certain datasets to boundary case testing, e.g. 'lambda', 'lambda-two-movie', 'aligned-lambda-one-zmw', 'empty-bam', 'empty-aligned-bam', 'bam-missing-pbi', 'aligned-bam-missing-pbi', 'rs-cmph5', 'rs-bam', etc. We can either:
- accept that as an inevitability, because it isn't too bad now
- have a separate "internal-test" file repo with these arbitrary combinations
- disallow boundary case test files entirely or within reason, and force users to develop their own or generate them at test-time or
- develop a set of characteristics and allow for more advanced indexing (something like the pbi recarray).
We'll probably go with option 1, as it is what we have already and seems to work, but it is something to think about.
from pbcore.
I mostly agree, with the caveat that we should restrict this repo to files that adhere to our specs (which includes the empty BAM, I think - anyway, that's used in a lot of different tests - but not, for example, an AlignmentSet with missing pbi, although a BAM missing a pbi is potentially valid). We might want to clearly distinguish the corner cases from canonical examples, for example by dumping them in subdirectories (so AlignmentSet/other/empty_aligned.subreads.bam
, etc.).
from pbcore.
Would adding a "description" field help address this issue?
from pbcore.
Yes, we probably need that anyway.
from pbcore.
In version 1, the general idea I was going for would be to group files by whatever metric or grouping that make sense for your application or usecase within a bundle JSON file.
Such as grouping by size, or "bad files" using huge.json, and bad-alignmentsets.json.
However, I am open to other approaches to group within a bundle.
from pbcore.
Updated with description https://gist.github.com/mpkocher/79333f8e1f9059914b8bdcdee3f17cea
from pbcore.
Related Issues (20)
- Can I use CCS module in pbcore?
- util/Process.py could be a little better
- Systematic tagging of new releases and upload to PyPI HOT 1
- I am trying to run Quiver I get this error HOT 2
- InvalidDataSetIOError HOT 4
- SMART Analysis - QUIVER - pbtranscript.py ", line 35, in <module> from pbcore.util.ToolRunner import PBMultiToolRunner HOT 4
- [Issue] Distribution of encoded IPDs does not match the technical specification
- Test failures with NumPy 1.14 HOT 1
- Please use proper release tags HOT 3
- KeyError BASECALLERVERSION HOT 4
- inputs to genomicconsensus with arrow algorithm HOT 1
- test failure with numpy 1.16: dtype of to_begin must be compatible with input ary HOT 2
- Please port to Python3 HOT 5
- pyxb no longer supported upstream HOT 2
- Read group string with barcode attached cannot be converted to int. HOT 1
- Wrong info in mapping.xml
- Install Quiver HOT 1
- Numpy 1.11 throws VisibleDeprecationWarnings HOT 1
- Push SA 3.1.0 compatible version to pypi HOT 1
- pbcore.io.merge_gffs_sorted should clean up after itself HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pbcore.