Comments (2)
TL;DR: don't touch the behavior for basic file types; negotiate what you want, for datasets.
For basic biofx file types, I consider a file to consist of "headers/metadata" (which can include, for example, the movie table in the cmp.h5 format, or the text headers in a GFF and "records".
A file can be empty in two possible ways: 1) truly empty (zero length) or 2) contain no records. There is no distinction for some file types like FASTA that contain no metadata, but the distinction is important for file types that do.
A reader for a file type that requires metadata will rightly generate an error if metadata is not found--the file is ill-formed. But in any format, the mere absence of records is not intrinsically an error. It may be an error in the context of an application, but not at the library level.
Pbcore's basic readers mostly adheres to this principle. The one exception is the cmp.h5 reader, which will throw an exception on trying to open an empty file. The reason for that is that we don't get empty cmp.h5 files right, in our ecosystem. When I've seen empty cmp.h5 files, they invariably are lacking important metadata tables. Ideally this would have been fixed upstream, but the cmp.h5 ship has already sailed now.
Since the dataset concept bridges the application and data file levels, you are welcome to say that "by spec", a dataset of such and such type may not be empty, and then you can have the dataset reader throw exception on finding such a file. But honestly then the problem is at the writer level---such an invalid file should never be written. Adding the checks on the reader side is then just good defensive programming.
from pbcore.
I will suggest one other possibility. We need to spell out the reader API that we intend client code to use. openAlignmentDataset
, etc. We can add additional API methods like openNonemptyAlignmentDataset
, to centralize the error checking, for client code that requires nonempty.
from pbcore.
Related Issues (20)
- Can I use CCS module in pbcore?
- util/Process.py could be a little better
- Systematic tagging of new releases and upload to PyPI HOT 1
- I am trying to run Quiver I get this error HOT 2
- InvalidDataSetIOError HOT 4
- SMART Analysis - QUIVER - pbtranscript.py ", line 35, in <module> from pbcore.util.ToolRunner import PBMultiToolRunner HOT 4
- [Issue] Distribution of encoded IPDs does not match the technical specification
- Test failures with NumPy 1.14 HOT 1
- Please use proper release tags HOT 3
- KeyError BASECALLERVERSION HOT 4
- inputs to genomicconsensus with arrow algorithm HOT 1
- test failure with numpy 1.16: dtype of to_begin must be compatible with input ary HOT 2
- Please port to Python3 HOT 5
- pyxb no longer supported upstream HOT 2
- Read group string with barcode attached cannot be converted to int. HOT 1
- Wrong info in mapping.xml
- Install Quiver HOT 1
- Numpy 1.11 throws VisibleDeprecationWarnings HOT 1
- Push SA 3.1.0 compatible version to pypi HOT 1
- pbcore.io.merge_gffs_sorted should clean up after itself HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pbcore.