Consistent Model of empty Readers and DataSets about pbcore

[dalexander]

TL;DR: don't touch the behavior for basic file types; negotiate what you want, for datasets.

For basic biofx file types, I consider a file to consist of "headers/metadata" (which can include, for example, the movie table in the cmp.h5 format, or the text headers in a GFF and "records".

A file can be empty in two possible ways: 1) truly empty (zero length) or 2) contain no records. There is no distinction for some file types like FASTA that contain no metadata, but the distinction is important for file types that do.

A reader for a file type that requires metadata will rightly generate an error if metadata is not found--the file is ill-formed. But in any format, the mere absence of records is not intrinsically an error. It may be an error in the context of an application, but not at the library level.

Pbcore's basic readers mostly adheres to this principle. The one exception is the cmp.h5 reader, which will throw an exception on trying to open an empty file. The reason for that is that we don't get empty cmp.h5 files right, in our ecosystem. When I've seen empty cmp.h5 files, they invariably are lacking important metadata tables. Ideally this would have been fixed upstream, but the cmp.h5 ship has already sailed now.

Since the dataset concept bridges the application and data file levels, you are welcome to say that "by spec", a dataset of such and such type may not be empty, and then you can have the dataset reader throw exception on finding such a file. But honestly then the problem is at the writer level---such an invalid file should never be written. Adding the checks on the reader side is then just good defensive programming.

[dalexander]

I will suggest one other possibility. We need to spell out the reader API that we intend client code to use. openAlignmentDataset, etc. We can add additional API methods like openNonemptyAlignmentDataset, to centralize the error checking, for client code that requires nonempty.