Support reading from cloud stores about sgkit-vcf

[hammer]

@horta it looks like htslib is using libcurl to read from GCS https://github.com/samtools/htslib/pull/446/commits

[horta]

Thanks Jeff!

bgen C library open a file with this interface: struct bgen_file* bgen_file_open(char const* filepath);

I wonder if we could just have something like: struct bgen_file* bgen_file_open(struct ifstream *stream);

Since there is no command line that uses bgen C library, I think it is better to have python handle S3/GCS/URL/etc. authentication etc., and pass to the C level only a stream of bytes that can be fseek'd and fread'd.

I googled a bit and I didnt find much. I would try to avoid C++ (specially at the public interface) mostly to facilitate its use by Python.

[jeromekelleher]

Htslib had to work really hard to get support for remote URLs in there using libcurl - it was a big, deep change to the library which required a lot of work abstracting out the htsfile interface. I think it'd be very difficult to monkey-patch remote URL support onto bgen reader (if that's what you're talking about here @horta).

[horta]

That is what I have in mind. bgen library does fread, fseek, fclose, fopen, and fwrite. It assumes that the underlying file is fseekable and that serial access is better than random access.

An URL file is not fseekable but I'm assuming that S3 files might be. Is that true? Looking at the AWS SDK for Python, it does not seem to be the case: https://github.com/boto/botocore/blob/c168486392658dbddd78d130999f85fc0faf0a51/botocore/response.py#L29

However, AWS SDK for PHP seems to be the case: https://docs.aws.amazon.com/sdk-for-php/v3/developer-guide/s3-stream-wrapper.html
Reading the section "Opening Seekable Streams" on the other hand indicates that is just a local buffered fseek (fake fseek, which amounts to nothing).

If S3 connections is not fseekable, I don't think it makes sense to suport reading S3 files as it would amount to download it in practice...

[horta]

GCS seems to provide arbitrary chunk reading: https://stackoverflow.com/questions/14248333/google-cloud-storage-seeking-within-files
That is not fseek but a wrapper around it could provide a fseek operation that does not amount to download the whole file but mostly the parts you are interested in.

[horta]

S3 is the same: https://stackoverflow.com/questions/45108722/is-there-s3-range-read-function-that-allows-to-read-assigned-byte-range-from-a

[horta]

For future reference, it would be great if someone developed a BytesIO abstraction using this function https://libcloud.readthedocs.io/en/stable/storage/api.html#libcloud.storage.base.StorageDriver.download_object_range_as_stream from libcloud. It would make it possible to read cloud files as it where local with possibly not much bandwidth waste. That seems too good of an idea to not have any library out there doing it or possiblity I'm missing something.