Add `Tiled` data sources to browse Bluesky runs about silx

[danielballan]

Thanks @t20100. I agree that the blocking I/O sounds like the hard part. We may have to live with a synchronous I/O for now and just make sure that timeouts return control to the user in the event of connection issues.

Adding an h5web Provider for Tiled is also interesting. This has been on our radar since we opened an Issue in Tiled in September 2021. It might be about time to do it. One perhaps unique capability this could add is the ability to view specfiles, TIFFs, and other formats, which Tiled can serve through a unified HDF5-ish abstraction.

I think PyMca is serving a particular cluster of requirements though, so we would pursue this in addition to PyMca integration.

[vasole]

Just to comment that if the prefix is removed, it would simplify things at the PyMca side too because I had already foreseen to handle URLs exclusively via the silx abstraction.

[padraic-shafer]

Refined goals

Discussion on 2024-04-09 @whs92 @danielballan @AbbyGi @vshekar @padraic-shafer

1. Use an isolated "Open" dialog or similar entrypoint that can cope with paginated access to large Catalogs, generating a smaller dataset (a tiled client with filters applied) that can be passed down to the rest of the silx/PyMca stack. This can also hand down authentication state.
2. Fit Tiled nodes into HDF5 abstraction up to some limit (~1000). Tabular data from Tiled is just a Group of 1-dimensional arrays.
3. Focus on 'primary' and 'baseline' streams to start, with an eye on "tab per stream" and whether that fits.
4. Have a switch for polling live data. (This can later be refactored to use websockets, once Tiled supports that.)
5. Ensure HTTP I/O does not lock up or crash the app.

@vasole @t20100 @linupi
Because we weren't able to find a suitable time yet for all of us to meet live--and it sounds like it might be a couple weeks until that's possible--what do you think about this approach? Do you foresee particular difficulties or incompatibilities in fitting this into the architecture of silx?

[t20100]

Hi,

Thanks for the summary!

For the silx part, it makes sense to me and the proof-of-concept was very simple to implement. However, I still have a shallow understanding of tiled.
For now my main concern would be point 5 "Ensure HTTP I/O does not lock up or crash the app." since the hdf5-like API and silx view are built around synchronous access to the data, and I'm not convinced this is easy to change.

[t20100]

BTW, you might want to have a look at h5web, a web-based HDF5 data viewer my colleagues @axelboc and @loichuder developed and maintain. It is available as a JupyterLab extension, a VSCode extension and powers HDF5 online viewing of the ESRF "data portal" and the https://myhdf5.hdfgroup.org/ online viewer (thanks to h5wasm).
This again aims at supporting HDF5 files but the access to the data is abstracted through Providers (for now there's 3 for the HDFGroup's HSDS server, h5wasm and our h5grove a small server tailored for h5web), so there may be a way to adapt it to tiled.
As opposed to silx view, it's natively asynchronous.

[t20100]

We may have to live with a synchronous I/O for now and just make sure that timeouts return control to the user in the event of connection issues.

Sounds good to me.

[t20100]

I just made some update to the silx branch with basic tiled support, and opened PR #4121.

Compared to the previous poc version:

• The tiled: prefix no longer works (changed to tiled- to avoid URL parsing issues): This is not compatible with current support in pymca (vasole/pymca#1074)
• But, this prefix is no longer needed and should be removed IMO
• There is a way to limit the number of retrieved entries per container.

Feedbacks welcomed!

[t20100]

tiled- prefix removed.
Also reworked the TiledDataset to inherit directly from commonh5.Dataset and added a tile Cache.