spaceml-org / helionb-sdoml Goto Github PK

Hi all,

Thanks for the great work to provide more suitable primitives for Machine Learning in Heliophysics!

It would be great to have a Pytorch DataLoader for the SDO ML v2 dataset in order to ease data loading and facilitate reproducibility.

Ideally the DataLoader should:

• allow to access header information as well as optional labels during training/validation/testing
• provide straightforward extensibility for a custom set of labels (e.g. flaring/non-flaring) in addition to the already existing header information
• implement a set of default transformations that are suitable for the different channels (clamping, normalizing)
• provide a strategy to downsample the data temporally
• provide a way to downsample the data spatially (target size)
• provide a suitable strategy to split the data into train/validation sets
• provide a guideline how to split data between the test and train/val sets
• ...?

A potential starting point can be found here: https://github.com/i4Ds/awesome-helio/pull/10/files (still a few TODOs) - feedback is welcome.

Cheers,
Marius