pauljwright / sdoml

package for the sdoml dataset

SDOML dataset

SDOML is an open-source package for working with the SDOML Dataset (sdoml.org).

⚠️ This package is in development. Currently, it is intended for internal use only. The syntax is subject to change, and the documentation is incomplete. ⚠️

Installation

If you'd like to help develop the SDOML package, or just want to try out the package, you will need to install it from GitHub. The best way to do this is to create a new python virtual environment (either with pipenv or conda). Once you have that virtual environment:

$ git clone https://https://github.com/PaulJWright/sdoml.git
$ cd sdoml
$ pip install -e .

To install the optional extras required for testing, this can be performed with pip as below (for bash, and zsh)

$ pip install -e .[test]

~ pip install -e '.[test]'

pytest and coverage

coverage run -m pytest
coverage html

circleci

You can run circleci locally (https://circleci.com/docs/local-cli/), with brew, for example:

brew install circleci

and validate the file with

circleci config validate

License

This project is Copyright (c) Paul J. Wright and licensed under the terms of the Apache Software License 2.0 license. This package is based upon the Openastronomy packaging guide which is licensed under the BSD 3-clause licence. See the licenses folder for more information.

Contributing

We love contributions! sdoml is open source, built on open source, and we'd love to have you hang out in our community.

Imposter syndrome disclaimer: We want your help. No, really.

There may be a little voice inside your head that is telling you that you're not ready to be an open source contributor; that your skills aren't nearly good enough to contribute. What could you possibly offer a project like this one?

We assure you - the little voice in your head is wrong. If you can write code at all, you can contribute code to open source. Contributing to open source projects is a fantastic way to advance one's coding skills. Writing perfect code isn't the measure of a good developer (that would disqualify all of us!); it's trying to create something, making mistakes, and learning from those mistakes. That's how we all improve, and we are happy to help others learn.

Being an open source contributor doesn't just mean writing code, either. You can help out by writing documentation, tests, or even giving feedback about the project (and yes - that includes giving feedback about the contribution process). Some of these contributions may be the most valuable to the project as a whole, because you're coming to the project with fresh eyes, so you can see the errors and assumptions that seasoned contributors have glossed over.

Note: This disclaimer was originally written by Adrienne Lowe for a PyCon talk, and was adapted by sdoml based on its use in the README file for the MetPy project.

sdoml's Issues

Demonstration NN training

create a new example in the documentations to demonstrate how you go about using sdoml for training/inference of a ML model

Add support for non-consecutive years/months

e.g.

[1,2,3,9,10,11,12]

should be split into two consecutive blocks.

[
    [1,2,3], 
    [9,10,11,12],
]

Need to also ensure that each month has the same number of samples

Notice: You might be using enable_deprecations_as_exceptions that is removed in astropy v6

This is an automated issue from Astropy. You might be using code removed in astropy v6.0.dev from astropy/astropy#14670 in the following files:

* sdoml/tests/test_example.py

Please use https://docs.pytest.org/en/stable/how-to/capture-warnings.html instead.

If this issue is opened in error or irrelevant, feel free to close. We apologize for any inconvenience caused.

tidy up `pd.DataFrame` handling

devise an efficient way to handle building a co-temporal dataframe that can be indexed based on required years/months/days.

Consider implementation of `mypy`

https://mypy-lang.org/

Reinstate support for GCS

Support is currently only available for AWS due to a few factors including:

1. Anonymous access (no need for the CLI and authentication)
2. Free egress (want people to use this on the cloud, but this is a nice benefit)

---

Previous approach for loading data in was as follows (but now only requires root/channels:

from sdoml.sources import DataSource
data_to_load = {
   "HMI": {
         "storage_location": "gcs",
         "root": "fdl-sdoml-v2/sdomlv2_hmi_small.zarr/",
         "channels": ["Bx", "By", "Bz"],
   },
   "AIA": {
         "storage_location": "gcs",
         "root": "fdl-sdoml-v2/sdomlv2_small.zarr/",
         "channels": ["94A", "131A"],
   },
   "EVE": {
         "storage_location": "gcs",
         "root": "fdl-sdoml-v2/sdomlv2_eve.zarr/",
         "channels": ["O V", "Fe XI"],
   },
}
datasource_arr = [DataSource(instrument=k, meta=v) for k, v in data_to_load.items()]