kevin218 / astraeus

Astraeus is a general-purpose tool that manages your data using Xarray structures and reads/writes data from/to your exoplanet data reduction pipeline.

Astraeus

A tool for standardizing I/O when reducing and analyzing exoplanet data

Astraeus is a general-purpose tool that manages your data using Xarray structures and reads/writes data from/to your exoplanet data reduction pipeline. By using consistent formats and keywords across pipelines, users should be able to exchange and compare results easier than ever before! Astraeus makes use of Xarray, an open source Python package that uses labelled multi-dimensional arrays (think of Pandas in N dimensions).

Xarray is commonly used for data analysis in geoscience and makes use of two core data structures, called a DataArray and a Dataset. The former is like a NumPy array, except with coordinates, labels, and attributes. The latter is simply a collection of DataArrays. Both structures can easily be written to and loaded from HDF5 files. This means, even if you are not using Python, you can still read an HDF5 file written by Astraeus and maintain the full useability of your data.

Installation & Example Usage

Up-to-date installation instructions and helpful examples can be found at https://kevin218.github.io/Astraeus/.

astraeus's Issues

Need to make transposed dataarrays for Eureka

As mentioned in kevin218/Eureka#603 (comment), there is currently an issue with the connection between Eureka! and Astraeus. The required code to permit custom S3 inputs into Eureka's Stage 4 includes this snippet:

    "custom_dataset['time'] = time\n",
    "custom_dataset['time'].attrs['time_units'] = time_units\n",
    "custom_dataset['x'] = x\n",
    "custom_dataset['optspec'] = (['time', 'x'], optspec)\n",
    "custom_dataset['optspec'].attrs['flux_units'] = flux_units\n",
    "custom_dataset['optspec'].attrs['time_units'] = time_units\n",
    "custom_dataset['optspec'].attrs['wave_units'] = wave_units\n",
    "custom_dataset['opterr'] = (['time', 'x'], opterr)\n",
    "custom_dataset['opterr'].attrs['flux_units'] = flux_units\n",
    "custom_dataset['opterr'].attrs['time_units'] = time_units\n",
    "custom_dataset['opterr'].attrs['wave_units'] = wave_units\n",
    "custom_dataset['wave_1d'] = (['x'], wave_1d)\n",
    "custom_dataset['wave_1d'].attrs['wave_units'] = wave_units"

I tried using Astraeus' built-in functions to avoid having to explicitly do all that, but there is an issue with doing so. The SpecData files should have the format optspec = (time, x), but using Astraeus' makeLCDA function on the optspec array I get the transpose: (wavelength, time). In addition to the transpose issue, I'm also not totally sure whether having wavelength instead of x would break things within Eureka!'s Stage 4. Using the makeFluxLikeDA function also doesn't work since it requires that the input array be three dimensional (time, x, y) while we only have/want (time, x).

One possibility is that we may not be using Astraeus functions when making our Stage 3 outputs, but I believe we are, and the fact that the makeFluxLikeDA function puts time as the first index while makeLCDA puts time as the last index suggests to me that the issue is the result of inconsistencies in the formatting/organization of Astraeus' arrays.

[Bug]: Issues with Eureka!'s S4 Astraeus Saving

I am currently getting an issue with Eureka!'s Stage 4 save files, where the save files are successfully made, but then it seems an error occurs when the automated garbage collection tries to close the files (which seem to have already been closed). I suspect this is a bug with Astraeus, but I am confused why this is only happening with Eureka!'s Stage 4. Any thought @kevin218?

Traceback:

Saving results
Finished writing to /Volumes/DataDrive/WASP-12b_WFC3/G102/visit11/Stage4/S4_2022-06-29_wfc3_run2/ap5_bg5/S4_wfc3_ap5_bg5_SpecData.h5
Finished writing to /Volumes/DataDrive/WASP-12b_WFC3/G102/visit11/Stage4/S4_2022-06-29_wfc3_run2/ap5_bg5/S4_wfc3_ap5_bg5_LCData.h5
Exception ignored in: <function CachingFileManager.__del__ at 0x180af74c0>
Traceback (most recent call last):
  File "/Users/tjbell1/miniconda3/envs/eureka/lib/python3.9/site-packages/xarray/backends/file_manager.py", line 244, in __del__
  File "/Users/tjbell1/miniconda3/envs/eureka/lib/python3.9/site-packages/xarray/backends/file_manager.py", line 222, in close
  File "/Users/tjbell1/miniconda3/envs/eureka/lib/python3.9/site-packages/h5netcdf/core.py", line 1125, in close
  File "/Users/tjbell1/miniconda3/envs/eureka/lib/python3.9/site-packages/h5py/_hl/files.py", line 525, in close
TypeError: bad operand type for unary ~: 'NoneType'
Exception ignored in: <function File.close at 0x180eceb80>
Traceback (most recent call last):
  File "/Users/tjbell1/miniconda3/envs/eureka/lib/python3.9/site-packages/h5netcdf/core.py", line 1125, in close
  File "/Users/tjbell1/miniconda3/envs/eureka/lib/python3.9/site-packages/h5py/_hl/files.py", line 525, in close
TypeError: bad operand type for unary ~: 'NoneType'`