astropy / asdf-astropy

This is the extension library for ASDF to provide support for Astropy.

Astropy

The Astropy Project (https://astropy.org/) is a community effort to develop a single core package for Astronomy in Python and foster interoperability between Python astronomy packages. This repository contains the core package which is intended to contain much of the core functionality and some common tools needed for performing astronomy and astrophysics with Python.

Table of Contents

• Installation
• Contributing
• Getting Started with GitHub Codespaces
• Supporting the Project
• License

Installation

Releases are registered on PyPI, and development is occurring at the project's GitHub page.

For detailed installation instructions, see the online documentation or docs/install.rst in this source distribution.

To install astropy from PyPI, use:

pip install astropy

Contributing

The Astropy Project is made both by and for its users, so we welcome and encourage contributions of many kinds. Our goal is to keep this a positive, inclusive, successful, and growing community by abiding with the Astropy Community Code of Conduct.

More detailed information on contributing to the project or submitting feedback can be found on the contributions page. A summary of contribution guidelines can also be used as a quick reference when you are ready to start writing or validating code for submission.

Getting Started with GitHub Codespaces

Codespaces is a cloud development environment supported by GitHub. None of the Astropy build machinery depends on it, but it is a convenient way to quickly get started doing development on Astropy.

To get started, create a codespace for this repository by clicking this:

A codespace will open in a web-based version of Visual Studio Code. The dev container is fully configured with software needed for this project. For help, see the GitHub Codespaces Support page.

Note: Dev containers is an open spec which is supported by GitHub Codespaces and other tools.

Supporting the Project

The Astropy Project is sponsored by NumFOCUS, a 501(c)(3) nonprofit in the United States. You can donate to the project by using the link above, and this donation will support our mission to promote sustainable, high-level code base for the astronomy community, open code development, educational materials, and reproducible scientific research.

License

Astropy is licensed under a 3-clause BSD style license - see the LICENSE.rst file.

asdf-astropy's Issues

Parameters for Pix2Sky and Sky2Pix ConicOrthomorphic are invalid

Parameters for Conic Orthomorphic models are invalid and this leads to crashes in gwcs tests using astropy/astropy#12558.

The reason it was not causing issues before astropy/astropy#12558 is that previous code (or current as of now on astropy's master) did delay setting WCS projection parameters until model was evaluated. The proposed PR on astropy sets projection models at class initialization leading to crashes in gwcs tests that import asdf_astropy/converters/transform/tests/test_transform.py.

@nden To verify that even with current astropy master (pre- astropy/astropy#12558) add:

    m = astropy_models.Pix2Sky_ConicOrthomorphic(sigma=89.5, delta=0.5)
    m(0, 0)

at the top of create_single_models() and try to run gwcs tests with this modification.

The reason is that theta1 = sigma + delta and it is equal to 90 degrees which leads to singularities in the formula for C in https://docs.astropy.org/en/stable/api/astropy.modeling.projections.Pix2Sky_ConicOrthomorphic.html

If one wants to use something different from default values, then theta1 should be such that singularities are avoided. I would suggest

astropy_models.Pix2Sky_ConicOrthomorphic(sigma=88, delta=1)  # or any delta < 2

Missing converter for `astropy.coordinates.representation.geodetic`

There does not appear to be a converter for the following

WGS84GeodeticRepresentation
WGS72GeodeticRepresentation
GRS80GeodeticRepresentation

Now found in astropy.coordinates.representation.geodetic
https://github.com/astropy/astropy/blob/9ec5277724150ea178282f5dfbd9ca8782f89021/astropy/coordinates/representation/geodetic.py#L103-L121

This causes the current test suite to fail as discussed in this PR:
#181

Support MaskedQuantity

Astropy (v5.0+) has the Masked class (https://docs.astropy.org/en/stable/utils/masked/index.html), which can produce MaskedQuantity. It would be great if ASDF could serialize Masked objects.

Review of what else should go in this package

What other tags should be moved from astropy here?

astropy.io.asdf.tags:
coordinates
fits
table
time
unit/equivalency

`dtype` is not preserved roundtrip by `Quantity`

We currently do not preserve the dtype for Quantity objects roundtrip. E.G.

import asdf
import astropy.units as u

amp33 = u.Quantity(np.zeros((5, 5), dtype=np.uint16), dtype=np.uint16)
print(f"{amp33.dtype=}")

af = asdf.AsdfFile({"amp33": amp33})
af.write_to('test.asdf')

with asdf.open('test.asdf', mode="rw") as ff:
    new_amp33 = ff["amp33"]

print(f"{new_amp33.dtype=}")

returns

amp33.dtype=dtype('uint16')
new_amp33.dtype=dtype('float64')

So the dtype is not making it round trip. I believe this is because Quantity automatically casts the dtype to float64 unless otherwise specified. This means the QuantityConverter needs to add a dtype argument here:

asdf-astropy/asdf_astropy/converters/unit/quantity.py

Line 30 in 9b5bd0f

return Quantity(value, unit=node["unit"], copy=False)

matching the dtype of value if it is a ndarray.

Schechter1D model error

With the release of astropy 5.1, the Schechter1D model is not being detected as having a Converter. One was added in #67; however, this work is incomplete. The Converter will not be active until asdf-transform-schemas has a new release which includes the changes from asdf-format/asdf-transform-schemas#53.

Additional, version checks need to be included in the tests in order to account for this.

Models with input_units_equivalencies does not serialized correctly in asdf

Description

After the resolution of astropy/astropy#8059, we can serialize the unit equivalencies in ASDF. However, it seems that the astropy models asdf types/schemas are not updated accordingly to dump the Model.input_units_equivalencies attributes along side with the model.

The issue is demonstrated in this MWE:

#! /usr/bin/env python
import astropy
from astropy.modeling import models
import asdf
import numpy as np
import astropy.units as u
import tempfile


if __name__ == "__main__":
    print(f"astropy version: {astropy.__version__}")
    print(f"asdf version: {asdf.__version__}")

    mat = np.array([
        [1., 0.],
        [0., -1.]])

    values_in = [2, 1] * u.pix
    af = models.AffineTransformation2D(mat * u.deg, translation=[0, 0] * u.deg)
    af.input_units_equivalencies = {
            'x': u.pixel_scale(1 * u.arcsec / u.pix),
            'y': u.pixel_scale(1 * u.arcsec / u.pix),
            }
    print(f'\naf model:\n{af}')

    # dump to asdf object
    with tempfile.NamedTemporaryFile() as fp:
        asdf.AsdfFile({
            'af': af,
            'af_equiv': af.input_units_equivalencies
            }).write_to(fp, all_array_storage='inline')
        with open(fp.name, 'rb') as fo:
            print(f"\naf model serialized:\n{fo.read().decode()}")
        with asdf.open(fp.name) as fo:
            af_rt = fo.tree['af']
            af_equiv = fo.tree['af_equiv']

    print(f'\naf model round trip:\n{af_rt}')

    print(
            f'af has input units equiv: '
            f'{af.input_units_equivalencies is not None}')
    print(
            f'af_roundtrip has input units equiv: '
            f'{af_rt.input_units_equivalencies is not None}')
    print(
            f'af has input units equiv same as '
            f'af_equiv round tripped separately: '
            f'{af.input_units_equivalencies == af_equiv}')

    def check_model(mdl):
        try:
            values_out = mdl(*values_in)
            return np.allclose(
                [
                    values_out[0].to("arcsec").value,
                    values_out[1].to("arcsec").value
                    ],
                [2, -1])
        except Exception as e:
            return f"Error: {e}"

    print("check model eval af: ")
    print(f"  {check_model(af)}")
    print("check model eval af roundtrip: ")
    print(f"  {check_model(af_rt)}")

The output of me running this script is:

astropy version: 3.2.3
asdf version: 2.3.3

af model:
Model: AffineTransformation2D
Inputs: ('x', 'y')
Outputs: ('x', 'y')
Model set size: 1
Parameters:
    matrix [2,2] translation [2]
        deg            deg
    ------------ ---------------
     1.0 .. -1.0      0.0 .. 0.0

af model serialized:
#ASDF 1.0.0
#ASDF_STANDARD 1.3.0
%YAML 1.1
%TAG ! tag:stsci.edu:asdf/
--- !core/asdf-1.1.0
asdf_library: !core/software-1.0.0 {author: Space Telescope Science Institute, homepage: 'http://github.com/spacetelescope/asdf',
  name: asdf, version: 2.3.3}
history:
  extensions:
  - !core/extension_metadata-1.0.0
    extension_class: astropy.io.misc.asdf.extension.AstropyAsdfExtension
    software: {name: astropy, version: 3.2.3}
  - !core/extension_metadata-1.0.0
    extension_class: astropy.io.misc.asdf.extension.AstropyExtension
    software: {name: astropy, version: 3.2.3}
  - !core/extension_metadata-1.0.0
    extension_class: asdf.extension.BuiltinExtension
    software: {name: asdf, version: 2.3.3}
af: !transform/affine-1.2.0
  matrix: !unit/quantity-1.1.0
    unit: !unit/unit-1.0.0 deg
    value: !core/ndarray-1.0.0
      data:
      - [1.0, 0.0]
      - [0.0, -1.0]
      datatype: float64
      shape: [2, 2]
  translation: !unit/quantity-1.1.0
    unit: !unit/unit-1.0.0 deg
    value: !core/ndarray-1.0.0
      data: [0.0, 0.0]
      datatype: float64
      shape: [2]
af_equiv:
  x: !<tag:astropy.org:astropy/units/equivalency-1.0.0>
  - kwargs_names: [pixscale]
    kwargs_values:
    - !unit/quantity-1.1.0 {unit: !unit/unit-1.0.0 arcsec pixel-1, value: 1.0}
    name: pixel_scale
  y: !<tag:astropy.org:astropy/units/equivalency-1.0.0>
  - kwargs_names: [pixscale]
    kwargs_values:
    - !unit/quantity-1.1.0 {unit: !unit/unit-1.0.0 arcsec pixel-1, value: 1.0}
    name: pixel_scale
...


af model round trip:
Model: AffineTransformation2D
Inputs: ('x', 'y')
Outputs: ('x', 'y')
Model set size: 1
Parameters:
    matrix [2,2] translation [2]
        deg            deg
    ------------ ---------------
     1.0 .. -1.0      0.0 .. 0.0
af has input units equiv: True
af_roundtrip has input units equiv: False
af has input units equiv same as af_equiv round tripped separately: True
check model eval af:
  True
check model eval af roundtrip:
  Error: AffineTransformation2D: Units of input 'x', pix (unknown), could not be converted to required input units of deg (angle)

Release in preparation for astropy 6.0

When changes in astropy 6.0 are reaching a conclusion (perhaps after the feature freeze) we should release asdf-astropy to have a version hopefully compatible with the upcoming astropy 6.0.

Does this package have RTD docs?

If it does, it is not in repo info on top right, nor in Readme.

`Time` values do not always round trip perfectly

Due to how we are generating the representation for astropy.time.Time objects, some numerical inaccuracies accumulate. This results in the time value read from asdf for a astropy.time.Time object slightly differing the value of the original object. The reason for this is that several of the available time formats require a conversion into a format which we write to ASDF (for human readability) and then that format will get converted to the original format, which is a process that generates numerical errors.

A demonstration of this is the need for a atol and the use of isclose instead of == in this test:
https://github.com/WilliamJamieson/asdf-astropy/blob/889005badaea78a6feb59f72574d3b9b299292b9/asdf_astropy/converters/time/tests/test_time.py#L139-L153 (fixing an incorrectly written test as part of #132).

Currently, there are two possible solutions that I see:

1. astropy.time.Time should be written under a new schema (say http://astropy.org/schemas/astropy/time/time-1.0.0) rather than the one built into ASDF. This schema should instead write the base-representation (using val1 and val2 directly) for the astropy.time.Time object rather than attempting to write the object in a human-readable format. @perrygreenfield suggested that maybe ASDF should support converters including a comment alongside this to make it human readable.
2. As proposed by @eslavich, use the decimal built-in type of python to exactly convert val1 and val2 into their exact representation for the universal time value, and write that value as an exact string. Then forcing the read in decimal followed by a computation in decimal to extract val1 and val2. Maybe include a flag in the existing time schema to indicate this mode should be used.

Other solutions may exist, which should be discussed as part of this issue.

Access table metadata without loading data

Hello =)

Is it possible to access only the metadata of e.g. Qtables stored in the .asdf file-format without loading the data itself into memory? I tried to use QTable.read() with a find_table key-word but tree["data"]["meta"] will result in an error since the meta-data alone does not represent a table.

In the asdf documentation they state that "Array data remains unloaded until it is explicitly accessed". Therefore I thought it might be possible.

Here is a try using the example-data (QTable) from #118 :

# For the export of example data see issue 118
import asdf

# Is this only loading the meta-data into memory?
with asdf.open("Nr42_fluxgenerator.asdf") as af:
    meta = af["data"].meta

Or is a simple

meta = QTable.read("Nr42_fluxgenerator.asdf").meta 

already doing the job without reading the table data into memory? Otherwise, would it be useful to be implemented into astropy somehow?

Failed tests: cannot represent an object: RepresenterError

Hi,

During the upgrade of dependencies chain for Guix astro packages, I've faced with error
when asdf-astropy is failed to pass unit tests with new [email protected]

<...>
_____________________ test_serialization[version6-time18] ______________________
[gw2] linux -- Python 3.10.7 /gnu/store/jh59fh48mcffyz5wmsjj0p96xkkflbz0-python-wrapper-3.10.7/bin/python

time = <Time object: scale='utc' format='ymdhms' value=(2010, 3, 1, 0, 0, 0.)>
version = AsdfVersion('1.6.0')
tmp_path = PosixPath('/tmp/guix-build-python-asdf-astropy-0.5.0.drv-0/pytest-of-nixbld/pytest-0/popen-gw2/test_serialization_version6_ti2')

    @pytest.mark.parametrize("time", create_times())
    @pytest.mark.parametrize("version", asdf.versioning.supported_versions)
    def test_serialization(time, version, tmp_path):
        file_path = tmp_path / "test.asdf"
        with asdf.AsdfFile(version=version) as af:
            af["time"] = time
>           af.write_to(file_path)

asdf_astropy/converters/time/tests/test_time.py:55:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/gnu/store/f3b1r7ivvz64qx00bmqbbxqz3y8cak71-python-asdf-3.1.0/lib/python3.10/site-packages/asdf/_asdf.py:1231: in write_to
    self._serial_write(fd, pad_blocks, include_block_index)
/gnu/store/f3b1r7ivvz64qx00bmqbbxqz3y8cak71-python-asdf-3.1.0/lib/python3.10/site-packages/asdf/_asdf.py:998: in _serial_write
    self._write_tree(tree, fd, pad_blocks)
/gnu/store/f3b1r7ivvz64qx00bmqbbxqz3y8cak71-python-asdf-3.1.0/lib/python3.10/site-packages/asdf/_asdf.py:970: in _write_tree
    yamlutil.dump_tree(
/gnu/store/f3b1r7ivvz64qx00bmqbbxqz3y8cak71-python-asdf-3.1.0/lib/python3.10/site-packages/asdf/yamlutil.py:408: in dump_tree
    yaml.dump_all(
/gnu/store/5waxhk0gwydl7gq6jlzhy56pry8pvjf2-python-pyyaml-6.0/lib/python3.10/site-packages/yaml/__init__.py:241: in dump_all
    dumper.represent(data)
/gnu/store/5waxhk0gwydl7gq6jlzhy56pry8pvjf2-python-pyyaml-6.0/lib/python3.10/site-packages/yaml/representer.py:27: in represent
    node = self.represent_data(data)
/gnu/store/f3b1r7ivvz64qx00bmqbbxqz3y8cak71-python-asdf-3.1.0/lib/python3.10/site-packages/asdf/yamlutil.py:39: in represent_data
    node = super().represent_data(data)
/gnu/store/5waxhk0gwydl7gq6jlzhy56pry8pvjf2-python-pyyaml-6.0/lib/python3.10/site-packages/yaml/representer.py:48: in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
/gnu/store/f3b1r7ivvz64qx00bmqbbxqz3y8cak71-python-asdf-3.1.0/lib/python3.10/site-packages/asdf/yamlutil.py:59: in represent_mapping
    node = super(AsdfDumper, dumper).represent_mapping(None, mapping.data, flow_style)
/gnu/store/5waxhk0gwydl7gq6jlzhy56pry8pvjf2-python-pyyaml-6.0/lib/python3.10/site-packages/yaml/representer.py:118: in represent_mapping
    node_value = self.represent_data(item_value)
/gnu/store/f3b1r7ivvz64qx00bmqbbxqz3y8cak71-python-asdf-3.1.0/lib/python3.10/site-packages/asdf/yamlutil.py:39: in represent_data
    node = super().represent_data(data)
/gnu/store/5waxhk0gwydl7gq6jlzhy56pry8pvjf2-python-pyyaml-6.0/lib/python3.10/site-packages/yaml/representer.py:58: in represent_data
    node = self.yaml_representers[None](self, data)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <asdf.yamlutil.AsdfDumper object at 0x27c1cc0>
data = <Time object: scale='utc' format='ymdhms' value=(2010, 3, 1, 0, 0, 0.)>

    def represent_undefined(self, data):
>       raise RepresenterError("cannot represent an object", data)
E       yaml.representer.RepresenterError: ('cannot represent an object', <Time object: scale='utc' format='ymdhms' value=(2010, 3, 1, 0, 0, 0.)>)

/gnu/store/5waxhk0gwydl7gq6jlzhy56pry8pvjf2-python-pyyaml-6.0/lib/python3.10/site-packages/yaml/representer.py:231: RepresenterError
=========================== short test summary info ============================
FAILED asdf_astropy/converters/time/tests/test_time.py::test_serialization[version6-time7]
FAILED asdf_astropy/converters/time/tests/test_time.py::test_serialization[version6-time3]
FAILED asdf_astropy/converters/time/tests/test_time.py::test_serialization[version6-time15]
FAILED asdf_astropy/converters/time/tests/test_time.py::test_serialization[version6-time4]
FAILED asdf_astropy/converters/time/tests/test_time.py::test_serialization[version6-time5]
FAILED asdf_astropy/converters/time/tests/test_time.py::test_serialization[version6-time11]
FAILED asdf_astropy/converters/time/tests/test_time.py::test_serialization[version6-time6]
FAILED asdf_astropy/converters/time/tests/test_time.py::test_serialization[version6-time12]
FAILED asdf_astropy/converters/time/tests/test_time.py::test_serialization[version6-time13]
FAILED asdf_astropy/converters/time/tests/test_time.py::test_serialization[version6-time19]
FAILED asdf_astropy/converters/time/tests/test_time.py::test_serialization[version6-time1]
FAILED asdf_astropy/converters/time/tests/test_time.py::test_serialization[version6-time2]
FAILED asdf_astropy/converters/time/tests/test_time.py::test_serialization[version6-time14]
FAILED asdf_astropy/converters/time/tests/test_time.py::test_serialization[version6-time0]
FAILED asdf_astropy/converters/time/tests/test_time.py::test_serialization[version6-time9]
FAILED asdf_astropy/converters/time/tests/test_time.py::test_serialization[version6-time10]
FAILED asdf_astropy/converters/time/tests/test_time.py::test_serialization[version6-time8]
FAILED asdf_astropy/converters/time/tests/test_time.py::test_serialization[version6-time20]
FAILED asdf_astropy/converters/time/tests/test_time.py::test_serialization[version6-time16]
FAILED asdf_astropy/converters/time/tests/test_time.py::test_serialization[version6-time17]
FAILED asdf_astropy/converters/time/tests/test_time.py::test_serialization[version6-time18]
======================= 21 failed, 3981 passed in 22.61s =======================

[email protected]

• [email protected]
• [email protected]
• [email protected]
• [email protected]
• [email protected]
• [email protected]
• [email protected]
• [email protected]
• [email protected]
• [email protected]
• [email protected]
• [email protected]
• [email protected]

Fixing AstropyDeprecationWarning by using asdf-astropy causes real failure

When running the stdatamodels unit tests (from master), one gets the following deprecation warning:

$ pytest
========================================= test session starts ==========================================
platform darwin -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0
rootdir: /Users/jdavies/dev/stdatamodels, configfile: setup.cfg, testpaths: tests, src
plugins: asdf-2.14.2, doctestplus-0.12.1, openfiles-0.5.0
collected 184 items                                                                                    

tests/test_filetype.py ............                                                              [  6%]
tests/test_fits.py .......................................                                       [ 27%]
tests/test_history.py ...                                                                        [ 29%]
tests/test_models.py .................................                                           [ 47%]
tests/test_resources.py ........                                                                 [ 51%]
tests/test_s3_utils.py .....                                                                     [ 54%]
tests/test_schema.py ...............                                                             [ 62%]
tests/test_storage.py .                                                                          [ 63%]
tests/test_util.py ...........................                                                   [ 77%]
tests/test_validation.py ....xx......x......x.x..................                                [ 99%]
src/stdatamodels/util.py .                                                                       [100%]

=========================================== warnings summary ===========================================
tests/test_history.py::test_history_from_model_to_fits
tests/test_history.py::test_history_from_model_to_fits
  /Users/jdavies/miniconda3/envs/stdatamodels/lib/python3.10/site-packages/astropy/io/misc/asdf/types.py:72: AstropyDeprecationWarning: ASDF functionality for astropy is being moved out of the astropy package to the new asdf-astropy package. Please use this package instead. astropy.io.misc.asdf is deprecated since astropy 5.1 and will be removed in a future release.
    warnings.warn(create_asdf_deprecation_warning())

tests/test_history.py::test_history_from_model_to_fits
  /Users/jdavies/miniconda3/envs/stdatamodels/lib/python3.10/site-packages/astropy/io/misc/asdf/types.py:77: AstropyDeprecationWarning: ASDF functionality for astropy is being moved out of the astropy package to the new asdf-astropy package. Please use this package instead. astropy.io.misc.asdf is deprecated since astropy 5.1 and will be removed in a future release.
    warnings.warn(create_asdf_deprecation_warning())

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================= 179 passed, 5 xfailed, 3 warnings in 11.20s ==============================

These deprecations come from astropy itself:

$ pytest tests/test_history.py -W error::astropy.utils.exceptions.AstropyDeprecationWarning
========================================= test session starts ==========================================
platform darwin -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0
rootdir: /Users/jdavies/dev/stdatamodels, configfile: setup.cfg
plugins: asdf-2.14.2, doctestplus-0.12.1, openfiles-0.5.0
collected 3 items                                                                                      

tests/test_history.py .F.                                                                        [100%]

=============================================== FAILURES ===============================================
___________________________________ test_history_from_model_to_fits ____________________________________

tmpdir = local('/private/var/folders/t1/md4315cx5kl3zjv8fkgmtvl586jmhc/T/pytest-of-jdavies/pytest-17/test_history_from_model_to_fit0')

    def test_history_from_model_to_fits(tmpdir):
        tmpfits = str(tmpdir.join('tmp.fits'))
        m = DataModel()
        m.history = [HistoryEntry({
            'description': 'First entry',
            'time': Time(datetime.datetime.now())})]
        m.history.append(HistoryEntry({
            'description': 'Second entry',
            'time': Time(datetime.datetime.now())
        }))
>       m.save(tmpfits)

tests/test_history.py:57: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/stdatamodels/model_base.py:523: in save
    self.to_fits(output_path, *args, **kwargs)
src/stdatamodels/model_base.py:648: in to_fits
    ff = AsdfInFits(hdulist, tree)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/fits_embed.py:150: in __init__
    super().__init__(tree=tree, **kwargs)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/asdf.py:177: in __init__
    self.tree = tree
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/asdf.py:589: in tree
    self._validate(asdf_object, custom=bool(tree))
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/asdf.py:617: in _validate
    tagged_tree = yamlutil.custom_tree_to_tagged_tree(tree, self)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/yamlutil.py:261: in custom_tree_to_tagged_tree
    return treeutil.walk_and_modify(
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/treeutil.py:420: in walk_and_modify
    return _recurse(top)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/treeutil.py:407: in _recurse
    result = _handle_children(result, json_id)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/treeutil.py:377: in _handle_children
    return _handle_generator(result)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/treeutil.py:276: in _handle_generator
    result = next(generator)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/treeutil.py:304: in _handle_mapping
    value = _recurse(value, json_id)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/treeutil.py:407: in _recurse
    result = _handle_children(result, json_id)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/treeutil.py:377: in _handle_children
    return _handle_generator(result)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/treeutil.py:276: in _handle_generator
    result = next(generator)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/treeutil.py:304: in _handle_mapping
    value = _recurse(value, json_id)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/treeutil.py:407: in _recurse
    result = _handle_children(result, json_id)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/treeutil.py:377: in _handle_children
    return _handle_generator(result)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/treeutil.py:276: in _handle_generator
    result = next(generator)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/treeutil.py:337: in _handle_mutable_sequence
    result.append(_recurse(value, json_id))
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/treeutil.py:407: in _recurse
    result = _handle_children(result, json_id)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/treeutil.py:377: in _handle_children
    return _handle_generator(result)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/treeutil.py:276: in _handle_generator
    result = next(generator)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/treeutil.py:304: in _handle_mapping
    value = _recurse(value, json_id)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/treeutil.py:406: in _recurse
    result = _handle_callback(node, json_id)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/treeutil.py:283: in _handle_callback
    result = callback(node)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/yamlutil.py:258: in _walker
    return tag.to_tree_tagged(obj, ctx)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cls = <class 'astropy.io.misc.asdf.tags.time.time.TimeType'>
node = <Time object: scale='utc' format='datetime' value=2022-12-12 12:17:45.455596>
ctx = <asdf.fits_embed.AsdfInFits object at 0x7fdff4d03b80>

    @classmethod
    def to_tree_tagged(cls, node, ctx):
>       warnings.warn(create_asdf_deprecation_warning())
E       astropy.utils.exceptions.AstropyDeprecationWarning: ASDF functionality for astropy is being moved out of the astropy package to the new asdf-astropy package. Please use this package instead. astropy.io.misc.asdf is deprecated since astropy 5.1 and will be removed in a future release.

../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/astropy/io/misc/asdf/types.py:72: AstropyDeprecationWarning
======================================= short test summary info ========================================
FAILED tests/test_history.py::test_history_from_model_to_fits - astropy.utils.exceptions.AstropyDeprecationWarning: ASDF functionality for astropy is being moved o...
===================================== 1 failed, 2 passed in 0.45s ======================================

But if I pip install asdf-astropy and rerun the test, I get an actual validation error:

$ pytest tests/test_history.py
========================================= test session starts ==========================================
platform darwin -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0
rootdir: /Users/jdavies/dev/stdatamodels, configfile: setup.cfg
plugins: asdf-2.14.2, doctestplus-0.12.1, openfiles-0.5.0
collected 3 items                                                                                      

tests/test_history.py .F.                                                                        [100%]

=============================================== FAILURES ===============================================
___________________________________ test_history_from_model_to_fits ____________________________________

tmpdir = local('/private/var/folders/t1/md4315cx5kl3zjv8fkgmtvl586jmhc/T/pytest-of-jdavies/pytest-18/test_history_from_model_to_fit0')

    def test_history_from_model_to_fits(tmpdir):
        tmpfits = str(tmpdir.join('tmp.fits'))
        m = DataModel()
        m.history = [HistoryEntry({
            'description': 'First entry',
            'time': Time(datetime.datetime.now())})]
        m.history.append(HistoryEntry({
            'description': 'Second entry',
            'time': Time(datetime.datetime.now())
        }))
>       m.save(tmpfits)

tests/test_history.py:57: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/stdatamodels/model_base.py:523: in save
    self.to_fits(output_path, *args, **kwargs)
src/stdatamodels/model_base.py:648: in to_fits
    ff = AsdfInFits(hdulist, tree)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/fits_embed.py:150: in __init__
    super().__init__(tree=tree, **kwargs)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/asdf.py:177: in __init__
    self.tree = tree
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/asdf.py:589: in tree
    self._validate(asdf_object, custom=bool(tree))
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/asdf.py:618: in _validate
    schema.validate(tagged_tree, self, reading=reading)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/schema.py:682: in validate
    validator.validate(instance)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = Validator(schema={}, format_checker=None)
args = ({'history': {'entries': [{'description': 'First entry', 'time': {'value': '2022-12-12T12:20:17.336', 'base_format': '....fits', 'model_type': 'DataModel'}, '_fits_hash': 'b8bf102cca29ade468c8db8c7dab0c0a8f7e9110d5b398ba436b361789642b44'},)
kwargs = {}
error = <ValidationError: "{'entries': [{'description': 'First entry', 'time': {'value': '2022-12-12T12:20:17.336', 'base_form...time': {'value': '2022-12-12T12:20:17.339', 'base_format': 'datetime'}}]} is not valid under any of the given schemas">

    def validate(self, *args, **kwargs):
        for error in self.iter_errors(*args, **kwargs):
>           raise error
E           jsonschema.exceptions.ValidationError: {'entries': [{'description': 'First entry', 'time': {'value': '2022-12-12T12:20:17.336', 'base_format': 'datetime'}}, {'description': 'Second entry', 'time': {'value': '2022-12-12T12:20:17.339', 'base_format': 'datetime'}}]} is not valid under any of the given schemas
E           
E           Failed validating 'anyOf' in schema['properties']['history']:
E               {'anyOf': [{'items': [{'$ref': 'history_entry-1.0.0'}],
E                           'type': 'array'},
E                          {'$ref': '#/definitions/history-1.1.0'}],
E                'description': 'A log of transformations that have happened to the '
E                               'file.  May\n'
E                               'include such things as data collection, data '
E                               'calibration\n'
E                               'pipelines, data analysis etc.\n'}
E           
E           On instance['history']:
E               {'entries': [{'description': 'First entry',
E                             'time': {'base_format': 'datetime',
E                                      'value': '2022-12-12T12:20:17.336'}},
E                            {'description': 'Second entry',
E                             'time': {'base_format': 'datetime',
E                                      'value': '2022-12-12T12:20:17.339'}}]}

../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/jsonschema/validators.py:314: ValidationError
======================================= short test summary info ========================================
FAILED tests/test_history.py::test_history_from_model_to_fits - jsonschema.exceptions.ValidationError: {'entries': [{'description': 'First entry', 'time': {'value'...
===================================== 1 failed, 2 passed in 0.81s ======================================

Not sure if this is a bug here, or in stdatamodels.

TST: Use GitHub Actions

Please see https://groups.google.com/g/astropy-dev/c/45frnTEAX-U

MNT: Python 3.7 support status?

asdf-astropy/setup.cfg

Line 17 in 5d3b444

python_requires = >=3.6

setup.cfg allows Python 3.7 but it seems untested in your CI matrix. Either test against it or drop support.

Add CITATION

As @dhomeier suggested in astropy/astropy.github.com#490 (comment) , please add a CITATION file with appropriate citation. Thanks!

MNT: Drop Python 3.6

Python 3.6 has already reached EOL.

asdf-astropy/setup.cfg

Line 17 in 5d3b444

python_requires = >=3.6

Remove implied dependency of `asdf_astropy.testing.helpers` on `pytest`

This import

asdf-astropy/asdf_astropy/testing/helpers.py

Line 3 in 175805e

from astropy.tests.helper import assert_quantity_allclose

is creating a dependency on pytest for asdf_astropy.testing.helpers, the is undesirable see spacetelescope/roman_datamodels#79.

Support non-VOunits

(note that this issue has been opened after the PR to fix it)

@PaulHuwe has requested that we consider supporting non-VOunits which are part of astropy as a more general thing. For example I had to do a hack job in spacetelescope/roman_datamodels#109 in order to support a few non-VOunits for the Roman Space Telescope datamodels.

Initially, I was opposed to adding general support for non-VOunits to ASDF due to the possibility of confusion of unit symbol meanings. However, after some consideration I have come to the conclusion that we can indeed support non-VOunits which originate from astropy in a way that ensures that no confusion can occur.

My plan for doing this is simple, units will be written under two different unit tags:

• tag:tag:stsci.edu:asdf/unit/unit-1.0.0 (the original tag) for VOunits.
• tag:astropy.org:astropy/units/unit-1.0.0 for non-VOunits specified by astropy.

Support for this will be slightly complicated as the object types in astropy for units are not divided between VOunits and non-VOunits. However, theoretically a select_tag method can be implemented to distinguish them.

Support serialization of new astropy 5.0 models

Support for the following models needs to be implemented here:

astropy.modeling.functional_models.ArcCosine1D
astropy.modeling.functional_models.ArcSine1D
astropy.modeling.functional_models.ArcTangent1D
astropy.modeling.functional_models.Cosine1D
astropy.modeling.functional_models.Tangent1D
astropy.modeling.spline.Spline1D

`assert_model_roundtrip` should be a `helper`

A few packages are importing directly from the asdf_astropy unit tests, the assert_model_roundtrip function, see this example. Things within unit tests are normally considered private; however, this function seems useful for testing converters in other contexts.

I think the assert_..._roundtrip functions should be moved to https://github.com/astropy/asdf-astropy/blob/main/asdf_astropy/testing/helpers.py so that they are properly available for other packages to use.

Allow lazy loading of Quantity instances created during deserialization

Currently the Quantity converter calls _make_array before creating the Quantity:

asdf-astropy/asdf_astropy/converters/unit/quantity.py

Line 33 in 0c50251

value = value._make_array()

When the array is stored as an internal ASDF block this will force the block to be loaded at the time of deserialization.

Initial attempts to enable lazy loading by removing the above mentioned line do not prevent the ASDF block from being loaded possibly because NDArrayType is not a subclass of ndarray and astropy will attempt to iterate through the array during loading:
https://github.com/astropy/astropy/blob/79dd50f61ef5e0a568dd42549319db38fefd0bc7/astropy/units/quantity.py#L495
triggering loading of the ASDF block.

DOC: Add link to RTD site on repo About

Would be nice to have https://asdf-astropy.readthedocs.io/ listed there.

Support `MagUnits`

Support for MagUnit style units has been requested for the Roman Space Telescope output files. I think this is a reasonable request given we are working to resolve #144 (non-VOunits).

This should be relatively straight forward as we all MagUnit units essentially wrap a "physical unit" which we should be able to serialize already (for VOunits, and non-VOunits if #142 is merged). The heart of a schema for this would be:

type: object
properties:
  unit:
     oneOf:
       - tag: "tag:stsci.edu:asdf/unit/unit-1.0.0"
       - tag: "tag:astropy.org:astropy/units/unit-1.0.0"

Once the schema is setup it should be relatively straightforward to get the Converter integrated and working.

@mhvk or @nstarman as astropy.units maintainers do I need to do anything besides wrapping the physical unit inside the MagUnit?

ASDF for cosmologies

Intended as a reminder to myself. But if anyone wants to just do it... :)

DOC: Conversion guide

If someone was already using astropy.io.misc.asdf, what do they need to do to use this new package? Even if it is a one-line import change, this should be documented.

Does the data block needs to be binary?

Hi all,

I would like to save my astropy QTables with the asdf format to also save the meta dictionary and units. The only thing that holds me back is comparability with other users who might not want to use python to evaluate the data.

Would it be possible to store the data (relatively small) after the yaml header in a human-readable format like ecsv does? Or is there a hard reason why it needs to be a binary block?

PS: I found the ASDF-option to save inline arrays inside the yaml, but I think it is not accessible via QTable.write() and it seems not very human-readable nor easy to extract with GUI-software..
PS2: using ecsv with QTable.read() it does not import the meta dictionary as a dictionary and asdf seems more future-proof.

DOC: Add links to schema packages when their documentation is finished.

Once:

• asdf-format/asdf-transform-schemas#58
• asdf-format/asdf-coordinates-schemas#7

have been resolved, asdf-astropy documentation needs to link to these.

Not all `astropy.time` formats appear to be supported correctly.'

In looking at the coverage reports for asdf-astropy I noticed that the astropy.time time-formats covered by:

asdf-astropy/asdf_astropy/converters/time/time.py

Lines 22 to 27 in 816dc57

	if obj.format == "byear":
	obj = Time(obj, format="byear_str")
	elif obj.format == "jyear":
	obj = Time(obj, format="jyear_str")
	elif obj.format in ("fits", "datetime", "plot_date"):
	obj = Time(obj, format="isot")

are not being properly tested by the asdf-astropy unit tests. However, when I added test-cases to cover these formats, I discovered that times using these formats do not round-trip through asdf using asdf-astropy. Note that this extends to when one uses astropy.io.misc.asdf as well.

We claim that asdf-astropy will produce an identical astropy object to that which was serialized for all supported astropy objects. This is clearly not happening at this time.

Add support for multiple tags in a single FrameConverter

Don't assume that tag is always a string.

Converter needs to be written for `Schechter1D` model

The Schechter1D model was introduced to astropy in astropy/astropy#13116, a converter needs to be added for this model.

Circular dependency with astropy?

I see this in astropy/astropy#14784 (https://github.com/astropy/astropy/actions/runs/6114052860/job/16594815510?pr=14784)

Collecting astropy>=5.0.4 (from asdf-astropy==0.4.1.dev54+gcb80bd8)
  Downloading astropy-5.3.3.tar.gz (7.8 MB)

This makes no sense because CI in astropy should not result in any "Downloading astropy-5.3.3.tar.gz". I wonder if asdf-astropy should stop declaring astropy as required dependency...

asdf-astropy/pyproject.toml

Line 23 in cb80bd8

"astropy>=5.0.4",

Add links to the docs on the Github page and readme

I couldn't find the docs until I searched the web for them :)

Warnings from package building

The 0.4.0 publish action is pointing out some warnings originating from setuptools:
See

https://github.com/astropy/asdf-astropy/actions/runs/4471646987/jobs/7856805781#step:7:729

It looks like:

asdf-astropy/pyproject.toml

Line 62 in b3dafcc

packages = ["asdf_astropy", "asdf_astropy.resources"]

may be misconfigured.

I believe the the documentation for this is in: https://setuptools.pypa.io/en/latest/userguide/package_discovery.html#package-discovery-and-namespace-packages

Request for large datasets that use ASDF

Hi @eslavich, where are some large datasets stored in ASDF format?

Add an ASDF schema for the TETE coordinate frame

Description

Not all coordinate frames in astropy.coordinates.builtin_frames have ASDF serialization schemas. This means that it may be impossible to write data with properly-defined gWCS objects to disk, either as pure ASDF files or, as DRAGONS (Gemini DR) does, as FITS files with ASDF-like extensions. One of those missing is astropy.coordinates.TETE and it would be helpful to have a schema for this.

Additional context

I don't claim to understand all the nuances of coordinate schemes so perhaps there's a reason why some coordinate frames have schemas and some don't, or perhaps it's simply limited resources to add each new frame? But I've got some data with RADESYS="GAPPT" and I'd like to make an ASDF-serializable gWCS object from it. There's an archived discussion on GAPPT here but @StuartLittlefair informs me that TETE is the appropriate reference frame to use.

DOC: Getting started should have a quick-start example

"Getting Started" looks pretty sparse:

What happens after installation? Would be nice to actually have "getting started to use the package" example too, not just installation.

setting compression on individual columns of astropy table

Is there a syntax to set compression on individual columns of an astropy table? In the following example, using all_array_compression compresses the columns, but using AsdfFile.set_array_compression() does not.

import numpy as np
from astropy.table import Table
import asdf

t = Table(data=dict(col=np.ones(1)))

with asdf.AsdfFile(tree=dict(table=t)) as af:
    af.write_to('test.asdf')

with asdf.AsdfFile(tree=dict(table=t)) as af:
    af.write_to('test_compressed.asdf',
        all_array_compression='zlib',
    )

with asdf.AsdfFile(tree=dict(table=t)) as af:
    af.set_array_compression(af['table']['col'], 'zlib')  # this has no effect
    af.write_to('test_compressed_col.asdf')

Comparing test.asdf with test_compressed_col.asdf, we see that they have identical checksums (and there's no zlib tag at the beginning of the binary block). So the set_array_compression had no effect.

I've tried variants of this like

set_array_compression(af['table']['col'].base, 'zlib')
set_array_compression(af['table']['col'].data, 'zlib')

but I couldn't get it to work.

I did dig around in the source code a bit, and it looked to me like it's trying to compare the ultimate ndarray base to check if two arrays are the same, but maybe a copy is being made somewhere that's thwarting this detection.

Implement support for astropy.modeling.physical_models.NFW

Neither the schema nor the Converter class exist yet.

MNT: Rename default branch from master to main

Please rename your default branch from master to main as part of astropy/astropy-project#151 , preferably by 2021-03-22. Also a friendly reminder to check documentation, workflows, etc., and update them accordingly. Please don't forget to communicate this change to your users and stakeholders. To summarize:

• Rename branch from master to main, preferably by 2021-03-22.
• Update documentation, workflows, etc., accordingly. -- See #13
• Communicate this change to your users and stakeholders.

Once this is taken care of, you may close this issue.

This is an automated issue. If this is opened in error, please let @pllim know!

time-1.0.0 tag is not being converted

With the release of astropy 5.1, asdf support by astropy has now been deprecated. This has exposed a hidden bug in asdf-astropy, namely that the time-1.0.0 tag is not being converted by asdf-astropy but was falling back on the astropy based converters.

Investigate options for saving tables with standard asdf table tag

asdf-astropy currently does not implement an asdf core table writer:

asdf-astropy/asdf_astropy/converters/table/table.py

Lines 47 to 58 in b532105

	class AsdfTableConverter(Converter):
	tags = ("tag:stsci.edu:asdf/core/table-*",)
	types = ()
	def to_yaml_tree(self, obj, tag, ctx):
	msg = "astropy does not support writing astropy.table.Table with the ASDF table-1.0.0 tag"
	raise NotImplementedError(msg)
	def from_yaml_tree(self, node, tag, ctx):
	from astropy.table import Table
	return Table(node["columns"], meta=node.get("meta"))

Investigate options for triggering asdf-astropy to write a restricted table that is compliant with the core table schema to aid in producing files that might be more easily supported by non-python implementations of asdf.

different output compared to astropy.io.misc.asdf

As discussed in this PR spacetelescope/stdatamodels#104 astropy.io.misc.asdf and asdf-astropy will output different yaml for certain time formats.

It appears that asdf-astropy will always output an object even when a string is read (as seen in this gist: https://gist.github.com/braingram/b369a6326e299088fb98bcef38a01c7d

I don't necessarily see this as an issue as the object format is more descriptive and less ambiguous than the string formatted time (at the expense of slightly more yaml). It may however create issues with downstream packages.

If this is intended behavior, is this documented?

Copy semantics change in numpy 2.0 caused incompatibility with astropy v6.1rc

Please see:

• astropy/astropy#16167
• astropy/astropy-integration-testing#16
• https://numpy.org/devdocs/numpy_2_0_migration_guide.html#adapting-to-changes-in-the-copy-keyword

E       ValueError: Unable to avoid copy while creating an array as requested.
E       If using `np.array(obj, copy=False)` replace it with `np.asarray(obj)` to allow a copy when needed (no behavior change in NumPy 1.x).
E       For more details, see https://numpy.org/devdocs/numpy_2_0_migration_guide.html#adapting-to-changes-in-the-copy-keyword.

Example logs:

https://github.com/astropy/astropy-integration-testing/actions/runs/8568043943/job/23485755895?pr=16

asdf==3.1.0
asdf-astropy==0.6.0
asdf_coordinates_schemas==0.3.0
asdf_standard==1.1.1
asdf_transform_schemas==0.5.0
asdf_unit_schemas==0.2.0

https://github.com/astropy/astropy-integration-testing/actions/runs/8568043943/job/23485756300?pr=16

asdf==3.1.0
asdf-astropy @ git+https://github.com/astropy/asdf-astropy.git@f7a29bab1b1ddfbebe3aea9b60acff7fbd2ee956
asdf_coordinates_schemas==0.3.0
asdf_standard==1.1.1
asdf_transform_schemas==0.5.0
asdf_unit_schemas==0.2.0

cc @astrofrog @neutrinoceros