GithubHelp home page GithubHelp logo

asdf-format / sphinx-asdf Goto Github PK

View Code? Open in Web Editor NEW
2.0 7.0 9.0 246 KB

Plugin for sphinx for generating documentation from ASDF schemas

License: BSD 3-Clause "New" or "Revised" License

Python 100.00%

sphinx-asdf's Introduction

sphinx-asdf

CI Status

sphinx-asdf is a plugin for sphinx that enables generation of documentation from ASDF schemas.

Installation

To install the latest release on PyPI:

$ pip install sphinx-asdf

The latest development version is available from the main branch on github. To clone the project:

$ git clone https://github.com/asdf-format/sphinx-asdf

To install:

$ cd sphinx-asdf
$ pip install .

To install in development mode:

$ pip install -e .

Usage

The sphinx-asdf plugin provides two sphinx directives, asdf-autoschemas and asdf-schema.

The plugin also provides several configuration variables:

  • asdf_schema_path
  • asdf_schema_standard_prefix
  • asdf_schema_reference_mappings

Basic Example

Consider a package with the following layout as an example:

mypackage/
   setup.py
   mypackage/
      schemas/
         example.org/
            custom/
               foo/
                  a.yaml
                  b.yaml
                  c.yaml
               bar/
                  x.yaml
                  y.yaml
                  z.yaml
      ...
   docs/
      conf.py
      schemas.rst

This package provides schemas with tags that all have the prefix of tag:example.org/custom. The layout of the schema directory reflects this naming convention. We wish to provide documentation for all of our schemas.

First, we will add configuration variables to our docs/conf.py file:

# This variable indicates the top-level directory containing schemas.
# The path is relative to the location of conf.py in the package
asdf_schema_path = "../mypackage/schemas"
# This variable indicates the standard prefix that is common to all schemas
# provided by the package.
asdf_schema_standard_prefix = "example.org/custom"

The variables set in the docs/conf.py file indicate to sphinx-asdf where to locate the schemas based on the names we will use in the documentation.

Now, we use the asdf-autoschemas directive in docs/schemas.rst to create a table of contents for the schema documentation:

.. asdf-autoschemas::

   foo/a
   foo/b
   foo/c
   bar/x
   bar/y
   bar/z

Each item in the list represents the name of a schema to be included in the documents. The names are not paths to the schema files and should not include the file extension. Resolution of the names to actual schema files is handled by the asdf_schema_path and asdf_schema_standard_prefix configuration variables.

We can also use multiple asdf-autoschemas directives if we wish:

These schemas are part of foo:

.. asdf-autoschemas::

   foo/a
   foo/b
   foo/c

And these schemas are part of bar:

.. asdf-autoschemas::

   bar/x
   bar/y
   bar/z

When sphinx-build runs, the sphinx-asdf plugin will automatically generate schema documentation for each of the schemas listed in each asdf-autoschemas directive. In the documentation, each asdf-autoschemas directive will be replaced with a table of contents that contains links to each of the listed schema documents.

Directive settings

While the asdf_schema_path and asdf_schema_standard_prefix configuration variables set global schema resolution settings, it is also possible to set per-directive settings. For example, we could have done the following:

.. asdf-autoschemas::
   :schema_root: ../mypackage/schemas
   :standard_prefix: example.org/custom

This would eliminate the need to set global settings. It also allows any global settings to be overridden on a per-directive basis.

Note

The :schema_root: argument requires a path that is relative to the sphinx configuration file.

External References

Schema references to other schemas in the same package are automatically converted to links in the generated documentation. (This assumes that all of the references correspond to schemas that are explicitly generated by an asdf-autoschemas directive). However, sometimes it is necessary to resolve references to schemas in other packages. The asdf_schema_reference_mapping configuration variable is provided for this purpose. It enables a mapping between references that begin with a particular prefix to a URL indicating another package's documentation.

For example, to enable references to the ASDF Standard documentation to be resolved as links, include the following in your docs/conf.py file:

asdf_schema_reference_mappings = [
    (
        "tag:stsci.edu:asdf",
        "http://asdf-standard.readthedocs.io/en/latest/generated/stsci.edu/asdf/",
    ),
]

Inline documentation

The asdf-autoschemas directive automatically generates individual schema documentation pages and creates a table of contents. However, sometimes it may be useful to include schema documentation inline inside another document. The asdf-schema directive is provided for this purpose.

Using the package described above as an example, the asdf-schema directive can be used to document a single schema inside of another document:

.. asdf-schema::

   foo/a

The behavior of the asdf-schema directive is also governed by the asdf_schema_path and asdf_schema_standard_prefix global configuration variables. The directive also accepts the same :schema_root: and :standard_prefix: arguments as asdf-autoschemas (see Directive settings above) for per-directive configuration.

Contributing

We welcome feedback and contributions to the project. Contributions of code, documentation, or general feedback are all appreciated. Please follow the contributing guidelines to submit an issue or a pull request.

We strive to provide a welcoming community to all of our users by abiding to the Code of Conduct.

sphinx-asdf's People

Contributors

astrofrog avatar braingram avatar cagtayfabry avatar dependabot[bot] avatar drdavella avatar eslavich avatar kmacdonald-stsci avatar mdboom avatar nden avatar pllim avatar pre-commit-ci[bot] avatar williamjamieson avatar

Stargazers

 avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

sphinx-asdf's Issues

Formatting bug when rendering raw ASDF

When an ASDF file's content is injected into documentation like this:

.. asdf:: test.asdf

The resulting rendered HTML looks like this:

test.asdfb'#ASDF' 1.5.0
%YAML 1.1
%TAG ! tag:stsci.edu:asdf/
# etc

Something is going on in that first line, the single quotes aren't actually part of the content and the filename shouldn't be run-on with the ASDF header.

Build Error on Python 3.11

When I make the asdf-astropy docs locally in a Python 3.11 virtual env I consistently get the following error:

Extension error (sphinx_asdf.connections):
Handler <function autogenerate_schema_docs at 0x131564f40> for event 'builder-inited' threw an exception (exception: global flags not at the start of the expression at position 441)

I can reproduce this consistently (across newly created clean envs).

pip list:

Package                       Version              Editable project location
----------------------------- -------------------- -----------------------------------------------------
alabaster                     0.7.13
asdf                          2.15.0
asdf-astropy                  0.4.1.dev25+gc6608be /Users/wjamieson/workspaces/asdf-astropy/asdf-astropy
asdf-coordinates-schemas      0.2.0
asdf-standard                 1.0.3
asdf-transform-schemas        0.3.0
asdf-unit-schemas             0.1.0
astropy                       5.3.1
astropy-sphinx-theme          1.1
attrs                         23.1.0
Babel                         2.12.1
certifi                       2023.5.7
charset-normalizer            3.2.0
contourpy                     1.1.0
coverage                      7.2.7
cycler                        0.11.0
docutils                      0.18.1
fonttools                     4.41.0
graphviz                      0.20.1
hypothesis                    6.81.1
idna                          3.4
imagesize                     1.4.1
importlib-metadata            6.8.0
iniconfig                     2.0.0
Jinja2                        3.1.2
jmespath                      1.0.1
jsonschema                    4.17.3
kiwisolver                    1.4.4
MarkupSafe                    2.1.3
matplotlib                    3.7.2
mistune                       3.0.1
numpy                         1.25.1
numpydoc                      1.5.0
packaging                     23.1
Pillow                        10.0.0
pip                           23.1.2
pluggy                        1.2.0
psutil                        5.9.5
pyerfa                        2.0.0.3
Pygments                      2.15.1
pyparsing                     3.0.9
pyrsistent                    0.19.3
pytest                        7.4.0
pytest-arraydiff              0.5.0
pytest-astropy                0.10.0
pytest-astropy-header         0.2.2
pytest-cov                    4.1.0
pytest-doctestplus            0.13.0
pytest-filter-subpackage      0.1.2
pytest-mock                   3.11.1
pytest-openfiles              0.5.0
pytest-remotedata             0.4.0
python-dateutil               2.8.2
PyYAML                        6.0
requests                      2.31.0
scipy                         1.11.1
semantic-version              2.10.0
setuptools                    65.5.0
six                           1.16.0
snowballstemmer               2.2.0
sortedcontainers              2.4.0
Sphinx                        6.2.1
sphinx-asdf                   0.2.0
sphinx-astropy                1.9.1
sphinx-automodapi             0.15.0
sphinx-bootstrap-theme        0.8.1
sphinx-gallery                0.13.0
sphinx-rtd-theme              1.2.2
sphinxcontrib-applehelp       1.0.4
sphinxcontrib-devhelp         1.0.2
sphinxcontrib-htmlhelp        2.0.1
sphinxcontrib-jquery          4.1
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          1.0.3
sphinxcontrib-serializinghtml 1.1.5
toml                          0.10.2
tomli                         2.0.1
urllib3                       2.0.3
zipp                          3.16.1

`rename` or `refname`

As found in #80

elif shorten:
# TODO this should probably be:
# refname = schema_id
# as it was previously
# rename = schema_id
# and ruff cleaned this up as unused. However, changing it
# to refname breaks some downstream packages
pass

Determine if this should be refname and why it breaks downstream packages.

sphinx-asdf safe for parallel build

I encountered the following error running sphinx / sphinx-asdf with -j auto flag for parallel processing

Warning, treated as error:
the sphinx_asdf extension does not declare if it is safe for parallel reading, assuming it isn't - please ask the extension author to check and make it explicit

This seems to be the relevant part regarding extensions in the sphinx documentation:
https://www.sphinx-doc.org/en/master/extdev/index.html#extension-metadata

Any idea if sphinx-asdf is safe for parallel reading and where to add the metadata?

`RemovedInSphinx70Warning: sphinx.io.read_doc() is deprecated.`

It looks like there are a few deprecation errors in RTD builds on Python 3.11:
https://readthedocs.org/projects/jwst-pipeline/builds/20117878/

/home/docs/checkouts/readthedocs.org/user_builds/jwst-pipeline/envs/7552/lib/python3.11/site-packages/sphinx_asdf/connections.py:25: RemovedInSphinx70Warning: sphinx.io.read_doc() is deprecated.
  doctree = read_doc(env.app, env, env.doc2path(docname))

doctree = read_doc(env.app, env, env.doc2path(docname))

It looks like read_doc() was deprecated in Sphinx 5.0.0 b1:
https://www.sphinx-doc.org/en/master/changes.html#id170

Parsing all files add a huge overhead

To find the asdf directive it seems that this plugin relies on fully parsing all the rst files, which on astropy introduces a huge overhead:

find_autoasdf_directives /home/simon/Dev/astro/astropy/docs/api/astropy.wcs.wcsapi.BaseHighLevelWCS.rst : 0.19 sec.
find_autoasdf_directives /home/simon/Dev/astro/astropy/docs/api/astropy.tests.helper.check_pickling_recovery.rst : 0.03 sec.
find_autoasdf_directives /home/simon/Dev/astro/astropy/docs/units/standard_units.rst : 0.08 sec.
find_autoasdf_directives /home/simon/Dev/astro/astropy/docs/api/astropy.utils.misc.JsonCustomEncoder.rst : 0.08 sec.
find_autoasdf_directives /home/simon/Dev/astro/astropy/docs/api/astropy.nddata.PartialOverlapError.rst : 0.03 sec.
find_autoschema_references : 217.85 sec.

I don't know much about Sphinx plugins API but I'm surprised that there is no better way to proceed ?

For reference the code involved here:
https://github.com/spacetelescope/sphinx-asdf/blob/64ed11ab19a007bd1861c82cc2710f1e6ec61286/sphinx_asdf/connections.py#L19-L48

Preserve sphinx theme set in project configuration

Currently the generated schema documentation does not conform to the theme that is set in the project's docs/conf.py. This is because we use the sphinx bootstrap stylesheets to generate schema documentation, so they get pulled in regardless of whether the project is using sphinx-bootstrap-theme or not.

I'm not sure whether this will be possible and/or how much effort it is worth.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.