astrodbtoolkit / astrodb_utils

astrodb-utils

The following tables are expected by AstroDB Toolkit and the AstroDB_scripts package:

• Sources
• Publications
• Names
• Telescopes
• Instruments You may modify these tables, but doing so may decrease the interoperability of your database with other tools.

Developer Setup Instructions

• Make new environment with Python=3.10
• Install dependencies using an editable install:

  pip install -e ".[test]"
  

• In the astrodb_utils/tests/ directory, clone the astrodb-template-db repo:

  git clone https://github.com/astrodbtoolkit/astrodb-template-db.git
  

• Be sure to run tests from the top level directory.

astrodb_utils's Issues

Add `update_version` function.

A version of this function is here: https://github.com/kelle/SIMPLE-db/blob/script-reorg/scripts/updates/2023/update_version.py

Open readthedocs here?

Make example scripts for ingesting many things

Remove all functions which handle ingesting multiple things (eg spectra/spectrum). Instead, give examples which demonstrate how to do make the loops and use the single item functions.

Focus on scripts which ingest ONE thing.

Update `ingest_source` function to provide guidance on Simbad resolvable source names

[moved from https://github.com/SIMPLE-AstroDB/SIMPLE-db/issues/135]

Prompt user to double check the primary identifiers being added and provide guidance on how to change but keep simbad resolvable. E.g., swap the primary identifier to be S Ori 70 rather than [BZR99] S Ori 70

add description and reference to ingest_instrument script

Current organization of repos and packages

https://app.mural.co/t/stsciinternal7446/m/stsciinternal7446/1706043059814/0e06fd6ca3ce8ecbefec953a6345323edcf652cc?sender=u9972c6cc42643b1ed44a3448

calling structure of functions with many inputs

Right now, we are using positional arguments for functions ingesting data. For functions with a lot of inputs, this becomes quite unwieldy. Should we have a different method? maybe a dictionary?

Add "protected tables" documentation

Similar to astrodbtoolkit/astrodb-template-db#20

Add units expectations to ingest functions

In SIMPLE, we assume units and note what the expectation is in the docs. This is extremely error prone. Other options

• Include units in the column name. e.g., parallax_mas, parallax_error_mas
• Include units as a column in all tables with a measurement. We could then have a test to make sure that that string corresponds to a astropy unit.

Change filter width column to `effective_width_angstroms`

In the PhotometryFilters table, we ingest the filter width provided by the SVO service. SVO provides both the FWHM and the effective width. Does anyone have strong or informed opinions about which ones of these we should use for "width"?

Figure out REFERENCE_TABLES

Right now, the REFERENCE_TABLES definition in utils.py (line 72) is superseding any other definition. We need to figure out how to handle passing this variable.
https://github.com/astrodbtoolkit/astrodb_scripts/blob/6d93f7c6daef2bfa77d3bdc19a81f6f00678731c/astrodb_scripts/utils.py#L72

Coordinate transform script

[moved from https://github.com/SIMPLE-AstroDB/SIMPLE-db/issues/264]

We should make a "helper" script which takes the coordinates from the Sources table and uses whatever astrometry (proper motion) to transform coordinates to a specific date.

I would use astropy.

• see Evolving coordinate positions between epochs: Part 3: Coordinates Tutorial
• apply_space_motion()

Update the Documentation and Modify Ingest Publication Function

Moved from SIMPLE-AstroDB/SIMPLE-db#346

In creating some changes, we need to update the documentation and also modify the ingest publication function to work with the new system of author codes.

@shererynA - do you remember what this was about? Could you please provide an example?

Write tests related to the `adopted` column

[Condensed from https://github.com/SIMPLE-AstroDB/SIMPLE-db/issues/84]

Tests should fail if there are multiple measurements set as 'adopted'. That is, if set to false/null they would not trigger a failure, just when you have 2 or more entries set to true.

[On hold until a table with an adopted column is added to the template database: https://github.com/astrodbtoolkit/astrodb-template-db/issues/37)

RTD organization

I have some questions about how we want to organize the readthedocs. A few classic pages are:

• Installation
• Getting started — this is what I have the most questions about, I guess. What does a "quickstart" look like for this project? Maybe #31 goes here?
• API (auto-generated docstrings, etc.)

We could also have:

• developer information (#11 could go here)
• FAQ

What do y'all think?

Improve `find_publication` function

Right now, the find_publication function does not find Gaia and Gaia23, it only finds Gaia. It should find both. There is currently a skipped test.

Fix test configuration

For some reason, running pytest doesn't work. Error is
ModuleNotFoundError: No module named 'tests.test_utils'.

Probably something not quite right to pyproject.toml and might be related to src layout.

provide instructions on how to setup ADS TOKEN

[moved from https://github.com/SIMPLE-AstroDB/SIMPLE-db/issues/152]

For the ingest_publication function, an ADS TOKEN environment variable should be set. We should provide instructions on how to do that.

make account on ADS and get the token from ADS: https://ui.adsabs.harvard.edu/help/api/

Then add it via an environment variable in your .bashrc or whatever init file you have.

export ADS_TOKEN="<your token>"

Setup releases and putting in pypi

Make this repo pip installable and have releases and version numbers.

Add photometry related scripts

Move from SIMPLE the scripts which:

• Add photometry
• Query SVO
• assign UCD

How to add/where to put test data

in order to test these scripts, we need a database. Gotta figure out how that all fits in. Would be nice to have a folder structure which makes it clear that the data in here is for testing only. Maybe nested under tests/?

Rename this package and repo `astrodb_utils`

In SIMPLE, we decided that scripts are "one-off" scripts and utils are the helper functions. In keeping consistent with that thinking, this package should be named astrodb_utils.

Write helper function to check for values

A lot of the ingest scripts contain similar code to look to see if values already exist in the database. (e.g., telescope, instrument, filter, etc.) I think we could use a helper function here. Something like this

def in_database(db, value, table, field):
       match = (
            db.query(db.table)
            .filter(db.table.c.field == value)
            .table()
        )

        if len(match) == 0:
            msg = f"Value {value} not found in {table} table."
            return false
        else
            return true

I'm not exactly sure how to compose the pointers in the query (f"db.{table}.c.{field}?) but I think the answer is in here: https://docs.sqlalchemy.org/en/20/faq/sqlexpressions.html

Question re:"*" usage in functions

What is * supposed to be in:

astrodb_utils/astrodb_utils/utils.py

Line 122 in 640b6bf

def find_source_in_db(db, source, *, ra=None, dec=None, search_radius=60.0):

That's used throughout several functions, but it's not clear to me what it represents or how it's meant to be used. If this is meant to be *args isn't it missing the args part of it so it can be referred to in the code?

hyphen vs. underscore?

Hyphen is more consistent with, e.g., SIMPLE-db, but we can't import modules with a hyphen without importlib I believe...

dynamically infer docs info from toml

I've hardcoded a lot of docs info in conf.py. Doesn't need to be that way. Flagging so that I remember to fix!

modify init

from utils import *

Use `sqlalchemy` classes for ingesting instead of dictionary

Takes more advantage of the schema and is more ...direct. @dr-rodriguez to provide an example.

Consolidate database loading and generating scripts

The generate_database.py creates the database. It's used a lot. Not sure if should live in this package or as a custom script. @dr-rodriguez ?