desihub / desidatamodel Goto Github PK
View Code? Open in Web Editor NEWThe DESI data model.
License: BSD 3-Clause "New" or "Revised" License
The DESI data model.
License: BSD 3-Clause "New" or "Revised" License
This is a meta ticket to capture a list of questions leftover from PR #78.
exposures.fits
, test-tiles.fits
.pix
and simpix
files.TARGETS
.TRUTH
table no longer has CONTAM_TARGET
.NUMOBS_INIT
and PRIORITY_INIT
int64
, when NUMOBS_MORE
is int32
.2**32
or 2**64
exposures?OBSCONDITIONS
table especially.tile-TILEID-FIELDNUM.rst
but the filestile-TILEID.fits
.skies.rst
get named that? It's called sky.fits
.DESI_TARGET
file.NUMOBS_INIT
as int64
, but it is int32
in other files.PRIORITY
?ALLMASK_[GRZ]
columns float32
? The description is pretty clearSEED
keyword in truth table looks dubious. Why not an integer with a comment?FIBERASSIGN
table is radically different from othertile
have to have the same keywords?Data model (.rst) files need to follow a certain format to ensure that they can be parsed by data model validation software. One or more unit tests should be written to guarantee this.
As of PR #90, the description of (1x|4x)_depth
tiles was incomplete. These have a different directory structure from the rest of the tiles.
There are still references to brick-based reduction files, especially in the DESI_SPECTRO_REDUX
section. Can these be removed? In general, what brick-based models need to be retained, and what can be removed?
Next time we review the DESI_TARGET
data model, remove deprecated files such as stdstar.rst
Please review the descriptions in the data model for these files:
DESI_SPECTRO_REDUX/SPECPROD/exposures/NIGHT/EXPID
DESI_SPECTRO_REDUX/SPECPROD/preproc/NIGHT/EXPID
DESI_SPECTRO_REDUX/SPECPROD/calibnight/NIGHT
Note that Issue #106 concerns how to support the somewhat complex structure of the psfnight files in the desidatamodel
automation, but is not related to the descriptions of HDUs, header keywords or columns within those files.
Recent runs of generate_model
are printing char stream
instead of e.g. char[8]
when describing string-valued columns in binary tables. This may be an artifact of the Numpy version used.
Hi,
We have currently the working version of the datamodel for some parts of the MWS output https://github.com/segasai/desidatamodel/tree/mwbranch
Currently I put our output files in DESI_SPECTRO_REDUX/SPECPROD/spectra-NSIDE/PIXGROUP/PIXNUM/
But presumably we want to place them in some other folder like
DESI_MWS_REDUX .
What would be the steps necessary to start merging our branch with the main desidatamodel ?
At least all the travis tests are passing. (I also assume that we do want our datamodel merged).
Sergey
Also CC to @callendeprieto
Is the mockobserving branch still required?
Update datamodel as needed to match the results of software release 17.12 in /project/projectdirs/desi/datachallenge/reference_runs/17.12 .
In some cases where they differ, the datamodel may reflect what we really want. Make sure that the datamodel is clear that this isn't yet implemented, and make sure that there is a ticket for getting the code to implement the desired datamodel.
The data model for psfnight and related files is moderately complex, involving variable-sized image HDUs, as well as data tables that contain two-dimensional, array-valued columns that are also variable-sized. See also desihub/desispec#1778.
@weaverba137 @sbailey I want to add a description of the MTL files generated by this code in desitarget.
Is it enough to include a new mtl.rst
file under doc/DESI_TARGET
or is there anything else that should be done?
The files owned by surveysim that probably need documented data models are:
Survey per-tile and per-night statistics written by surveysim.stats.SurveyStatistics.save
Simulated exposure metadata written bysurveysim.exposures.ExposureList.save
(copied from desihub/surveysim#60)
The data model path DESI_TARGET/fiberassign
is outdated and should be updated:
DESI_SPECTRO_DATA/NIGHT/EXPID/fiberassign-EXPID.rst
. After those files appear in the raw data, they are checked into the tiles
product, so the fiberassign data model in that area can refer to the raw data version.DESI_TARGET/tiles/TILES_VERSION/TILEXX/fiberassign-TILEID.rst
. DESI_TARGET/tiles
is simply a svn checkout of the full tiles
product.The DESI raw data files will have a set of 30 HDUs that are more-or-less identical except for the EXTNAME, B0, B1, B2, ..., R1, R2, ..., Z8, Z9
. Allow all of the data model metadata to be expressed by a single HDU description, rather than repeat the same HDU description 30 times.
Note that this has nothing to do with the INHERIT
keyword in raw data HDUs. Treat that keyword as an ordinary, but not-trivial, required HDU keyword.
It would be helpful if the datamodel could include optional collapsable sections. e.g. we inherit hundreds of keywords from raw data + fiberassign and it would be nice to only show those upon request so that the webpage can focus on the data portions of the files without scrolling through pages and pages of keywords first.
A specific reminder to update the data model for the survey operations MTL files.
Update data model based upon 18.3 reference run at NERSC in:
/global/projecta/projectdirs/desi/datachallenge/reference_runs/18.3
A number of files have the same format but a new location as itemized in
Just like desihub/specter#14, desidatamodel uses fitsio, but doesn't actually need to.
Please review the data model for the top-level exposures-SPECPROD
and tiles-SPECPROD
files. The structural description of these files should be complete, but descriptions of the HDUs, header keywords and columns need work.
Is anything happening with the average_flux_calib
branch?
I'm trying to make sure that all units defined in FITS files (BUNIT
and TUNIT
keywords) conform to the FITS standard. Unfortunately 'electron' is not valid.
See doc/DESI_SPECTRO_REDUX/SPECPROD/exposures/NIGHT/EXPID/calib-CAMERA-EXPID.rst
for an example. Would adu
be an acceptable substitute?
We originally dreamed of having the desidatamodel formats be both human and machine readable such that code could verify if the datamodel files were correct. We found it hard to make something that was human-friendly while also being strict and complete enough for computer parsing. And in the meantime our datamodel has gotten out of sync with our data files. I suggest a pragmatic middle ground:
Write a script to run on a production directory and verify that:
Print a brief report for discrepancies, e.g.
blat/foo/bar/quat.fits missing from datamodel
blat/foo/bar/quiz.fits datamodel missing description of HDU EXTNAME=BIZBAT
Once a file of type X has been checked, all other files of that type could be skipped โ i.e. we don't need to verify 300000 frame files all have the same format. I think all of our files have the form directory/prefix-*.*
โ i.e. you can parse and key off of "prefix" since we don't reuse the same prefix for different kinds of files in different locations.
Even keeping the files and datamodel in sync at that level would be very useful. Checking header keywords, float32 vs. float64, etc. could come later.
The interface might look something like
checkmodel /project/projectdirs/desi/spectro/redux/dc3 --root DESI_SPECTRO_REDUX/PRODNAME
checkmodel /project/projectdirs/desi/target --root DESI_TARGET
I hope this is viable. Actually trying to implement it could reveal any gotchas.
The file doc/DESI_TARGET/fiberassign/tilemeasure.rst
was added on June 13 to master
by @forero (I think...) but it causes Travis tests to fail with a couple warnings / errors (see below). Please fix or move to a branch while in development!
% sphinx-build doc junk
[snip]
building [html]: targets for 109 source files that are out of date
updating environment: 109 added, 0 changed, 0 removed
reading sources... [100%] index
/Users/ioannis/repos/desihub/desidatamodel/doc/DESI_TARGET/fiberassign/tilemeasure.rst:27: ERROR: Unknown target name: "hdu2".
looking for now-outdated files... none found
pickling environment... done
checking consistency... /Users/ioannis/repos/desihub/desidatamodel/doc/DESI_TARGET/fiberassign/tilemeasure.rst:: WARNING: document isn't included in any toctree
done
preparing documents... done
writing output... [100%] index
generating indices... genindex py-modindex
highlighting module code... [100%] desidatamodel.check
writing additional pages... search
copying static files... done
copying extra files... done
dumping search index in English (code: en) ... done
dumping object inventory... done
build succeeded, 2 warnings.
DESI FITS files EXTNAME requirements:
i.e. we reserve the right to re-order or insert HDUs; accessing them by name won't break but accessing them by number might.
For discussion: a potential special case is blank HDU 0, where the "real" data are in HDU 1 (with an EXTNAME!) because they are a binary table that the FITS standard doesn't allow in HDU 0, e.g. the targets catalog. In formal FITS nomenclature I don't even think that HDU 0 is an "extension" but the standard doesn't forbid giving it an EXTNAME anyway (thankfully). I suggest our HDU 0 EXTNAME policy should be:
I suggest that check_model
and unit tests check (3) (the only "shall") and do a best-effort basis on (1) and (2), but if that gets too irritating or there are messy corner cases, don't worry about it.
Comments in PR #68 have a list of known files with HDU 0 that have no EXTNAME.
Currently both header keyword tables and data tables use the same CSS styling. If sphinx can support it, it would be UX helpful to have them visually distinguishable, e.g. with different background colors so that you can quickly tell when scrolling if you are looking at a data table or a header keyword table.
In order to properly interpret the fiberassign HDUs --
https://desidatamodel.readthedocs.io/en/latest/DESI_TARGET/fiberassign/tile.html
one has to read DESI-0530 and DESI-1049 on DocDB.
It would be nice to have the "essential" information self-contained in the desidatamodel
page, especially for the POTENTIAL
HDU, which is a fairly short and straightforward.
Also: note that the extension name has changed: POTENTIAL_ASSIGNMENTS
--> POTENTIAL
I noticed a few problems with the PROTODESI_DATA files that should be addressed before that branch is merged.
toctree
section. The links should simply be e.g. pdFVC
not pdFVC/index
.The DESI_SPECTRO_DATA
section needs updating, in particular, the coordinates files urgently need description.
The data model for the fiberassign tile files has changed and needs to be updated for release 18.6. See also desihub/desitest#12.
We will likely need to expand the set of human-readable directory and file names. For reference, these are things like NIGHT
and EXPID
, which stand in for harder-to-read regular expressions. The existing set is here.
It looks like we no longer use NSIDE
as of everest, but I'm more interested in what to call:
(sv1|sv2|sv3|main)
= SURVEY
?(backup|bright|dark|other)
= CONDITIONS
?(cumulative|perexp|pernight)
= TILETYPE
?[14]x_depth
= TILETYPE
or does that have such a different data model that it requires a separate section?To think about this another way, do we need this structure in the data model:
or can we get away with this:
And similar question for the healpix directory.
Targeting everest first, make sure that all file types give no warnings from fitsverify
.
Please review the data model for the qso_mgii
and qso_qn
files in both healpix and tiles directories. The structural description of these files should be complete, but descriptions of the HDUs, header keywords and columns need work. If necessary it is possible for files to refer to other files, or for HDUs to refer to other HDUs, to reduce duplication of effort.
The data file data/targets/targets.dat should really be a YAML file. As far as I can tell it is already perfectly valid YAML.
This is especially important because the format of targets.dat is significantly different from all other *.dat files in that directory.
It is always a good thing to set EXTNAME
in FITS file HDUs, so that they can referred to by name. However, we haven't been very careful about enforcing this. Therefore I propose:
EXTNAME
recorded for every HDU. Stern warning if not.EXTNAME
is not set.EXTNAME
is not set in one or the other or both.The data model files in the DESI_SPECTRO_(DATA|REDUX|SIM)
sections map cleanly onto directories in the recent reference runs. However, data model files in the DESI_TARGET
section do not. This includes fiber assignment files.
The desimodel files are lightly documented in the desimodel/doc directory and in DESI-0847, but we should add more detailed data model documentation to desidatamodel. This would be a new top-level DESIMODEL directory, at the same level as DESI_SPECTRO_REDUX, etc.
Update the data model for software release 18.11 based on files in /project/projectdirs//desi/datachallenge/reference_runs/18.11/
Lots of formats changed, in particular
After merging #102, a few files still needed updates to descriptions of header keywords in fuji
and guadalupe
.
Many data files contain fiberassign/fibermap table HDUs. To the extent possible, and leveraging the optional table and HDU cross-reference capability implemented in #102, reduce the number of separate descriptions of these tables.
--warning-is-error
.Some files may have entire optional HDUs, and, in some situations, data tables may have missing/optional columns. We can add human-readable documentation describing situations and files where HDUs or columns may be absent. However, there also needs to be a machine-readable mechanism for marking optional items.
One suggestion: create an RST comment that is not rendered into the final document, but would nevertheless be machine readable. For example:
.. Optional Columns - COLUMNA, COLUMNB, COLUMNC
Please review the data model for the zall-(pix|tilecumulative)
files.
In desidatamodel/doc/QUICKLOOK/*.rst, we need the information of which scalar is being tested in *_NORMAL_RANGE and *_WARN_RANGE when the *_STATUS is created. This info was available in older versions of datamodel.
The top level index.rst file explicitly says that the imaging data model is not included in this package, so there is no point in keeping the doc/imaging directory. Sphinx is complaining about not being able to find a link to this directory.
How similar are Nightwatch outputs to Quicklook? That is, could we get away with renaming the current Quicklook directory and tweaking some of the files? Also, does this transition make #49 moot?
The files owned by desisurvey that probably need documented data models are:
Ephemerides written by desisurvey.ephem.Ephemerides
Design HAs written by desisurvey.scripts.surveyinit.calculate_initial_plan
Afternoon planning state written by desisurvey.plan.Planner.save
Tile scheduler state written by desisurvey.scheduler.Scheduler.save
(copied from desihub/desisurvey#91)
Currently there is a way to say "this data model is the same as this other data model", but it would be nice to have this functionality at the HDU level, not just the file level.
Review the end-to-end dataflow for data model consistency. The same concept should have the same name if it appears in multiple files, e.g. EXTNAME=WAVE vs. WAVELENGTH, or SN vs SNR vs S_N. Identify mismatches and propose necessary changes.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.