sbailey / surveyqa

DESI Survey QA Dashboard

surveyqa

This repository is for exploratory work on a DESI survey QA dashboard.

Authors

• Stephen Bailey
• Ana Lyons
• Ruhi Doshi
• William Sheu

surveyqa's Issues

Add options to only run a subset of nights

Add option to surveyqa.core.makeplots to take a list of nights to process. If None, generate all nights in the input exposures. e.g. something like

def makeplots(exposures, tiles, outdir, nights=None):
    # ...
    if nights is None:
        nights = sorted(set(exposures['NIGHT']))
    # ...
    for night in nights:
        # ...

This also means that passing in an empty list will skip generating any nightly plots (useful for faster debugging when iterating on the summary page). Make sure to update docstring for usage of nights. I'll file a separate ticket about how to handle the night-to-night linkage.

Add option to surveyqa.core.makeplots to skip generating the summary page (useful for faster debugging when iterating on the nightly plots).

add moon location and phase to nightly skyplot

On the skyplot of each nightly page, add the location of the moon at midnight on that night, and preferably draw its phase (full, partial, new...). Seeing where the moon is can help explain why the tile selection algorithm was avoiding a part of the sky.

The moon locations and phase might be calculable from routines in astropy. If not and/or if that is slow, it might be more effective to pre-calculate the moon at midnight for every night for the next 10 years and cache that in a dictionary in a json file that is just read and used.

https://github.com/desihub/desisurvey/blob/master/py/desisurvey/ephem.py may have some useful code, but overall that is trying to calculate more than we need here so don't just copy it in full.

Add hour-angle histograms and timeseries

Add code for calculating the "hour angle" of each exposure, which is basically a measure of how far off you are from observing each tile as it passes through the meridian going directly overhead from south to north. Background:

The "local sidereal time" (LST) is a measure of time relative to the distant stars instead of to the sun, i.e. LST depends upon the longitude of where you are on earth and what is overhead right now. The hour angle of a particular exposure = LST - RA, where RA is the Right Ascension of wherever the telescope is pointing.

Detailed routines are available in astropy, but the approximations at https://aa.usno.navy.mil/faq/docs/GAST.php are good enough for here. Converting units from hours to degrees, using MJD instead of JD, and using the longitude of Kitt Peak, I get:

D = MJD - 51544.5
LST = (168.86072948111115 + 360.98564736628623 * D) % 360

Please double check me on those numbers, cross checking to http://www-kpno.kpno.noao.edu/Info/Mtn_Weather/ while noting that MJD = JD - 2400000.5 .

Calculate the hour angle for every exposure and include histograms of that on the summary page and timeseries and histograms on the nightly pages. Coordinate with @ana-lyons and issue #24 for adding placeholders for those plots while @williyamshoe works on adding the calculations to enable those plots.

Watch out for the RA and LST wraparound around 0/360. The hour angle of a DESI exposure should always be between [-60, 60], not something like 359.

Old-school "hour angles" are measured in hours (thus the name...) but let's do everything in degrees.

(re)add hover to sky plot

On the summary page, re-add the hover tooltips to the sky plot, showing the tileid, night of observation(s), and exposure ID(s).

update prev/next links for adjacent nights when updating a single night

This is closely related to issue #47, but it is potentially tricker and messier and I'm not sure exactly how we want to do this, so implement #47 first and then we'll deal with this in a separate branch + PR:

When updating a single night without regenerating all nights, check if the previous and next nights also need to be updated so that their prev/next links are correct. The logic might be something like this:

if output file for this night already exists:
    assume links are correct from earlier run and update only that night
else:
    generate the page for this night
    if previous night output file exists:
        update it so that its "next" link points to this new night
    else:
        do nothing (i.e. don't generate a *new* file for a prev night, but update any pre-existing ones)

    if next night output file exists:
        update it so that its "prev" link points to this new night
    else:
        do nothing

The primary use case here is when we are N+1 nights into the survey and just got new data, we can update the plots for night N without having to re-generate all previous N nights, while still having the Nth night updated so that its "next" link will work.

Summary page layout requests

Here are some layout requests for the summary page. These are motivated by two things:

• trying to fit more information per page so that someone can get an overview in a glance with minimal scrolling
• further polishing the graphical design and layout for overall clarity and professionalism of presentation

Reduce the width of the sky plot and 2 progress plots so that they can fit in a single row of a window that is ~1000 pixels wide. Consider moving the toolbars to the top instead of the sides to help with this.

Reduce the width of the colorbars on the plots that have them (maybe; let's see how that looks).

Reduce the amount of whitespace within and between the gray section title bars so that more content fits vertically on a single screen without scrolling.

Update the various observing conditions plots to all have the same size, matched to the size of the progress plots too. Experiment with a grid layout such that the sky plot is ~2x wider than the progress plots, so that when the window fits those 3 in a single row, the observing conditions histograms fit 4 across. See https://960.gs and https://960.gs/demo for some potential dimensions to work with, though targeting a 960 pixel-wide window might be a bit narrow; we'll have to experiment and see. e.g. something like:

The titles of the histograms are redundant with the x-axis labels. I'm inclined to drop the titles, but that might require adjusting the padding between plots so that the labels clearly go with the axis of the plot above instead of looking like it might be the title of the plot below.

Drop "distribution" y-axis labels of the histograms. If we are going to label the y-axis of the histograms, it would be more informative to have "counts per (binsize)" but that may just use up valuable horizontal space.

Histograms probably don't need any of the tools. Let's try removing those to save space.

AIRMASS -> Airmass, TRANSP -> Atmospheric transparency, SEEING -> Seeing.

All of these requests may be adjusted after we try them out and see what they look like.

gridlines vs. no gridlines for summary page histograms

which looks best?

Feedback on nightly sky plot

This is a great example of the type of inefficiency that we are looking for in these plots. Note how we are jumping back and forth across the galaxy (requiring large telescope movements that take extra time) instead of observing 3 tiles on one side before doing a single large move and then continuing.

Add color coding or some other way of showing what order the tiles were observed in (without hovering to get exposure IDs or otherwise knowing enough observational astronomy to know which way the sky rotates).

Bigger points for the observed tiles could help make the hover easier.

Perhaps in tension with the first request: would be nice to color-code or shape-code the tiles by DARK/GRAY/BRIGHT program to get a quick view of what kinds of things got observed.

Add placeholders for remaining plots

@ana-lyons please add placeholder code/figures for the remaining plots, including

• a function that returns the figure (for now returning a dummy light gray plot of the appropriate size with a title like "PLACEHOLDER: (name of metric)")
• placeholder divs in the HTML template
• the appropriate script/div variables generated and passed into the jinja2.Template.render() call

The goal should be that updating those plots with the real version should be just updating the function, without having to also update the HTML template and Template.render() call (which could cause merge conflicts with other changes).

I'm going to submit some other layout requests, but please make this ticket your top development priority to head off potential merge conflicts / coordination headaches with other work.

Cross reference this to the notes I distributed a few weeks ago (I don't have them in front of me), but I think the missing plots are:

• summary page:
  • sky brightness
  • hour angle
  • total summed exposure time per tile (current exptime plot is per exposure, which is also useful but different)
• per night pages:
  • transparency vs. time
  • exptime (per exposure) histogram, to match exptime vs. time plot
    • I don't think we need total exptime per tile on the nightly pages
  • sky brightness vs. time and histogram
  • hour angle vs. time and histogram

bokeh warnings about different column lengths

After the latest updates (I think from @williyamshoe in commit 79fa701 but I'm not entirely sure) I'm getting Bokeh warnings:

/Users/sbailey/anaconda3/lib/python3.6/site-packages/bokeh/models/sources.py:110: BokehUserWarning: ColumnDataSource's columns must be of the same length. Current lengths: ('date', 4093), ('x', 2), ('y', 2)
  "Current lengths: %s" % ", ".join(sorted(str((k, len(v))) for k, v in data.items())), BokehUserWarning))

The plots still show up, albeit in a different place on the summary page than before (I preferred the old location with the progress plots at the top next to the sky plot).

Please use pull requests for future updates so that we can vet changes before they get into master rather than fixing problems after-the-fact.

Feedback on survey progress plots

Suggestions for survey progress plots:

• Progress should be "Fraction" 0-1 or "Percentage" 0-100.
• Progress should be normalized to the full set of tiles, not just the tiles observed to date. i.e. 200 days into the survey we shouldn't be at "1" yet.
• Add light gray progress line for what we would have achieved if we were perfectly on schedule to finish in 5 years (i.e. a straight line from (startdate,0) to (startdate+5years, 1).
  • on-schedule trend lines for the #tiles vs. time would be normalized to the total number of tiles in each program
• Date axis will need work to be readable

Per-night page layout requests

People liked these surveyqa pages enough that they suggested that during operations we should update the current night page after every new exposure, and having that page permanently displayed on one of the control room monitors. :) That means that we'll want to fit as much as possible on a single page without scrolling, without it becoming unreadable or ugly. Layout suggestions:

Reduce whitespace of upper title and navigation bar. Perhaps move navigation to be next to the title instead of under it.

Try moving the sky plot toolbar to the top instead of side. Drop the toolbars of the histograms.

Make horizontal size of the skyplot graph match the horizontal size of the timeseries plots so that they nicely visually align. Ditto for horizontal size of exposure type counts plot and the histograms. Similarly match the vertical size of the plots that are next to each other, something like:

SEEING -> Seeing, AIRMASS -> Airmass, etc.

Drop "Distribution" y-axis label on histograms (though in general the rule of "always label your axes" remains a good rule, even if I'm requesting to violate it here).

We will likely need to iterate on the balance between size, readability, and visual appeal, so work on this separately from issue #25 (summary page layout) so that progress on one doesn't get bogged down by iterations on the other.

Add sunrise/sunset and twilight times to nightly timeseries plots

Add the times of sunset, sunrise, and evening and morning 12 degree and 18 degree twilights to the nightly timeseries plots. Let's try these as light gray bars so that they are visible but visually don't dominate the plot.

Like #27, this information may be available via astropy, and compare with https://github.com/desihub/desisurvey/blob/master/py/desisurvey/ephem.py, but it may be more effective to pre-calculate and cache these numbers for every night for the next 10 years instead of re-calculating the same thing every time we run the code.

enable viewing plots even with no internet

We have an operational requirement that everything at the telescope can work, even if it is cut off from the internet to the outside world. This currently isn't the case with our plots because they load bokeh from https://cdn.pydata.org/bokeh/release/bokeh-1.0.2.min.js (and similar links for bokeh tables and .css files).

Two pieces:

1. When generating html pages, check if we already have those files in the output directory. If not, fetch them from cdn.pydata.org. OK to use the external "requests" library instead of the lower-level "http.client" library.
2. Update the html output to fall back to the local copy if it can't fetch the standard copy

See https://weblogs.asp.net/jongalloway/using-cdn-hosted-jquery-with-a-local-fall-back-copy for a discussion of how to fall back to a local copy. It basically tries to load from the external server, checks if it got anything, and if not it loads from the local source (which you have pre-cached via (1)).

The reason for pre-caching in (1) instead of directly including the bokeh*.js files in our repo is that we want to support users having slightly different versions of the bokeh library and we have to keep those in sync. i.e. whoever is generating the html pages has a particular version of bokeh, and the html has to point to the matching version of the .js library.

Color coding of nightly skypathplots

Now that PR #22 fixed the ordering of the skypathplots, the order of observations is more clear and the color coding for ordering may not be needed. It is still hard to tell the dark vs. gray vs. bright point shapes, so let's try using colors for that instead.

i.e. let's make a branch and experiment with color coding the skypathplot points by dark/gray/bright. Still connect them by a line, and consider drawing an extra circle or something around the first or last (not both) exposure of the night.

Nightly skypath plots

The hover tool of the nightly skypath plot uses the figure's intrinsic coordinate system ($x, $y) to represent the RA and DEC
I was checking them against the RA and DEC recorded in the data source, and they are similar but not quite exact
Is this something which needs to be updated?

tiles vs. time progress normalization

Something is wrong in the normalization and trend lines for the tiles vs. time plot:

In this case, there should be 3 trend lines, one for each of DARK, GRAY, and BRIGHT with different numbers of total tiles after 5 years for each program:

In [5]: for p in set(t['PROGRAM']):
   ...:     print(p, np.count_nonzero(t['PROGRAM'] == p))
   ...:     
BRIGHT 6028
GRAY 2005
DARK 8038

It also looks like the plot is showing the number of exposures for each program instead of the number of tiles:

In [8]: for p in set(exp['PROGRAM']):
   ...:     ii = exp['PROGRAM'] == p
   ...:     nexp = np.count_nonzero(ii)
   ...:     ntiles = len(set(exp['TILEID'][ii]))
   ...:     print(p, nexp, ntiles)
   ...:     
CALIB 810 1
GRAY 850 296
BRIGHT 1703 785
DARK 4093 1595

e.g. the DARK line should have reached 1595, not 4093.

Ordering bug in nightly skypath plot

There is some ordering bug in the nightly skypath plots, such that the lines connecting the observations aren't in order of increasing MJD. e.g. from night 20191206:

The two observations in the lower right are expids 177,178 and 179,180 (i.e. back-to-back) and yet they are not directly connected by a line.

I suspect that the code is assuming that the exposures list is sorted by MJD (which it is in the original input) but that something else along the way is changing that order. Perhaps this join in get_skypathplot doesn't preserve original input order:

    tiles_and_exps = join(exposures, tiles['STAR_DENSITY', 'EXPOSEFAC', 'OBSCONDITIONS', 'TILEID'], keys='TILEID')

I think a tiles_and_exps.sort("MJD") would fix this, but please try adding that and testing to make sure that the displayed paths are indeed in increasing expid (and MJD) order.

Feedback on summary histograms

Overall looks good. A few comments:

• no need to specify "all non-calib exposures" in title (which runs out of space anyway)

• exposures per tile is an integer quantity, so pick a binning where the bins are 1 unit wide.

Feedback on nightly trend plots

Feedback on nightly trend plots (X vs. time), e.g.

• exclude the calibration exposures from these (i.e. only plot PROGRAM='SCIENCE' exposures). That will get ride of the first N points with airmass=seeing=0.
• reduce the height so that they are 2-3x wider than high (so we can see more plots on the screen).
  • the overall layout code (from @ana-lyons) should make the size choices, not the underlying plotting (from @rdoshi99). I haven't checked the code to see if that is already happening.
• time axis should be relative to local Kitt Peak time (AZ = UTC-7 hours). I might change my mind about this.