desihub / desiperf

Tools for monitoring DESI Instrument Performance

desiperf

Tools for monitoring DESI Instrument Performance

See google doc for more description: https://docs.google.com/document/d/1I7UqqHTs2U1vuIGsLK8OrTWVK29r0CoDxNgy69c8Zc8/edit

Directory structure for (most) modules in desihub

• bin/: executable scripts; should be added to $PATH
• py/desiperf/: for modules that one would import; py/ should be added to $PYTHONPATH to enable python import desiperf to find the desiperf directory
• doc/: documentation
• etc/: for small config files or other things that don't fit elsewhere

To run InstPerfApp on local machine (temporary)

• cd py/desiperf/
• bokeh serve --show instperfapp
• output will be found at http://localhost:5006/instperfapp

desiperf's Issues

LinAlg error with some attributes

When calculating the trend line with certain attributes, can get this error:

error handling message Message 'EVENT' (revision 1) content: '{"event_name":"button_click","event_values":{"model_id":"1032"}}': LinAlgError('SVD did not converge in Linear Least Squares')

Change detector data

Instead of having repeat rows for each amp, create columns for each.

Add new attributes to tooltips

Include expid, sequence, obstype, flavor, program and date_obs to each tooltip in addition to selected attribute.

Focus Attribute

I noticed that the Focus attribute (under the Observation catagory) can not be plotted. Each datapoint for Focus looks to be a list of 6 values (for example here is the first data point [1136.6, -358.0, 718.5, -7.0, 20.0, 173.6]). If we want to be able to plot the Focus we will need to split this up, which is doable but I don't know what any of the individual values mean, so I don't know which one to take.
Or should we plot all six separately?

Identify telemetry attributes that aren't useful

Add to bokeh tools

Decide on set of bokeh tools for plots and add these

Capability to bin data on plots

To better identify trends, we will likely want to bin data over different time periods. Including a bokeh slider to interactively bin data will be useful.

add etc data to data_handler

Take data that was generated by Paul Martini and make it available to the throughput page. This needs to be complete before plots can be made on the throughput page.

Select for sets of attributes

The current list of attributes is too long to look through. There should be a list of groups of attributes. Here are some options:

Exposures:
['EXPID' 'data_location' 'targtra' 'targtdec' 'skyra' 'skydec' 'deltara'
'deltadec' 'reqtime' 'exptime' 'flavor' 'program' 'lead' 'focus'
'airmass' 'mountha' 'zd' 'mountaz' 'domeaz' 'spectrographs' 's2n'
'transpar' 'skylevel' 'zenith' 'mjd_obs' 'date_obs' 'night' 'moonra'
'moondec' 'parallactic' 'mountel' 'sequence' 'obstype']

GFA data
['ccdtemp_mean' 'hotpeltier_mean' 'coldpeltier_mean' 'filter_mean'
'humid2_mean' 'humid3_mean' 'fpga_mean' 'camerahumid_mean'
'cameratemp_mean']

Guider
['combined_x' 'combined_y' 'guider_time_recorded']
['duration' 'expid' 'seeing' 'frames' 'meanx' 'meany' 'meanx2' 'meany2'
'meanxy' 'maxx' 'maxy']

Telemetry
['air_flow' 'air_temp' 'truss_temp' 'air_in_temp' 'flowrate_in'
'mirror_temp' 'probe1_temp' 'probe2_temp' 'air_dewpoint' 'air_out_temp'
'decbore_temp' 'flowrate_out' 'hinge_s_temp' 'hinge_w_temp'
'mirror_status' 'glycol_in_temp' 'servo_setpoint' 'topring_s_temp'
'topring_w_temp' 'truss_etb_temp' 'truss_ett_temp' 'truss_ntb_temp'
'truss_ntt_temp' 'truss_stb_temp' 'truss_sts_temp' 'truss_stt_temp'
'truss_tsb_temp' 'truss_tsm_temp' 'truss_tst_temp' 'truss_wtb_temp'
'truss_wtt_temp' 'casscage_i_temp' 'casscage_o_temp' 'chimney_ib_temp'
'chimney_im_temp' 'chimney_it_temp' 'chimney_os_temp' 'chimney_ow_temp'
'glycol_out_temp' 'mirror_avg_temp' 'mirror_eib_temp' 'mirror_eit_temp'
'mirror_eob_temp' 'mirror_eot_temp' 'mirror_nib_temp' 'mirror_nit_temp'
'mirror_nob_temp' 'mirror_not_temp' 'mirror_rtd_temp' 'mirror_sib_temp'
'mirror_sit_temp' 'mirror_sob_temp' 'mirror_sot_temp' 'mirror_wib_temp'
'mirror_wit_temp' 'mirror_wob_temp' 'mirror_wot_temp' 'probe1_humidity'
'probe2_humidity' 'primarycell_i_temp' 'primarycell_o_temp'
'mirror_desired_temp' 'telescope_timestamp' 'centersection_i_temp'
'centersection_o_temp' 'gust' 'split' 'dewpoint' 'humidity' 'pressure'
'wind_speed' 'temperature' 'wind_direction' 'tower_timestamp' 'C_floor'
'SCR_roof' 'platform' 'LCR_floor' 'lights_low' 'shack_wall' 'stairs_mid'
'LCR_ceiling' 'lights_high' 'mirror_cover' 'stairs_lower' 'stairs_upper'
'utility_room' 'LCR_ambient_N' 'LCR_ambient_S' 'shack_ceiling'
'shutter_lower' 'shutter_upper' 'dome_timestamp' 'telescope_base'
'utility_N_wall' 'dome_back_lower' 'dome_back_upper' 'dome_left_lower'
'dome_left_upper' 'SCR_E_wall_coude' 'SCR_roof_ambient'
'dome_right_lower' 'dome_right_upper' 'LCR_N_wall_inside'
'LCR_W_wall_inside' 'LCR_N_wall_outside' 'LCR_W_wall_outside'
'SCR_E_wall_computer']

Hexapod
['rot_rate' 'hex_status' 'rot_offset' 'rot_enabled' 'rot_interval'
'hex_trim_0' 'hex_position_0' 'hex_trim_1' 'hex_position_1' 'hex_trim_2'
'hex_position_2' 'hex_trim_3' 'hex_position_3' 'hex_trim_4'
'hex_position_4' 'hex_trim_5' 'hex_position_5' 'hex_tweak']

ADC
['adc_home1' 'adc_home2' 'adc_nrev1' 'adc_nrev2' 'adc_angle1' 'adc_angle2']

FVC
['shutter_open' 'fan_on' 'temp_degc' 'exptime_sec' 'psf_pixels'
'fvc_time_recorded'

SPECTROGRAPH
['nir_camera_temp_mean' 'nir_camera_humidity_mean' 'red_camera_temp_mean'
'red_camera_humidity_mean' 'blue_camera_temp_mean'
'blue_camera_humidity_mean' 'bench_cryo_temp_mean' 'bench_nir_temp_mean'
'bench_coll_temp_mean' 'ieb_temp_mean']

add plots to the throughput page

This issue can be started after issue #10 is completed.

In throughput.py page, add plots that use the etc data. Make plots similar to the other pages.

• attr1 vs. attr2
• attr1 vs. exposure or time
• attr2 vs exposure or time

attr1/attr2: 'estimated_snr', 'goal_snr', 'seeing', 'transparency', 'skylevel', 'max_exposure_time'

Include drop down menus for the attributes so they can be selected.

Document origin of telemetry data

It would be great to add a help features that describes the origin of the various quantities that may be plotted through the app. Maybe we could add a page that is called "Glossary" that lists all of the parameters? Or even better would be a separate page on the desi wiki that has this information, as it should be of broader interest. In that case, we could have a help feature that references the wiki page. It would be an awesome, heroic task for someone to create such a telemetry glossary.

Collection of bugs

• Fix positioner data and plots. Positioner 595 doesn't work; OFFSET_FINAL not in file
• airmass, air_temp_diff, wind_speed not in data
• Guiding and tput plots not changing with options
• Detector Noise data issue with CAMERA_TEMP

Add button to save selected data

When data has been selected with the lasso tool, have a button where you can save the subset of data to a csv file on disk for further analysis

Create simulated throughput data

Decide on what data we will want to plot/analyze and create a data set of simulated throughput values.

add code to get individual positioner data

Swap axes on Detector Noise Performance page

The detector noise performance currently plots the parameters in Menu Option 1 (e.g. READNOISE, BIAS) on the x-axis of the three panels, and the parameters in Menu Option 2 (e.g. EXPID) on the y-axis. It would be more natural to plot quantities such as READNOISE and BIAS on the y-axis vs. EXPID (and eventually time) on the x-axis, rather than vice versa, as EXPID is likely to be common to all three panels (and channels).

Sharable URL with full request

A useful tool would be to be able to share an URL that contain a full request.

This would be useful to

• share findings in emails/slack/slides ...
• create links to standard monitoring plots in wiki pages or elsewhere, like spectrograph throughput vs time (once we have some throughput saved in nightwatch QA).

Make telemetry page searchable

Now, a full table shows all the telemetry and their location, etc. We should make this searchable.

Better arrangement of items on pages

Pages are currently quite messy. Come up with a better organization for each page.

Add focalplane scatter plot

Add block of code for plotting values by positioner in the focalplane.

Load telemetry file dynamically

There are ways to pull the data from a google sheet and read it in. We should do this for the telemetry page so it can always be up to speed. Currently, as a stop measure, we download the google file as a csv and load it that way.

Spectrograph Page Cleanup

On the Spectrograph page, when the Category is "Spectrograph" I think it does not make sense to have the attribute be specific to an individual camera (although I know that is the way the data are stored). For example, if I set "X Category" to Observation and "Y Category" to Spectrograph and then choose to plot EXPID vs. humidity, I have to set the Y Attribute to "nir_camera_humidity_mean" for all three plots. I think it would be more useful to have 'camera_humidity' and 'camera_temp' and then plot the nir_camera_humidity_mean in the "NIR Detectors" panel, red_camera_humidity_mean in the "Red Detectors" panel, etc. At least I can't forsee wanting to plot nir_camera_humidity_mean vs. properties of the other two cameras.

Note that this isn't true for all attributes under the "Spectrograph" category. The bench temps (and ieb temp) are not specific to cameras.

attrs all CAPS or lower()

Style of attributes in lists are all over the place. Make all lower or upper case.

Change Landing Page

Decide what should be on the front page, whether or not everything is enabled.

Add some indication that the app is working to generate a new plot

Sometimes the app takes a while to update a plot after clicking the 'Plot' button. It would be helpful to have some indicator that the app is working on generating the new plot. At present it is unclear if it failed or if generating the plot is simply in progress. Related to this, I've noticed the "Data Overview" updates well before the plot does in some cases. This is not necessarily bad, but might add to a user's confusion.

Data file sizes

Hi @parfa30 -- would you take a look the changes in my branch to accommodate a .fits.gz file instead of the .csv files? I'm getting errors (so please don't merge this), and would appreciate it if you see something obviously wrong with what I've done. Thanks!

Combine Focalplane and Guiding pages into one FocalPlane page

Use radio buttons to limit attribute selections. For example, press a button that says "guiding" or something and then the drop down list includes just attributes in that area. As it stands now, there are just too many options to choose through and is confusing.

Guiding and Focalplane pages use the exact same data essentially.

Add data to positioner files and guider files

Additional data is needed for certain plots. I will add this data , but implementation of how the data is compiled in the future may change.

Add binning to Detector page

Several things on other pages aren't yet on the detector page.

Add trend line to plots

There should be an option to add a trend line to plots and calculate the slope, etc.

Can't start up the application

When I run ' _bokeh serve --show .\instperfapp_ ' I receive the following error in the terminal:

2020-10-01 13:57:26,469 Error running application handler <bokeh.application.handlers.directory.DirectoryHandler object at 0x0000022789545DA0>: cannot convert float NaN to integer
File "main.py", line 60, in data_text:
fp = ['FocalPlane Data', int(min(fps.NIGHT)), int(max(fps.NIGHT)), int(min(fps.EXPID)), int(max(fps.EXPID))] Traceback (most recent call last):
File "C:\Users\socce\Anaconda3\lib\site-packages\bokeh\application\handlers\code_runner.py", line 197, in run
exec(self._code, module.dict)
File "C:\Users\socce\Desktop\KPNO\DESI\desiperf\py\desiperf\instperfapp\main.py", line 84, in
df = data_text(ds, fps)
File "C:\Users\socce\Desktop\KPNO\DESI\desiperf\py\desiperf\instperfapp\main.py", line 60, in data_text
fp = ['FocalPlane Data', int(min(fps.NIGHT)), int(max(fps.NIGHT)), int(min(fps.EXPID)), int(max(fps.EXPID))]
ValueError: cannot convert float NaN to integer

so either fps.NIGHT or fps.EXPID has a NaN, but I don't know how we want to handle them.

Detector selections on Amps

In detector/qa data, everything is arranged by AMPS. Maybe color code the data in each plot by Amp.

Get x_axis_label and y_axis_label names to change when selecting new data

Both Paul and I have tried to have the x_axis_label and y_axis_label on the plots to change when the selections have changed but so far have not had luck. This stackoverflow has some suggestions: https://stackoverflow.com/questions/24131501/x-and-y-axis-labels-for-bokeh-figure

Selection based on flavor

Flavors include science, dark, etc. We should be able to look at only certain data based on this.

issues selecting some attributes

Experienced this error in Positioner plots, but might show up in others. A couple issues:

1. Sometimes when select a new attribute an Re-Plot, nothing happens
2. With some attributes, get this error:
   error: ValueError('On entry to DLASCL parameter number 4 had an illegal value')

add info about saved selected data

Show where the data is saved when hit "save selected data" button

lines to identify changes in survey

One simple idea for tracking some of the dates when things changed may be to have vertical lines plotted in the time axes to denote when changes happened, i.e., beginning of the different mini-SVs, shutdown/restart for Covid-19, implementation of anti-collision software, etc.

Identify what makes startup and re-plotting go so slow

Not sure if we can fix it but it would be good to know what the culprit is.

Binned data not quite right in spectrograph plots

Looks like binned data is for the Blue. plots only and repeating on the red and infrared plots.

Add page for Focus

Add tab/page for data from donut analysis and focus

Improve positioner data

The data_script.py doesn't build the positioner files. These need to be improved.

Spectral trace coordinates and wavelength calibration vs. environmental parameters

Include dx (trace shift) and dy (shift in wavelength direction) from each camera for each spectroscopic exposure. This is to enable studies of changes in where the spectra are dispersed on the detector vs. environmental conditions, especially in the Shack. Note this is motivated by Julien'Guy's discovery of a correlation between trace shift and Shack humidity in the blue and red cameras.

Deal with outliers

We might want a way to identify outliers and remove them from certain analyses.