desihub / desinightlog

DESI Night Log webapp for observers

desinightlog

This is referred to as both the DESI NightLog as well as the DESI Nightly Input (DNI)

This code is running at KPNO on the desi-server and at NERSC. The applications can be accessed here:

• https://nightlog.desi.lbl.gov/ObserverReport
• http://desi-www.kpno.noao.edu:5006/ObserverReport

Details below describe additional information on how these applications are being run.

KPNO:

• Running under dosmanager@desi-server. This is only accessible by the DESI ICS team
• Nightlogs are saved to /software/www2/html/nightlogs/
• Code running from /software/products/desinightlog-main/
• Logs for running the Nightlog: /data/msdos/desinightlog/logs
• Directory structure is set up daily at 12pm MST using a cronjob to ensure permissions are correctly set.
• msdos@desi-server:/data_local/home/msdos/parkerf/desilo/nl_dir.sh

NERSC:

• Running as a Rancher2 Spin Application
• Nightlogs are saved to /global/cfs/cdirs/desi/survey/ops/nightlogs
• Code running from /global/common/software/desi/users/parkerf/desinightlog/py/desinightlog
• Logs for running Nightlog:

Files are transferred between NERSC and KPNO according to https://github.com/desihub/desitransfer

desinightlog's Issues

FROM: Email address should be adjusted on nightly reports

The end of night report currently comes from [email protected]. Since noao.edu no longer technically exists, there is a risk that end-of-night reports could be flagged by spam filters as coming from a fictitious address.[email protected] would probably work fine.

Nightlog data are being modified after sync is complete

In the process of trying to recover nightlog sync data after the restoration of line internet to KPNO, I discovered that files at KPNO were being modified after the sync of those files to NERSC was complete. The timestamps indicate that sometimes this happened days after the conclusion of the night.

The nightlog sync only targets nightlog data from the current night, so if a file is modified after the end of that night, that change will not get synced.

Why are files being modified in this way, and is it expected that late-modified files will never get updated at NERSC?

Handling Out of Disk Errors (and other 'failure to create log' errors)

On Dec. 31, 2022, the disks on the KPNO computers were filled up by the creation of some of the initial nightlog files. Once the issue with disk space was resolved, the log was unable to handle the fact that not all of the nightlog files were created and repeatedly failed to add entries with errors from pandas stating that there were "No columns to parse from file."

Solve issue by Adding a try/except block to handle out of disk space errors and no columns to parse errors.

Loading issues from problem tab only loads problem description, not alarm ID or resolution

When entering a time in the problems tab and loading a problem, it should load all available information including the problem resolution and any alarm ID that was provided. Currently it only loads the problem description

NERSC Log one hour off

Find the time zone issue causing NERSC log to persistently be one hour off of KPNO time. Possible issue with AZ not doing Daylight savings time.

Reduce Redundant Errors

Add a "last exception reported" attribute to the automatic callback function so that we don't log the same error on loop every 30 seconds. This both eats disk space and makes it harder to find the actual cause of some issues.

This attribute should also reset if no exception is reported to prevent false suppression of error messages.

>/< signs being stored correctly but not correctly escaped in HTML display

While entering a night summary which included greater than/less than signs, the summary was being cut off at the first appearance of the greater than/less than sign. However, while loading the summary, the entire text appeared. This should mean that the issue is in how the characters are being escaped in the HTML file display.

Images from night log are only visible in the night log

It is hard to see images that the observers post to the night log. It would be good if images added to the night log could make it into the following locations:

• the NERSC night log
• the night log email
• the Night QA night log (https://data.desi.lbl.gov/desi/spectro/redux/daily/nightqa/20220219/nightqa-20220219.html)

Presently I believe that images posted in the night log only are visible through the KPNO night log, which is only accessible over the VPN and makes understanding reported issues challenging.

Nightlog submission button change color on press

In order to prevent repeated presses and repeated emails, have the button for nightlog submission and email sending change color immediately on button press

Bad Exposure List and updated QA

If an exposure is incorrectly marked as bad and then updated later, it will remain in the bad exposures list.

Nonobserver must enter name two separate times for exposures and problem entries

Name should be stored in log/session memory for both tabs when entered in either.

Night log is not being sent out

The night log has been emailed to the collaboration only irregularly since 20221012. I did not receive a log on the Oct 13, 14, 15, 17, 18, or 19th. Only the 16th came through.

Likewise https://data.desi.lbl.gov/desi/survey/ops/nightlogs/ is not being populated, so we are not seeing the night logs as part of daily operations.

Various issues with merging SO and LO comments on KPNO nightlog

Support observer on 2/17-2/19 noticed several peculiarities with DQ comment submissions.

First, SO data quality comments overwrite LO comments when both submitted by exposure number.

Second, SO comment number N doesn't appear until comment N+1 is submitted. This is occasionally coupled with successfully submitted SO comments disappearing upon the submission of new SO comments.