desihub / desisurveyops

Scripts and code for DESI Survey Operations

desisurveyops

Scripts and code for DESI Survey Operations

desisurveyops's Issues

Problematic observations from recent engineering time, 20220413, 20220414, 20220416

This was discussed on Slack and by email, but I am unclear as to the status. The beginning of the night on 20220413 featured engineering observations on main survey tiles. We need to mark some of these exposures as bad.

QA for observations 20619/130091 and 25297/130094 look fine to me, though the night log reports that 25297/130094 did not have a turbulence correction applied. I think we should keep these two observations.

QA for 22170/130095 looks terrible and this exposure needs to be marked bad and the tile reprocessed. This duplicates #43 and #40 .

Night log reports observations of a few other tiles that may have already been marked bad and do not make the nightly QA (e.g., 22202, 22211, 22203, 20638). Because there is no QA, I have not inspected these tiles.

On 20220414 the observers started observing backup tiles to avoid problems like the above. But then they moved to BRIGHT tile 23075 and did more test observations. It looks like we got 4 back to back short exposures there.
https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/20375/20220414/tile-qa-20375-thru20220414.png
QA looks fine to me and I don't see major positioning problems like those we saw for other fast sequencer test observations. It's certainly not normal survey observations to have 4 short exposures on a BRIGHT tile, though, so we could consider marking all of these as bad. The rest of 20220414 looks fine to me.

On 20220417, I don't see anything dubious in the night log. There's a weird exposure of 42426 in the nightqa. It's weird because:

• the QA only shows one observation on 20220416, so why am I seeing it on 20220417? (note: tile was also observed on 20220417, so maybe the observation on 20220417 was marked bad, but I'm not sure I want to see it in that case?)
• Positioning is bad on all petals except P0, so we'd want to mark the exposure as bad?
  It looks to me like the 20220416 exposure on this tile was good, but the 20220417 exposure was bad. But the QA script includes the flagged bad exposure on 20220417, making the positioning on all petals look bad despite the fact that this exposure was marked bad? @araichoor .
  https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/42426/20220416/tile-qa-42426-thru20220416.png
  https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/42426/20220417/tile-qa-42426-thru20220417.png
  I don't know why we would have failed to position on all petals; that never happens normally?

Nothing else exceptionally alarming.

Short version: let's mark 22170/130095 bad and reprocess. Let's discuss 23075, 20619, 25297, but probably keep all of these; their QA is fine. Let's check to see if there's a bug in the QA script where data from bad, excluded exposures are included in the positioner offset plots, as suggested by 42426.

Dark tiles on 20220219 observed have slight excess at z > 5

20220219 had bright moon and great seeing, and had a number of dark tiles observed. Some remaining moon-induced sky gradients lead to a bit of signal in the spectra that get identified as z > 5 spectra based ~entirely on the continuum as far as I can tell. This doesn't look to affect LyA rates and I am marking these as good, with a few exceptions. But these would benefit from future reprocessing with a more flexible sky model.

https://data.desi.lbl.gov/desi/spectro/redux/daily/nightqa/20220219/tileqa-20220219.pdf
Affected: 1610, 1611, 1650, 2563, 3143, 6055, 7067, 7217, 7893, 7914, 8801, 10006, 11566.

TILEID 8801 looks especially bad and has a weird clump on P1 and has been left unsure.
https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/8801/20220219/tile-qa-8801-thru20220219.png

Tileid=3298: accumulation of Z~0.25 for FIBER around 1900

This TILEID=3298 has been observed on 20220519 with EXPID=134023.
It has been marked as good.

Nevertheless, there is an accumulation of Z~0.25 for 1890 < FIBER < 1912 (approximate range).
https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/3298/20220509/tile-qa-3298-thru20220509.png

There is a drop in the flux at lambda~4660A (you can see it when a different z is picked), which is often interpreted at a negative oii flux:
tmp-3298-thru20220509-1890fiber1911.pdf

Maybe some fibers in that range could be marked as bad.

Original slack message: https://desisurvey.slack.com/archives/C01HNN87Y7J/p1652218526122329

Tileid=21809 marked as unsure; large sky residuals, possibly due to weird scattered light path?

This TILEID=21809 has been observed on 20220512 with EXPID=134448.
It has been marked as "unsure".

It has ZSKYPDF alert for petal=6,7,8.
The n(z) look a bit awkward.
https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/21809/20220512/tile-qa-21809-thru20220512.png

The sframesky has some brightish lines in the z7 camera, but I can t figure out where that does come from (no bright star in the viewer); the neighboring exposures have nothing like tht.
https://data.desi.lbl.gov/desi/spectro/redux/daily/nightqa/20220512/sframesky-20220512.pdf

The sky monitor also has significantly brighter sky there (it's the outlier at 8:00):

Don't know if that s related, but the cryostat pressure for sm8 (=petal=7) has some fluctuations before that exposure:
https://telemetry.desi.lbl.gov/d/xcYVcw3Zk/spectrographs-cryostats?orgId=2&from=1652403600000&to=1652443200000

Original slack message: https://desisurvey.slack.com/archives/C01HNN87Y7J/p1652480247392209

z3 CTE issue tiles benefitting from future reprocessing

z3 had CTE issues from 20211126 - 20211202 (inclusive). These are minor and could be largely fixed with the per-sector fits.

Tile 23224 rejects fibers in patch of high EBV

Tile 23224 observed on 2022-02-12 rejects a patch of hundreds of fibers in a patch of high Galactic extinction for failing the effective time cut. We had planned to replace this EFFTIME cut to be corrected for the median EBV for the tile, rather than the per-object EBV. That would recover these fibers.

I had marked this tile as QA='good'. If one wanted to back this out, is it sufficient to edit the surveyops/ops/tiles-specstatus.ecsv file to have QA='unsure' before it gets archived?

Bright tile 23736 with suspicious low-z pileup

Bright tile 23736 observed on 20220318 has a low redshift z=0.25 pileup that might be real, but it looks suspicious because it's exclusively located on petal 7 and does not extend across neighboring petals 6 and 8.

Tile 25744 has petals 3,4,9 rejected for large STARRMS

Most of the signal for tile 25744 comes from the exposure 20220121/00119660. Petals 3,4,9 are rejected due to a large STARRMS > 0.20 (dispersion in the standard star model fits) from the file /global/cfs/cdirs/desi/spectro/redux/daily/exposures/20220121/00119660/exposure-qa-00119660.fits

PETAL_LOC STARRMS

---

    0 0.14335816
    1 0.14170471
    2 0.07650502
    3 0.20699991
    4 0.25244614
    5 0.13656339
    6 0.10696009
    7 0.16174984
    8 0.12091315
    9 0.20128961

Could the cause of this be the standard star fitting being pulled by the very poor data in the previous exposure 20220120/00119533 on the same tile?

Exposure 130095 on tile 22170 did not have fiber correction move

Exposure 130095 on BRIGHT tile 22211 22170 (night 20220413) did not have fiber correction move and should be rejected. I didn't catch this until seeing the QA plot, combining those data with exposure 130266 on night 20220414.

That first exposure should be marked bad, and the cumulative tile reductions removed. We'll need another exposure on this tile to complete it.

22170 is marked unsure

@djschlegel marked tile 22170 as unsure; recording this here. QA looks pretty horrid; both exposures probably need to be marked bad?
https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/22170/20220414/tile-qa-22170-thru20220414.png

Low-EFFTIME fibers near the edge of petal 1 from 20220326

4 tiles (TILEIDs 8808, 11015, 11046, 11542) from 20220326 have fibers near the edge of petal 1 with low EFFTIME:

https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/8808/20220326/tile-qa-8808-thru20220326.png
https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/11015/20220326/tile-qa-11015-thru20220326.png
https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/11046/20220326/tile-qa-11046-thru20220326.png
https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/11542/20220326/tile-qa-11542-thru20220326.png

These tiles are marked as unsure in the QA.

Discussions of this issue on slack:
https://desisurvey.slack.com/archives/C01HNN87Y7J/p1648414581954449

Turbulence corrections from 20220507 - 20220513 were problematic

The fixed, stationary locations of the stuck positioners were incorrect for the nights of 20220507-20220513. We could repeat the turbulence corrections for these nights using the now known positions. We would need to think about how we want to incorporate that information into the pipeline to propagate it into the final data. The current original source is the coordinates files in data/YEARMMDD/EXPID.

A similar mechanism could be applied to all pre-turbulence correction data.

Only a few fibers per exposure move by more than 30 microns, though there are a few tens with corrections more than 20 microns.

Bright tile 26226 observed with high read noise

Bright tile 26266 observed on 20220318 over three different exposures was reported by the observers to have high read noise on all of them.

The redshifts as a function of FIBERID look patchier than usual. They seem to have a pattern that changes petal by petal.

24638 has weird sky fit on r3; bad trace fit

No obvious impact on QA, but the sframes look pretty bad on r3.
https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/24638/20220329/tile-qa-24638-thru20220329.png

https://data.desi.lbl.gov/desi/users/raichoor/main-status/sframesky/sframesky-20220329-00127995.png

Acquisition failed on tile 2219

Field acquisition failed on tile 2219 exposure 128339 on April 1. This is a high-Galactic-latitude DARK tile. Conditions were good with 1.3 arcsec seeing in clear, dark skies: https://www.legacysurvey.org/viewer?ra=187.319&dec=36.81&layer=ls-dr9&zoom=8&desifoot=187.3190,36.5810

Steve Kent's response: Yes, low star density. Several other fields in the same region succeeded, so this one was just over the edge. There are several tuning parameters in the star finding and matching code, and I think allowing fainter stars is the easiest fix (the current limit is 18th mag). Note that the parameters were tuned in the days when the GFA CCDs were often in their noisy state and PM still succeeds when that happens. Given how rare the failures have been, I'd suggest waiting for a few more to occur before making any change (which is eay to do - Klaus needs to edit a configuration file but no reinstall of code is needed.)

We could change this configuration file (to a faint limit of 18.5 or 19.0) and try re-observing at this exact location again. I suggest we design BACKUP tile 42546 which has exactly the same coordinates and observe during engineering time. (The corresponding BRIGHT tile has slightly different coordinates due to our small offsets to some tile centers to avoid bright stars on disabled fibers, and we've already observed it anyway.)

Bright tile 26068 has an entire petal with low effective time, not obvious why

Bright tile 26068 observed on 20220317 shows petal 4 has having an unusual low effective time. It's not obvious to me why that's the case. Redshifts look OK. I marked that tile as unsure.

Tile 9497 half amp marked bad due to spurious jump

David Schlegel has documented the cause of the spurious jump here.
desihub/desispec#1619

This causes the tile to lose a half amp. I plan on marking the tile as good anyway, so we'll reobserve objects in that half amp. Tagging @djschlegel and @sbailey in case they want to overrule.

Weird behavior in r1 of TILE 5648 on 20220324; ion pump glow contamination

EXPID 127342-127345 are affected by this issue, although only 127342 (TILEID 5648) shows noticeable effects on the redshifts.

QA for tile 5648, showing many objects with z=~0.09 in petal 1:

Example of a z=~0.09 spectrum in tile 5648:

Expid 113678 not propagated into tsnr-exposures.fits / exposures-daily.fits

Expid 113678 is a bad exposure on tile 21281 for which the ETC estimates 0.085 s of effective exposure time. This exposure looks to have been rejected by the pipeline (EFFTIME_SPEC = 0), but it doesn't have a record in the tsnr-exposures file and so the NTS believes that this tile has 0.085 s of effective time, when it should have 0 seconds of effective exposure time. Can we get an entry in tsnr-exposures.fits recording the fact that the pipeline has rejected this tile? Thanks!

Bias variation in b8 causes jump in spectra

The bias variation in b8 causes jump in spectra, which results in some features in sky Z vs. FIBER plot and lower-than-normal success rate of LRGs, and it's particularly noticeable for the night of 2022-02-02:

Sky Z vs. FIBER plot: https://data.desi.lbl.gov/desi/spectro/redux/daily/nightqa/20220202/skyzfiber-20220202.png

Example of affected spectrum:

BRIGHT tiles that hang off footprint and have weird redshift distribution

There are a few otherwise fine tiles that hang off the edge of the footprint and have a weird redshift distribution. I don't understand what is going on with these. I don't know where we're getting off footprint redshifts? I don't know if these are failures of real targets or just weird targets?

There may be many more cases that never got marked unsure.

Affected tiles: 20682, 21347, 23401, 25506

https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/20682/20211211/tile-qa-20682-thru20211211.png
https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/21347/20220109/tile-qa-21347-thru20220109.png
https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/23401/20211218/tile-qa-23401-thru20211218.png
https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/25506/20211211/tile-qa-25506-thru20211211.png

I'm leaving these as unsure; at least they're all on the edge of the footprint so they aren't so problematic from a depth-first perspective.

z6 instability leads to redshift pileup

z6 CTE or amp offset that could likely be fixed with sector code.

https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/10941/20211106/tile-qa-10941-thru20211106.png
https://data.desi.lbl.gov/desi/users/raichoor/main-status/sframesky/sframesky-20211106-00107649.png

Tileid= 21984 marked as unsure

Tileid=21984 has been observed with expids=134791-92 on 20220514.
It has several r- and z-band sky alerts (petals 4-5-6-7-8-9); and a suspicious bgs_faint n(z), despite having nominally ~1.5 more effime than required (efftime=265s):
https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/21984/20220514/tile-qa-21984-thru20220514.png

If relevant, on the same night, all backup exposures from expid=134789 onwards were flagged as "unsure" by the observers in the nightlog.
Besides, these expids=134791-92 were preceded/succeded with backup observations, so conditions might have been variable.

Tile 23225 on 20220206 has petal 8 rejected for low throughput

Tile 23255 on 20220206 has petal 8 rejected for low throughput with no obvious reason why.

SP1 spectra from night of 2022-04-13 show different absorption dip due to new collimator

One of the new replacement collimators was installed for the night of 2022-04-13 only, on SP1 (SM10). It has a deep absorption at 420 nm, which is bluer than the dip in the original collimators. Only BRIGHT tiles were observed on that one night. The reduced data do show uncorrected flux-calibration residuals around this new dip, although the BGS redshifts appear robust to this. A couple of representative calibrated spectra attached.

Bright tile 26056 redshifts wrong due to overscan cosmic on b1

Bright tile 26056 observed in one exposure (expid 123247) on 2022-02-21 shows many spurious redshifts at z=0.14-0.15. This is due to a cosmic landing next to (or on?) the overscan of B1 amplifier D. Cosmic is at (2045,3707) in the raw image. The row-by-row bias subtraction results in large negative (and significant) values at ~5600 Ang, which is eventually fit as H-beta in absorption for most fibers in the range 500-749.

This will have to be solved by using both the bias region and the overscan region to filter out the effects of cosmics independent of bias level jumps.

Exposure ID 129062 on tile 24668 was out-of-focus

Another exposure at the very beginning of the night that looks like it was out-of-focus causing petal 7 to mostly be dropped:

https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/24668/20220407/tile-qa-24668-thru20220407.png

Tileid 9934 marked as unsure

Tileid 9934 was observed with exposure 136310, 136311 and 136312 on 20220522.
It apperead as a clear outlier on the Lya plot. The redshifts as a function of the fiberid looks clumpy in some locations.

In the nightlog we can read: "SPLIT on DARK tile 9934, exposure 136311. Survey speed dropped from 85% to 40%, presumed to be seeing related."

Tile 22230 on 20220211 rejects half of R4 due to CTE in overscan

The single exposure on tile 22230 on 2022-02-11 (expid 122156) rejects half of P4 (fibers 2250-2499) due to a small CTE from a cosmic effecting the overscan region, resulting in a too-large overscan jump (OSTEPB=6.07).

This has previously been seen in r8-D, z8-D, z3-D, z5-A, as described in the desispec issue #1619 (desihub/desispec#1619). However, it had not been identified previously on r4-B.

Below is the raw image centered on the bright cosmic at location (2219,1036).

sky issue on petal 2 tile 26071

alerts on the tile qa page and doesn't look good on the sframe; no obvious impact on redshifts

Exposure 137368 should be marked as bad due to missing guide info

The guider crashed on exposure 137368 on tile 10589 on 2022-05-30. Night log states:
GUIDE5 GFA image size error followed by second failed automatic revere attempt. Loss of guiding mid-exposure.; Action: Stopped exposure and read out data for DARK tile 10589 exposure [137368] once guiding was confirmed as lost.

This results in no guider cube being written for the exposure. We've been marking such exposures as bad, since we lose the metadata such as the guider-computed effective times. The last such instance was exposure 133140 on 20220503, triggered from a failure of GUIDE7.

Tileid 25103 marked as unsure

Tileid 25103 was observed with exposure 136464 on 20220523. It has multiple sky substraction warnings and a strong redshift clumping around z=0.75 across five petals (although the sky substraction warnings and the clumping might be unrelated).

The sky substraction on petal 7 is definitely problematic

EXPID 109653 is bad

We need to mark EXPID 109653 on TILEID 22381 as bad, and then reprocess TILEID 22381.

This is a tile that we marked "bad" months ago but then forgot about. @akremin, could you get this one, too? Thanks!

Tile 11103 marked unsure -- Z7 sky chi2

Expnum 137248 on 2022-05-29. The tile qa plot looked fine to me, but the sframesky QA plot shows a broad glowing back in R7 and Z7, and the Z7 one triggered a warning on the QA plot.

Tile 4215 marked as unsure: lots of flagged fibers on P4. Poor positioning due to fast sequencer & long slew before fix.

Bright tile 22889 with clustered sources around z=3.0-3.5

Bright tile 22889 observed on 20220314 shows a bunch of clustered sources around z=3.0-3.5 on petal 4.

z=1.8 and 5.5 redshift pileups on tile 7009; likely bad sky residuals

This tile got a ton of EFFTIME in pretty poor conditions. Redshift distribution shows clear pileups at z=1.8 and 5.5, as well as SKYCHI2PDF warnings.
https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/7009/20211215/tile-qa-7009-thru20211215.png

I suspect that the sky variation is just too large and things are being misfit as high redshifts. We either need to improve the sky modeling or mark the component exposures as bad so that the tile may be reobserved.

Possibly the dark time equivalent of #4 .

Exposure ID 128784 on tile 23108 was out-of-focus

Exposure ID 128784 on tile 23108 was out-of-focus, leading to a high RMS across petals on that exposure. Here is the tile:

https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/23108/20220405/tile-qa-23108-thru20220405.png (this is the bad exposure)
https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/23108/20220406/tile-qa-23108-thru20220406.png

Here's a plot of the relative focus from Eddie:

See also desihub/desispec#1738.

When >90% of a petal's fibers are masked, we should mask the whole petal

Tile 3533 will soon be marked "good", but we want to handle this case better.

It looks like >90% of the fibers on this petal are masked due to low EFFTIME. In this case, we want to throw the petal out rather than keeping the few fibers that survive the futs.

https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/3533/20211204/tile-qa-3533-thru20211204.png

Tile 24211 has pile-up at z=1.8 on P7

Exposure 120202 on 2022-01-25 (tile 24211) has a pile-up of spurious z=1.8 redshifts on petal 7.

Tiles with ~large sky residuals that need to be marked bad or better processed

These are currently "unsure" and typically have lots of significant redshifts at z=1.8. Often have SKYCHI2PDF warning. May just need to have their component exposures marked bad so that NTS will reobserve them.

Affected: 20140, 21321, 23276, 24633, 24634, 26147

e.g.
https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/20140/20211117/tile-qa-20140-thru20211117.png

sframe files show lots of sky amplitude variation from fiber to fiber that is either a gradient or cloudy?
https://data.desi.lbl.gov/desi/users/raichoor/main-status/sframesky/sframesky-20211117-00109233.png

https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/20140/20211117/tile-qa-20140-thru20211117.png
https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/21321/20220114/tile-qa-21321-thru20220114.png
https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/23276/20211117/tile-qa-23276-thru20211117.png
https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/24633/20211219/tile-qa-24633-thru20211219.png
https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/24634/20211219/tile-qa-24634-thru20211219.png
https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/26147/20211219/tile-qa-26147-thru20211219.png

Tile 2295 on 20220211 has low throughput petals 2,4,7

Tile 2295 on 2022-02-11 has low throughput on petals 2,4,7 causing a smattering of fibers to be rejected on those petals. Note this was observed with a very bright sky (sky=7!), although otherwise very good conditions (photometric, 0.9 arcsec seeing).

Bright tile 21348 with strong z=0.6 pileup for sky fibers

Bright 21348 observed on 20220314 shows a strong z=0.6 pileup for sky fibers on petal 7

Tile 4402 rejects half fibers on r8 due to CTE in amp region

Tile 4402 on 2022-02-05 expid 121487 which rejects fibers 4250-4499 due to a cosmic appearing at location (2197,3718) of the r8 CCD. This has OSTEPD=5.7 in the frame file. Corresponding spectro issue is desihub/desispec#1619 .

Below is a cutout of the raw image centered on (2197,3718).

High read noise reported on R3B for expid 130556 due to CTE

Exposure 130556 on BACKUP tile 42424 observed 2022-04-16 reports a high read noise on amplifier R3B with OBSRDNB = 10.58 e- (or 6.38 ADU). This is apparently due to a bright (but not saturating) star next to the overscan, and hysteresis/CTE pushing down the counts by approx 30 ADU immediately after that star.

Attached is a cut throw a portion of the raw image showing the effect where the star is at column 2197.

Weird bright feature leads to spurious redshifts on 10184/116955

Weird bright feature on r8; presumably a hot row or something?
https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/10184/20220106/tile-qa-10184-thru20220106.png
Pretty clear in sframe:
https://data.desi.lbl.gov/desi/spectro/redux/daily/nightqa/20220106/sframesky-20220106.pdf

Leaving this as unsure for now.

Tile 25270 on 20220206 has petal 7 rejected for low throughput

Tile 25270 on 20220206 has petal 7 rejected for low throughput with no obvious reason why.

Dubious z=2.5 objects on all petals on tile 4403

There's a dubious set of z=2.5 objects on all petals on TILEID 4403.
https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/4403/20220113/tile-qa-4403-thru20220113.png

My suspicion is that this is the fiberflat / humidity code, but I am not sure.
https://data.desi.lbl.gov/desi/spectro/redux/daily/nightqa/20220113/sframesky-20220113.pdf
It is remarkable how perfect the humidity correction is at the beginning of the night, and then how it suddenly gets worse for the rest of the night. If this is actually the cause, then most of the night would benefit from reprocessing.

I can't mark this tile as good since this presumably affects LyA decisions, so this one is relatively high priority.

Cals from 20211226 are problematic

We had problematic calibration images on 20211226 and have marked all tiles from that night as unsure until we are able to decide if this night is salvageable. Tiles affected:
2576 3238 3270 3607 6028 6376 6744 8764 11778 22955 23474 24146 25715 41253 41260 41484 41757 41758 41773
(note that some of these are backup 4xxxx series and are less important, but there are a number of dark and bright tiles as well)

Update: Title is incorrect, 20211226, I have decided that we should have included the dashes in the night strings as a matter of policy. Darn.

Garbage redshifts on P4 on tile 10357

P4 shows clear pileups of garbage redshifts around z=0.25 and z=0.5.
https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/10357/20211210/tile-qa-10357-thru20211210.png

This doesn't look like it affects LyA and I will mark it good.

I am having trouble seeing what is causing them; the sframe files look pretty good.
https://data.desi.lbl.gov/desi/spectro/redux/daily/nightqa/20211210/sframesky-20211210.pdf