Weird bright feature leads to spurious redshifts on 10184/116955 about desisurveyops

[Waelthus]

Looks like r8-D had a hot row indeed

[schlafly]

@julienguy , @sbailey , do we have a policy for what we want to do with these? Options:

1. Mark the row as bad and reprocess
2. Mark the amp as bad and reprocess
3. Mark the exposure as bad and reobserve

This is rare enough that (3), while wasteful, isn't completely crazy. But we should make a policy.

[julienguy]

2. is what the current pipeline is set up to do: either we mark 'manually' this amplifier as bad in the exposures table, or the code auto-detects a bad amplifier with an overscan step measurement greater than 5 electron (this apparently did not work well for that exposure).

Looking at this exposure, the header keyword OSTEPD (overscan step for amplifier D) has a value of 4.67 electrons, and the threshold to declare the amplifier bad is set at 5 electrons. However, because OSTEPD>2, overscan subtraction per row was triggered, see in header:
OMETHD = 'PER_ROW ' / use average overscan per row

Quite unexpectedly, it is the overscan subtraction per row that generates this horizontal band (!) ...

• this is caused by a negative trail from a large charge deposit from a cosmic ray or radioactive decay bleeding in the overscan region (see image below)
• we have in place this 'dark trail correction' (see PR desihub/desispec#798 and issue desihub/desispec#796), but it comes with calibration parameters that have to be tuned for each CCD.
• I am opening a ticket on this in desispec.

[djschlegel]

The underlying issue is CTE from cosmics (desihub/desispec#1619). I've added this to that ticket as another example. I think it's common enough on Z8-D, Z3-D, Z5-8 that we don't want to throw out these data, but rather figure out how to mitigate in the spectro pipeline.

[sbailey]

@schlafly it appears that tile 10184 was flagged as GOOD and thus got slurped up in the archiving on January 25 (last Tuesday) before the fix in PR desihub/desispec#1636 and before reprocessing could have fixed this.

Please confirm whether that is ok (i.e. "good enough"), or whether we need to reprocesses this petal and re-MTL update this tile (to be interleaved with Fuji). I think the danger is the possibility of LyA QSOs getting overridden by bogus low-z galaxy fits due to this overscan dip feature.

[djschlegel]

Note there have been no overlapping DARK tile observations of this tile between when it was observed (20220106) through last night (20220130), so it would even be possible to just back out these MTL updates and load new ones.

[araichoor]

For what is worth, the newlya identified Ly-a diagnosis for this tile looks ok:

It is the point at (x,y) ~ (2.6, 80) in this plot: the y-value is not significantly below the expectation.
Though we are talking here about one petal only, i.e. one tenth of stats, so not sure how much the effect would be visible.

[ashleyjross]

I certainly meant to mark it "unsure" at the time (I remember discussing with Eddie). Sorry if somehow I marked it good! Given there hasn't been any overlapping tiles, I'm in favor of backing out the updates.

[schlafly]

No, it looks like I marked it as good with the commit "Moving more tiles with marginal problems documented at desisurveyops from unsure to good." I don't think it was my intent to get this one---I didn't mark this problem as "doesnotblockmtl"---sorry. For my two cents it doesn't mess with LyA decisions and we can safely ignore. If we want to reincorporate, I'd rather do that through the "normal" MTL reincorporation mechanism than backing out---it just seems safer and more like standard procedure.

[geordie666]

My two cents: I agree with Eddie that in general, now we have the reprocessing code for MTL, we should re-archive these sorts of tiles and then apply the MTL reprocessing code, instead of backing tiles out.

Backing out is tempting, in this case, as it's so simple. But, I'd rather have a standard procedure where once a month (say) we re-archive and re-process any problematic tiles. Otherwise we'll be tempted to end up back in a situation where we're doing something "special" on a semi-regular basis.

[ashleyjross]

I agree that using the standard procedure is much better in general. On this tile, I think we would ultimately wind up losing about half of petal 8, but they are almost all marked as good observations in the ledger? This seems on the upper end of issues we should be ok with, but I'm ok with whatever the group decision for this case. @github.com>

[schlafly]

No, the software fix just fixes this and the data is fine, so nothing is lost if the LyA decisions are okay.

And we can discuss again at the survey-ops telecon, but I understand the new standard to be that if we're not messing up LyA decisions, we can proceed with MTL updates. So marking half a petal later as bad is something that I would say need not hold up MTL updates.

[ashleyjross]

Ok, right, it should eventually almost all be ok, so yeah, this should clearly be fine for doing nothing right now (i.e., keeping it "good"). On the second point, I guess we do need some threshold for when it is a likely unflagged hardware issue. The most MTL-update friendly threshold would be >70% of the focal plane being ok (matching the threshold to pass a tile for flagged issues)?

[schlafly]

Fixed in software, added to closed issues, and closing.