Create hooks for automatically reading dark images about csxtools

[mrakitin]

https://github.com/bluesky/bluesky-darkframes may be a solution to the problem.

[leofang]

I had a discussion with @wen-hu recently on this, and I don't think this is possible unless the scan IDs of the dark images are recorded somewhere, say in the RE plan. In particular, there could be several scans that share the same set of dark images, and unfortunately the (hypothetical) hook would have no way to infer this without it being stored in Databroker.

I took it upon myself to close this issue. Please feel free to reopen if anyone thinks I am wrong above. Thanks.

[mpmdean]

I don't really see why this is said to be impossible. There are many options, but my first reaction would be that when collecting a dark via:
ct_dark()
and ct_dark_all()
the macros automatically pass md=dict(dark=True) or similar to label the scans as dark in the databroker.

We could then search backwards in time from the header describing the data and find use the first dark scan that it finds.

[leofang]

https://github.com/bluesky/bluesky-darkframes may be a solution to the problem.

Thanks, @mrakitin. I wasn't aware of this. This info is enough for reopening this issue.

I don't really see why this is said to be impossible. There are many options, but my first reaction would be that when collecting a dark via:
ct_dark()
and ct_dark_all()
the macros automatically pass md=dict(dark=True) or similar to label the scans as dark in the databroker.

We could then search backwards in time from the header describing the data and find use the first dark scan that it finds.

I am not sure I understand your reply correctly, but there are two potential issues based on my (very limited) understanding -- correct me if I'm wrong:

1. Any hook created from this point is incompatible with existing scans, unless one goes through the entire CSX db and places the labels
2. Search backward in time is not always valid. Sometimes dark images are taken after the regular scan is finished, so in this case you need to search forward.

[mpmdean]

For 1.

I think the function should provide options

1. Specify the dark header
2. Search for the header
3. Don't to a dark correction
4. Specify a light field correction
5. Search for a light field correction
6. Don't do a light field correction

Since the option to specify the dark and light corrections would remain, even if the search functions were not backward compatible, they would still be very useful for future data.

2. One can equally well write a search for the more recent dark forwards or backwards.

[ambarb]

db[122918].start['fccd']

returns all the information for a dark scan. This work was started when I left to SIX and not completed.

[ambarb]

additionally, I added inout to the baseline so someone can troubleshoot or confirm that the beam is being block.

I am happy to discus what the original intention of the issue report was and why it is needed. we need to be careful in how itis implemented because it relates to the 5-way light source project and the automatic digestion images for this an other projects that rely heavily on automation.

[leofang]

@mpmdean The "function" you suggested definitely works. But it'd no longer be a "hook" that is fully automated; substantial human intervention would be needed, and in the end one might as well just type in the dark IDs directly (as we do now). I am as skeptical as @ambarb in the automation regard.

[mpmdean]

If the function was created any user who takes care to collect the dark images with labels will have an automated experience for all their future experiments.

I'd emphasize that users currently have to keep track of four other scans to provide a full correction (darks field correction 3 gains plus a light field correction).

I'd argue that there's not much point in having the databroker infrastructure if we don't leverage it for this type of task.

[leofang]

Yes, I agree 100% that we need to write some info related to dark + light into Databroker as the first step.

[ambarb]

We do this already. Things are there to confirm what is a dark image (double redancy available for different devices) and what the gain setting is.

One way to limit the database searching is to use the start document key associated with the user group +/- 2 days or maybe the sample description. I tried this search already for data 6 months ago and it was as if the data was collected that day. I was looking for the energy calibration reference to compare that day to 6 months ago....so different use case, but same idea.

Another consideration, that is a 2nd order, but much harder problem is the flat field correction. It would be better if we could store the result in the database instead of calculating each time we process light images.

What do you guys thingk so far? Want to start with the first problem?

[ambarb]

looks like there is something better already used at one beamline and tested at another. @jklynch @mrakitin are working on the details. Dark images will be done first. Discussed with @danielballan to have flat field result already processed and added to table data for each scan.

[ambarb]

some recent progress with @danielballan @jklynch and @mrakitin . Preliminary data with just one dark stream (autogain) are available for testing. WE would also like to retain current functionality to point to either primary stream data or dark stream data in a user defined scan number.

branch is in @danielballan fork

tested multiple darks without setting gain, and nothing alarming happened. work can proceed with incorporating this.

then on to flat field.

[mpmdean]

I'm trying to find the branch in question and https://github.com/danielballan/csxtools looks like it hasn't been updated in a rather long time.

[ambarb]

The hooks are created in the profile so the database is marked correctly. There is no code yet in any fork of csxtools that I know about to deal with these new data streams. As far as updating the beam line's python profile to create new streams, 1 of 4 streams are complete in dan's repo of the profile. It is still a "prototype"

[mrakitin]

See NSLS-II-CSX/profile_collection#32 for details.

[ambarb]

#89 addresses this