Comments (17)
https://github.com/bluesky/bluesky-darkframes may be a solution to the problem.
from csxtools.
I had a discussion with @wen-hu recently on this, and I don't think this is possible unless the scan IDs of the dark images are recorded somewhere, say in the RE plan. In particular, there could be several scans that share the same set of dark images, and unfortunately the (hypothetical) hook would have no way to infer this without it being stored in Databroker.
I took it upon myself to close this issue. Please feel free to reopen if anyone thinks I am wrong above. Thanks.
from csxtools.
I don't really see why this is said to be impossible. There are many options, but my first reaction would be that when collecting a dark via:
ct_dark()
and ct_dark_all()
the macros automatically pass md=dict(dark=True) or similar to label the scans as dark in the databroker.
We could then search backwards in time from the header describing the data and find use the first dark scan that it finds.
from csxtools.
https://github.com/bluesky/bluesky-darkframes may be a solution to the problem.
Thanks, @mrakitin. I wasn't aware of this. This info is enough for reopening this issue.
I don't really see why this is said to be impossible. There are many options, but my first reaction would be that when collecting a dark via:
ct_dark()
and ct_dark_all()
the macros automatically pass md=dict(dark=True) or similar to label the scans as dark in the databroker.We could then search backwards in time from the header describing the data and find use the first dark scan that it finds.
I am not sure I understand your reply correctly, but there are two potential issues based on my (very limited) understanding -- correct me if I'm wrong:
- Any hook created from this point is incompatible with existing scans, unless one goes through the entire CSX db and places the labels
- Search backward in time is not always valid. Sometimes dark images are taken after the regular scan is finished, so in this case you need to search forward.
from csxtools.
For 1.
I think the function should provide options
- Specify the dark header
- Search for the header
- Don't to a dark correction
- Specify a light field correction
- Search for a light field correction
- Don't do a light field correction
Since the option to specify the dark and light corrections would remain, even if the search functions were not backward compatible, they would still be very useful for future data.
- One can equally well write a search for the more recent dark forwards or backwards.
from csxtools.
db[122918].start['fccd']
returns all the information for a dark scan. This work was started when I left to SIX and not completed.
from csxtools.
additionally, I added inout
to the baseline so someone can troubleshoot or confirm that the beam is being block.
I am happy to discus what the original intention of the issue report was and why it is needed. we need to be careful in how itis implemented because it relates to the 5-way light source project and the automatic digestion images for this an other projects that rely heavily on automation.
from csxtools.
@mpmdean The "function" you suggested definitely works. But it'd no longer be a "hook" that is fully automated; substantial human intervention would be needed, and in the end one might as well just type in the dark IDs directly (as we do now). I am as skeptical as @ambarb in the automation regard.
from csxtools.
If the function was created any user who takes care to collect the dark images with labels will have an automated experience for all their future experiments.
I'd emphasize that users currently have to keep track of four other scans to provide a full correction (darks field correction 3 gains plus a light field correction).
I'd argue that there's not much point in having the databroker infrastructure if we don't leverage it for this type of task.
from csxtools.
Yes, I agree 100% that we need to write some info related to dark + light into Databroker as the first step.
from csxtools.
We do this already. Things are there to confirm what is a dark image (double redancy available for different devices) and what the gain setting is.
One way to limit the database searching is to use the start document key associated with the user group +/- 2 days or maybe the sample description. I tried this search already for data 6 months ago and it was as if the data was collected that day. I was looking for the energy calibration reference to compare that day to 6 months ago....so different use case, but same idea.
Another consideration, that is a 2nd order, but much harder problem is the flat field correction. It would be better if we could store the result in the database instead of calculating each time we process light images.
What do you guys thingk so far? Want to start with the first problem?
from csxtools.
looks like there is something better already used at one beamline and tested at another. @jklynch @mrakitin are working on the details. Dark images will be done first. Discussed with @danielballan to have flat field result already processed and added to table data for each scan.
from csxtools.
some recent progress with @danielballan @jklynch and @mrakitin . Preliminary data with just one dark stream (autogain) are available for testing. WE would also like to retain current functionality to point to either primary stream data or dark stream data in a user defined scan number.
branch is in @danielballan fork
tested multiple darks without setting gain, and nothing alarming happened. work can proceed with incorporating this.
then on to flat field.
from csxtools.
I'm trying to find the branch in question and https://github.com/danielballan/csxtools looks like it hasn't been updated in a rather long time.
from csxtools.
The hooks are created in the profile so the database is marked correctly. There is no code yet in any fork of csxtools that I know about to deal with these new data streams. As far as updating the beam line's python profile to create new streams, 1 of 4 streams are complete in dan's repo of the profile. It is still a "prototype"
from csxtools.
See NSLS-II-CSX/profile_collection#32 for details.
from csxtools.
#89 addresses this
from csxtools.
Related Issues (20)
- make csx tools load multiple scan_ids HOT 6
- image_stack_to_movie is broken HOT 1
- get_fastccd_images cannot process all instances of recorded data HOT 2
- roi argument for get_fastccd_images() is broken ... plus some flatfield dependency discussion HOT 23
- KeyError: 'AD_HDF5' for 1.14 for get_fastccd_flatfield() and get_fastccd_images() HOT 7
- Should we change default behavior get_fastccd_images to use a flatfield of (almost) all ones? HOT 3
- get_fastccd_images() does not work with roi given as input HOT 3
- swmr says it is enabled but it doesn't seem to work HOT 1
- Strange error related to dark image scan with x2 gain HOT 3
- Logging module no longer works
- fccd flatfield function doesn't handle the broken part of the camera very well HOT 5
- Remove hard-coded rows and columns because this changes HOT 1
- Potential place for controlling dark image searching HOT 1
- flat field issue HOT 3
- conda packaging for Python 3.10 HOT 2
- Fix deprecation warnings on tests. HOT 6
- ENH more automation of flatfield correction HOT 1
- homework for new helpers modules HOT 1
- helpers.overscan copy-paste error for right-side of detector
- flake8 style needed for new modules
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from csxtools.