Useful python tools for CSX (23-ID)
Home Page: http://nsls-ii-csx.github.io/csxtools
License: Other
get_fastccd_images calls the decepricated get_images function rather than accessing via the databroker
using CSX on srv2 (current) kernal:
from databroker import DataBroker as db
from csxtools.utils import get_fastccd_images
slicerator = get_fastccd_images(db[108225])
Processing without dark images
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/csxtools/utils.py:181: UserWarning: Images and get_images are deprecated. Use Header.data('fccd_image') instead.
images = header.db.get_images(header, tag)
image_stack_to_movie() suggestions
tests/test_image.py::test_images_mean
/nsls2/users/abarbour/Repos/csxtools/tests/test_image.py:85: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
x = np.array([np.repeat(ii*np.ones(ii*100, dtype=np.float32), 400).reshape(
tests/test_image.py::test_images_sum
/nsls2/users/abarbour/Repos/csxtools/tests/test_image.py:92: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
x = np.array([np.repeat(ii*np.ones(ii*100, dtype=np.float32), 400).reshape(
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.htmlWhen roi coordinates are passed to
get_fastccd_images(h, (bgnd8, bgnd2, bgnd1), flat=ff, tag='fccd_image', roi = (0, 0, 50, 50))
the function returns an AttributeError.
AttributeError Traceback (most recent call last)
in
/opt/conda_envs/analysis-2019-3.0-ml/lib/python3.7/site-packages/csxtools/utils.py in get_fastccd_images(light_header, dark_headers, flat, gain, tag, roi)
92
93 tt = ttime.time()
---> 94 b = get_images_to_3D(bgnd_events, dtype=np.uint16)
95 logger.info("Image conversion took %.3f seconds",
96 ttime.time() - tt)
/opt/conda_envs/analysis-2019-3.0-ml/lib/python3.7/site-packages/csxtools/utils.py in get_images_to_3D(images, dtype)
173
174 """
--> 175 im = np.vstack([np.asarray(im, dtype=dtype) for im in images])
176 return im
177
/opt/conda_envs/analysis-2019-3.0-ml/lib/python3.7/site-packages/csxtools/utils.py in (.0)
173
174 """
--> 175 im = np.vstack([np.asarray(im, dtype=dtype) for im in images])
176 return im
177
/opt/conda_envs/analysis-2019-3.0-ml/lib/python3.7/site-packages/slicerator/init.py in (.0)
472
473 def iter(self):
--> 474 return (self._get(i) for i in range(len(self)))
475
476 def getitem(self, i):
/opt/conda_envs/analysis-2019-3.0-ml/lib/python3.7/site-packages/slicerator/init.py in _get(self, key)
459 # We need to copy here: else any _proc_func that acts inplace would
460 # change the ancestor value.
--> 461 return self._proc_func(*(copy(a[key]) for a in self._ancestors))
462
463 def repr(self):
/opt/conda_envs/analysis-2019-3.0-ml/lib/python3.7/site-packages/slicerator/init.py in proc_func(*x)
683 if all_pipe:
684 def proc_func(x):
--> 685 return func((x + args), **kwargs)
686
687 return Pipeline(proc_func, *ancestors)
/opt/conda_envs/analysis-2019-3.0-ml/lib/python3.7/site-packages/csxtools/utils.py in _crop_images(image, roi)
207 @pipeline
208 def _crop_images(image, roi):
--> 209 return _crop(image, roi)
210
211
/opt/conda_envs/analysis-2019-3.0-ml/lib/python3.7/site-packages/csxtools/utils.py in _crop(image, roi)
214 # Assuming ROI is specified in the "rotated" (correct) orientation
215 roi = [image_shape[-2]-roi[3], roi[0], image_shape[-1]-roi[1], roi[2]]
--> 216 return image.T[roi[1]:roi[3], roi[0]:roi[2]].T
217
218
AttributeError: 'ImageStack' object has no attribute 'T'
Lines 74 to 77 in 52ff964
get_fastccd_images() is provisioned to search database. I've developed some code that does this, but I envisioned using it this new function to find the arguments as input to get_fastccd_images(). This way, it is easier to track provenance of corrected images.
Open for thoughts etc.
The search I've developed is quite fast (uses databroker to do the search) and it is controlled by a time window (default is 30 minutes before or 30 minutes after the light scan in question).
Note we may use the exception handling here instead to deal with (None, None, None) as this is not currently allowed. Currently, if you do not want to subtract ANY darks (and just correct the gain settings), then None not (None, None, None) is the value required for the dark scans argument.
@patryk-w figured this out and has some temporary code he is using.
to move forward, do we want a bandaid function that we deprecate later?
Or do we want to alter the existing flatfield function to be configurable for bad areas of the ccd?
options in clude using argments to exclude areas or referencing a bad pixel mask.
@wen-hu needs the flatfield support to process some ptychography datasets, but I found an error with the latest csxtools (0.1.15):
>>> from csxtools.utils import get_fastccd_flatfield, get_fastccd_images, get_images_to_4D
>>> import numpy as np
>>> from databroker import Broker
>>>
>>> db = Broker.named('csx')
>>> dark0_f = 127832; dark1_f = dark0_f + 1; dark2_f = dark0_f +2
>>> flat_im = get_fastccd_flatfield(db[127831], dark=(db[dark0_f], db[dark1_f], db[dark2_f]))
/home/leofang/conda_envs/ptycho_exp_Jan2020/lib/python3.6/site-packages/csxtools/utils.py:190: UserWarning: Images and get_images are deprecated. Use Header.data('fccd_image') instead.
images = header.db.get_images(header, tag)
/home/leofang/conda_envs/ptycho_exp_Jan2020/lib/python3.6/site-packages/csxtools/utils.py:271: RuntimeWarning: invalid value encountered in greater
flat[flat > limits[1]] = np.nan
Flatfield correction removed 79554 pixels (8.29 %)
>>>
>>> flat_im.shape
(1000, 960)
>>> silcerator = get_fastccd_images(db[127849], dark_headers=(db[127846], db[127847], db[127848]), flat=flat_im, roi=[0,0,1000,960])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/leofang/conda_envs/ptycho_exp_Jan2020/lib/python3.6/site-packages/csxtools/utils.py", line 94, in get_fastccd_images
b = get_images_to_3D(bgnd_events, dtype=np.uint16)
File "/home/leofang/conda_envs/ptycho_exp_Jan2020/lib/python3.6/site-packages/csxtools/utils.py", line 175, in get_images_to_3D
im = np.vstack([np.asarray(im, dtype=dtype) for im in images])
File "/home/leofang/conda_envs/ptycho_exp_Jan2020/lib/python3.6/site-packages/csxtools/utils.py", line 175, in <listcomp>
im = np.vstack([np.asarray(im, dtype=dtype) for im in images])
File "/home/leofang/conda_envs/ptycho_exp_Jan2020/lib/python3.6/site-packages/slicerator/__init__.py", line 474, in <genexpr>
return (self._get(i) for i in range(len(self)))
File "/home/leofang/conda_envs/ptycho_exp_Jan2020/lib/python3.6/site-packages/slicerator/__init__.py", line 461, in _get
return self._proc_func(*(copy(a[key]) for a in self._ancestors))
File "/home/leofang/conda_envs/ptycho_exp_Jan2020/lib/python3.6/site-packages/slicerator/__init__.py", line 685, in proc_func
return func(*(x + args), **kwargs)
File "/home/leofang/conda_envs/ptycho_exp_Jan2020/lib/python3.6/site-packages/csxtools/utils.py", line 209, in _crop_images
return _crop(image, roi)
File "/home/leofang/conda_envs/ptycho_exp_Jan2020/lib/python3.6/site-packages/csxtools/utils.py", line 216, in _crop
return image.T[roi[1]:roi[3], roi[0]:roi[2]].T
AttributeError: 'ImageStack' object has no attribute 'T'Seems like image in _crop is assumed to be a numpy array but it actually isn't (or is not handled correctly by @pipeline). Perhaps we can hot-fix it by not doing a rotate90 or np.rot90 to avoid transpose (T) operation to the flatfield? But not sure if this would break existing code, or if existing code just never worked 🤣
Any thoughts? @danielballan @mrakitin
The current srv1 kernal on CSX cannot specify more than one dark image in get_fastccd_images
`
from databroker import DataBroker as db
from csxtools.utils import get_fastccd_images
old_scan_no = 85528
slicerator = get_fastccd_images(db[old_scan_no], (db[old_scan_no-1, old_scan_no+1], None, None))
`
Throws an error
AttributeError Traceback (most recent call last)
in ()
3
4 old_scan_no = 85528
----> 5 slicerator = get_fastccd_images(db[old_scan_no], (db[old_scan_no-1, old_scan_no+1], None, None))
/opt/conda_envs/analysis-2018-1.0/lib/python3.6/site-packages/csxtools/utils.py in get_fastccd_images(light_header, dark_headers, flat, gain, tag, roi)
85 # Get the images
86
---> 87 bgnd_events = _get_images(d, tag, roi)
88
89 # We assume that all images are for the background
/opt/conda_envs/analysis-2018-1.0/lib/python3.6/site-packages/csxtools/utils.py in _get_images(header, tag, roi)
179 def _get_images(header, tag, roi=None):
180 t = ttime.time()
--> 181 images = header.db.get_images(header, tag)
182 t = ttime.time() - t
183 logger.info("Took %.3f seconds to read data using get_images", t)
AttributeError: 'list' object has no attribute 'db'
I believe this comes from this change:
-def _get_images(header, tag, roi=None, handler_override=None):
+def _get_images(header, tag, roi=None):
t = ttime.time()
This scan is a step scan, mid-way or more through. Using get_fastccd_images()
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/csxtools/utils.py:181: UserWarning: Images and get_images are deprecated. Use Header.data('fccd_image') instead.
images = header.db.get_images(header, tag)
---------------------------------------------------------------------------
OSError Traceback (most recent call last)
<ipython-input-48-9e807f3dbb64> in <module>()
1 #%%capture
----> 2 images = get_fastccd_images(h, (bgnd8[0], bgnd2, bgnd1), flat=None)
3
4 stack = get_images_to_4D(images)
5
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/csxtools/utils.py in get_fastccd_images(light_header, dark_headers, flat, gain, tag, roi)
119 logger.info("Computed dark images in %.3f seconds", ttime.time() - t)
120
--> 121 events = _get_images(light_header, tag, roi)
122
123 # Ok, so lets return a pims pipeline which does the image conversion
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/csxtools/utils.py in _get_images(header, tag, roi)
179 def _get_images(header, tag, roi=None):
180 t = ttime.time()
--> 181 images = header.db.get_images(header, tag)
182 t = ttime.time() - t
183 logger.info("Took %.3f seconds to read data using get_images", t)
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/databroker/_core.py in get_images(self, headers, name, stream_name, handler_registry)
1712 headers=headers,
1713 name=name, stream_name=stream_name,
-> 1714 handler_registry=handler_registry)
1715
1716 def get_resource_uids(self, header):
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/databroker/_core.py in __init__(self, mds, reg, es, headers, name, handler_registry, handler_override, stream_name)
606 self.handler_registry = defaultdict(lambda: handler_override)
607 with self.reg.handler_context(self.handler_registry) as reg:
--> 608 example_frame = reg.retrieve(first_uid)
609 # Try to duck-type as a numpy array, but fall back as a general
610 # Python object.
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/databroker/assets/base_registry.py in retrieve(self, datum_id)
176 return self._api.retrieve(self._datum_col, datum_id,
177 self._datum_cache, self.get_spec_handler,
--> 178 logger)
179
180 def get_datum(self, datum_id):
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/databroker/assets/core.py in retrieve(col, datum_id, datum_cache, get_spec_handler, logger)
63 def retrieve(col, datum_id, datum_cache, get_spec_handler, logger):
64 datum = _get_datum_from_datum_id(col, datum_id, datum_cache, logger)
---> 65 handler = get_spec_handler(datum['resource'])
66 return handler(**datum['datum_kwargs'])
67
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/databroker/assets/base_registry.py in get_spec_handler(self, resource)
260 if root:
261 rpath = os.path.join(root, rpath)
--> 262 ret = handler(rpath, **kwargs)
263 h_cache[key] = ret
264 return ret
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/databroker/assets/handlers.py in __init__(self, filename, frame_per_point)
201 super(AreaDetectorHDF5Handler, self).__init__(
202 filename=filename, key=hardcoded_key,
--> 203 frame_per_point=frame_per_point)
204
205
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/databroker/assets/handlers.py in __init__(self, filename, key, frame_per_point)
151 self._dataset = None
152 self._data_objects = {}
--> 153 self.open()
154
155 def get_file_list(self, datum_kwarg_gen):
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/databroker/assets/handlers.py in open(self)
173 return
174
--> 175 self._file = h5py.File(self._filename, 'r')
176
177 def close(self):
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/h5py/_hl/files.py in __init__(self, name, mode, driver, libver, userblock_size, swmr, **kwds)
310 with phil:
311 fapl = make_fapl(driver, libver, **kwds)
--> 312 fid = make_fid(name, mode, userblock_size, fapl, swmr=swmr)
313
314 if swmr_support:
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/h5py/_hl/files.py in make_fid(name, mode, userblock_size, fapl, fcpl, swmr)
140 if swmr and swmr_support:
141 flags |= h5f.ACC_SWMR_READ
--> 142 fid = h5f.open(name, flags, fapl=fapl)
143 elif mode == 'r+':
144 fid = h5f.open(name, h5f.ACC_RDWR, fapl=fapl)
h5py/_objects.pyx in h5py._objects.with_phil.wrapper()
h5py/_objects.pyx in h5py._objects.with_phil.wrapper()
h5py/h5f.pyx in h5py.h5f.open()
OSError: Unable to open file (file is already open for write (may use <h5clear file> to clear file consistency flags))For this
scans = [60144,60145,60146]
dark8s = [60148,60147,60147]
nth = 2
print(f'Using S-{scans[nth]} with dark S-{dark8s[nth]} ')
stack = get_fastccd_images(db[scans[nth]], (db[dark8s[nth]], None, None), tag='fccd_image') #fccd_image_lightfield or fccd_image
images = stack[0]the last data set has darks with requested number of images = 2500, but the scan was stopped before it finished (2133). This will be a problem for data collected with swimmer, data that we ask DAMA to edit the database because we are missing the table information, and so on. In this case, we have the table information. I'm not sure what happens if we ^C a motor scan....maybe its okay to use the data up to a certain point, but if the last point doesn't have the full number of images, then this will be problematic.
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-53-edc6e725dc72> in <module>()
1 nth = 2
2 print(f'Using S-{db[scans[nth]].start.scan_id} with dark S-{db[dark8s[nth]].start.scan_id} ')
----> 3 stack = get_fastccd_images(db[scans[nth]], (db[dark8s[nth]], None, None), tag='fccd_image') #fccd_image_lightfield or fccd_image
4 images = stack[0]
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/csxtools/utils.py in get_fastccd_images(light_header, dark_headers, flat, gain, tag, roi)
92
93 tt = ttime.time()
---> 94 b = get_images_to_3D(bgnd_events, dtype=np.uint16)
95 logger.info("Image conversion took %.3f seconds",
96 ttime.time() - tt)
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/csxtools/utils.py in get_images_to_3D(images, dtype)
173
174 """
--> 175 im = np.vstack([np.asarray(im, dtype=dtype) for im in images])
176 return im
177
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/csxtools/utils.py in <listcomp>(.0)
173
174 """
--> 175 im = np.vstack([np.asarray(im, dtype=dtype) for im in images])
176 return im
177
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/numpy/core/numeric.py in asarray(a, dtype, order)
490
491 """
--> 492 return array(a, dtype, copy=False, order=order)
493
494
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/slicerator.py in <genexpr>(.0)
224
225 def __iter__(self):
--> 226 return (self._get(i) for i in self.indices)
227
228 def __len__(self):
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/slicerator.py in _get(self, key)
204
205 def _get(self, key):
--> 206 return self._ancestor[key]
207
208 def _map_index(self, key):
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/slicerator.py in __getitem__(self, i)
185 indices, new_length = key_to_indices(i, len(self))
186 if new_length is None:
--> 187 return self._get(indices)
188 else:
189 return cls(self, indices, new_length, propagate_attrs)
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/pims/base_frames.py in __getitem__(self, key)
146 """__getitem__ is handled by Slicerator. In all pims readers, the data
147 returning function is get_frame."""
--> 148 return self.get_frame(key)
149
150 def __iter__(self):
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/databroker/assets/handlers.py in get_frame(self, i)
35
36 def get_frame(self, i):
---> 37 return Frame(self._dataset[self._start + i], frame_no=i)
38
39 def __len__(self):
h5py/_objects.pyx in h5py._objects.with_phil.wrapper()
h5py/_objects.pyx in h5py._objects.with_phil.wrapper()
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/h5py/_hl/dataset.py in __getitem__(self, args)
474
475 # Perform the dataspace selection.
--> 476 selection = sel.select(self.shape, args, dsid=self.id)
477
478 if selection.nselect == 0:
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/h5py/_hl/selections.py in select(shape, args, dsid)
92
93 sel = SimpleSelection(shape)
---> 94 sel[args]
95 return sel
96
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/h5py/_hl/selections.py in __getitem__(self, args)
259 return self
260
--> 261 start, count, step, scalar = _handle_simple(self.shape,args)
262
263 self._id.select_hyperslab(start, count, step)
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/h5py/_hl/selections.py in _handle_simple(shape, args)
449 else:
450 try:
--> 451 x,y,z = _translate_int(int(arg), length)
452 s = True
453 except TypeError:
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/h5py/_hl/selections.py in _translate_int(exp, length)
469
470 if not 0<=exp<length:
--> 471 raise ValueError("Index (%s) out of range (0-%s)" % (exp, length-1))
472
473 return exp, 1, 1
ValueError: Index (2134) out of range (0-2133)In the discussion of
I noticed that pixels without photons are different values if the flat field is not applied to get_fastccd_images(). For consistency in working with other codes/GUIs not in csxtools, should we change the behavior to match if a flatfield is not provided?
Using an array of all 1's as the flatfield will do this. We can do it manually without changing the function, but it is one more thing to explain to the user. Thoughts?
The normalized images with flatfield applied converts areas with no photons to nan. This is an issue with sk-beam's correlation functions if these pixels are included in an roi, but the problem is already present for flatfield normalized images. And in fact, this conversion to nan is better than letting someone calculate an auto-correlation function with pixels that are essentially zero.
In a way, the flatfield normalization is acting as a pixel mask for pixels that should not be used. This is kinda cool, without having to carry around a separate array for this mask.
Can anyone think of any other problems/benefits or things we should keep in mind before making a decision?
https://github.com/NSLS-II-CSX/csxtools/blob/master/examples/Correct_FastCCD_Images.ipynb
I cannot reproduce this notebook. Logging code is still in the functions of CSXtools (and the level is set to the same value ("info").
Some minor work is needed.
Address bits of code that are less likely to fail in testing so far but could fail if conditions are met (flow control choices - most common seem to work ok)
As per @hyperrealist in #89
find_possible_darks() will run into an unhandled exception at this line if header.stop["exit_status"] == "abort" since exp_time will not be defined in that case.find_possible_darks() may return a pandas DataFrame here or a python dictionary here depending on the execution path.And then also:
np.copy() from fccd.py and overscan.pyFar future for a new issue after this one is addressed:
Possible corrupt data for gain2???
Using None or db[354022] works for bgnd2, but the scan that is supposed to be used results in an error (db[135026]).
scans = [134931, 134958, 134914, 134910, 135020, 135024, ]
bgnd1 = [134934, 134961, 134917, 134913, 135023, 135027, ]#None #
bgnd2 = [134933, 134960, 134916, 134912, 135022, 135026, ]#None #
bgnd8 = [134932, 134959, 134915, 134911, 135021, 135025, ]
ii = 5 #which scan index
scan_no = scans[ii]
bgnd1 = db[bgnd1[ii]]
bgnd2 = db[bgnd2[ii]]
bgnd8 = db[bgnd8[ii],]
#images = get_fastccd_images(h, (bgnd8[0],None,None), flat=None) # **works** 135024
#images = get_fastccd_images(h, (bgnd8[0],None, bgnd1), flat=None) # works 135024
images = get_fastccd_images(h, (bgnd8[0],bgnd2,None), flat=None) # doesn't work for 135024
#images = get_fastccd_images(h, (bgnd8[0], bgnd2, bgnd1), flat=None) # doesn't work for 135024
stack = get_images_to_4D(images) / ADUpPH
print('Raw image size is: ',stack.shape)/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/csxtools/utils.py:181: UserWarning: Images and get_images are deprecated. Use Header.data('fccd_image') instead.
images = header.db.get_images(header, tag)
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-227-35c7fa5e722f> in <module>()
2 #images = get_fastccd_images(h, (bgnd8[0],None,None), flat=None) # works 135024
3 #images = get_fastccd_images(h, (bgnd8[0],None, bgnd1), flat=None) # works 135024
----> 4 images = get_fastccd_images(h, (bgnd8[0],bgnd2,None), flat=None) # doesn't work for 135024
5 #images = get_fastccd_images(h, (bgnd8[0], bgnd2, bgnd1), flat=None) # doesn't work for 135024
6
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/csxtools/utils.py in get_fastccd_images(light_header, dark_headers, flat, gain, tag, roi)
92
93 tt = ttime.time()
---> 94 b = get_images_to_3D(bgnd_events, dtype=np.uint16)
95 logger.info("Image conversion took %.3f seconds",
96 ttime.time() - tt)
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/csxtools/utils.py in get_images_to_3D(images, dtype)
173
174 """
--> 175 im = np.vstack([np.asarray(im, dtype=dtype) for im in images])
176 return im
177
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/csxtools/utils.py in <listcomp>(.0)
173
174 """
--> 175 im = np.vstack([np.asarray(im, dtype=dtype) for im in images])
176 return im
177
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/numpy/core/numeric.py in asarray(a, dtype, order)
490
491 """
--> 492 return array(a, dtype, copy=False, order=order)
493
494
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/slicerator.py in <genexpr>(.0)
224
225 def __iter__(self):
--> 226 return (self._get(i) for i in self.indices)
227
228 def __len__(self):
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/slicerator.py in _get(self, key)
204
205 def _get(self, key):
--> 206 return self._ancestor[key]
207
208 def _map_index(self, key):
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/slicerator.py in __getitem__(self, i)
185 indices, new_length = key_to_indices(i, len(self))
186 if new_length is None:
--> 187 return self._get(indices)
188 else:
189 return cls(self, indices, new_length, propagate_attrs)
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/pims/base_frames.py in __getitem__(self, key)
146 """__getitem__ is handled by Slicerator. In all pims readers, the data
147 returning function is get_frame."""
--> 148 return self.get_frame(key)
149
150 def __iter__(self):
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/databroker/assets/handlers.py in get_frame(self, i)
35
36 def get_frame(self, i):
---> 37 return Frame(self._dataset[self._start + i], frame_no=i)
38
39 def __len__(self):
h5py/_objects.pyx in h5py._objects.with_phil.wrapper()
h5py/_objects.pyx in h5py._objects.with_phil.wrapper()
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/h5py/_hl/dataset.py in __getitem__(self, args)
474
475 # Perform the dataspace selection.
--> 476 selection = sel.select(self.shape, args, dsid=self.id)
477
478 if selection.nselect == 0:
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/h5py/_hl/selections.py in select(shape, args, dsid)
92
93 sel = SimpleSelection(shape)
---> 94 sel[args]
95 return sel
96
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/h5py/_hl/selections.py in __getitem__(self, args)
259 return self
260
--> 261 start, count, step, scalar = _handle_simple(self.shape,args)
262
263 self._id.select_hyperslab(start, count, step)
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/h5py/_hl/selections.py in _handle_simple(shape, args)
449 else:
450 try:
--> 451 x,y,z = _translate_int(int(arg), length)
452 s = True
453 except TypeError:
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/h5py/_hl/selections.py in _translate_int(exp, length)
469
470 if not 0<=exp<length:
--> 471 raise ValueError("Index (%s) out of range (0-%s)" % (exp, length-1))
472
473 return exp, 1, 1
ValueError: Index (1) out of range (0-0)At one point, widgets were broken in jupytherhub, but now we are conda env associated with 2018-3. This function looks to be very broken at this point.
ValueError: Invalid file object: <_io.BufferedReader name=65> is the Exception, but I tried with just 10 images that were just a portion of the FastCCD image, so the problem is not with the input array.
Also not there are some requested enhancements for usability: #57
image_stack_to_movie(sub_images[0:10], vmin=np.percentile(sub_images[1],30), vmax=np.percentile(sub_images[1],90))
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-193-d87213fc5ab6> in <module>()
1 #THIS IS CURRENTLY BROKEN
2 #
----> 3 image_stack_to_movie(sub_images[0:10], vmin=np.percentile(sub_images[1],30), vmax=np.percentile(sub_images[1],90))
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/csxtools/ipynb/animation.py in image_stack_to_movie(images, frames, vmin, vmax, figsize, cmap, fps)
102 plt.close(anim._fig)
103 # return anim.to_html5_video()
--> 104 return HTML(_anim_to_html(anim, fps))
105
106
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/csxtools/ipynb/animation.py in _anim_to_html(anim, fps)
115 anim.save(f.name, fps=fps,
116 extra_args=['-vcodec', 'libx264',
--> 117 '-pix_fmt', 'yuv420p'])
118 video = open(f.name, "rb").read()
119 anim._encoded_video = base64.b64encode(video)
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/matplotlib/animation.py in save(self, filename, writer, fps, dpi, codec, bitrate, extra_args, metadata, extra_anim, savefig_kwargs)
1198 # TODO: See if turning off blit is really necessary
1199 anim._draw_next_frame(d, blit=False)
-> 1200 writer.grab_frame(**savefig_kwargs)
1201
1202 # Reconnect signal for first draw if necessary
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/contextlib.py in __exit__(self, type, value, traceback)
97 value = type()
98 try:
---> 99 self.gen.throw(type, value, traceback)
100 except StopIteration as exc:
101 # Suppress StopIteration *unless* it's the same exception that
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/matplotlib/animation.py in saving(self, fig, outfile, dpi, *args, **kwargs)
239 yield self
240 finally:
--> 241 self.finish()
242
243
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/matplotlib/animation.py in finish(self)
365 def finish(self):
366 '''Finish any processing for writing the movie.'''
--> 367 self.cleanup()
368
369 def grab_frame(self, **savefig_kwargs):
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/matplotlib/animation.py in cleanup(self)
403 def cleanup(self):
404 '''Clean-up and collect the process used to write the movie file.'''
--> 405 out, err = self._proc.communicate()
406 self._frame_sink().close()
407 _log.debug('MovieWriter -- Command stdout:\n%s', out)
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/subprocess.py in communicate(self, input, timeout)
841
842 try:
--> 843 stdout, stderr = self._communicate(input, endtime, timeout)
844 finally:
845 self._communication_started = True
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/subprocess.py in _communicate(self, input, endtime, orig_timeout)
1503 selector.register(self.stdin, selectors.EVENT_WRITE)
1504 if self.stdout:
-> 1505 selector.register(self.stdout, selectors.EVENT_READ)
1506 if self.stderr:
1507 selector.register(self.stderr, selectors.EVENT_READ)
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/selectors.py in register(self, fileobj, events, data)
349
350 def register(self, fileobj, events, data=None):
--> 351 key = super().register(fileobj, events, data)
352 poll_events = 0
353 if events & EVENT_READ:
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/selectors.py in register(self, fileobj, events, data)
235 raise ValueError("Invalid events: {!r}".format(events))
236
--> 237 key = SelectorKey(fileobj, self._fileobj_lookup(fileobj), events, data)
238
239 if key.fd in self._fd_to_key:
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/selectors.py in _fileobj_lookup(self, fileobj)
222 """
223 try:
--> 224 return _fileobj_to_fd(fileobj)
225 except ValueError:
226 # Do an exhaustive search.
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/selectors.py in _fileobj_to_fd(fileobj)
37 except (AttributeError, TypeError, ValueError):
38 raise ValueError("Invalid file object: "
---> 39 "{!r}".format(fileobj)) from None
40 if fd < 0:
41 raise ValueError("Invalid file descriptor: {}".format(fd))
ValueError: Invalid file object: <_io.BufferedReader name=65>conda-forge/csxtools-feedstock#3 failed with a missing import. Need to resolve it on the package level and release a new version with the fix.
import: 'csxtools'
/home/conda/feedstock_root/build_artifacts/csxtools_1642620788862/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.10/site-packages/pims/image_reader.py:26: RuntimeWarning: PIMS image_reader.py could not find scikit-image. Falling back to matplotlib's imread(), which uses floats instead of integers. This may break your scripts.
(To ignore this warning, include the line "warnings.simplefilter("ignore", RuntimeWarning)" in your script.)
warnings.warn(RuntimeWarning(ski_preferred))
Traceback (most recent call last):
File "/home/conda/feedstock_root/build_artifacts/csxtools_1642620788862/test_tmp/run_test.py", line 2, in <module>
import csxtools
File "/home/conda/feedstock_root/build_artifacts/csxtools_1642620788862/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.10/site-packages/csxtools/__init__.py", line 3, in <module>
from .utils import (get_fastccd_images, get_fastccd_timestamps) # noqa F401
File "/home/conda/feedstock_root/build_artifacts/csxtools_1642620788862/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.10/site-packages/csxtools/utils.py", line 3, in <module>
from pims import pipeline
File "/home/conda/feedstock_root/build_artifacts/csxtools_1642620788862/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.10/site-packages/pims/__init__.py", line 1, in <module>
from pims.api import *
File "/home/conda/feedstock_root/build_artifacts/csxtools_1642620788862/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.10/site-packages/pims/api.py", line 19, in <module>
from .cine import Cine # noqa
File "/home/conda/feedstock_root/build_artifacts/csxtools_1642620788862/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.10/site-packages/pims/cine.py", line 29, in <module>
from collections import Iterable
ImportError: cannot import name 'Iterable' from 'collections' (/home/conda/feedstock_root/build_artifacts/csxtools_1642620788862/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.10/collections/__init__.py)
Tests failed for csxtools-0.1.16-py310hb5077e9_1.tar.bz2 - moving package to /home/conda/feedstock_root/build_artifacts/brokenConda packages for csx analysis from 2019-1.2 works okay on this and other functions.
databroker 0.12.0
csxtools 0.13.0
Using conda packages for csx analysis from 2019-3.0.
databroker 0.13.0
csxtools 0.14.0
Will show flatfield function and then standard images functions (without flatfield). Note that flatfield uses get_fastccd_images() so this is likely the problem. This is different from what @leofang described #68
#0.3s acq - same stats as above
bgnd1 = db[117332+3]
bgnd2 = db[117332+2]
bgnd8 = db[117332+1]
flat = db[117332]
ff = get_fastccd_flatfield(flat, (bgnd8, bgnd2, bgnd1))KeyError Traceback (most recent call last)
<ipython-input-17-162df5f3ae04> in <module>
14 #ff = get_fastccd_flatfield(flat, (bgnd8, None, None))#, limits=(0.8, 1.2))
15
---> 16 ff = get_fastccd_flatfield(flat, (bgnd8, bgnd2, bgnd1))
~/conda_envs/csx-dev-analysis/lib/python3.7/site-packages/csxtools/utils.py in get_fastccd_flatfield(light, dark, flat, limits)
291 Flatfield correction
292 """
--> 293 images = get_images_to_3D(get_fastccd_images(light, dark, flat))
294 images = stackmean(images)
295 flat = calculate_flatfield(images, limits)
~/conda_envs/csx-dev-analysis/lib/python3.7/site-packages/csxtools/utils.py in get_fastccd_images(light_header, dark_headers, flat, gain, tag, roi)
92
93 tt = ttime.time()
---> 94 b = get_images_to_3D(bgnd_events, dtype=np.uint16)
95 logger.info("Image conversion took %.3f seconds",
96 ttime.time() - tt)
~/conda_envs/csx-dev-analysis/lib/python3.7/site-packages/csxtools/utils.py in get_images_to_3D(images, dtype)
173
174 """
--> 175 im = np.vstack([np.asarray(im, dtype=dtype) for im in images])
176 return im
177
~/conda_envs/csx-dev-analysis/lib/python3.7/site-packages/csxtools/utils.py in <listcomp>(.0)
173
174 """
--> 175 im = np.vstack([np.asarray(im, dtype=dtype) for im in images])
176 return im
177
~/conda_envs/csx-dev-analysis/lib/python3.7/site-packages/databroker/_core.py in data(self, field, stream_name, fill)
490 for event in self.events(stream_name=stream_name,
491 fields=[field],
--> 492 fill=fill):
493 yield event['data'][field]
494
~/conda_envs/csx-dev-analysis/lib/python3.7/site-packages/databroker/_core.py in events(self, stream_name, fields, fill)
380 ev_gen = self.db.get_events([self], stream_name=stream_name,
381 fields=fields, fill=fill)
--> 382 for ev in ev_gen:
383 yield ev
384
~/conda_envs/csx-dev-analysis/lib/python3.7/site-packages/databroker/_core.py in get_events(self, headers, stream_name, fields, fill, handler_registry)
1512 stream_name=stream_name,
1513 fill=fill,
-> 1514 handler_registry=handler_registry):
1515 if name == 'event':
1516 yield doc
~/conda_envs/csx-dev-analysis/lib/python3.7/site-packages/databroker/_core.py in get_documents(self, headers, stream_name, fields, fill, handler_registry)
1606 datum)
1607
-> 1608 doc = proc_gen.send(doc)
1609
1610 yield name, self.prepare_hook(name, doc)
~/conda_envs/csx-dev-analysis/lib/python3.7/site-packages/databroker/_core.py in _fill_events_coro(self, descriptors, fields, inplace)
2056 d_id = data[dk]
2057 data[dk] = (registry_map[(desc_id, dk)]
-> 2058 .retrieve(d_id))
2059 filled[dk] = d_id
2060
~/conda_envs/csx-dev-analysis/lib/python3.7/site-packages/databroker/assets/base_registry.py in retrieve(self, datum_id)
176 return self._api.retrieve(self._datum_col, datum_id,
177 self._datum_cache, self.get_spec_handler,
--> 178 logger)
179
180 def get_datum(self, datum_id):
~/conda_envs/csx-dev-analysis/lib/python3.7/site-packages/databroker/assets/core.py in retrieve(col, datum_id, datum_cache, get_spec_handler, logger)
63 def retrieve(col, datum_id, datum_cache, get_spec_handler, logger):
64 datum = _get_datum_from_datum_id(col, datum_id, datum_cache, logger)
---> 65 handler = get_spec_handler(datum['resource'])
66 return handler(**datum['datum_kwargs'])
67
~/conda_envs/csx-dev-analysis/lib/python3.7/site-packages/databroker/assets/base_registry.py in get_spec_handler(self, resource)
245
246 spec = resource['spec']
--> 247 handler = self.handler_reg[spec]
248
249 key = (str(resource['uid']), handler.__name__)
~/conda_envs/csx-dev-analysis/lib/python3.7/collections/__init__.py in __getitem__(self, key)
912 except KeyError:
913 pass
--> 914 return self.__missing__(key) # support subclasses that define __missing__
915
916 def get(self, key, default=None):
~/conda_envs/csx-dev-analysis/lib/python3.7/collections/__init__.py in __missing__(self, key)
904
905 def __missing__(self, key):
--> 906 raise KeyError(key)
907
908 def __getitem__(self, key):
KeyError: 'AD_HDF5'#%%capture
images = get_fastccd_images(h, (bgnd8[0], bgnd2, bgnd1), flat=None)
stack = get_images_to_4D(images)---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
<ipython-input-11-19cab0555ecf> in <module>
1 #%%capture
----> 2 images = get_fastccd_images(h, (bgnd8[0], bgnd2, bgnd1), flat=None)
3 #stack = get_images_to_4D(images)
4
5 #### JUST ONE IMAGE PER POINT - this may work instead #
~/conda_envs/csx-dev-analysis/lib/python3.7/site-packages/csxtools/utils.py in get_fastccd_images(light_header, dark_headers, flat, gain, tag, roi)
92
93 tt = ttime.time()
---> 94 b = get_images_to_3D(bgnd_events, dtype=np.uint16)
95 logger.info("Image conversion took %.3f seconds",
96 ttime.time() - tt)
~/conda_envs/csx-dev-analysis/lib/python3.7/site-packages/csxtools/utils.py in get_images_to_3D(images, dtype)
173
174 """
--> 175 im = np.vstack([np.asarray(im, dtype=dtype) for im in images])
176 return im
177
~/conda_envs/csx-dev-analysis/lib/python3.7/site-packages/csxtools/utils.py in <listcomp>(.0)
173
174 """
--> 175 im = np.vstack([np.asarray(im, dtype=dtype) for im in images])
176 return im
177
~/conda_envs/csx-dev-analysis/lib/python3.7/site-packages/databroker/_core.py in data(self, field, stream_name, fill)
490 for event in self.events(stream_name=stream_name,
491 fields=[field],
--> 492 fill=fill):
493 yield event['data'][field]
494
~/conda_envs/csx-dev-analysis/lib/python3.7/site-packages/databroker/_core.py in events(self, stream_name, fields, fill)
380 ev_gen = self.db.get_events([self], stream_name=stream_name,
381 fields=fields, fill=fill)
--> 382 for ev in ev_gen:
383 yield ev
384
~/conda_envs/csx-dev-analysis/lib/python3.7/site-packages/databroker/_core.py in get_events(self, headers, stream_name, fields, fill, handler_registry)
1512 stream_name=stream_name,
1513 fill=fill,
-> 1514 handler_registry=handler_registry):
1515 if name == 'event':
1516 yield doc
~/conda_envs/csx-dev-analysis/lib/python3.7/site-packages/databroker/_core.py in get_documents(self, headers, stream_name, fields, fill, handler_registry)
1606 datum)
1607
-> 1608 doc = proc_gen.send(doc)
1609
1610 yield name, self.prepare_hook(name, doc)
~/conda_envs/csx-dev-analysis/lib/python3.7/site-packages/databroker/_core.py in _fill_events_coro(self, descriptors, fields, inplace)
2056 d_id = data[dk]
2057 data[dk] = (registry_map[(desc_id, dk)]
-> 2058 .retrieve(d_id))
2059 filled[dk] = d_id
2060
~/conda_envs/csx-dev-analysis/lib/python3.7/site-packages/databroker/assets/base_registry.py in retrieve(self, datum_id)
176 return self._api.retrieve(self._datum_col, datum_id,
177 self._datum_cache, self.get_spec_handler,
--> 178 logger)
179
180 def get_datum(self, datum_id):
~/conda_envs/csx-dev-analysis/lib/python3.7/site-packages/databroker/assets/core.py in retrieve(col, datum_id, datum_cache, get_spec_handler, logger)
63 def retrieve(col, datum_id, datum_cache, get_spec_handler, logger):
64 datum = _get_datum_from_datum_id(col, datum_id, datum_cache, logger)
---> 65 handler = get_spec_handler(datum['resource'])
66 return handler(**datum['datum_kwargs'])
67
~/conda_envs/csx-dev-analysis/lib/python3.7/site-packages/databroker/assets/base_registry.py in get_spec_handler(self, resource)
245
246 spec = resource['spec']
--> 247 handler = self.handler_reg[spec]
248
249 key = (str(resource['uid']), handler.__name__)
~/conda_envs/csx-dev-analysis/lib/python3.7/collections/__init__.py in __getitem__(self, key)
912 except KeyError:
913 pass
--> 914 return self.__missing__(key) # support subclasses that define __missing__
915
916 def get(self, key, default=None):
~/conda_envs/csx-dev-analysis/lib/python3.7/collections/__init__.py in __missing__(self, key)
904
905 def __missing__(self, key):
--> 906 raise KeyError(key)
907
908 def __getitem__(self, key):
KeyError: 'AD_HDF5'see discussion in #77 for minimum_ADU suggestion. For FCCD, beamline staff recoomend collecting data wiht average 1000 ADU per pixel for a robust flatfield data set.
Should csxtools be able to handle passing more than I scanid to db as below?
from databroker import DataBroker as db
from csxtools.utils import get_fastccd_images
slicerator = get_fastccd_images(db[108225, 108226])
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-5-5adf5db257b5> in <module>()
2 from csxtools.utils import get_fastccd_images
3
----> 4 slicerator = get_fastccd_images(db[108225, 108226])
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/csxtools/utils.py in get_fastccd_images(light_header, dark_headers, flat, gain, tag, roi)
119 logger.info("Computed dark images in %.3f seconds", ttime.time() - t)
120
--> 121 events = _get_images(light_header, tag, roi)
122
123 # Ok, so lets return a pims pipeline which does the image conversion
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/csxtools/utils.py in _get_images(header, tag, roi)
179 def _get_images(header, tag, roi=None):
180 t = ttime.time()
--> 181 images = header.db.get_images(header, tag)
182 t = ttime.time() - t
183 logger.info("Took %.3f seconds to read data using get_images", t)
AttributeError: 'list' object has no attribute 'db'
We might consider raising a warning if stacksum encounters NaN's:
"stacksum encountered NaN values and excluded these values from the sum effectively assuming NaN = 0.
Consider using the number of points, which is returned as the second argument in the tuple, to renormalize the image."
@stuwilkins Are you ready to tag a release? If so, I will deploy it on Jupyterhub. cc @vivekthampy
Right now get_fastccd_images gets the whole stack of images into one numpy array. Think it might be useful to have that broken up by events to plot scans (such as energy scans).
Working on this right now.
The latest csxtools version on srv1 seems to have a bug in the image_stack_to_movie function.
import numpy as np
import matplotlib.pyplot as plt
from csxtools.ipynb import image_stack_to_movie
%matplotlib nbagg
X, Y = np.meshgrid(np.linspace(-1, 1), np.linspace(-1, 1))
def gauss(X, Y, A, sigma):
return A*np.exp(-(X**2+Y**2)/(2*sigma**2))
As = np.linspace(0.8, 1.2, 10)
sigmas = np.linspace(0.5, 0.7, 10)
stack = np.array([gauss(X, Y, A, sigma) for A, sigma in zip(As, sigmas)])
image_stack_to_movie(stack)
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-3-8dcbc2cc7342> in <module>()
----> 1 image_stack_to_movie(stack)
/opt/conda_envs/analysis-2018-1.0/lib/python3.6/site-packages/csxtools/ipynb/animation.py in image_stack_to_movie(images, frames, vmin, vmax, figsize, cmap, fps)
102 plt.close(anim._fig)
103 # return anim.to_html5_video()
--> 104 return HTML(_anim_to_html(anim, fps))
105
106
/opt/conda_envs/analysis-2018-1.0/lib/python3.6/site-packages/csxtools/ipynb/animation.py in _anim_to_html(anim, fps)
115 anim.save(f.name, fps=fps,
116 extra_args=['-vcodec', 'libx264',
--> 117 '-pix_fmt', 'yuv420p'])
118 video = open(f.name, "rb").read()
119 anim._encoded_video = base64.b64encode(video)
/opt/conda_envs/analysis-2018-1.0/lib/python3.6/site-packages/matplotlib/animation.py in save(self, filename, writer, fps, dpi, codec, bitrate, extra_args, metadata, extra_anim, savefig_kwargs)
1061 # TODO: See if turning off blit is really necessary
1062 anim._draw_next_frame(d, blit=False)
-> 1063 writer.grab_frame(**savefig_kwargs)
1064
1065 # Reconnect signal for first draw if necessary
/opt/conda_envs/analysis-2018-1.0/lib/python3.6/contextlib.py in __exit__(self, type, value, traceback)
97 value = type()
98 try:
---> 99 self.gen.throw(type, value, traceback)
100 except StopIteration as exc:
101 # Suppress StopIteration *unless* it's the same exception that
/opt/conda_envs/analysis-2018-1.0/lib/python3.6/site-packages/matplotlib/animation.py in saving(self, *args, **kw)
287 yield self
288 finally:
--> 289 self.finish()
290
291 def _run(self):
/opt/conda_envs/analysis-2018-1.0/lib/python3.6/site-packages/matplotlib/animation.py in finish(self)
307 def finish(self):
308 'Finish any processing for writing the movie.'
--> 309 self.cleanup()
310
311 def grab_frame(self, **savefig_kwargs):
/opt/conda_envs/analysis-2018-1.0/lib/python3.6/site-packages/matplotlib/animation.py in cleanup(self)
346 def cleanup(self):
347 'Clean-up and collect the process used to write the movie file.'
--> 348 out, err = self._proc.communicate()
349 self._frame_sink().close()
350 verbose.report('MovieWriter -- '
/opt/conda_envs/analysis-2018-1.0/lib/python3.6/subprocess.py in communicate(self, input, timeout)
836
837 try:
--> 838 stdout, stderr = self._communicate(input, endtime, timeout)
839 finally:
840 self._communication_started = True
/opt/conda_envs/analysis-2018-1.0/lib/python3.6/subprocess.py in _communicate(self, input, endtime, orig_timeout)
1492 selector.register(self.stdin, selectors.EVENT_WRITE)
1493 if self.stdout:
-> 1494 selector.register(self.stdout, selectors.EVENT_READ)
1495 if self.stderr:
1496 selector.register(self.stderr, selectors.EVENT_READ)
/opt/conda_envs/analysis-2018-1.0/lib/python3.6/selectors.py in register(self, fileobj, events, data)
349
350 def register(self, fileobj, events, data=None):
--> 351 key = super().register(fileobj, events, data)
352 poll_events = 0
353 if events & EVENT_READ:
/opt/conda_envs/analysis-2018-1.0/lib/python3.6/selectors.py in register(self, fileobj, events, data)
235 raise ValueError("Invalid events: {!r}".format(events))
236
--> 237 key = SelectorKey(fileobj, self._fileobj_lookup(fileobj), events, data)
238
239 if key.fd in self._fd_to_key:
/opt/conda_envs/analysis-2018-1.0/lib/python3.6/selectors.py in _fileobj_lookup(self, fileobj)
222 """
223 try:
--> 224 return _fileobj_to_fd(fileobj)
225 except ValueError:
226 # Do an exhaustive search.
/opt/conda_envs/analysis-2018-1.0/lib/python3.6/selectors.py in _fileobj_to_fd(fileobj)
37 except (AttributeError, TypeError, ValueError):
38 raise ValueError("Invalid file object: "
---> 39 "{!r}".format(fileobj)) from None
40 if fd < 0:
41 raise ValueError("Invalid file descriptor: {}".format(fd))
ValueError: Invalid file object: <_io.BufferedReader name=52>
'''Lines 318 to 319 in 52ff964
two options:
logger.warning("csxtools.utils.fccd_mask() planned for deprecation. Contact beamline staff ([email protected]) if you use this function.")The calculate_flatfield function has an issue when it is used with the current state of the CSX detector.
The problem is that more than 1/2 the detector is insensitive to light, so the median is not representative of the true data. The line below then means that the data and limits get re-scaled to unreal values.
Line 268 in 52ff964
@stuwilkins The program crashes if the background scan headers have events with different number of images. The crashing function is get_images_to_4D.
https://gist.github.com/vivekthampy/71d110d899f68a61150a
One solution that works for me is using the the function below instead of get_images_to_4D.
def convert_to_3D(images):
if isinstance(images, np.ndarray):
if images.ndim == 3 or images.ndim == 2:
return images
elif images.ndim == 4:
shape = images.shape
return np.reshape(images, (shape[0]*shape[1], shape[2], shape[3]))
else:
ims = images[0]
for im in images[1:]:
ims = np.vstack((ims, im))
return imsShould swap the text for missing dark images regarding bgnd2 and bgnd1. Instead, maybe look at gain bits directly for WARNING text. In example below, 59112 is gain 2 setting.
In [25]:
bgnd8 = DataBroker['6364f4e8-d0b2-42e3-b999-2708c56ef1b6'] #[59111]
bgnd2 = None #DataBroker[59112]
bgnd1 = DataBroker['165e034b-83de-4769-b80f-447194e701c5'] #None # DataBroker[''] #[]
data = DataBroker['2e693cc7-27b3-4ac2-90ef-f3d1319d3066'] #[59110]
In [26]:
stack = get_fastccd_images(data, (bgnd8, bgnd2, bgnd1), tag='fccd_image') #fccd_image_lightfield
images = stack[0]
WARNING:csxtools.utils:Missing dark image for gain setting 1
VS
In [27]:
bgnd8 = DataBroker['6364f4e8-d0b2-42e3-b999-2708c56ef1b6'] #[59111]
bgnd2 = DataBroker['165e034b-83de-4769-b80f-447194e701c5'] #None #DataBroker[59112]
bgnd1 = None # DataBroker[''] #[]
data = DataBroker['2e693cc7-27b3-4ac2-90ef-f3d1319d3066'] #[59110]
In [*]:
stack = get_fastccd_images(data, (bgnd8, bgnd2, bgnd1), tag='fccd_image') #fccd_image_lightfield
images = stack[0]
WARNING:csxtools.utils:Missing
dark image for gain setting 2
csxtools/csxtools/helpers/overscan.py
Line 106 in 04250be
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.