volkerh / lattice_lightsheet_deskew_deconv Goto Github PK

View Code? Open in Web Editor NEW

21.0 21.0 3.0 8.9 MB

Open-source, GPU accelerated code for deskewing and deconvolving lattice light sheet data

License: Other

Python 100.00%

lattice_lightsheet_deskew_deconv's People

Contributors

Stargazers

Watchers

Forkers

peterzs zeroth russtorres

lattice_lightsheet_deskew_deconv's Issues

Support additional output formats Big Data Viewer, Imaris, .h5, Adaptive Particle Representation

Not urgent, but relatively easy to add output format options for Big Data Viewer and Imaris hdf5 via https://github.com/tlambert03/imarispy (should be easy).

Think about supporting adaptive particle representation as output to alleviate the storage problem
https://github.com/AdaptiveParticles/LibAPR (no clue how difficult integration is, only heard about it yesterday).

Architecture not well-suited for multiprocessing due to pickling limitations

Attempted to parallelize processing of images and ran into some frustrations. (Learned some more about the inner workings of Python in the process though).

The way I tried to achieve thiswas by using a multiprocessing pool (see https://github.com/VolkerH/Lattice_Lightsheet_Deskew_Deconv/blob/parallel_read_write/lls_dd/process_llsm_experiment.py)

        lock=Lock()
        # throws an error ... must be moved to a top-level function (at least on windows) :(
        # https://stackoverflow.com/questions/48046862/python-django-multiprocessing-error-attributeerror-cant-pickle-local-object
        def process_indexrow(irow):
            index, row = irow
            wavelength = row.wavelength
            self.process_file(
                lock,
                pathlib.Path(row.file),
                deskew_func,
                rotate_func,
                deconv_functions[wavelength],
            )

        indexrowl = list(subset_files.iterrows())
        with Pool(4) as p:
r = list(tqdm.tqdm(p.imap(process_indexrow, indexrowl), total=len(indexrowl)))

The GPU functions in ExperimentProcesser.processfile() (which I don't want to run in parallel, to prevent running out of GPU memory) were protected by the Lock(). Still, the file reading and writing should be handled by several processes from the multiprocessing pool, thus implementing pipelining that ensures that there is a steady stream of data ready to be passed to the GPU for processing and that the GPU doesn't sit idle during IO. That was the idea.

It turns out that this approach does not work at all, as multiprocessing uses pickle to serialize the functions that are being called and the data that is being passed. Apparently pickle can only serialize top-level functions, not class methods. (There are numerous stackoverflow posts on this topic, e.g. this one https://stackoverflow.com/questions/8804830/python-multiprocessing-picklingerror-cant-pickle-type-function). One recommended approach is to use pathos which implements a similar interface in pathos.multiprocessing but uses a different package for serialization called dill which can also serialize class methods and lambda functions.

However, even with pathos (or dill) I ran into serialization issues. dill seems to pass some objects to pickle for serialization and I ended up with this error message (shortened):

    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\site-packages\dill\_dill.py", line 902, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\pickle.py", line 847, in _batch_setitems
    save(v)
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\pickle.py", line 496, in save
    rv = reduce(self.proto)
TypeError: can't pickle SwigPyObject objects

So some of the data or functions I parallelize wrap C/C++ using SWIG and those wrapped functions/objects can't be serialized with pickle or dill.
So what options do I have? Looking at the dask documentation (http://distributed.dask.org/en/latest/serialization.html) it appears that dask supports several methods for serialization (including pickle) but I'm not sure any of those will handle class methods and SwigPyObjects that wrap C++ code.

So the take-away messages for me are:

the current architecture with class methods that have partially evaluated functions as arguments is probably not a good one when trying to use multiprocessing.
seperate functions that are based on SWIG wrappers or have SWIG objects as arguments from those that do not require SWIG, so that the latter can be used with multiprocessing.

Maybe @jni has some advice?

openCL error

---------------------------------------------------------------------------
MemoryError                               Traceback (most recent call last)
<ipython-input-7-6b02bb31308f> in <module>
----> 1 ep.process_stack_subfolder('Stack_10_drp1_dendra2skl_mScarletdrp1')

~/Lattice_Lightsheet_Deskew_Deconv/Python/process_llsm_experiment.py in process_stack_subfolder(self, stack_name)
    265             if self.do_deconv:
    266                 # determine wavelength for file and pick corresponding PSF
--> 267                 self.process_file(pathlib.Path(row.file), deskew_func, rotate_func, deconv_functions[wavelength])
    268             else:
    269                 self.process_file(pathlib.Path(row.file), deskew_func, rotate_func)

~/Lattice_Lightsheet_Deskew_Deconv/Python/process_llsm_experiment.py in process_file(self, infile, deskew_func, rotate_func, deconv_func)
    176                     self.create_MIP(deconv_deskewed.astype(self.output_dtype), outfiles["deconv/deskew/MIP"])
    177             if self.do_deconv_rotate:
--> 178                 deconv_rotated = rotate_func(deconv_raw)
    179                 write_tiff_createfolder(outfiles["deconv/rotate"], deconv_rotated.astype(self.output_dtype))
    180                 if self.do_MIP:

~/Lattice_Lightsheet_Deskew_Deconv/Python/gputools_wrapper.py in affine_transform_gputools(input, matrix, offset, output_shape, output, order, mode, cval, prefilter)
     58     with warnings.catch_warnings():
     59         warnings.simplefilter("ignore")
---> 60         result = gputools.affine(data=input, mat=matrix, mode=mode, interpolation=interpolation)
     61 
     62     if needs_crop:

~/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/gputools/transforms/transformations.py in affine(data, mat, mode, interpolation)
     81     prog.run_kernel("affine3",
     82                     data.shape[::-1], None,
---> 83                     d_im, res_g.data, mat_inv_g.data)
     84 
     85     return res_g.get()

~/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/gputools/core/oclprogram.py in run_kernel(self, name, global_size, local_size, *args, **kwargs)
     44             self._kernel_dict[name] = getattr(self,name)
     45 
---> 46         self._kernel_dict[name](self._dev.queue,global_size, local_size,*args,**kwargs)
     47 
     48 

~/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/pyopencl/__init__.py in kernel_call(self, queue, global_size, local_size, *args, **kwargs)
    813         # __call__ can't be overridden directly, so we need this
    814         # trampoline hack.
--> 815         return self._enqueue(self, queue, global_size, local_size, *args, **kwargs)
    816 
    817     def kernel_capture_call(self, filename, queue, global_size, local_size,

<generated code> in enqueue_knl_affine3(self, queue, global_size, local_size, arg0, arg1, arg2, global_offset, g_times_l, wait_for)

MemoryError: clEnqueueNDRangeKernel failed: MEM_OBJECT_ALLOCATION_FAILURE

Think about deployment for end-users

My initial thought was that - once batch processing works - users could run a jupyter notebook in which
they set input and output path and select other options via ipywidgets. Like a mininmalist GUI.

In a sense, what I want is a GUI provided on the user's local machine and the data and the processing happening on the server. In a sense that is how I currently use Jupyter on the cluster.

imjoy.io has shown a very nice way of how to do this even better and more user-friendly.
The deconv/deskew engine (once it is finished) could simply be implemented as a plugin to imjoy.io with a plugin backend runnning on a powerful GPU node.

However, I am not quite sure whether it provides GUI options to set parameters yet, e.g. (but one can quickly go to the code of the plugin and edit it on the fly - even on a mobile phone).

A crazy idea at this stage but something to revisit later.

Add scale to processed tiffs

embed scaling/unit information (pixel / mm in X,Y,Z) in tiff tags in a way that is understood by Fiji.

add tests

time to start adopting some proper software engineering practices

Performance in batch mode

For this particular dataset, the naive approach takes about 1s per image including read/write if I use this naive approach. I can see the GPU utilization going up and down as well.

In my commandline batch tool processing that same dataset is almost a factor of 4 slower:

That code does an additional affine transform and MIP but that should not make a significant difference. Maybe the overhead is due to passing around partially evaluated functions.

PSF subpixel fitting

code for finding centre of bead for creating the PSF is crude and is lacking subpixel fitting
Look at something like this (extend to 3D)
https://github.com/david-hoffman/gaussfitter
or some of the fitting code for PALM/STORM detections for accurate localization

reuse PSF

when processing several stacks in an experiment, psf processing happens for every stack

Not enough GPU ram ... make a "slow" option

both gputools (which used reika for fft) and flowdec run out of GPU memory when trying to process a stack of size (151, 800, 600).

Depending on what I am trying do exactly the error message in tensorflow either shows up when initializing the batch cufft plan or later on when allocating space for a tensor.

One of the error messages I saw is that it was trying to allocate a tensor of size 256, 1024, 1024. When I crop the volume by 23 pixes in Z (this would correspond to 128 z slices), everything works fine. When I only crop 22 pixels it fails.

flowdec rounds up the sizes to the next size where the fastest FFT can be performed. This appears to be very generous rounding. It would be nice to be able to trade some speed for the ability to process such volumes. I should look into adding an option to round up to the next size for which an FFT can be performed, even if it is not optimal in terms of speed.

PSF management - lattice pattern in file name

Currently, we don't seem to have lattice pattern information in our PSF filenames.
This seems to differ from other systems, e.g. the LLSpy documentation refers to a particular naming scheme for PSF acquisitions.

TODO:

check how our file names differ
see whether we can adopt the same naming system

Real-Time deskew visualization

Opening this issue to continue the discussion with @dmilkie from here:
hammerlab/flowdec#12

I don't think we'll get the deconv running in realtime even with flowdec it still takes a few seconds per stack on our HPC node with a Tesla K80.

I have only recently gotten the Janelia cudaDeconv binary to run on our HPC cluster, but haven't used it much. Never got it to work with LLSpy. Not having access to the source code made me look into alternatives for the deconv and I found flowdec. I haven't benchmarked it but it doesn't feel like it is orders of magnitudes faster than cudaDeconv.

Performing the deconvolution directly on the raw data with a skewed seems to bring a better speed improvement for me as no deconvolution is performed on empty fill voxels.

I discussed some ideas for near-live deskewed volume visualization with David, who runs our LLS.
One idea was to help users orient themselves and navigate around a larger sample by stiching any volumes already captured together and visualize them together with the current position. This would require having information about the stage position in the metadata though and as far as I could see it is not currently recorded (at least in the version of the Labview software that we are using). It was on my TODO list to look at the Labview code to see whether I can add this bit (not very good with Labview though and need to get a license first - currently using the runtime only).

Artefacts on RTX 2080 Ti

Noticed all sorts of weird artefacts on some images processed with the RTX 2080Ti.
Happens both with flowdec and with gputools deconvolution, but not on all images.
Some cellphone shots:

Just running the same data set on the K80 cards on our cluster and so far not seeing such problems. Starting to think this is a hardware issue. Might have one of those defective RTX2080 Ti cards, probably memory errors.

Allow deskewing when no PSF is present

Currently, one of the first things the program does is to scan the Experiment folders for expected files and to parse them.
When no PSF files are found, the program quits with an error message, making it impossible to run deskew only (for which PSFs are not required).

add sample image data sets

include gputools.deconv as deconvolution backend

While looking for solutions to also perform GPU-accelerated decon on non-CUDA GPUs I just noticed that gputools also has a RL deconvolution built on the Reikna OpenCL fft. A first test in a Jupyter notebook showed that this works as well (although it seems to produce a few more ringing artefacts) and could be included as an alternative deconvolution backend.

Took about 7 seconds for 10 iterations on my laptop's on-chip GPU. (this is without the I/O)

Include as backend.

psf z_galvo step extraction fails for 20190802 dataset

something about the PSF metadata appears to be malformed in this particular dataset. which causes PSF generation to crash when trying to deconvolve.
TODO: catch exception and try to recover or at least provide useful error message

fixed camera background subtraction - clip at 0, allow negative ? Also include for PSF.

Existing solutions subtract a fixed camera offset value from each stack (around greyvalue 100 for Orca Flash).
There are two options to handle this:

Allow negative values. This seems reasonable to me. Needs change of dtype from np.uint16 to np.int16 or np.float. Currently, some processing steps convert the output array to floating point anyway and I need to cast the output to np.uint16 before saving. Not sure where this is happening, probably already with the affine transformations. Not sure how flowdec handles negative values.
Clip negative values to 0 after background subtraction. Can keep input as np.uint16

PSF background subtraction currently happens by estimating the background in areas away from the bead (median of values in top and bottom Z-slice of PSF values). In that case I'm clipping as I don't want to have negative values in the PSF (doesn't make physical sense). If I subtract a fixed camera offset, I shoud probably also for the PSF.

fix paths to illustrations in jupyter notebooks

Run notebooks in binder or colab ?

Would be nice if one could run the notebooks in binder. However it appears there is no free binder option with GPU support.
Google colab has GPU support but not sure whether it is possible to install pyopengl.

let people know about python lls deskew deconv code

Hi,
just wanted to make you aware of this repo. Still very early stages but enough there to bring it out in the open. Still uploading files in the coming days/weeks (travelling at the moment)q
@jni @tlambert03 @maweigert @uschmidt83 @eric-czech

artefacts introduced by rotate to coverslip

noticed that rotate to coverslip introduces some "stripy" artefacts (similar to interlacing).
(even when not deconvolving).
Not sure whether this is an artefact of the volume renderer or is actually in the data.
(most likely cause would be an error in resampling)

Visualize Richardson Lucy iterations through observer callback

Use the flowdec observer callback to visualize the RL iterations.

Option 1:
Visualize while they are happening by updating an existing jupyter widget (maybe a bokeh image)

Option 2:
Record all intermediate results, visualize with an ipywidget slider to go through iterations

Fail gracefully when out of GPU memory

If there is not enough GPU memory to process the data, fail gracefully
and proceed to the next data set when running in batch mode

Depending on how large the stack is, GPU memory errors occur

when allocating the FFT plan
during building the tensorflow graph
in pyopencl during the affine transforms

Example for case 1:

019-04-16 10:31:35.730608: F tensorflow/stream_executor/cuda/cuda_fft.cc:444] failed to initialize batched cufft plan with customized allocator: 
Aborted (core dumped)

Example for case 2:

fill in

Example for case 3:

ME Stack5

pyopencl._cl.MemoryError: clEnqueueNDRangeKernel failed: MEM_OBJECT_ALLOCATION_FAILURE

Assertion error in tensorflow

During batch processing I observed the following error.

My guess is that the there is a tiff file with inconsistent dimensions in the folder (incomplete stack).
Need to check for that.

---------------------------------------------------------------------------
InvalidArgumentError                      Traceback (most recent call last)
~/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)
   1333     try:
-> 1334       return fn(*args)
   1335     except errors.OpError as e:

~/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/tensorflow/python/client/session.py in _run_fn(feed_dict, fetch_list, target_list, options, run_metadata)
   1318       return self._call_tf_sessionrun(
-> 1319           options, feed_dict, fetch_list, target_list, run_metadata)
   1320 

~/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/tensorflow/python/client/session.py in _call_tf_sessionrun(self, options, feed_dict, fetch_list, target_list, run_metadata)
   1406         self._session, options, feed_dict, fetch_list, target_list,
-> 1407         run_metadata)
   1408 

InvalidArgumentError: assertion failed: [45 512 256] [76 512 256]
	 [[{{node assert_greater_equal/Assert/Assert}} = Assert[T=[DT_INT32, DT_INT32], summarize=3, _device="/job:localhost/replica:0/task:0/device:CPU:0"](assert_greater_equal/All/_27, Shape_2/_29, Shape_3/_31)]]

During handling of the above exception, another exception occurred:

InvalidArgumentError                      Traceback (most recent call last)
<ipython-input-8-5a9a874ff0ea> in <module>
----> 1 ep.process_all()

~/Lattice_Lightsheet_Deskew_Deconv/Python/process_llsm_experiment.py in process_all(self)
    272         """
    273         for stack in tqdm.tqdm(self.ef.stacks):
--> 274             self.process_stack_subfolder(stack)

~/Lattice_Lightsheet_Deskew_Deconv/Python/process_llsm_experiment.py in process_stack_subfolder(self, stack_name)
    263             if self.do_deconv:
    264                 # determine wavelength for file and pick corresponding PSF
--> 265                 self.process_file(pathlib.Path(row.file), deskew_func, rotate_func, deconv_functions[wavelength])
    266             else:
    267                 self.process_file(pathlib.Path(row.file), deskew_func, rotate_func)

~/Lattice_Lightsheet_Deskew_Deconv/Python/process_llsm_experiment.py in process_file(self, infile, deskew_func, rotate_func, deconv_func)
    166             # write settings/metadata file to subfolder
    167         if self.do_deconv:
--> 168             deconv_raw = deconv_func(vol_raw)
    169             # TODO: write deconv settings
    170             if self.do_deconv_deskew:

~/Lattice_Lightsheet_Deskew_Deconv/Python/deconvolution.py in deconv_volume(vol, psf, deconvolver, n_iter, observer)
     21     """
     22     aq = fd_data.Acquisition(data=vol, kernel=psf)
---> 23     return deconvolver.run(aq, niter=n_iter).data
     24 
     25 

~/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/flowdec/restoration.py in run(self, acquisition, niter, session_config)
    214     def run(self, acquisition, niter, session_config=None):
    215         input_kwargs = dict(niter=niter, pad_mode=self.pad_mode, pad_min=self.pad_min, start_mode=self.start_mode)
--> 216         res = self._run(acquisition, input_kwargs, session_config=session_config)
    217         return DeconvolutionResult(res['result'], info={k: v for k, v in res.items() if k != 'result'})
    218 

~/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/flowdec/restoration.py in _run(self, acquisition, input_kwargs, session_config)
     81             data_dict = {self.graph.inputs[k]:v for k, v in acquisition.to_feed_dict().items()}
     82             args_dict = {self.graph.inputs[k]:v for k, v in input_kwargs.items() if v is not None}
---> 83             res = sess.run(self.graph.outputs, feed_dict={**data_dict, **args_dict})
     84             return res
     85 

~/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/tensorflow/python/client/session.py in run(self, fetches, feed_dict, options, run_metadata)
    927     try:
    928       result = self._run(None, fetches, feed_dict, options_ptr,
--> 929                          run_metadata_ptr)
    930       if run_metadata:
    931         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

~/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/tensorflow/python/client/session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
   1150     if final_fetches or final_targets or (handle and feed_dict_tensor):
   1151       results = self._do_run(handle, final_targets, final_fetches,
-> 1152                              feed_dict_tensor, options, run_metadata)
   1153     else:
   1154       results = []

~/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/tensorflow/python/client/session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
   1326     if handle is None:
   1327       return self._do_call(_run_fn, feeds, fetches, targets, options,
-> 1328                            run_metadata)
   1329     else:
   1330       return self._do_call(_prun_fn, handle, feeds, fetches)

~/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)
   1346           pass
   1347       message = error_interpolation.interpolate(message, self._graph)
-> 1348       raise type(e)(node_def, op, message)
   1349 
   1350   def _extend_graph(self):

InvalidArgumentError: assertion failed: [45 512 256] [76 512 256]
	 [[node assert_greater_equal/Assert/Assert (defined at /home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/flowdec/restoration.py:245)  = Assert[T=[DT_INT32, DT_INT32], summarize=3, _device="/job:localhost/replica:0/task:0/device:CPU:0"](assert_greater_equal/All/_27, Shape_2/_29, Shape_3/_31)]]

Caused by op 'assert_greater_equal/Assert/Assert', defined at:
  File "/home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/ipykernel_launcher.py", line 16, in <module>
    app.launch_new_instance()
  File "/home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/traitlets/config/application.py", line 658, in launch_instance
    app.start()
  File "/home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/ipykernel/kernelapp.py", line 505, in start
    self.io_loop.start()
  File "/home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/tornado/platform/asyncio.py", line 132, in start
    self.asyncio_loop.run_forever()
  File "/home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/asyncio/base_events.py", line 422, in run_forever
    self._run_once()
  File "/home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/asyncio/base_events.py", line 1434, in _run_once
    handle._run()
  File "/home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/asyncio/events.py", line 145, in _run
    self._callback(*self._args)
  File "/home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/tornado/ioloop.py", line 758, in _run_callback
    ret = callback()
  File "/home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/tornado/stack_context.py", line 300, in null_wrapper
    return fn(*args, **kwargs)
  File "/home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/tornado/gen.py", line 1233, in inner
    self.run()
  File "/home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run
    yielded = self.gen.send(value)
  File "/home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 357, in process_one
    yield gen.maybe_future(dispatch(*args))
  File "/home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/tornado/gen.py", line 326, in wrapper
    yielded = next(result)
  File "/home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 267, in dispatch_shell
    yield gen.maybe_future(handler(stream, idents, msg))
  File "/home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/tornado/gen.py", line 326, in wrapper
    yielded = next(result)
  File "/home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 534, in execute_request
    user_expressions, allow_stdin,
  File "/home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/tornado/gen.py", line 326, in wrapper
    yielded = next(result)
  File "/home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/ipykernel/ipkernel.py", line 294, in do_execute
    res = shell.run_cell(code, store_history=store_history, silent=silent)
  File "/home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/ipykernel/zmqshell.py", line 536, in run_cell
    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
  File "/home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2819, in run_cell
    raw_cell, store_history, silent, shell_futures)
  File "/home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2845, in _run_cell
    return runner(coro)
  File "/home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/IPython/core/async_helpers.py", line 67, in _pseudo_sync_runner
    coro.send(None)
  File "/home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3020, in run_cell_async
    interactivity=interactivity, compiler=compiler, result=result)
  File "/home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3191, in run_ast_nodes
    if (yield from self.run_code(code, result)):
  File "/home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3267, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-8-5a9a874ff0ea>", line 1, in <module>
    ep.process_all()
  File "/home/vhil0002/Lattice_Lightsheet_Deskew_Deconv/Python/process_llsm_experiment.py", line 274, in process_all
    self.process_stack_subfolder(stack)
  File "/home/vhil0002/Lattice_Lightsheet_Deskew_Deconv/Python/process_llsm_experiment.py", line 219, in process_stack_subfolder
    deconvolver = init_rl_deconvolver()
  File "/home/vhil0002/Lattice_Lightsheet_Deskew_Deconv/Python/deconvolution.py", line 7, in init_rl_deconvolver
    return tfd_restoration.RichardsonLucyDeconvolver(n_dims=3, start_mode="input").initialize()
  File "/home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/flowdec/restoration.py", line 62, in initialize
    self.graph = self._get_tf_graph()
  File "/home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/flowdec/restoration.py", line 58, in _get_tf_graph
    inputs, outputs = self._build_tf_graph()
  File "/home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/flowdec/restoration.py", line 245, in _build_tf_graph
    message='Data shape must be >= kernel shape', data=[tf.shape(datah), tf.shape(kernh)])
  File "/home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/tensorflow/python/ops/check_ops.py", line 717, in assert_greater_equal
    return control_flow_ops.Assert(condition, data, summarize=summarize)
  File "/home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/tensorflow/python/util/tf_should_use.py", line 189, in wrapped
    return _add_should_use_warning(fn(*args, **kwargs))
  File "/home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 159, in Assert
    return gen_logging_ops._assert(condition, data, summarize, name="Assert")
  File "/home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/tensorflow/python/ops/gen_logging_ops.py", line 52, in _assert
    name=name)
  File "/home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "/home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
    op_def=op_def)
  File "/home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1770, in __init__
    self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): assertion failed: [45 512 256] [76 512 256]
	 [[node assert_greater_equal/Assert/Assert (defined at /home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/flowdec/restoration.py:245)  = Assert[T=[DT_INT32, DT_INT32], summarize=3, _device="/job:localhost/replica:0/task:0/device:CPU:0"](assert_greater_equal/All/_27, Shape_2/_29, Shape_3/_31)]]

black code

https://github.com/ambv/black

bug: incompatible shapes

==========================

Processor for experiment folder:
Matt_Eramo/20190301_dlp2_Matt_image_tests_v1
Output folder:
output_mat

Processing options:
==================:
skip existing output files: False
Perform max intensity projections: False
max intensity projection output: montage
Perform deskew: False
Rotate to coverslip: False
Perform deconvolution: True
Devonvolution backend: flowdec
Number of iterations for deconvolution: 20
Perform deskew after deconvolution: False
Rotate to coverslip after deconvolution: True
Background subtraction value for camera: 100
processing stack number 5
/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/lls_dd-2019.3a1-py3.6.egg/lls_dd/process_llsm_experiment.py:385: UserWarning: Fix write_func stuff to include compression and units
  warnings.warn("Fix write_func stuff to include compression and units")
/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/lls_dd-2019.3a1-py3.6.egg/lls_dd/process_llsm_experiment.py:463: UserWarning: more than one PSF found. Taking first one
  warnings.warn(f"more than one PSF found. Taking first one")
  0%|                                                                                                     | 0/100 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
    return fn(*args)
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [105,840,625] vs. [105,840,624]
	 [[{{node while/truediv}} = RealDiv[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](while/zeros_like/Enter, while/irfft3d)]]
	 [[{{node result/_149}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_591_result", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/vhil0002/anaconda3/envs/newllsm/bin/lls_dd", line 11, in <module>
    load_entry_point('lls-dd==2019.3a1', 'console_scripts', 'lls_dd')()
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/lls_dd-2019.3a1-py3.6.egg/lls_dd/cmdline.py", line 156, in process
    ep.process_stack_subfolder(processcmd.ef.stacks[int(number)])
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/lls_dd-2019.3a1-py3.6.egg/lls_dd/process_llsm_experiment.py", line 517, in process_stack_subfolder
    deconv_functions[wavelength],
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/lls_dd-2019.3a1-py3.6.egg/lls_dd/process_llsm_experiment.py", line 341, in process_file
    deconv_raw = deconv_func(vol_raw)
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/lls_dd-2019.3a1-py3.6.egg/lls_dd/deconvolution.py", line 62, in deconv_volume
    result = deconvolver.run(aq, niter=n_iter, session_config=config)
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/flowdec-1.0.7-py3.6.egg/flowdec/restoration.py", line 216, in run
    res = self._run(acquisition, input_kwargs, session_config=session_config)
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/flowdec-1.0.7-py3.6.egg/flowdec/restoration.py", line 83, in _run
    res = sess.run(self.graph.outputs, feed_dict={**data_dict, **args_dict})
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
    run_metadata)
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [105,840,625] vs. [105,840,624]
	 [[node while/truediv (defined at /home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/flowdec-1.0.7-py3.6.egg/flowdec/restoration.py:301)  = RealDiv[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](while/zeros_like/Enter, while/irfft3d)]]
	 [[{{node result/_149}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_591_result", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Caused by op 'while/truediv', defined at:
  File "/home/vhil0002/anaconda3/envs/newllsm/bin/lls_dd", line 11, in <module>
    load_entry_point('lls-dd==2019.3a1', 'console_scripts', 'lls_dd')()
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/lls_dd-2019.3a1-py3.6.egg/lls_dd/cmdline.py", line 156, in process
    ep.process_stack_subfolder(processcmd.ef.stacks[int(number)])
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/lls_dd-2019.3a1-py3.6.egg/lls_dd/process_llsm_experiment.py", line 495, in process_stack_subfolder
    pad_mode=self.flowdec_pad_mode, pad_min=pad_min, real_domain_fft=True
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/lls_dd-2019.3a1-py3.6.egg/lls_dd/deconvolution.py", line 19, in init_rl_deconvolver
    n_dims=3, start_mode="input", **kwargs
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/flowdec-1.0.7-py3.6.egg/flowdec/restoration.py", line 62, in initialize
    self.graph = self._get_tf_graph()
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/flowdec-1.0.7-py3.6.egg/flowdec/restoration.py", line 58, in _get_tf_graph
    inputs, outputs = self._build_tf_graph()
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/flowdec-1.0.7-py3.6.egg/flowdec/restoration.py", line 316, in _build_tf_graph
    result = tf.while_loop(cond, body, [1, decon], parallel_iterations=1)[1]
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3291, in while_loop
    return_same_structure)
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3004, in BuildLoop
    pred, body, original_loop_vars, loop_vars, shape_invariants)
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2939, in _BuildLoop
    body_result = body(*packed_vars_for_body)
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/flowdec-1.0.7-py3.6.egg/flowdec/restoration.py", line 301, in body
    blur1 = tf.where(conv1 < self.epsilon, tf.zeros_like(datat), datat / conv1, name='blur1')
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 866, in binary_op_wrapper
    return func(x, y, name=name)
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 974, in _truediv_python3
    return gen_math_ops.real_div(x, y, name=name)
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/tensorflow/python/ops/gen_math_ops.py", line 5989, in real_div
    "RealDiv", x=x, y=y, name=name)
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
    op_def=op_def)
  File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1770, in __init__
    self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Incompatible shapes: [105,840,625] vs. [105,840,624]
	 [[node while/truediv (defined at /home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/flowdec-1.0.7-py3.6.egg/flowdec/restoration.py:301)  = RealDiv[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](while/zeros_like/Enter, while/irfft3d)]]
	 [[{{node result/_149}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_591_result", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

CPU version — ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'

The Jupyter notebook (VolkerH/Lattice_Lightsheet_Deskew_Deconv/01_Lattice_Light_Sheet_Deconvolution.ipynb) produces the error when processing the following cell:

from lls_dd.transform_helpers import *
from lls_dd.transforms import *
import numpy as np
import tifffile
## Depending on whether you want to use ndimage or gputools, uncomment one of the following
## two lines
from scipy.ndimage import affine_transform
#from lls_dd.gputools_wrapper import affine_transform_gputools as affine_transform
from numpy.linalg import inv
from skimage.feature import peak_local_max
from skimage.filters import gaussian
import tensorflow as tf
from flowdec import restoration as tfd_restoration
from flowdec import data as fd_data
import matplotlib.pyplot as plt

The following output were produced:

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
ImportError: numpy.core.multiarray failed to import

The above exception was the direct cause of the following exception:

SystemError                               Traceback (most recent call last)
/Users/fdjutant/miniconda/envs/llsdd/lib/python3.6/importlib/_bootstrap.py in _find_and_load(name, import_)

SystemError: <class '_frozen_importlib._ModuleLockManager'> returned a result with an error set

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
ImportError: numpy.core._multiarray_umath failed to import

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
ImportError: numpy.core.umath failed to import

The installation process run smoothly following the instruction for environment-no-CUDA.yml. It seems that the problem appears when importing tensorflow. The notebook 00_Lattice_Light_Sheet_Deskew.ipynb run without any issue.

psf generation - change to two stage affine to avoid artefacts

psf generation currently still does scaling and skewing in a single affine transformation.
This leads to interpolation between voxel with different physical spacing (see discussion here #22 ). This has already been fixed for rotate to coverlip but not for the psf generation.

volkerh / lattice_lightsheet_deskew_deconv Goto Github PK

lattice_lightsheet_deskew_deconv's People

Contributors

Stargazers

Watchers

Forkers

lattice_lightsheet_deskew_deconv's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs