Architecture not well-suited for multiprocessing due to pickling limitations about lattice_lightsheet_deskew_deconv

[VolkerH]

So the culprit with the SwigPyObjects is in the deconvolution function, most likely tensorflow. If I leave out deconvolution there is one more problem: the lock doesn't work as I intented. After removing the lock I can do parallel deskew/rotate.

[VolkerH]

cloesed by accident, reopening

[jni]

@VolkerH multiprocessing is very primitive, and for most functions these days that use NumPy you should use dask or at least threaded. Here's a PR where I parallelised a PyTorch inference pipeline using dask and threading locks to prevent GPU oversubscription:

https://github.com/saalfeldlab/simpleference/pull/3/files

dask uses cloudpickle which is much more powerful than pickle, although swigpyobjects sounds hard to pickle for sure.

[VolkerH]

@jni thanks a lot. I will see whether cloudpickle in dask helps. Probably will let this sit for a few days though.
I shoule be able to avoid pickeling the SwigPyObjects as the tensorflow part is the one that I don't want to run in parallel anyway. However, I will have to restructure my code and break up process_file into its different parts (reading, deskewing, deconvolving and writing).

[VolkerH]

there is also some pyopencl stuff that can't be pickled.
__
for obj in iterable:
File "/home/vhil0002/.conda/pkgs/cache/su62_scratch/volker_conda/newllsm/lib/python3.6/site-packages/multiprocess/pool.py", line 735, in next
raise value
multiprocess.pool.MaybeEncodingError: Error sending result: '<multiprocess.pool.ExceptionWithTraceback object at 0x7fd90fd115f8>'. Reason: 'TypeError("can't pickle pyopencl._cl._ErrorRecord objects",)'

[VolkerH]

Ok ... I was impatient. Rather than testing dask I just did a quick test to see whether cloudpickle can deal with the processing functions:

        print("Cloudpickling deskew_function")
        cpdsf=cloudpickle.dumps(deskew_func)
        print("Cloudpickling rotate_func")
        cprotf=cloudpickle.dumps(rotate_func)
        print("Cloudpickling deconv_functions")
        cpdcf=cloudpickle.dumps(deconv_functions)

Output was:

Cloudpickling deskew_function
Cloudpickling rotate_func
Cloudpickling deconv_functions
Traceback (most recent call last):
  File "C:\Users\Volker\Anaconda3\envs\spimenv\Scripts\lls_dd-script.py", line 11, in <module>
    load_entry_point('lls-dd==2019.3a1', 'console_scripts', 'lls_dd')()
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\site-packages\click\core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\site-packages\click\core.py", line 717, in main
    rv = self.invoke(ctx)
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\site-packages\click\core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\site-packages\click\core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\site-packages\click\core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\site-packages\click\decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\site-packages\click\core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\site-packages\lls_dd-2019.3a1-py3.6.egg\lls_dd\cmdline.py", line 151, in process
    ep.process_stack_subfolder(processcmd.ef.stacks[int(number)])
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\site-packages\lls_dd-2019.3a1-py3.6.egg\lls_dd\process_llsm_experiment.py", line 525, in process_stack_subfolder
    cpdcf=cloudpickle.dumps(deconv_functions)
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\site-packages\cloudpickle\cloudpickle.py", line 961, in dumps
    cp.dump(obj)
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\site-packages\cloudpickle\cloudpickle.py", line 267, in dump
    return Pickler.dump(self, obj)
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\pickle.py", line 409, in dump
    self.save(obj)
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\pickle.py", line 631, in save_reduce
    self._batch_setitems(dictitems)
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\pickle.py", line 847, in _batch_setitems
    save(v)
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\pickle.py", line 634, in save_reduce
    save(state)
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\pickle.py", line 751, in save_tuple
    save(element)
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\pickle.py", line 847, in _batch_setitems
    save(v)
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\pickle.py", line 634, in save_reduce
    save(state)
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\pickle.py", line 847, in _batch_setitems
    save(v)
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\pickle.py", line 634, in save_reduce
    save(state)
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\pickle.py", line 847, in _batch_setitems
    save(v)
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\pickle.py", line 634, in save_reduce
    save(state)
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\pickle.py", line 847, in _batch_setitems
    save(v)
  File "C:\Users\Volker\Anaconda3\envs\spimenv\lib\pickle.py", line 496, in save
    rv = reduce(self.proto)
TypeError: can't pickle _thread.RLock objects

So in contrast to dill, cloudpickle fails at serializing _thread.RLock, presumably also from tensorflow. Not sure whether this happens before or after serializing the swig stuff.

[jni]

Hmm. I'm not sure whether:

• PyTorch has better pickle compatibility, or
• Somehow dask doesn't need to pickle that function because workers can import directly?

[VolkerH]

Hmmm. Doesn't really matter as the GPU computation is the part I don't want to run in multiple processes anyway.
I have decided to use the pattern implemented here, adatped to either pathos or dask so that class methods can be serialized:
https://pypi.org/project/multiprocessing-generator/
It is exacyly what I want:

Up to 100 elements ahead of what is consumed will be fetched by the generator in the background, which is useful when the producer and the consumer do not use the same resources (for instance network vs. CPU).

Currently process_file takes a path and reads the file. I will seperate out the file reading and change it into process_volume. Then the file reading can be handled by such a prefetching generator. For file writing I can trivially start a new process for each file.

[VolkerH]

closing this until I have more time to work on it