CPU version — ModuleNotFoundError: No module named 'numpy.core._multiarray_umath' about lattice_lightsheet_deskew_deconv

[VolkerH]

Hmmm .... just googled the error message and it seems that this is some interaction between tensorflow and numpy, similar to what is reported here
numpy/numpy#12977

I will try and reproduce but don't have time today.
In the environment-no-CUDA.yml I seem to have pinned the numpy version to 1.15.* but not the tensorflow version, so probably the most recent version of tensorflow doesn't play nicely with the older numpy. Can you try removing the version pin
or alternatively try upgrading numpy?

[fdjutant]

I tried to upgrade the numpy to the most recent one. The conflict seems to be worse since the Python is 3.6. Please see below. Thanks, Volker!

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
/Users/fdjutant/miniconda/envs/llsdd/lib/python3.6/site-packages/numpy/core/__init__.py in <module>()
     21 try:
---> 22     from . import multiarray
     23 except ImportError as exc:

/Users/fdjutant/miniconda/envs/llsdd/lib/python3.6/site-packages/numpy/core/multiarray.py in <module>()
     11 
---> 12 from . import overrides
     13 from . import _multiarray_umath

/Users/fdjutant/miniconda/envs/llsdd/lib/python3.6/site-packages/numpy/core/overrides.py in <module>()
      6 
----> 7 from numpy.core._multiarray_umath import (
      8     add_docstring, implement_array_function, _get_implementing_args)

ImportError: dlopen(/Users/fdjutant/miniconda/envs/llsdd/lib/python3.6/site-packages/numpy/core/_multiarray_umath.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libopenblas.dylib
  Referenced from: /Users/fdjutant/miniconda/envs/llsdd/lib/python3.6/site-packages/numpy/core/_multiarray_umath.cpython-36m-darwin.so
  Reason: image not found

During handling of the above exception, another exception occurred:

ImportError                               Traceback (most recent call last)
<ipython-input-1-32f6fcd52b11> in <module>()
----> 1 from lls_dd.transform_helpers import *
      2 from lls_dd.transforms import *
      3 import numpy as np
      4 import tifffile
      5 ## Depending on whether you want to use ndimage or gputools, uncomment one of the following

/Users/fdjutant/miniconda/envs/llsdd/lib/python3.6/site-packages/lls_dd-2020.7a1-py3.6.egg/lls_dd/__init__.py in <module>()
----> 1 from .process_llsm_experiment import ExperimentProcessor
      2 from .experiment_folder import Experimentfolder

/Users/fdjutant/miniconda/envs/llsdd/lib/python3.6/site-packages/lls_dd-2020.7a1-py3.6.egg/lls_dd/process_llsm_experiment.py in <module>()
      2 import logging
      3 import pathlib
----> 4 import tifffile
      5 import tqdm
      6 import numpy as np

/Users/fdjutant/miniconda/envs/llsdd/lib/python3.6/site-packages/tifffile/__init__.py in <module>()
      1 # tifffile/__init__.py
      2 
----> 3 from .tifffile import __doc__, __all__, __version__, main
      4 from .tifffile import *

/Users/fdjutant/miniconda/envs/llsdd/lib/python3.6/site-packages/tifffile/tifffile.py in <module>()
    603 from concurrent.futures import ThreadPoolExecutor
    604 
--> 605 import numpy
    606 
    607 try:

/Users/fdjutant/miniconda/envs/llsdd/lib/python3.6/site-packages/numpy/__init__.py in <module>()
    138     from . import _distributor_init
    139 
--> 140     from . import core
    141     from .core import *
    142     from . import compat

/Users/fdjutant/miniconda/envs/llsdd/lib/python3.6/site-packages/numpy/core/__init__.py in <module>()
     46 """ % (sys.version_info[0], sys.version_info[1], sys.executable,
     47         __version__, exc)
---> 48     raise ImportError(msg)
     49 finally:
     50     for envkey in env_added:

ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.6 from "/Users/fdjutant/miniconda/envs/llsdd/bin/python"
  * The NumPy version is: "1.19.2"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: dlopen(/Users/fdjutant/miniconda/envs/llsdd/lib/python3.6/site-packages/numpy/core/_multiarray_umath.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libopenblas.dylib
  Referenced from: /Users/fdjutant/miniconda/envs/llsdd/lib/python3.6/site-packages/numpy/core/_multiarray_umath.cpython-36m-darwin.so
  Reason: image not found

[VolkerH]

Hi,

I just set up a new environment from my .yml file and I can't reproduce the error locally.

The error message after your update comes from the same module but is a bit more enlightenting.
It seems the library image for libopenblas is missing (see below).
I wonder whether it is a Mac OS specific issue. What version of OS X do you have?
Unfortunately, I currently don't have access to a Mac.

ImportError: dlopen(/Users/fdjutant/miniconda/envs/llsdd/lib/python3.6/site-packages/numpy/core/_multiarray_umath.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libopenblas.dylib
  Referenced from: /Users/fdjutant/miniconda/envs/llsdd/lib/python3.6/site-packages/numpy/core/_multiarray_umath.cpython-36m-darwin.so
  Reason: image not found

Normally conda should take care of putting the libraries in the right place.
Can you edit the .yml file and remove conda-forge from the channels and set up a clean environment?
IIRC, the conda-forge channel provides optimized versions of some numerical libraries, maybe that has something to do with it.
I am just speculating here as I am not that familiar with exact resolution of library imports on the different platforms.
Maybe also check your environment whether you have some LD_LIBRARY_PATH that might have priority over the one used by conda.

[fdjutant]

Hi,

Sorry for the very late response. The conflict still persists after removing conda-forge from the environment-no-CUDA.yml.

I managed to use a Windows computer in the lab, and that works now. I will just be using Windows. Thank you so much for the help and the code!

[VolkerH]

Hi @fdjutant,

thanks for letting me know that it works for you on Windows. Apologies that I can't help further with Mac at this time.

Any feedback is welcome, in particular if you end up using this regularly.

[fdjutant]

Hi @VolkerH,

I forgot to mention that the numpy version in the environment-no-cuda.yml file needs to be removed in order to make it work. Thanks again!