Issue running poismf about poismf

[david-cortes]

Ok, could reproduce it now, issue is with the compilation linker step which for some reason cannot link to the libmkl_rt.so file even though it exists, and the problem is not limited to python.

Will investigate about it to see how to correct it.

[david-cortes]

Thanks for the bug report. Could you post the full output of:

python setup.py build_ext --inplace --force

python setup.py install --force

[sara-02]

For sudo python3 setup.py build_ext --inplace --force

running build_ext
Installation: Using BLAS library found in:
/home/sarah/.local/lib/libmkl_rt.so


cythoning poismf/poismf_c_wrapper.pyx to poismf/poismf_c_wrapper.c
/home/sarah/.local/lib/python3.6/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /home/sarah/poismf/poismf/poismf_c_wrapper.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)
building 'poismf.poismf_c_wrapper' extension
C compiler: x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC

creating build/temp.linux-x86_64-3.6
creating build/temp.linux-x86_64-3.6/poismf
creating build/temp.linux-x86_64-3.6/src
compile options: '-D_FOR_PYTHON -DHAS_MKL -I/home/sarah/.local/lib/python3.6/site-packages/numpy/core/include -I/home/sarah/.local/include -I/home/sarah/.local/lib/python3.6/site-packages/findblas/ -I/usr/include/python3.6m -c'
extra options: '-O2 -fopenmp -march=native -std=c99'
x86_64-linux-gnu-gcc: poismf/poismf_c_wrapper.c
x86_64-linux-gnu-gcc: src/nonnegcg.c
src/nonnegcg.c: In function ‘minimize_nonneg_cg’:
src/nonnegcg.c:271:72: warning: ‘grad_prev’ may be used uninitialized in this function [-Wmaybe-uninitialized]
     beta += ( x[i] <= 0 )? 0 : grad_curr[i] * (grad_curr[i] - grad_prev[i]);
                                                               ~~~~~~~~~^~~
src/nonnegcg.c:270:62: warning: ‘direction_prev’ may be used uninitialized in this function [-Wmaybe-uninitialized]
     theta += ( x[i] <= 0 )? 0 : grad_curr[i] * direction_prev[i];
                                                ~~~~~~~~~~~~~~^~~
src/nonnegcg.c:274:9: warning: ‘grad_prev_norm_sq’ may be used uninitialized in this function [-Wmaybe-uninitialized]
    beta /= grad_prev_norm_sq;
    ~~~~~^~~~~~~~~~~~~~~~~~~~
In file included from /home/sarah/.local/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1830:0,
                 from /home/sarah/.local/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                 from /home/sarah/.local/lib/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from poismf/poismf_c_wrapper.c:598:
/home/sarah/.local/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it with " \
  ^~~~~~~
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.6/poismf/poismf_c_wrapper.o build/temp.linux-x86_64-3.6/src/nonnegcg.o -o /home/sarah/poismf/poismf/poismf_c_wrapper.cpython-36m-x86_64-linux-gnu.so -fopenmp -L/home/sarah/.local/lib -l:libmkl_rt.so

For sudo python3 setup.py install --force

running install
running bdist_egg
running egg_info
writing poismf.egg-info/PKG-INFO
writing dependency_links to poismf.egg-info/dependency_links.txt
writing requirements to poismf.egg-info/requires.txt
writing top-level names to poismf.egg-info/top_level.txt
reading manifest file 'poismf.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'poismf.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build/lib.linux-x86_64-3.6
creating build/lib.linux-x86_64-3.6/poismf
copying poismf/__init__.py -> build/lib.linux-x86_64-3.6/poismf
running build_ext
Installation: Using BLAS library found in:
/home/sarah/.local/lib/libmkl_rt.so


skipping 'poismf/poismf_c_wrapper.c' Cython extension (up-to-date)
building 'poismf.poismf_c_wrapper' extension
C compiler: x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC

compile options: '-D_FOR_PYTHON -DHAS_MKL -I/home/sarah/.local/lib/python3.6/site-packages/numpy/core/include -I/home/sarah/.local/include -I/home/sarah/.local/lib/python3.6/site-packages/findblas/ -I/usr/include/python3.6m -c'
extra options: '-O2 -fopenmp -march=native -std=c99'
x86_64-linux-gnu-gcc: poismf/poismf_c_wrapper.c
x86_64-linux-gnu-gcc: src/nonnegcg.c
src/nonnegcg.c: In function ‘minimize_nonneg_cg’:
src/nonnegcg.c:271:72: warning: ‘grad_prev’ may be used uninitialized in this function [-Wmaybe-uninitialized]
     beta += ( x[i] <= 0 )? 0 : grad_curr[i] * (grad_curr[i] - grad_prev[i]);
                                                               ~~~~~~~~~^~~
src/nonnegcg.c:270:62: warning: ‘direction_prev’ may be used uninitialized in this function [-Wmaybe-uninitialized]
     theta += ( x[i] <= 0 )? 0 : grad_curr[i] * direction_prev[i];
                                                ~~~~~~~~~~~~~~^~~
src/nonnegcg.c:274:9: warning: ‘grad_prev_norm_sq’ may be used uninitialized in this function [-Wmaybe-uninitialized]
    beta /= grad_prev_norm_sq;
    ~~~~~^~~~~~~~~~~~~~~~~~~~
In file included from /home/sarah/.local/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1830:0,
                 from /home/sarah/.local/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                 from /home/sarah/.local/lib/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from poismf/poismf_c_wrapper.c:598:
/home/sarah/.local/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it with " \
  ^~~~~~~
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.6/poismf/poismf_c_wrapper.o build/temp.linux-x86_64-3.6/src/nonnegcg.o -o build/lib.linux-x86_64-3.6/poismf/poismf_c_wrapper.cpython-36m-x86_64-linux-gnu.so -fopenmp -L/home/sarah/.local/lib -l:libmkl_rt.so
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/poismf
copying build/lib.linux-x86_64-3.6/poismf/__init__.py -> build/bdist.linux-x86_64/egg/poismf
copying build/lib.linux-x86_64-3.6/poismf/poismf_c_wrapper.cpython-36m-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/egg/poismf
byte-compiling build/bdist.linux-x86_64/egg/poismf/__init__.py to __init__.cpython-36.pyc
creating stub loader for poismf/poismf_c_wrapper.cpython-36m-x86_64-linux-gnu.so
byte-compiling build/bdist.linux-x86_64/egg/poismf/poismf_c_wrapper.py to poismf_c_wrapper.cpython-36.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying poismf.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying poismf.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying poismf.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying poismf.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying poismf.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt
zip_safe flag not set; analyzing archive contents...
poismf.__pycache__.poismf_c_wrapper.cpython-36: module references __file__
creating 'dist/poismf-0.1.4-py3.6-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing poismf-0.1.4-py3.6-linux-x86_64.egg
removing '/usr/local/lib/python3.6/dist-packages/poismf-0.1.4-py3.6-linux-x86_64.egg' (and everything under it)
creating /usr/local/lib/python3.6/dist-packages/poismf-0.1.4-py3.6-linux-x86_64.egg
Extracting poismf-0.1.4-py3.6-linux-x86_64.egg to /usr/local/lib/python3.6/dist-packages
poismf 0.1.4 is already the active version in easy-install.pth

Installed /usr/local/lib/python3.6/dist-packages/poismf-0.1.4-py3.6-linux-x86_64.egg
Processing dependencies for poismf==0.1.4
Searching for findblas==0.1.3.4
Best match: findblas 0.1.3.4
Adding findblas 0.1.3.4 to easy-install.pth file

Using /home/sarah/.local/lib/python3.6/site-packages
Searching for Cython==0.29.14
Best match: Cython 0.29.14
Adding Cython 0.29.14 to easy-install.pth file
Installing cygdb script to /usr/local/bin
Installing cython script to /usr/local/bin
Installing cythonize script to /usr/local/bin

Using /home/sarah/.local/lib/python3.6/site-packages
Searching for pandas==0.25.3
Best match: pandas 0.25.3
Adding pandas 0.25.3 to easy-install.pth file

Using /home/sarah/.local/lib/python3.6/site-packages
Searching for numpy==1.17.4
Best match: numpy 1.17.4
Adding numpy 1.17.4 to easy-install.pth file
Installing f2py script to /usr/local/bin
Installing f2py3 script to /usr/local/bin
Installing f2py3.6 script to /usr/local/bin

Using /home/sarah/.local/lib/python3.6/site-packages
Searching for pytz==2019.3
Best match: pytz 2019.3
Adding pytz 2019.3 to easy-install.pth file

Using /home/sarah/.local/lib/python3.6/site-packages
Searching for python-dateutil==2.8.1
Best match: python-dateutil 2.8.1
Adding python-dateutil 2.8.1 to easy-install.pth file

Using /home/sarah/.local/lib/python3.6/site-packages
Searching for six==1.13.0
Best match: six 1.13.0
Adding six 1.13.0 to easy-install.pth file

Using /home/sarah/.local/lib/python3.6/site-packages
Finished processing dependencies for poismf==0.1.4

[david-cortes]

Strange case. Looks from the output that it compiled correctly and generated the following file in the first case (build_ext):

/home/sarah/poismf/poismf/poismf_c_wrapper.cpython-36m-x86_64-linux-gnu.so

If you do indeed see that file, then it should be possible to import it by running python in this folder:

/home/sarah/poismf

(e.g. cd /home/sarah/poismf && python, then inside python from poismf import PoisMF)

If it still complains that the file is not there, only reason I can think of is that somehow you have different python versions under python, python3, sudo python, in which case you'd have to use the same command for installing and running it.

From the second one (install), issue is that it gets installed in a system folder, but then you run python locally in that same folder, and it sees that there's a local module poismf, so it imports that module instead of the system one, but the compiled file is not there as it was generated in the installation folder.

Solution in the 2nd case is to launch python in a different folder, or use build_ext if you want to launch it from the same folder. Also I don't think you should be using sudo for that, as otherwise if you then run python as a user it will again try to import packages from a different folder.

[sara-02]

Hey so for the cd /home/sarah/poismf && python command upon importing I am getting another error

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/sarah/poismf/poismf/__init__.py", line 4, in <module>
    from .poismf_c_wrapper import run_pgd, _predict_multiple, _predict_factors
ImportError: libmkl_rt.so: cannot open shared object file: No such file or directory

I used python3 in all of my commands for installing and running, so I hope that should not be a problem.

[sara-02]

For the second case, I opened a new terminal and tried the following, still getting the same error as in the above case

python3
Python 3.6.8 (default, Oct  7 2019, 12:59:55) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from poismf import PoisMF
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/poismf-0.1.4-py3.6-linux-x86_64.egg/poismf/__init__.py", line 4, in <module>
    from .poismf_c_wrapper import run_pgd, _predict_multiple, _predict_factors
ImportError: libmkl_rt.so: cannot open shared object file: No such file or directory
>>> 

The reason I had to use sudo was the commnd python setup.py install was giving an permission denied error.

[sara-02]

For sanity testing, I tried running it on R, and it seems to be working without any error.

head(predict(model, 1)) ## predict the whole row 1
[1] 3.248912e-08 2.351517e-08 2.005383e-08 3.683958e-08 2.652832e-08
[6] 3.473318e-08
> 
> #all predictions for new row/user/doc
> head(predict(model, data.frame(col_ix = c(1,2,3), count = c(4,5,6)) ))
[1] 8.896338e-06 6.439046e-06 5.491243e-06 1.008760e-05 7.264121e-06
[6] 9.510819e-06

[david-cortes]

I see now: thing is, this package depends on some BLAS library as backend for linear algebra operations. In your case, you have a local install of MKL which gets chosen as the library to link to. In ubuntu I think R will by default install OpenBLAS as backend.

Now, the BLAS library to link to is determined through package findblas, which will prefer MKL over OpenBLAS if it can find it. But for some reason, after it links to the file /home/sarah/.local/lib/libmkl_rt.so, it then cannot find it (does that file exist in your computer?).

I have no idea how that happened (gcc shows the correct arguments: -L/home/sarah/.local/lib -l:libmkl_rt.so), and no idea how to solve it (this is managed by setuptools). Only possible solutions I can think of are:

• Pre-load the library: LD_PRELOAD=/home/sarah/.local/lib/libmkl_rt.so python, maybe LD_LIBRARY_PATH=/home/sarah/.local/lib python, or both at the same time.
• Force a relinkage to the libmkl_rt.so in your local folder (complicated), or to the same libblas.so used by R (probably something like /usr/lib/x86_64-linux-gnu/openblas/libblas.so).
• Install openblas (sudo aptitude install libopenblas-dev) if not already installed, then in setup.py change this line:

from findblas.distutils import build_ext_with_blas

to this:

from Cython.Distutils import build_ext as build_ext_with_blas

then in the same file

e.extra_link_args += ['-fopenmp']

to

e.extra_link_args += ['-fopenmp', '-lcblas']

EDIT: still wrong for the last one, would then need editing the file pgd.c by changing #include "findblas.h" to #include "cblas.h", and add to the setup file the path in which the headers are, which is yet another source of issues.

---

Also just tested it in a default conda install in debian 10, seemed to work fine there.

[david-cortes]

BTW: if you see that this file does not exist: /home/sarah/.local/lib/libmkl_rt.so
Then it's an issue with this other package: https://www.github.com/david-cortes/findblas
If the file is not there, would appreciate it if you could open an issue in that other package describing how you installed python and numpy/scipy.

[patrickfletcher]

I am having the same error:
from poismf import PoisMF File "/home/patrick/.local/lib/python3.6/site-packages/poismf/__init__.py", line 4, in <module> from .poismf_c_wrapper import run_pgd, _predict_multiple, _predict_factors ImportError: libmkl_rt.so: cannot open shared object file: No such file or directory

I'm using Linux Mint 18.3. Installed mkl-devel using:
python3 -m pip install mkl-devel --user
then poismf similarly. libmkl_rt.so exists in the ./local/lib folder, and the LD_PRELOAD/LD_LIBRARY_PATH suggestion above didn't change the error message.

[david-cortes]

@patrickfletcher : Thanks for the report. Will better try it under Ubuntu now.

In the meantime, could you please find the generated .so file (something like poismf_c_wrapper.cpython-37m-x86_64-linux-gnu.so, under the same path as __init__.py (/home/patrick/.local/lib/python3.6/site-packages/poismf/)), run ldd on it (linux command) – e.g.:

ldd /home/patrick/.local/lib/python3.6/site-packages/poismf/poismf_c_wrapper*.so

and paste the full output in here.

[patrickfletcher]

ldd /home/patrick/.local/lib/python3.6/site-packages/poismf/poismf_c_wrapper*.so linux-vdso.so.1 (0x00007fff981a2000) libmkl_rt.so => not found libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007fd342b66000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd342775000) /lib64/ld-linux-x86-64.so.2 (0x00007fd342fb0000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd342571000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd342352000)

[david-cortes]

@patrickfletcher : Another thing: how did you install NumPy? E.g.: conda, pip, apt-get/aptitude, compiled from source, etc.

[patrickfletcher]

Yes, I have numpy installed: numpy==1.18.1 - I installed via
python3 -m pip -U numpy --user

[patrickfletcher]

Thanks for your prompt response! :)

[david-cortes]

I wasn’t able to figure out why it fails to install. Only solution I found is to either use a Python install from conda, or to uninstall the mkl-devel and numpy from pip and install instead the ones from system packages, or to use openblas.

The problem nevertheless also seems to affect NumPy somehow (installs but is not able to find MKL), so I opened an issue there: numpy/numpy#15372

[patrickfletcher]

OK, interesting. I'll try the other solutions when I get a chance. Thanks!

[david-cortes]

@patrickfletcher : I've pushed an update to package findblas (dependency of this) which should solve the issue. Could you please update to the latest version (python3 -m pip install findblas==0.1.4) and then confirm if this package installs and loads correctly? (You'll need to run pip3 uninstall poismf first if you installed)

[patrickfletcher]

It works now! I did as you said - upgraded findblas, then I rebuilt poismf (not use the cached whl).

Thanks!