linking against imported library about cmake-build-extension

[diegoferigo]

Hi @vincentberenz, the link is not working, can you please replicate your SO question in this issue? I found it in the google cache for the moment.

First, are you sure that the license of the *.so library allows being redistributed in your PyPI package? This is essential to know in order to avoid legal problems. My reply below assumes this is allowed.

Providing a working solution is not trivial since there are many subtleties of dynamic linking (rpath in primis, see here and here) that would take long to explain. Furthermore, packaging for PyPI disrupts pretty much everything since tools like pypa/auditwheel change exactly the linking configuration of CMake in order to allow vendoring third-party libraries.

Without explaining too much, in my opinion you should proceed as follows:

• Create a pure CMake project of your bindings that allows being installed in your system (i.e. it finds the manifacturer .so wherever it is on disk, is able to compile your code, is able to install your .so so that you can use locally your bindings). All done without pip.
• Add to your project setuptools support though this repo. It should not affect the CMake workflow. You would just find the shared library of your bindings packaged in a wheel. Note that this process does not touch the manifacturer shared library, that should be found in the system by the dynamic loader (use ldd on your bindings library to understand if it was found).
• Now, the wheel generated by this process could work on your system, but it does not in other systems, since it does not include the manifacturer library. Luckily, you can use auditwheel that will detect that something is missing from the wheel and it fixes it by 1) vendoring the missing .so file, and 2) updating the RPATH of your bindings library.

Yes, it could be confusing if this is all new to you, but I tried to outline steps that could be followed one by one. Make sure that all of them are properly working before passing to the next one.

Few notes:

• I noticed that you keep the manifacturer libdrivers.so in the sources tree. This is fine (as long as you can redistribute it), but in order to use it from there, you either have to install it in a folder known to the dynamic loaded (/usr/local/lib maybe?) or add your sources dir to LD_LIBRARY_PATH (dirty option but ok for testing).
• If you do this, at least locally it should work. Note that currently the relative path ../../lib/libdrivers.so should be met.

[vincentberenz]

Hi @diegoferigo, thanks so much for your kind answer.

I updated my issue to a new question, with more details: https://stackoverflow.com/questions/73053313/cmake-specifying-the-install-relative-path-to-imported-shared-library (also copied below for convenience). I have attempted to play with the rpath.

For now I put the drivers.so file in a "proper" folder (/usr/local/lib) and find it in cmake, and things work fine. But I would like final users not to have to download / copy the file, especially if they do not have sudo rights. But I'll keep this as plan B, if I remain stuck.

I also came accross auditwheel. I did not look carefully at it yet. I will, but ideally I'd like things to get properly installed with only pip install .

Working on a pure cmake first makes a lot of sense, I will work on it.

Thanks again for your help. I realize this is more an issue with me not knowing setuptools/cmake enough, rather than an issue with cmake-build-extension (but I hope this could be useful to other users as well).

stackoverlflow question:

---

I would like to link against an imported shared library:

add_library(drivers SHARED IMPORTED)
set_target_properties(
  drivers
  PROPERTIES IMPORTED_LOCATION
  ${CMAKE_CURRENT_SOURCE_DIR}/lib/libdrivers.so
  )

The library I would like to link against is a pybind11 binding library. I try to setup rpath so that libdrivers.so will be found in the same directory as the bindings library.

pybind11_add_module(pybind11_bindings MODULE srcpy/bindings.cpp)
set_target_properties(pybind11_bindings PROPERTIES
    LIBRARY_OUTPUT_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/myproject
    OUTPUT_NAME "bindings"
    SKIP_BUILD_RPATH FALSE
     BUILD_WITH_INSTALL_RPATH TRUE
  )
target_link_libraries(pybind11_bindings
  PRIVATE drivers
  )
set_target_properties(pybind11_bindings PROPERTIES 
INSTALL_RPATH "\$ORIGIN")
set_target_properties(pybind11_bindings PROPERTIES INSTALL_RPATH_USE_LINK_PATH TRUE)

Things are installed:

install(IMPORTED_RUNTIME_ARTIFACTS drivers
  LIBRARY DESTINATION ${PMYROJECT_INSTALL_PREFIX}
  COMPONENT bindings
  )

install(
    TARGETS pybind11_bindings
    COMPONENT bindings
    LIBRARY DESTINATION ${MYPROJECT_INSTALL_PREFIX}
    ARCHIVE DESTINATION ${MYPROJECT_INSTALL_PREFIX}
    RUNTIME DESTINATION ${MYPROJECT_INSTALL_PREFIX}
    )

To complicate things a bit, this is part of a setuptools project installed with pip and using cmake-build-extension: https://github.com/diegoferigo/cmake-build-extension

After pip install . is called, the site-packages folder has the expected content, i.e. site-packages/myproject contains bindings.cpython-38-x86_64-linux-gnu.so and libdrivers.so.

but:

ldd bindings.cpython-38-x86_64-linux-gnu.so

../../lib/libdrivers.so => not found

Also:

objdump -x bindings.cpython-310-x86_64-linux-gnu.so | grep RPATH

does not provide any output, so it does not seem the rpath has been set as desired.

Instead, objdump -x bindings.cpython-310-x86_64-linux-gnu.so output:

Dynamic Section:
  NEEDED               ../../lib/libdrivers.so

Anything I may be doing wrong ?

[diegoferigo]

Thanks again for your help. I realize this is more an issue with me not knowing setuptools/cmake enough, rather than an issue with cmake-build-extension (but I hope this could be useful to other users as well).

Understanding the linking model is quite complicated at first for everyone. For those interesting to know more, How To Write Shared Libraries is a good primer.

For now I put the drivers.so file in a "proper" folder (/usr/local/lib) and find it in cmake, and things work fine. But I would like final users not to have to download / copy the file, especially if they do not have sudo rights. But I'll keep this as plan B, if I remain stuck.

I also came accross auditwheel. I did not look carefully at it yet. I will, but ideally I'd like things to get properly installed with only pip install .

I think in my previous reply I did one step further than what you probably had in mind, I considered the case in which you also want to upload your package to PyPI in form of a wheel. In this case, tools like auditwheel come in the picture. They allow packagers to fix the wheels before uploading them, so that users using pip install <package> would get a working wheel.

In your target setup, if I got it correctly, you don't want to assume that the closed-source .so library is already installed by the user, and also you want to avoid the need to download it manually. To this end, what I typically end implementing in this setting, is relying on CMake. Your CMake project should:

• Call find_package(...) to check if the shared library is found in the system (you might have to write a custom find package logic, there are many guides for this).
• If found in the system, find_package would have populated a new imported target and you bindings can link against the closed-source library.
• If not found in the system and it can be downloaded publicly, you can use resources like FetchContent to automate the download of the closed-source library in your build folder. This would prevent vendoring the shared library in your repo.
• If not found in the system and it cannot be downloaded, you can vendor the closed-source library and manually create an imported target. In this case, the closed-source library becomes part of your project, and when you install your project in the system, also the closed-source library is installed.

When all of this will be in place, you can move forward with this repo, and pip install . will work as expected. The caveat is that if the library is found in the system, your Python package would not have it inside (but the dynamic loader is able to find it from the system), and instead, if is not found in the system and your CMake projects installs it alongside your bindings, your Python package would have it inside. Both configuration would work locally.

Then, for deployment, you can generate the wheel as per documentation, and process it with auditwheel. It will fix only the case in which the closed-source library would not be part of your Python package in the installation example of the last paragraph.

[vincentberenz]

Thanks again for your help !

I gave a try to auditwheel, and saw it would be quite a lot of effort to make it work, especially since I'll need things to work on several systems. I am a bit surprised that something so simple on the surface turns out to be so complicated :)