diegoferigo / cmake-build-extension Goto Github PK

View Code? Open in Web Editor NEW

56.0 6.0 15.0 156 KB

Setuptools extension to build and package CMake projects

Home Page: https://pypi.org/project/cmake-build-extension/

License: MIT License

Python 20.16% CMake 4.03% SWIG 1.32% C 72.48% C++ 2.02%

cmake setuptools extension python bindings pybind pybind11 swig

cmake-build-extension's People

Contributors

Stargazers

Watchers

Forkers

litghost consonantspork alexus37 externalrepositories iguessthislldo yssource cosunae musicinmybrain claudioperez kdewald mwittgen carlsondc-ceva roy-kid xiretza

cmake-build-extension's Issues

write init.py into a wrong folder

The extension tries to write the init.py into the wrong folder. I don't know how to change this behavior

      ==> Building:
      $ cmake --build /home/xxx/work/cpp/build/temp.linux-x86_64-cpython-310_Blah --config Release
      
      ==> Installing:
      $ cmake --install /home/xxx/work/cpp/build/temp.linux-x86_64-cpython-310_Blan
      
      error: [Errno 2] No such file or directory: '/home/xxx/work/cpp/build/lib.linux-x86_64-cpython-310/blah/__init__.py'
      [end of output]

The build folder should be temp.linux-x86_64-cpython-310_Blah, but it tries to write the __init__.py file into /home/xxx/work/cpp/build/lib.linux-x86_64-cpython-310/blah/

the above error is triggered by write_top_level_init=init_py, in setup.py

Evaluate if recommend `~=` usage for dependencies versions

PEP440 defines, among other things, the ~= operator for declaring compatible releases. Here below the reported example:

# Equivalent
~= 2.2
>= 2.2, == 2.*

# Equivalent
~= 1.4.5
>= 1.4.5, == 1.4.*

This could be helpful when used together with our cmake_depends_on option.

Example: ProjectA and ProjectB use cmake-build-extension (or provide a similar package structure like pybind11, casadi, etc), and ProjectA depends on ProjectB. Using ~= in the dependency specification inside ProjectA metadata could provide a more robust setting to possible ABI / API breaks of future ProjectB versions.

Note that these depends could be only build requires (defined in the pyproject.toml) and not install_requires.

In fact, it is not clear what would happen in a Python application MyApplication uses both ProjectA, that requires when building a specific version of ProjectB, and a new ProjectC, that as well requires when building a specific version of ProjectB, possibly different. I suspect that if the two ProjectB versions have different ABIs, the import order of ProjectA and ProjectC matters.

After analyzing the problem, the README should be updated accordingly with this best-practice.

linking against imported library

Apologies, this is not an issue, but rather a question:
using cmake-build-extension, is it possible to link the generated bindings to some imported library ? (imported: downloaded *.so file with no source available).
More details :
https://stackoverflow.com/questions/73022834/cmake-setuptools-linking-against-an-imported-library-cmake-build-extension

sdist doesn't package source_dir

Per Python documentation

The following files are included in a source distribution by default:

all C source files mentioned in the ext_modules or libraries setup() arguments

However, when I use cmake-build-extension, I have to manually add the source into sdist.

Example cannot be imported in Windows if python >= 3.8

Some while ago, I marked the test as allowed to fail, but it would be wise to open an issue to track the problem.

The SWIG example compiles fine in all Window versions, generating the following files:

adding 'mymath/_bindings.pyd'
adding 'mymath/bindings.py'
adding 'mymath/libbindings.dll.a'
adding 'mymath/include/mymath.h'
adding 'mymath/lib/libmymath.a'
adding 'mymath-0.0.0.dist-info/METADATA'
adding 'mymath-0.0.0.dist-info/WHEEL'
adding 'mymath-0.0.0.dist-info/top_level.txt'
adding 'mymath-0.0.0.dist-info/RECORD'

However, when it is imported in the tests, it fails with the following error (more details in any CI job):

=================================== ERRORS ====================================
____________________ ERROR collecting tests/test_mymath.py ____________________
ImportError while importing test module 'D:\a\cmake-build-extension\cmake-build-extension\examples\swig\tests\test_mymath.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
c:\hostedtoolcache\windows\python\3.9.5\x64\lib\importlib\__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests\test_mymath.py:3: in <module>
    import mymath.bindings
c:\hostedtoolcache\windows\python\3.9.5\x64\lib\site-packages\mymath\bindings.py:18: in <module>
    from . import _bindings
E   ImportError: DLL load failed while importing _bindings: The specified module could not be found.
=========================== short test summary info ===========================
ERROR tests/test_mymath.py
!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
============================== 1 error in 0.37s ===============================

I have no idea what could cause this error. I marked as allowed to fail also the new Python 3.9 job.

I also caught the following in the Python 3.7 job:

C:\Users\RUNNER~1\AppData\Local\Temp\pip-build-env-t1xho44w\overlay\Lib\site-packages\wheel\bdist_wheel.py:82: RuntimeWarning: Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect
    warn=(impl == 'cp')):
  C:\Users\RUNNER~1\AppData\Local\Temp\pip-build-env-t1xho44w\overlay\Lib\site-packages\wheel\bdist_wheel.py:87: RuntimeWarning: Config variable 'WITH_PYMALLOC' is unset, Python ABI tag may be incorrect
    sys.version_info < (3, 8))) \

The second warning disappears if Python >= 3.8.

Extend the example with also pybind11 bindings

Pybind11, especially for big projects that use modern C++ features, is much better than SWIG. On the other hand, if the C++ project is thought from the beginning with SWIG support in mind, using SWIG prevent manually binding each class and each class method, which is quite nice.

This being said, I see a wider and wider usage of pybind11 over SWIG in C++ / Python hybrid projects, and it would be useful extending the example to show how pybind11 can be used in this setup.

Probably the most simple implementation is:

Removing examples/swig/ subfolder and store the example in example/
Renaming example/bindings to example/bindings_swig/
Add a new example/bindings_pybind11/
Add CMake logic to select either the SWIG or the pybind11 bindings
Test both cases separately

This way, the tiny C++ mymath library can be shared by the two systems. And, nice point, the example would provide numpy support for both, that is something that I never found in any other project and, in my experience, is quite challenging to grasp for new users.

Expose binaries of the CMake project

This is a longstanding problem that basically all build-extension implementation I'm aware of suffer. Currently, all the binaries of the CMake projects are correctly installed in the package folder that is installed inside the active site-packages, typically in /path/to/site-packages/<package_name>/bin/. However, they cannot be exposed directly to the user, i.e. there's not simple way to get them either installed or symlinked in the bin folder that is part of the PATH.

Setuptools supports console scripts (setuptools, python packaging, entry points), but the system works only for calling python modules or functions.

With some python magic, we can add a new option (similar to the automatic creation of the __init__.py file) that installs somewhere in the package a __main__ magic file that acts as a Python shim around the C++ executables. In this way, the executables are exposed to the user by the package manager, that cleanly handles the installation and removal of the Python shim that will be created outside the site-packages folder.

Downstream packages inherit setuptools_scm behavior

Due to this issue: pypa/setuptools_scm#653 and brokenness in setuptools itself it is currently not possible to turn setuptools_scm off if it is installed inside the build environment.

That means if I depend on cmake-build-extension for only the purpose of building CMake extensions and not using the setuptools_scm features I have to set up a bunch of pre-emptive excludes that setuptools_scm is so kind to add for me.

I would suggest that until upstream setuptools and setuptools_scm get their act together that you make setuptools_scm and exta and opt-in via cmake-build-extension[scm] or something similar.

Adjust supported Python versions

Remove Python 3.6 since its EOL in Dec 2021
Add Python 3.11

Multiple calls operating on the same build folder leak resources from isolated environment

Hello,

Ninja is currently the only generator supported by cmake-build-extension:

cmake-build-extension/src/cmake_build_extension/build_extension.py

Line 78 in bb836e4

raise RuntimeError("Required command 'ninja' not found")

Unfortunately it is not installed on all platform and a great benefit of CMake is to be agnostic of the final build tool that is used.

It would be nice to add a configuration option that select a different generator in replacement of Ninja (e.g. Make)

Would SWIG example `normalize_numpy` leak memory?

Here's a quote from the wrapper in bindings_swig/bingins.i:

    void normalize_numpy(double* in_1, unsigned size_in_1,
                         double** out_1, int* size_out_1)
    {
        const std::vector<double> vector(in_1, in_1 + size_in_1);

        auto result = mymath::normalize(vector);

        *out_1 = static_cast<double*>(malloc(result.size() * sizeof(double)));
        std::copy(result.begin(), result.end(), *out_1);

        *size_out_1 = result.size();
    }

I can see malloc. The question is when the vector is going to be deallocated on the Python side? Or would numpy do it implicitly (perhaps using Python's GC)?

Python 3.7 on Windows fails in CI

This issue is a reminder of the failure in CI related to the Python 3.7 job on Windows. Refer to #38 for the PR that marked this job as optional in CI.

Failure

============================= test session starts =============================
platform win32 -- Python 3.7.9, pytest-7.1.3, pluggy-1.0.0 -- c:\hostedtoolcache\windows\python\3.7.9\x64\python.exe
cachedir: .pytest_cache
rootdir: D:\a\cmake-build-extension\cmake-build-extension\example, configfile: setup.cfg, testpaths: tests
plugins: icdiff-0.6
Windows fatal exception: code 0xc0000139

Thread 0x00001094 (most recent call first):
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1043 in create_module
  File "<frozen importlib._bootstrap>", line 583 in module_from_spec
  File "<frozen importlib._bootstrap>", line 670 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 967 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 983 in _find_and_load
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1035 in _handle_fromlist
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\mymath_pybind11\__init__.py", line 4 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 728 in exec_module
  File "<frozen importlib._bootstrap>", line 677 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 967 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 983 in _find_and_load
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 953 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 983 in _find_and_load
  File "D:\a\cmake-build-extension\cmake-build-extension\example\tests\test_pybind11.py", line 3 in <module>
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\assertion\rewrite.py", line 168 in exec_module
  File "<frozen importlib._bootstrap>", line 677 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 967 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 983 in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006 in _gcd_import
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\importlib\__init__.py", line 127 in import_module
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\pathlib.py", line 533 in import_path
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\python.py", line 608 in _importtestmodule
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\python.py", line 519 in _getobj
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\python.py", line 301 in obj
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\python.py", line 536 in _inject_setup_module_fixture
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\python.py", line 522 in collect
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\runner.py", line 369 in <lambda>
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\runner.py", line 338 in from_call
  File "D:\a\cmake-build-extension\cmake-build-extension\example\tests\test_swig.py", line 3 in <module>
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\assertion\rewrite.py", line 168 in exec_module
  File "<frozen importlib._bootstrap>", line 677 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 967 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 983 in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006 in _gcd_import
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\importlib\__init__.py", line 127 in import_module
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\pathlib.py", line 533 in import_path
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\python.py", line 608 in _importtestmodule
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\python.py", line 519 in _getobj
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\python.py", line 301 in obj
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\python.py", line 536 in _inject_setup_module_fixture
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\python.py", line 522 in collect
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\runner.py", line 369 in <lambda>
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\runner.py", line 338 in from_call
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\runner.py", line 369 in pytest_make_collect_report
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\runner.py", line 537 in collect_one_node
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\main.py", line 824 in genitems
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\main.py", line 657 in perform_collect
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\main.py", line 332 in pytest_collection
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\main.py", line 321 in _main
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\main.py", line 268 in wrap_session
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\main.py", line 315 in pytest_cmdline_main
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\config\__init__.py", line 165 in main
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\config\__init__.py", line 187 in console_main
  File "C:\hostedtoolcache\windows\Python\3.7.9\x64\Scripts\pytest.exe\__main__.py", line 7 in <module>
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\runpy.py", line 85 in _run_code
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\runpy.py", line 193 in _run_module_as_main
collecting ... collected 0 items / 2 errors

=================================== ERRORS ====================================
___________________ ERROR collecting tests/test_pybind11.py ___________________
ImportError while importing test module 'D:\a\cmake-build-extension\cmake-build-extension\example\tests\test_pybind11.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
c:\hostedtoolcache\windows\python\3.7.9\x64\lib\importlib\__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests\test_pybind11.py:3: in <module>
    import mymath_pybind11.bindings as mymath
c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\mymath_pybind11\__init__.py:4: in <module>
    from . import bindings
E   ImportError: DLL load failed: The specified procedure could not be found.
_____________________ ERROR collecting tests/test_swig.py _____________________
ImportError while importing test module 'D:\a\cmake-build-extension\cmake-build-extension\example\tests\test_swig.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
c:\hostedtoolcache\windows\python\3.7.9\x64\lib\importlib\__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests\test_swig.py:3: in <module>
    import mymath_swig.bindings as mymath
c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\mymath_swig\__init__.py:4: in <module>
    from . import bindings
c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\mymath_swig\bindings.py:18: in <module>
    from . import _swig_bindings
E   ImportError: DLL load failed: The specified procedure could not be found.
!!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!
============================== 2 errors in 0.45s ==============================
Error: Process completed with exit code 1.

Using cmake extension modules alongside other extension modules

Unless I'm misinterpreting something, using BuildExtension as cmdclass['build_ext'] in setup.py makes any non-cmake extension modules completely ineffective, because BuildExtension.run() simply ignores them:

cmake-build-extension/src/cmake_build_extension/build_extension.py

Line 82 in 2c3d822

 cmake_extensions = [e for e in self.extensions if isinstance(e, CMakeExtension)] 

Is my understanding correct? How would I use this project if I also have e.g. cython extension modules?

Allow disabling `CMakeExtension`s through environment variable

It could be useful introducing a new env var like CMAKE_BUILD_EXTENSION_ENABLED=<0|1> to disable the compilation of the build extensions.

As an alternative, it could be replaced by a new --no-cmake-build-extension option of build_ext.

This allows for instance installing only Python resources excluding C++ resources. This is already the default behavior in editable mode, but this feature would allow obtaining the same also for non-editable mode.

Publish to conda-forge

Hi,
thanks for this useful project.
I was wondering if you could also publish the package to conda-forge so it is possible to use it in conda_env.yaml files.

name: my_env

channels:
  - pytorch
  - conda-forge

dependencies:
  - python=3.7
  - pip
  - cmake_build_extension

Thanks!

Use gitignore when packaging with custom sdist command

The GitSdistFolder class should exclude bundling files that match the gitignore.

Right now it selects all the files excluding the .git folder.

Should always check ninja in run() of BuildExtension?

cmake-build-extension/src/cmake_build_extension/build_extension.py

Line 92 in fea4503

if shutil.which("ninja") is None:

In this line, I wonder if the ninja always be checked if I already specify another generator like Unix Makefiles?

PS: no matter what generator I specify, it always throw an error:

==> Configuring:
$ cmake -S /tmp/build-via-sdist-p3l2l10i/xtensor_python_test-0.0.1 -B /tmp/build-via-sdist-p3l2l10i/xtensor_python_test-0.0.1/build/temp.linux-x86_64-cpython-310_Pybind11Bindings -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX:PATH=/tmp/build-via-sdist-p3l2l10i/xtensor_python_test-0.0.1/build/lib.linux-x86_64-cpython-310/mymath_pybind11 -DPython3_ROOT_DIR=/tmp/build-env-yxeuhf6l -DCALL_FROM_SETUP_PY:BOOL=ON -DBUILD_SHARED_LIBS:BOOL=OFF -DEXAMPLE_WITH_SWIG:BOOL=OFF -DEXAMPLE_WITH_PYBIND11:BOOL=ON

==> Building:
$ cmake --build /tmp/build-via-sdist-p3l2l10i/xtensor_python_test-0.0.1/build/temp.linux-x86_64-cpython-310_Pybind11Bindings --config Release

==> Installing:
$ cmake --install /tmp/build-via-sdist-p3l2l10i/xtensor_python_test-0.0.1/build/temp.linux-x86_64-cpython-310_Pybind11Bindings

CMake Error: Could not create named generator "Unix Makefiles"

Thanks in advance if you can help me!

Update `README.md` to use the more common `src` code folder

It currently uses python.

https://setuptools.readthedocs.io/en/latest/userguide/declarative_config.html#using-a-src-layout

diegoferigo / cmake-build-extension Goto Github PK

cmake-build-extension's People

Contributors

Stargazers

Watchers

Forkers

cmake-build-extension's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs