novocaine / rust-python-ext

Distutils helpers for rust Python extensions

rust-python-ext

Setuptools helpers for rust Python extensions.

Compile and distribute Python extensions written in rust as easily as if they were written in C.

Well, maybe easier - it's rust.

Example

setup.py

from setuptools import setup
from rust_ext import build_rust_cmdclass, install_lib_including_rust

setup(name='hello-rust',
    version='1.0',
    cmdclass={
        # This enables 'setup.py build_rust', and makes it run 
        # 'cargo extensions/cargo.toml' before building your package.
        'build_rust': build_rust_cmdclass('extensions/cargo.toml'),
        # This causes your rust binary to be automatically installed
        # with the package when install_lib runs (including when you 
        # run 'setup.py install'.
        'install_lib': install_lib_including_rust
    },
    packages=['hello_rust'],
    # rust extensions are not zip safe, just like C-extensions.
    zip_safe=False
)

You can optionally pass additional arguments to cargo through build_rust_cmdclass - see https://github.com/novocaine/rust-python-ext/blob/master/rust_ext/__init__.py.

Result:

➜  example git:(master) ✗ python setup.py install

.. yada yada yada ..

running build_rust
cargo build --manifest-path extensions/cargo.toml --release
    Updating registry `https://github.com/rust-lang/crates.io-index`
    Updating git repository `https://github.com/alexcrichton/pkg-config-rs.git`
 Downloading regex v0.1.38
   Compiling pkg-config v0.3.5 (https://github.com/alexcrichton/pkg-config-rs.git#42f1704b)
   Compiling regex-syntax v0.1.2
   Compiling rustc-serialize v0.3.15
   Compiling memchr v0.1.3
   Compiling aho-corasick v0.2.1
   Compiling regex v0.1.38
   Compiling python27-sys v0.0.6 (file:///Users/jsalter/dev/rust-cpython/python27-sys)
   Compiling num v0.1.25
   Compiling cpython v0.0.1 (file:///Users/jsalter/dev/rust-cpython)
   Compiling hello-world v0.0.1 (file:///Users/jsalter/Documents/dev/rust-ext/example/extensions)

.. yada yada yada ..

Installed /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/hello_rust-1.0-py2.7.egg
Processing dependencies for hello-rust==1.0
Finished processing dependencies for hello-rust==1.0

➜  example git:(master) ✗ python
Python 2.7.9 (v2.7.9:648dcafa7e5f, Dec 10 2014, 10:10:46)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information
>>> import hello_rust
>>> hello_rust.hello()
Rust says: Hello Python!

Getting Started

Install:

pip install https://github.com/novocaine/rust-python-ext/zipball/master

Note that I didn't ever upload this extension to pypi - the version that's there has been put there has been uploaded by someone else and is old.

Compile the example:

cd example
python setup.py install
python -c 'import hello_rust; hello_rust.hello()'

If you are not pretty confident with Python C extensions already, it is recommended that you base your project off the code in the example directory. This gives you a sensible layout and something that is already compiling.

Notes

• Supports Python 2.7 and Python 3.6 on Linux and OS X (tested by travis CI)

• Unlike distutils, rust-python-ext delegates all rust build decisions to cargo. So you can't pass compiler args to the compiler from setup.py. This is by design. Cargo's awesome - use that. You can however pass args to cargo which might then influence what it does.

• If you want to access the python C API from rust, use https://github.com/dgrunwald/rust-cpython. The example dir contains a project that shows how this is done.

• If you don't explicitly pass ext_name to build_rust_cmdclass, your extensions will be be named according to your lib's name in cargo.toml, with the lib prefix stripped out so that it looks like a regular Python module as per the c-ext convention. If you want it to start with lib or be named something else, pass a value to ext_name.

• This should interop just fine with other C-exts or cython being in the package, although I haven't tested it. The cmdclass approach is minimally invasive and is how, I believe, the setuptools god intends things to be. There is no monkey-patching or hacking of distutils internals.

• As per the above, you don't have to use the supplied cmdclass helper for install_lib if you don't want to, it just means that install will automatically trigger build_rust.

• You can use setup.py develop to put your module's code on PYTHONPATH without installing it, as you can with other extensions. This automatically enables --inplace.

TODO

• Windows
• An example using CFFI and/or ctypes
• clean command

rust-python-ext's Issues

Why is `lib` prefix removed from library name?

After building a rust library, the lib prefix is removed, e.g. libfoo.so is renamed to foo.so. Why is this being done? I had a hard time working around this on a project I'm working on. Eventually, I resorted to naming my library such that the output file is liblibfoo.so.

Here is my setup.py and my Cargo.toml.

I'm probably missing something obvious... would appreciate any feedback you can provide.

Documentation Does Not Explain Required Feature-Hooks

I discovered through trial and error, and looking at the code, and figuring out how Rust handles features....etc.. that this project requires the inclusion of top-level features in the user's package, python27-sys and python3-sys, which shadow and include the cpython features of the same names. Without these hooks, if I'm not mistaken, rust-ext can't build a library correctly according to the python version that runs the setup.py.

The docs should probably explain a bit of the magic behind rust-ext; I'm happy to contribute, when I've finally got a stable project built on rust-ext in my repertoire!

some help getting up and running with python3?

I have spent some time trying to get this working with python3 and I'm relatively close but I do need some help. I think this maybe should go as a bug report to rust-cpython but i wanted to check here first. build_rust works but then when I go to import i get

ImportError: dynamic module does not define module export function (PyInit__python_boilerplate)

I did some searching around and it looks like the function should be defined with the build options that were given eg it did call cargo build --manifest-path extensions/Cargo.toml --release --features python3-sys and looking through the macro definitions in rust-cpython it seems that for some reason the conditional compilation stuff isn't happening correctly and the config parameter is not being set to python3 all the way through

Build/Install in Develop Mode?

Hi, thanks for this module, it's a great step forward for Rust to have great python bindings, and the beginnings of a good distribution story.

I'm trying to integrate a rust module with a Python module using this tool, and one of the most uncomfortable things is: It doesn't seem to work in develop mode.

This is extra awkward because Rust is installed under my user but seems not to work with sudo when called by this lib. I suspect many other nightly-using rust devs will do likewise and install to user, not system.

Any hints as to how to get this running in develop mode to make the dev cycle a bit more comfortable?

setup.py clean should call cargo clean

I think this makes sense. I'm not too sure if it is easy to do though as I'm not too familiar with python packaging.

Also on a side note, is there anyway to pass to setup.py the list of features we want to build and install with cargo?

Could not compile on Rust 1.10 nightly

Ubuntu 16.04/GNOME

$ rustc --version
rustc 1.10.0-nightly (6974800c6 2016-05-08)

$ cargo build
   Compiling utf8-ranges v0.1.3
   Compiling interpolate_idents v0.0.3
error: unresolved import `rustc::plugin::Registry`. Could not find `plugin` in `rustc` [--explain E0432]
 --> /home/herzog/.cargo/registry/src/github.com-1ecc6299db9ec823/interpolate_idents-0.0.3/src/lib.rs:6:5
6 |> use rustc::plugin::Registry;
  |>     ^^^^^^^^^^^^^^^^^^^^^^^

error: unresolved import `syntax::ast::TokenTree::TtToken`. There is no `TtToken` in `syntax::ast::TokenTree`. Did you mean to use `Token`? [--explain E0432]
 --> /home/herzog/.cargo/registry/src/github.com-1ecc6299db9ec823/interpolate_idents-0.0.3/src/lib.rs:8:36
8 |> use syntax::ast::TokenTree::{self, TtToken};
  |>                                    ^^^^^^^

error: unresolved import `syntax::parse::token::IdentStyle`. There is no `IdentStyle` in `syntax::parse::token` [--explain E0432]
  --> /home/herzog/.cargo/registry/src/github.com-1ecc6299db9ec823/interpolate_idents-0.0.3/src/lib.rs:11:53
11 |> use syntax::parse::token::{self, DelimToken, Token, IdentStyle};
   |>                                                     ^^^^^^^^^^

error: unresolved import `syntax::diagnostic::FatalError`. Could not find `diagnostic` in `syntax` [--explain E0432]
  --> /home/herzog/.cargo/registry/src/github.com-1ecc6299db9ec823/interpolate_idents-0.0.3/src/parser_any_macro.rs:14:13
14 |>         use syntax::diagnostic::FatalError;
   |>             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/home/herzog/.cargo/registry/src/github.com-1ecc6299db9ec823/interpolate_idents-0.0.3/src/parser_any_macro.rs:14:13: 14:43: note: in this expansion of panictry! (defined in /home/herzog/.cargo/registry/src/github.com-1ecc6299db9ec823/interpolate_idents-0.0.3/src/parser_any_macro.rs)

error: unresolved import `syntax::diagnostic::FatalError`. Could not find `diagnostic` in `syntax` [--explain E0432]
  --> /home/herzog/.cargo/registry/src/github.com-1ecc6299db9ec823/interpolate_idents-0.0.3/src/parser_any_macro.rs:14:13
14 |>         use syntax::diagnostic::FatalError;
   |>             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/home/herzog/.cargo/registry/src/github.com-1ecc6299db9ec823/interpolate_idents-0.0.3/src/parser_any_macro.rs:14:13: 14:43: note: in this expansion of panictry! (defined in /home/herzog/.cargo/registry/src/github.com-1ecc6299db9ec823/interpolate_idents-0.0.3/src/parser_any_macro.rs)

error: `TtToken` is not an enum variant, struct or const [E0418]
  --> /home/herzog/.cargo/registry/src/github.com-1ecc6299db9ec823/interpolate_idents-0.0.3/src/lib.rs:31:26
31 |>                         &TtToken(ref span, Token::Ident(ref ident, IdentStyle::Plain)) => {
   |>                          ^^^^^^^

error: method `make_impl_items` has an incompatible type for trait:
 expected struct `syntax::ast::ImplItem`,
    found struct `syntax::ptr::P` [--explain E0053]
  --> /home/herzog/.cargo/registry/src/github.com-1ecc6299db9ec823/interpolate_idents-0.0.3/src/parser_any_macro.rs:74:5
74 |>     fn make_impl_items(self: Box<ParserAnyMacro<'a>>)
   |>     ^

error: method `make_stmts` has an incompatible type for trait:
 expected struct `syntax::codemap::Spanned`,
    found struct `syntax::ptr::P` [--explain E0053]
  --> /home/herzog/.cargo/registry/src/github.com-1ecc6299db9ec823/interpolate_idents-0.0.3/src/parser_any_macro.rs:88:5
88 |>     fn make_stmts(self: Box<ParserAnyMacro<'a>>)
   |>     ^

error: aborting due to 2 previous errors
Build failed, waiting for other jobs to finish...
error: Could not compile `interpolate_idents`.

To learn more, run the command again with --verbose.

Enable Travis builds

Work started in the PR #8

Cargo --debug option is not existing any more

In Cargo 0.16.0 there is no --debug flag and passing second argument to build_rust_cmdclass should behavior should be changed.

Now if you pass True as second arg it will yields error --debug flag not defined.

Options for distributing without rustc?

Hi,
I'm interested in your project because I'm looking for an alternative to C++ extensions. I'm now wondering how the odds are for distributing an extension that I would write.
In C++, I could build a wheel for Windows and Mac and then expect that Linux users would have a compiler ready.
With Rust, however, this may not work that way.
The wheel building part for Win/Mac may still work out (will it?)
I'm most worried about the Linux part since rustc will just not be available everywhere for the foreseeable future. How well does Lbinary distribution work for Linux with Rust? It's not going to be better than C, is it?

Kind regads,
Matthias