Inclusion of libsodium in pysodium about pysodium

[ereOn]

Closing this as I found a better alternative and this discussion has apparently died.

[stef]

what is a binary wheel?

[ereOn]

I am referring to PEP 427 -- The Wheel Binary Package Format 1.0.

Here is the relevant paragraph:

Python needs a package format that is easier to install than sdist. Python's sdist packages are defined by and require the distutils and setuptools build systems, running arbitrary code to build-and-install, and re-compile, code just so it can be installed into a new virtualenv. This system of conflating build-install is slow, hard to maintain, and hinders innovation in both build systems and installers.

Wheel attempts to remedy these problems by providing a simpler interface between the build system and the installer. The wheel binary package format frees installers from having to know about the build system, saves time by amortizing compile time over many installations, and removes the need to install a build system in the target environment.

Basically, this would allow providing a package that works right out of pip install on all (if not most) platforms, without having to install libsodium (which frankly on Windows, like many things, is too much trouble).

[stef]

hmm, what are the pros and cons for pysodium?

[ereOn]

@stef Well, consider this example:

This is my experience so far with pysodium when developing a multi-platform asyncio-native implementation of ZeroMQ.

pip install pysodium

I then tried to used only to discover I had to install libsodium externally.

On Linux, I basically had to apt-get install libsodium. On OSX, I had to brew install libsodium. So far, easy enough.

On Windows, I had to compile my own libsodium.dll and put it in some folder. I then had to add that folder to the PATH. Surely, the lack of standard and package manager on Windows is to blame, but from a user perspective, using pysodium could (should ?) be OS agnostic. I should just have to do the pip install and the fact it uses a C library underneath should be an implementation detail.

For instance, pyzmq does exactly what I suggest, and embeds the precompiled binaries for Windows, OSX and compiles it on Linux on package install. As a result, I can do pip install zmq and use the package. The fact it relies on a C library is hidden to the user.

Now to answer to your pros/cons list:

Pros

• Better user experience.
• OS agnosticity (that may not be a word. Forgive my french).
• No incompatibility with the installed library: we always use one we control.
• Related to the previous point: ability to use different versions in different virtual environments without messing with PATH/LD_LIBRARY_PATH.

Cons

• No easy way for the user to control which version of libsodium to use. If a new libsodium is released, and we don't release a new pysodium, the user has to wait or fork/patch.
• A slightly more complex setup.py: cffi makes this easy enough, but there is still some work to do. Maintenance should be low though.

[stef]

hmmm, thx. bring it on.

[ereOn]

@stef : Great ! I'll work on that and keep you posted. May be a while before I have something worth submitting.

[evadot]

No easy way for the user to control which version of libsodium to use. If a new libsodium is released, and we don't release a new pysodium, the user has to wait or fork/patch.

That sucks.
I don't understand why would you want this. It seems normal to me that if you want to use any python wrapper for a C library you need the library on the system.
Changing stuff just for Windows because it sucks a package doesn't seems the way to go for me.

[stef]

hmmm, i also don't like the idea of packaging an extra version of binary libraries. can this be avoided somehow?

[ereOn]

@evadot You clearly misread me.

This is not for Windows only. This is an annoyance for all operating systems. It just happens to be slightly less painful on Linux and OSX. Now you may not care about Windows personally, but alas some people have or want to support all major operating systems, and it would be childish to only care about the ones we prefer (if it were up to me, I'd remove Windows completely from the equation but I can't - even though, my request would still stand).

Many Python libraries took that exact path for the reasons I mentionned. Now, that effectively would make pysodium a standalone-library instead of a wrapper. Which may indeed not be desired, and that's ultimately @stef 's decision. Hence my question before putting down all the work.

As I said, I'm completely OK if that ends-up with a no. What I need is a clear stand on this so I can know how to move forward with my other projects that depend on pysodium.

Now @stef, regarding alternatives, we could imagine having an alternative target install, as ipython does using extras_requires.

So that if you type:

pip install pysodium

You end-up with the wrapper, exactly as it is today. No changes whatsoever.

And if you type:

pip install pysodium[standalone]

You end-up with the wrapper installed, and an additional Python package (let's call it - pysodium-binaries) that I would maintain. When this package is detected by pysodium, it shall use it instead of ctypes to call the various libsodium methods. An externally available libsodium would not be needed anymore in that case.

I think this approach is better as it keeps a perfect backward compatibility with existing code, adds the new functionality for those who need it while keeping the necessary changes in pysodium to a minimum. At the same time, the pysodium-binaries library can evolve at its own pace and things are less tightly coupled.

To sum up, the changes needed in pysodium would be:

• Update setup.py to handle the optional variant.
• Change pysodium/__init__.py to detect and use pysodium-binaries when available.
• Update the README.md file to indicate that a variant is available and why.

Let me know what you think of this. Cheers !

[evadot]

The problem I see with this is :

1. A new libsodium is released with new functionality
2. pysodium gets updated but not pysodium-binaries (or the other way around)
3. People complain because they don't understand why they can't use new stuff while others can.

Packaging libsodium directly with pysodium is a no-go for me, this is a wrapper python package, not a full blown library.

[stef]

I prefer OS wide libsodiums, which are handled by the OS packaging system. The problem with bundled binary libs is, that if there's a security fix, you have to update all bundled copies in addition to the OS wide-libs. But i also see the need sometimes for diverse virtual envs with varying versions of the same libs. It would be nice to support both, the sane default with reusing existing system libs, and a way to also "pin" the lib in a virtual env.

[stef]

oh i forgot the 3rd option: whether packaging libsodium with pysodium directly or as a separate python package is i good idea i'm unsure.

[ereOn]

@evadot

3. People complain because they don't understand why they can't use new stuff while others can.

So in your opinion, it is better to have an annoying user experience all the time rather than having some hypothetical user complaints in the rare occurence where libsodium's API changes ?

Also, in this specific event pysodium would have to be maintained in his current form anyway. You can't imply that this maintenance cost is null for pysodium and too big for pysodium-binaries: that is just unfair reasoning.

What you may be missing here - I don't know how familiar or unfamiliar you are with Python wheels - is that on Linux, you'd have to recompile the Python package binary parts anyway, because binary wheels do not exist on Linux. Put otherwise, you'd always end-up with the most up-to-date libsodium library in every case. On Windows and OSX, you'd have a choice : either install the wheel and enjoy the precompiled libsodium benefits, or recompile the Python package from source and use a more up-to-date version if you need it and want to pay the maintenance overhead.

All this of course, if you - as a user - want to use pysodium as a standalone library and not a wrapper.

Packaging libsodium directly with pysodium is a no-go for me, this is a wrapper python package, not a full blown library.

That is precisely what my last suggestion addresses: people that want to use it as a wrapper, as it is now, still can, without changing a thing. People that require convenience and ease-of-use, be it for development or any other valid reason - being stuck on Windows being one of those - install the standalone variant. We'd have the best of both worlds and more importantly, happy users.

@stef : I was merely suggesting the external Python library to minimize the maintenance impact on your side. Now if you'd prefer to have it all in one package, fine by me as well. I don't really care how : I care more about the user experience and the security aspects.