Installation broken on macOS 10.15.7 and ports python3 or apple python3 about python-mbedtls

[Synss]

Hi @earandel and thank you for your report.

I actually develop on MacOS (Catalina) so I know it works on that platform. However, I do not package for MacOS. For that, you should probably see with the MacPorts guys.

Let me have a look anyway. I can probably tell you what is wrong ;)

Cheers,
Mathias

[Synss]

Note: MacOS is not supported--so this is not a bug.

[Synss]

I install everything from source using the scripts under scripts. They are documented.

Maybe simpler in your case (It seemed to work--but keep in mind that my system is tainted because I develop here):

# Get a local copy of the sources
git clone https://github.com/Synss/python-mbedtls.git pymbedtls
cd pymbedtls

# Create a virtual environment where you can install python-mbedtls
python3 -m venv venv
. ./venv/bin/activate
pip install cython

# You probably already have that next one
port install mbedtls

# Now, you need the path where MacPorts installs things on your system: the MacPorts root
port contents mbedtls

# $libdir is the MacPorts root: the part of the path up to .../include/mbedtls or .../lib/mbedtls, etc. above
libdir=PATH_TO_THE_MACPORTS_ROOT
C_INCLUDE_PATH="$libdir/include" LIBRARY_PATH="$libdir/lib" DYLD_LIBRARY_PATH="/usr/lib:$libdir/lib" python setup.py install

But you should really contact MacPorts maintainers or contribute a port. I'd gladly help.

[earandel]

Thank you very much for this... the build nearly worked with the above. It failed with the following error:

build/3.6.12/temp.macosx-10.15-x86_64-3.6/pyrex/mbedtls/_platform.c:608:10: fatal error: 
      'mbedtls/platform_util.h' file not found
#include "mbedtls/platform_util.h"
         ^~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.

I worked around it by creating a symlink in the venv/include dir to the mbedtls directory in the ports include directory. I'm not sure why it failed on this particular header file because it got quite a ways through the build steps before hitting this error so I'm guessing it was able to include some of the mbedtls header files. Anyway, after that change and the above instructions from you, I was able to complete the build and installation in the virtualenv. The import of the mbedtls works now so hopefully everything will work now... I have some other dependency issues to work out with some other python modules before I can test further.

I did try the virtualenv approach before as well but didn't install cython or setup the above env vars so ended up with the same result as installing on the system.

If macOS is not officially support, this issue can be closed then... I mistook the comment under the Installation section mentioning testing on Linux and macOS to mean the platforms were supported. Even if not officially supported, your unofficial support was fantastic in resolving my issue! :) Thanks again.

[Synss]

If macOS is not officially support, this issue can be closed then...

macOS is tested and I would fix bugs quickly. I just do not do packaging—because I believe most people interested in this library use MacPorts or Homebrew. I guess it would not be very difficult to build wheel for that platform. Maybe after I refactored my TravisCI config.

[Synss]

Hi! Issue #36 reported missing files in the archive. That could very well be the cause for the failing installation.

[Synss]

Hi! It was indeed a bug and should now be fixed in 1.4.1. Could you please try once more and report here? Thank you!

[earandel]

Ok, I tried installing without the virtualenv. It got quite a ways into the build but failed trying to find mbedtls/platform_util.h which is in the /opt/local/include directory installed by MacPorts. Interestingly enough, it did find the dynamic library early on in the build process now which is in the /opt/local/lib directory so this is an improvement. I had to manually work around the include file issue before when installing manually using the virtualenv so something must not be quite right with the include rules since it isn't adding /opt/local/include to the compile command.

# pip3 install python-mbedtls
Collecting python-mbedtls
  Downloading python-mbedtls-1.4.1.tar.gz (121 kB)
     |################################| 121 kB 786 kB/s 
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
    Preparing wheel metadata ... done
Collecting certifi
  Downloading certifi-2020.11.8-py2.py3-none-any.whl (155 kB)
     |################################| 155 kB 828 kB/s 
Building wheels for collected packages: python-mbedtls
  Building wheel for python-mbedtls (PEP 517) ... error
  ERROR: Command errored out with exit status 1:
   command: /opt/local/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 /opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmp_ox3i77p
       cwd: /private/tmp/pip-install-6kztug_3/python-mbedtls
  Complete output (53 lines):
    loading: '/opt/local/lib/libmbedtls.dylib'
    mbedtls version: mbed TLS 2.24.0
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/3.6.12
  creating build/3.6.12/lib.macosx-10.15-x86_64-3.6
  creating build/3.6.12/lib.macosx-10.15-x86_64-3.6/mbedtls
  copying src/mbedtls/hashlib.py -> build/3.6.12/lib.macosx-10.15-x86_64-3.6/mbedtls
  copying src/mbedtls/__init__.py -> build/3.6.12/lib.macosx-10.15-x86_64-3.6/mbedtls
  copying src/mbedtls/secrets.py -> build/3.6.12/lib.macosx-10.15-x86_64-3.6/mbedtls
  copying src/mbedtls/hmac.py -> build/3.6.12/lib.macosx-10.15-x86_64-3.6/mbedtls
  creating build/3.6.12/lib.macosx-10.15-x86_64-3.6/mbedtls/cipher
  copying src/mbedtls/cipher/ARIA.py -> build/3.6.12/lib.macosx-10.15-x86_64-3.6/mbedtls/cipher
  copying src/mbedtls/cipher/ARC4.py -> build/3.6.12/lib.macosx-10.15-x86_64-3.6/mbedtls/cipher
  copying src/mbedtls/cipher/AES.py -> build/3.6.12/lib.macosx-10.15-x86_64-3.6/mbedtls/cipher
  copying src/mbedtls/cipher/__init__.py -> build/3.6.12/lib.macosx-10.15-x86_64-3.6/mbedtls/cipher
  copying src/mbedtls/cipher/DES3.py -> build/3.6.12/lib.macosx-10.15-x86_64-3.6/mbedtls/cipher
  copying src/mbedtls/cipher/Camellia.py -> build/3.6.12/lib.macosx-10.15-x86_64-3.6/mbedtls/cipher
  copying src/mbedtls/cipher/CHACHA20.py -> build/3.6.12/lib.macosx-10.15-x86_64-3.6/mbedtls/cipher
  copying src/mbedtls/cipher/Blowfish.py -> build/3.6.12/lib.macosx-10.15-x86_64-3.6/mbedtls/cipher
  copying src/mbedtls/cipher/DES3dbl.py -> build/3.6.12/lib.macosx-10.15-x86_64-3.6/mbedtls/cipher
  copying src/mbedtls/cipher/DES.py -> build/3.6.12/lib.macosx-10.15-x86_64-3.6/mbedtls/cipher
  running build_ext
  cythoning src/mbedtls/_platform.pyx to build/3.6.12/temp.macosx-10.15-x86_64-3.6/pyrex/mbedtls/_platform.c
  creating build/3.6.12/temp.macosx-10.15-x86_64-3.6
  creating build/3.6.12/temp.macosx-10.15-x86_64-3.6/pyrex
  creating build/3.6.12/temp.macosx-10.15-x86_64-3.6/pyrex/mbedtls
  cythoning src/mbedtls/mpi.pyx to build/3.6.12/temp.macosx-10.15-x86_64-3.6/pyrex/mbedtls/mpi.c
  cythoning src/mbedtls/pk.pyx to build/3.6.12/temp.macosx-10.15-x86_64-3.6/pyrex/mbedtls/pk.c
  cythoning src/mbedtls/tls.pyx to build/3.6.12/temp.macosx-10.15-x86_64-3.6/pyrex/mbedtls/tls.c
  cythoning src/mbedtls/version.pyx to build/3.6.12/temp.macosx-10.15-x86_64-3.6/pyrex/mbedtls/version.c
  cythoning src/mbedtls/_random.pyx to build/3.6.12/temp.macosx-10.15-x86_64-3.6/pyrex/mbedtls/_random.c
  cythoning src/mbedtls/hkdf.pyx to build/3.6.12/temp.macosx-10.15-x86_64-3.6/pyrex/mbedtls/hkdf.c
  cythoning src/mbedtls/x509.pyx to build/3.6.12/temp.macosx-10.15-x86_64-3.6/pyrex/mbedtls/x509.c
  cythoning src/mbedtls/exceptions.pyx to build/3.6.12/temp.macosx-10.15-x86_64-3.6/pyrex/mbedtls/exceptions.c
  cythoning src/mbedtls/_ringbuf.pyx to build/3.6.12/temp.macosx-10.15-x86_64-3.6/pyrex/mbedtls/_ringbuf.c
  cythoning src/mbedtls/_md.pyx to build/3.6.12/temp.macosx-10.15-x86_64-3.6/pyrex/mbedtls/_md.c
  cythoning src/mbedtls/cipher/_cipher.pyx to build/3.6.12/temp.macosx-10.15-x86_64-3.6/pyrex/mbedtls/cipher/_cipher.c
  creating build/3.6.12/temp.macosx-10.15-x86_64-3.6/pyrex/mbedtls/cipher
  building 'mbedtls._platform' extension
  creating build/3.6.12/temp.macosx-10.15-x86_64-3.6/build
  creating build/3.6.12/temp.macosx-10.15-x86_64-3.6/build/3.6.12
  creating build/3.6.12/temp.macosx-10.15-x86_64-3.6/build/3.6.12/temp.macosx-10.15-x86_64-3.6
  creating build/3.6.12/temp.macosx-10.15-x86_64-3.6/build/3.6.12/temp.macosx-10.15-x86_64-3.6/pyrex
  creating build/3.6.12/temp.macosx-10.15-x86_64-3.6/build/3.6.12/temp.macosx-10.15-x86_64-3.6/pyrex/mbedtls
  /usr/bin/clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -pipe -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -I/opt/local/Library/Frameworks/Python.framework/Versions/3.6/include/python3.6m -c build/3.6.12/temp.macosx-10.15-x86_64-3.6/pyrex/mbedtls/_platform.c -o build/3.6.12/temp.macosx-10.15-x86_64-3.6/build/3.6.12/temp.macosx-10.15-x86_64-3.6/pyrex/mbedtls/_platform.o
  build/3.6.12/temp.macosx-10.15-x86_64-3.6/pyrex/mbedtls/_platform.c:608:10: fatal error: 'mbedtls/platform_util.h' file not found
  #include "mbedtls/platform_util.h"
           ^~~~~~~~~~~~~~~~~~~~~~~~~
  1 error generated.
  error: command '/usr/bin/clang' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for python-mbedtls
Failed to build python-mbedtls
ERROR: Could not build wheels for python-mbedtls which use PEP 517 and cannot be installed directly

# ls -l /opt/local/include/mbedtls/platform_util.h 
-rw-r--r--  1 root  wheel  7727 Aug 27 04:31 /opt/local/include/mbedtls/platform_util.h

[Synss]

Now, the easiest way for you would be to find a simple MacPorts Python port and adapt it to python-mbedtls. You probably only have to change the filename and version and maybe a couple of links and checksums.

Anyway, compiling from source, you will need to tell clang where the headers and the libraries are. You can check at my (shell script)[https://github.com/Synss/python-mbedtls/blob/master/scripts/install-pymbedtls.sh] for how I do that during development.

I also use MacPorts but I install it locally so the following paths may not be entirely correct.

Anyway, for the compilation, you will need at least C_INCLUDE_PATH and LIBRARY_PATH and to use the library, you will need DYLD_LIBRARY_PATH.

So in your case, probably

export C_INCLUDE_PATH=/opt/local/include
export LIBRARY_PATH=/opt/local/lib
export DYLD_LIBRARY_PATH=$LIBRARY_PATH
pip3 install python-mbedtls
# ...
# python3
# import mbedtls

But really, you should do it with MacPorts ;)

[Synss]

I could write a port but I do not want to submit it or maintain it (because of my limited time–nothing else). Would you be interested in maintaining it for MacPorts?

I would write an initial, working port and the maintenance burden would probably be limited to checking for your releases, updating the version, and reaching out to me when it does not work.

[earandel]

Strangely enough, if I set all the env vars you suggested above, the installation fails with an error that it cannot find the _iconv symbol in libiconv.2.dylib. However, if I just do

export C_INCLUDE_PATH=/opt/local/include
pip3 install python-mbedtls
# ...
# python3
# import mbedtls

Everything completes successfully.

As for maintaining a port for this, let me read up on what is involved in maintaining a port. I don't have a lot of extra time but it might not be too time consuming or difficult.

[Synss]

Great that it finally worked for you as well!

[crifan]

Have fixed: mbedtls/_platform.c:1205:10: fatal error: 'mbedtls/platform_util.h' file not found

• Refer: Mbed TLS documentation
• Steps
  • download git src: git clone https://github.com/Mbed-TLS/mbedtls.git
  • build and install: make and make install
    • default install path: /usr/local

Final worked:

export C_INCLUDE_PATH=/usr/local/include
export LIBRARY_PATH=/usr/local/lib/

pip3 install python-mbedtls

[second-string]

Don't forget to pip install the requirements for mbedtls if it's not installing properly. Before running make and make install:
cd mbedtls
python3 -m venv .venv
source .venv/bin/activate
pip3 install -r scripts/basic.requirements.txt