libc++ conflict with libstdc++ with Linux and Anaconda about py-tree-sitter

[boba-and-beer]

I don't have Anaconda installed and I encountered the same issue so I don't think it's an anaconda-specific problem. Not a maintainer, but I think creating a PR would help resolve the problem for a fair few others as well (including me!)

Edit: Managed to fix it on my side. For anyone else experiencing similar issues,
navigate it to:
https://github.com/tree-sitter/py-tree-sitter/blob/master/tree_sitter/__init__.py#L46

Change:

        if cpp:
            if find_library("c++"):
                compiler.add_library("c++")
            elif find_library("stdc++"):
                compiler.add_library("stdc++")

to

compiler.add_library("stdc++")

and then install it again. (Alternatively - just edit it inside your environment for installed packages)

[sogaiu]

FWIW, I think there might also be some relevant info in this issue: #41

[maxbrunsfeld]

Has anyone come up with a general solution to this that could be incorporated into the library? It would be great if we could somehow detect what version of the C++ standard library the current Python interpreter is linked against (if any).

[narpfel]

I believe the problem is caused by the fact that distutils uses cc as the driver to link the final shared object, so the C++ standard library (libstdc++ or libc++, depending on the platform) is not linked automatically. If c++ were used, the compiler should automatically select the correct (default) STL for the current platform.

I’m not sure if this is the proper API to change the compiler executable in distutils, but this (somewhat hacky?) patch fixes the problem for me:

diff --git a/tree_sitter/__init__.py b/tree_sitter/__init__.py
index cc94ac9..b2e23d4 100644
--- a/tree_sitter/__init__.py
+++ b/tree_sitter/__init__.py
@@ -3,6 +3,7 @@
 from ctypes import cdll, c_void_p
 from ctypes.util import find_library
 from distutils.ccompiler import new_compiler
+from distutils.unixccompiler import UnixCCompiler
 from os import path
 from platform import system
 from tempfile import TemporaryDirectory
@@ -43,11 +44,8 @@ class Language:
         ]
 
         compiler = new_compiler()
-        if cpp:
-            if find_library("c++"):
-                compiler.add_library("c++")
-            elif find_library("stdc++"):
-                compiler.add_library("stdc++")
+        if isinstance(compiler, UnixCCompiler):
+            compiler.compiler_cxx[0] = "c++"
 
         if max(source_mtimes) <= output_mtime:
             return False
@@ -69,7 +67,11 @@ class Language:
                         extra_preargs=flags,
                     )[0]
                 )
-            compiler.link_shared_object(object_paths, output_path)
+            compiler.link_shared_object(
+                object_paths,
+                output_path,
+                target_lang="c++" if cpp else "c",
+            )
         return True
 
     def __init__(self, library_path, name):

This should let the C++ compiler link the same STL as is expected by scanner.cc. I don’t have access to a Mac so I only tested this on Linux.

[maxbrunsfeld]

If anyone has the time and ability to thoroughly test this (or any other) approach, I'd definitely be open to merging it. I think it's fine as long as it can be shown to work on macOS, some common linux distro, and windows.

[narpfel]

@maxbrunsfeld What would you consider a thorough test for this? Would passing the test suite be enough? As one of the tests uses tree-sitter-python (which has a scanner.cc file), compiling and loading the resulting tree sitter library should be sufficient to find out if the correct STL was linked.

Because the py-tree-sitter CI currently does not test on Mac, I set up a GitHub action to test this on Ubuntu (with and without libc++ installed), Windows and Mac here. CI results here. (Note that the build failure for PyPy on Mac is not related to the patch, but also present on current master.)

If this is OK for you, I will make a PR. Should I include the GitHub action?

[maxbrunsfeld]

What would you consider a thorough test for this? Would passing the test suite be enough?

👍 Yes.

Note that the build failure for PyPy on Mac is not related to the patch, but also present on current master

That's strange; I personally develop on a mac, and the tests run fine for me. The error message is interesting:

clang: warning: include path for libstdc++ headers not found; pass '-stdlib=libc++' on the command line to use the libc++ standard library instead [-Wstdlibcxx-not-found]
tests/fixtures/tree-sitter-python/src/scanner.cc:2:10: fatal error: 'vector' file not found
#include <vector>
         ^~~~~~~~

It looks like, for some reason, the flag -stdlib=libstdc++ is being passed to clang, indicating that it should use the non-default C++ standard library.

[narpfel]

I reran the tests without --quiet. This is the compiler invocation on PyPy/Mac that errors:

cc -arch x86_64 -fPIC -Itests/fixtures/tree-sitter-python/src -c tests/fixtures/tree-sitter-python/src/scanner.cc -o /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpg8_knoprtree_sitter_language/tests/fixtures/tree-sitter-python/src/scanner.o

and on Python3.8/Mac:

cc -fPIC -Itests/fixtures/tree-sitter-python/src -c tests/fixtures/tree-sitter-python/src/scanner.cc -o /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpqvfc5o5vtree_sitter_language/tests/fixtures/tree-sitter-python/src/scanner.o

So the only difference is -arch? I don’t see why this should influence clang’s include path, though I don’t know Apple Clang at all.

[maxbrunsfeld]

Oh, I think it's because we should be using c++ instead of cc as the executable?

[maxbrunsfeld]

I also think it's fine to just not include PyPy on mac in our CI matrix. I'm sorry, I misunderstood you and didn't realize that there was a successful macOS build using Python3.8.

[narpfel]

Oh, I think it's because we should be using c++ instead of cc as the executable?

It shouldn’t matter. cc as the compiler driver usually selects the right compiler based on the extension.

I also think it's fine to just not include PyPy on mac in our CI matrix.

Done.