Hi.

The bug is related to https://llvm.org/doxygen/namespacellvm_1_1sys.html#aab375071f641a086c0d7067635ccd3dc
https://github.com/llvm/llvm-project/blob/213aea4c5836934771eb97eb97e4c964053a8596/llvm/include/llvm/Support/Host.h#L36

$cmake -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -G Ninja ..
-- The C compiler identification is Clang 11.0.0
-- The CXX compiler identification is Clang 11.0.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Using llvm-config: /usr/bin/llvm-config
-- LLVM prefix: /usr
-- Clang cmake directory: /usr/lib/cmake/clang
-- Clang link dylib: OFF
-- Performing Test LLVM_LIBSTDCXX_MIN
-- Performing Test LLVM_LIBSTDCXX_MIN - Success
-- Performing Test LLVM_LIBSTDCXX_SOFT_ERROR
-- Performing Test LLVM_LIBSTDCXX_SOFT_ERROR - Success
-- Performing Test C_SUPPORTS_FPIC
-- Performing Test C_SUPPORTS_FPIC - Success
-- Performing Test CXX_SUPPORTS_FPIC
-- Performing Test CXX_SUPPORTS_FPIC - Success
-- Building with -fPIC
-- Performing Test SUPPORTS_FVISIBILITY_INLINES_HIDDEN_FLAG
-- Performing Test SUPPORTS_FVISIBILITY_INLINES_HIDDEN_FLAG - Success
-- Performing Test C_SUPPORTS_WERROR_DATE_TIME
-- Performing Test C_SUPPORTS_WERROR_DATE_TIME - Success
-- Performing Test CXX_SUPPORTS_WERROR_DATE_TIME
-- Performing Test CXX_SUPPORTS_WERROR_DATE_TIME - Success
-- Performing Test C_SUPPORTS_WERROR_UNGUARDED_AVAILABILITY_NEW
-- Performing Test C_SUPPORTS_WERROR_UNGUARDED_AVAILABILITY_NEW - Success
-- Performing Test CXX_SUPPORTS_WERROR_UNGUARDED_AVAILABILITY_NEW
-- Performing Test CXX_SUPPORTS_WERROR_UNGUARDED_AVAILABILITY_NEW - Success
-- Performing Test LINKER_SUPPORTS_COLOR_DIAGNOSTICS
-- Performing Test LINKER_SUPPORTS_COLOR_DIAGNOSTICS - Success
-- Performing Test C_SUPPORTS_FNO_FUNCTION_SECTIONS
-- Performing Test C_SUPPORTS_FNO_FUNCTION_SECTIONS - Success
-- Performing Test C_SUPPORTS_FFUNCTION_SECTIONS
-- Performing Test C_SUPPORTS_FFUNCTION_SECTIONS - Success
-- Performing Test CXX_SUPPORTS_FFUNCTION_SECTIONS
-- Performing Test CXX_SUPPORTS_FFUNCTION_SECTIONS - Success
-- Performing Test C_SUPPORTS_FDATA_SECTIONS
-- Performing Test C_SUPPORTS_FDATA_SECTIONS - Success
-- Performing Test CXX_SUPPORTS_FDATA_SECTIONS
-- Performing Test CXX_SUPPORTS_FDATA_SECTIONS - Success
CMake Warning (dev) at /usr/lib/cmake/llvm/HandleLLVMOptions.cmake:852 (option):
Policy CMP0077 is not set: option() honors normal variables. Run "cmake
--help-policy CMP0077" for policy details. Use the cmake_policy command to
set the policy and suppress this warning.

For compatibility with older versions of CMake, option is clearing the
normal variable 'LLVM_ENABLE_EH'.
Call Stack (most recent call first):
CMakeLists.txt:59 (include)
This warning is for project developers. Use -Wno-dev to suppress it.

CMake Warning (dev) at /usr/lib/cmake/llvm/HandleLLVMOptions.cmake:853 (option):
Policy CMP0077 is not set: option() honors normal variables. Run "cmake
--help-policy CMP0077" for policy details. Use the cmake_policy command to
set the policy and suppress this warning.

For compatibility with older versions of CMake, option is clearing the
normal variable 'LLVM_ENABLE_RTTI'.
Call Stack (most recent call first):
CMakeLists.txt:59 (include)
This warning is for project developers. Use -Wno-dev to suppress it.

CMake Warning (dev) at /usr/local/share/cmake/Modules/CMakeDependentOption.cmake:39 (option):
Policy CMP0077 is not set: option() honors normal variables. Run "cmake
--help-policy CMP0077" for policy details. Use the cmake_policy command to
set the policy and suppress this warning.

For compatibility with older versions of CMake, option is clearing the
normal variable 'LLVM_EXPORT_SYMBOLS_FOR_PLUGINS'.
Call Stack (most recent call first):
/usr/lib/cmake/llvm/HandleLLVMOptions.cmake:1006 (CMAKE_DEPENDENT_OPTION)
CMakeLists.txt:59 (include)
This warning is for project developers. Use -Wno-dev to suppress it.

-- Looking for os_signpost_interval_begin
-- Looking for os_signpost_interval_begin - not found
-- Linker detection: LLD
-- Clang resources directory: /usr/lib/clang/11.0.0/include
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of __int128_t
-- Check size of __int128_t - done
-- Check size of _Complex
-- Check size of _Complex - done
-- Found PythonInterp: /usr/local/bin/python3.7 (found version "3.7.9")
-- Found PythonLibs: /usr/local/lib/libpython3.7m.so (found version "3.7.9")
-- Configuring done
-- Generating done
-- Build files have been written to: /home/oceanfish81/dragonffi/build
oceanfish81@gollvm:~/dragonffi/build $ ninja -j 8
[5/60] Building CXX object CMakeFiles/dffi.dir/lib/dffi_api.cpp.o
FAILED: CMakeFiles/dffi.dir/lib/dffi_api.cpp.o
/usr/bin/clang++ -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Ddffi_EXPORTS -I../include -Iinclude -I../third-party -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -w -fdiagnostics-color -ffunction-sections -fdata-sections -fPIC -fvisibility=hidden -std=c++14 -MD -MT CMakeFiles/dffi.dir/lib/dffi_api.cpp.o -MF CMakeFiles/dffi.dir/lib/dffi_api.cpp.o.d -o CMakeFiles/dffi.dir/lib/dffi_api.cpp.o -c ../lib/dffi_api.cpp
../lib/dffi_api.cpp:112:21: error: no member named 'getProcessTriple' in namespace 'llvm::sys'
return llvm::sys::getProcessTriple();
~~~~~~~~~~~^
1 error generated.

Here is what I have under Qemu/on OpenSUSE:

oceanfish81@riscv:/home/oceanfish81/dragonffi/build/tests> ./anon_members
WARNING: This target JIT is not designed for the host you are running. If bad things happen, please choose a different -march switch.

oceanfish81@riscv:/home/oceanfish81/dragonffi/build/tests> ./anon_struct
WARNING: This target JIT is not designed for the host you are running. If bad things happen, please choose a different -march switch.
LLVM ERROR: Relocation type not implemented yet!
Aborted (core dumped)

oceanfish81@riscv:/home/oceanfish81/dragonffi/build/tests> ./anon_union
WARNING: This target JIT is not designed for the host you are running. If bad things happen, please choose a different -march switch.
LLVM ERROR: Relocation type not implemented yet!
Aborted (core dumped)

oceanfish81@riscv:/home/oceanfish81/dragonffi/build/tests> ./array
WARNING: This target JIT is not designed for the host you are running. If bad things happen, please choose a different -march switch.

oceanfish81@riscv:/home/oceanfish81/dragonffi/build/tests> ./asm_redirect
WARNING: This target JIT is not designed for the host you are running. If bad things happen, please choose a different -march switch.
LLVM ERROR: Relocation type not implemented yet!
Aborted (core dumped)

oceanfish81@riscv:/home/oceanfish81/dragonffi/build/tests> ./attrs
WARNING: This target JIT is not designed for the host you are running. If bad things happen, please choose a different -march switch.

oceanfish81@riscv:/home/oceanfish81/dragonffi/build/tests> ./bool
WARNING: This target JIT is not designed for the host you are running. If bad things happen, please choose a different -march switch.

oceanfish81@riscv:/home/oceanfish81/dragonffi/build/tests> ./cconv
WARNING: This target JIT is not designed for the host you are running. If bad things happen, please choose a different -march switch.
LLVM ERROR: Relocation type not implemented yet!
Aborted (core dumped)

oceanfish81@riscv:/home/oceanfish81/dragonffi/build/tests> ./compile
WARNING: This target JIT is not designed for the host you are running. If bad things happen, please choose a different -march switch.

oceanfish81@riscv:/home/oceanfish81/dragonffi/build/tests> ./compile_error
WARNING: This target JIT is not designed for the host you are running. If bad things happen, please choose a different -march switch.
/__dffi_private/anon_cu_0.c:1:1: error: unknown type name 'this'
this is not valid C code
^
/__dffi_private/anon_cu_0.c:1:8: error: expected ';' after top level declarator
this is not valid C code
^
;

/__dffi_private/anon_cu_1.c:1:1: error: unknown type name 'still'
still invalide code
^
/__dffi_private/anon_cu_1.c:1:15: error: expected ';' after top level declarator
still invalide code
^
;

oceanfish81@riscv:/home/oceanfish81/dragonffi/build/tests> ./decl
WARNING: This target JIT is not designed for the host you are running. If bad things happen, please choose a different -march switch.
coucou
coucou
LLVM ERROR: Relocation type not implemented yet!
Aborted (core dumped)

oceanfish81@riscv:/home/oceanfish81/dragonffi/build/tests> ./dlopen
Unable to open (null)

oceanfish81@riscv:/home/oceanfish81/dragonffi/build/tests> ./enum
WARNING: This target JIT is not designed for the host you are running. If bad things happen, please choose a different -march switch.
V10 = 10
V4 = 4
V1 = 1
V0 = 0

oceanfish81@riscv:/home/oceanfish81/dragonffi/build/tests> ./func_ptr
WARNING: This target JIT is not designed for the host you are running. If bad things happen, please choose a different -march switch.
LLVM ERROR: Relocation type not implemented yet!
Aborted (core dumped)

oceanfish81@riscv:/home/oceanfish81/dragonffi/build/tests> ./includes
terminate called after throwing an instance of 'std::logic_error'
what(): basic_string::_M_construct null not valid
Aborted (core dumped)

oceanfish81@riscv:/home/oceanfish81/dragonffi/build/tests> ./inline
WARNING: This target JIT is not designed for the host you are running. If bad things happen, please choose a different -march switch.

oceanfish81@riscv:/home/oceanfish81/dragonffi/build/tests> ./multiple_defs
WARNING: This target JIT is not designed for the host you are running. If bad things happen, please choose a different -march switch.

oceanfish81@riscv:/home/oceanfish81/dragonffi/build/tests> ./stdint
WARNING: This target JIT is not designed for the host you are running. If bad things happen, please choose a different -march switch.

oceanfish81@riscv:/home/oceanfish81/dragonffi/build/tests> ./struct
WARNING: This target JIT is not designed for the host you are running. If bad things happen, please choose a different -march switch.
LLVM ERROR: Relocation type not implemented yet!
Aborted (core dumped)

oceanfish81@riscv:/home/oceanfish81/dragonffi/build/tests> ./system_headers
WARNING: This target JIT is not designed for the host you are running. If bad things happen, please choose a different -march switch.
LLVM ERROR: Relocation type not implemented yet!
Aborted (core dumped)

oceanfish81@riscv:/home/oceanfish81/dragonffi/build/tests> ./typedef
WARNING: This target JIT is not designed for the host you are running. If bad things happen, please choose a different -march switch.
LLVM ERROR: Relocation type not implemented yet!
Aborted (core dumped)

oceanfish81@riscv:/home/oceanfish81/dragonffi/build/tests> ./union
WARNING: This target JIT is not designed for the host you are running. If bad things happen, please choose a different -march switch.
LLVM ERROR: Relocation type not implemented yet!
Aborted (core dumped)

oceanfish81@riscv:/home/oceanfish81/dragonffi/build/tests> ./varargs
WARNING: This target JIT is not designed for the host you are running. If bad things happen, please choose a different -march switch.
LLVM ERROR: Relocation type not implemented yet!
Aborted (core dumped)

Surely there is no actual port - and not all tests failed.
I might check more tests, if they are available.

This is a great library, thank you.

I need to support ARM64 processors also running Linux. While it's not tested on that platform, is it expected to work? Or does extra code need to be written to support that platform?

I'd like to request support for booleans. I think it's the DW_ATE_boolean enum variant?

It does not appear in the switch statement inside dffi_impl.cpp / CUImpl::getTypeFromDIType

Hello.
Here is what I have on

$ uname -a
Linux debian 5.9.0-1-powerpc64 #1 SMP Debian 5.9.1-1 (2020-10-17) ppc64 GNU/Linux

with

$ ld.lld --version
LLD 11.0.1 (compatible with GNU linkers)

and

$ clang --version
Debian clang version 11.0.1-+rc2-1
:

$ cmake -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_BUILD_TYPE=MinSizeRel -DCMAKE_CXX_FLAGS=' -fuse-ld=lld ' .. -G Ninja
-- The C compiler identification is Clang 11.0.1
-- The CXX compiler identification is Clang 11.0.1
-- Detecting C compiler ABI info
make: *** [Makefile:103: all] Error 2
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - failed
-- Check for working CXX compiler: /usr/bin/clang++
-- Check for working CXX compiler: /usr/bin/clang++ - broken
CMake Error at /usr/share/cmake-3.18/Modules/CMakeTestCXXCompiler.cmake:59 (message):
The C++ compiler

"/usr/bin/clang++"

is not able to compile a simple test program.

It fails with the following output:

Change Dir: /home/oceanfish81/dragonffi/release/CMakeFiles/CMakeTmp

Run Build Command(s):/usr/local/bin/ninja cmTC_45259 && [1/2] Building CXX object CMakeFiles/cmTC_45259.dir/testCXXCompiler.cxx.o
clang: warning: argument unused during compilation: '-fuse-ld=lld' [-Wunused-command-line-argument]
[2/2] Linking CXX executable cmTC_45259
FAILED: cmTC_45259
: && /usr/bin/clang++ -fuse-ld=lld  CMakeFiles/cmTC_45259.dir/testCXXCompiler.cxx.o -o cmTC_45259   && :
ld.lld: error: /usr/bin/../lib/gcc/powerpc64-linux-gnu/10/../../../powerpc64-linux-gnu/crt1.o: ABI version 1 is not supported
clang: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.

Is it possible to print the signature of a loaded function? For example, gettimeofday is tricky because the arguments are pointers to structs. Is it possible to get this signature back:

int gettimeofday(struct timeval *restrict tp, void *restrict tzp);

And even better that timeval is this? (taken from sys/time.h)

struct timeval {
             time_t       tv_sec;   /* seconds since Jan. 1, 1970 */
             suseconds_t  tv_usec;  /* and microseconds */
     };

I get stuck in not knowing how to proceed from the "QualType" for the arguments.

   >>> import pydffi
   >>> FFI = pydffi.FFI()
   >>> CU = FFI.cdef("#include <sys/time.h>")
   >>> CU.funcs.gettimeofday.type().returnType.kind
   BasicKind.Int
   >>> CU.funcs.gettimeofday.type().params[0]
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
   TypeError: Unable to convert function return value to a Python type! The signature was
   	(arg0: pydffi.FunctionType) -> List[dffi::QualType].   

Hi.
I wasn't able to run one of the tests, on ppc64 (Qemu).

oceanfish81@debian:~/dragonffi/release/tests$ ./includes
terminate called after throwing an instance of 'std::logic_error'
what(): basic_string::_M_construct null not valid
Aborted

Here is my environment:

$ uname -a
Linux debian 5.9.0-1-powerpc64 #1 SMP Debian 5.9.1-1 (2020-10-17) ppc64 GNU/Linux

$ cat /proc/cpuinfo
processor : 0
cpu : POWER7 (architected), altivec supported
clock : 1000.000000MHz
revision : 2.3 (pvr 003f 0203)

processor : 1
cpu : POWER7 (architected), altivec supported
clock : 1000.000000MHz
revision : 2.3 (pvr 003f 0203)

processor : 2
cpu : POWER7 (architected), altivec supported
clock : 1000.000000MHz
revision : 2.3 (pvr 003f 0203)

processor : 3
cpu : POWER7 (architected), altivec supported
clock : 1000.000000MHz
revision : 2.3 (pvr 003f 0203)

processor : 4
cpu : POWER7 (architected), altivec supported
clock : 1000.000000MHz
revision : 2.3 (pvr 003f 0203)

processor : 5
cpu : POWER7 (architected), altivec supported
clock : 1000.000000MHz
revision : 2.3 (pvr 003f 0203)

processor : 6
cpu : POWER7 (architected), altivec supported
clock : 1000.000000MHz
revision : 2.3 (pvr 003f 0203)

processor : 7
cpu : POWER7 (architected), altivec supported
clock : 1000.000000MHz
revision : 2.3 (pvr 003f 0203)

timebase : 512000000
platform : pSeries
model : IBM pSeries (emulated by qemu)
machine : CHRP IBM pSeries (emulated by qemu)
MMU : Hash

$ cat /etc/cloud-release
ID="genericcloud"
VERSION="202010221749"

$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux bullseye/sid"
NAME="Debian GNU/Linux"
ID=debian

I am running Qemu in such a way:

qemu-system-ppc64w -smp 8 -hda ppc64_deb_cloud.vmdk -netdev tap,id=mynet0,ifname=tap1 -device e1000,netdev=mynet0 -machine pseries-2.7 -m 4G

qemu-system-ppc64 --version
QEMU emulator version 5.1.92 (v5.2.0-rc2-11843-gf571c4ffb5-dirty)
Copyright (c) 2003-2020 Fabrice Bellard and the QEMU Project developers

Hi.

I tried generating random numbers by seeding the RNG using srand with return value from time(0).

Here's the script

I would like your help in understanding why I couldn't successfully seed the RNG using time function. I left information about the exception and different ways of achieving my goal I've tried. If something is not clear or you need more information, please let me know.

pydffi was installed with pip install pydffi.
OS:
Linux machine 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04) x86_64 GNU/Linux
Python version: Python 3.6.3

Thank you!

I didn't find any documentation on your tests - so I was not even able to suggest any expanded logs, on a per-test basis.

I am unsure if that is about a test suite - it looks like it is some sequence of tests, each of which checks some specific cutting edge aspect.

I have compressed my build folder https://github.com/advancedwebdeveloper/dragonffi_ppc64/raw/master/dragonffi_release.zip
.
Files are available on Github directly https://github.com/advancedwebdeveloper/dragonffi_ppc64

Here is my environment:

$ uname -a
Linux debian 5.9.0-1-powerpc64 #1 SMP Debian 5.9.1-1 (2020-10-17) ppc64 GNU/Linux

$ cat /proc/cpuinfo
processor : 0
cpu : POWER7 (architected), altivec supported
clock : 1000.000000MHz
revision : 2.3 (pvr 003f 0203)

processor : 1
cpu : POWER7 (architected), altivec supported
clock : 1000.000000MHz
revision : 2.3 (pvr 003f 0203)

processor : 2
cpu : POWER7 (architected), altivec supported
clock : 1000.000000MHz
revision : 2.3 (pvr 003f 0203)

processor : 3
cpu : POWER7 (architected), altivec supported
clock : 1000.000000MHz
revision : 2.3 (pvr 003f 0203)

processor : 4
cpu : POWER7 (architected), altivec supported
clock : 1000.000000MHz
revision : 2.3 (pvr 003f 0203)

processor : 5
cpu : POWER7 (architected), altivec supported
clock : 1000.000000MHz
revision : 2.3 (pvr 003f 0203)

processor : 6
cpu : POWER7 (architected), altivec supported
clock : 1000.000000MHz
revision : 2.3 (pvr 003f 0203)

processor : 7
cpu : POWER7 (architected), altivec supported
clock : 1000.000000MHz
revision : 2.3 (pvr 003f 0203)

timebase : 512000000
platform : pSeries
model : IBM pSeries (emulated by qemu)
machine : CHRP IBM pSeries (emulated by qemu)
MMU : Hash

$ cat /etc/cloud-release
ID="genericcloud"
VERSION="202010221749"

$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux bullseye/sid"
NAME="Debian GNU/Linux"
ID=debian

I am running Qemu in such a way:

qemu-system-ppc64w -smp 8 -hda ppc64_deb_cloud.vmdk -netdev tap,id=mynet0,ifname=tap1 -device e1000,netdev=mynet0 -machine pseries-2.7 -m 4G

qemu-system-ppc64 --version
QEMU emulator version 5.1.92 (v5.2.0-rc2-11843-gf571c4ffb5-dirty)
Copyright (c) 2003-2020 Fabrice Bellard and the QEMU Project developers

I have tacked such errors:

oceanfish81@debian:~/dragonffi/release/tests$ ./includes
terminate called after throwing an instance of 'std::logic_error'
what(): basic_string::_M_construct null not valid
Aborted

$./dlopen
Segmentation fault

$ ./compile_error
/__dffi_private/anon_cu_0.c:1:1: error: unknown type name 'this'
this is not valid C code
^
/__dffi_private/anon_cu_0.c:1:8: error: expected ';' after top level declarator
this is not valid C code
^
;

/__dffi_private/anon_cu_1.c:1:1: error: unknown type name 'still'
still invalide code
^
/__dffi_private/anon_cu_1.c:1:15: error: expected ';' after top level declarator
still invalide code
^
;

I hope I didn't forget anything.

Here is what I got:

$ ./compile_error
/__dffi_private/anon_cu_0.c:1:1: error: unknown type name 'this'
this is not valid C code
^
/__dffi_private/anon_cu_0.c:1:8: error: expected ';' after top level declarator
this is not valid C code
^
;

/__dffi_private/anon_cu_1.c:1:1: error: unknown type name 'still'
still invalide code
^
/__dffi_private/anon_cu_1.c:1:15: error: expected ';' after top level declarator
still invalide code
^
;

.
Description of my environment could be obtained here

I am looking to call C functions and access C structs using llvmlite, python bindings for the LLVM API (numba/llvmlite#385).

This seems to be similar to the problem this project is trying to solve. Is it possible to use this package to generate LLVM (or even better build it with llvmlite), instead of actually doing the function calls?

Basically, the use case is that I am wrapping C libraries in Numba and need to be able to call functions from them, create structs, and modify their contents.

Somewhat inevitable, a happy internal user of dragonffi has asked about calling C++ code directly. For example, rather than wrapping member functions with a C wrapper (taking this explicitly) they'd like to new an object and invoke a member.

I imagine you have considered calling C++ code. How big a project would it be to add such support via clang/llvm. Are there showstopping gotchas due to complexity of C++? I'm just trying to understand the scope. Thanks!

Installed pydffi with "pip3 install pydffi" on Debian testing. Seems to be version 0.4.0. Trying to "import pydffi" gives error with Python 3.7.3. I have also installed llvm-dev Debian package (seems to be version 1:7.0-47.1) and llvm (version 1:7.0-47.1). I have also installed llvmlite 0.28.0 with pip3.

~/p/ec/testing>python3
Python 3.7.3 (default, Apr 3 2019, 05:39:12)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

import pydffi
Traceback (most recent call last):
File "", line 1, in
ImportError: /j/home/ylo/.local/lib/python3.7/site-packages/pydffi.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN4llvm32createAggressiveInstCombinerPassEv

I can't install dragonffi with Python 3.8. It does not seems to contain a wheel for python 3.8: https://pypi.org/project/pydffi/#files
(using pip 20.1.1)

Is it possible to create one ?

llvm seemed to build successfully but I hit an error when building dragonffi

cmake -DCMAKE_BUILD_TYPE=release -DLLVM_CONFIG=../llvm/llvm-5.0.1.src/build/bin/llvm-config ..                             (numba-xnd)
-- The C compiler identification is AppleClang 9.1.0.9020039
-- The CXX compiler identification is AppleClang 9.1.0.9020039
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Using llvm-config: /Users/saul/p/dragonffi/llvm/llvm-5.0.1.src/build/bin/llvm-config
-- Performing Test C_SUPPORTS_FPIC
-- Performing Test C_SUPPORTS_FPIC - Success
-- Performing Test CXX_SUPPORTS_FPIC
-- Performing Test CXX_SUPPORTS_FPIC - Success
-- Building with -fPIC
-- Performing Test SUPPORTS_FVISIBILITY_INLINES_HIDDEN_FLAG
-- Performing Test SUPPORTS_FVISIBILITY_INLINES_HIDDEN_FLAG - Success
-- Performing Test C_SUPPORTS_WERROR_DATE_TIME
-- Performing Test C_SUPPORTS_WERROR_DATE_TIME - Success
-- Performing Test CXX_SUPPORTS_WERROR_DATE_TIME
-- Performing Test CXX_SUPPORTS_WERROR_DATE_TIME - Success
-- Performing Test CXX_SUPPORTS_CXX11
-- Performing Test CXX_SUPPORTS_CXX11 - Success
-- Clang resources directory: /Users/saul/p/dragonffi/llvm/llvm-5.0.1.src/build/lib/clang/6.0.0
CMake Error at llvm/llvm-5.0.1.src/cmake/modules/LLVM-Config.cmake:175 (message):
  Target AArch64 is not in the set of libraries.
Call Stack (most recent call first):
  CMakeLists.txt:121 (llvm_map_components_to_libnames)


-- Configuring incomplete, errors occurred!
See also "/Users/saul/p/dragonffi/build/CMakeFiles/CMakeOutput.log".

EDIT: here is the log: https://gist.github.com/saulshanabrook/7c7ca86d26e02a73ea4655e5fd499370

Hello.
Here is what I got:

$./dlopen
Segmentation fault
.
This came from the same build, which was explained in #56

This line
call C functions thought the provided APIs and bindings.
should have through, not thought. i.e.:
call C functions through the provided APIs and bindings.

See https://bugzilla.opensuse.org/show_bug.cgi?id=1180748

Hello Adrien :)

This is rather tough issue (I think), so I'm obviously not asking for a quick resolution.

In DragonFFI, and AFAIK (correct me if I'm wrong), functions that have a callback (function pointer of some sort) need the callback to be in C.

Question

The question is simple, but I doubt the implementation would be (it seems rather complicated, I tried to understand how CPython is doing is but it looks complicated).

• Would it be technically possible to have python callbacks in DragonFFI, rather than compiled C callbacks?

DragonFFI

Below is a DragonFFI example with the CreateThread Windows API, which takes a function pointer as its 3rd parameter:

(note that I'm using some kind of wrapper around dragonFFI to make my life easier, but it's not hard to get).

thread_proc = """
#include <errhandlingapi.h>

DWORD WINAPI MyThreadProc(LPVOID lpParameter) 
{   
    SetLastError(0x1337);
    return 0;
}
"""

def main():
    # init stuff for dragonFFI wrapper
    include_base = pathlib.Path(r"C:\Program Files (x86)\Windows Kits\10\Include\10.0.19041.0")
    ffi = DragonFFI()
    ffi.include_bases = [include_base, ]
    ffi.set_includes(["windef.h", "processthreadsapi.h"])
    ffi.cdef(additional_text=thread_proc)

    nullptr = ffi.types.LPVOID()

    # pointer to thread function.
    pthreadproc = pydffi.ptr(ffi.funcs.MyThreadProc)

    ffi.funcs.SetLastError(0x42)

    input("press <enter> to call CreateThread.")
    err = ffi.funcs.GetLastError()
    print(f"LastErr (windows): {err.value:#x}")

    # call thread
    ffi.funcs.CreateThread(nullptr, 0, pthreadproc, nullptr, 0, nullptr)
    dffi_err = pydffi.getLastError()
    print(f"LastErr (pydffi): {dffi_err:#x}")

Debugged it with a native debugger, and everything's working as expected.

Python ctypes

Python ctypes has the possibility to wrap a python function so it can be called by a ctypes function. Below is a slightly different example, but still with CreateThread (see the wrapped function pointer called LPTHREAD_START_ROUTINE):

def my_thread(parameter):
    k32 = ctypes.WinDLL("kernel32", use_last_error=True)
    get_current_thread_id = k32.GetCurrentThreadId
    get_current_thread_id.argtypes = ()
    get_current_thread_id.restype = ctypes.c_uint32

    tid = get_current_thread_id()
    print(f"[THREAD] I'm a thread! My TID is: {tid:#x}")

    # wait a bit.
    sleep(1.0)

    print("[THREAD] Exiting.")

    return 0


def test_ctypes():
    k32 = ctypes.WinDLL("kernel32", use_last_error=True)

    # LPTHREAD_START_ROUTINE passed as 3rd arg to CreateThread.
    LPTHREAD_START_ROUTINE = ctypes.WINFUNCTYPE(
        ctypes.c_uint32,  # return type
        ctypes.c_void_p  # param type.
    )

    # CreateThread
    create_thread = k32.CreateThread
    create_thread.argtypes = (
        ctypes.c_void_p,
        ctypes.c_size_t,
        LPTHREAD_START_ROUTINE,  # callback
        ctypes.c_void_p,
        ctypes.c_uint32,
        ctypes.POINTER(ctypes.c_uint32)
    )
    create_thread.restype = ctypes.c_void_p

    # call CreateThread
    callback_func = LPTHREAD_START_ROUTINE(my_thread)
    tid = ctypes.c_uint32()
    create_thread(None, 0, callback_func, None, 0, ctypes.byref(tid))

    print(f"[MAIN] created TID: {tid.value:#x}")

    sleep(2.0)

    print("[MAIN] Exiting")

    return 0

Some pointers

• Python implementation of WINFUNCTYPE ; seems to be a wrapper around _CFuncPtr.
• _CFuncPtr defined as from _ctypes import CFuncPtr as _CFuncPtr at the top of ctypes.py.
• CFuncPtr defined here in ctypes.c.

I'm not sure of the whole inner workings machinery behind CFuncPtr...

Sorry for the looooong issue, Thanks a lot!

Hi folks,

Support for js should be cool, any plan?

Tks,
Cong

Hi Adrien :)

I'm really enjoying doing things with dragonffi on Windows. I have hit a roadblock and don't know exactly how to deal with it (although I did find a way to circumvent the problem).

Intro

I'm calling the NtQuerySystemInformation function with the undocumented SystemBigPoolInformation InfoClass (0x42).

The call looks like this:

r = ffi.NtQuerySystemInformation(
    0x42,  # info class
    pbuffer,  # output buffer
    0x1000,  # size of the output buffer
    pout)  # returned size

The structure returned by the API is a SYSTEM_BIGPOOL_INFORMATION and it looks like this:

typedef struct _SYSTEM_BIGPOOL_ENTRY 
{
	union {
		PVOID VirtualAddress;
		ULONG_PTR NonPaged : 1;
	};
	ULONG_PTR SizeInBytes;
	union {
		UCHAR Tag[4];
		ULONG TagUlong;
	};
} SYSTEM_BIGPOOL_ENTRY, *PSYSTEM_BIGPOOL_ENTRY;

typedef struct _SYSTEM_BIGPOOL_INFORMATION {
	ULONG Count;
	SYSTEM_BIGPOOL_ENTRY AllocatedInfo[ANYSIZE_ARRAY];
} SYSTEM_BIGPOOL_INFORMATION, *PSYSTEM_BIGPOOL_INFORMATION;

The Count field tells how much SYSTEM_BIGPOOL_ENTRY there are in the AllocatedInfo field. So far, so good in python:

    # cast pointer to buffer to _SYSTEM_BIGPOOL_INFORMATION pointer 
    sbi = pydffi.cast(pydffi.ptr(buffer), ffi.types.PSYSTEM_BIGPOOL_INFORMATION)
    print("SYSTEM_BIGPOOL_INFORMATION count: {0} ({0:#x})".format(sbi.obj.Count))

    # get the allocated info array
    allocated_info = sbi.obj.AllocatedInfo

Problem

Now I can't access any of the unnamed unions field in the SYSTEM_BIGPOOL_ENTRY structure:

    # doesn't work...
    print(f"SYSTEM_BIGPOOL_INFORMATION: {allocated_info[0].VirtualAddress.value:#x}")
    print(f"SYSTEM_BIGPOOL_INFORMATION: {allocated_info[1].VirtualAddress.value:#x}")
    print(f"SYSTEM_BIGPOOL_INFORMATION: {allocated_info[2].VirtualAddress.value:#x}")

Workaround

If I change the structure to have named unions it does work without any problems:

C:

    typedef struct _SYSTEM_BIGPOOL_ENTRY 
    {
        union {
            PVOID VirtualAddress;
            ULONG_PTR NonPaged : 1;
        }a;  // name
        ULONG_PTR SizeInBytes;
        union {
            UCHAR Tag[4];
            ULONG TagUlong;
        }b;  // name
    } SYSTEM_BIGPOOL_ENTRY, *PSYSTEM_BIGPOOL_ENTRY;

python:

    print(f"allocated_info[0].VA: {allocated_info[0].a.VirtualAddress.value:#x}")
    print(f"allocated_info[1].VA: {allocated_info[1].a.VirtualAddress.value:#x}")
    print(f"allocated_info[2].VA: {allocated_info[2].a.VirtualAddress.value:#x}")

Question

Is there any way to deal with unnamed unions / structures or do I explicitly have to name them (I guess unnamed union / structures is a MS extension?)

Thanks a lot!

Firstly - thx for sharing this project, must have been a large amount of work!

I'm on Monterey with an M1 - python 3.9 running in Pycharm.

I think I've built it okay - I end up a single .so .../dragonffi/build/bindings/python/pydffi.so.

I export LLVM_CONFIG=/opt/homebrew/opt/llvm@11/bin/llvm-config but when I call import pydffi I get:

Fatal Python error: take_gil: PyMUTEX_LOCK(gil->mutex) failed
Python runtime state: unknown

Process finished with exit code 134 (interrupted by signal 6: SIGABRT)

Looking for some help getting this working if poss. Thx

$ make -j32
[ 1%] Building CXX object CMakeFiles/dffi.dir/lib/dffi_llvm_wrapper.cpp.o
[ 3%] Building CXX object CMakeFiles/dffi.dir/lib/dffi_impl_clang.cpp.o
[ 4%] Building CXX object CMakeFiles/dffi.dir/lib/dffi_impl.cpp.o
/home/oceanfish81/dragonffi/dragonffi/lib/dffi_llvm_wrapper.cpp:34:42: error: cannot initialize a parameter of type 'llvm::CloneFunctionChangeType' with an rvalue of type 'bool'
llvm::CloneFunctionInto(NewF, F, VMap, true, Returns);
^~~~
/usr/local/include/llvm/Transforms/Utils/Cloning.h:159:73: note: passing argument to parameter 'Changes' here
ValueToValueMapTy &VMap, CloneFunctionChangeType Changes,
^
/home/oceanfish81/dragonffi/dragonffi/lib/dffi_impl_clang.cpp:94:8: warning: comparison of different enumeration types in switch statement ('clang::CallingConv' and 'dffi::CallingConv') [-Wenum-compare-switch]
case CC_PreserveAll:
^~~~~~~~~~~~~~
/home/oceanfish81/dragonffi/dragonffi/lib/dffi_impl_clang.cpp:91:8: warning: comparison of different enumeration types in switch statement ('clang::CallingConv' and 'dffi::CallingConv') [-Wenum-compare-switch]
case CC_PreserveMost:
^~~~~~~~~~~~~~~
/home/oceanfish81/dragonffi/dragonffi/lib/dffi_impl_clang.cpp:88:8: warning: comparison of different enumeration types in switch statement ('clang::CallingConv' and 'dffi::CallingConv') [-Wenum-compare-switch]
case CC_Swift:
^~~~~~~~
/home/oceanfish81/dragonffi/dragonffi/lib/dffi_impl_clang.cpp:85:8: warning: comparison of different enumeration types in switch statement ('clang::CallingConv' and 'dffi::CallingConv') [-Wenum-compare-switch]
case CC_OpenCLKernel:
^~~~~~~~~~~~~~~
/home/oceanfish81/dragonffi/dragonffi/lib/dffi_impl_clang.cpp:84:8: warning: comparison of different enumeration types in switch statement ('clang::CallingConv' and 'dffi::CallingConv') [-Wenum-compare-switch]
case CC_SpirFunction:
^~~~~~~~~~~~~~~
/home/oceanfish81/dragonffi/dragonffi/lib/dffi_impl_clang.cpp:81:8: warning: comparison of different enumeration types in switch statement ('clang::CallingConv' and 'dffi::CallingConv') [-Wenum-compare-switch]
case CC_X86RegCall:
^~~~~~~~~~~~~
/home/oceanfish81/dragonffi/dragonffi/lib/dffi_impl_clang.cpp:78:8: warning: comparison of different enumeration types in switch statement ('clang::CallingConv' and 'dffi::CallingConv') [-Wenum-compare-switch]
case CC_X86_64SysV:
^~~~~~~~~~~~~
/home/oceanfish81/dragonffi/dragonffi/lib/dffi_impl_clang.cpp:75:8: warning: comparison of different enumeration types in switch statement ('clang::CallingConv' and 'dffi::CallingConv') [-Wenum-compare-switch]
case CC_Win64:
^~~~~~~~
/home/oceanfish81/dragonffi/dragonffi/lib/dffi_impl_clang.cpp:72:8: warning: comparison of different enumeration types in switch statement ('clang::CallingConv' and 'dffi::CallingConv') [-Wenum-compare-switch]
case CC_IntelOclBicc:
^~~~~~~~~~~~~~~
/home/oceanfish81/dragonffi/dragonffi/lib/dffi_impl_clang.cpp:69:8: warning: comparison of different enumeration types in switch statement ('clang::CallingConv' and 'dffi::CallingConv') [-Wenum-compare-switch]
case CC_AAPCS_VFP:
^~~~~~~~~~~~
/home/oceanfish81/dragonffi/dragonffi/lib/dffi_impl_clang.cpp:66:8: warning: comparison of different enumeration types in switch statement ('clang::CallingConv' and 'dffi::CallingConv') [-Wenum-compare-switch]
case CC_AAPCS:
^~~~~~~~
/home/oceanfish81/dragonffi/dragonffi/lib/dffi_impl_clang.cpp:63:8: warning: comparison of different enumeration types in switch statement ('clang::CallingConv' and 'dffi::CallingConv') [-Wenum-compare-switch]
case CC_X86Pascal:
^~~~~~~~~~~~
/home/oceanfish81/dragonffi/dragonffi/lib/dffi_impl_clang.cpp:60:8: warning: comparison of different enumeration types in switch statement ('clang::CallingConv' and 'dffi::CallingConv') [-Wenum-compare-switch]
case CC_X86VectorCall:
^~~~~~~~~~~~~~~~
/home/oceanfish81/dragonffi/dragonffi/lib/dffi_impl_clang.cpp:57:8: warning: comparison of different enumeration types in switch statement ('clang::CallingConv' and 'dffi::CallingConv') [-Wenum-compare-switch]
case CC_X86ThisCall:
^~~~~~~~~~~~~~
/home/oceanfish81/dragonffi/dragonffi/lib/dffi_impl_clang.cpp:54:8: warning: comparison of different enumeration types in switch statement ('clang::CallingConv' and 'dffi::CallingConv') [-Wenum-compare-switch]
case CC_X86FastCall:
^~~~~~~~~~~~~~
/home/oceanfish81/dragonffi/dragonffi/lib/dffi_impl_clang.cpp:51:8: warning: comparison of different enumeration types in switch statement ('clang::CallingConv' and 'dffi::CallingConv') [-Wenum-compare-switch]
case CC_X86StdCall:
^~~~~~~~~~~~~
/home/oceanfish81/dragonffi/dragonffi/lib/dffi_impl_clang.cpp:42:8: warning: comparison of different enumeration types in switch statement ('clang::CallingConv' and 'dffi::CallingConv') [-Wenum-compare-switch]
case CC_C:
^~~~
/home/oceanfish81/dragonffi/dragonffi/lib/dffi_impl_clang.cpp:41:11: warning: enumeration value 'CC_AArch64VectorCall' not handled in switch [-Wswitch]
switch (Info.getCC()) {
^
/home/oceanfish81/dragonffi/dragonffi/lib/dffi_impl.cpp:183:21: error: no matching constructor for initialization of 'driver::Driver'
Driver_.reset(new driver::Driver{"dummy", TripleStr, *Diags_, Overlay});
^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/local/include/clang/Driver/Driver.h:310:3: note: candidate constructor not viable: no known conversion from 'IntrusiveRefCntPtrllvm::vfs::OverlayFileSystem' to 'std::string' (aka 'basic_string') for 4th argument
Driver(StringRef ClangExecutable, StringRef TargetTriple,
^
/usr/local/include/clang/Driver/Driver.h:59:7: note: candidate constructor (the implicit copy constructor) not viable: requires 1 argument, but 4 were provided
class Driver {
^
/usr/local/include/clang/Driver/Driver.h:59:7: note: candidate constructor (the implicit move constructor) not viable: requires 1 argument, but 4 were provided
/home/oceanfish81/dragonffi/dragonffi/lib/dffi_impl.cpp:235:7: error: no member named 'ThreadModel' in 'clang::CodeGenOptions'
CGO.ThreadModel = "posix";

1 error generated.
make[2]: *** [CMakeFiles/dffi.dir/build.make:80: CMakeFiles/dffi.dir/lib/dffi_llvm_wrapper.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
18 warnings generated.
2 errors generated.
make[2]: *** [CMakeFiles/dffi.dir/build.make:93: CMakeFiles/dffi.dir/lib/dffi_impl.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:235: CMakeFiles/dffi.dir/all] Error 2
make: *** [Makefile:84: all] Error 2

$ uname -a
Linux ams1-c2-large-arm-01 5.4.0-40-generic

Hi Adrien :)

Would it be difficult to have a mechanism for preserving the "last error" for different systems / runtimes?

For example the Windows API relies heavily on APIs returning a NULL / FALSE / 0 / -1 value on error. So if anything went wrong while calling an API, the developer must use the GetLastError API to know what went really wrong and decide accordingly on how to handle the error. The "last error" (which is per-thread on Windows) is overridden almost every time an API is called (this also includes system calls).

Obviously a similar system is also used for errno.

While using dragonffi there's no guarantee that this "last error" has been preserved, so if anything goes wrong it might be difficult for the caller to actually get a sensible feedback from the system / runtime.

ctypes offers an ad hoc mechanism in the form of ctypes.get_last_error for Windows APIs and ctypes.get_errno for errno.

Do you think there's a possibility for such a mechanism to be implemented in dragonffi?

Thanks!

Traceback (most recent call last):
File "<pyshell#269>", line 1, in
import pydffi
ImportError: /opt/intelpython3/lib/python3.6/site-packages/pydffi.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN4llvm32createAggressiveInstCombinerPassEv

I am not sure where to go with this. Does anyone know where I might start ... ?