Hello,

This feature request is a spin-off off #50.

I was wondering if something like this can be made to work:

import nimterop/cimport

# ...

const
  xlmIncludePath = getEnv("XCELIUM_ROOT") / ".." / "include"
static:
  doAssert fileExists(xlmIncludePath / "svdpi.h")
  doAssert fileExists(xlmIncludePath / "svdpi_compatibility.h")

const
  headersvdpiLocal = xlmIncludePath / "svdpi.h"
type
  svOpenArrayHandle* = pointer
proc svGetArrElemPtr*(a1: svOpenArrayHandle; indx1: cint): pointer {.
    importc: "svGetArrElemPtr", header: headersvdpiLocal.}
proc svGetArrElemPtr1*(a1: svOpenArrayHandle; indx1: cint): pointer {.
    importc: "svGetArrElemPtr1", header: headersvdpiLocal.}
proc svGetArrElemPtr2*(a1: svOpenArrayHandle; indx1: cint; indx2: cint): pointer {.
    importc: "svGetArrElemPtr2", header: headersvdpiLocal.}
proc svGetArrElemPtr3*(a1: svOpenArrayHandle; indx1: cint; indx2: cint; indx3: cint): pointer {.
    importc: "svGetArrElemPtr3", header: headersvdpiLocal.}

cAddSearchDir(xlmIncludePath)
cImport(cSearchPath("svdpi.h"))

The idea is that cImport skips defining svOpenArrayHandle, svGetArrElemPtr, etc. which already got defined before cImport got called.

Without that, right now I get the below expected error:

/home/kmodi/stow/pkgs/nim/devel/lib/core/macros.nim(528, 3) Error: redefinition of 'svOpenArrayHandle'; previous declaration here: ../../svdpi.nim(22, 3)

So nimterop needs to check if an identifier is already defined, and proceed adding its definition only if that identifier doesn't already exist. Is that possible?

void
  bar(void),
  *foo(void),
  baz(void);

proc bar*() {.importc: "bar", header: headertest.}
proc foo*(): pointer {.importc: "foo", header: headertest.}
proc baz*(): pointer {.importc: "baz", header: headertest.}

As you see above in the Nim output, all functions declared after a function that returns a pointer now also erroneously return a pointer.

(translation_unit 1 1 44
 (declaration 1 1 44
  (primitive_type 1 1 4)
  (function_declarator 2 3 9
   (identifier 2 3 3)
   (parameter_list 2 6 6
    (parameter_declaration 2 7 4
     (primitive_type 2 7 4)
    )
   )
  )
  (pointer_declarator 3 3 10
   (function_declarator 3 4 9
    (identifier 3 4 3)
    (parameter_list 3 7 6
     (parameter_declaration 3 8 4
      (primitive_type 3 8 4)
     )
    )
   )
  )
  (function_declarator 4 3 9
   (identifier 4 3 3)
   (parameter_list 4 6 6
    (parameter_declaration 4 7 4
     (primitive_type 4 7 4)
    )
   )
  )
 )
)

[EDIT] still trying to make up my mind whether this is an actual bug or not; maybe the correct answer is for user to set both PATH and NIMBLE_DIR to be consistent; need to think

nimble develop in a nimterop clone shd work in isolation (ie not pickup an external toast since toast is built by this package)

problem 1

export NIMBLE_DIR=$HOME/.nimble_fake4
git clone https://github.com/genotrance/nimterop
cd nimterop
nimble develop
nimble build
nimble test

   Warning: Using env var NIM_LIB_PREFIX: /Users/timothee/git_clone//nim//Nim/
  Executing task test in /private/tmp/d34/nimterop/nimterop.nimble
nim c -r tests/tnimterop_c.nim
Hint: used config file '/Users/timothee/git_clone/nim/Nim/config/nim.cfg' [Conf]
Hint: used config file '/Users/timothee/.config/nim/nim.cfg' [Conf]
Hint: used config file '/Users/timothee/git_clone/nim/Nim/config/config.nims' [Conf]
/private/tmp/d34/nimterop/tests/tnimterop_c.nim(2, 16) Error: cannot open file: nimterop/cimport
stack trace: (most recent call last)
/private/tmp/d34/nimterop/nimterop.nimble(24) testTask
/private/tmp/d34/nimterop/nimterop.nimble(17) execCmd
/Users/timothee/git_clone/nim/Nim/lib/system/nimscript.nim(241) exec
/Users/timothee/git_clone/nim/Nim/lib/system/nimscript.nim(241, 7) Error: unhandled exception: FAILED: nim c -r tests/tnimterop_c.nim

problem: toast is generated in ./toast and reffered to as toast so isn't picked up
(and err msg doesn't tell u that, needs debugging)

problem 2

if we omit:

export NIMBLE_DIR=$HOME/.nimble_fake4

then if a user has a toast already built in /Users/timothee/.nimble//bin/toast that one will be picked up instead of the one we're building

this leads to hard to diagnose errors

solution

nimble test and other commands shd be smart about how to refer to toast, maybe via an absolute path computed from stuff like:

• getCurrentSource etc

it should "do the right thing" regardless nimble install or nimble develop is used

workaround

• nimble install to update /Users/timothee/.nimble//bin/toast then nimble develop again

typedef struct foo { int foo[8]; };

toast --preprocess --pnim /tmp/test.c

{.experimental: "codeReordering".}
SIGSEGV: Illegal storage access. (Attempt to read from nil?)

toast --preprocess --past /tmp/test.c

(translation_unit 1 1 35
 (type_definition 1 1 35
  (struct_specifier 1 9 26
   (type_identifier 1 16 3)
   (field_declaration_list 1 20 15
    (field_declaration 1 22 11
     (primitive_type 1 22 3)
     (array_declarator 1 26 6
      (field_identifier 1 26 3)
      (number_literal 1 30 1)
     )
    )
   )
  )
  (type_identifier 1 35 0)
 )
)
��z�#d�z�#d8�z�#d��XGV)x�z�#��XGVEMPT��XGVI��z�#d@�z�#d�z�#d��z�#�z�#d��z�#d��z�#��XGVEMPTY��z�#d��z�#d��z�#d��XGVprelude��XGVdoc��XGVcommand��XGVmain��XGVargs��XGVoptionsH�z�#d@�z�Error: unhandled exception: cannot write string to file [IOError]

The following 2 definitions from SDL2 cause a clash in Nim namespace:

typedef enum
{
    ...
} SDL_GLContextResetNotification;

typedef enum
{
    ...
    SDL_GL_CONTEXT_RESET_NOTIFICATION,
    ....
} SDL_GLattr;

same input example as #71

Hint: conversion from FREE_IMAGE_FORMAT to itself is pointless [ConvFromXtoItselfNotNeeded]

comes from this:

type FREE_IMAGE_FORMAT* = distinct int
converter enumToInt(en: FREE_IMAGE_FORMAT): int {.used.} = en.int

for reference: https://gitter.im/nimgen/Lobby?at=5bf5dfa087c4b86bccf7de41

nimble install 'http://github.com/genotrance/nimtreesitter?subdir=treesitter'
Downloading http://github.com/genotrance/nimtreesitter using git (subdir is 'treesitter')
Setting up Git repo
Checking out artifacts
Resetting Git repo
File doesn't exist: utf8proc_data.c
stack trace: (most recent call last)
/private/tmp/nimble_87594/githubcom_genotrancenimtreesitter/treesitter/treesitter.nimble(28) installBefore
/private/tmp/nimble_87594/githubcom_genotrancenimtreesitter/treesitter/treesitter.nimble(25) setupTask
/Users/timothee/git_clone/nim/Nim/lib/system/nimscript.nim(237) exec
/Users/timothee/git_clone/nim/Nim/lib/system/nimscript.nim(237, 7) Error: unhandled exception: FAILED: nimgen treesitter.cfg

note: 2nd time, it seemed to work; not sure what it failed 1st time

• use assert/doAssert/raise instead of quit (https://forum.nim-lang.org/t/4042)

• use quoteShell everywhere relevant; adding " is not always correct and also read uglier eg:

      {.passC: "-I\"/Users/foo/include\"".}
     #  whereas quoteShell would, here, give simply
      {.passC: "-I/Users/foo/include".}

• group all globals in nimterop/globals.nim into a single object

• pass global to procs that need it instead of having globals

• each un-handled AST element should call a (user-settable) callback instead of just doing nothing, eg via
  else: return and else: discard

• tests: use unittest check,require instead of assert or doAssert

• use type AstRef = ref Ast instead of ref Ast everywhere (removed the Ast altogether)

• inconsistent: sometimes you use default c, sometimes default cpp;

proc process(path: string, mode="cpp", pretty = true)
macro cIncludeCcAddStdDir*(mode = "c")

ideally, you'd have a single default define once (DRY)

let modeDefault = "c" 
proc process(path: string, mode=modeDefault, pretty = true)
macro cIncludeCcAddStdDir*(mode = modeDefault)

controversial

• generate translations by operating on NimNode instead of string; it's less error prone; parseStmt can always be used when needed (also can be used during this transition)
  pFunctionDeclarator for example is a bit ugly using string manipulations

EDIT: actually a proc with NimNode, IIRC, has to be compileTime only ; that would be a limitation compared to using strings which have flexibility of being usable at either runtime or compiletime; need to think more...
although maybe Nim could be patched to allow runtime NimNode, from which we can get a string (eg via repr) in case user wants a generated nim code at runtime

genotrance: now that toast does all of it, this isn't possible anymore.

feedback that maybe belongs to treesitter

• class seems wrongly accepted for C header files

genotrance; not supported yet by toast - C++ can be tracked separately.

comparison with calypso approach

• can nimterop eventually allow mapping C++ function templates to nim generic procs ? eg, mapping std::vector<T>
  that's feasible with https://wiki.dlang.org/Calypso in D

genotrance: need to see what tree-sitter generates in this case but should be feasible

• some ideas from https://github.com/Syniurge/Calypso/issues could be reused

scratch below here

• option to return generated nim code as a string (user can then do whatever, eg dump it to stdout)
• option add line info (pointing to .h sources) as doc comment for each translated declaration
• option add debug line info (pointing to .h sources) (eg as dwarf?) as done in Nim code, which allows you to have a nice debug experience under gdb/lldb
• allow (as option) specifying language, eg: cImport("foo.h", lang = c) ; cImport("foo.h", lang = cpp)
• allow cFlag(“-Dfoo -Dbar -Ifoobar ”) instead of cDefine (and maybe even cIncludeDir ) etc

cImport() already has flags which can do this

• can nimterop replace py2nim? cf https://forum.nim-lang.org/t/2705

No plans to cover Python - already pyimport and nimpy exist

• how can treesitter correctly disambiguate C++ code that depends on semantic analysis, eg : << which could mean operator << or double < template

Will see how this works once we start C++ work

links

• https://news.ycombinator.com/item?id=9640912

you can even build a Nim macro that does a "Go get" and Nim developers can use Golang libraries right in their code as imports

Likewise with Go, will track separately if we ever get to it.

after instructions in README:
after this:
nimble install http://github.com/genotrance/nimterop

Error: Build failed for package: nimterop
        ... Details:
        ... Execution failed with exit code 1
        ... Command: "/Users/timothee/.nimble//bin/nim" c --noBabelPath -d:release --path:"/Users/timothee/git_clone/nim/nimtreesitter/treesitter"  --path:"/Users/timothee/.nimble/pkgs/nimgen-0.5.0"  --path:"/Users/timothee/.nimble/pkgs/c2nim-#3ec45c24585ebaed"  --path:"/Users/timothee/git_clone/nim/Nim"  --path:"/Users/timothee/.nimble/pkgs/regex-0.10.0"  --path:"/Users/timothee/.nimble/pkgs/unicodedb-0.6.0"  --path:"/Users/timothee/.nimble/pkgs/unicodeplus-0.4.0"  --path:"/Users/timothee/.nimble/pkgs/unicodedb-0.6.0"  --path:"/Users/timothee/git_clone/nim/nimtreesitter/treesitter_c"  --path:"/Users/timothee/.nimble/pkgs/nimgen-0.5.0"  --path:"/Users/timothee/.nimble/pkgs/c2nim-#3ec45c24585ebaed"  --path:"/Users/timothee/git_clone/nim/Nim"  --path:"/Users/timothee/.nimble/pkgs/regex-0.10.0"  --path:"/Users/timothee/.nimble/pkgs/unicodedb-0.6.0"  --path:"/Users/timothee/.nimble/pkgs/unicodeplus-0.4.0"  --path:"/Users/timothee/.nimble/pkgs/unicodedb-0.6.0"  --path:"/Users/timothee/git_clone/nim/nimtreesitter/treesitter"  --path:"/Users/timothee/.nimble/pkgs/nimgen-0.5.0"  --path:"/Users/timothee/.nimble/pkgs/c2nim-#3ec45c24585ebaed"  --path:"/Users/timothee/git_clone/nim/Nim"  --path:"/Users/timothee/.nimble/pkgs/regex-0.10.0"  --path:"/Users/timothee/.nimble/pkgs/unicodedb-0.6.0"  --path:"/Users/timothee/.nimble/pkgs/unicodeplus-0.4.0"  --path:"/Users/timothee/.nimble/pkgs/unicodedb-0.6.0"  --path:"/Users/timothee/git_clone/nim/nimtreesitter/treesitter_cpp"  --path:"/Users/timothee/.nimble/pkgs/nimgen-0.5.0"  --path:"/Users/timothee/.nimble/pkgs/c2nim-#3ec45c24585ebaed"  --path:"/Users/timothee/git_clone/nim/Nim"  --path:"/Users/timothee/.nimble/pkgs/regex-0.10.0"  --path:"/Users/timothee/.nimble/pkgs/unicodedb-0.6.0"  --path:"/Users/timothee/.nimble/pkgs/unicodeplus-0.4.0"  --path:"/Users/timothee/.nimble/pkgs/unicodedb-0.6.0"  --path:"/Users/timothee/git_clone/nim/nimtreesitter/treesitter"  --path:"/Users/timothee/.nimble/pkgs/nimgen-0.5.0"  --path:"/Users/timothee/.nimble/pkgs/c2nim-#3ec45c24585ebaed"  --path:"/Users/timothee/git_clone/nim/Nim"  --path:"/Users/timothee/.nimble/pkgs/regex-0.10.0"  --path:"/Users/timothee/.nimble/pkgs/unicodedb-0.6.0"  --path:"/Users/timothee/.nimble/pkgs/unicodeplus-0.4.0"  --path:"/Users/timothee/.nimble/pkgs/unicodedb-0.6.0"  --path:"/Users/timothee/.nimble/pkgs/regex-0.10.0"  --path:"/Users/timothee/.nimble/pkgs/unicodedb-0.6.0"  --path:"/Users/timothee/.nimble/pkgs/unicodeplus-0.4.0"  --path:"/Users/timothee/.nimble/pkgs/unicodedb-0.6.0"  -o:"/tmp/nimble_16894/githubcom_genotrancenimterop/toast" "/tmp/nimble_16894/githubcom_genotrancenimterop/toast.nim"
        ... Output: Hint: used config file '/Users/timothee/git_clone/nim/Nim/config/nim.cfg' [Conf]
        ... Hint: used config file '/Users/timothee/.config/nim/nim.cfg' [Conf]
        ... Hint: used config file '/Users/timothee/.config/nim/config.nims' [Conf]
        ... /Users/timothee/.config/nim/config.nims [config.nims used]
        ... Hint: used config file '/private/tmp/nimble_16894/githubcom_genotrancenimterop/config.nims' [Conf]
        ... Error: execution of an external compiler program 'clang -c  -w -I"/Users/timothee/git_clone/nim/nimtreesitter/treesitter/treesitter/include" -I"/Users/timothee/git_clone/nim/nimtreesitter/treesitter/treesitter/include/tree_sitter" -I"/Users/timothee/git_clone/nim/nimtreesitter/treesitter/treesitter/src" -I"/Users/timothee/git_clone/nim/nimtreesitter/treesitter/treesitter/utf8proc" -DUTF8PROC_STATIC -std=gnu++14 -O3  -I/Users/timothee/git_clone/nim/Nim/lib -o /Users/timothee/.cache/nim/toast_r/parsercpp.o /Users/timothee/git_clone/nim/nimtreesitter/treesitter_cpp/treesitter/../treesitter/cpp/src/parser.c' failed with exit code: 1
        ... error: invalid argument '-std=gnu++14' not allowed with 'C'

as discussed here https://gitter.im/nimgen/Lobby?at=5bf638386621313894fbddac

links

https://stackoverflow.com/questions/2224334/gcc-dump-preprocessor-defines

i think cIncludeDir("include”) is being mis-used; IMO cCompile and cImport should be given actual paths where sources (h and c) live, and cIncludeDir would only be used to resolve where to find bar.h if a h or c file contains #include bar.h

Note: further discussed here: https://gitter.im/nimgen/Lobby?at=5bf5f541a115c91ef77149b9

Wondering how much work it'd be to abstract away the source AST to allow using clang's AST instead of treesitter's AST

clang -Xclang -ast-dump -fno-color-diagnostics -fsyntax-only tests/include/test.h

clang -Xclang -ast-dump -fno-color-diagnostics -fsyntax-only tests/include/test.h
TranslationUnitDecl 0x7f8d3882bce8 <<invalid sloc>> <invalid sloc>
|-TypedefDecl 0x7f8d3882c260 <<invalid sloc>> <invalid sloc> implicit __int128_t '__int128'
| `-BuiltinType 0x7f8d3882bf80 '__int128'
|-TypedefDecl 0x7f8d3882c2d0 <<invalid sloc>> <invalid sloc> implicit __uint128_t 'unsigned __int128'
| `-BuiltinType 0x7f8d3882bfa0 'unsigned __int128'
|-TypedefDecl 0x7f8d3882c5a8 <<invalid sloc>> <invalid sloc> implicit __NSConstantString 'struct __NSConstantString_tag'
| `-RecordType 0x7f8d3882c3b0 'struct __NSConstantString_tag'
|   `-Record 0x7f8d3882c328 '__NSConstantString_tag'
|-TypedefDecl 0x7f8d3882c640 <<invalid sloc>> <invalid sloc> implicit __builtin_ms_va_list 'char *'
| `-PointerType 0x7f8d3882c600 'char *'
|   `-BuiltinType 0x7f8d3882bd80 'char'
...

motivation

dealing with treesitter's shortcomings, especially wrt lack of semantic analysis making it hard to deal with C++ templates etc

note

it may be even easier to write a binary using libclang that outputs the AST in json directly instead of having to custom parse it in nimterop

typedef struct _Kernel { char name; } Kernel;

type
  Kernel* {.importc: "struct _Kernel", header: headertest, bycopy.} = object
    name*: cchar

(translation_unit 1 1 45
 (type_definition 1 1 45
  (struct_specifier 1 9 29
   (type_identifier 1 16 7)
   (field_declaration_list 1 24 14
    (field_declaration 1 26 10
     (primitive_type 1 26 4)
     (field_identifier 1 31 4)
    )
   )
  )
  (type_identifier 1 39 6)
 )
)

If Kernel is a pointer, the output breaks.

typedef struct _Kernel { char name; } *Kernel;

(translation_unit 1 1 46
 (type_definition 1 1 46
  (struct_specifier 1 9 29
   (type_identifier 1 16 7)
   (field_declaration_list 1 24 14
    (field_declaration 1 26 10
     (primitive_type 1 26 4)
     (field_identifier 1 31 4)
    )
   )
  )
  (pointer_declarator 1 39 7
   (type_identifier 1 40 6)
  )
 )
)

No Nim output.

typedef unsigned long long ULL;

gets translated to

type
  ULL* = cuclonglong

But it should be culonglong.
Possibly due to the naive translation from long to clong here
https://github.com/genotrance/nimterop/blob/4d57be55ac5e0be8870bd7479e08448a079da2be/nimterop/getters.nim#L54-L58

C features

• handling nested struct

• maybe relevant: [TODO] Nim should allow nested types · Issue #7449 · nim-lang/Nim

C++ features

• template, simple case:
  template<typename T> class Foo{}; typedef Foo<int> FooInt

Hello,

I am using nimterop built from master and nim built from devel.

I am using nimterop this way:

import nimterop/cimport

cDebug()

# Below doesn't work
cIncludeDir("/cad/adi/apps/cadence/xlm/linux/current/tools/include")
# Below doesn't work
# cIncludeDir(/cad/adi/apps/cadence/xlm/linux/current/tools/include)
# Below doesn't work
# {.passC: "-I/cad/adi/apps/cadence/xlm/linux/current/tools/include".}

# Below manual definition of s_vpi_vecval is a workaround for
# https://github.com/genotrance/nimterop/issues/47.
type
  s_vpi_vecval* {.importc: "s_vpi_vecval".} = object
    aval: uint32
    bval: uint32

cImport("svdpi.h")

The svdpi.h and the other header file needed are in that directory path that I have in the above example, but still they cannot be found. I get this error:

{.passC: "-I\"/cad/adi/apps/cadence/xlm/linux/current/tools/include\"".}
stack trace: (most recent call last)
../../../../.nimble/pkgs/nimterop-0.1.0/nimterop/cimport.nim(232, 24) cImport
../../../../.nimble/pkgs/nimterop-0.1.0/nimterop/cimport.nim(21, 11) findPath
../../../../stow/pkgs/nim/devel/lib/system.nim(3989, 20) failedAssertImpl
../../../../stow/pkgs/nim/devel/lib/system.nim(3982, 11) raiseAssert
../../../../stow/pkgs/nim/devel/lib/system.nim(3016, 7) sysFatal
svdpi.nim(16, 8) template/generic instantiation of `cImport` from here
../../../../stow/pkgs/nim/devel/lib/system.nim(3016, 7) Error: unhandled exception: /home/kmodi/.nimble/pkgs/nimterop-0.1.0/nimterop/cimport.nim(21, 5) `(not fail)` File or directory not found: svdpi.h

The only way I can get the compilation to work is if I copy the .h files and put them in the directory containing the above .nim file.

Am I missing something in the cIncludeDir syntax?

toast clashes with an ffmpeg dependency for GSM compression, so machines with an improperly setup path would cause nimterop to fail with a strange error message complaining about invalid arguments.

Maybe c2ast would be a fitting name? If nimterop plans to support more languages than C/C++ that would be inaccurate though. astgen or genast perhaps? ast2nim?

// uncomment this and no ERROR will be generated
// #whateverstuff TEST_INT 512

asdfasdf

before uncommenting

(translation_unit 0 45
 (comment 0 3)
 (comment 4 34)
 (ERROR 36 44
  (identifier 36 44)
 )
)

after uncommenting

(translation_unit 0 42
 (comment 0 3)
 (preproc_call 4 32
  (preproc_directive 4 18)
  (preproc_arg 18 31)
 )
 (expression_statement 33 41
  (identifier 33 41)
 )
)

cmd:
./toast --past --source:/Users/timothee/git_clone/nim/nimterop/tests/include/test_bug2.h

Very interested in this repo; is there a list of existing C and C++ libraries that can be handled? (besides https://github.com/genotrance/nimterop/blob/master/tests/include/test.h)
"currently 0" is ok as an answer if that's the case, I understand it's an early release

analog to what nim does to C files eg:

/* Generated by Nim Compiler v0.19.9 */
/*   (c) 2018 Andreas Rumpf */
/* The generated code is subject to the original license. */
/* Compiled for: MacOSX, amd64, clang */
/* Command for C compiler:
   clang -c  -w -g  -I/Users/timothee/git_clone/nim/Nim/lib -I/Users/timothee/git_clone/nim/timn/tests/nim/all -o /tmp/foo/bar.c.o /tmp/foo/bar.c */

it could show all relevant info eg:

generated at D20190124T190319 (or other human+machine friendly format for timestamp)
# replace w full path to toast?
toast --pnim --preprocess --defines+=FORCE --includeDirs+=/private/tmp/d34/nimterop/tests/include /private/tmp/d34/nimterop/tests/include/test.h

Example header file: svdpi.h

Example lines from that:

/*
 * Return a pointer to an element of the array
 * or NULL if index outside the range or null pointer
 */
XXTERN void *svGetArrElemPtr(const svOpenArrayHandle, int indx1, ...);

/* specialized versions for 1-, 2- and 3-dimensional arrays: */
XXTERN void *svGetArrElemPtr1(const svOpenArrayHandle, int indx1);
XXTERN void *svGetArrElemPtr2(const svOpenArrayHandle, int indx1, int indx2);
XXTERN void *svGetArrElemPtr3(const svOpenArrayHandle, int indx1, int indx2,
        int indx3);
...

From cDebug() output, I see that these are getting translated by nimterop to:

proc svGetArrElemPtr*(a1: svOpenArrayHandle; indx1: cint) {.
    importc: "svGetArrElemPtr", header: headersvdpi.}
proc svGetArrElemPtr1*(a1: svOpenArrayHandle; indx1: cint) {.
    importc: "svGetArrElemPtr1", header: headersvdpi.}
proc svGetArrElemPtr2*(a1: svOpenArrayHandle; indx1: cint; indx2: cint) {.
    importc: "svGetArrElemPtr2", header: headersvdpi.}
proc svGetArrElemPtr3*(a1: svOpenArrayHandle; indx1: cint; indx2: cint; indx3: cint) {.
    importc: "svGetArrElemPtr3", header: headersvdpi.}
...

So the void pointer return types are interpreted as just void.

Should these be:

proc svGetArrElemPtr*(a1: svOpenArrayHandle; indx1: cint): pointer {.
    importc: "svGetArrElemPtr", header: headersvdpi.}
proc svGetArrElemPtr1*(a1: svOpenArrayHandle; indx1: cint): pointer {.
    importc: "svGetArrElemPtr1", header: headersvdpi.}
proc svGetArrElemPtr2*(a1: svOpenArrayHandle; indx1: cint; indx2: cint): pointer {.
    importc: "svGetArrElemPtr2", header: headersvdpi.}
proc svGetArrElemPtr3*(a1: svOpenArrayHandle; indx1: cint; indx2: cint; indx3: cint): pointer {.
    importc: "svGetArrElemPtr3", header: headersvdpi.}

?

import std/os
import nimterop/cimport

# replace this with path to include in your system
const incDir = getEnv("homebrew_D") # /Users/timothee/homebrew/

cDebug()
cAddSearchDir(incDir/"include")
cImport cSearchPath "FreeImage.h"

gives:

Error: undeclared identifier: 'wchar_t'

Right now, time_t is hard-coded to int32. Once devel becomes a release and perhaps n-1, use time_t.Time instead.

nim-lang/Nim#10319

cc @zestyr

Functions that return array pointers get skipped. If they're part of a multi-line declaration, all of them get skipped.

PixelWand
  *DestroyPixelWand(PixelWand *),
  **DestroyPixelWands(PixelWand **,const size_t);
PixelWand *NewPixelWand(void);
PixelWand **NewPixelWands(const size_t);

proc NewPixelWand*(): ptr PixelWand {.importc: "NewPixelWand", header: headerwand.}

(translation_unit 1 1 165
 (declaration 1 1 93
  (type_identifier 1 1 9)
  (pointer_declarator 2 3 30
   (function_declarator 2 4 29
    (identifier 2 4 16)
    (parameter_list 2 20 13
     (parameter_declaration 2 21 11
      (type_identifier 2 21 9)
      (abstract_pointer_declarator 2 31 1)
     )
    )
   )
  )
  (pointer_declarator 3 3 46
   (pointer_declarator 3 4 45
    (function_declarator 3 5 44
     (identifier 3 5 17)
     (parameter_list 3 22 27
      (parameter_declaration 3 23 12
       (type_identifier 3 23 9)
       (abstract_pointer_declarator 3 33 2
        (abstract_pointer_declarator 3 34 1)
       )
      )
      (parameter_declaration 3 36 12
       (type_qualifier 3 36 5)
       (primitive_type 3 42 6)
      )
     )
    )
   )
  )
 )
 (declaration 4 1 30
  (type_identifier 4 1 9)
  (pointer_declarator 4 11 19
   (function_declarator 4 12 18
    (identifier 4 12 12)
    (parameter_list 4 24 6
     (parameter_declaration 4 25 4
      (primitive_type 4 25 4)
     )
    )
   )
  )
 )
 (declaration 5 1 40
  (type_identifier 5 1 9)
  (pointer_declarator 5 11 29
   (pointer_declarator 5 12 28
    (function_declarator 5 13 27
     (identifier 5 13 13)
     (parameter_list 5 26 14
      (parameter_declaration 5 27 12
       (type_qualifier 5 27 5)
       (primitive_type 5 33 6)
      )
     )
    )
   )
  )
 )
)

git rev-parse HEAD
1e086d5

nimble installWithDeps
nimble test

Hint:  [Link]
Undefined symbols for architecture x86_64:
  "__test_call_param_", referenced from:
      _NimMainModule in nimterop_tnimterop_c.c.o
  "_test_call_param2", referenced from:
      _NimMainModule in nimterop_tnimterop_c.c.o
  "_test_call_param3", referenced from:
      _NimMainModule in nimterop_tnimterop_c.c.o
  "_test_call_param4", referenced from:
      _NimMainModule in nimterop_tnimterop_c.c.o
  "_test_call_param5", referenced from:
      _NimMainModule in nimterop_tnimterop_c.c.o
  "_test_call_param6", referenced from:
      _NimMainModule in nimterop_tnimterop_c.c.o
  "_test_call_param7", referenced from:
      _NimMainModule in nimterop_tnimterop_c.c.o
  "_test_call_param8", referenced from:
      _NimMainModule in nimterop_tnimterop_c.c.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Error: execution of an external program failed: 'g++   -o /Users/timothee/git_clone/nim/nimterop/tests/tnimterop_c  /Users/timothee/.cache/nim/tnimterop_c_d/runtime.c.o /Users/timothee/.cache/ni...

EDIT: using -f to force rebuild worked:

nim c -r -f tests/tnimterop_c.nim

(so may be hard to reproduce...)

is that a nim bug where nim should recompile because something changed, but didn't recompile?

ImageMagick uses the type in these two functions:

ThrowMagickExceptionList(ExceptionInfo *, const char *, const char *,
  const size_t, const ExceptionType, const char *, const char *, va_list),

LogMagickEventList(const LogEventType, const char *, const char *, const size_t,
  const char *, va_list);

First one in exception.c uses the value here (operands is the va_list):
n = vsprintf(reason+length, format, operands);

Currently nimterop just keeps the va_list type, but it's undeclared in Nim.

Maybe something like

type va_list {.importc: "va_list", header:"<stdarg.h>".} = object

• is that a bug?
  if gStateRT.mode.len != 0: gStateRT.mode = “cpp"

• globals.nim:
  State* = object
  => State* = ref object

• this change:
  node.children.len
  =>
  tsNodeNamedChildCount()
  seems works; can we add back a nicer API (where one can use len) ?

• README needs to be updated to reflect Move AST to Nim processing into toast commit

• s/staticExec/gorgeEx/g (staticExec less safe, ignores exit code)

the approach "just use nim cpp" doesn't work, eg, after #14 :

nim c -r tests/tnimterop_c.nim # ok 
nim cpp -r tests/tnimterop_c.nim # fail

My system headers define a type called __intptr_t, causing troubles:

ico@platdoos:~/external/nimterop(master)$ cat t.h 
typedef long int __intptr_t;
ico@platdoos:~/external/nimterop(master)$ ./toast -n t.h
{.experimental: "codeReordering".}
const
  headert = "t.h"

type
  ptr int* = clong

follow up for #11:
constants works but not C macros, see test case in #30

[EDIT] examples

#define foobar1 30 # case1: litteral: already handled
#define foobar2 3.1415 / 2 # case2: symbol free expression
#define foobar3(x) x*x + 1 # case3: refers to just parameters symbol `x`
#define foobar4(x) a * x # case4: refers to external symbol `a` ; harder; could be impossible to map depending on what `a` is

note: C allows pretty much anything in macros, these are legal but shouldn't be mapped:
```c
#define foo whatever_that_does_not_exist
#define COMMA  ,
#define EXTERNC  extern "C"
#define BEGIN {

behavior: same as C macros

nim templates seem the most appropriate mapping from C macros, and have same semantics (code substitution) with regard to duplicate evaluation as C macros, which we may wanna preserve during mapping eg:

#define foo(x) x*x

var counter = 0
proc identity[T](x: T): T =
  counter.inc
  x

static: doAssert foo(identity(3)) == 3*3
doAssert counter == 2

proposal

for simplicity we can:

• case1: automatically map litterals (integral, float, string) (already do except for string see #75 (comment))

• case2, case3: automatically map macros that only refer to params

• case3: disable mapping these by default

  • but call user defined plugin onUmapped(symbol) (with a enum field called reason)
  • allow user to specify which ones to map and using direct source translation, and use a template for these
  • user can always provide full definition with another mean

nimterop should allow generating doc comments for each decl generated; and this could be customizable (eg via an enum)

• pretty sure treesitter exposes comments, so might as well use them to generate doc comments, that way we can have half decent (or more) docs for free

• doc comment could also specify the corresponding declaration from source function

• furthermore, w the plugin idea we discussed, a wrapper writer can provide his definition for a symbol inside onSymbol(..), and can be creative about it in a way that'd be impossible for nimterop to handle, for eg:

• using doxygen

• custom weird formats like ncurses; not sure where doc comments are for symbols for ncurses ; maybe https://github.com/mirror/ncurses/blob/master/doc/ncurses-intro.doc ( see also https://invisible-island.net/ncurses/man/curs_printw.3x.html);

• @rnowley how did u insert doc comments in https://github.com/rnowley/nim-ncurses/blob/master/ncurses.nim ?

nimble testext
Error: cannot open 'tests/tnimteropext.nim'

See https://irclogs.nim-lang.org/21-01-2019.html#08:05:37

char* is currently converted to ptr cchar, but cstring should be used instead.

char** can be mapped to the cstringArray type.

cstringArray {...} = ptr UncheckedArray[cstring]
This is binary compatible to the type char** in C. The array's high value is large enough to disable bounds checking in practice. Use cstringArrayToSeq to convert it into a seq[string].

The following enum is ignored by nimterop:

enum modbus_recoverymode
{ 
    MODBUS_ERROR_RECOVERY_NONE = (1<<0),
    MODBUS_ERROR_RECOVERY_LINK = (1<<1),
};

current code:

import nimterop/cimport

cDebug()
cDisableCaching()

cDefine("FORCE")
cIncludeDir "$projpath/include"
cAddSearchDir "$projpath/include"
cCompile cSearchPath("test.c")
cImport cSearchPath "test.h"

proposal

import nimterop/cimport

importForeign(
  debug = true,
  caching = false,
  defines = ["FORCE"],
  includes = ["$projpath/include"],
  searchDir = ["$projpath/include"],
  imports = ["$searchPath/test.h"], # $searchPath means use searchPath to resolve
  compiles = ["$searchPath/test.c"],
)

benefits

• a single macro is used (note: cPlugin may need to remain a macro as it takes an AST (plugin code) as input)
• no need for globals
• simpler implementation, easier API for users

links

• https://github.com/Syniurge/Calypso/blob/documentation/calypso_proposal_pragma.md (pretty related !)
• https://github.com/genotrance/nimterop/blob/master/nimterop/globals.nim
• #66 (comment)

I think we need to improve the entire gState situation in the future. The global is out of control and there's only more going to get added.

• some points raised in #10 (comment)

migration path

• migrate these: https://github.com/search?q=nimterop

otherwise, easy to miss then leads to further odd-ness

From readline.h:

typedef struct undo_list {
  struct undo_list *next;
  int start, end;               /* Where the change took place. */
  char *text;                   /* The text to insert, if undoing a delete. */
} UNDO_LIST;

is translated to:

type
  UNDO_LIST* {.importc: "UNDO_LIST", header: headert, bycopy.} = object
    next*: ptr undo_list
    start*: cint
    `end`*: cint
    text*: cstring

UNDO_LIST != undo_list

Throws off ast.nim which expects a specific tree structure.

struct foo { int foo[8+1]; };
struct foo { int foo[__MAGICNUM+1]; };

First will work, second too but __MAGICNUM won't get cleaned up.

from https://ptpb.pw/-vx0/nim

proc svDpiVersion*(): cstring {.importc: "svDpiVersion", header: headersvdpi.}
proc svGetBitselBit*(s: ptr svBitVecVal; i: cint): svBit {.importc: "svGetBitselBit",
    header: headersvdpi.}
...

=>

{.push header: "headersvdpi".}
proc svDpiVersion*(): cstring {.importc: "$1".}
proc svGetBitselBit*(s: ptr svBitVecVal; i: cint): svBit {.importc: "$1".}
...
{.pop.}

toast myheader.h is cached using getCacheValue(myheader.h) which seems wrong as it doesnt' track myheader.h's dependencies;

this is related to #29 and nim-lang/RFCs#510 but different I think, in the sense that even if nim-lang/RFCs#510 is fixed, the current code won't work since getCacheValue will return a "incomplete" hash that will bypass a call to gorgeEx(cmd, cache=getCacheValue(fullpath)) in getToast

proposal

we can achive both accuracy (recompile when needed) and speed (no overhead for that check) as follows:

replace (result, ret) = gorgeEx(cmd, cache=getCacheValue(fullpath)) with:

let hash = getHash(cmd, fullpath)
let indexFile = getTempDir/"nimterop"/("index." &  hash & ".json")
var workNeeded = false
if not exists indexFile:
  let deps = getDeps(fullpath) #  via `clang -M` (a proc required to implement https://github.com/nim-lang/RFCs/issues/510)
  # decide whether to strip system deps (via https://github.com/nim-lang/RFCs/issues/510)
  # also same lastModificationTimes for each deps
  indexFile.writeFile %* (deps: deps, cmd: cmd, file: fullpath)
  workNeeded = true
else:
  workNeeded = anyDepsIsNewer(indexFile.lastModificationTime) # that's the overhead; seems fast enough
  if workNeeded: updateIndex(indexFile)
if workNeeded:
  (result, ret) = gorgeEx(cmd)

Alternatively we can use 1 single db file (eg sql / mongdob) instead of json files

Causes missing structs in ImageMagick.

struct foo { int foo[8 + 1]; };

(translation_unit 1 1 31
 (struct_specifier 1 1 30
  (type_identifier 1 8 3)
  (field_declaration_list 1 12 19
   (field_declaration 1 14 15
    (primitive_type 1 14 3)
    (array_declarator 1 18 10
     (field_identifier 1 18 3)
     (math_expression 1 22 5
      (number_literal 1 22 1)
      (number_literal 1 26 1)
     )
    )
   )
  )
 )
)

This is the offending line, if I put that on its on in a file, toast will eat my CPU and never finish:

cat > t.h <<EOT
typedef int rl_command_func_t PARAMS((int, int));
EOT
./toast -np r.h 

Hello,

I installed nimterop using nimble.

Here is the minimum working example:

# libdpi.nim
import nimterop/cimport

cImport("svdpi.h")

# Input: none
# Return: none
proc hello() {.exportc.} =
  echo "svdpi version: " & svDpiVersion()
  echo "Hello from Nim!"

and the svdpi.h (this is a publicly shareable file, so you can add this to your tests too if you like).

On running:

nim c --out:libdpi.so --app:lib libdpi.nim

I get this error:

Hint: unicodeplus [Processing]
Importing /home/kmodi/sandbox/systemverilog/dpi_c/nim_svdpi/svdpi.h
toast --pnim --preprocess /home/kmodi/sandbox/systemverilog/dpi_c/nim_svdpi/svdpi.h
stack trace: (most recent call last)
../../../../.nimble/pkgs/nimterop-0.1.0/nimterop/cimport.nim(230, 22) cImport
../../../../.nimble/pkgs/nimterop-0.1.0/nimterop/cimport.nim(90, 11) getToast
../../../../stow/pkgs/nim/devel/lib/system.nim(3989, 20) failedAssertImpl
../../../../stow/pkgs/nim/devel/lib/system.nim(3982, 11) raiseAssert
../../../../stow/pkgs/nim/devel/lib/system.nim(3016, 7) sysFatal
libdpi.nim(4, 8) template/generic instantiation of `cImport` from here
../../../../stow/pkgs/nim/devel/lib/system.nim(3016, 7) Error: unhandled exception: /home/kmodi/.nimble/pkgs/nimterop-0.1.0/nimterop/cimport.nim(90, 12) `exitCode == 0` 1