Allow the debugger to be configured on or off.

This happens when I'm running ./example.

And if you'll look at ldd output, you'll get the same:

% ldd example
        linux-vdso.so.1 (0x00007ffd9fbeb000)
        libcalc.so => not found
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5c1f7a7000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f5c1f9dc000)

To solve this issue I have to run program like this:

LD_LIBRARY_PATH=`pwd` ./example

Probably this should be added to README?

On most systems, specific code can be executed at shared library load time (as per DllMain on Windows or .init for Linux/macOS). The use of this would free us from having to initialize manually, and would save us the risk of clients trying to initialize multiple times. On the other hand, to do this we may need a somewhat stricter policy for how the core file is distributed relative to the shared library.

I was unable to run the example unfortunately. I use roswell so I cloned into ~/.roswell/local-projects. Then I navigated to ~/.roswell/local-projects/sbcl-librarian/example and

make SBCL_SRC=$HOME/.roswell/src/sbcl-2.2.2-arm64-darwin

But it couldn't find the '#:libcalc system so I changed line 3 in example/script.lisp to

(asdf:load-asd (truename "./libcalc.asd"))

Because sbcl still couldn't find sbcl-librarian on its own I tried runing script.lisp with roswell directly

$ ros run -l script.lisp -q  
...

debugger invoked on a PACKAGE-DOES-NOT-EXIST in thread
#<THREAD "main thread" RUNNING {700863FBB3}>:
  The name "SBCL-LIBRARIAN/EXAMPLE/LIBCALC" does not designate any package.

I don't know how to deal with this one. The readme gives some instructions but they are all the way at the end of the process, it would be nice to have some instructions that start at "where do I clone this project?".

All of the generated C code is undocumented. We could improve this by

• get the docstr from the associated lisp function
• add a documentation field to the :function spec
• ???

After the commit 04f7e39 attempt to build a libcalc example fails with this trace:

CL_SOURCE_REGISTRY="/home/art/projects/sbcl-librarian//" /home/art/.roswell/src/sbcl-2.3.2//run-sbcl.sh --script "script.lisp"
While evaluating the form starting at line 1, column 0
  of #P"/home/art/projects/sbcl-librarian/examples/libcalc/libcalc.asd":
Unhandled MISSING-DEPENDENCY in thread #<SB-THREAD:THREAD "main thread" RUNNING {1004490283}>: Component #:BORDEAUX-THREADS not found, required by #<SYSTEM "sbcl-librarian">

Backtrace for: #<SB-THREAD:THREAD "main thread" RUNNING {1004490283}>
0: (SB-DEBUG::DEBUGGER-DISABLED-HOOK Component #:BORDEAUX-THREADS not found, required by #<SYSTEM "sbcl-librarian"> #<unused argument> :QUIT T)
1: (SB-DEBUG::RUN-HOOK SB-EXT:*INVOKE-DEBUGGER-HOOK* Component #:BORDEAUX-THREADS not found, required by #<SYSTEM "sbcl-librarian">)
2: (INVOKE-DEBUGGER Component #:BORDEAUX-THREADS not found, required by #<SYSTEM "sbcl-librarian">)
3: (ERROR MISSING-DEPENDENCY :REQUIRED-BY #<SYSTEM "sbcl-librarian"> :REQUIRES #:BORDEAUX-THREADS)
4: (ASDF/FIND-COMPONENT:RESOLVE-DEPENDENCY-NAME #<SYSTEM "sbcl-librarian"> #:BORDEAUX-THREADS NIL)
5: (ASDF/PLAN:MAP-DIRECT-DEPENDENCIES #<PREPARE-OP > #<SYSTEM "sbcl-librarian"> #<FUNCTION (LAMBDA (ASDF/PLAN::O ASDF/PLAN::C) :IN ASDF/PLAN:TRAVERSE-ACTION) {100129BE9B}>)
6: ((LAMBDA NIL :IN ASDF/PLAN:TRAVERSE-ACTION))
7: ((LAMBDA NIL :IN ASDF/ACTION:CALL-WHILE-VISITING-ACTION))
8: (ASDF/PLAN:TRAVERSE-ACTION #<SEQUENTIAL-PLAN {1001276833}> #<PREPARE-OP > #<SYSTEM "sbcl-librarian"> T)
9: (ASDF/PLAN:MAP-DIRECT-DEPENDENCIES #<PREPARE-OP > #<CL-SOURCE-FILE "sbcl-librarian" "package"> #<FUNCTION (LAMBDA (ASDF/PLAN::O ASDF/PLAN::C) :IN ASDF/PLAN:TRAVERSE-ACTION) {100129BCEB}>)
10: ((LAMBDA NIL :IN ASDF/PLAN:TRAVERSE-ACTION))
11: ((LAMBDA NIL :IN ASDF/ACTION:CALL-WHILE-VISITING-ACTION))

Possible solution

Is to add https://github.com/sionescu/bordeaux-threads as submodule and to update Makefile to ensure that submodules are in sync. Actually not only bordeaux-threads should be cloned. Here is the full list of commands I did to solve the problem:

cd ~/projects/sbcl-librarian/

mkdir bundle
cd bundle

git clone https://github.com/sionescu/bordeaux-threads
git clone https://gitlab.common-lisp.net/alexandria/alexandria
git clone https://github.com/lmj/global-vars
git clone https://github.com/trivial-features/trivial-features
git clone https://github.com/trivial-garbage/trivial-garbage

When SBCL's garbage collector is invoked, it seems to mess with memory in such a way that my shared library fails (with some fairly obscure errors).

I've created a repo on github with a minimal example project which reproduces the error (on Linux, haven't tried other OSs). The error, in this example, happens when parsing a ~100KB JSON file after a call to (sb-ext:gc), although the same class of error will occur if the GC is triggered indirectly. Smaller files can trigger the error if the dynamic space size is reduced.

The project has a dependency on quilc. I assume the error can be reproduced without quilc, but, in my limited testing, including quilc as a dependency causes the error 100% of the time.

I don't know enough about SBCL and it's garbage collector to be sure, but it seems as though GC is affecting the pointer that is received by my parse function such that it points to invalid memory.

This is the error I am seeing currently, though it has been different in the past

gcc test-parse.c -o test-parse -lsbcl -lparsejson -L. -I.
;
; compilation unit aborted
;   caught 1 fatal ERROR condition
ERROR The value
  0
is not of type
  SB-C::CONSTRAINT
error: The value
  0
is not of type
  SB-C::CONSTRAINT

Requires compiling SBCL

0. cd sbcl-src
1. sh make.sh
2. sh make-shared-library.sh

Then need to set LD_LIBRARY_PATH=/path/to/sbcl-src/src/runtime when compiling the executable. When running the executable, LD_LIBRARY_PATH=/path/to/sbcl-librian/example:/path/to/sbcl-src/src/runtime.

Line 42 in the parse function:

(find-package "SBCL-LIBRARIAN-EXAMPLE")

The package name is actually sbcl-librarian/example/libcalc. It looks like the name was changed in this commit but the line mentioned above wasn't updated to match.

P.S. The README note regarding -pagezero_size on OS X should probably mention that it applies to intel-based Macs. It's not needed (and doesn't work) on Apple Silicon.

I tried this out using Windows 10 via Parallels for macOS. I'm using msys2, with gcc 11.2.0 from the latest mingw-w64.

I built SBCL from source (commit 02635ecb29aed48e2f97cae25817b51656224f36) with no apparent problems. I also built libsbcl.so, just via the ordinary make-shared-library.sh.

Using sbcl-librarian, I'm able to produce libcalc.core. However, when I try the following example (not building another dll, but just linking against libsbcl.so) I get an error

example.c

#include "libcalc.h"
err_t (*calc_int_literal)(int value, expr_type *result);
err_t (*calc_int_literal_value)(expr_type expr, int *result);
err_t (*calc_int_literal_p)(expr_type obj, int *result);
err_t (*calc_sum_expression)(expr_type left, expr_type right, expr_type *result);
err_t (*calc_sum_expression_left_arg)(expr_type expr, expr_type *result);
err_t (*calc_sum_expression_right_arg)(expr_type expr, expr_type *result);
err_t (*calc_sum_expression_p)(expr_type expr, int *result);
err_t (*calc_simplify)(expr_type expr, expr_type *result);
err_t (*calc_parse)(char * source, expr_type *result);
err_t (*calc_expression_to_string)(expr_type expr, char * *result);
void (*release_handle)(void *handle);
int (*handle_eq)(void *a, void *b);
extern int initialize_lisp(int argc, char **argv);


int main(int argc, char** argv) {
  return initialize_lisp(argc, argv);
}

I compile with cc example.c libsbcl.so

I run with ./a.exe --core libcalc.core and see the following

$ ./a.exe --core libcalc.core
This is SBCL 2.1.0, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.

SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses.  See the CREDITS and COPYING files in the
distribution for more information.
Unhandled SB-KERNEL::UNDEFINED-ALIEN-VARIABLE-ERROR in thread #<SB-THREAD:THREAD "main thread" RUNNING
                                                                 {10029F0003}>:
  Attempt to access an undefined alien variable.

Backtrace for: #<SB-THREAD:THREAD "main thread" RUNNING {10029F0003}>
0: (SB-DEBUG::DEBUGGER-DISABLED-HOOK #<SB-KERNEL::UNDEFINED-ALIEN-VARIABLE-ERROR {1002A05913}> #<unused argument> :QUIT T)
1: (SB-DEBUG::RUN-HOOK *INVOKE-DEBUGGER-HOOK* #<SB-KERNEL::UNDEFINED-ALIEN-VARIABLE-ERROR {1002A05913}>)
2: (INVOKE-DEBUGGER #<SB-KERNEL::UNDEFINED-ALIEN-VARIABLE-ERROR {1002A05913}>)
3: (ERROR SB-KERNEL::UNDEFINED-ALIEN-VARIABLE-ERROR)
4: ("foreign function: #x7FF81914DC7A")
5: ("foreign function: #x7FF81914DD50")
6: ((SETF SB-SYS:SAP-REF-SAP) :INVALID-VALUE-FOR-UNESCAPED-REGISTER-STORAGE :INVALID-VALUE-FOR-UNESCAPED-REGISTER-STORAGE :INVALID-VALUE-FOR-UNESCAPED-REGISTER-STORAGE)
7: (SB-EVAL::EVAL-PROGN ((SETF (SB-SYS:SAP-REF-SAP SB-ALIEN::SAP (/ SB-ALIEN::OFFSET SB-VM:N-BYTE-BITS)) SB-ALIEN::VALUE) (SB-ALIEN-INTERNALS:NATURALIZE SB-ALIEN::VALUE (QUOTE #<SB-ALIEN-INTERNALS:ALIEN-POINTER-TYPE (* T)>))) #<SB-EVAL::ENV {1002A05083}>)
8: ((SETF SB-ALIEN-INTERNALS:%ALIEN-VALUE) #<SB-ALIEN-INTERNALS:ALIEN-VALUE :SAP #X20220B70 :TYPE (* T)> #.(SB-SYS:INT-SAP #X2019D380000) 0 #<SB-ALIEN-INTERNALS:ALIEN-POINTER-TYPE (* T)>)
9: (SB-IMPL::START-LISP)
10: ("foreign function: #x7FF81914DC7A")
11: ("foreign function: #x7FF81912445E")

unhandled condition in --disable-debugger mode, quitting
Missing required foreign symbol 'log1p'
<??? type 199>

I'm not sure what this log1p business is about, but I did notice that it's referenced here https://github.com/sbcl/sbcl/blob/d35ad6f9bee212499cb827fec3195bf2064c10a2/src/runtime/mswin64.def

When we want to pass in a C-allocated array, we have to use :pointer as the type in sbcl-librarian:define-api and then manually reinterpret that pointer as a lisp list.

It would be convenient if we supported a type such as e.g. (:list :int) which would automatically do the unpacking before handing the list to the associated Lisp function.

This will make it easier to distribute lisp libraries.

Here I found an article showing how to link a binary file into the library and access it's data:
https://balau82.wordpress.com/2012/02/19/linking-a-binary-blob-with-gcc/

Problem

Writing the API specification for a library can be tedious and potentially error-prone, especially when exporting a large amount of functions. This decreases the velocity at which we can deliver new and existing Lisp projects to downstream alien language users.

Solution

SBCL's sb-introspect extension contains two functions that we can leverage to synthesize API export specifications directly from Lisp function objects:

• function-lambda-list, which we can use to get the arity and argument names for a function.
• function-type, which we can use to get the compiler-inferred type signature for the function.

Inferring types

A clear issue with using compiler-inferred type information for writing function export specifications is that in many cases the compiler cannot infer a more specific type than t for arguments or return values. This does not preclude the generation of bindings, since the t type can just be mapped to a generic handle type in the function export spec.

Nevertheless, better bindings can be generated if the compiler can infer more specific types since we can replace generic handles with more specific foreign types such as :int or :string. This can be achieved by declaring types more extensively throughout Lisp code, which has the added benefits of improving documentation and potentially improving performance.

Possible interface

One possible way this could look is like this:

(defmacro define-api-from-package (name (&key package ...))
  ...)

The macro can iterate through the external symbols of the provided package using do-external-symbols and produce an export specification for each symbol bound to a function using sb-introspect as described before.

Problem

I want to be able to export a Lisp function under a custom name.

As an example, Quilc defines the compiler-hook function in the #:cl-quil package. When building a shared library for quilc, I want to map this function to the C function quilc_compile_quil.

Problem

We are blind to breakages in the example library, such as the one described here, that result from API changes/new features.

Solution

Write some tests that check that the example library builds and runs correctly. Unit tests for individual features are also probably good idea.

I've been struggling with building the example binary for the last few hours. I've tried the solutions outlined in #21 and #8 without success. I've tried using the version of sbcl made available by my system's package manager, using Roswell and building it from source.

The sticking point seems to consistently be the lack of availability of libcalc. I've tried reworking the Makefile flags and compiling manually but my C toolchain skills are ... pretty much nonexistent.

Has anyone had success building and running the example on Debian/Ubuntu Linux?

Currently, using sbcl-librarian to build shared libraries requires (1) building SBCL from source and then (2) building the SBCL runtime into either a shared library (Linux and macOS) or a static library for further linking (Windows). On macOS and Linux, downstream users also require a copy of the runtime shared library on their systems.

To reduce the burden of building SBCL from source, we should distribute both shared library and static library builds of the SBCL runtime on macOS, Linux, and Windows. A good initial distribution channel for these built binaries is Conda since we anticipate many users wanting to wrap Lisp code into Python libraries.

The implementation plan is as follows:

1. Update the Linux and macOS sbcl recipes on conda-forge to at least 2.1.10 which adds support for foreign callables. Changes need to be made here.
2. Add a Windows build to sbcl-feedstock.
3. Either: (1) create a new libsbcl conda package that uses the updated sbcl package to bootstrap the build or (2) add libsbcl shared and static libraries to the existing sbcl conda package.

Problem

Users currently have to build SBCL from source in order to use sbcl-librarian, which is burdensome. Even after the user builds SBCL, their environment needs to be set up in such a way as to avoid mismatched core/FASL versions and linking errors.

Solution

Distribute a Conda package called libsbcl on conda-forge that packages a single {.dll,.so,.dylib} shared library shrinkwrapping the SBCL runtime, a base core file, and a library load time hook that initializes the runtime.

Notes

• TODO: specify shrinkwrapping procedure
• Instructions to add a recipe to conda-forge

Any idea what the problem is? SBCL 2.0.1

❯ sbcl --eval "(progn (asdf:load-system :sbcl-librarian) (load \"example.lisp\"))"
This is SBCL 2.0.1.debian, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.

SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses.  See the CREDITS and COPYING files in the
distribution for more information.
; compiling file "/home/mgsk/hackery/lisp/sbcl-librarian/function.lisp" (written 20 JAN 2022 10:09:13 PM):
; compiling (IN-PACKAGE #:SBCL-LIBRARIAN)
; compiling (DEFUN CANONICAL-SIGNATURE ...)
; compiling (DEFUN C-FUNCTION-DECLARATION ...); 
; caught ERROR:
;   READ error during COMPILE-FILE:
;   
;     Symbol "DEFINE-ALIEN-CALLABLE" not found in the SB-ALIEN package.
;   
;       Line: 64, Column: -1, File-Position: 2882
;   
;       Stream: #<SB-INT:FORM-TRACKING-STREAM for "file /home/mgsk/hackery/lisp/sbcl-librarian/function.lisp" {100357A1A3}>

; compilation aborted after 0:00:00.008

debugger invoked on a UIOP/LISP-BUILD:COMPILE-FILE-ERROR in thread
#<THREAD "main thread" RUNNING {1000560083}>:
  COMPILE-FILE-ERROR while
  compiling #<CL-SOURCE-FILE "sbcl-librarian" "function">

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [RETRY                        ] Retry
                                     compiling #<CL-SOURCE-FILE "sbcl-librarian" "function">.
  1: [ACCEPT                       ] Continue, treating
                                     compiling #<CL-SOURCE-FILE "sbcl-librarian" "function">
                                     as having been successful.
  2:                                 Retry ASDF operation.
  3: [CLEAR-CONFIGURATION-AND-RETRY] Retry ASDF operation after resetting the
                                     configuration.
  4:                                 Retry ASDF operation.
  5:                                 Retry ASDF operation after resetting the
                                     configuration.
  6: [CONTINUE                     ] Ignore runtime option --eval "(progn (asdf:load-system :sbcl-librarian) (load \"example.lisp\"))".
  7: [ABORT                        ] Skip rest of --eval and --load options.
  8:                                 Skip to toplevel READ/EVAL/PRINT loop.
  9: [EXIT                         ] Exit SBCL (calling #'EXIT, killing the process).

(UIOP/LISP-BUILD:CHECK-LISP-COMPILE-RESULTS NIL T T "~/asdf-action::format-action/" ((#<ASDF/LISP-ACTION:COMPILE-OP > . #<ASDF/LISP-ACTION:CL-SOURCE-FILE "sbcl-librarian" "function">)))
   error finding frame source: Bogus form-number: the source file has probably
                               changed too much to cope with.
   source: NIL
0] 

Problem

Handle types are currently implemented as typedefs to void *, which means that a C compiler cannot do type checking on handles.

For example, here is a C program that uses the C bindings for Quilc, violates the function type signatures specified by libquilc, but compiles without any issues:

#include <stdio.h>
#include <stdlib.h>

#include "libquilc.h"

void die(char *msg) {
  printf("%s\n", msg);
  exit(1);
}

int main(int argc, char **argv) {
  init("libquilc.core");

  char source[256];
  chip_specification chip_spec;

  fgets(source, 256, stdin);

  /*
   * The function call below should not compile, since the function `quilc_parse_quil` has the signature
   * 
   *     error_t (*quilc_parse_quil)(char* source, quil_program *result)
   * 
   * but we are calling it with a `chip_specification *` as the second argument.
   */
  if (quilc_parse_quil(source, &chip_spec) != ERROR_SUCCESS)
    die("unable to parse quil");

  lisp_release_handle(chip_spec);

  return 0;
}

Solution

Implement handles as typedefs to dummy struct pointers. For example, we can redefine the quil_program and chip_specification handle types as such:

struct quil_program__ { int unused; }; typedef struct quil_program__ *quil_program;
struct chip_specification__ { int unused; }; typedef struct chip_specification__ *chip_specification;

Now the example program given above causes the compiler to emit a warning:

test.c:19:32: warning: incompatible pointer types passing 'chip_specification *' (aka 'struct chip_specification__ **') to
      parameter of type 'quil_program *' (aka 'struct quil_program__ **') [-Wincompatible-pointer-types]
  if (quilc_parse_quil(source, &chip_spec) != ERROR_SUCCESS)
                               ^~~~~~~~~~
1 warning generated.

Notably, this technique of implementing opaque handles is standard practice in Win32 [1, 2].

The current strategy for initialization (call initialize_lisp with argc and argv containing core file name) exhibits unpredictable behavior on Windows, because the runtime code actually recovers command line arguments via system calls. Cf. GetCommandLineW() in https://github.com/sbcl/sbcl/blob/master/src/runtime/runtime.c#L593 and elsewhere

What this means in practice is that if you call initialize_lisp on Windows at library load time with anything other than runtime options (e.g. with the toplevel option --noinform), you may segfault. On my machine the error is like

(in libcalc.c)

int calc_init(char *core) {
  char *init_args[] = {"", "--core", core, "--noinform"};
  return initialize_lisp(4, init_args); }

(in Python)

>>> import libcalc
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\ejdavis\quicklisp\local-projects\sbcl-librarian\example\libcalc.py", line 24, in <module>
    libcalc.calc_init(str(libpath.parent / 'libcalc.core').encode('utf-8'))
OSError: exception: access violation writing 0x0000000000008060

which is fixed by dropping the --noinform argument.

After testing, it is found that the interaction between Python and SBCL is too expensive. Is there a solution now?

`expr = byref(etl.expr_type())
s = "123123".encode('utf-8')

stime = time.time()

for _ in range(10000):
etl.etl_process(s, expr) #SBCL empty function implementation

print(time.time() - stime) #--> 1.56 s

`