marcoheisig / cl-mpi

MPI bindings for Common Lisp with many useful extras

cl-mpi

cl-mpi provides convenient CFFI bindings for the Message Passing Interface (MPI). MPI is typically used in High Performance Computing to utilize big parallel computers with thousands of cores. It features minimal communication overhead with a latency in the range of microseconds. In comparison to the C or FORTRAN interface of MPI, cl-mpi relieves the programmer from working with raw pointers to memory and a plethora of mandatory function arguments.

If you have questions or suggestions, feel free to contact me ([email protected]).

cl-mpi has been tested with MPICH, MPICH2, IntelMPI and Open MPI.

Usage

An MPI program must be launched with mpirun or mpiexec. These commands spawn multiple processes depending on your system and commandline parameters. Each process is identical, except that it has a unique rank that can be queried with (MPI-COMM-RANK). The ranks are assigned from 0 to (- (MPI-COMM-SIZE) 1). A wide range of communication functions is available to transmit messages between different ranks. To become familiar with cl-mpi, see the examples directory.

The easiest way to deploy and run cl-mpi applications is by creating a statically linked binary. To do so, create a separate ASDF system like this:

(defsystem :my-mpi-app
  :depends-on (:cl-mpi)
  :defsystem-depends-on (:cl-mpi-asdf-integration)
  :class :mpi-program
  :build-operation :static-program-op
  :build-pathname "my-mpi-app"
  :entry-point "my-mpi-app:main"
  :serial t
  :components
  ((:file "foo") (:file "bar")))

and simply run

(asdf:make :my-mpi-app)

on the REPL. Note that not all Lisp implementation support the creation of statically linked binaries (actually, we only tested SBCL so far). Alternatively, you can try to use uiop:dump-image to create binaries.

Further remark: If the creation of statically linked binaries with SBCL fails with something like “undefined reference to main”, your SBCL is probably not built with the :sb-linkable-runtime feature. You are affected by this when (find :sb-linkable-runtime *features*) returns NIL. In that case, you have to compile SBCL yourself, which is as simple as executing the following commands, where SOMEWHERE is the desired installation folder

git clone git://git.code.sf.net/p/sbcl/sbcl
cd sbcl
sh make.sh --prefix=SOMEWHERE --fancy --with-sb-linkable-runtime --with-sb-dynamic-core
cd tests && sh run-tests.sh
sh install.sh

Testing

To run the test suite:

./scripts/run-test-suite.sh all

or

./scripts/run-test-suite.sh YOUR-FAVOURITE-LISP

Performance

cl-mpi makes no additional copies of transmitted data and has therefore the same bandwidth as any other language (C, FORTRAN). However the convenience of error handling, automatic inference of the message types and safe computation of memory locations adds a little overhead to each message. The exact overhead varies depending on the Lisp implementation and platform but is somewhere around 1000 machine cycles.

Summary:

• latency increase per message: 400 nanoseconds (SBCL on a 2.4GHz Intel i7-5500U)
• bandwidth unchanged

Authors

• Alex Fukunaga
• Marco Heisig

Special Thanks

This project was funded by KONWIHR (The Bavarian Competence Network for Technical and Scientific High Performance Computing) and the Chair for Applied Mathematics 3 of Prof. Dr. Bänsch at the FAU Erlangen-Nürnberg.

Big thanks to Nicolas Neuss for all the useful suggestions.

cl-mpi's Issues

Can't build today

I can't build today. I don't really grok the error, though. http://report.quicklisp.org/2016-05-19/failure-report/cl-mpi.html#cl-mpi has a log.

Unhandled SB-KERNEL:SIMPLE-PACKAGE-ERROR in thread #<SB-THREAD:THREAD "main thread" RUNNING {100334F653}>: 
These symbols are not accessible in the STATIC-VECTORS package:
(MPI::WITH-STATIC-VECTORS)

Roswell script

While providing a set of scripts under scripts/ is good by itself, also consider using a new feature of "roswell script", which easily makes the command installable and available anywhere on the system: https://github.com/snmsts/roswell/wiki/2.-Roswell-as-a-Scripting-Environment

It is useful for a system management task with CRON as well as for a scientific purpose, like under the TORQUE/PBS batch scheduler.

BTW, AlexF is my current PhD advisor :)

cl-mpi and debian 8

I'm trying to build on debian 8. It installs mpi.h in /usr/lib/openmpi/openmpi/include/mpi.h. What should I do to help cl-openmpi find it?

Implement the remaining collective operations

Not all collective MPI operations are currently supported. Someone should fix this...

Increase the testsuite

There should be more test cases. Especially the serial version should strive to have as much coverage as possible.

One sided communication

MPI 3.0 introduced many primitives for one-sided communication (get/put/windows...). It would be nice to have access to those from cl-mpi.

Add support for static linking

Currently, cl-mpi creates a small shared C library to interface with the host MPI. This is inconvenient on distributed systems, because it requires recompilation on each node. Someone should add an option of using ASDF's static-program-op to create independent binaries.

Add a distributed debugger

Debugging MPI applications is painful. It would be nice to have an optional extension of cl-mpi that provides some means for distributed debugging.

A possible implementation would be to provide a WITH-MPI-DEBUGGER macro that creates a separate communicator for debug communication and handles all conditions by broadcasting them over the network.

Provide MPI examples

It would be nice to have some introductory examples like a simple Lattice-Boltzmann code as a starting point for new users.

MPI IO

Find a way to nicely integrate MPI IO into Common Lisp.

cl-mpi test failure

Hi

I tired running the test script on a mac as well as linux box (Fedora) with sbcl 1.5.3. Both machines had mpich2 installed but the test are failing.

Output

=========================================
Building sbcl (SBCL 1.5.3.110-13ba7ac00) image ...
...sbcl image complete
=========================================

=========================================
Testing sbcl...
== Testing CL-MPI ==
Machine: X86-64 Intel(R) Core(TM) i5-5350U CPU @ 1.80GHz
Lisp:    SBCL 1.5.3.110-13ba7ac00
MPI:     MPICH 45149472
cl-mpi:  cl-mpi 1.0

Running test suite MPI-SERIAL-TESTS
 Running test MPI-WTIME ..
 Running test MPI-INIT ..
 Running test PROCESSOR-NAME ..
 Running test SERIAL-GROUPS .....
 Running test MPI-BUFFERING ..
 Running test MPI-CONTEXT ..
 Running test SERIAL-MPI-SENDRECV X
 Running test SERIAL-MPI-ISEND .......
 Running test MPI-PROBE ....
 Running test MPI-IPROBE ....
 Running test MPI-WAIT-AND-TEST .....
 Did 36 checks.
    Pass: 35 (97%)
    Skip: 0 ( 0%)
    Fail: 1 ( 2%)

 Failure Details:
 --------------------------------
 SERIAL-MPI-SENDRECV []:
      Unexpected Error: #<MPI-ERROR-CONDITION {1002872453}>
Invalid argument, error stack:
MPI_Sendrecv(236): MPI_Sendrecv(sbuf=0x7ff99fe04420, scount=10, MPI_LONG, dest=0, stag=42, rbuf=0x7ff99fe06230, rcount=10, MPI_LONG, src=0, rtag=42, comm=0x84000007, status=0x0) failed
MPI_Sendrecv(115): Null pointer in parameter status..
 --------------------------------


Running test suite MPI-PARALLEL-TESTS
 Running test MPI-BARRIER
 Running test MPI-RING X
 Did 8 checks.
    Pass: 0 ( 0%)
    Skip: 7 (87%)
    Fail: 1 (12%)

 Failure Details:
 --------------------------------
 MPI-RING [Send a Common Lisp datastructure through all nodes.]:
      Unexpected Error: #<MPI-ERROR-CONDITION {10049BCA23}>
Invalid argument, error stack:
MPI_Recv(171): MPI_Recv(buf=0x7ff99fc089f0, count=7, MPI_INT, src=3, tag=MPI_ANY_TAG, MPI_COMM_WORLD, status=0x0) failed
MPI_Recv(88).: Null pointer in parameter status..
 --------------------------------

 Skip Details:
 MPI-REDUCE [Perform different reductions with MPI-Reduce]:
     Dependencies not satisfied.
 MPI-ALLREDUCE [Perform different reductions with MPI-Allreduce.]:
     Dependencies not satisfied.
 NONBLOCKING [Nonblocking communication with MPI-I[SEND,RECV], MPI-WAIT and
MPI-WAITALL]:
     Dependencies not satisfied.
 MPI-ALLGATHER [Use mpi-allgather to generate a vector of all ranks.]:
     Dependencies not satisfied.
 MPI-BCAST [Use mpi-bcast to broadcast a single number.]:
     Dependencies not satisfied.
 SEND-SUBSEQUENCE [Send only a subsequence of an array]:
     Dependencies not satisfied.
 MPI-SENDRECV [Send a Common Lisp datastructure through all nodes using mpi-sendrecv.]:
     Dependencies not satisfied.

In direct memory block for handle type GROUP, 1 handles are still allocated
In direct memory block for handle type GROUP, 1 handles are still allocated
In direct memory block for handle type GROUP, 1 handles are still allocated
In direct memory block for handle type REQUEST, 1 handles are still allocated
In direct memory block for handle type GROUP, 1 handles are still allocated
[0] 48 at [0x00007ff99fc0ade0], src/mpi/group/grouputil.c[80]
[3] 48 at [0x00007ff48270e4e0], src/mpi/group/grouputil.c[80]
[1] 32 at [0x00007fda2e550240], src/mpid/ch3/src/ch3u_handle_recv_pkt.c[251]
[1] 48 at [0x00007fda2e70f9f0], src/mpi/group/grouputil.c[80]
[2] 48 at [0x00007fc9c9408780], src/mpi/group/grouputil.c[80]
...sbcl testing complete

I followed the instruction to build sbcl in the README file.

Test leaking memory

The test suite code is still leaking memory when I run this in MPICH full debug mode. The MPI_group is not being freed.

In direct memory block for handle type GROUP, 1 handles are still allocated
In direct memory block for handle type GROUP, 1 handles are still allocated
[1] 24 at [0x00000000025e19a0], src/mpi/group/grouputil.c[74]
[0] 24 at [0x00000000016829a0], src/mpi/group/grouputil.c[74]
...sbcl testing complete

I isolated the problem to this line in the file

serial-tests.lisp

 54 (test (mpi-context :depends-on mpi-init)
 55   (is (mpi-comm-free (mpi-comm-dup)))
 56   (let ((c1 *standard-communicator*)
 57         (c2 (mpi-comm-dup *standard-communicator*))
 58         (c3 (mpi-comm-create (mpi-comm-group *standard-communicator*)  ;; Here it is
 59                              :comm *standard-communicator*)))

I think an intermediate comm-group is begin created but not freed when you call mpi-comm-group. I am just starting to learn CL so could not dig deeper.

[Feedback] Using CL-MPI

So far, CL-MPI has been working for our uses. A few things we've run into so far:

• We don't use cl-launch. Our programs are sufficiently complicated that we deliver/save executables.

• The saved executables seem to depend on some dynamic library that CFFI (or something else) generates. This was pretty inconvenient. When we deploy the application across many machines, we don't want to have to find or figure out this rogue dynamic library. As a "quick fix", we've just installed and compiled CL-MPI on each node. Not particularly efficient, though.

• The API is pretty straightforward if you know MPI, but it could still use better documentation. For instance, I had to read the source code to MPI-PROBE since status structs aren't explicitly exposed in the API (which is OK). Documentation is something we might be able to contribute to.

• It's extremely hard for us to debug MPI lisp apps right now. The main problem is that some process signals an error and it completely hoses the system and there's no backtrace. This might be my own ignorance on how to run and use MPI. (I'm definitely a beginner at it.)

Those are some thoughts for now. My general hope is that the polish can increase. :)

Add support for a parallel mpi-based repl?

I have been thinking about implementing a repl for MPI based projects.

Some past work was for instance the STAR/MPI project.

I don't know how it should actually look like, but AFAIK right now cl-mpi does not really allow for it? I could be wrong though.

References

• STAR/MPI Paper

ECL: CL-MPI fails to load: undefined symbol: ompi_mpi_op_prod

Table of Contents

1. Problem
2. Steps to reproduce
3. Versions
4. Backtrace

Problem

cl-mpi fails to load on ECL 16.1.3

Condition of type: LOAD-FOREIGN-LIBRARY-ERROR
Unable to load foreign library (WRAP).
  LOAD-FOREIGN-MODULE: Could not load foreign module "/home/ma/.cache/common-lisp/ecl-16.1.3-unknown-linux-x64/home/ma/src/repos/cl-mpi/mpi/wrap.so" (Error: "/home/ma/.cache/common-lisp/ecl-16.1.3-unknown-linux-x64/home/ma/src/repos/cl-mpi/mpi/wrap.so: undefined symbol: ompi_mpi_op_prod")

I did not dig in to this error too deeply because I am not really an ECL user, just trying to get the test-suite to pass :-).

Steps to reproduce

Start an ECL repl from the command line (not SLIME, see #22), then run (ql:quickload :cl-mpi).

Versions

ecl 16.1.3
cl-launch 4.1.4
cffi 0.20.0
cl-mpi 3e0e518
openmpi 3.1.3
gcc 8.3.0
linux 4.19.37-1-lts

Backtrace

ECL (Embeddable Common-Lisp) 16.1.3 (git:UNKNOWN)
Copyright (C) 1984 Taiichi Yuasa and Masami Hagiya
Copyright (C) 1993 Giuseppe Attardi
Copyright (C) 2000 Juan J. Garcia-Ripoll
Copyright (C) 2016 Daniel Kochmanski
ECL is free software, and you are welcome to redistribute it
under certain conditions; see file 'Copyright' for details.
Type :h for Help.
Top level in: #<process TOP-LEVEL>.
> (ql:quickload :cl-mpi)
To load "cl-mpi":
  Load 1 ASDF system:
    cl-mpi
; Loading "cl-mpi"
[package mpi]; mpicc -o /home/ma/.cache/common-lisp/ecl-16.1.3-unknown-linux-x64/home/ma/src/repos/cl-mpi/mpi/grovel__grovel-tmp8CUM5SC2.o -c -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -fno-plt -fPIC -D_THREAD_SAFE -Dlinux -fPIC -I/home/ma/opt/quicklisp/dists/quicklisp/software/cffi_0.20.0/ /home/ma/.cache/common-lisp/ecl-16.1.3-unknown-linux-x64/home/ma/src/repos/cl-mpi/mpi/grovel__grovel.c
; mpicc -o /home/ma/.cache/common-lisp/ecl-16.1.3-unknown-linux-x64/home/ma/src/repos/cl-mpi/mpi/grovel__grovel-tmpVJPNC3UW -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -fno-plt -fPIC -D_THREAD_SAFE -Dlinux /home/ma/.cache/common-lisp/ecl-16.1.3-unknown-linux-x64/home/ma/src/repos/cl-mpi/mpi/grovel__grovel.o
; /home/ma/.cache/common-lisp/ecl-16.1.3-unknown-linux-x64/home/ma/src/repos/cl-mpi/mpi/grovel__grovel /home/ma/.cache/common-lisp/ecl-16.1.3-unknown-linux-x64/home/ma/src/repos/cl-mpi/mpi/grovel__grovel.grovel-tmp.lisp
..; mpicc -o /home/ma/.cache/common-lisp/ecl-16.1.3-unknown-linux-x64/home/ma/src/repos/cl-mpi/mpi/wrap__wrapper-tmpTDNN23GC.o -c -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -fno-plt -fPIC -D_THREAD_SAFE -Dlinux -fPIC -I/home/ma/opt/quicklisp/dists/quicklisp/software/cffi_0.20.0/ /home/ma/.cache/common-lisp/ecl-16.1.3-unknown-linux-x64/home/ma/src/repos/cl-mpi/mpi/wrap__wrapper.c
/home/ma/.cache/common-lisp/ecl-16.1.3-unknown-linux-x64/home/ma/src/repos/cl-mpi/mpi/wrap__wrapper.c: In function ‘cl_mpi_get_mpi_ub_cffi_wrap’:
/home/ma/.cache/common-lisp/ecl-16.1.3-unknown-linux-x64/home/ma/src/repos/cl-mpi/mpi/wrap__wrapper.c:214:3: warning: ‘ompi_mpi_ub’ is deprecated: MPI_UB is deprecated in MPI-2.0 [-Wdeprecated-declarations]
   return MPI_UB;
   ^~~~~~
In file included from /home/ma/.cache/common-lisp/ecl-16.1.3-unknown-linux-x64/home/ma/src/repos/cl-mpi/mpi/wrap__wrapper.c:6:
/usr/include/mpi.h:913:56: note: declared here
 OMPI_DECLSPEC extern struct ompi_predefined_datatype_t ompi_mpi_ub __mpi_interface_deprecated__("MPI_UB is deprecated in MPI-2.0");
                                                        ^~~~~~~~~~~
/home/ma/.cache/common-lisp/ecl-16.1.3-unknown-linux-x64/home/ma/src/repos/cl-mpi/mpi/wrap__wrapper.c: In function ‘cl_mpi_get_mpi_lb_cffi_wrap’:
/home/ma/.cache/common-lisp/ecl-16.1.3-unknown-linux-x64/home/ma/src/repos/cl-mpi/mpi/wrap__wrapper.c:219:3: warning: ‘ompi_mpi_lb’ is deprecated: MPI_LB is deprecated in MPI-2.0 [-Wdeprecated-declarations]
   return MPI_LB;
   ^~~~~~
In file included from /home/ma/.cache/common-lisp/ecl-16.1.3-unknown-linux-x64/home/ma/src/repos/cl-mpi/mpi/wrap__wrapper.c:6:
/usr/include/mpi.h:912:56: note: declared here
 OMPI_DECLSPEC extern struct ompi_predefined_datatype_t ompi_mpi_lb __mpi_interface_deprecated__("MPI_LB is deprecated in MPI-2.0");
                                                        ^~~~~~~~~~~
.
Condition of type: LOAD-FOREIGN-LIBRARY-ERROR
Unable to load foreign library (WRAP).
  LOAD-FOREIGN-MODULE: Could not load foreign module "/home/ma/.cache/common-lisp/ecl-16.1.3-unknown-linux-x64/home/ma/src/repos/cl-mpi/mpi/wrap.so" (Error: "/home/ma/.cache/common-lisp/ecl-16.1.3-unknown-linux-x64/home/ma/src/repos/cl-mpi/mpi/wrap.so: undefined symbol: ompi_mpi_op_prod")

Available restarts:

1. (RETRY) Try loading the foreign library again.
2. (USE-VALUE) Use another library instead.
3. (TRY-RECOMPILING) Recompile wrap and try loading it again
4. (RETRY) Retry loading FASL for #<mpi-wrapper-file "cl-mpi" "mpi" "wrap">.
5. (ACCEPT) Continue, treating loading FASL for #<mpi-wrapper-file "cl-mpi" "mpi" "wrap"> as having been successful.
6. (RETRY) Retry ASDF operation.
7. (CLEAR-CONFIGURATION-AND-RETRY) Retry ASDF operation after resetting the configuration.
8. (ABORT) Give up on "cl-mpi"
9. (RESTART-TOPLEVEL) Go back to Top-Level REPL.

Broken at SI:BYTECODES. [Evaluation of: (QUICKLISP-CLIENT:QUICKLOAD :CL-MPI)] In: #<process TOP-LEVEL>.

Solving the Stub Problem (TM)

One idea I had was to remove the stub library from the Lisp build chain entirely, and just depend on the user of the system to compile and install this stub themselves. It makes a (ql:quickload ...) inconvenient, but it makes the life of the actual target user of an MPI application a lot more convenient. Alternatively, there can be an ASDF system that does the building as-is right now, but it's separated out.

This way you can just build the stub C .so on the target machine of interest, which has all of the MPI stuff installed anyway, and dynamic linkage to the MPI of choice can happen through this stub.

The only problem I see is dealing with the groveling. Groveling is importing build-time information into the Lisp system, which may be invalid on the target system. Perhaps assuming different MPI implementations from build-to-execute is a little bit insane though. But maybe it is something that can be slogged through, since the grovel definitions are so small?

Comments welcome @marcoheisig

ECL fails to load CL-MPI :OUTPUT argument to RUN-PROGRAM does not have a file handle

Table of Contents

1. Problem
2. Versions
3. Steps to reproduce
4. Backtraces
5. Roswell and SLIME, too

Problem

The run-test-suite.sh script fails for ECL when attempting to build the test image with cl-launch.

I don't think this is a cl-mpi bug, but I'm filing this here to document it in case anyone else runs into this error while attempting to run the test suite.

The executive summary is that attempting to run the test suite or load cl-mpi in slime-repl fails under ECL with an error like:

:OUTPUT argument to RUN-PROGRAM does not have a file handle:
#<broadcast stream 0000555c0a748640>

It looks like this might be a bad interaction between ECL and cffi-grovel. See the following possibly-related bugs:

https://bugs.launchpad.net/cffi/+bug/1647237
https://bugs.launchpad.net/asdf/+bug/1663350
https://gitlab.com/embeddable-common-lisp/ecl/issues/357

Note that the same or similar errors affect the roswell/cl-mpi.ros script and the slime repl.

See below the full backtrace and more details.

Versions

ecl 16.1.3
cl-launch 4.1.4
cffi 0.20.0
cl-mpi 3e0e518
openmpi 3.1.3
gcc 8.3.0
linux 4.19.37-1-lts

Steps to reproduce

./scripts/run-test-suite.sh ecl

Backtraces

The first failure occurs when attempting load static-vectors.

./scripts/run-test-suite.sh ecl
=========================================
Building ecl (ECL 16.1.3) image ...
; gcc -o /home/ma/.cache/common-lisp/ecl-16.1.3-unknown-linux-x64/home/ma/opt/quicklisp/dists/quicklisp/software/static-vectors-v1.8.3/src/ffi-types__grovel-tmpPZJMZEP7.o -c -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -fno-plt -fPIC -D_THREAD_SAFE -Dlinux -fPIC -I/home/ma/opt/quicklisp/dists/quicklisp/software/cffi_0.20.0/ /home/ma/.cache/common-lisp/ecl-16.1.3-unknown-linux-x64/home/ma/opt/quicklisp/dists/quicklisp/software/static-vectors-v1.8.3/src/ffi-types__grovel.c
Fatal condition:
:OUTPUT argument to RUN-PROGRAM does not have a file handle:
#<broadcast stream 000055c46b74d500>
0: (#<bytecompiled-function CL-LAUNCH::BUILD-PROGRAM> ((CL-LAUNCH::PROGRAM-SYS "cl-launch-program") (CL-LAUNCH::DEPENDENCIES "cl-launch-build-1") (CL-LAUNCH::OP . ASDF/BUNDLE:PROGRAM-OP) (CL-LAUNCH::STANDALONE . T) (C:*SUPPRESS-COMPILER-NOTES* . T) (C:*SUPPRESS-COMPILER-WARNINGS* . T) (UIOP/IMAGE:QUIT . T) (CL-LAUNCH::INIT . "(cl:in-package :cl-mpi-test-suite)(unwind-protect (run-cl-mpi-test-suite) (mpi-finalize) (uiop:quit))") (CL-LAUNCH::FINAL) (RESTART) (CL-LAUNCH::BUILD (:LOAD-SYSTEM "cl-mpi-test-suite")) (CL-LAUNCH::DUMP . "./scripts/ecl.image")))
1: (#<bytecompiled-function 000055c46a3c9370> ((UIOP/IMAGE:QUIT . T) (CL-LAUNCH::INIT . "(cl:in-package :cl-mpi-test-suite)(unwind-protect (run-cl-mpi-test-suite) (mpi-finalize) (uiop:quit))") (CL-LAUNCH::FINAL) (RESTART) (CL-LAUNCH::DUMP . "./scripts/ecl.image") (CL-LAUNCH::BUILD (:LOAD-SYSTEM "cl-mpi-test-suite")) (CL-LAUNCH::SOURCE-REGISTRY) (CL-LAUNCH::QUICKLISP . T)))
2: (#<bytecompiled-function CL-LAUNCH::RUN> ((UIOP/IMAGE:QUIT . T) (CL-LAUNCH::INIT . "(cl:in-package :cl-mpi-test-suite)(unwind-protect (run-cl-mpi-test-suite) (mpi-finalize) (uiop:quit))") (CL-LAUNCH::FINAL) (RESTART) (CL-LAUNCH::DUMP . "./scripts/ecl.image") (CL-LAUNCH::BUILD (:LOAD-SYSTEM "cl-mpi-test-suite")) (CL-LAUNCH::SOURCE-REGISTRY) (CL-LAUNCH::QUICKLISP . T)))
3: (#<bytecompiled-function SI:BYTECODES> NIL)
4: (#<bytecompiled-function SI:BYTECODES> ((SI::OUTPUT-FILE . T) (SI::C-FILE) (SI::H-FILE) (SI::DATA-FILE) (SI::VERBOSE) (SI::SYSTEM-P) (EXT:QUIT)))
5: (#<bytecompiled-function SI:BYTECODES> NIL)
6: (NIL NIL)
Above backtrace due to this condition:
:OUTPUT argument to RUN-PROGRAM does not have a file handle:
#<broadcast stream 000055c46b74d500>
:OUTPUT argument to RUN-PROGRAM does not have a file handle:
#<broadcast stream 000055c46b74d500>
Failed to dump an image
...ecl image creation failed
=========================================

This only appears to be an issue when running under cl-launch though. If I start an ECL repl from the command line, then (ql:quickload :static-vectors) completes successfully.

If I then try to run the cl-mpi test suite again, I get a similar grovelling failure when loading cl-mpi.

./scripts/run-test-suite.sh ecl
=========================================
Building ecl (ECL 16.1.3) image ...
; mpicc -o /home/ma/.cache/common-lisp/ecl-16.1.3-unknown-linux-x64/home/ma/src/repos/cl-mpi/mpi/grovel__grovel-tmpEG1C6YMO.o -c -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -fno-plt -fPIC -D_THREAD_SAFE -Dlinux -fPIC -I/home/ma/opt/quicklisp/dists/quicklisp/software/cffi_0.20.0/ /home/ma/.cache/common-lisp/ecl-16.1.3-unknown-linux-x64/home/ma/src/repos/cl-mpi/mpi/grovel__grovel.c
Fatal condition:
:OUTPUT argument to RUN-PROGRAM does not have a file handle:
#<broadcast stream 0000555c0a748640>
0: (#<bytecompiled-function CL-LAUNCH::BUILD-PROGRAM> ((CL-LAUNCH::PROGRAM-SYS "cl-launch-program") (CL-LAUNCH::DEPENDENCIES "cl-launch-build-1") (CL-LAUNCH::OP . ASDF/BUNDLE:PROGRAM-OP) (CL-LAUNCH::STANDALONE . T) (C:*SUPPRESS-COMPILER-NOTES* . T) (C:*SUPPRESS-COMPILER-WARNINGS* . T) (UIOP/IMAGE:QUIT . T) (CL-LAUNCH::INIT . "(cl:in-package :cl-mpi-test-suite)(unwind-protect (run-cl-mpi-test-suite) (mpi-finalize) (uiop:quit))") (CL-LAUNCH::FINAL) (RESTART) (CL-LAUNCH::BUILD (:LOAD-SYSTEM "cl-mpi-test-suite")) (CL-LAUNCH::DUMP . "./scripts/ecl.image")))
1: (#<bytecompiled-function 0000555c093c6370> ((UIOP/IMAGE:QUIT . T) (CL-LAUNCH::INIT . "(cl:in-package :cl-mpi-test-suite)(unwind-protect (run-cl-mpi-test-suite) (mpi-finalize) (uiop:quit))") (CL-LAUNCH::FINAL) (RESTART) (CL-LAUNCH::DUMP . "./scripts/ecl.image") (CL-LAUNCH::BUILD (:LOAD-SYSTEM "cl-mpi-test-suite")) (CL-LAUNCH::SOURCE-REGISTRY) (CL-LAUNCH::QUICKLISP . T)))
2: (#<bytecompiled-function CL-LAUNCH::RUN> ((UIOP/IMAGE:QUIT . T) (CL-LAUNCH::INIT . "(cl:in-package :cl-mpi-test-suite)(unwind-protect (run-cl-mpi-test-suite) (mpi-finalize) (uiop:quit))") (CL-LAUNCH::FINAL) (RESTART) (CL-LAUNCH::DUMP . "./scripts/ecl.image") (CL-LAUNCH::BUILD (:LOAD-SYSTEM "cl-mpi-test-suite")) (CL-LAUNCH::SOURCE-REGISTRY) (CL-LAUNCH::QUICKLISP . T)))
3: (#<bytecompiled-function SI:BYTECODES> NIL)
4: (#<bytecompiled-function SI:BYTECODES> ((SI::OUTPUT-FILE . T) (SI::C-FILE) (SI::H-FILE) (SI::DATA-FILE) (SI::VERBOSE) (SI::SYSTEM-P) (EXT:QUIT)))
5: (#<bytecompiled-function SI:BYTECODES> NIL)
6: (NIL NIL)
Above backtrace due to this condition:
:OUTPUT argument to RUN-PROGRAM does not have a file handle:
#<broadcast stream 0000555c0a748640>
:OUTPUT argument to RUN-PROGRAM does not have a file handle:
#<broadcast stream 0000555c0a748640>
Failed to dump an image
...ecl image creation failed
=========================================

Roswell and SLIME, too

The same issue appears to affect the roswell/cl-mpi.ros script (after minor modifications to remove references to obsolete cl-mpi-benchmarks).

./roswell/cl-mpi.ros
; gcc -o /home/ma/.cache/common-lisp/ecl-16.1.3-unknown-linux-x64/home/ma/.roswell/lisp/quicklisp/dists/quicklisp/software/static-vectors-v1.8.3/src/ffi-types__grovel-tmpYY7JI19B.o -c -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -fno-plt -fPIC -D_THREAD_SAFE -Dlinux -fPIC -I/home/ma/.roswell/lisp/quicklisp/dists/quicklisp/software/cffi_0.20.0/ /home/ma/.cache/common-lisp/ecl-16.1.3-unknown-linux-x64/home/ma/.roswell/lisp/quicklisp/dists/quicklisp/software/static-vectors-v1.8.3/src/ffi-types__grovel.c
An error occurred during initialization:
:OUTPUT argument to RUN-PROGRAM does not have a file handle:
#<broadcast stream 000056509e51adc0>.

Likewise when attempting to load cl-mpi from a slime repl:

:INPUT argument to RUN-PROGRAM does not have a file handle:
#<a SWANK/GRAY::SLIME-INPUT-STREAM>
   [Condition of type CFFI-GROVEL:GROVEL-ERROR]