spirali / aislinn

Dynamic verifier for MPI programs

About me 🧚‍♀️

• I am a developer and research engineer at ACS Research Group and IT4Innovations.
• I am interested in AI Safety, HPC, and formal verification.
• I have PhD in Computer Science.
• I work mostly in Rust and Python; sometimes in TypeScript. In past C, C++, Haskell.
• I love digital painting and playing the cello.

Selected projects where I am main author/co-author

• HyperQueue - user-friendly and scalable job scheduler for supercomputers
• Nelsie - slide making software
• Rain - framework for large distributed pipelines
• Interlab - toolkit for multi-agent interactions
• RSDS - Dask server reimplemented in Rust
• Nedoc - non-evaluating documentation generator for Python
• Haydi - Python framework for generating discrete structures)
• Aislinn - dynamic verifier for MPI programs
• Estee - simulator for task-based workflow schedulers
• ORCO - Python package for defining, executing, and persisting computations
• Replay-cache - replay cache for LangChain
• RMahjong - Riichi Mahjong
• LabLab - A simple image anotation tool

aislinn's Issues

AttributeError: 'module' object has no attribute 'InvalidPersistentRequest'

Hello,

The following program raises an error in Aislinn:

#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

#define buff_size 128

int main(int argc, char **argv) {
  int nprocs = -1;
  int rank = -1;

  MPI_Init(&argc, &argv);
  MPI_Comm_size(MPI_COMM_WORLD, &nprocs);
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
  printf("Hello from rank %d \n", rank);

  if (nprocs < 2)
    printf("MBI ERROR: This test needs at least 2 processes to produce a bug!\n");

  int dest = (rank == nprocs - 1) ? (0) : (rank + 1);
  int src = (rank == 0) ? (nprocs - 1) : (rank - 1); 

  int buf1[buff_size]; MPI_Request req1;
  int buf2[buff_size]; MPI_Request req2;
  MPI_Send_init(buf1, buff_size, MPI_INT, dest, 0, MPI_COMM_WORLD, &req1);
  MPI_Start(&req1);
  MPI_Irecv(buf2, buff_size, MPI_INT, src, 0, MPI_COMM_WORLD, &req2); 
  
	 /* MISSING: MPI_Wait(&req1, MPI_STATUS_IGNORE); */ 
	 MPI_Wait(&req2, MPI_STATUS_IGNORE);  
	MPI_Request_free(&req1);
	MPI_Request_free(&req2);

  MPI_Finalize();
  printf("Rank %d finished normally\n", rank);
  return 0;
}

Here is the output of Aislinn in this case:

 $ aislinn -p=2 MissingWait_Send_init_Irecv_nok
==AN== INFO: Aislinn v0.5.0
Traceback (most recent call last):
  File "/builds/quinson/mbi2/tools/aislinn-git/bin/../src/aislinn/aislinn.py", line 359, in <module>
    main()
  File "/builds/quinson/mbi2/tools/aislinn-git/bin/../src/aislinn/aislinn.py", line 315, in main
    if not generator.run():
  File "/builds/quinson/mbi2/tools/aislinn-git/src/aislinn/mpi/generator.py", line 195, in run
    self.main_cycle()
  File "/builds/quinson/mbi2/tools/aislinn-git/src/aislinn/mpi/generator.py", line 184, in main_cycle
    context.process_run_result(c.finish_async())
  File "/builds/quinson/mbi2/tools/aislinn-git/src/aislinn/mpi/context.py", line 150, in process_run_result
    if self.handle_call(result[1], result[2:]):
  File "/builds/quinson/mbi2/tools/aislinn-git/src/aislinn/mpi/context.py", line 123, in handle_call
    return call.run(self, args)
  File "/builds/quinson/mbi2/tools/aislinn-git/src/aislinn/mpi/mpicalls.py", line 921, in run
    for i in xrange(len(args))])
  File "/builds/quinson/mbi2/tools/aislinn-git/src/aislinn/mpi/mpicalls.py", line 240, in MPI_Request_free
    request = check.check_persistent_request(context, request_id, False, 1)
  File "/builds/quinson/mbi2/tools/aislinn-git/src/aislinn/mpi/check.py", line 166, in check_persistent_request
    context.add_error_and_throw(errormsg.InvalidPersistentRequest(
AttributeError: 'module' object has no attribute 'InvalidPersistentRequest'

Thanks for this tool,
Mt

AssertionError: assert marks.issuperset(m)

Hello,

here is another simple MPI code that raises an exception in Aislinn:

#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

#define buff_size 128

int main(int argc, char **argv) {
  int nprocs = -1;
  int rank = -1;

  MPI_Init(&argc, &argv);
  MPI_Comm_size(MPI_COMM_WORLD, &nprocs);
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
  printf("Hello from rank %d \n", rank);

  if (nprocs < 2)
    printf("MBI ERROR: This test needs at least 2 processes to produce a bug.\n");

  int dbs = sizeof(int)*nprocs; /* Size of the dynamic buffers for alltoall and friends */
  MPI_Request req1; MPI_Status sta1; int sum1, val1 = 1;
  int buf2[buff_size];MPI_Request req2;MPI_Status sta2;
  int root = 0;

  if (rank % 2) {
    MPI_Ireduce(&sum1, &val1, 1, MPI_INT, MPI_SUM, 0, MPI_COMM_WORLD, &req1); MPI_Wait(&req1,&sta1); /* MBIERROR1 */
    
  } else {
     /* MBIERROR2 */
    
  }
  
  MPI_Finalize();
  printf("Rank %d finished normally\n", rank);
  return 0;
}

Note that this code is actually buggy: Odd ranks call MPI_Ireduce while even ranks do not call any collective

Here is the resulting error message:

 $ aislinn -p=2 CollCallOrder_Ireduce_none_nok
==AN== INFO: Aislinn v0.5.0
Traceback (most recent call last):
  File "/builds/quinson/mbi2/tools/aislinn-git/bin/../src/aislinn/aislinn.py", line 359, in <module>
    main()
  File "/builds/quinson/mbi2/tools/aislinn-git/bin/../src/aislinn/aislinn.py", line 315, in main
    if not generator.run():
  File "/builds/quinson/mbi2/tools/aislinn-git/src/aislinn/mpi/generator.py", line 200, in run
    self.ndsync_check()
  File "/builds/quinson/mbi2/tools/aislinn-git/src/aislinn/mpi/generator.py", line 212, in ndsync_check
    message = NdsyncChecker(self, self.statespace).run()
  File "/builds/quinson/mbi2/tools/aislinn-git/src/aislinn/mpi/ndsync.py", line 72, in run
    self.process_node(node, markings)
  File "/builds/quinson/mbi2/tools/aislinn-git/src/aislinn/mpi/ndsync.py", line 113, in process_node
    assert marks.issuperset(m)
AssertionError

Concurrency error in your implementation of Allgatherv?

Hello,

I get an 'Invalid write' error in any code that contains a call to MPI_Allgatherv(), such as the following. This code is supposed to be correct.

#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

#define buff_size 128

int main(int argc, char **argv) {
  int nprocs = -1;
  int rank = -1;

  MPI_Init(&argc, &argv);
  MPI_Comm_size(MPI_COMM_WORLD, &nprocs);
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
  printf("Hello from rank %d \n", rank);

  if (nprocs < 2)
    printf("MBI ERROR: This test needs at least 2 processes to produce a bug.\n");

  int dbs = sizeof(int)*nprocs; /* Size of the dynamic buffers for alltoall and friends */
  int *rbuf1 = malloc(dbs), *rcounts1=malloc(dbs),  *displs1=malloc(dbs);
  for (int i = 0; i < nprocs; i++) {
    rcounts1[i] = 1;
    displs1[i] = 2 * (nprocs - (i + 1));
  }

  if (rank % 2) {
    MPI_Allgatherv(&rank, 1, MPI_INT, rbuf1, rcounts1, displs1, MPI_INT, MPI_COMM_WORLD);  
  } else {
    MPI_Allgatherv(&rank, 1, MPI_INT, rbuf1, rcounts1, displs1, MPI_INT, MPI_COMM_WORLD);
    
  }

  free(rbuf1);free(rcounts1);free(displs1);
  
  
  MPI_Finalize();
  printf("Rank %d finished normally\n", rank);
  return 0;
}

The produced stack trace (in the HTML) reads as follows:

0x10CF47: aislinn_call (aislinn.c:61)
0x10B9EE: MPI_Allgatherv (mpi_generated.c:643)
0x108CD9: main (CollCorrect_Allgatherv.c:59)

AssertionError: self.datatype == datatype

Hello again,

here is another buggy code raising an exception in Aislinn. The code is wrong because not all the ranks use the same operation in the Reduce.

#include <mpi.h>
#include <stdio.h>

#define buff_size 128


int main(int argc, char **argv) {
  int nprocs = -1;
  int rank = -1;

  MPI_Init(&argc, &argv);
  MPI_Comm_size(MPI_COMM_WORLD, &nprocs);
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
  printf("Hello from rank %d \n", rank);

  if (nprocs < 2)
    printf("MBI ERROR: This test needs at least 2 processes to produce a bug!\n");

  MPI_Op op = MPI_SUM;
  if (rank % 2)
    op = MPI_MAX;

  int sum1, val1 = 1;
  MPI_Reduce(&sum1, &val1, 1, MPI_INT, op, 0, MPI_COMM_WORLD); /* MBIERROR */
	

  MPI_Finalize();
  printf("Rank %d finished normally\n", rank);
  return 0;
}

Here is the produced output:

$ aislinn-cc -g /builds/quinson/mbi2/gencodes/CollOpMatching_Reduce_nok.c -o CollOpMatching_Reduce_nok
$ aislinn -p=2 CollOpMatching_Reduce_nok
==AN== INFO: Aislinn v0.5.0
Traceback (most recent call last):
  File "/builds/quinson/mbi2/tools/aislinn-git/bin/../src/aislinn/aislinn.py", line 359, in <module>
    main()
  File "/builds/quinson/mbi2/tools/aislinn-git/bin/../src/aislinn/aislinn.py", line 315, in main
    if not generator.run():
  File "/builds/quinson/mbi2/tools/aislinn-git/src/aislinn/mpi/generator.py", line 195, in run
    self.main_cycle()
  File "/builds/quinson/mbi2/tools/aislinn-git/src/aislinn/mpi/generator.py", line 184, in main_cycle
    context.process_run_result(c.finish_async())
  File "/builds/quinson/mbi2/tools/aislinn-git/src/aislinn/mpi/context.py", line 150, in process_run_result
    if self.handle_call(result[1], result[2:]):
  File "/builds/quinson/mbi2/tools/aislinn-git/src/aislinn/mpi/context.py", line 123, in handle_call
    return call.run(self, args)
  File "/builds/quinson/mbi2/tools/aislinn-git/src/aislinn/mpi/mpicalls.py", line 921, in run
    for i in xrange(len(args))])
  File "/builds/quinson/mbi2/tools/aislinn-git/src/aislinn/mpi/mpicalls.py", line 423, in MPI_Reduce
    args)
  File "/builds/quinson/mbi2/tools/aislinn-git/src/aislinn/mpi/mpicalls.py", line 790, in call_collective_operation
    context, comm, op_class, blocking, args)
  File "/builds/quinson/mbi2/tools/aislinn-git/src/aislinn/mpi/globalstate.py", line 116, in call_collective_operation
    op.enter(context, comm, args)
  File "/builds/quinson/mbi2/tools/aislinn-git/src/aislinn/mpi/collectives.py", line 66, in enter
    self.enter_main(context, comm, args)
  File "/builds/quinson/mbi2/tools/aislinn-git/src/aislinn/mpi/collectives.py", line 653, in enter_main
    self.datatype == datatype)
AssertionError