galoisinc / pirate

Main repository for the Galois/Two Six Pirate Project for DARPA GAPS

PIRATE

This is main software repository for the Pirate distributed-computing technologies. Our goal is to develop technologies that help make it easier to build efficient, safe and secure distributed systems.

The main artifacts available are:

• The pirate-llvm compiler.
• The libpirate library in this repo, and
• Demonstration applications that use the above tools.

Trying out the system is very easy -- we provide a Docker image with our tools, documentation, and demos preinstalled, and provide online html documentation. The docker image can be installed and run by running

docker run -it pirateteam/ubuntu

This will download the image from Docker Hub, and start an interactive bash session in the root account. The pirate directory contains the demos, and HTML documentaton. The compiler toolchain and libpirate are pre-installed in /usr/local.

If you are not able to run Docker in your environment, we provide tarball archives of the most recent master builds for Linux versions:

• Centos 7
• Ubuntu 20.04

We also upload artifacts of our libraries as part of continuous integration.

If you want to build the source yourself, the pirate-llvm repository has links to instructions for building it, and instructions for building the other components are available below.

This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. HR0011-19-C-0103.

Building

To build the libpirate library:

$ cd pirate
$ mkdir build
$ cd build
$ cmake ..
$ make

To build the channel demo application that uses libpirate:

$ cd pirate
$ mkdir build
$ cd build
$ cmake -DCHANNEL_DEMO=ON ..
$ make

To build all the demo applications that use libpirate:

$ cd pirate
$ mkdir build
$ cd build
$ cmake -DGAPS_DEMOS=ON ..
$ make

Additional cmake Options

Invoke with

$ cmake -D<OPTION_NAME>=<OPTON_VALUE> ..

• PIRATE_LAUNCHER build the application launcher (default ON)
• PIRATE_GETOPT build resource loader library (default ON)
• GAPS_DISABLE use the standard compiler installed on the system instead of the GAPS LLVM compiler (default OFF)
• PIRATE_UNIT_TEST enable compilation of libpirate unit tests (requires googletest v1.10 or greater, default OFF)
• CHANNEL_DEMO enable compilation of GAPS channel application (default OFF)
• GAPS_DEMOS enable compilation of all GAPS demo applications (default OFF)
• GAPS_BENCH enable compilation of GAPS benchmark applications (default OFF)
• PIRATE_SHMEM_FEATURE support shared memory channels (requires libpthread and librt, default OFF)
• BUILD_ALL enables PIRATE_SHMEM_FEATURE, PIRATE_UNIT_TEST, GAPS_DEMOS, and GAPS_BENCH (default OFF)
• SINGLE_BINARY encrypt and combine application binaries into a single executable (default OFF)

libpirate

Pirate primitives layer. The PIRATE core primitives layer will provide a series of capabilities for executing PIRATE executables on TA1 hardware. At minimum, there are four basic primitives that must be supported: configuring TA1 hardware, loading code and data onto the appropriate CPU, implementing channel send and receive calls, and resource cleanup / data wipe on termination.

libpirate currently implements GAPS channels using Linux named pipes, a character device driver, a Unix domain socket, shared memory, network communication, or userspace IO. Benchmarks are available on the wiki.

demos

channel_demo

Provide a simple application for validating GAPS channel API and functionality, continuous integration of the PIRATE library, facilitating hardware integration, and debugging. The channel demo has a reader and a writer that can be configured to use different channel types and transmit different patterns of data. (more information)

simple_demo

Adapted a simple webserver as a demonstration of an application using GAPS channels. Manually separated the program into two executables. The low side application is a webserver that sends http requests to the high side application. The high side application filters the html response before sending it to the low side application. Uses the libpirate API. (more information)

Adapted from http://www.cs.cmu.edu/afs/cs/academic/class/15213-s00/www/class28/tiny.c

The build script for simple_demo includes an experiment that builds a single executable that contains both the low-side and high-side executables.

time_demo

Trusted timestamping is the process of tracking the time that data was created or modified. A trusted timestamp generally identifies the data that timestamped (typically by a secure hash of the data), the time that the data was timestamped, and a digital signature or other evidence that the timestamp should be trusted. This is typically a digitial signature signed by a trusted third party, but could include additional evidence such as information needed to locate the timestamp in a blockchain ledger. The additional information could be used to provide alternate methods of establishing trust if the private key associated with the digital signature is lost. (more information)

pirate's Issues

Passing too-short `len` into `pirate_mercury_get_channel_description` causes buffer overflow

Calling pirate_mercury_get_channel_description with session.message_count > 0 and a buffer too short to contain the message ids causes a buffer overflow, even when len correctly reflects the buffer size. The problem is the following code (lines 294-299 of mercury.c):

    for (uint32_t i = 0; i < param->session.message_count; ++i) {
        wr += wr_sz;
        len -= wr_sz;
        wr_sz = snprintf(wr, len, ",%u", param->session.messages[i]);
        ret_sz += wr_sz;
    }

Note that, from printf(3),

If the output [of snprintf] was truncated ..., then the return value is the number of characters (excluding the terminating null byte) which would have been written to the final string if enough space had been available.

If len is too short, it is decreased past 0 (len is signed), then converted to size_t, causing an overflow. In this case snprintf will write past the end of the buffer.

`mtu` is ignored in `pirate_mercury_get_channel_description`

pirate_mercury_get_channel_description does not add mtu=XXX to the description string when it is present in the params struct, contrary to what the docs in libpirate.h suggest.

Build pal statically and include in CI-generated archives.

We are currently building pal dynamically and dynamically linking in libcyaml. We need to build it statically so that it does not add any runtime dependencies to the target machine and validate it is working when run on a minimal centos7 system.

O_NONBLOCK on stream-based channels

The current implementation of stream-based channels (unix pipes, unix sockets, tcp sockets) is incorrect when the O_NONBLOCK flag is used to open the channel. On pirate_read() if the channel contains only a partial datagram, then EAGAIN or EWOULDBLOCK errors can be returned after consuming a partial datagram from the channel. The channel would now be in an error state with a partial datagram in the channel. On pirate_write() if the channel is nearly full, then EAGAIN or EWOULDBLOCK errors can be returned after sending a partial datagram to the channel. In both cases the non-blocking behavior causes partial datagrams to appear. Which is not allowed as all pirate channels have a datagram abstraction.

select, poll, and epoll on these channels is working. It is valid to use the stream channels in blocking mode and use the wait functions to test whether the channels are available for reading or writing. Also using a UDP channel in non-blocking mode is another existing workaround.

There are several potential solutions to this problem.

1. Do not allow O_NONBLOCK on unix pipes, unix sockets, and tcp sockets. This is somewhat of a disappointment. We'd like to support all potential use cases across all channels. I think I've decided on a variation on this approach. But a compromise variation that I think will be tolerated.

2. Allow these channel types to be opened with the O_NONBLOCK flag. On each call to pirate_read() the first read() system call is non-blocking. If the entire datagram is consumed then we are done. If a partial datagram is read then fcntl() is invoked to temporarily change the file descriptor into blocking mode. The remainder of the datagram is consumed in blocking mode. Finally fcntl() is used to change the file descriptor back into non-blocking mode. Identically behavior for pirate_write(). I don't like this solution. There's the overhead of switching between blocking and non-blocking mode. Also it goes against the programmers intent. They requested non-blocking I/O and we are providing blocking I/O in some cases.

3. Use a temporary buffer for pirate_read(). If a partial datagram is read from a pirate_read() then place the partial datagram into the temporary buffer. Return the complete datagram in the subsequent request to pirate_read() that completes the end of the datagram. I like this solution. But I can't come up with an analogous solution for pirate_write().

4. Do not allow O_NONBLOCK on unix sockets and tcp sockets.
   a. We might be able to allow O_NONBLOCK on named pipes but only if fcntl can be used to set O_DIRECT on named pipes. O_DIRECT on anonymous pipes can be used to place them into "datagram mode". Which is the semantics that we want. I'm not sure whether this is possible on named pipes. See https://man7.org/linux/man-pages/man2/pipe.2.html. We have unit tests for the datagram semantics of channels so this should be easy to test.
   b. Add a new channel type for the SOCK_SEQPACKET socket type. This provides a sequenced, reliable, two-way connection based data transmission path for datagrams of fixed maximum length. We could support O_NONBLOCK on this channel type. It would be an alternative to unix sockets and tcp sockets for non-blocking I/O.

I'm leaning towards solution (4) which is basically solution (1) with the workarounds of O_DIRECT on named pipes (assuming that works) and adding the SOCK_SEQPACKET channel type.

The 'assemble' tasks for cutting a release can fail

I'm trying to issue a new release that includes the PDL deployment tool. The tasks that run on a release (but not on a typical merge/PR) have been failing for me with linker errors. You can see the failure here and I've copied the relevant log below.

2021-02-15T18:09:32.5139546Z -- The C compiler identification is Clang 10.0.0
2021-02-15T18:09:32.5638912Z -- The CXX compiler identification is Clang 10.0.0
2021-02-15T18:09:32.5714695Z -- Check for working C compiler: /usr/local/bin/cc
2021-02-15T18:09:32.6493334Z -- Check for working C compiler: /usr/local/bin/cc -- works
2021-02-15T18:09:32.6506216Z -- Detecting C compiler ABI info
2021-02-15T18:09:32.7446392Z -- Detecting C compiler ABI info - done
2021-02-15T18:09:32.7566909Z -- Detecting C compile features
2021-02-15T18:09:32.7567949Z -- Detecting C compile features - done
2021-02-15T18:09:32.7604009Z -- Check for working CXX compiler: /usr/local/bin/c++
2021-02-15T18:09:32.8474141Z -- Check for working CXX compiler: /usr/local/bin/c++ -- works
2021-02-15T18:09:32.8487699Z -- Detecting CXX compiler ABI info
2021-02-15T18:09:32.9342935Z -- Detecting CXX compiler ABI info - done
2021-02-15T18:09:32.9501883Z -- Detecting CXX compile features
2021-02-15T18:09:32.9507602Z -- Detecting CXX compile features - done
2021-02-15T18:09:32.9614471Z -- Found OpenSSL: /usr/lib/x86_64-linux-gnu/libcrypto.so (found version "1.1.1f")  
2021-02-15T18:09:32.9791037Z -- Freespace library not found
2021-02-15T18:09:32.9791788Z -- Ffmpeg libraries not found
2021-02-15T18:09:32.9792442Z -- Pigpio library not found
2021-02-15T18:09:32.9793366Z -- Jpeg library found
2021-02-15T18:09:32.9793915Z -- X11 library found
2021-02-15T18:09:32.9819562Z -- Configuring done
2021-02-15T18:09:33.0440795Z -- Generating done
2021-02-15T18:09:33.0483020Z -- Build files have been written to: /root/build
2021-02-15T18:09:33.6925366Z Scanning dependencies of target low
2021-02-15T18:09:33.7056633Z [  1%] Building C object simple_demo/CMakeFiles/low.dir/src/demo.c.o
2021-02-15T18:09:33.8524609Z [  1%] Building C object simple_demo/CMakeFiles/low.dir/src/tiny.c.o
2021-02-15T18:09:33.9351369Z [  2%] Linking C executable low/low
2021-02-15T18:09:33.9642189Z ld.lld: error: undefined symbol: pirate_init_channel_param
2021-02-15T18:09:33.9643025Z >>> referenced by demo.c:513 (/root/demos/simple_demo/src/demo.c:513)
2021-02-15T18:09:33.9643757Z >>>               CMakeFiles/low.dir/src/demo.c.o:(main_low)
2021-02-15T18:09:33.9654555Z 
2021-02-15T18:09:33.9655198Z ld.lld: error: undefined symbol: pirate_open_param
2021-02-15T18:09:33.9655962Z >>> referenced by demo.c:516 (/root/demos/simple_demo/src/demo.c:516)
2021-02-15T18:09:33.9656694Z >>>               CMakeFiles/low.dir/src/demo.c.o:(main_low)
2021-02-15T18:09:33.9657399Z >>> referenced by demo.c:522 (/root/demos/simple_demo/src/demo.c:522)
2021-02-15T18:09:33.9658146Z >>>               CMakeFiles/low.dir/src/demo.c.o:(main_low)
2021-02-15T18:09:33.9658549Z 
2021-02-15T18:09:33.9659034Z ld.lld: error: undefined symbol: pirate_close
2021-02-15T18:09:33.9659755Z >>> referenced by demo.c:550 (/root/demos/simple_demo/src/demo.c:550)
2021-02-15T18:09:33.9660738Z >>>               CMakeFiles/low.dir/src/demo.c.o:(main_low)
2021-02-15T18:09:33.9661494Z >>> referenced by demo.c:554 (/root/demos/simple_demo/src/demo.c:554)
2021-02-15T18:09:33.9662212Z >>>               CMakeFiles/low.dir/src/demo.c.o:(main_low)
2021-02-15T18:09:33.9662917Z >>> referenced by demo.c:550 (/root/demos/simple_demo/src/demo.c:550)
2021-02-15T18:09:33.9664019Z >>>               CMakeFiles/low.dir/src/demo.c.o:(main_low)
2021-02-15T18:09:33.9664728Z >>> referenced by demo.c:554 (/root/demos/simple_demo/src/demo.c:554)
2021-02-15T18:09:33.9665482Z >>>               CMakeFiles/low.dir/src/demo.c.o:(main_low)
2021-02-15T18:09:33.9665891Z 
2021-02-15T18:09:33.9666375Z ld.lld: error: undefined symbol: pirate_write
2021-02-15T18:09:33.9667095Z >>> referenced by demo.c:116 (/root/demos/simple_demo/src/demo.c:116)
2021-02-15T18:09:33.9667866Z >>>               CMakeFiles/low.dir/src/demo.c.o:(webserver_thread)
2021-02-15T18:09:33.9668619Z >>> referenced by demo.c:122 (/root/demos/simple_demo/src/demo.c:122)
2021-02-15T18:09:33.9669484Z >>>               CMakeFiles/low.dir/src/demo.c.o:(webserver_thread)
2021-02-15T18:09:33.9669937Z 
2021-02-15T18:09:33.9670412Z ld.lld: error: undefined symbol: pirate_read
2021-02-15T18:09:33.9671129Z >>> referenced by demo.c:130 (/root/demos/simple_demo/src/demo.c:130)
2021-02-15T18:09:33.9671880Z >>>               CMakeFiles/low.dir/src/demo.c.o:(webserver_thread)
2021-02-15T18:09:33.9672648Z >>> referenced by demo.c:140 (/root/demos/simple_demo/src/demo.c:140)
2021-02-15T18:09:33.9673419Z >>>               CMakeFiles/low.dir/src/demo.c.o:(webserver_thread)
2021-02-15T18:09:33.9674170Z >>> referenced by demo.c:152 (/root/demos/simple_demo/src/demo.c:152)
2021-02-15T18:09:33.9675110Z >>>               CMakeFiles/low.dir/src/demo.c.o:(webserver_thread)
2021-02-15T18:09:33.9687969Z clang-10: error: linker command failed with exit code 1 (use -v to see invocation)
2021-02-15T18:09:33.9718113Z make[2]: *** [simple_demo/CMakeFiles/low.dir/build.make:99: simple_demo/low/low] Error 1
2021-02-15T18:09:33.9723442Z make[1]: *** [CMakeFiles/Makefile2:376: simple_demo/CMakeFiles/low.dir/all] Error 2
2021-02-15T18:09:33.9726868Z make: *** [Makefile:84: all] Error 2
2021-02-15T18:09:34.1178882Z ##[error]Process completed with exit code 2.