jameslikeslinux / rdma_dist

Experimental RDMA Distribution Driver for Erlang

RDMA Distribution Driver for Erlang

This is an alternative distribution driver for the Erlang virtual machine. It implements a native protocol for RDMA devices such as InfiniBand. This can potentially improve messaging latency and throughput compared to the default TCP distribution protocol.

Building

This software can be added as a dependency in your project using Rebar or Mix. It requires OFED to be installed. If it is installed somewhere outside of the default locations, set the CFLAGS and LDFLAGS environmental variables appropriately before compiling.

Using

The software's ebin directory must be added to the Erlang VM's code path. Then you can tell the VM to use the driver by passing the -proto_dist rdma option. Here is an example:

• Erlang:

  % erl +K true -pa rdma_dist/ebin -proto_dist rdma -name [email protected]
  

• Elixir:

  % iex --erl "+K true -pa rdma_dist/ebin -proto_dist rdma -name [email protected]"
  

Kernel polling (+K true) is recommended. If the RDMA device isn't the default interface on the system, explicitly name it in the -name option. For InfiniBand devices, you must have an IP-over-IB device set up---it will be used for connection management. Once the connection is established, the driver will use native RDMA verbs.

Testing

Tests can be run with make test.

This software has seen limited testing over the following devices:

• Qlogic QDR InfiniBand
• Mellanox QDR InfiniBand

No real-world applications have been tested using this distribution driver. If you have success with other RDMA devices or large Erlang applications, let me know :)

rdma_dist's Issues

Problems compiling the C source

Dear Sir

Thank you for creating this project! I would like to try this out on my local HPC cluster. I can't claim the following is an issue with the project code: I imagine it is an issue with my knowledge of the local HPC cluster setup.

I am having problems compiling the C source. This gist has the rebar.config I am using, and the error output from rebar compile: https://gist.github.com/llaisdy/9c98fdc2ab94b4abb20a

I have added various include and lib paths to the rebar.config. I am compiling with gcc4.8 and erlang 17.1

My C is rudimentary, but the error messages look like the compiler can't find certain libraries --- e.g. I think some of the definitions it can't find are in stddef.h --- but stddef.h is in the -I paths specified.

The HPC system I'm working on seems to have different versions of header files like stddef.h in different places. Also, if I change the order of the -I paths in the CFLAGS parameter, I get different error messages.

I wonder if the compiler is finding incorrect versions of the libraries it needs, or if the three separate paths I am giving in rebar.config cause conflicts.

Does anything obvious leap out at you about these error messages?

Can you suggest a way to include all and only the header files that I need?

With thanks and best wishes

Ivan