RDMA like RoCE must support cuda10? about byteps

[ymjiang]

RDMA does not necessarily needs cuda10. For now we only provide docker files with cuda10 that support RDMA. But you can try to build your own cuda9+RDMA image.

[yangwenhuan]

@ymjiang

[ymjiang]

The images we provide in the tutorials need cuda10. But that does not mean RDMA needs cuda10. Like I said, you can build your own images with cuda9, and make sure to install RDMA drivers properly.

[yangwenhuan]

@ymjiang , I used RoCE, but got 'unreachable' error, did it matter with network bonding?

[ymjiang]

Did you set DMLC_PS_ROOT_URI and DMLC_INTERFACE correctly according to your RDMA device?

Can you run through this benchmark with RDMA? Remember to use make -j USE_RDMA=1 for building and export DMLC_ENABLE_RDMA=1 for running.

[yangwenhuan]

@ymjiang , I am not sure about DMLC_INTERFACE, how to check it?

[ymjiang]

Looks like you are using docker. When you launched the docker container, did you add the --device arguments as shown in the tutorial? It should carry your RDMA device information on the host machine.

[yangwenhuan]

@ymjiang , I am not sure about DMLC_INTERFACE, how to check it?

@ymjiang , this is the screenshot of my host machine, not in docker.

[yangwenhuan]

@ymjiang , I used RoCE, but got 'unreachable' error, did it matter with network bonding?

this screenshot is in docker, and I did specify --device option:

docker run -it --net=host --device /dev/infiniband/rdma_cm --device /dev/infiniband/issm0 --device /dev/infiniband/ucm0 --device /dev/infiniband/umad0 --device /dev/infiniband/uverbs0 --cap-add IPC_LOCK bytepsimage/byteps_server_rdma bash

[bobzhuyb]

This may be due to bonding. I am not sure. Can you try ib_write_bw with rdma_cm enabled? It should be ib_write_bw -z, or ib_write_bw --com_rdma_cm

Can you also check the configuration details of bonding here https://community.mellanox.com/s/article/howto-create-linux-bond--lag--interface-over-infiniband-network

Quote:

Since rdma_cm does not support MAC enslavement, fail_over_mac=1 (or active) should be added to the bond interface configuration in case of Ethernet interface.

[yangwenhuan]

This may be due to bonding. I am not sure. Can you try ib_write_bw with rdma_cm enabled? It should be ib_write_bw -z, or ib_write_bw --com_rdma_cm

Can you also check the configuration details of bonding here https://community.mellanox.com/s/article/howto-create-linux-bond--lag--interface-over-infiniband-network

Quote:

Since rdma_cm does not support MAC enslavement, fail_over_mac=1 (or active) should be added to the bond interface configuration in case of Ethernet interface.

I had tried before, it seems ok.

[bobzhuyb]

@yangwenhuan As I said, please use the rdma_cm option when running.ib_send_bw