[Bug]: NCCL hangs and causes timeout about vllm

[youkaichao]

FYI i recently added a page of debugging tips for this problem: https://docs.vllm.ai/en/latest/getting_started/debugging.html

[wjj19950828]

In addition to these two switches --disable-custom-all-reduce and --enforce-eager, is there any other solution to solve this problem? Thanks~

[wjj19950828]

FYI i recently added a page of debugging tips for this problem: https://docs.vllm.ai/en/latest/getting_started/debugging.html

Thanks for your reply~

I tried the test code you provided below, and I haven't encountered any related problems so far. This problem does occur accidentally. Do you have any suggestions?

# save it as `test.py` , and run it with `NCCL_DEBUG=TRACE torchrun --nproc-per-node=8 test.py`
# adjust `--nproc-per-node` to the number of GPUs you want to use.
import torch
import torch.distributed as dist
dist.init_process_group(backend="nccl")
data = torch.FloatTensor([1,] * 128).to(f"cuda:{dist.get_rank()}")
dist.all_reduce(data, op=dist.ReduceOp.SUM)
torch.cuda.synchronize()
value = data.mean().item()
assert value == dist.get_world_size()