RuntimeError: CUDA error: out of memory about graphormer

[huahuaai]

this is the log
Using backend: pytorch
2021-12-26 07:39:05 | INFO | fairseq.distributed.utils | distributed init (rank 2): tcp://localhost:18027
2021-12-26 07:39:06 | INFO | fairseq.distributed.utils | distributed init (rank 0): tcp://localhost:18027
2021-12-26 07:39:06 | INFO | fairseq.distributed.utils | distributed init (rank 6): tcp://localhost:18027
2021-12-26 07:39:06 | INFO | torch.distributed.distributed_c10d | Added key: store_based_barrier_key:1 to store for rank: 6
2021-12-26 07:39:06 | INFO | fairseq.distributed.utils | distributed init (rank 3): tcp://localhost:18027
2021-12-26 07:39:06 | INFO | torch.distributed.distributed_c10d | Added key: store_based_barrier_key:1 to store for rank: 3
2021-12-26 07:39:06 | INFO | fairseq.distributed.utils | distributed init (rank 1): tcp://localhost:18027
2021-12-26 07:39:06 | INFO | torch.distributed.distributed_c10d | Added key: store_based_barrier_key:1 to store for rank: 1
2021-12-26 07:39:06 | INFO | fairseq.distributed.utils | distributed init (rank 4): tcp://localhost:18027
2021-12-26 07:39:06 | INFO | torch.distributed.distributed_c10d | Added key: store_based_barrier_key:1 to store for rank: 4
2021-12-26 07:39:06 | INFO | fairseq.distributed.utils | distributed init (rank 7): tcp://localhost:18027
2021-12-26 07:39:06 | INFO | torch.distributed.distributed_c10d | Added key: store_based_barrier_key:1 to store for rank: 7
2021-12-26 07:39:06 | INFO | fairseq.distributed.utils | distributed init (rank 5): tcp://localhost:18027
2021-12-26 07:39:06 | INFO | torch.distributed.distributed_c10d | Added key: store_based_barrier_key:1 to store for rank: 5
2021-12-26 07:39:06 | INFO | torch.distributed.distributed_c10d | Added key: store_based_barrier_key:1 to store for rank: 2
2021-12-26 07:39:06 | INFO | torch.distributed.distributed_c10d | Added key: store_based_barrier_key:1 to store for rank: 0
2021-12-26 07:39:06 | INFO | torch.distributed.distributed_c10d | Rank 0: Completed store-based barrier for 8 nodes.
2021-12-26 07:39:06 | INFO | fairseq.distributed.utils | initialized host ubuntu as rank 0
2021-12-26 07:39:06 | INFO | torch.distributed.distributed_c10d | Rank 4: Completed store-based barrier for 8 nodes.
2021-12-26 07:39:06 | INFO | fairseq.distributed.utils | initialized host ubuntu as rank 4
2021-12-26 07:39:06 | INFO | torch.distributed.distributed_c10d | Rank 5: Completed store-based barrier for 8 nodes.
2021-12-26 07:39:06 | INFO | torch.distributed.distributed_c10d | Rank 2: Completed store-based barrier for 8 nodes.
2021-12-26 07:39:06 | INFO | fairseq.distributed.utils | initialized host ubuntu as rank 5
2021-12-26 07:39:06 | INFO | fairseq.distributed.utils | initialized host ubuntu as rank 2
2021-12-26 07:39:06 | INFO | torch.distributed.distributed_c10d | Rank 6: Completed store-based barrier for 8 nodes.
2021-12-26 07:39:06 | INFO | fairseq.distributed.utils | initialized host ubuntu as rank 6
2021-12-26 07:39:06 | INFO | torch.distributed.distributed_c10d | Rank 7: Completed store-based barrier for 8 nodes.
2021-12-26 07:39:06 | INFO | torch.distributed.distributed_c10d | Rank 1: Completed store-based barrier for 8 nodes.
2021-12-26 07:39:06 | INFO | torch.distributed.distributed_c10d | Rank 3: Completed store-based barrier for 8 nodes.
2021-12-26 07:39:06 | INFO | fairseq.distributed.utils | initialized host ubuntu as rank 7
2021-12-26 07:39:06 | INFO | fairseq.distributed.utils | initialized host ubuntu as rank 3
2021-12-26 07:39:06 | INFO | fairseq.distributed.utils | initialized host ubuntu as rank 1
Traceback (most recent call last):
File "/home/huazixu/anaconda3/envs/mol/bin/fairseq-train", line 8, in
sys.exit(cli_main())
File "/home/huazixu/anaconda3/envs/mol/lib/python3.9/site-packages/fairseq_cli/train.py", line 528, in cli_main
distributed_utils.call_main(cfg, main)
File "/home/huazixu/anaconda3/envs/mol/lib/python3.9/site-packages/fairseq/distributed/utils.py", line 344, in call_main
torch.multiprocessing.spawn(
File "/home/huazixu/anaconda3/envs/mol/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
File "/home/huazixu/anaconda3/envs/mol/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 188, in start_processes
while not context.join():
File "/home/huazixu/anaconda3/envs/mol/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 150, in join
raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 4 terminated with the following error:
Traceback (most recent call last):
File "/home/huazixu/anaconda3/envs/mol/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap
fn(i, *args)
File "/home/huazixu/anaconda3/envs/mol/lib/python3.9/site-packages/fairseq/distributed/utils.py", line 322, in distributed_main
cfg.distributed_training.distributed_rank = distributed_init(cfg)
File "/home/huazixu/anaconda3/envs/mol/lib/python3.9/site-packages/fairseq/distributed/utils.py", line 272, in distributed_init
dist.all_reduce(torch.zeros(1).cuda())
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

[huahuaai]

it‘s not useful to decrease the batchsize

[dongZheX]

it‘s not useful to decrease the batchsize

Hello, have you solved this problem?

[huahuaai]

it‘s not useful to decrease the batchsize

Hello, have you solved this problem?
yes it‘s just my devices’ problem