Comments (4)
this is the log
Using backend: pytorch
2021-12-26 07:39:05 | INFO | fairseq.distributed.utils | distributed init (rank 2): tcp://localhost:18027
2021-12-26 07:39:06 | INFO | fairseq.distributed.utils | distributed init (rank 0): tcp://localhost:18027
2021-12-26 07:39:06 | INFO | fairseq.distributed.utils | distributed init (rank 6): tcp://localhost:18027
2021-12-26 07:39:06 | INFO | torch.distributed.distributed_c10d | Added key: store_based_barrier_key:1 to store for rank: 6
2021-12-26 07:39:06 | INFO | fairseq.distributed.utils | distributed init (rank 3): tcp://localhost:18027
2021-12-26 07:39:06 | INFO | torch.distributed.distributed_c10d | Added key: store_based_barrier_key:1 to store for rank: 3
2021-12-26 07:39:06 | INFO | fairseq.distributed.utils | distributed init (rank 1): tcp://localhost:18027
2021-12-26 07:39:06 | INFO | torch.distributed.distributed_c10d | Added key: store_based_barrier_key:1 to store for rank: 1
2021-12-26 07:39:06 | INFO | fairseq.distributed.utils | distributed init (rank 4): tcp://localhost:18027
2021-12-26 07:39:06 | INFO | torch.distributed.distributed_c10d | Added key: store_based_barrier_key:1 to store for rank: 4
2021-12-26 07:39:06 | INFO | fairseq.distributed.utils | distributed init (rank 7): tcp://localhost:18027
2021-12-26 07:39:06 | INFO | torch.distributed.distributed_c10d | Added key: store_based_barrier_key:1 to store for rank: 7
2021-12-26 07:39:06 | INFO | fairseq.distributed.utils | distributed init (rank 5): tcp://localhost:18027
2021-12-26 07:39:06 | INFO | torch.distributed.distributed_c10d | Added key: store_based_barrier_key:1 to store for rank: 5
2021-12-26 07:39:06 | INFO | torch.distributed.distributed_c10d | Added key: store_based_barrier_key:1 to store for rank: 2
2021-12-26 07:39:06 | INFO | torch.distributed.distributed_c10d | Added key: store_based_barrier_key:1 to store for rank: 0
2021-12-26 07:39:06 | INFO | torch.distributed.distributed_c10d | Rank 0: Completed store-based barrier for 8 nodes.
2021-12-26 07:39:06 | INFO | fairseq.distributed.utils | initialized host ubuntu as rank 0
2021-12-26 07:39:06 | INFO | torch.distributed.distributed_c10d | Rank 4: Completed store-based barrier for 8 nodes.
2021-12-26 07:39:06 | INFO | fairseq.distributed.utils | initialized host ubuntu as rank 4
2021-12-26 07:39:06 | INFO | torch.distributed.distributed_c10d | Rank 5: Completed store-based barrier for 8 nodes.
2021-12-26 07:39:06 | INFO | torch.distributed.distributed_c10d | Rank 2: Completed store-based barrier for 8 nodes.
2021-12-26 07:39:06 | INFO | fairseq.distributed.utils | initialized host ubuntu as rank 5
2021-12-26 07:39:06 | INFO | fairseq.distributed.utils | initialized host ubuntu as rank 2
2021-12-26 07:39:06 | INFO | torch.distributed.distributed_c10d | Rank 6: Completed store-based barrier for 8 nodes.
2021-12-26 07:39:06 | INFO | fairseq.distributed.utils | initialized host ubuntu as rank 6
2021-12-26 07:39:06 | INFO | torch.distributed.distributed_c10d | Rank 7: Completed store-based barrier for 8 nodes.
2021-12-26 07:39:06 | INFO | torch.distributed.distributed_c10d | Rank 1: Completed store-based barrier for 8 nodes.
2021-12-26 07:39:06 | INFO | torch.distributed.distributed_c10d | Rank 3: Completed store-based barrier for 8 nodes.
2021-12-26 07:39:06 | INFO | fairseq.distributed.utils | initialized host ubuntu as rank 7
2021-12-26 07:39:06 | INFO | fairseq.distributed.utils | initialized host ubuntu as rank 3
2021-12-26 07:39:06 | INFO | fairseq.distributed.utils | initialized host ubuntu as rank 1
Traceback (most recent call last):
File "/home/huazixu/anaconda3/envs/mol/bin/fairseq-train", line 8, in
sys.exit(cli_main())
File "/home/huazixu/anaconda3/envs/mol/lib/python3.9/site-packages/fairseq_cli/train.py", line 528, in cli_main
distributed_utils.call_main(cfg, main)
File "/home/huazixu/anaconda3/envs/mol/lib/python3.9/site-packages/fairseq/distributed/utils.py", line 344, in call_main
torch.multiprocessing.spawn(
File "/home/huazixu/anaconda3/envs/mol/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
File "/home/huazixu/anaconda3/envs/mol/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 188, in start_processes
while not context.join():
File "/home/huazixu/anaconda3/envs/mol/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 150, in join
raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:
-- Process 4 terminated with the following error:
Traceback (most recent call last):
File "/home/huazixu/anaconda3/envs/mol/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap
fn(i, *args)
File "/home/huazixu/anaconda3/envs/mol/lib/python3.9/site-packages/fairseq/distributed/utils.py", line 322, in distributed_main
cfg.distributed_training.distributed_rank = distributed_init(cfg)
File "/home/huazixu/anaconda3/envs/mol/lib/python3.9/site-packages/fairseq/distributed/utils.py", line 272, in distributed_init
dist.all_reduce(torch.zeros(1).cuda())
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
from graphormer.
it‘s not useful to decrease the batchsize
from graphormer.
it‘s not useful to decrease the batchsize
Hello, have you solved this problem?
from graphormer.
it‘s not useful to decrease the batchsize
Hello, have you solved this problem?
yes it‘s just my devices’ problem
from graphormer.
Related Issues (20)
- Implementation of Graphormer based on pytorch geometric HOT 5
- Pretrained OCP20 Graphormer3D model HOT 1
- Feature Request: New Model
- python setup.py build_ext --inplace bug
- install.sh Issue, no access to sudo, using pip21.1 HOT 1
- OC20 checkpoint
- Own dataset problem while dataset.map()
- Unable to download pretrain model.HTTP error 409 HOT 2
- No support for custom datasets in ogb format? HOT 1
- Binary pretrained model can't train a multi-class classifier?
- no setup.py and --user-feature param HOT 1
- How to create a submission file for the graphormer model trained on the OC20 task
- Working Dependencies 2024? HOT 1
- Custom dataset
- Can this model be used in the field of traffic prediction?
- Lack of diversity in the 1ake example prediction by DiG HOT 8
- Feature representations for new Proteins in DiG HOT 11
- Missing file train_cli.sh HOT 2
- json.decoder.JSONDecodeError ??? HOT 1
- Missing training scripts for proteins and protein-ligands HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from graphormer.