Faild to train useing GPU with M3gnet model about matgl

[shyuep]

Your torch default device needs to be the very first call. Not after all the other model setup has occurred. Or you can use with torch.device("cuda") before all your code.

[shyuep]

I should add that you should post such questions on the Discussions section, not in as an Issue in GitHub. Issues are meant for actual bug reports.

[SmallBearC]

Thank you very very much for your guidance I have tried as follows: from __future__ import annotations import torch torch.set_default_device('cuda') #torch.device('cuda') import os import shutil import numpy as np import pytorch_lightning as pl from dgl.data.utils import split_dataset from pymatgen.core import Structure from matgl.ext.pymatgen import Structure2Graph, get_element_list from matgl.graph.data import M3GNetDataset, MEGNetDataset, MGLDataLoader, collate_fn, collate_fn_efs from matgl.models import M3GNet, MEGNet from matgl.utils.training import ModelLightningModule, PotentialLightningModule #mp.set_start_method('spawn') #torch.set_default_device('cpu') if __name__ == '__main__': #&nbsp; &nbsp; torch.device('cuda') &nbsp; &nbsp; stru0 = Structure.from_file("../../strus/0/POSCAR") &nbsp; &nbsp; stru1 = Structure.from_file("../../strus/1/POSCAR") &nbsp; &nbsp; ...... Or from __future__ import annotations import torch #torch.set_default_device('cuda') #torch.device('cuda') import os import shutil import numpy as np import pytorch_lightning as pl from dgl.data.utils import split_dataset from pymatgen.core import Structure from matgl.ext.pymatgen import Structure2Graph, get_element_list from matgl.graph.data import M3GNetDataset, MEGNetDataset, MGLDataLoader, collate_fn, collate_fn_efs from matgl.models import M3GNet, MEGNet from matgl.utils.training import ModelLightningModule, PotentialLightningModule #mp.set_start_method('spawn') #torch.set_default_device('cpu') if __name__ == '__main__': &nbsp; &nbsp; torch.set_default_device('cuda') &nbsp; &nbsp; torch.device('cuda') &nbsp; &nbsp; stru0 = Structure.from_file("../../strus/0/POSCAR") &nbsp; &nbsp; stru1 = Structure.from_file("../../strus/1/POSCAR") &nbsp; &nbsp; ...... But it all had the same error: Original Traceback (most recent call last): &nbsp; File "/home/lycui/anaconda3/envs/mgl/lib/python3.9/site-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop &nbsp; &nbsp; data = fetcher.fetch(index) &nbsp; File "/home/lycui/anaconda3/envs/mgl/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch &nbsp; &nbsp; data = [self.dataset[idx] for idx in possibly_batched_index] &nbsp; File "/home/lycui/anaconda3/envs/mgl/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 51, in <listcomp&gt; &nbsp; &nbsp; data = [self.dataset[idx] for idx in possibly_batched_index] &nbsp; File "/home/lycui/anaconda3/envs/mgl/lib/python3.9/site-packages/dgl/data/utils.py", line 425, in __getitem__ &nbsp; &nbsp; return self.dataset[self.indices[item]] &nbsp; File "/home/lycui/anaconda3/envs/mgl/lib/python3.9/site-packages/matgl/graph/data.py", line 321, in __getitem__ &nbsp; &nbsp; self.state_attr[idx], RuntimeError: CUDA error: initialization error CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. When I use the torch.device('cuda') as follows: from __future__ import annotations import torch torch.device('cuda') #torch.device('cuda') import os import shutil import numpy as np import pytorch_lightning as pl from dgl.data.utils import split_dataset from pymatgen.core import Structure from matgl.ext.pymatgen import Structure2Graph, get_element_list from matgl.graph.data import M3GNetDataset, MEGNetDataset, MGLDataLoader, collate_fn, collate_fn_efs from matgl.models import M3GNet, MEGNet from matgl.utils.training import ModelLightningModule, PotentialLightningModule #mp.set_start_method('spawn') #torch.set_default_device('cpu') if __name__ == '__main__': #&nbsp; &nbsp; torch.device('cuda') &nbsp; &nbsp; stru0 = Structure.from_file("../../strus/0/POSCAR") &nbsp; &nbsp; stru1 = Structure.from_file("../../strus/1/POSCAR") &nbsp; &nbsp; ...... Or from __future__ import annotations import torch #torch.set_default_device('cuda') #torch.device('cuda') import os import shutil import numpy as np import pytorch_lightning as pl from dgl.data.utils import split_dataset from pymatgen.core import Structure from matgl.ext.pymatgen import Structure2Graph, get_element_list from matgl.graph.data import M3GNetDataset, MEGNetDataset, MGLDataLoader, collate_fn, collate_fn_efs from matgl.models import M3GNet, MEGNet from matgl.utils.training import ModelLightningModule, PotentialLightningModule #mp.set_start_method('spawn') #torch.set_default_device('cpu') if __name__ == '__main__': &nbsp; &nbsp;&nbsp;torch.device('cuda') &nbsp; &nbsp; torch.device('cuda') &nbsp; &nbsp; stru0 = Structure.from_file("../../strus/0/POSCAR") &nbsp; &nbsp; stru1 = Structure.from_file("../../strus/1/POSCAR") &nbsp; &nbsp; ...... I get this error: File "/home/lycui/anaconda3/envs/mgl/lib/python3.9/site-packages/matgl/models/_m3gnet.py", line 224, in forward &nbsp; &nbsp; g.edata["bond_vec"] = bond_vec File "/home/lycui/anaconda3/envs/mgl/lib/python3.9/site-packages/dgl/view.py", line 227, in __setitem__ &nbsp; &nbsp; self._graph._set_e_repr(self._etid, self._edges, {key: val}) File "/home/lycui/anaconda3/envs/mgl/lib/python3.9/site-packages/dgl/heterograph.py", line 4460, in _set_e_repr &nbsp; &nbsp; raise DGLError( dgl._ffi.base.DGLError: Cannot assign edge feature "bond_vec" on device cpu to a graph on device cuda:0. Call DGLGraph.to() to copy the graph to the same device. So I'm confused, I don't know what to do. Here are some packages version: dgl&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;1.1.0+cu117 cryptography&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 41.0.1 matgl&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.6.2 pytorch-lightning&nbsp; &nbsp; 2.0.4 torch&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;2.0.1 tqdm&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;4.65.0 pymatgen&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 2023.5.31 ------------------ 原始邮件 ------------------ 发件人: "materialsvirtuallab/matgl" ***@***.***&gt;; 发送时间:&nbsp;2023年6月25日(星期天) 上午8:32 ***@***.***&gt;; ***@***.******@***.***&gt;; 主题:&nbsp;Re: [materialsvirtuallab/matgl] Faild to train useing GPU with M3gnet model (Issue #94) Your torch default device needs to be the very first call. Not after all the other model setup has occurred. Or you can use with torch.device("cuda") before all your code. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***&gt;

[chiku-parida]

Are you able to solve this problem? Please let me know.
I have tried like below declaring CUDA in the beginning still the error persists...

'''
if torch.cuda.is_available():
device = 'cuda'
else:
device = 'cpu'

print(f'The available device is {device}')
'''

[txy159]

Hi, @SmallBearC @chiku-parida, Are you able to solve this problem ? I also get the same errors.