Comments (11)
Use the following command to train a model with GPU:1
bash ./scripts/train.sh 1 --config <config>
from pointr.
@yuxumin OK. Thanks. But when I change into GPU:1, I still met the OOM when running the command: bash ./scripts/train.sh 2 --config ./cfgs/PCN_models/GRNet.yaml --exp_name example
init.py", line 22, in forward
min_x, max_x, min_y, max_y, min_z, max_z, gt_cloud)
RuntimeError: CUDA out of memory. Tried to allocate 512.00 MiB (GPU 0; 31.75 GiB total capacity; 29.93 GiB already allocated; 451.50 MiB free; 29.99 GiB reserved in total by PyTorch)
It seems that the code still run on GPU: 0 ? I'm not sure.
from pointr.
It seems that the code still run on GPU: 0 ? I'm not sure.
The code in train.sh is CUDA_VISIBLE_DEVICES=${GPUS} python main.py ${PY_ARGS}
. I am sure that the code run on GPU:1
bash ./scripts/train.sh 2 --config ./cfgs/PCN_models/GRNet.yaml --exp_name example
And this will put the model and data on GPU:2
from pointr.
@yuxumin OK. I see. I try to run other GPU Cluster nodes, but I met an error like this:
anaconda3/envs/pytorch-PoinTr/lib/python3.7/site-packages/knn_cuda/init.py", line 15, in load_cpp_ext
assert torch.cuda.is_available(), "torch.cuda.is_available() is False."
AssertionError: torch.cuda.is_available() is False.
I use cuda11.1 and pytorch==1.8.0, it seems that the GPU is not available now.
from pointr.
@yuxumin Which file to change batchsize of GRNet.yaml and TopNet.yaml?
from pointr.
bash ./scripts/train.sh 2 --config ./cfgs/PCN_models/GRNet.yaml --exp_name example
This code means to use the config file in cfgs/PCN_models/GRNet.yaml
from pointr.
@yuxumin Is this: total_bs : 32 in cfgs/PCN_models/GRNet.yaml to change the batch size, right?
from pointr.
yes
from pointr.
@yuxumin I have change it into 2, the OOM still occurs.
from pointr.
GRNet requires to calculate gridding loss during the training, which takes a lot of memory of GPU. Can you try to train other models, rather than GRNet
from pointr.
Close it since no response. Feel free to re-open it if problems still exist
from pointr.
Related Issues (20)
- 大神您好,你的研究工作对我启发很大,我的自己的数据归一化到[-0.5,0.5],每一个模型10000个点,按照PCN数据集的格式放入PoinTr网络进行训练,遇到上面的问题,我感到非常的困惑,可以帮我看看这是什么原因么? HOT 3
- The pretrained model of ShapeNet55 HOT 2
- 你好,请问有没有提供利用ckpt-last.pth进行预测的代码呢 HOT 2
- memory leak in the training code HOT 1
- 你好,demo下的目录结构应该是如何? HOT 2
- 定义PoinTr模型似乎只能在cuda环境
- Can you provide the Projected_ShapeNet-55_noise dataset? HOT 3
- If my use case is to generate only one object and I have about 1500 data. Which model is best for this? AdaPointr or Pointr???
- 分布式多gpu训练报错
- 训练效果不好 HOT 1
- How do I run this network to solve AttributeError: 'NoneType' object has no attribute 'items'
- Question about the unused parameters in Adapointr HOT 2
- normalization at inference time vs training time HOT 2
- How to use pre-trained models? HOT 4
- Pretrained AdaPoinTr models HOT 3
- The question about the code of GRNet in this repository. HOT 2
- No module named 'PoinTr' found HOT 1
- 大佬你好,注意到你的shapenet2pc只有采样到8192分辨率,那残缺点云是如何制作的呢
- Training error: ModuleNotFoundError: No module named 'pointnet2_ops' HOT 2
- Question about the query ranking.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pointr.