nadeemlab / cir Goto Github PK

I started to train the Mesh Only model from scratch. The model is often not updated for several hours during training.Have you ever encountered such a problem? After training 200 epochs from the beginning, the effect will be worse than that mentioned in the paper (I guess it is because of the training equipment, for example, I only used a 3090)
In addition,you mentioned ran multiple training with different parameters on a GPU before. What parameters have you modified

I used 3090 (24GB) in training. I found that the video memory only used 7GB, and the CPU usage was very low. It took me about 10 minutes to train an epoch. I read your paper and found that you used four A6000. How do you improve the video memory and CPU usage to improve the training speed? In addition, I would like to ask how to load the previously trained model to continue training

Hello All,

I encountered the following problems when running test.py. Have you ever encountered this situation.
Environment：
PyTorch 1.6.0
Python 3.8(ubuntu18.04)
Cuda 10.1（RTX2080Ti）

problems :
Initialize optimizer
Load pre-processed data
LIDC training dataset
LIDC validation dataset
LIDC testing dataset
LUNGx training dataset
LUNGx testing dataset
Initialize evaluator

LIDC_123vs45

training
Evaluation: 0%| | 0/648 [00:00<?, ?sample/s]

validation
Evaluation: 0%| | 0/163 [00:00<?, ?sample/s]

testing
Evaluation: 0%| | 0/72 [00:00<?, ?sample/s]

LUNGx

training
Evaluation: 0%| | 0/10 [00:00<?, ?sample/s]

validation

testing
Evaluation: 0%| | 0/73 [00:00<?, ?sample/s]
wandb: Waiting for W&B process to finish... (success).
wandb:
wandb: Synced Experiment_1/trial_1/epoch_180: https://wandb.ai/kingjames12138/CIR_test/runs/31ko97oh
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
wandb: Find logs at: ./experiments/MICCAI2022/Experiment_001/trial_1/wandb/run-20220922_145012-31ko97oh/logs