Training speed is too slow about giga

[Steve-Tod]

I guess you are right, the speed of io is too slow. One thing you can try to verify this guess is to iterate the dataloader without training and check the speed.

[huaji98]

yes,I found that loading the data took about the same amount of time as before with training.

[Steve-Tod]

Do you get a similar speed when training giga-aff? If not, then the IO bottleneck is the occupancy data loading. The default size of each occupancy data file is 100K points, but we only sample 2048 points. So one thing you can try is to decrease the size and increase the number of occupancy files when generating data:

python scripts/save_occ_data_parallel.py /path/to/raw/data {number_of_points_per_file} {number_of_files} --num-proc 40

[huaji98]

Do you get a similar speed when training giga-aff? If not, then the IO bottleneck is the occupancy data loading. The default size of each occupancy data file is 100K points, but we only sample 2048 points. So one thing you can try is to decrease the size and increase the number of occupancy files when generating data:

python scripts/save_occ_data_parallel.py /path/to/raw/data {number_of_points_per_file} {number_of_files} --num-proc 40

Thank you for your reply. I found a more powerful computer, and now the training speed is normal！