GithubHelp home page GithubHelp logo

synsin0 / srcn3d Goto Github PK

View Code? Open in Web Editor NEW
55.0 3.0 9.0 127 KB

Official implementation of SRCN3D: Sparse R-CNN 3D Surround-View Cameras 3D Object Detection and Tracking for Autonomous Driving

License: MIT License

Python 99.47% Shell 0.53%

srcn3d's Issues

How long it takes to train the SRCN3D-VoV model?

Thanks for your great work!

Can you share how long it takes to train the SRCN3D-V2-99 model? I use 4 A100-SXM4-80GB and the log information shows it takes 16 days (as shown below). For more information, I set samples_per_gpu=1, workers_per_gpu=16.

I notice in the log file you provided, it takes 5 days to train the model on 3 GeForce RTX 3090 GPUS with samples_per_gpu=1, workers_per_gpu=4. I'm not sure what're the problems here.

------------------------------------------------------------
sys.platform: linux
Python: 3.7.16 (default, Jan 17 2023, 22:20:44) [GCC 11.2.0]
CUDA available: True
GPU 0,1,2,3: NVIDIA A100-SXM4-80GB
CUDA_HOME: /nvme/share/cuda-11.3/
NVCC: Build cuda_11.3.r11.3/compiler.29920130_0
GCC: gcc (GCC) 7.3.0
PyTorch: 1.10.2+cu111
PyTorch compiling details: PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.2.3 (Git Hash 7336ca9f055cf1bfa13efb658fe15dc9b41f0740)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX512
  - CUDA Runtime 11.1
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
  - CuDNN 8.0.5
  - Magma 2.5.2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.10.2, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, 

TorchVision: 0.11.3+cu111
OpenCV: 4.7.0
MMCV: 1.4.0
MMCV Compiler: GCC 7.3
MMCV CUDA Compiler: 11.3
MMDetection: 2.21.0
MMSegmentation: 0.21.1
MMDetection3D: 0.17.2+4385995
------------------------------------------------------------
2023-02-24 21:22:33,864 - mmdet - INFO - workflow: [('train', 1)], max: 24 epochs
2023-02-24 21:22:33,864 - mmdet - INFO - Checkpoints will be saved to /nvme/konglingdong/models/RoboDet/zoo/SRCN3D/work_dirs/srcn3d_v2-99_roi7_nusc_dd3d by HardDiskBackend.
2023-02-24 21:29:40,961 - mmdet - INFO - Epoch [1][50/7033]	lr: 7.973e-05, eta: 16 days, 16:17:18, time: 8.540, data_time: 0.233, memory: 22634, loss_cls: 1.3677, loss_bbox: 2.4627, d0.loss_cls: 1.3751, d0.loss_bbox: 2.1025, d1.loss_cls: 1.5197, d1.loss_bbox: 2.0489, d2.loss_cls: 1.4078, d2.loss_bbox: 2.0845, d3.loss_cls: 1.3781, d3.loss_bbox: 2.2114, d4.loss_cls: 1.3406, d4.loss_bbox: 2.3242, loss: 21.6232, grad_norm: 226.7225
2023-02-24 21:36:37,719 - mmdet - INFO - Epoch [1][100/7033]	lr: 9.307e-05, eta: 16 days, 11:22:22, time: 8.335, data_time: 0.022, memory: 22634, loss_cls: 1.2072, loss_bbox: 2.0843, d0.loss_cls: 1.1952, d0.loss_bbox: 1.8563, d1.loss_cls: 1.1928, d1.loss_bbox: 1.8148, d2.loss_cls: 1.1846, d2.loss_bbox: 1.7957, d3.loss_cls: 1.1924, d3.loss_bbox: 1.8352, d4.loss_cls: 1.1766, d4.loss_bbox: 1.9282, loss: 18.4634, grad_norm: 21.2113
2023-02-24 21:43:31,938 - mmdet - INFO - Epoch [1][150/7033]	lr: 1.064e-04, eta: 16 days, 8:52:09, time: 8.285, data_time: 0.023, memory: 22634, loss_cls: 1.0610, loss_bbox: 1.7132, d0.loss_cls: 1.0973, d0.loss_bbox: 1.7770, d1.loss_cls: 1.0334, d1.loss_bbox: 1.6853, d2.loss_cls: 1.0238, d2.loss_bbox: 1.6414, d3.loss_cls: 1.0449, d3.loss_bbox: 1.6432, d4.loss_cls: 1.0437, d4.loss_bbox: 1.6758, loss: 16.4400, grad_norm: 18.7006
2023-02-24 21:50:26,253 - mmdet - INFO - Epoch [1][200/7033]	lr: 1.197e-04, eta: 16 days, 7:34:27, time: 8.286, data_time: 0.023, memory: 22634, loss_cls: 0.9462, loss_bbox: 1.6390, d0.loss_cls: 0.9866, d0.loss_bbox: 1.7255, d1.loss_cls: 0.9085, d1.loss_bbox: 1.6240, d2.loss_cls: 0.9007, d2.loss_bbox: 1.5843, d3.loss_cls: 0.9236, d3.loss_bbox: 1.5828, d4.loss_cls: 0.9309, d4.loss_bbox: 1.6090, loss: 15.3611, grad_norm: 17.9067
2023-02-24 21:57:22,841 - mmdet - INFO - Epoch [1][250/7033]	lr: 1.331e-04, eta: 16 days, 7:10:50, time: 8.332, data_time: 0.023, memory: 22634, loss_cls: 0.8951, loss_bbox: 1.5424, d0.loss_cls: 0.9228, d0.loss_bbox: 1.6664, d1.loss_cls: 0.8340, d1.loss_bbox: 1.5674, d2.loss_cls: 0.8440, d2.loss_bbox: 1.5314, d3.loss_cls: 0.8612, d3.loss_bbox: 1.5194, d4.loss_cls: 0.8752, d4.loss_bbox: 1.5255, loss: 14.5849, grad_norm: 17.6487
2023-02-24 22:04:17,271 - mmdet - INFO - Epoch [1][300/7033]	lr: 1.464e-04, eta: 16 days, 6:32:34, time: 8.289, data_time: 0.024, memory: 22634, loss_cls: 0.8020, loss_bbox: 1.4868, d0.loss_cls: 0.8497, d0.loss_bbox: 1.6088, d1.loss_cls: 0.7373, d1.loss_bbox: 1.5021, d2.loss_cls: 0.7377, d2.loss_bbox: 1.4631, d3.loss_cls: 0.7551, d3.loss_bbox: 1.4590, d4.loss_cls: 0.7785, d4.loss_bbox: 1.4716, loss: 13.6517, grad_norm: 16.6638
2023-02-24 22:11:12,080 - mmdet - INFO - Epoch [1][350/7033]	lr: 1.597e-04, eta: 16 days, 6:06:18, time: 8.296, data_time: 0.027, memory: 22634, loss_cls: 0.8134, loss_bbox: 1.4780, d0.loss_cls: 0.8504, d0.loss_bbox: 1.5855, d1.loss_cls: 0.7426, d1.loss_bbox: 1.4718, d2.loss_cls: 0.7502, d2.loss_bbox: 1.4447, d3.loss_cls: 0.7661, d3.loss_bbox: 1.4480, d4.loss_cls: 0.7853, d4.loss_bbox: 1.4685, loss: 13.6046, grad_norm: 17.5405
2023-02-24 22:18:09,001 - mmdet - INFO - Epoch [1][400/7033]	lr: 1.731e-04, eta: 16 days, 5:59:40, time: 8.338, data_time: 0.027, memory: 22634, loss_cls: 0.7625, loss_bbox: 1.5228, d0.loss_cls: 0.8284, d0.loss_bbox: 1.6058, d1.loss_cls: 0.7117, d1.loss_bbox: 1.4891, d2.loss_cls: 0.7116, d2.loss_bbox: 1.4638, d3.loss_cls: 0.7260, d3.loss_bbox: 1.4648, d4.loss_cls: 0.7392, d4.loss_bbox: 1.4888, loss: 13.5144, grad_norm: 18.1703
2023-02-24 22:25:02,758 - mmdet - INFO - Epoch [1][450/7033]	lr: 1.864e-04, eta: 16 days, 5:33:15, time: 8.275, data_time: 0.022, memory: 22634, loss_cls: 0.7690, loss_bbox: 1.5465, d0.loss_cls: 0.8215, d0.loss_bbox: 1.6058, d1.loss_cls: 0.7022, d1.loss_bbox: 1.4927, d2.loss_cls: 0.7101, d2.loss_bbox: 1.4712, d3.loss_cls: 0.7257, d3.loss_bbox: 1.4843, d4.loss_cls: 0.7345, d4.loss_bbox: 1.5171, loss: 13.5806, grad_norm: 17.1787
2023-02-24 22:31:58,973 - mmdet - INFO - Epoch [1][500/7033]	lr: 1.997e-04, eta: 16 days, 5:24:32, time: 8.324, data_time: 0.022, memory: 22634, loss_cls: 0.7312, loss_bbox: 1.4822, d0.loss_cls: 0.7620, d0.loss_bbox: 1.5425, d1.loss_cls: 0.6664, d1.loss_bbox: 1.4275, d2.loss_cls: 0.6700, d2.loss_bbox: 1.4164, d3.loss_cls: 0.6780, d3.loss_bbox: 1.4368, d4.loss_cls: 0.7051, d4.loss_bbox: 1.4570, loss: 12.9751, grad_norm: 16.8601
2023-02-24 22:38:53,071 - mmdet - INFO - Epoch [1][550/7033]	lr: 2.000e-04, eta: 16 days, 5:05:20, time: 8.282, data_time: 0.022, memory: 22634, loss_cls: 0.7254, loss_bbox: 1.4267, d0.loss_cls: 0.7760, d0.loss_bbox: 1.5204, d1.loss_cls: 0.6622, d1.loss_bbox: 1.3974, d2.loss_cls: 0.6677, d2.loss_bbox: 1.3753, d3.loss_cls: 0.6751, d3.loss_bbox: 1.3855, d4.loss_cls: 0.6996, d4.loss_bbox: 1.4017, loss: 12.7130, grad_norm: 16.6110
2023-02-24 22:45:47,151 - mmdet - INFO - Epoch [1][600/7033]	lr: 2.000e-04, eta: 16 days, 4:48:00, time: 8.281, data_time: 0.023, memory: 22634, loss_cls: 0.6976, loss_bbox: 1.4603, d0.loss_cls: 0.7609, d0.loss_bbox: 1.5215, d1.loss_cls: 0.6539, d1.loss_bbox: 1.4148, d2.loss_cls: 0.6571, d2.loss_bbox: 1.3959, d3.loss_cls: 0.6719, d3.loss_bbox: 1.4021, d4.loss_cls: 0.6773, d4.loss_bbox: 1.4282, loss: 12.7416, grad_norm: 16.0229
2023-02-24 22:52:42,571 - mmdet - INFO - Epoch [1][650/7033]	lr: 2.000e-04, eta: 16 days, 4:38:15, time: 8.309, data_time: 0.023, memory: 22634, loss_cls: 0.6874, loss_bbox: 1.4336, d0.loss_cls: 0.7615, d0.loss_bbox: 1.5044, d1.loss_cls: 0.6509, d1.loss_bbox: 1.3947, d2.loss_cls: 0.6541, d2.loss_bbox: 1.3754, d3.loss_cls: 0.6663, d3.loss_bbox: 1.3873, d4.loss_cls: 0.6723, d4.loss_bbox: 1.4108, loss: 12.5987, grad_norm: 17.6562
2023-02-24 22:59:36,758 - mmdet - INFO - Epoch [1][700/7033]	lr: 2.000e-04, eta: 16 days, 4:23:53, time: 8.284, data_time: 0.023, memory: 22634, loss_cls: 0.6724, loss_bbox: 1.4318, d0.loss_cls: 0.7489, d0.loss_bbox: 1.5110, d1.loss_cls: 0.6346, d1.loss_bbox: 1.4080, d2.loss_cls: 0.6329, d2.loss_bbox: 1.3947, d3.loss_cls: 0.6455, d3.loss_bbox: 1.3976, d4.loss_cls: 0.6543, d4.loss_bbox: 1.4140, loss: 12.5458, grad_norm: 16.1715
2023-02-24 23:06:31,005 - mmdet - INFO - Epoch [1][750/7033]	lr: 2.000e-04, eta: 16 days, 4:10:43, time: 8.285, data_time: 0.023, memory: 22634, loss_cls: 0.6718, loss_bbox: 1.3878, d0.loss_cls: 0.7450, d0.loss_bbox: 1.4765, d1.loss_cls: 0.6367, d1.loss_bbox: 1.3629, d2.loss_cls: 0.6326, d2.loss_bbox: 1.3478, d3.loss_cls: 0.6425, d3.loss_bbox: 1.3570, d4.loss_cls: 0.6539, d4.loss_bbox: 1.3732, loss: 12.2879, grad_norm: 16.7400
2023-02-24 23:13:25,990 - mmdet - INFO - Epoch [1][800/7033]	lr: 2.000e-04, eta: 16 days, 4:00:56, time: 8.300, data_time: 0.023, memory: 22634, loss_cls: 0.6775, loss_bbox: 1.4184, d0.loss_cls: 0.7393, d0.loss_bbox: 1.4938, d1.loss_cls: 0.6299, d1.loss_bbox: 1.3779, d2.loss_cls: 0.6275, d2.loss_bbox: 1.3604, d3.loss_cls: 0.6465, d3.loss_bbox: 1.3746, d4.loss_cls: 0.6580, d4.loss_bbox: 1.3933, loss: 12.3970, grad_norm: 16.6003
2023-02-24 23:20:19,991 - mmdet - INFO - Epoch [1][850/7033]	lr: 2.000e-04, eta: 16 days, 3:48:14, time: 8.280, data_time: 0.023, memory: 22634, loss_cls: 0.6935, loss_bbox: 1.4177, d0.loss_cls: 0.7489, d0.loss_bbox: 1.4804, d1.loss_cls: 0.6554, d1.loss_bbox: 1.3682, d2.loss_cls: 0.6551, d2.loss_bbox: 1.3519, d3.loss_cls: 0.6592, d3.loss_bbox: 1.3650, d4.loss_cls: 0.6714, d4.loss_bbox: 1.3897, loss: 12.4564, grad_norm: 19.0236
2023-02-24 23:27:16,623 - mmdet - INFO - Epoch [1][900/7033]	lr: 2.000e-04, eta: 16 days, 3:44:23, time: 8.333, data_time: 0.023, memory: 22634, loss_cls: 0.6691, loss_bbox: 1.3087, d0.loss_cls: 0.7370, d0.loss_bbox: 1.4456, d1.loss_cls: 0.6414, d1.loss_bbox: 1.3175, d2.loss_cls: 0.6403, d2.loss_bbox: 1.2987, d3.loss_cls: 0.6456, d3.loss_bbox: 1.2988, d4.loss_cls: 0.6552, d4.loss_bbox: 1.2995, loss: 11.9574, grad_norm: 17.6854
2023-02-24 23:34:10,584 - mmdet - INFO - Epoch [1][950/7033]	lr: 2.000e-04, eta: 16 days, 3:32:20, time: 8.279, data_time: 0.023, memory: 22634, loss_cls: 0.6252, loss_bbox: 1.3668, d0.loss_cls: 0.7023, d0.loss_bbox: 1.4554, d1.loss_cls: 0.5966, d1.loss_bbox: 1.3405, d2.loss_cls: 0.5911, d2.loss_bbox: 1.3265, d3.loss_cls: 0.6045, d3.loss_bbox: 1.3349, d4.loss_cls: 0.6132, d4.loss_bbox: 1.3494, loss: 11.9064, grad_norm: 15.6669
2023-02-24 23:41:04,520 - mmdet - INFO - Exp name: srcn3d_v2-99_roi7_nusc_dd3d.py

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.