I am following the instruction on ReadMe to run training part. When I execute "python tools/plain_train_net.py --config-file "configs/smoke_gn_vector.yaml", I get an error "RuntimeError: received 0 items of ancdata"
Command Line Args: Namespace(ckpt=None, config_file='configs/smoke_gn_vector.yaml', dist_url='tcp://127.0.0.1:50153', eval_only=False, machine_rank=0, num_gpus=1, num_machines=1, opts=[])
[2020-06-19 18:10:56,601] smoke INFO: Using 1 GPUs
[2020-06-19 18:10:56,601] smoke INFO: Collecting environment info
[2020-06-19 18:10:57,952] smoke INFO:
PyTorch version: 1.4.0
Is debug build: No
CUDA used to build PyTorch: 10.0
OS: Ubuntu 16.04.6 LTS
GCC version: (Ubuntu 5.5.0-12ubuntu1~16.04) 5.5.0 20171010
CMake version: version 3.5.1
Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 10.0.130
GPU models and configuration:
GPU 0: GeForce GTX 1080 Ti
GPU 1: GeForce GTX 1080 Ti
GPU 2: GeForce GTX 1080 Ti
GPU 3: TITAN Xp
Nvidia driver version: 418.87.01
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.2
Versions of relevant libraries:
[pip3] numpy==1.18.1
[pip3] torch==1.4.0
[pip3] torchfile==0.1.0
[pip3] torchvision==0.5.0
[conda] blas 1.0 mkl
[conda] mkl 2020.1 217
[conda] mkl-service 2.3.0 py37he904b0f_0
[conda] mkl_fft 1.1.0 py37h23d657b_0
[conda] mkl_random 1.1.1 py37h0573a6f_0
[conda] pytorch 1.4.0 py3.7_cuda10.0.130_cudnn7.6.3_0 pytorch
[conda] torchvision 0.5.0 py37_cu100 pytorch
Pillow (7.1.2)
[2020-06-19 18:10:57,952] smoke INFO: Namespace(ckpt=None, config_file='configs/smoke_gn_vector.yaml', dist_url='tcp://127.0.0.1:50153', eval_only=False, machine_rank=0, num_gpus=1, num_machines=1, opts=[])
[2020-06-19 18:10:57,952] smoke INFO: Loaded configuration file configs/smoke_gn_vector.yaml
[2020-06-19 18:10:57,953] smoke INFO:
MODEL:
WEIGHT: "catalog://ImageNetPretrained/DLA34"
INPUT:
FLIP_PROB_TRAIN: 0.5
SHIFT_SCALE_PROB_TRAIN: 0.3
DATASETS:
DETECT_CLASSES: ("Car", "Cyclist", "Pedestrian")
TRAIN: ("kitti_train",)
TEST: ("kitti_test",)
TRAIN_SPLIT: "trainval"
TEST_SPLIT: "test"
SOLVER:
BASE_LR: 2.5e-4
STEPS: (10000, 18000)
MAX_ITERATION: 25000
IMS_PER_BATCH: 32
[2020-06-19 18:10:57,953] smoke INFO: Running with config:
CUDNN_BENCHMARK: True
DATALOADER:
ASPECT_RATIO_GROUPING: False
NUM_WORKERS: 4
SIZE_DIVISIBILITY: 0
DATASETS:
DETECT_CLASSES: ('Car', 'Cyclist', 'Pedestrian')
MAX_OBJECTS: 30
TEST: ('kitti_test',)
TEST_SPLIT: test
TRAIN: ('kitti_train',)
TRAIN_SPLIT: trainval
INPUT:
FLIP_PROB_TRAIN: 0.5
HEIGHT_TEST: 384
HEIGHT_TRAIN: 384
PIXEL_MEAN: [0.485, 0.456, 0.406]
PIXEL_STD: [0.229, 0.224, 0.225]
SHIFT_SCALE_PROB_TRAIN: 0.3
SHIFT_SCALE_TRAIN: (0.2, 0.4)
TO_BGR: True
WIDTH_TEST: 1280
WIDTH_TRAIN: 1280
MODEL:
BACKBONE:
BACKBONE_OUT_CHANNELS: 64
CONV_BODY: DLA-34-DCN
DOWN_RATIO: 4
FREEZE_CONV_BODY_AT: 0
USE_NORMALIZATION: GN
DEVICE: cuda
GROUP_NORM:
DIM_PER_GP: -1
EPSILON: 1e-05
NUM_GROUPS: 32
SMOKE_HEAD:
DEPTH_REFERENCE: (28.01, 16.32)
DIMENSION_REFERENCE: ((3.88, 1.63, 1.53), (1.78, 1.7, 0.58), (0.88, 1.73, 0.67))
LOSS_ALPHA: 2
LOSS_BETA: 4
LOSS_TYPE: ('FocalLoss', 'DisL1')
LOSS_WEIGHT: (1.0, 10.0)
NUM_CHANNEL: 256
PREDICTOR: SMOKEPredictor
REGRESSION_CHANNEL: (1, 2, 3, 2)
REGRESSION_HEADS: 8
USE_NMS: False
USE_NORMALIZATION: GN
SMOKE_ON: True
WEIGHT: catalog://ImageNetPretrained/DLA34
OUTPUT_DIR: ./tools/logs
PATHS_CATALOG: /home/robot1/Shukai/SMOKE/smoke/config/paths_catalog.py
SEED: -1
SOLVER:
BASE_LR: 0.00025
BIAS_LR_FACTOR: 2
CHECKPOINT_PERIOD: 20
EVALUATE_PERIOD: 20
IMS_PER_BATCH: 32
LOAD_OPTIMIZER_SCHEDULER: True
MASTER_BATCH: -1
MAX_ITERATION: 25000
OPTIMIZER: Adam
STEPS: (10000, 18000)
TEST:
DETECTIONS_PER_IMG: 50
DETECTIONS_THRESHOLD: 0.25
IMS_PER_BATCH: 1
PRED_2D: True
SINGLE_GPU_TEST: True
[2020-06-19 18:10:57,954] smoke.utils.envs INFO: Using a generated random seed 58027627
[2020-06-19 18:11:00,129] smoke.utils.check_point INFO: Loading checkpoint from catalog://ImageNetPretrained/DLA34
[2020-06-19 18:11:00,129] smoke.utils.check_point INFO: catalog://ImageNetPretrained/DLA34 points to http://dl.yf.io/dla/models/imagenet/dla34-ba72cf86.pth
[2020-06-19 18:11:00,130] smoke.utils.check_point INFO: url http://dl.yf.io/dla/models/imagenet/dla34-ba72cf86.pth cached in /home/robot1/.torch/models/dla34-ba72cf86.pth
[2020-06-19 18:11:00,163] smoke.utils.model_serialization INFO: backbone.body.base.base_layer.0.weight loaded from base_layer.0.weight of shape (16, 3, 7, 7)
[2020-06-19 18:11:00,164] smoke.utils.model_serialization INFO: backbone.body.base.base_layer.1.bias loaded from base_layer.1.bias of shape (16,)
[2020-06-19 18:11:00,164] smoke.utils.model_serialization INFO: backbone.body.base.base_layer.1.weight loaded from base_layer.1.weight of shape (16,)
[2020-06-19 18:11:00,164] smoke.utils.model_serialization INFO: backbone.body.base.level0.0.weight loaded from level0.0.weight of shape (16, 16, 3, 3)
[2020-06-19 18:11:00,164] smoke.utils.model_serialization INFO: backbone.body.base.level0.1.bias loaded from level0.1.bias of shape (16,)
[2020-06-19 18:11:00,164] smoke.utils.model_serialization INFO: backbone.body.base.level0.1.weight loaded from level0.1.weight of shape (16,)
[2020-06-19 18:11:00,164] smoke.utils.model_serialization INFO: backbone.body.base.level1.0.weight loaded from level1.0.weight of shape (32, 16, 3, 3)
[2020-06-19 18:11:00,164] smoke.utils.model_serialization INFO: backbone.body.base.level1.1.bias loaded from level1.1.bias of shape (32,)
[2020-06-19 18:11:00,164] smoke.utils.model_serialization INFO: backbone.body.base.level1.1.weight loaded from level1.1.weight of shape (32,)
[2020-06-19 18:11:00,164] smoke.utils.model_serialization INFO: backbone.body.base.level2.project.0.weight loaded from level2.project.0.weight of shape (64, 32, 1, 1)
[2020-06-19 18:11:00,164] smoke.utils.model_serialization INFO: backbone.body.base.level2.project.1.bias loaded from level2.project.1.bias of shape (64,)
[2020-06-19 18:11:00,164] smoke.utils.model_serialization INFO: backbone.body.base.level2.project.1.weight loaded from level2.project.1.weight of shape (64,)
[2020-06-19 18:11:00,164] smoke.utils.model_serialization INFO: backbone.body.base.level2.root.conv.weight loaded from level2.root.conv.weight of shape (64, 128, 1, 1)
[2020-06-19 18:11:00,164] smoke.utils.model_serialization INFO: backbone.body.base.level2.tree1.conv1.weight loaded from level2.tree1.conv1.weight of shape (64, 32, 3, 3)
[2020-06-19 18:11:00,164] smoke.utils.model_serialization INFO: backbone.body.base.level2.tree1.conv2.weight loaded from level2.tree1.conv2.weight of shape (64, 64, 3, 3)
[2020-06-19 18:11:00,164] smoke.utils.model_serialization INFO: backbone.body.base.level2.tree2.conv1.weight loaded from level2.tree2.conv1.weight of shape (64, 64, 3, 3)
[2020-06-19 18:11:00,164] smoke.utils.model_serialization INFO: backbone.body.base.level2.tree2.conv2.weight loaded from level2.tree2.conv2.weight of shape (64, 64, 3, 3)
[2020-06-19 18:11:00,164] smoke.utils.model_serialization INFO: backbone.body.base.level3.project.0.weight loaded from level3.project.0.weight of shape (128, 64, 1, 1)
[2020-06-19 18:11:00,164] smoke.utils.model_serialization INFO: backbone.body.base.level3.project.1.bias loaded from level3.project.1.bias of shape (128,)
[2020-06-19 18:11:00,164] smoke.utils.model_serialization INFO: backbone.body.base.level3.project.1.weight loaded from level3.project.1.weight of shape (128,)
[2020-06-19 18:11:00,164] smoke.utils.model_serialization INFO: backbone.body.base.level3.tree1.project.0.weight loaded from level3.tree1.project.0.weight of shape (128, 64, 1, 1)
[2020-06-19 18:11:00,164] smoke.utils.model_serialization INFO: backbone.body.base.level3.tree1.project.1.bias loaded from level3.tree1.project.1.bias of shape (128,)
[2020-06-19 18:11:00,164] smoke.utils.model_serialization INFO: backbone.body.base.level3.tree1.project.1.weight loaded from level3.tree1.project.1.weight of shape (128,)
[2020-06-19 18:11:00,164] smoke.utils.model_serialization INFO: backbone.body.base.level3.tree1.root.conv.weight loaded from level3.tree1.root.conv.weight of shape (128, 256, 1, 1)
[2020-06-19 18:11:00,164] smoke.utils.model_serialization INFO: backbone.body.base.level3.tree1.tree1.conv1.weight loaded from level3.tree1.tree1.conv1.weight of shape (128, 64, 3, 3)
[2020-06-19 18:11:00,164] smoke.utils.model_serialization INFO: backbone.body.base.level3.tree1.tree1.conv2.weight loaded from level3.tree1.tree1.conv2.weight of shape (128, 128, 3, 3)
[2020-06-19 18:11:00,164] smoke.utils.model_serialization INFO: backbone.body.base.level3.tree1.tree2.conv1.weight loaded from level3.tree1.tree2.conv1.weight of shape (128, 128, 3, 3)
[2020-06-19 18:11:00,164] smoke.utils.model_serialization INFO: backbone.body.base.level3.tree1.tree2.conv2.weight loaded from level3.tree1.tree2.conv2.weight of shape (128, 128, 3, 3)
[2020-06-19 18:11:00,164] smoke.utils.model_serialization INFO: backbone.body.base.level3.tree2.root.conv.weight loaded from level3.tree2.root.conv.weight of shape (128, 448, 1, 1)
[2020-06-19 18:11:00,164] smoke.utils.model_serialization INFO: backbone.body.base.level3.tree2.tree1.conv1.weight loaded from level3.tree2.tree1.conv1.weight of shape (128, 128, 3, 3)
[2020-06-19 18:11:00,164] smoke.utils.model_serialization INFO: backbone.body.base.level3.tree2.tree1.conv2.weight loaded from level3.tree2.tree1.conv2.weight of shape (128, 128, 3, 3)
[2020-06-19 18:11:00,164] smoke.utils.model_serialization INFO: backbone.body.base.level3.tree2.tree2.conv1.weight loaded from level3.tree2.tree2.conv1.weight of shape (128, 128, 3, 3)
[2020-06-19 18:11:00,164] smoke.utils.model_serialization INFO: backbone.body.base.level3.tree2.tree2.conv2.weight loaded from level3.tree2.tree2.conv2.weight of shape (128, 128, 3, 3)
[2020-06-19 18:11:00,164] smoke.utils.model_serialization INFO: backbone.body.base.level4.project.0.weight loaded from level4.project.0.weight of shape (256, 128, 1, 1)
[2020-06-19 18:11:00,165] smoke.utils.model_serialization INFO: backbone.body.base.level4.project.1.bias loaded from level4.project.1.bias of shape (256,)
[2020-06-19 18:11:00,165] smoke.utils.model_serialization INFO: backbone.body.base.level4.project.1.weight loaded from level4.project.1.weight of shape (256,)
[2020-06-19 18:11:00,165] smoke.utils.model_serialization INFO: backbone.body.base.level4.tree1.project.0.weight loaded from level4.tree1.project.0.weight of shape (256, 128, 1, 1)
[2020-06-19 18:11:00,165] smoke.utils.model_serialization INFO: backbone.body.base.level4.tree1.project.1.bias loaded from level4.tree1.project.1.bias of shape (256,)
[2020-06-19 18:11:00,165] smoke.utils.model_serialization INFO: backbone.body.base.level4.tree1.project.1.weight loaded from level4.tree1.project.1.weight of shape (256,)
[2020-06-19 18:11:00,165] smoke.utils.model_serialization INFO: backbone.body.base.level4.tree1.root.conv.weight loaded from level4.tree1.root.conv.weight of shape (256, 512, 1, 1)
[2020-06-19 18:11:00,165] smoke.utils.model_serialization INFO: backbone.body.base.level4.tree1.tree1.conv1.weight loaded from level4.tree1.tree1.conv1.weight of shape (256, 128, 3, 3)
[2020-06-19 18:11:00,165] smoke.utils.model_serialization INFO: backbone.body.base.level4.tree1.tree1.conv2.weight loaded from level4.tree1.tree1.conv2.weight of shape (256, 256, 3, 3)
[2020-06-19 18:11:00,165] smoke.utils.model_serialization INFO: backbone.body.base.level4.tree1.tree2.conv1.weight loaded from level4.tree1.tree2.conv1.weight of shape (256, 256, 3, 3)
[2020-06-19 18:11:00,165] smoke.utils.model_serialization INFO: backbone.body.base.level4.tree1.tree2.conv2.weight loaded from level4.tree1.tree2.conv2.weight of shape (256, 256, 3, 3)
[2020-06-19 18:11:00,165] smoke.utils.model_serialization INFO: backbone.body.base.level4.tree2.root.conv.weight loaded from level4.tree2.root.conv.weight of shape (256, 896, 1, 1)
[2020-06-19 18:11:00,165] smoke.utils.model_serialization INFO: backbone.body.base.level4.tree2.tree1.conv1.weight loaded from level4.tree2.tree1.conv1.weight of shape (256, 256, 3, 3)
[2020-06-19 18:11:00,165] smoke.utils.model_serialization INFO: backbone.body.base.level4.tree2.tree1.conv2.weight loaded from level4.tree2.tree1.conv2.weight of shape (256, 256, 3, 3)
[2020-06-19 18:11:00,165] smoke.utils.model_serialization INFO: backbone.body.base.level4.tree2.tree2.conv1.weight loaded from level4.tree2.tree2.conv1.weight of shape (256, 256, 3, 3)
[2020-06-19 18:11:00,165] smoke.utils.model_serialization INFO: backbone.body.base.level4.tree2.tree2.conv2.weight loaded from level4.tree2.tree2.conv2.weight of shape (256, 256, 3, 3)
[2020-06-19 18:11:00,165] smoke.utils.model_serialization INFO: backbone.body.base.level5.project.0.weight loaded from level5.project.0.weight of shape (512, 256, 1, 1)
[2020-06-19 18:11:00,165] smoke.utils.model_serialization INFO: backbone.body.base.level5.project.1.bias loaded from level5.project.1.bias of shape (512,)
[2020-06-19 18:11:00,165] smoke.utils.model_serialization INFO: backbone.body.base.level5.project.1.weight loaded from level5.project.1.weight of shape (512,)
[2020-06-19 18:11:00,165] smoke.utils.model_serialization INFO: backbone.body.base.level5.root.conv.weight loaded from level5.root.conv.weight of shape (512, 1280, 1, 1)
[2020-06-19 18:11:00,165] smoke.utils.model_serialization INFO: backbone.body.base.level5.tree1.conv1.weight loaded from level5.tree1.conv1.weight of shape (512, 256, 3, 3)
[2020-06-19 18:11:00,165] smoke.utils.model_serialization INFO: backbone.body.base.level5.tree1.conv2.weight loaded from level5.tree1.conv2.weight of shape (512, 512, 3, 3)
[2020-06-19 18:11:00,165] smoke.utils.model_serialization INFO: backbone.body.base.level5.tree2.conv1.weight loaded from level5.tree2.conv1.weight of shape (512, 512, 3, 3)
[2020-06-19 18:11:00,165] smoke.utils.model_serialization INFO: backbone.body.base.level5.tree2.conv2.weight loaded from level5.tree2.conv2.weight of shape (512, 512, 3, 3)
[2020-06-19 18:11:00,191] smoke.data.datasets.kitti INFO: Initializing KITTI trainval set with 7481 files loaded
[2020-06-19 18:11:00,191] smoke.trainer INFO: Start training
Traceback (most recent call last):
File "tools/plain_train_net.py", line 100, in <module>
args=(args,),
File "/home/robot1/Shukai/SMOKE/smoke/engine/launch.py", line 56, in launch
main_func(*args)
File "tools/plain_train_net.py", line 88, in main
train(cfg, model, device, distributed)
File "tools/plain_train_net.py", line 53, in train
arguments
File "/home/robot1/Shukai/SMOKE/smoke/engine/trainer.py", line 58, in do_train
for data, iteration in zip(data_loader, range(start_iter, max_iter)):
File "/home/robot1/anaconda3/envs/SMOKE/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/home/robot1/anaconda3/envs/SMOKE/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/home/robot1/anaconda3/envs/SMOKE/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/home/robot1/anaconda3/envs/SMOKE/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/home/robot1/anaconda3/envs/SMOKE/lib/python3.7/multiprocessing/queues.py", line 113, in get
return _ForkingPickler.loads(res)
File "/home/robot1/anaconda3/envs/SMOKE/lib/python3.7/site-packages/torch/multiprocessing/reductions.py", line 294, in rebuild_storage_fd
fd = df.detach()
File "/home/robot1/anaconda3/envs/SMOKE/lib/python3.7/multiprocessing/resource_sharer.py", line 58, in detach
return reduction.recv_handle(conn)
File "/home/robot1/anaconda3/envs/SMOKE/lib/python3.7/multiprocessing/reduction.py", line 185, in recv_handle
return recvfds(s, 1)[0]
File "/home/robot1/anaconda3/envs/SMOKE/lib/python3.7/multiprocessing/reduction.py", line 161, in recvfds
len(ancdata))
RuntimeError: received 0 items of ancdata