Default process group has not been initialized, please make sure to call init_process_group about mmrazor

[humu789]

You can try to add the parameter: --launcher in your running command to perform init_dist

[humu789]

Can you tell me how much GPUs to use in your experiments

[xingyueye]

@humu789 Hi, I encountered this problem as well with single GPU. It seems because that torch.distributed is called when not been initialized in "mmrazor/mmrazor/core/utils/broadcast.py"
dist.broadcast(dir_tensor, src)
And the initialization process is ignored in "./tools/mmdet/search_mmdet.py"(and other search scripts)

    if args.launcher == 'none':
        distributed = False
    else:
        distributed = True
        init_dist(args.launcher, **cfg.dist_params)

[humu789]

@xingyueye Yes, we are trying to solve it better. You can quick run your experiment by referring to it.

You can try to add the parameter: --launcher in your running command to perform init_dist

[daitranskku]

I had the same issue with two TeslaV100. I tried —launch, but I still get the same error. Is anyone resolving these issues? Thanks

[humu789]

@daitranskku Can you provide details of the command to perform your experiment?

[daitranskku]

@humu789 I tried step 3 with the command below.
Also, I added --launcher pytorch, and it produced "KeyError: 'RANK"

Step 3

python tools/mmdet/search_mmdet.py \
configs/step3.py \
work_dirs/step2/latest.pth 

---

022-02-16 09:25:59,017 - mmdet - INFO - Environment info:

sys.platform: linux
Python: 3.7.11 (default, Jul 27 2021, 14:32:16) [GCC 7.5.0]
CUDA available: True
GPU 0,1: NVIDIA Tesla V100-PCIE-32GB
CUDA_HOME: /usr/local/cuda
NVCC: Build cuda_11.3.r11.3/compiler.29920130_0
GCC: gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-4)
PyTorch: 1.10.1+cu113
PyTorch compiling details: PyTorch built with:

• GCC 7.3
• C++ Version: 201402
• Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
• Intel(R) MKL-DNN v2.2.3 (Git Hash 7336ca9f055cf1bfa13efb658fe15dc9b41f0740)
• OpenMP 201511 (a.k.a. OpenMP 4.5)
• LAPACK is enabled (usually provided by MKL)
• NNPACK is enabled
• CPU capability usage: AVX512
• CUDA Runtime 11.3
• NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
• CuDNN 8.2
• Magma 2.5.2
• Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.3, CUDNN_VERSION=8.2.0, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.10.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,
  TorchVision: 0.11.2+cu113
  OpenCV: 4.5.5
  MMCV: 1.4.3
  MMCV Compiler: GCC 7.3
  MMCV CUDA Compiler: 11.3
  MMDetection: 2.20.0+0dd407a

---

2022-02-16 09:25:59,018 - mmdet - INFO - Distributed training: False
2022-02-16 09:26:00,414 - mmdet - INFO - Config:
dataset_type = 'COCODataset'
data_root = 'data/coco/'
img_norm_cfg = dict(
mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
train_pipeline = [
dict(type='LoadImageFromFile'),
dict(type='LoadAnnotations', with_bbox=True),
dict(type='Resize', img_scale=(1333, 800), keep_ratio=True),
dict(type='RandomFlip', flip_ratio=0.5),
dict(
type='Normalize',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
to_rgb=True),
dict(type='Pad', size_divisor=32),
dict(type='DefaultFormatBundle'),
dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'])
]
test_pipeline = [
dict(type='LoadImageFromFile'),
dict(
type='MultiScaleFlipAug',
img_scale=(1333, 800),
flip=False,
transforms=[
dict(type='Resize', keep_ratio=True),
dict(type='RandomFlip'),
dict(
type='Normalize',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
to_rgb=True),
dict(type='Pad', size_divisor=32),
dict(type='ImageToTensor', keys=['img']),
dict(type='Collect', keys=['img'])
])
]
data = dict(
samples_per_gpu=5,
workers_per_gpu=10,
train=dict(
type='CocoDataset',
ann_file=
'.json',
img_prefix=
'',
pipeline=[
dict(type='LoadImageFromFile'),
dict(type='LoadAnnotations', with_bbox=True),
dict(type='Resize', img_scale=(1333, 800), keep_ratio=True),
dict(type='RandomFlip', flip_ratio=0.5),
dict(
type='Normalize',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
to_rgb=True),
dict(type='Pad', size_divisor=32),
dict(type='DefaultFormatBundle'),
dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'])
],
classes=('15', '10', '16', '18', '13', '09', '12', '17', '11', '19',
'14')),
val=dict(
type='CocoDataset',
ann_file=
'.json',
img_prefix=
'/',
pipeline=[
dict(type='LoadImageFromFile'),
dict(
type='MultiScaleFlipAug',
img_scale=(1333, 800),
flip=False,
transforms=[
dict(type='Resize', keep_ratio=True),
dict(type='RandomFlip'),
dict(
type='Normalize',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
to_rgb=True),
dict(type='Pad', size_divisor=32),
dict(type='ImageToTensor', keys=['img']),
dict(type='Collect', keys=['img'])
])
],
classes=('15', '10', '16', '18', '13', '09', '12', '17', '11', '19',
'14')),
test=dict(
type='CocoDataset',
ann_file=
'.json',
img_prefix=
'/',
pipeline=[
dict(type='LoadImageFromFile'),
dict(
type='MultiScaleFlipAug',
img_scale=(1333, 800),
flip=False,
transforms=[
dict(type='Resize', keep_ratio=True),
dict(type='RandomFlip'),
dict(
type='Normalize',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
to_rgb=True),
dict(type='Pad', size_divisor=32),
dict(type='ImageToTensor', keys=['img']),
dict(type='Collect', keys=['img'])
])
],
classes=('15', '10', '16', '18', '13', '09', '12', '17', '11', '19',
'14'),
test_mode=True))
evaluation = dict(interval=1, metric='bbox')
optimizer = dict(type='SGD', lr=0.02, momentum=0.9, weight_decay=0.0001)
optimizer_config = dict(grad_clip=None)
lr_config = dict(
policy='step',
warmup='linear',
warmup_iters=500,
warmup_ratio=0.001,
step=[8, 11])
runner = dict(type='EpochBasedRunner', max_epochs=12)
checkpoint_config = dict(interval=1)
log_config = dict(interval=50, hooks=[dict(type='TextLoggerHook')])
custom_hooks = [dict(type='NumClassCheckHook')]
dist_params = dict(backend='nccl')
log_level = 'INFO'
load_from = None
resume_from = None
workflow = [('train', 1)]
norm_cfg = dict(type='BN', requires_grad=True)
model = dict(
type='mmdet.FasterRCNN',
backbone=dict(
type='mmcls.SearchableShuffleNetV2',
norm_cfg=dict(type='BN', requires_grad=True),
out_indices=(0, 1, 2, 3),
widen_factor=1.0,
with_last_layer=False),
neck=dict(
type='FPN',
norm_cfg=dict(type='BN', requires_grad=True),
in_channels=[64, 160, 320, 640],
out_channels=256,
num_outs=5),
rpn_head=dict(
type='RPNHead',
in_channels=256,
feat_channels=256,
anchor_generator=dict(
type='AnchorGenerator',
scales=[8],
ratios=[0.5, 1.0, 2.0],
strides=[4, 8, 16, 32, 64]),
bbox_coder=dict(
type='DeltaXYWHBBoxCoder',
target_means=[0.0, 0.0, 0.0, 0.0],
target_stds=[1.0, 1.0, 1.0, 1.0]),
loss_cls=dict(
type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0),
loss_bbox=dict(type='L1Loss', loss_weight=1.0)),
roi_head=dict(
type='StandardRoIHead',
bbox_roi_extractor=dict(
type='SingleRoIExtractor',
roi_layer=dict(type='RoIAlign', output_size=7, sampling_ratio=0),
out_channels=256,
featmap_strides=[4, 8, 16, 32]),
bbox_head=dict(
type='Shared4Conv1FCBBoxHead',
norm_cfg=dict(type='BN', requires_grad=True),
in_channels=256,
fc_out_channels=1024,
roi_feat_size=7,
num_classes=80,
bbox_coder=dict(
type='DeltaXYWHBBoxCoder',
target_means=[0.0, 0.0, 0.0, 0.0],
target_stds=[0.1, 0.1, 0.2, 0.2]),
reg_class_agnostic=False,
loss_cls=dict(
type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0),
loss_bbox=dict(type='L1Loss', loss_weight=1.0))),
train_cfg=dict(
rpn=dict(
assigner=dict(
type='MaxIoUAssigner',
pos_iou_thr=0.7,
neg_iou_thr=0.3,
min_pos_iou=0.3,
match_low_quality=True,
ignore_iof_thr=-1),
sampler=dict(
type='RandomSampler',
num=256,
pos_fraction=0.5,
neg_pos_ub=-1,
add_gt_as_proposals=False),
allowed_border=-1,
pos_weight=-1,
debug=False),
rpn_proposal=dict(
nms_pre=2000,
max_per_img=1000,
nms=dict(type='nms', iou_threshold=0.7),
min_bbox_size=0),
rcnn=dict(
assigner=dict(
type='MaxIoUAssigner',
pos_iou_thr=0.5,
neg_iou_thr=0.5,
min_pos_iou=0.5,
match_low_quality=False,
ignore_iof_thr=-1),
sampler=dict(
type='RandomSampler',
num=512,
pos_fraction=0.25,
neg_pos_ub=-1,
add_gt_as_proposals=True),
pos_weight=-1,
debug=False)),
test_cfg=dict(
rpn=dict(
nms_pre=1000,
max_per_img=1000,
nms=dict(type='nms', iou_threshold=0.7),
min_bbox_size=0),
rcnn=dict(
score_thr=0.05,
nms=dict(type='nms', iou_threshold=0.5),
max_per_img=100)))
mutator = dict(
type='OneShotMutator',
placeholder_mapping=dict(
all_blocks=dict(
type='OneShotOP',
choices=dict(
shuffle_3x3=dict(
type='ShuffleBlock',
norm_cfg=dict(type='BN', requires_grad=True),
kernel_size=3),
shuffle_5x5=dict(
type='ShuffleBlock',
norm_cfg=dict(type='BN', requires_grad=True),
kernel_size=5),
shuffle_7x7=dict(
type='ShuffleBlock',
norm_cfg=dict(type='BN', requires_grad=True),
kernel_size=7),
shuffle_xception=dict(
type='ShuffleXception',
norm_cfg=dict(type='BN', requires_grad=True))))))
algorithm = dict(
type='DetNAS',
architecture=dict(
type='MMDetArchitecture',
model=dict(
type='mmdet.FasterRCNN',
backbone=dict(
type='mmcls.SearchableShuffleNetV2',
norm_cfg=dict(type='BN', requires_grad=True),
out_indices=(0, 1, 2, 3),
widen_factor=1.0,
with_last_layer=False),
neck=dict(
type='FPN',
norm_cfg=dict(type='BN', requires_grad=True),
in_channels=[64, 160, 320, 640],
out_channels=256,
num_outs=5),
rpn_head=dict(
type='RPNHead',
in_channels=256,
feat_channels=256,
anchor_generator=dict(
type='AnchorGenerator',
scales=[8],
ratios=[0.5, 1.0, 2.0],
strides=[4, 8, 16, 32, 64]),
bbox_coder=dict(
type='DeltaXYWHBBoxCoder',
target_means=[0.0, 0.0, 0.0, 0.0],
target_stds=[1.0, 1.0, 1.0, 1.0]),
loss_cls=dict(
type='CrossEntropyLoss', use_sigmoid=True,
loss_weight=1.0),
loss_bbox=dict(type='L1Loss', loss_weight=1.0)),
roi_head=dict(
type='StandardRoIHead',
bbox_roi_extractor=dict(
type='SingleRoIExtractor',
roi_layer=dict(
type='RoIAlign', output_size=7, sampling_ratio=0),
out_channels=256,
featmap_strides=[4, 8, 16, 32]),
bbox_head=dict(
type='Shared4Conv1FCBBoxHead',
norm_cfg=dict(type='BN', requires_grad=True),
in_channels=256,
fc_out_channels=1024,
roi_feat_size=7,
num_classes=80,
bbox_coder=dict(
type='DeltaXYWHBBoxCoder',
target_means=[0.0, 0.0, 0.0, 0.0],
target_stds=[0.1, 0.1, 0.2, 0.2]),
reg_class_agnostic=False,
loss_cls=dict(
type='CrossEntropyLoss',
use_sigmoid=False,
loss_weight=1.0),
loss_bbox=dict(type='L1Loss', loss_weight=1.0))),
train_cfg=dict(
rpn=dict(
assigner=dict(
type='MaxIoUAssigner',
pos_iou_thr=0.7,
neg_iou_thr=0.3,
min_pos_iou=0.3,
match_low_quality=True,
ignore_iof_thr=-1),
sampler=dict(
type='RandomSampler',
num=256,
pos_fraction=0.5,
neg_pos_ub=-1,
add_gt_as_proposals=False),
allowed_border=-1,
pos_weight=-1,
debug=False),
rpn_proposal=dict(
nms_pre=2000,
max_per_img=1000,
nms=dict(type='nms', iou_threshold=0.7),
min_bbox_size=0),
rcnn=dict(
assigner=dict(
type='MaxIoUAssigner',
pos_iou_thr=0.5,
neg_iou_thr=0.5,
min_pos_iou=0.5,
match_low_quality=False,
ignore_iof_thr=-1),
sampler=dict(
type='RandomSampler',
num=512,
pos_fraction=0.25,
neg_pos_ub=-1,
add_gt_as_proposals=True),
pos_weight=-1,
debug=False)),
test_cfg=dict(
rpn=dict(
nms_pre=1000,
max_per_img=1000,
nms=dict(type='nms', iou_threshold=0.7),
min_bbox_size=0),
rcnn=dict(
score_thr=0.05,
nms=dict(type='nms', iou_threshold=0.5),
max_per_img=100)),
pretrained=None)),
mutator=dict(
type='OneShotMutator',
placeholder_mapping=dict(
all_blocks=dict(
type='OneShotOP',
choices=dict(
shuffle_3x3=dict(
type='ShuffleBlock',
norm_cfg=dict(type='BN', requires_grad=True),
kernel_size=3),
shuffle_5x5=dict(
type='ShuffleBlock',
norm_cfg=dict(type='BN', requires_grad=True),
kernel_size=5),
shuffle_7x7=dict(
type='ShuffleBlock',
norm_cfg=dict(type='BN', requires_grad=True),
kernel_size=7),
shuffle_xception=dict(
type='ShuffleXception',
norm_cfg=dict(type='BN', requires_grad=True)))))),
pruner=None,
distiller=None,
retraining=False)
find_unused_parameters = True
classes = ('15', '10', '16', '18', '13', '09', '12', '17', '11', '19', '14')
searcher = dict(
type='EvolutionSearcher',
metrics='bbox',
score_key='bbox_mAP',
constraints=dict(flops=300000000.0),
candidate_pool_size=50,
candidate_top_k=10,
max_epoch=20,
num_mutation=20,
num_crossover=20)
work_dir = 'work_dirs/step3'

2022-02-16 09:26:10,328 - mmdet - INFO - build search...
2022-02-16 09:26:10,329 - mmdet - INFO - start search...
2022-02-16 09:26:10,329 - mmdet - INFO - Experiment setting:
2022-02-16 09:26:10,329 - mmdet - INFO - candidate_pool_size: 50
2022-02-16 09:26:10,329 - mmdet - INFO - candidate_top_k: 10
2022-02-16 09:26:10,329 - mmdet - INFO - num_crossover: 20
2022-02-16 09:26:10,329 - mmdet - INFO - num_mutation: 20
2022-02-16 09:26:10,329 - mmdet - INFO - mutate_prob: 0.1
2022-02-16 09:26:10,329 - mmdet - INFO - max_epoch: 20
2022-02-16 09:26:10,329 - mmdet - INFO - score_key: bbox_mAP
2022-02-16 09:26:10,329 - mmdet - INFO - constraints: {'flops': 300000000.0}
2022-02-16 09:26:10,329 - mmdet - INFO - ####################################################################################################

[humu789]

@daitranskku distributed is False in your log, which means that init_dist is not been executed in search_mmdet.py. You can try to check whther --launcher is used correctly.

[daitranskku]

Thanks @humu789 . I added -m torch.distributed.launch and its working.