Comments (9)
You can try to add the parameter: --launcher
in your running command to perform init_dist
from mmrazor.
Can you tell me how much GPUs to use in your experiments
from mmrazor.
@humu789 Hi, I encountered this problem as well with single GPU. It seems because that torch.distributed is called when not been initialized in "mmrazor/mmrazor/core/utils/broadcast.py"
dist.broadcast(dir_tensor, src)
And the initialization process is ignored in "./tools/mmdet/search_mmdet.py"(and other search scripts)
if args.launcher == 'none':
distributed = False
else:
distributed = True
init_dist(args.launcher, **cfg.dist_params)
from mmrazor.
@xingyueye Yes, we are trying to solve it better. You can quick run your experiment by referring to it.
You can try to add the parameter:
--launcher
in your running command to performinit_dist
from mmrazor.
I had the same issue with two TeslaV100. I tried —launch
, but I still get the same error. Is anyone resolving these issues? Thanks
from mmrazor.
@daitranskku Can you provide details of the command to perform your experiment?
from mmrazor.
@humu789 I tried step 3 with the command below.
Also, I added --launcher pytorch
, and it produced "KeyError: 'RANK"
Step 3
python tools/mmdet/search_mmdet.py \
configs/step3.py \
work_dirs/step2/latest.pth
022-02-16 09:25:59,017 - mmdet - INFO - Environment info:
sys.platform: linux
Python: 3.7.11 (default, Jul 27 2021, 14:32:16) [GCC 7.5.0]
CUDA available: True
GPU 0,1: NVIDIA Tesla V100-PCIE-32GB
CUDA_HOME: /usr/local/cuda
NVCC: Build cuda_11.3.r11.3/compiler.29920130_0
GCC: gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-4)
PyTorch: 1.10.1+cu113
PyTorch compiling details: PyTorch built with:
- GCC 7.3
- C++ Version: 201402
- Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v2.2.3 (Git Hash 7336ca9f055cf1bfa13efb658fe15dc9b41f0740)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- LAPACK is enabled (usually provided by MKL)
- NNPACK is enabled
- CPU capability usage: AVX512
- CUDA Runtime 11.3
- NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
- CuDNN 8.2
- Magma 2.5.2
- Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.3, CUDNN_VERSION=8.2.0, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.10.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,
TorchVision: 0.11.2+cu113
OpenCV: 4.5.5
MMCV: 1.4.3
MMCV Compiler: GCC 7.3
MMCV CUDA Compiler: 11.3
MMDetection: 2.20.0+0dd407a
2022-02-16 09:25:59,018 - mmdet - INFO - Distributed training: False
2022-02-16 09:26:00,414 - mmdet - INFO - Config:
dataset_type = 'COCODataset'
data_root = 'data/coco/'
img_norm_cfg = dict(
mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
train_pipeline = [
dict(type='LoadImageFromFile'),
dict(type='LoadAnnotations', with_bbox=True),
dict(type='Resize', img_scale=(1333, 800), keep_ratio=True),
dict(type='RandomFlip', flip_ratio=0.5),
dict(
type='Normalize',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
to_rgb=True),
dict(type='Pad', size_divisor=32),
dict(type='DefaultFormatBundle'),
dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'])
]
test_pipeline = [
dict(type='LoadImageFromFile'),
dict(
type='MultiScaleFlipAug',
img_scale=(1333, 800),
flip=False,
transforms=[
dict(type='Resize', keep_ratio=True),
dict(type='RandomFlip'),
dict(
type='Normalize',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
to_rgb=True),
dict(type='Pad', size_divisor=32),
dict(type='ImageToTensor', keys=['img']),
dict(type='Collect', keys=['img'])
])
]
data = dict(
samples_per_gpu=5,
workers_per_gpu=10,
train=dict(
type='CocoDataset',
ann_file=
'.json',
img_prefix=
'',
pipeline=[
dict(type='LoadImageFromFile'),
dict(type='LoadAnnotations', with_bbox=True),
dict(type='Resize', img_scale=(1333, 800), keep_ratio=True),
dict(type='RandomFlip', flip_ratio=0.5),
dict(
type='Normalize',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
to_rgb=True),
dict(type='Pad', size_divisor=32),
dict(type='DefaultFormatBundle'),
dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'])
],
classes=('15', '10', '16', '18', '13', '09', '12', '17', '11', '19',
'14')),
val=dict(
type='CocoDataset',
ann_file=
'.json',
img_prefix=
'/',
pipeline=[
dict(type='LoadImageFromFile'),
dict(
type='MultiScaleFlipAug',
img_scale=(1333, 800),
flip=False,
transforms=[
dict(type='Resize', keep_ratio=True),
dict(type='RandomFlip'),
dict(
type='Normalize',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
to_rgb=True),
dict(type='Pad', size_divisor=32),
dict(type='ImageToTensor', keys=['img']),
dict(type='Collect', keys=['img'])
])
],
classes=('15', '10', '16', '18', '13', '09', '12', '17', '11', '19',
'14')),
test=dict(
type='CocoDataset',
ann_file=
'.json',
img_prefix=
'/',
pipeline=[
dict(type='LoadImageFromFile'),
dict(
type='MultiScaleFlipAug',
img_scale=(1333, 800),
flip=False,
transforms=[
dict(type='Resize', keep_ratio=True),
dict(type='RandomFlip'),
dict(
type='Normalize',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
to_rgb=True),
dict(type='Pad', size_divisor=32),
dict(type='ImageToTensor', keys=['img']),
dict(type='Collect', keys=['img'])
])
],
classes=('15', '10', '16', '18', '13', '09', '12', '17', '11', '19',
'14'),
test_mode=True))
evaluation = dict(interval=1, metric='bbox')
optimizer = dict(type='SGD', lr=0.02, momentum=0.9, weight_decay=0.0001)
optimizer_config = dict(grad_clip=None)
lr_config = dict(
policy='step',
warmup='linear',
warmup_iters=500,
warmup_ratio=0.001,
step=[8, 11])
runner = dict(type='EpochBasedRunner', max_epochs=12)
checkpoint_config = dict(interval=1)
log_config = dict(interval=50, hooks=[dict(type='TextLoggerHook')])
custom_hooks = [dict(type='NumClassCheckHook')]
dist_params = dict(backend='nccl')
log_level = 'INFO'
load_from = None
resume_from = None
workflow = [('train', 1)]
norm_cfg = dict(type='BN', requires_grad=True)
model = dict(
type='mmdet.FasterRCNN',
backbone=dict(
type='mmcls.SearchableShuffleNetV2',
norm_cfg=dict(type='BN', requires_grad=True),
out_indices=(0, 1, 2, 3),
widen_factor=1.0,
with_last_layer=False),
neck=dict(
type='FPN',
norm_cfg=dict(type='BN', requires_grad=True),
in_channels=[64, 160, 320, 640],
out_channels=256,
num_outs=5),
rpn_head=dict(
type='RPNHead',
in_channels=256,
feat_channels=256,
anchor_generator=dict(
type='AnchorGenerator',
scales=[8],
ratios=[0.5, 1.0, 2.0],
strides=[4, 8, 16, 32, 64]),
bbox_coder=dict(
type='DeltaXYWHBBoxCoder',
target_means=[0.0, 0.0, 0.0, 0.0],
target_stds=[1.0, 1.0, 1.0, 1.0]),
loss_cls=dict(
type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0),
loss_bbox=dict(type='L1Loss', loss_weight=1.0)),
roi_head=dict(
type='StandardRoIHead',
bbox_roi_extractor=dict(
type='SingleRoIExtractor',
roi_layer=dict(type='RoIAlign', output_size=7, sampling_ratio=0),
out_channels=256,
featmap_strides=[4, 8, 16, 32]),
bbox_head=dict(
type='Shared4Conv1FCBBoxHead',
norm_cfg=dict(type='BN', requires_grad=True),
in_channels=256,
fc_out_channels=1024,
roi_feat_size=7,
num_classes=80,
bbox_coder=dict(
type='DeltaXYWHBBoxCoder',
target_means=[0.0, 0.0, 0.0, 0.0],
target_stds=[0.1, 0.1, 0.2, 0.2]),
reg_class_agnostic=False,
loss_cls=dict(
type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0),
loss_bbox=dict(type='L1Loss', loss_weight=1.0))),
train_cfg=dict(
rpn=dict(
assigner=dict(
type='MaxIoUAssigner',
pos_iou_thr=0.7,
neg_iou_thr=0.3,
min_pos_iou=0.3,
match_low_quality=True,
ignore_iof_thr=-1),
sampler=dict(
type='RandomSampler',
num=256,
pos_fraction=0.5,
neg_pos_ub=-1,
add_gt_as_proposals=False),
allowed_border=-1,
pos_weight=-1,
debug=False),
rpn_proposal=dict(
nms_pre=2000,
max_per_img=1000,
nms=dict(type='nms', iou_threshold=0.7),
min_bbox_size=0),
rcnn=dict(
assigner=dict(
type='MaxIoUAssigner',
pos_iou_thr=0.5,
neg_iou_thr=0.5,
min_pos_iou=0.5,
match_low_quality=False,
ignore_iof_thr=-1),
sampler=dict(
type='RandomSampler',
num=512,
pos_fraction=0.25,
neg_pos_ub=-1,
add_gt_as_proposals=True),
pos_weight=-1,
debug=False)),
test_cfg=dict(
rpn=dict(
nms_pre=1000,
max_per_img=1000,
nms=dict(type='nms', iou_threshold=0.7),
min_bbox_size=0),
rcnn=dict(
score_thr=0.05,
nms=dict(type='nms', iou_threshold=0.5),
max_per_img=100)))
mutator = dict(
type='OneShotMutator',
placeholder_mapping=dict(
all_blocks=dict(
type='OneShotOP',
choices=dict(
shuffle_3x3=dict(
type='ShuffleBlock',
norm_cfg=dict(type='BN', requires_grad=True),
kernel_size=3),
shuffle_5x5=dict(
type='ShuffleBlock',
norm_cfg=dict(type='BN', requires_grad=True),
kernel_size=5),
shuffle_7x7=dict(
type='ShuffleBlock',
norm_cfg=dict(type='BN', requires_grad=True),
kernel_size=7),
shuffle_xception=dict(
type='ShuffleXception',
norm_cfg=dict(type='BN', requires_grad=True))))))
algorithm = dict(
type='DetNAS',
architecture=dict(
type='MMDetArchitecture',
model=dict(
type='mmdet.FasterRCNN',
backbone=dict(
type='mmcls.SearchableShuffleNetV2',
norm_cfg=dict(type='BN', requires_grad=True),
out_indices=(0, 1, 2, 3),
widen_factor=1.0,
with_last_layer=False),
neck=dict(
type='FPN',
norm_cfg=dict(type='BN', requires_grad=True),
in_channels=[64, 160, 320, 640],
out_channels=256,
num_outs=5),
rpn_head=dict(
type='RPNHead',
in_channels=256,
feat_channels=256,
anchor_generator=dict(
type='AnchorGenerator',
scales=[8],
ratios=[0.5, 1.0, 2.0],
strides=[4, 8, 16, 32, 64]),
bbox_coder=dict(
type='DeltaXYWHBBoxCoder',
target_means=[0.0, 0.0, 0.0, 0.0],
target_stds=[1.0, 1.0, 1.0, 1.0]),
loss_cls=dict(
type='CrossEntropyLoss', use_sigmoid=True,
loss_weight=1.0),
loss_bbox=dict(type='L1Loss', loss_weight=1.0)),
roi_head=dict(
type='StandardRoIHead',
bbox_roi_extractor=dict(
type='SingleRoIExtractor',
roi_layer=dict(
type='RoIAlign', output_size=7, sampling_ratio=0),
out_channels=256,
featmap_strides=[4, 8, 16, 32]),
bbox_head=dict(
type='Shared4Conv1FCBBoxHead',
norm_cfg=dict(type='BN', requires_grad=True),
in_channels=256,
fc_out_channels=1024,
roi_feat_size=7,
num_classes=80,
bbox_coder=dict(
type='DeltaXYWHBBoxCoder',
target_means=[0.0, 0.0, 0.0, 0.0],
target_stds=[0.1, 0.1, 0.2, 0.2]),
reg_class_agnostic=False,
loss_cls=dict(
type='CrossEntropyLoss',
use_sigmoid=False,
loss_weight=1.0),
loss_bbox=dict(type='L1Loss', loss_weight=1.0))),
train_cfg=dict(
rpn=dict(
assigner=dict(
type='MaxIoUAssigner',
pos_iou_thr=0.7,
neg_iou_thr=0.3,
min_pos_iou=0.3,
match_low_quality=True,
ignore_iof_thr=-1),
sampler=dict(
type='RandomSampler',
num=256,
pos_fraction=0.5,
neg_pos_ub=-1,
add_gt_as_proposals=False),
allowed_border=-1,
pos_weight=-1,
debug=False),
rpn_proposal=dict(
nms_pre=2000,
max_per_img=1000,
nms=dict(type='nms', iou_threshold=0.7),
min_bbox_size=0),
rcnn=dict(
assigner=dict(
type='MaxIoUAssigner',
pos_iou_thr=0.5,
neg_iou_thr=0.5,
min_pos_iou=0.5,
match_low_quality=False,
ignore_iof_thr=-1),
sampler=dict(
type='RandomSampler',
num=512,
pos_fraction=0.25,
neg_pos_ub=-1,
add_gt_as_proposals=True),
pos_weight=-1,
debug=False)),
test_cfg=dict(
rpn=dict(
nms_pre=1000,
max_per_img=1000,
nms=dict(type='nms', iou_threshold=0.7),
min_bbox_size=0),
rcnn=dict(
score_thr=0.05,
nms=dict(type='nms', iou_threshold=0.5),
max_per_img=100)),
pretrained=None)),
mutator=dict(
type='OneShotMutator',
placeholder_mapping=dict(
all_blocks=dict(
type='OneShotOP',
choices=dict(
shuffle_3x3=dict(
type='ShuffleBlock',
norm_cfg=dict(type='BN', requires_grad=True),
kernel_size=3),
shuffle_5x5=dict(
type='ShuffleBlock',
norm_cfg=dict(type='BN', requires_grad=True),
kernel_size=5),
shuffle_7x7=dict(
type='ShuffleBlock',
norm_cfg=dict(type='BN', requires_grad=True),
kernel_size=7),
shuffle_xception=dict(
type='ShuffleXception',
norm_cfg=dict(type='BN', requires_grad=True)))))),
pruner=None,
distiller=None,
retraining=False)
find_unused_parameters = True
classes = ('15', '10', '16', '18', '13', '09', '12', '17', '11', '19', '14')
searcher = dict(
type='EvolutionSearcher',
metrics='bbox',
score_key='bbox_mAP',
constraints=dict(flops=300000000.0),
candidate_pool_size=50,
candidate_top_k=10,
max_epoch=20,
num_mutation=20,
num_crossover=20)
work_dir = 'work_dirs/step3'
2022-02-16 09:26:10,328 - mmdet - INFO - build search...
2022-02-16 09:26:10,329 - mmdet - INFO - start search...
2022-02-16 09:26:10,329 - mmdet - INFO - Experiment setting:
2022-02-16 09:26:10,329 - mmdet - INFO - candidate_pool_size: 50
2022-02-16 09:26:10,329 - mmdet - INFO - candidate_top_k: 10
2022-02-16 09:26:10,329 - mmdet - INFO - num_crossover: 20
2022-02-16 09:26:10,329 - mmdet - INFO - num_mutation: 20
2022-02-16 09:26:10,329 - mmdet - INFO - mutate_prob: 0.1
2022-02-16 09:26:10,329 - mmdet - INFO - max_epoch: 20
2022-02-16 09:26:10,329 - mmdet - INFO - score_key: bbox_mAP
2022-02-16 09:26:10,329 - mmdet - INFO - constraints: {'flops': 300000000.0}
2022-02-16 09:26:10,329 - mmdet - INFO - ####################################################################################################
from mmrazor.
@daitranskku distributed
is False
in your log, which means that init_dist
is not been executed in search_mmdet.py
. You can try to check whther --launcher
is used correctly.
from mmrazor.
Thanks @humu789 . I added -m torch.distributed.launch
and its working.
from mmrazor.
Related Issues (20)
- I can't reproduce dfad results
- How to get started??
- [Bug] TypeError: 'NoneType' object is not iterable
- Try to reproduce CWD in VOC data set
- [Bug] (suggested temporary fix) Pytorch >= 2 causes mmrazor.engine to fail HOT 4
- [Bug] (suggested fix) `nn.Parameter` are not added to root after being traced in `mmrazor.models.task_utils.tracer.fx.custom_tracer.build_graphmodule()` HOT 2
- [Bug] (suggested fix) `mmrazor.models.algorithms.quantization.mm_architecture.MMArchitectureQuant.sync_qparams()` fails if there are modules present in other modes but not in forward `mode='tensor'` HOT 4
- I want to obtain the current epoch value and associate it with the custom distillation loss
- cannot use recorder to obtain panoptic_head info from mask2former
- [Bug] `mmrazor.engine.runner.quantization_loops.QATValLoop` calls `after_val_epoch` hook twice with different keys, causing `mmengine.hooks.checkpoint_hook._save_best_checkpoint()` to fail with `KeyError` for the `save_best` config
- [Bug] Custom Distillation MMSeg CWD loss nan problem
- When I use methodoutputs to access the results of assigner, I only obtain one sample
- Regarding tables and accuracy
- [Bug] (suggested fix) `mmrazor.models.algorithms.mm_architecture.MMArchitectureQuant.get_deploy_model()` fails if `predict` mode lacks nodes from the `model.quantizer.tracer.skipped_methods` configuration, but the architecture `quantizer.prepare(fp32_model)` has these nodes. HOT 4
- Is this a dead project ? HOT 1
- 我在用mmrazor通过yolov5-x蒸馏yolov5-s时候遇到了问题 HOT 1
- No Sign of activation quantization with QAT HOT 1
- MAP is stucked at 0 for Mobilenet V2 SSD QAT without pretrained model [Bug]
- [Docs] A100算力加持!书生大模型实战营第3期全面升级,趣味闯关模式等你开启
- Missing keys after RTMDET knowledge distillation HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mmrazor.