Thanks for your hard work. Much appreciated.
I try to run the command:
"python ./demo/D2Det_demo.py ./configs/D2Det/D2Det_instance_r101_fpn_2x.py ./D2Det-instance-res101.pth ./demo/demo.jpg --out ./demo/aa.jpg"
But aa.jpg only displays the detection results, NOT the segmentation results, Any ideas?

Another question: If I train with custom dataset, using the config "D2Det/D2Det_instance_r101_fpn_2x.py", Does the result contain segmentation mask? Do I need to add "mask_head" items to the configuration file?

Thank you!

Hi, thanks for sharing your code.

Do we need to install mmdetection and put this repo under it? I followed the instructions you provided but mmdetection folder does not have any D2Det folder in configs.

Hello I want to ask one question about your paper.
You say you use a pooling size of 7 × 7 (where k = 7) for classification, so "light-weight offset prediction only requires a k/2 ×k/2
sized RoIAlign" which means pass 3.5*3.5's feature map through fully connected layers ?

Hi,I want to know the inference speed of the network,Can you provide it?

hi！I'm trying to learn your work.but when I run the demo.py , the error happened. as you can see, the code can't perfom.
my environmental configuration：
python=3.7
pytorch=1.4
cuda = 10.0
mmcv=0.4.3

inds = nms_cuda.nms(dets_th, iou_thr) RuntimeError: CUDA error: invalid device function (launch_kernel at /opt/conda/conda-bld/pytorch_1579022060824/work/aten/src/ATen/native/cuda/Loops.cuh:103) frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x47 (0x7f4a9e223627 in /home/chenxiangbo/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/lib/libc10.so) frame #1: void at::native::gpu_index_kernel<__nv_dl_wrapper_t<__nv_dl_tag<void (*)(at::TensorIterator&, c10::ArrayRef<long>, c10::ArrayRef<long>), &(void at::native::index_kernel_impl<at::native::OpaqueType<8> >(at::TensorIterator&, c10::ArrayRef<long>, c10::ArrayRef<long>)), 1u>> >(at::TensorIterator&, c10::ArrayRef<long>, c10::ArrayRef<long>, __nv_dl_wrapper_t<__nv_dl_tag<void (*)(at::TensorIterator&, c10::ArrayRef<long>, c10::ArrayRef<long>), &(void at::native::index_kernel_impl<at::native::OpaqueType<8> >(at::TensorIterator&, c10::ArrayRef<long>, c10::ArrayRef<long>)), 1u>> const&) + 0x78d (0x7f4aaa4fc68d in /home/chenxiangbo/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/lib/libtorch.so) frame #2: <unknown function> + 0x571bf32 (0x7f4aaa4f5f32 in /home/chenxiangbo/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/lib/libtorch.so) frame #3: <unknown function> + 0x571c298 (0x7f4aaa4f6298 in /home/chenxiangbo/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/lib/libtorch.so) frame #4: <unknown function> + 0x16957eb (0x7f4aa646f7eb in /home/chenxiangbo/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/lib/libtorch.so) frame #5: at::native::index(at::Tensor const&, c10::ArrayRef<at::Tensor>) + 0x47e (0x7f4aa646b5ae in /home/chenxiangbo/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/lib/libtorch.so) frame #6: <unknown function> + 0x1c0155a (0x7f4aa69db55a in /home/chenxiangbo/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/lib/libtorch.so) frame #7: <unknown function> + 0x1c06023 (0x7f4aa69e0023 in /home/chenxiangbo/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/lib/libtorch.so) frame #8: <unknown function> + 0x3820d1a (0x7f4aa85fad1a in /home/chenxiangbo/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/lib/libtorch.so) frame #9: <unknown function> + 0x1c06023 (0x7f4aa69e0023 in /home/chenxiangbo/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/lib/libtorch.so) frame #10: at::Tensor::index(c10::ArrayRef<at::Tensor>) const + 0x191 (0x7f4ad50ea931 in /home/chenxiangbo/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/lib/libtorch_python.so) frame #11: nms_cuda(at::Tensor, float) + 0x84f (0x7f4a7b0e3bbc in /home/chenxiangbo/D2Det/demo/../mmdet/ops/nms/nms_cuda.cpython-37m-x86_64-linux-gnu.so) frame #12: nms(at::Tensor const&, float) + 0xee (0x7f4a7b0d217e in /home/chenxiangbo/D2Det/demo/../mmdet/ops/nms/nms_cuda.cpython-37m-x86_64-linux-gnu.so) frame #13: <unknown function> + 0x3c75b (0x7f4a7b0e175b in /home/chenxiangbo/D2Det/demo/../mmdet/ops/nms/nms_cuda.cpython-37m-x86_64-linux-gnu.so) frame #14: <unknown function> + 0x38ee0 (0x7f4a7b0ddee0 in /home/chenxiangbo/D2Det/demo/../mmdet/ops/nms/nms_cuda.cpython-37m-x86_64-linux-gnu.so) <omitting python frames>

From your paper, I know that the Discriminative RoI pooling is specifically designed for classification. But after reading your code, I've seen the Discriminative RoI pooling used in SingleRoIExtractor. In other words, it operates on both classification and regression tasks. Am I right in understanding this? In addition, in the ablation experiment, DRP is significantly improved compared to AP of the baseline. Is the increase mainly derived from Defromable ROI pooling or Adaptive pooling? Look forward to your answers.

Hello, thank you very much for your reply to the previous question, but I still get an error while compiling with pip install -v -e .
I have performed the following two installation steps, but each has the same problem:
method one:
1.conda create -n mmd1 Python=3.6
2.conda activate mmd1
3.conda install pytorch==1.1.0 torchvision==0.3.0 cudatoolkit=10.0 -c pytorch
4.pip install mmcv==0.4.3
5.pycocotools
cd /home/10102010/qy/detectron2_ubuntu/cocoapi-master/PythonAPI
python setup.py build_ext install
6.cd /home/10102010/qy/Projects_qy/D2Det-master
pip install -r requirements/build.txt
pip install -v -e . or python setup.py develop

Method Two:
1.conda create -n mmd Python=3.6
2.conda activate mmd
3.conda install pytorch==1.1.0 torchvision==0.3.0 cudatoolkit=9.0 -c pytorch
4.pip install mmcv==0.4.3
5. Install pycocotools
cd /home/10102010/qy/detectron2_ubuntu/cocoapi-master/PythonAPI
python setup.py build_ext install
6.cd /home/10102010/qy/Projects_qy/D2Det-master directory
pip install -r requirements/build.txt
pip install -v -e . or python setup.py develop

issue:
Running command python setup.py egg_info
which: no nvcc in (/home/10102010/qy/mm2.3envir/anaconda3/cd../envs/mmd/bin:/cm/shared/apps/gcc5/5.5.0/bin:/cm/shared/apps/slurm/current/sbodules/4.4.0//bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/sbin:/cm/local/apps/environment-modules/4.4.0/bin:/home/10102010
Traceback (most recent call last):
File "", line 1, in
File "/home/10102010/qy/Projects_qy/D2Det-master/setup.py", line 228, in
sources=['src/compiling_info.cpp']),
File "/home/10102010/qy/Projects_qy/D2Det-master/setup.py", line 97, in make_cuda_ext
raise EnvironmentError('CUDA is required to compile MMDetection!')
OSError: CUDA is required to compile MMDetection!
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
Exception information:
Traceback (most recent call last):
File "/home/10102010/qy/mm2.3envir/anaconda3/cd../envs/mmd/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 228, in _main
status = self.run(options, args)
File "/home/10102010/qy/mm2.3envir/anaconda3/cd../envs/mmd/lib/python3.6/site-packages/pip/_internal/cli/req_command.py", line 182, in wrappe
return func(self, options, args)
File "/home/10102010/qy/mm2.3envir/anaconda3/cd../envs/mmd/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 324, in run
reqs, check_supported_wheels=not options.target_dir
File "/home/10102010/qy/mm2.3envir/anaconda3/cd../envs/mmd/lib/python3.6/site-packages/pip/_internal/resolution/legacy/resolver.py", line 183
discovered_reqs.extend(self._resolve_one(requirement_set, req))
File "/home/10102010/qy/mm2.3envir/anaconda3/cd../envs/mmd/lib/python3.6/site-packages/pip/_internal/resolution/legacy/resolver.py", line 388
abstract_dist = self._get_abstract_dist_for(req_to_install)
File "/home/10102010/qy/mm2.3envir/anaconda3/cd../envs/mmd/lib/python3.6/site-packages/pip/_internal/resolution/legacy/resolver.py", line 326
return self.preparer.prepare_editable_requirement(req)
File "/home/10102010/qy/mm2.3envir/anaconda3/cd../envs/mmd/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 526, in pre
req, self.req_tracker, self.finder, self.build_isolation,
File "/home/10102010/qy/mm2.3envir/anaconda3/cd../envs/mmd/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 91, in _get
abstract_dist.prepare_distribution_metadata(finder, build_isolation)
File "/home/10102010/qy/mm2.3envir/anaconda3/cd../envs/mmd/lib/python3.6/site-packages/pip/_internal/distributions/sdist.py", line 40, in pre
self.req.prepare_metadata()
File "/home/10102010/qy/mm2.3envir/anaconda3/cd../envs/mmd/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 555, in prepar
self.metadata_directory = self._generate_metadata()
File "/home/10102010/qy/mm2.3envir/anaconda3/cd../envs/mmd/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 535, in _gener
details=self.name or "from {}".format(self.link)
File "/home/10102010/qy/mm2.3envir/anaconda3/cd../envs/mmd/lib/python3.6/site-packages/pip/internal/operations/build/metadata_legacy.py", li
command_desc='python setup.py egg_info',
File "/home/10102010/qy/mm2.3envir/anaconda3/cd../envs/mmd/lib/python3.6/site-packages/pip/internal/utils/subprocess.py", line 242, in call
raise InstallationError(exc_msg)
pip.internal.exceptions.InstallationError: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command ou
Removed file:///home/10102010/qy/Projects_qy/D2Det-master from build tracker '/tmp/pip-req-tracker-0ujmvsq'
Removed build tracker: '/tmp/pip-req-tracker-0ujmvsq'

hello,i can not find the operation of the Discriminative RoI Pooling in the code.

Hello, I see that your code has been modified in deform_conv.py relative to deform_conv.py in mmdetection. Would you like to know if you have rewritten mmdet\ops\dcn\src\deform_conv_cuda?

File "./demo/D2Det_demo.py", line 5, in
from mmdet.apis import init_detector, inference_detector, show_result_pyplot
File "/opt/coco/project/D2Det/demo/../mmdet/apis/init.py", line 1, in
from .inference import (async_inference_detector, inference_detector,
File "/opt/coco/project/D2Det/demo/../mmdet/apis/inference.py", line 11, in
from mmdet.core import get_classes
File "/opt/coco/project/D2Det/demo/../mmdet/core/init.py", line 7, in
from .post_processing import * # noqa: F401, F403
File "/opt/coco/project/D2Det/demo/../mmdet/core/post_processing/init.py", line 1, in
from .bbox_nms import multiclass_nms, multiclass_nms1
File "/opt/coco/project/D2Det/demo/../mmdet/core/post_processing/bbox_nms.py", line 3, in
from mmdet.ops.nms import nms_wrapper
File "/opt/coco/project/D2Det/demo/../mmdet/ops/init.py", line 7, in
from .nms import nms, soft_nms
File "/opt/coco/project/D2Det/demo/../mmdet/ops/nms/init.py", line 1, in
from .nms_wrapper import nms, soft_nms
File "/opt/coco/project/D2Det/demo/../mmdet/ops/nms/nms_wrapper.py", line 4, in
from . import nms_cpu, nms_cuda
ImportError: libtorch_cpu.so: cannot open shared object file: No such file or directory

I don't know how to solve it

Hello I want to ask one question about your paper. I'm a little confused about binary overlap prediction.
What is the specific operation mode of binary overlap prediction in dense local regression? Is the binary overlap prediction in front of or behind the dense local regression, or at the same time?

Excuse me, during my training of your D2Det instance model with iSAID dataset, cuda out of memory at 800 iterations. I cropped the original images into 800×800 pixels with stride 200 and used 1 Titan GPU with a batchsize of 2. Could you please give me some advice ?

@JialeCao001
Thank you for sharing such a great work! Can you please public the weights you had trained on UAVDT?
Thanks

I've test the model you provided on coco val2017, but I don't know how to test the model on coco test2017.could you tell me how to do that? I test the model on coco val2017 with below command line whose result is similar to yours.

./tools/dist_test.sh ./configs/D2Det/D2Det_instance_r101_fpn_2x.py ./weights/D2Det-instance-res101.pth 8 --eval bbox segm
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
loading annotations into memory...
loading annotations into memory...
loading annotations into memory...
Done (t=0.66s)
creating index...
index created!
Done (t=0.71s)
creating index...
index created!
loading annotations into memory...
loading annotations into memory...
loading annotations into memory...
loading annotations into memory...
loading annotations into memory...
Done (t=0.82s)
creating index...
index created!
Done (t=0.76s)
creating index...
Done (t=0.82s)
creating index...
index created!
index created!
Done (t=0.82s)
creating index...
Done (t=0.83s)
creating index...
Done (t=0.82s)
creating index...
index created!
index created!
index created!
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 5000/5000, 41.7 task/s, elapsed: 120s, ETA:     0s
Evaluating bbox...
Loading and preparing results...
DONE (t=1.23s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=50.66s).
Accumulating evaluation results...
DONE (t=6.82s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.440
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.629
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.476
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.267
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.479
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.583
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.354
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.557
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.580
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.375
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.620
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.756

Evaluating segm...
Loading and preparing results...
DONE (t=1.98s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *segm*
DONE (t=52.47s).
Accumulating evaluation results...
DONE (t=6.66s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.397
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.604
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.431
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.224
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.435
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.546
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.322
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.493
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.513
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.314
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.550
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.682

if I want to test the model on coco test2017, how could I do that? should I add the code in the file configs/D2Det/D2Det_instance_r101_fpn_2x.py and change them to the code next to that?
before:

ann_file=data_root + 'annotations/instances_val2017.json',
        img_prefix=data_root + 'val2017/',

after:

# ann_file=data_root + 'annotations/instances_val2017.json',
        # img_prefix=data_root + 'val2017/',
        ann_file=data_root + 'annotations/image_info_test-dev2017.json',
        img_prefix=data_root + 'test2017/',

I'm not sure if that's right. but after I do that, I run the command ./tools/dist_test.sh ./configs/D2Det/D2Det_instance_r101_fpn_2x.py ./weights/D2Det-instance-res101.pth 8 --eval bbox segm and I get the result as following, I don't know why. Could please tell me what mistakes I made and point them out?thank you

Evaluate annotation type *bbox*
DONE (t=74.21s).
Accumulating evaluation results...
DONE (t=13.37s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000

Evaluating segm...
Loading and preparing results...
DONE (t=6.91s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *segm*
DONE (t=76.09s).
Accumulating evaluation results...
DONE (t=13.09s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000

① Own data set is converted from voc to coco
② During the test, when the progress bar is printed out, the program is stuck, as shown in the figure

After my own analysis, the program is stuck in the function single_gpu_test as the following command line
for i, data in enumerate(data_loader):

enumerate(data_loader) cannot proceed smoothly

Do you know?

Please help me! Please!!!!
@JialeCao001

My environment is as follows:

I have read your paper. But the operation in the picture can not give a real reason for your effect,. In the case of no pre-defined box, the regression range of box is very wide. But your binary overlap prediction is a good tool. Your operation of picture may return a worse box, can you explain the real reason why it works better ?

I read that the source code is used：
“Conv2d(576, 4, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))”

But how do these four channels represent k*k dense position offsets?

As shown in the figure below, RPN has generated many proposals, and, due to the existence of anchors, the sizes of these proposals should be different. In the faster rcnn, the sizes are unified after roi Pooling. There is no ROI Pooling here. Why RPN How about direct output as k*k?

@JialeCao001
Thank you for sharing such a great work!
I have seen on your paper that you guys also made a comparison on iSAID dataset. Can you please public the weights you had trained on that dataset ?
Thanks in advance

Hi:
感谢开源，当我使用inference_demo.ipynb代码是 checkpoint 的模型D2Det-instance-res101.pth 会报以下的错误"d2det\mmdet\apis\inference.py", line 169, in show_result
img[mask] = img[mask] * 0.5 + color_mask * 0.5
IndexError: boolean index did not match indexed array along dimension 2; dimension is 3 but corresponding boolean dimension is 19"

Firstly, thank you for such a greate work. But I have a little problem when I reading your code. In the paper I know that the Discriminative RoI pooling create the Weighted RoI feature with F~ =W(F)⊙ F. But In your code the Weighted RoI feature is created by return F.avg_pool2d(x * weight, kernel, stride, padding) / F.avg_pool2d(weight, kernel, stride, padding).
I don't know why do it divide by F.avg_pool2d(weight, kernel, stride, padding)

Thanks for your great work. In your code, mask_on== false indicates instance segmentation task while mask_on ==True indicates object detection.
In D2Det.py

In D2Det_head.py

But when mask_on==false, the two deconv layers are not available.

Hello, I got the following error while training D2Det v2.1

Traceback (most recent call last):
File "/home/10102010/qy/ubuntu_mmd2.3/D2Det-mmdet2.1-master/tools/train.py", line 13, in
File "/home/10102010/qy/ubuntu_mmd2.3/D2Det-mmdet2.1-master/tools/train.py", line 13, in
from mmdet.apis import set_random_seed, train_detector
from mmdet.apis import set_random_seed, train_detector
File "/home/10102010/qy/ubuntu_mmd2.3/D2Det-mmdet2.1-master/mmdet/apis/init.py", line 1, in
File "/home/10102010/qy/ubuntu_mmd2.3/D2Det-mmdet2.1-master/mmdet/apis/init.py", line 1, in
from .inference import (async_inference_detector, inference_detector,
from .inference import (async_inference_detector, inference_detector,
File "/home/10102010/qy/ubuntu_mmd2.3/D2Det-mmdet2.1-master/mmdet/apis/inference.py", line 9, in
File "/home/10102010/qy/ubuntu_mmd2.3/D2Det-mmdet2.1-master/mmdet/apis/inference.py", line 9, in
from mmdet.core import get_classes
from mmdet.core import get_classes
File "/home/10102010/qy/ubuntu_mmd2.3/D2Det-mmdet2.1-master/mmdet/core/init.py", line 2, in
File "/home/10102010/qy/ubuntu_mmd2.3/D2Det-mmdet2.1-master/mmdet/core/init.py", line 2, in
from .bbox import * # noqa: F401, F403
from .bbox import * # noqa: F401, F403
File "/home/10102010/qy/ubuntu_mmd2.3/D2Det-mmdet2.1-master/mmdet/core/bbox/init.py", line 7, in
File "/home/10102010/qy/ubuntu_mmd2.3/D2Det-mmdet2.1-master/mmdet/core/bbox/init.py", line 7, in
from .samplers import (BaseSampler, CombinedSampler,
from .samplers import (BaseSampler, CombinedSampler,
File "/home/10102010/qy/ubuntu_mmd2.3/D2Det-mmdet2.1-master/mmdet/core/bbox/samplers/init.py", line 9, in
File "/home/10102010/qy/ubuntu_mmd2.3/D2Det-mmdet2.1-master/mmdet/core/bbox/samplers/init.py", line 9, in
from .score_hlr_sampler import ScoreHLRSampler
from .score_hlr_sampler import ScoreHLRSampler
File "/home/10102010/qy/ubuntu_mmd2.3/D2Det-mmdet2.1-master/mmdet/core/bbox/samplers/score_hlr_sampler.py", line 3, in
File "/home/10102010/qy/ubuntu_mmd2.3/D2Det-mmdet2.1-master/mmdet/core/bbox/samplers/score_hlr_sampler.py", line 3, in
from mmdet.ops import nms_match
from mmdet.ops import nms_match
File "/home/10102010/qy/ubuntu_mmd2.3/D2Det-mmdet2.1-master/mmdet/ops/init.py", line 3, in
File "/home/10102010/qy/ubuntu_mmd2.3/D2Det-mmdet2.1-master/mmdet/ops/init.py", line 3, in
from .corner_pool import CornerPool
from .corner_pool import CornerPool
File "/home/10102010/qy/ubuntu_mmd2.3/D2Det-mmdet2.1-master/mmdet/ops/corner_pool/init.py", line 1, in
File "/home/10102010/qy/ubuntu_mmd2.3/D2Det-mmdet2.1-master/mmdet/ops/corner_pool/init.py", line 1, in
from . import corner_pool
from . import corner_pool
File "/home/10102010/qy/ubuntu_mmd2.3/D2Det-mmdet2.1-master/mmdet/ops/corner_pool/corner_pool.py", line 4, in
File "/home/10102010/qy/ubuntu_mmd2.3/D2Det-mmdet2.1-master/mmdet/ops/corner_pool/corner_pool.py", line 4, in
from . import corner_pool_ext
from . import corner_pool_ext
ImportError: cannot import name 'corner_pool_ext'
ImportError: cannot import name 'corner_pool_ext'
Traceback (most recent call last):
File "/home/10102010/qy/mm2.3envir/anaconda3/cd../envs/mmd21/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/home/10102010/qy/mm2.3envir/anaconda3/cd../envs/mmd21/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/10102010/qy/mm2.3envir/anaconda3/cd../envs/mmd21/lib/python3.6/site-packages/torch/distributed/launch.py", line 263, in
main()
File "/home/10102010/qy/mm2.3envir/anaconda3/cd../envs/mmd21/lib/python3.6/site-packages/torch/distributed/launch.py", line 259, in main
cmd=cmd)
subprocess.CalledProcessError: Command '['/home/10102010/qy/mm2.3envir/anaconda3/cd../envs/mmd21/bin/python', '-u', '/home/10102010/qy/ubuntu_mmd2.3/D2Det-mmdet2.1-master/tools/train.py', '--local_rank=1', '/home/10102010/qy/ubuntu_mmd2.3/D2Det-mmdet2.1-master/configs/d2det/D2Det_detection_r50_fpn_2x.py', '--launcher', 'pytorch', '--work-dir']' returned non-zero exit status 1.

Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.

hello ,

thank you for the repo.

when i train the model, i just got the error like
"Traceback (most recent call last):
File "train.py", line 15, in
from mmdet import version
File "/home/adasone/yeohun/D2Det/tools/../mmdet/init.py", line 1, in
from .version import version, short_version
ModuleNotFoundError: No module named 'mmdet.version'
"

how can i fix it ?

Thanks.

Thanks for sharing your research/code with us. Many appreciated.
I tried to run:
python tools/test.py configs/D2Det/D2Det_instance_r101_fpn_2x.py checkpoints/D2Det-instance-res101.pth --show
loading annotations into memory...
Done (t=0.50s)
creating index...
index created!
Traceback (most recent call last):
File "tools/test.py", line 175, in
main()
File "tools/test.py", line 136, in main
shuffle=False)
TypeError: build_dataloader() missing 1 required positional argument: 'samples_per_gpu'

How to solve the is problem?

I followed the tutorial to set the D2Det enviorment by the dockerfile and wanted to train custom dataset.
the code and the steps are following:
folder :D2Det docker build -t mmdetection docker/
make the container form image mmdetection ,and upload the data
docker run --gpus all -it --name mm -v {DIR}/D2Det-master/data/:/mmdetection/data mmdetection
check the packages
version:pytorch 1.3.0 mmdet 2.10 python 3.6

and docker don't have mmcv packages

mmcv version link

the mmdetection/mmdet not exist D2Det/mmdet/ops
so try to copy D2D to docker
docker cp D2Det-master mm:/
python setup.py develop
and run tools/train.py

ImportError: cannot import name 'deform_conv_cuda'

the floder /mmdet/ops/dcn

and floder /mmdet/ops/dcn/src

what happen about it ?how can I solve it ?maybe the problem about compile
Hope your answers

@JialeCao001
Hi,thank you for sharing such a great work!
Did you provide the code for the iSAID instance segmentation in this project？For example: How to load the iSAID data for traning model？

Thank you for your reply

Thanks for sharing your research and code with us. Many appreciated.
I tried to train your model with KITTI dataset, the following errors occur every time:

2021-03-01 09:19:43,961 - mmdet - INFO - Epoch [1][440/1856] lr: 5.067e-06, eta: 8:32:39,
time: 0.400, data_time: 0.100, memory: 4264, loss_rpn_cls: 0.3128, loss_rpn_bbox: 0.2060, loss_cls: 0.2960, acc: 96.9043, loss_reg: 0.2682, loss_mask: 0.6795, loss: 1.7624
2021-03-01 09:19:47,923 - mmdet - INFO - Epoch [1][450/1856] lr: 5.177e-06, eta: 8:32:01,
time: 0.396, data_time: 0.095, memory: 4264, loss_rpn_cls: 0.3051, loss_rpn_bbox: 0.1585, loss_cls: 0.2783, acc: 96.7188, loss_reg: 0.2841, loss_mask: 0.6796, loss: 1.7056
2021-03-01 09:19:51,800 - mmdet - INFO - Epoch [1][460/1856] lr: 5.286e-06, eta: 8:31:11,
time: 0.388, data_time: 0.087, memory: 4264, loss_rpn_cls: 0.2838, loss_rpn_bbox: 0.1638, loss_cls: 0.2632, acc: 96.6406, loss_reg: 0.2799, loss_mask: 0.6799, loss: 1.6707
Traceback (most recent call last):
File "train.py", line 161, in
main()
File "train.py", line 157, in main
meta=meta)
File "/work_dirs/D2Det_mmdet2.1/mmdet/apis/train.py", line 179, in train_detector
runner.run(data_loaders, cfg.workflow, cfg.total_epochs)
File "/opt/conda/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 122, in run
epoch_runner(data_loaders[i], *kwargs)
File "/opt/conda/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 43, in train
self.call_hook('after_train_iter')
File "/opt/conda/lib/python3.7/site-packages/mmcv/runner/base_runner.py", line 282, in call_hook
getattr(hook, fn_name)(self) File "/opt/conda/lib/python3.7/site-packages/mmcv/runner/hooks/optimizer.py", line 21, in after_train_iter runner.outputs['loss'].backward()
File "/opt/conda/lib/python3.7/site-packages/torch/tensor.py", line 198, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/opt/conda/lib/python3.7/site-packages/torch/autograd/init.py", line 100, in backward allow_unreachable=True) # allow_unreachable flag
RuntimeError: shape mismatch: value tensor of shape [8, 256, 7, 7] cannot be broadcast to indexing result of shape [9, 256, 7, 7] (make_index_put_iterator at /opt/conda/conda-bld/pytorch_1587428398394/work/aten/src/ATen/native/TensorAdvancedIndexing.cpp:215)frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x4e (0x7f43abb90b5e in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: at::native::index_put_impl(at::Tensor&, c10::ArrayRefat::Tensor, at::Tensor const&, bool, bool) + 0x712 (0x7f43d38d0b82 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #2: + 0xee23de (0x7f43d3c543de in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)frame #3: at::native::index_put_(at::Tensor&, c10::ArrayRefat::Tensor, at::Tensor const&, bool) + 0x135 (0x7f43d38c0255 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #4: + 0xee210e (0x7f43d3c5410e in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)frame #5: + 0x288fa88 (0x7f43d5601a88 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)frame #6: torch::autograd::generated::IndexPutBackward::apply(std::vector<at::Tensor, std::allocatorat::Tensor >&&) + 0x251 (0x7f43d53cc201 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)frame #7: + 0x2ae8215 (0x7f43d585a215 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #8: torch::autograd::Engine::evaluate_function(std::shared_ptrtorch::autograd::GraphTask&, torch::autograd::Node, torch::autograd::InputBuffer&) + 0x16f3 (0x7f43d5857513 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #9: torch::autograd::Engine::thread_main(std::shared_ptrtorch::autograd::GraphTask const&, bool) + 0x3d2 (0x7f43d58582f2 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #10: torch::autograd::Engine::thread_init(int) + 0x39 (0x7f43d5850969 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #11: torch::autograd::python::PythonEngine::thread_init(int) + 0x38 (0x7f43d8b97558 in
/opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #12: + 0xc819d (0x7f43db5ff19d in /opt/conda/lib/python3.7/site-packages/torch/lib/../../../.././libstdc++.so.6)
frame #13: + 0x76db (0x7f43fbfdf6db in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #14: clone + 0x3f (0x7f43fbd0888f in /lib/x86_64-linux-gnu/libc.so.6)

This error occurs randomly in different iteration. In addition, every time the error occured, the first dimension size of the tensor [8, 256, 7, 7] is different.
Do you know the possible reasons for this error?

I have trained a model on isaid dataset. Now I want to test it's performance on test set using test.py, and have got a .pkl file after testing. While ISAID test sever needs to upload a .json file, what should I do? Should I write some code to transform the file format?

Hi, I have been trying to train D2Det with my custom dataset which is already in the COCO format. My dataset has 7 classes so I have edit num_classes = 7 in the bbox_head dict(). The other things I have changed it the data dict() for my dataset.
However, I always have this CUDA error:

/home/phong/miniconda3/envs/d2/bin/python /home/phong/data/Zalo2020/D2Det/tools/train.py ../configs/D2Det/zalo.py
2020-11-05 00:57:30,306 - mmdet - INFO - Environment info:
------------------------------------------------------------
sys.platform: linux
Python: 3.6.12 |Anaconda, Inc.| (default, Sep  8 2020, 23:10:56) [GCC 7.3.0]
CUDA available: True
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 10.1, V10.1.105
GPU 0: GeForce RTX 2080 Ti
GCC: gcc (Ubuntu 8.4.0-3ubuntu2) 8.4.0
PyTorch: 1.2.0
PyTorch compiling details: PyTorch built with:
  - GCC 7.3
  - Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v0.18.1 (Git Hash 7de7e5d02bf687f971e7668963649728356e0c20)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - NNPACK is enabled
  - CUDA Runtime 10.0
  - NVCC architecture flags: -gencode;arch=compute_35,code=sm_35;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_50,code=compute_50
  - CuDNN 7.6.2
  - Magma 2.5.1
  - Build settings: BLAS=MKL, BUILD_NAMEDTENSOR=OFF, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -DUSE_FBGEMM -DUSE_QNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Wno-stringop-overflow, DISABLE_NUMA=1, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=True, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, 

TorchVision: 0.4.0
OpenCV: 4.4.0
MMCV: 0.4.3
MMDetection: 1.1.0+a76781a
MMDetection Compiler: GCC 8.4
MMDetection CUDA Compiler: 10.1
------------------------------------------------------------

2020-11-05 00:57:30,306 - mmdet - INFO - Distributed training: False
2020-11-05 00:57:30,306 - mmdet - INFO - Config:
/home/phong/data/Zalo2020/D2Det/configs/D2Det/zalo.py
# model settings
model = dict(
    type='D2Det',
    pretrained='torchvision://resnet50',
    backbone=dict(
        type='ResNet',
        depth=50,
        num_stages=4,
        out_indices=(0, 1, 2, 3),
        frozen_stages=1,
        style='pytorch'),
    neck=dict(
        type='FPN',
        in_channels=[256, 512, 1024, 2048],
        out_channels=256,
        num_outs=5),
    rpn_head=dict(
        type='RPNHead',
        in_channels=256,
        feat_channels=256,
        anchor_scales=[8],
        anchor_ratios=[0.5, 1.0, 2.0],
        anchor_strides=[4, 8, 16, 32, 64],
        target_means=[.0, .0, .0, .0],
        target_stds=[1.0, 1.0, 1.0, 1.0],
        loss_cls=dict(
            type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0)),
    bbox_roi_extractor=dict(
        type='SingleRoIExtractor',
        roi_layer=dict(
            type='DeformRoIPoolingPack',
            out_size=7,
            sample_per_part=2,
            out_channels=256,
            no_trans=False,
            group_size=1,
            trans_std=0.1),
        out_channels=256,
        featmap_strides=[4, 8, 16, 32]),
    bbox_head=dict(
        type='SharedFCBBoxHead',
        with_reg=False,
        num_fcs=2,
        in_channels=256,
        fc_out_channels=1024,
        roi_feat_size=7,
        num_classes=7,
        target_means=[0., 0., 0., 0.],
        target_stds=[0.1, 0.1, 0.2, 0.2],
        reg_class_agnostic=False,
        loss_cls=dict(
            type='CrossEntropyLoss', use_sigmoid=False, loss_weight=2.0)),
    reg_roi_extractor=dict(
        type='SingleRoIExtractor',
        roi_layer=dict(type='RoIAlign', out_size=14, sample_num=2),
        out_channels=256,
        featmap_strides=[4, 8, 16, 32]),
    D2Det_head=dict(
        type='D2DetHead',
        num_convs=8,
        in_channels=256,
        norm_cfg=dict(type='GN', num_groups=36),
        MASK_ON=False))
# model training and testing settings
train_cfg = dict(
    rpn=dict(
        assigner=dict(
            type='MaxIoUAssigner',
            pos_iou_thr=0.7,
            neg_iou_thr=0.3,
            min_pos_iou=0.3,
            ignore_iof_thr=-1),
        sampler=dict(
            type='RandomSampler',
            num=256,
            pos_fraction=0.5,
            neg_pos_ub=-1,
            add_gt_as_proposals=False),
        allowed_border=0,
        pos_weight=-1,
        debug=False),
    rpn_proposal=dict(
        nms_across_levels=False,
        nms_pre=2000,
        nms_post=2000,
        max_num=2000,
        nms_thr=0.7,
        min_bbox_size=0),
    rcnn=dict(
        assigner=dict(
            type='MaxIoUAssigner',
            pos_iou_thr=0.5,
            neg_iou_thr=0.5,
            min_pos_iou=0.5,
            ignore_iof_thr=-1),
        sampler=dict(
            type='RandomSampler',
            num=512,
            pos_fraction=0.25,
            neg_pos_ub=-1,
            add_gt_as_proposals=True),
        pos_radius=1,
        pos_weight=-1,
        max_num_reg=192,
        debug=False))
test_cfg = dict(
    rpn=dict(
        nms_across_levels=False,
        nms_pre=1000,
        nms_post=1000,
        max_num=1000,
        nms_thr=0.7,
        min_bbox_size=0),
    rcnn=dict(
        score_thr=0.03, nms=dict(type='nms', iou_thr=0.5), max_per_img=125))
# dataset settings
dataset_type = 'CocoDataset'
data_root = '../data/zalo/za_traffic_2020/traffic_train/'
img_norm_cfg = dict(
    mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
train_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(type='LoadAnnotations', with_bbox=True),
    dict(type='Resize', img_scale=(1333, 800), keep_ratio=True),
    dict(type='RandomFlip', flip_ratio=0.5),
    dict(type='Normalize', **img_norm_cfg),
    dict(type='Pad', size_divisor=32),
    dict(type='DefaultFormatBundle'),
    dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels']),
]
test_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(
        type='MultiScaleFlipAug',
        img_scale=(1333, 800),
        flip=False,
        transforms=[
            dict(type='Resize', keep_ratio=True),
            dict(type='RandomFlip'),
            dict(type='Normalize', **img_norm_cfg),
            dict(type='Pad', size_divisor=32),
            dict(type='ImageToTensor', keys=['img']),
            dict(type='Collect', keys=['img']),
        ])
]
data = dict(
    imgs_per_gpu=1,
    workers_per_gpu=1,
    train=dict(
        type=dataset_type,
        ann_file=data_root + 'train_traffic_sign_dataset.json',
        img_prefix=data_root + 'images/',
        pipeline=train_pipeline),
    val=dict(
        type=dataset_type,
        ann_file=data_root + 'train_traffic_sign_dataset.json',
        img_prefix=data_root + 'images_val/',
        pipeline=test_pipeline),
    test=dict(
        type=dataset_type,
        ann_file=data_root + 'train_traffic_sign_dataset.json',
        img_prefix=data_root + 'images_val/',
        pipeline=test_pipeline))
evaluation = dict(interval=1, metric='bbox')
# optimizer
optimizer = dict(type='SGD', lr=0.01, momentum=0.9, weight_decay=0.0001)
optimizer_config = dict(grad_clip=None)
# learning policy
lr_config = dict(
    policy='step',
    warmup='linear',
    warmup_iters=1000,
    warmup_ratio=1.0 / 80,
    step=[20, 23])
checkpoint_config = dict(interval=1)
# yapf:disable
log_config = dict(
    interval=50,
    hooks=[
        dict(type='TextLoggerHook'),
        # dict(type='TensorboardLoggerHook')
    ])
# yapf:enable
# runtime settings
total_epochs = 24
dist_params = dict(backend='nccl')
log_level = 'INFO'
work_dir = './work_dirs/D2Det_detection_r50_fpn_2x'
load_from = None
resume_from = None
workflow = [('train', 1)]

2020-11-05 00:57:30,641 - mmdet - INFO - load model from: torchvision://resnet50
2020-11-05 00:57:30,781 - mmdet - WARNING - The model and loaded state dict do not match exactly

unexpected key in source state_dict: fc.weight, fc.bias

loading annotations into memory...
Done (t=0.05s)
creating index...
index created!
2020-11-05 00:57:33,651 - mmdet - INFO - Start running, host: phong@phong-Server, work_dir: /home/phong/data/Zalo2020/D2Det/tools/work_dirs/D2Det_detection_r50_fpn_2x
2020-11-05 00:57:33,651 - mmdet - INFO - workflow: [('train', 1)], max: 24 epochs
Traceback (most recent call last):
  File "/home/phong/data/Zalo2020/D2Det/tools/train.py", line 144, in <module>
    main()
  File "/home/phong/data/Zalo2020/D2Det/tools/train.py", line 140, in main
    meta=meta)
  File "/home/phong/data/Zalo2020/D2Det/tools/../mmdet/apis/train.py", line 111, in train_detector
    meta=meta)
  File "/home/phong/data/Zalo2020/D2Det/tools/../mmdet/apis/train.py", line 233, in _non_dist_train
    runner.run(data_loaders, cfg.workflow, cfg.total_epochs)
  File "/home/phong/miniconda3/envs/d2/lib/python3.6/site-packages/mmcv/runner/runner.py", line 359, in run
    epoch_runner(data_loaders[i], **kwargs)
  File "/home/phong/miniconda3/envs/d2/lib/python3.6/site-packages/mmcv/runner/runner.py", line 263, in train
    self.model, data_batch, train_mode=True, **kwargs)
  File "/home/phong/data/Zalo2020/D2Det/tools/../mmdet/apis/train.py", line 75, in batch_processor
    losses = model(**data)
  File "/home/phong/miniconda3/envs/d2/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/phong/miniconda3/envs/d2/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/home/phong/miniconda3/envs/d2/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/phong/data/Zalo2020/D2Det/tools/../mmdet/core/fp16/decorators.py", line 49, in new_func
    return old_func(*args, **kwargs)
  File "/home/phong/data/Zalo2020/D2Det/tools/../mmdet/models/detectors/base.py", line 137, in forward
    return self.forward_train(img, img_meta, **kwargs)
  File "/home/phong/data/Zalo2020/D2Det/tools/../mmdet/models/detectors/D2Det.py", line 157, in forward_train
    x[:self.reg_roi_extractor.num_inputs], pos_rois)
  File "/home/phong/miniconda3/envs/d2/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/phong/data/Zalo2020/D2Det/tools/../mmdet/core/fp16/decorators.py", line 127, in new_func
    return old_func(*args, **kwargs)
  File "/home/phong/data/Zalo2020/D2Det/tools/../mmdet/models/roi_extractors/single_level.py", line 96, in forward
    target_lvls = self.map_roi_levels(rois, num_levels)
  File "/home/phong/data/Zalo2020/D2Det/tools/../mmdet/models/roi_extractors/single_level.py", line 70, in map_roi_levels
    (rois[:, 3] - rois[:, 1] + 1) * (rois[:, 4] - rois[:, 2] + 1))
RuntimeError: CUDA error: device-side assert triggered
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:56: void ClassNLLCriterion_updateOutput_no_reduce_kernel(int, THCDeviceTensor<Dtype, 2, int, DefaultPtrTraits>, THCDeviceTensor<long, 1, int, DefaultPtrTraits>, THCDeviceTensor<Dtype, 1, int, DefaultPtrTraits>, Dtype *, int, int) [with Dtype = float]: block: [0,0,0], thread: [0,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:56: void ClassNLLCriterion_updateOutput_no_reduce_kernel(int, THCDeviceTensor<Dtype, 2, int, DefaultPtrTraits>, THCDeviceTensor<long, 1, int, DefaultPtrTraits>, THCDeviceTensor<Dtype, 1, int, DefaultPtrTraits>, Dtype *, int, int) [with Dtype = float]: block: [0,0,0], thread: [1,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:56: void ClassNLLCriterion_updateOutput_no_reduce_kernel(int, THCDeviceTensor<Dtype, 2, int, DefaultPtrTraits>, THCDeviceTensor<long, 1, int, DefaultPtrTraits>, THCDeviceTensor<Dtype, 1, int, DefaultPtrTraits>, Dtype *, int, int) [with Dtype = float]: block: [0,0,0], thread: [2,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:56: void ClassNLLCriterion_updateOutput_no_reduce_kernel(int, THCDeviceTensor<Dtype, 2, int, DefaultPtrTraits>, THCDeviceTensor<long, 1, int, DefaultPtrTraits>, THCDeviceTensor<Dtype, 1, int, DefaultPtrTraits>, Dtype *, int, int) [with Dtype = float]: block: [0,0,0], thread: [3,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: device-side assert triggered (insert_events at /pytorch/c10/cuda/CUDACachingAllocator.cpp:569)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x33 (0x7f920ac67273 in /home/phong/miniconda3/envs/d2/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x10f1e (0x7f920ae9ef1e in /home/phong/miniconda3/envs/d2/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)
frame #2: <unknown function> + 0x146b1 (0x7f920aea26b1 in /home/phong/miniconda3/envs/d2/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10::TensorImpl::release_resources() + 0x74 (0x7f920ac55354 in /home/phong/miniconda3/envs/d2/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #4: <unknown function> + 0x1c6b44 (0x7f9254ec5b44 in /home/phong/miniconda3/envs/d2/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #5: <unknown function> + 0x40686b (0x7f925510586b in /home/phong/miniconda3/envs/d2/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #6: <unknown function> + 0x406896 (0x7f9255105896 in /home/phong/miniconda3/envs/d2/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #7: <unknown function> + 0x19b9fe (0x55b4d8ad99fe in /home/phong/miniconda3/envs/d2/bin/python)
frame #8: <unknown function> + 0xf2da8 (0x55b4d8a30da8 in /home/phong/miniconda3/envs/d2/bin/python)
frame #9: <unknown function> + 0xf2ec7 (0x55b4d8a30ec7 in /home/phong/miniconda3/envs/d2/bin/python)
frame #10: <unknown function> + 0xf2787 (0x55b4d8a30787 in /home/phong/miniconda3/envs/d2/bin/python)
frame #11: <unknown function> + 0xf2617 (0x55b4d8a30617 in /home/phong/miniconda3/envs/d2/bin/python)
frame #12: <unknown function> + 0xf262d (0x55b4d8a3062d in /home/phong/miniconda3/envs/d2/bin/python)
frame #13: <unknown function> + 0xf262d (0x55b4d8a3062d in /home/phong/miniconda3/envs/d2/bin/python)
frame #14: <unknown function> + 0xf262d (0x55b4d8a3062d in /home/phong/miniconda3/envs/d2/bin/python)
frame #15: <unknown function> + 0xf262d (0x55b4d8a3062d in /home/phong/miniconda3/envs/d2/bin/python)
frame #16: <unknown function> + 0xf262d (0x55b4d8a3062d in /home/phong/miniconda3/envs/d2/bin/python)
frame #17: <unknown function> + 0xf262d (0x55b4d8a3062d in /home/phong/miniconda3/envs/d2/bin/python)
frame #18: <unknown function> + 0xf262d (0x55b4d8a3062d in /home/phong/miniconda3/envs/d2/bin/python)
frame #19: <unknown function> + 0xf262d (0x55b4d8a3062d in /home/phong/miniconda3/envs/d2/bin/python)
frame #20: PyDict_SetItem + 0x3da (0x55b4d8a774ba in /home/phong/miniconda3/envs/d2/bin/python)
frame #21: PyDict_SetItemString + 0x4f (0x55b4d8a7e4df in /home/phong/miniconda3/envs/d2/bin/python)
frame #22: PyImport_Cleanup + 0x99 (0x55b4d8ae3d49 in /home/phong/miniconda3/envs/d2/bin/python)
frame #23: Py_FinalizeEx + 0x61 (0x55b4d8b4e061 in /home/phong/miniconda3/envs/d2/bin/python)
frame #24: Py_Main + 0x35e (0x55b4d8b583ae in /home/phong/miniconda3/envs/d2/bin/python)
frame #25: main + 0xee (0x55b4d8a2243e in /home/phong/miniconda3/envs/d2/bin/python)
frame #26: __libc_start_main + 0xf3 (0x7f9264de10b3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #27: <unknown function> + 0x1c3d0b (0x55b4d8b01d0b in /home/phong/miniconda3/envs/d2/bin/python)


Process finished with exit code 134 (interrupted by signal 6: SIGABRT)

Is this something related to the annotations that I am using? One thing I have noticed is that in COCO dataset the bounding box information is float numbers but mine is int numbers. Would it be a possible error ?

Hi @JialeCao001,

Thanks for sharing this amazing work.

I have a question regarding the installation steps. Do we need to pull and build the latest code from "https://github.com/open-mmlab/mmdetection" or would it be sufficient to build the "mmdet" folder present within this repository.

I noticed that there are some latest changes in the "mmdetection" repository which are not compatible with D2Det.

Thanks

When I used the D2Det_detection_r101_fpn_dcn_2x configuration file, there was the following error:

2020-09-24 15:18:39,295 - mmdet - INFO - load model from: torchvision://resnet101
2020-09-24 15:18:39,669 - mmdet - WARNING - The model and loaded state dict do not match exactly

unexpected key in source state_dict: fc.weight, fc.bias

missing keys in source state_dict: layer2.0.conv2.conv_offset.weight, layer2.0.conv2.conv_offset.bias, layer2.1.conv2.conv_offset.weight, layer2.1.conv2.conv_offset.bias, layer2.2.conv2.conv_offset.weight, layer2.2.conv2.conv_offset.bias, layer2.3.conv2.conv_offset.weight, layer2.3.conv2.conv_offset.bias, layer3.0.conv2.conv_offset.weight, layer3.0.conv2.conv_offset.bias, layer3.1.conv2.conv_offset.weight, layer3.1.conv2.conv_offset.bias, layer3.2.conv2.conv_offset.weight, layer3.2.conv2.conv_offset.bias, layer3.3.conv2.conv_offset.weight, layer3.3.conv2.conv_offset.bias, layer3.4.conv2.conv_offset.weight, layer3.4.conv2.conv_offset.bias, layer3.5.conv2.conv_offset.weight, layer3.5.conv2.conv_offset.bias, layer3.6.conv2.conv_offset.weight, layer3.6.conv2.conv_offset.bias, layer3.7.conv2.conv_offset.weight, layer3.7.conv2.conv_offset.bias, layer3.8.conv2.conv_offset.weight, layer3.8.conv2.conv_offset.bias, layer3.9.conv2.conv_offset.weight, layer3.9.conv2.conv_offset.bias, layer3.10.conv2.conv_offset.weight, layer3.10.conv2.conv_offset.bias, layer3.11.conv2.conv_offset.weight, layer3.11.conv2.conv_offset.bias, layer3.12.conv2.conv_offset.weight, layer3.12.conv2.conv_offset.bias, layer3.13.conv2.conv_offset.weight, layer3.13.conv2.conv_offset.bias, layer3.14.conv2.conv_offset.weight, layer3.14.conv2.conv_offset.bias, layer3.15.conv2.conv_offset.weight, layer3.15.conv2.conv_offset.bias, layer3.16.conv2.conv_offset.weight, layer3.16.conv2.conv_offset.bias, layer3.17.conv2.conv_offset.weight, layer3.17.conv2.conv_offset.bias, layer3.18.conv2.conv_offset.weight, layer3.18.conv2.conv_offset.bias, layer3.19.conv2.conv_offset.weight, layer3.19.conv2.conv_offset.bias, layer3.20.conv2.conv_offset.weight, layer3.20.conv2.conv_offset.bias, layer3.21.conv2.conv_offset.weight, layer3.21.conv2.conv_offset.bias, layer3.22.conv2.conv_offset.weight, layer3.22.conv2.conv_offset.bias, layer4.0.conv2.conv_offset.weight, layer4.0.conv2.conv_offset.bias, layer4.1.conv2.conv_offset.weight, layer4.1.conv2.conv_offset.bias, layer4.2.conv2.conv_offset.weight, layer4.2.conv2.conv_offset.bias

Is it a problem of network structure?

it seems that the mask_scores = 0.3 * mask_iou + bbox_score

Excuse me, how can I realize the training of D2Det model with iSAID data set

Like this:

I can't find voc_eval.py or coco_eval.py. How do you calculate mAP?

我在利用iSAID数据训练D2Det的时候，一开始能训练起来，但是消耗的显存会随着训练历程产生波动，很容易出现out of memory的情况。请问这个是什么情况？即使我把resize的尺寸改小也会出现训练一段时间之后突然显存不足的情况。麻烦大家分析一下。

I download your pretrained model and test it with demo picture. But I got results that one object was recognized as many objects. How could I deal with it so that I could only get one?

Welcome update to OpenMMLab 2.0

I am Vansin, the technical operator of OpenMMLab. In September of last year, we announced the release of OpenMMLab 2.0 at the World Artificial Intelligence Conference in Shanghai. We invite you to upgrade your algorithm library to OpenMMLab 2.0 using MMEngine, which can be used for both research and commercial purposes. If you have any questions, please feel free to join us on the OpenMMLab Discord at https://discord.gg/amFNsyUBvm or add me on WeChat (van-sin) and I will invite you to the OpenMMLab WeChat group.

Here are the OpenMMLab 2.0 repos branches:

Attention: please create a new virtual environment for OpenMMLab 2.0.

I need to install the mmdetection? or just download the D2Det code and install the env?

Hello, thank you for opening up your code

I am trying to port your code to mmdetection==2.3.0, I have a few questions：
1）When I trained the model, loss convergence was normal, but recall and AP were very low

What part of the transplant was wrong with me

2）The method multi_class_nms1(multi_bboxes, multi_scores, ......）

max_per_img= 125. Arguably, multi_bboxes.shape[1]=len(multi_scores)=125. But

multi_scores = 81, and
for I in range(1, num_classes): multi_bboxes.shape[1]=125, It's not a complete cycle

Thanks for sharing your research/code with us. Much appreciated.

I tried to run the "instance segmentation" test with "--show" but it fails on my setup (see below for details).
"instance segmentaion" test with "--eval segm" works OK though.
Any ideas?

python tools/test.py configs/D2Det/D2Det_instance_r101_fpn_2x.py      checkpoints/D2Det-instance-res101.pth      --show

  File ".../code/01-D2Det/mmdet/models/detectors/base.py", line 178, in show_result
    img_show[mask] = img_show[mask] * 0.5 + color_mask * 0.5
IndexError: boolean index did not match indexed array along dimension 0; dimension is 800 but corresponding boolean dimension is 427
(01-D2Det) gpu_server% 


---------------------------------------------------------------------
NG
(01-D2Det) gpu_server%  python tools/test.py configs/D2Det/D2Det_instance_r101_fpn_2x.py      checkpoints/D2Det-instance-res101.pth      --show
---------------------------------------------------------------------
loading annotations into memory...
Done (t=0.64s)
creating index...
index created!
[                                                  ] 0/5000, elapsed: 0s, ETA:
(427, 640, 3) (800, 1199, 3) (1, 3)
Traceback (most recent call last):
  File "tools/test.py", line 175, in <module>
    main()
  File "tools/test.py", line 153, in main
    outputs = single_gpu_test(model, data_loader, args.show)
  File ".../code/01-D2Det/mmdet/apis/test.py", line 23, in single_gpu_test
    model.module.show_result(data, result)
  File ".../code/01-D2Det/mmdet/models/detectors/base.py", line 178, in show_result
    img_show[mask] = img_show[mask] * 0.5 + color_mask * 0.5
IndexError: boolean index did not match indexed array along dimension 0; dimension is 800 but corresponding boolean dimension is 427
(01-D2Det) gpu_server% 
----------------------------------------------------------------------

----------------------------------------------------------------------
OK 
(01-D2Det) gpu-server%  python tools/test.py configs/D2Det/D2Det_instance_r101_fpn_2x.py      checkpoints/D2Det-instance-res101.pth --eval segm
----------------------------------------------------------------------
loading annotations into memory...
Done (t=0.64s)
creating index...
index created!
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 5000/5000, 8.3 task/s, elapsed: 603s, ETA:     0s
Evaluating segm...
Loading and preparing results...
DONE (t=1.35s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *segm*
DONE (t=32.53s).
Accumulating evaluation results...
DONE (t=3.37s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.397
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.604
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.431
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.224
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.435
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.546
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.322
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.493
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.513
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.314
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.550
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.682
(01-D2Det) gpu_server% 
----------------------------------------------------------------------