ixiaohuihuihui / ao2-detr

AO2-DETR: Arbitrary-Oriented Object Detection Transformer

Hi there 👋

• 👩‍🏫 I'm currently an Assistant Professor at Shenzhen University.
• 🌻 I graduated with a Ph.D. from Peking University in July 2024.
• 🌱 I’m currently learning object detection.
• 📫 Feel free to reach me: [email protected]

ao2-detr's Issues

training log for Dota_v1.5

Hello! Could you provide logs for training results (specifically, the one that got 75.89 mAP) on Dota_v1.5 dataset?

checkpoints

can you tell me where the checkpoints saved in?

config/_base_/datatsets/dota_detection.py报错

这是报错信息
KeyError: 'DOTA10Dataset is not in the dataset registry'
DOTA10Dataset 是还没有写吗

Because the mmrotate version is updated, the setup.py file may be invalid

Just change the setup.py file in the new version of mmrotate to this project

use_dab

Hello, I'm interesting of AO2-DETR.
I wanna use dab setting, Can u tell me about config setting detail?

I just modify "./configs/deformable_detr/deformable_detr_twostage_refine_r50_16x2_50e_dota"

base = 'deformable_detr_refine_r50_16x2_50e_dota.py'
model = dict(bbox_head=dict(use_dab = True))

Thank u

Issue on Training

Hi, Thank you for sharing nice work.

I tried your code, but I got some error like below.
I made my own custom dataset(has 5 classes)
I can't figure out How to fix this problem.

If you know any way for this, please help me :)

+) Looks like the original mmdet/hungarian_assigner.py has only 4-dim factor(for normalize).

I think in ao2-detr cases, it looks like using additional "angle" option( gt_bboxes.append([x, y, w, h, a]), so It maybe need to use 5-dim. What should i modify on here?
did i missed something in installation step??

Welcome update to OpenMMLab 2.0

Welcome update to OpenMMLab 2.0

I am Vansin, the technical operator of OpenMMLab. In September of last year, we announced the release of OpenMMLab 2.0 at the World Artificial Intelligence Conference in Shanghai. We invite you to upgrade your algorithm library to OpenMMLab 2.0 using MMEngine, which can be used for both research and commercial purposes. If you have any questions, please feel free to join us on the OpenMMLab Discord at https://discord.gg/amFNsyUBvm or add me on WeChat (van-sin) and I will invite you to the OpenMMLab WeChat group.

Here are the OpenMMLab 2.0 repos branches:

	OpenMMLab 1.0 branch	OpenMMLab 2.0 branch
MMEngine		0.x
MMCV	1.x	2.x
MMDetection	0.x 、1.x、2.x	3.x
MMAction2	0.x	1.x
MMClassification	0.x	1.x
MMSegmentation	0.x	1.x
MMDetection3D	0.x	1.x
MMEditing	0.x	1.x
MMPose	0.x	1.x
MMDeploy	0.x	1.x
MMTracking	0.x	1.x
MMOCR	0.x	1.x
MMRazor	0.x	1.x
MMSelfSup	0.x	1.x
MMRotate	1.x	1.x
MMYOLO		0.x

Attention: please create a new virtual environment for OpenMMLab 2.0.

Is there no validation set in the experiment

Hi! Thanks for your work!
I notice that the validation set is the same as the training set in configs/base/datasets/dotav1.py, both of them are set as 'split_1024_dota1_0/trainval'.Emmm,it doesn't seem reasonable. So I wonder how to choose the best epoch in all the pth files.

Problem with convergence after applying FRM

In my case here, I got relatively identical results from baseline training just like these provided ones here: link, but I found that the FRM configs were commented.
When I uncommented the configs and tested on 2 consecutive FRM modules with 1-point sampling(as it was in the default setting) and 5-point sampling(as in the R3Det paper), that was a total of 4 sets of experiments, I got no convergence on any of these experiments. Specifically, either of them showed pool performance:

or eventually deteriorated to 0.

I made a slight change in the forwarding of FRM modules:

Is there any possible explanation for this?

frm setting

Hello, I am trying to do frm setting now.

when I set frm(feature refinement module), Like paper setting

bbox_head=dict(
type='RotatedDeformableDETRHead',
num_query=300,
num_classes=15,
in_channels=2048,
sync_cls_avg_factor=True,
as_two_stage=False,
frm_cfgs=[
dict(in_channels=256, featmap_strides=[16, 32, 64, 128])],

mAP is very low
train: train
test: val

baseline-> 62.4, baseline+frm -> 31.4

Could u tell me any idea??

When will the code be available？

When will the code be available?
Thanks.

在dotav1上map只有0.03

20220708_081013.log
这个是log，请问是什么问题呢

where is the completed code ?

hi, thank you for the good work! May I ask where is the completed open source code ?

How to solve nan value in cost?

I find the following code in rotated_hungarian_assigner.py

contain_nan = (True in np.isnan(cost_matrix))
if contain_nan:
    # a = 1
    print('Find!!!')

And I meet same problem. Especially happens in gwd/kld cost(gaussian loss).
I don't know why this happens & how to solve.

Port to huggingface

Hello ! great work form the whole team

We were wondering if you intend to release the weights in the HuggingFace hub so that we can use it more easily ?

We would very much like to use your approach on inclined table detection in PDFs

Cheers

dataset

HRSID dataset，what file is the ‘hrsid.z01’ file？ How to unzip it? thx.

精度问题

我跑了两个脚本deformable_detr_twostage_refine_r50_16x2_50e_dota.py，deformable_detr_twostage_refine_r50_16x2_50e_dota1_ms，在trainval上达到了86，但是放到官方评估上test只达到了70，我想问问有人出现这个情况吗，这是为什么呀差的有点多呀

RuntimeError: The size of tensor a (4) must match the size of tensor b (5) at non-singleton dimension 1

sys.platform: win32
Python: 3.8.10 | packaged by conda-forge | (default, Sep 13 2021, 21:12:34) [MSC v.1916 64 bit (AMD64)]
CUDA available: True
GPU 0: NVIDIA GeForce RTX 4060 Laptop GPU
CUDA_HOME: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1
NVCC: Cuda compilation tools, release 12.1, V12.1.105
MSVC: 用于 x64 的 Microsoft (R) C/C++ 优化编译器 19.37.32824 版
GCC: n/a
PyTorch: 1.11.0+cu113
PyTorch compiling details: PyTorch built with:

• C++ Version: 199711
• MSVC 192829337
• Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
• Intel(R) MKL-DNN v2.5.2 (Git Hash a9302535553c73243c632ad3c4c80beec3d19a1e)
• OpenMP 2019
• LAPACK is enabled (usually provided by MKL)
• CPU capability usage: AVX2
• CUDA Runtime 11.3
• NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61
  ,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37
• CuDNN 8.2
• Magma 2.5.4
• Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.3, CUDNN_VERSION=8.2.0, CXX_COMPILER=C:/actions-runner/_work/pytorch/pytorch/builder/windo
  ws/tmp_bin/sccache-cl.exe, CXX_FLAGS=/DWIN32 /D_WINDOWS /GR /EHsc /w /bigobj -DUSE_PTHREADPOOL -openmp:experimental -IC:/actions-runner/_work/pytorch/pytorch/bui
  lder/windows/mkl/include -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO, LAPAC
  K_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.11.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=OFF, USE_OPENMP=ON, USE_ROCM=OFF,

TorchVision: 0.12.0+cu113
OpenCV: 4.9.0
MMCV: 1.7.0
MMCV Compiler: MSVC 192829924
MMCV CUDA Compiler: 11.3
MMRotate: 0.3.4+

The command I executed ：python tools/train.py configs/deformable_detr/deformable_detr_twostage_refine_r50_16x2_50e_dota.py

the report error：
Traceback (most recent call last):
File "tools/train.py", line 182, in
main()
File "tools/train.py", line 171, in main
train_detector(
File "f:\pycharm\mmrotate\mmrotate\apis\train.py", line 145, in train_detector
runner.run(data_loaders, cfg.workflow)
File "F:\envs\mmrotate\lib\site-packages\mmcv\runner\epoch_based_runner.py", line 136, in run
epoch_runner(data_loaders[i], **kwargs)
File "F:\envs\mmrotate\lib\site-packages\mmcv\runner\epoch_based_runner.py", line 53, in train
self.run_iter(data_batch, train_mode=True, **kwargs)
File "F:\envs\mmrotate\lib\site-packages\mmcv\runner\epoch_based_runner.py", line 31, in run_iter
outputs = self.model.train_step(data_batch, self.optimizer,
File "F:\envs\mmrotate\lib\site-packages\mmcv\parallel\data_parallel.py", line 77, in train_step
return self.module.train_step(*inputs[0], **kwargs[0])
File "F:\envs\mmrotate\lib\site-packages\mmdet\models\detectors\base.py", line 248, in train_step
losses = self(**data)
File "F:\envs\mmrotate\lib\site-packages\torch\nn\modules\module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "F:\envs\mmrotate\lib\site-packages\mmcv\runner\fp16_utils.py", line 119, in new_func
return old_func(*args, **kwargs)
File "F:\envs\mmrotate\lib\site-packages\mmdet\models\detectors\base.py", line 172, in forward
return self.forward_train(img, img_metas, **kwargs)
File "f:\pycharm\mmrotate\mmrotate\models\detectors\rotated_detr.py", line 129, in forward_train
losses = self.bbox_head.forward_train(x, img_metas, gt_bboxes,
File "f:\pycharm\mmrotate\mmrotate\models\dense_heads\rotated_detr_head.py", line 606, in forward_train
losses = self.loss(*loss_inputs, gt_bboxes_ignore=gt_bboxes_ignore)
File "F:\envs\mmrotate\lib\site-packages\mmcv\runner\fp16_utils.py", line 208, in new_func
return old_func(*args, **kwargs)
File "f:\pycharm\mmrotate\mmrotate\models\dense_heads\rotated_deformable_detr_head.py", line 332, in loss
losses_cls, losses_bbox, losses_piou = multi_apply(
File "F:\envs\mmrotate\lib\site-packages\mmdet\core\utils\misc.py", line 30, in multi_apply
return tuple(map(list, zip(*map_results)))
File "f:\pycharm\mmrotate\mmrotate\models\dense_heads\rotated_detr_head.py", line 383, in loss_single
cls_reg_targets = self.get_targets(cls_scores_list, bbox_preds_list,
File "f:\pycharm\mmrotate\mmrotate\models\dense_heads\rotated_detr_head.py", line 488, in get_targets
bbox_weights_list, pos_inds_list, neg_inds_list) = multi_apply(
File "F:\envs\mmrotate\lib\site-packages\mmdet\core\utils\misc.py", line 30, in multi_apply
return tuple(map(list, zip(*map_results)))
File "f:\pycharm\mmrotate\mmrotate\models\dense_heads\rotated_detr_head.py", line 566, in _get_target_single
pos_gt_bboxes_normalized = sampling_result.pos_gt_bboxes / factor
RuntimeError: The size of tensor a (4) must match the size of tensor b (5) at non-singleton dimension 1

Can you help me solve this problem, thank you very much

Nice Job!

The code is still incomplete?

I have tried to run the deformable_detr_twostage_refine_r50_16x2_50e_dota15_refine.py (modify from deformable_detr_twostage_refine_r50_16x2_50e_dota_refine.py), but the running process always prompted "cuda error device-side assert triggered", but deformable_detr_r50_16x2_50e_dota15.py is ok when running.

the training did not converge

Why the training loss does not converged?

I'm trying to learning for DOTA-v1.0 dataset.

I use copy of your config file "deformable_detr_r50_16x2_50e_dota.py"
I just change the angle_version from 'oc' to 'le90'

#angle_version = 'oc'
angle_version = 'le90'

and modify the "datasets/dotav1.py" to DOTA-v1.0 single-scale dataset.
like below..

data_root = '/home/jovyan/dl_data/DOTA-v1.0-v1.5/split_ss_dota/'

data = dict(
    #samples_per_gpu=4,
    samples_per_gpu=2,
    workers_per_gpu=2,
    train=dict(
        type=dataset_type,
        # ann_file=data_root + 'trainval1024_ms/DOTA_trainval1024_ms.json',
        ann_file=data_root + 'train/annfiles/',
        img_prefix=data_root + 'train/images/',
        pipeline=train_pipeline),
    val=dict(
        type=dataset_type,
        ann_file=data_root + 'val/annfiles/',
        img_prefix=data_root + 'val/images/',
        pipeline=test_pipeline),
    test=dict(
        type=dataset_type,
        # ann_file='/data2/dailh/split_1024_dota1_0/test/' + 'images/',
        # img_prefix='/data2/dailh/split_1024_dota1_0/test/' + 'images/',
        # ann_file=data_root + 'trainval/annfiles/',
        # # img_prefix=data_root + 'trainval/images/',
        ann_file=data_root + 'test/images/',
        img_prefix=data_root + 'test/images/',
        # ann_file=data_root + 'trainval/annfiles/',
        # img_prefix=data_root + 'trainval/images/',
        pipeline=test_pipeline))

and I have a one GPU card, so I learn like that
"python ./tools/train.py ./configs/deformable_detr/deformable_detr_r50_16x2_50e_dota.py"

Then, I got a log like below

It looks the loss does not converged..
what's wrong?
why the loss value does not converged? what am I miss something?

[
2022-09-08 15:48:34,495 - mmrotate - INFO - Epoch [2][4860/4867] lr: 1.000e-04, eta: 1 day, 6:24:50, time: 0.465, data_time: 0.007, memory: 8238, loss_cls: 0.7085, loss_piou: 8.8948, loss_bbox: 1.5794, d0.loss_cls: 0.7612, d0.loss_piou: 8.8352, d0.loss_bbox: 1.5764, d1.loss_cls: 0.7387, d1.loss_piou: 8.6198, d1.loss_bbox: 1.5748, d2.loss_cls: 0.7116, d2.loss_piou: 8.8279, d2.loss_bbox: 1.5675, d3.loss_cls: 0.7338, d3.loss_piou: 8.9340, d3.loss_bbox: 1.5621, d4.loss_cls: 0.7109, d4.loss_piou: 8.7618, d4.loss_bbox: 1.5756, loss: 66.6741, grad_norm: 1346.9443
2022-09-08 15:48:37,819 - mmrotate - INFO - Saving checkpoint at 2 epochs
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 3066/3066, 12.9 task/s, elapsed: 238s, ETA: 0s2022-09-08 15:52:48,874 - mmrotate - INFO -
+--------------------+-------+--------+--------+-------+
| class | gts | dets | recall | ap |
+--------------------+-------+--------+--------+-------+
| plane | 4449 | 67402 | 0.739 | 0.485 |
| baseball-diamond | 358 | 22337 | 0.620 | 0.049 |
| bridge | 783 | 145008 | 0.134 | 0.017 |
| ground-track-field | 212 | 30593 | 0.387 | 0.020 |
| small-vehicle | 10579 | 66748 | 0.121 | 0.050 |
| large-vehicle | 8819 | 67548 | 0.100 | 0.006 |
| ship | 18537 | 92142 | 0.101 | 0.015 |
| tennis-court | 1512 | 14030 | 0.792 | 0.551 |
| basketball-court | 266 | 24922 | 0.368 | 0.098 |
| storage-tank | 4740 | 71717 | 0.262 | 0.022 |
| soccer-ball-field | 251 | 20395 | 0.355 | 0.071 |
| roundabout | 275 | 29838 | 0.582 | 0.130 |
| harbor | 4167 | 62320 | 0.161 | 0.012 |
| swimming-pool | 732 | 31287 | 0.321 | 0.022 |
| helicopter | 122 | 20213 | 0.492 | 0.003 |
+--------------------+-------+--------+--------+-------+
| mAP | | | | 0.103 |
+--------------------+-------+--------+--------+-------+
2022-09-08 15:52:48,961 - mmrotate - INFO - Exp name: deformable_detr_dota_copy.py
2022-09-08 15:52:48,961 - mmrotate - INFO - Epoch(val) [2][3066] mAP: 0.1034
2022-09-08 15:52:56,104 - mmrotate - INFO - Epoch [3][10/4867] lr: 1.000e-04, eta: 1 day, 6:24:22, time: 0.713, data_time: 0.245, memory: 8238, loss_cls: 0.8683, loss_piou: 11.0895, loss_bbox: 1.7575, d0.loss_cls: 0.8965, d0.loss_piou: 11.9468, d0.loss_bbox: 1.7808, d1.loss_cls: 0.8592, d1.loss_piou: 11.9596, d1.loss_bbox: 1.7579, d2.loss_cls: 0.8440, d2.loss_piou: 11.3708, d2.loss_bbox: 1.7564, d3.loss_cls: 0.8481, d3.loss_piou: 11.4637, d3.loss_bbox: 1.7504, d4.loss_cls: 0.9026, d4.loss_piou: 11.3918, d4.loss_bbox: 1.7500, loss: 84.9938, grad_norm: 2063.5599
2022-09-08 15:53:00,730 - mmrotate - INFO - Epoch [3][20/4867] lr: 1.000e-04, eta: 1 day, 6:24:16, time: 0.463, data_time: 0.007, memory: 8238, loss_cls: 0.5905, loss_piou: 8.3354, loss_bbox: 1.7153, d0.loss_cls: 0.6218, d0.loss_piou: 9.0859, d0.loss_bbox: 1.7256, d1.loss_cls: 0.5957, d1.loss_piou: 8.1240, d1.loss_bbox: 1.7231, d2.loss_cls: 0.5891, d2.loss_piou: 8.2217, d2.loss_bbox: 1.7177, d3.loss_cls: 0.6114, d3.loss_piou: 7.7628, d3.loss_bbox: 1.7129, d4.loss_cls: 0.5860, d4.loss_piou: 7.3536, d4.loss_bbox: 1.7171, loss: 62.7894, grad_norm: 911.2250
2022-09-08 15:53:05,344 - mmrotate - INFO - Epoch [3][30/4867] lr: 1.000e-04, eta: 1 day, 6:24:09, time: 0.461, data_time: 0.007, memory: 8238, loss_cls: 0.5810, loss_piou: 5.0878, loss_bbox: 1.4809, d0.loss_cls: 0.6305, d0.loss_piou: 5.2119, d0.loss_bbox: 1.4825, d1.loss_cls: 0.5855, d1.loss_piou: 5.8150, d1.loss_bbox: 1.4835, d2.loss_cls: 0.5663, d2.loss_piou: 5.8356, d2.loss_bbox: 1.4850, d3.loss_cls: 0.6103, d3.loss_piou: 5.2023, d3.loss_bbox: 1.4839, d4.loss_cls: 0.5880, d4.loss_piou: 5.8718, d4.loss_bbox: 1.4889, loss: 45.4907, grad_norm: 575.0752
]

KeyError: 'RotatedDeformableDETR is not in the models registry'

你好，我在运行你的代码时，出现了这个错误：KeyError: 'RotatedDeformableDETR is not in the models registry'
我看了你给其他人回答，问题还是没有解决。
我的代码目录如下：

我的运行环境如下：

我的运行命令为：
python tools/train.py configs/deformable_detr/deformable_detr_r50_16x2_50e_dota.py
我的运行结果如下：

麻烦解答一下，我要怎样才可以顺利运行，谢谢。

the mAP on HRSC2016 is close to 0

Hi, I tried to use this code to train on the HRSC2016 dataset, but the mAP is close to 0, and I can't find out what the problem is. Have you used this code to test on the HRSC2016 dataset?

KeyError: 'RotatedDeformableDETR is not in the models registry'

Hello,

I get the error above when I try to run this config file on a single GPU: deformable_detr_r50_16x2_50e_dota.py

I use this command to run: python tools/train.py configs/deformable_detr/deformable_detr_r50_16x2_50e_dota.py

This does not happen when I try to train using other config files. Also, multi-GPU training works fine with this same config file.

Here is the full error message:

"
Traceback (most recent call last):
File "tools/train.py", line 180, in
main()
File "tools/train.py", line 153, in main
model = build_detector(cfg.model, train_cfg=cfg.get('train_cfg'), test_cfg=cfg.get('test_cfg'))
File "/home//anaconda3/envs/research-ao2/lib/python3.7/site-packages/mmrotate/models/builder.py", line 56, in build_detector
cfg, default_args=dict(train_cfg=train_cfg, test_cfg=test_cfg))
File "/home//anaconda3/envs/research-ao2/lib/python3.7/site-packages/mmcv/utils/registry.py", line 237, in build
return self.build_func(*args, **kwargs, registry=self)
File "/home//anaconda3/envs/research-ao2/lib/python3.7/site-packages/mmcv/cnn/builder.py", line 27, in build_model_from_cfg
return build_from_cfg(cfg, registry, default_args)
File "/home//anaconda3/envs/research-ao2/lib/python3.7/site-packages/mmcv/utils/registry.py", line 62, in build_from_cfg
f'{obj_type} is not in the {registry.name} registry')
KeyError: 'RotatedDeformableDETR is not in the models registry'
"

Kindly help.

KeyError: 'RotatedDeformableDETR is not in the models registry'

I want to run the training for AO2-DETR . But the following error is coming while running the training.
Train command used: python train.py configs/deformable_detr/deformable_detr_twostage_refine_r50_16x2_50e_dota.py
How to resolve this error.
mmcv version = 1.6.2
mmdet version = 2.28.1
mmrotate version = 0.3.4

FRM extra questions

Hello, I am Yongjun park.

I am interesting about transformer structure in Aerial,
So, I have some extra questions.

I saw OPG- > OPG+FRM(mAP +4.87) in ur paper.

I don't implement OPG clearly yet, So I just test FRM now.
When I test (baseline(2stage&iterative box) + FRM), It is almost same mAP with only baseline(2stage&iterative box).

baseline mAP(train:train, test:val):

baseline+FRM mAP(train:train, test:val):

So, OPG&FRM relation is important?
Or, It can be increase enough even comparing baseline and baseline+FRM?

And last, When can u upload ao2 code?

I really wanna test ao2 quickly.

Thank you.

KeyError: 'RotatedDeformableDETR is not in the models registry'

你好，我在執行python ./tools/train.py config/path，碰到KeyError: 'RotatedDeformableDETR is not in the models registry'，
我的pytorch==1.9.0 + cuda102
MMCV : 1.7.1
MMROTATE: 0.3.4

MMROTATE的安裝方式是git clone mmrotate project，然後執行pip install -v -e .
但是還是會碰到KeyError: 'RotatedDeformableDETR is not in the models registry'，請問有任何想法可以幫助我嗎?
感謝回復