zhangyp15 / monoflex Goto Github PK

The validation AP curve in the training process seems very unstable. How to train the model and reproduce the results stably? I have tried fixing the random seed, but it does not work.

您好，我在kitti-split1上用单卡进行训练，batch_size=4，在val的3694张图上测试：

[2021-06-17 06:25:15,960] monoflex.inference INFO: metric = R40
[2021-06-17 06:25:15,961] monoflex.inference INFO:
Car [email protected], 0.70, 0.70:
bbox AP:93.1365, 88.1290, 80.6364
bev  AP:27.4066, 19.9626, 16.8124
3d   AP:19.3602, 13.9580, 11.9678
aos  AP:93.01, 87.77, 79.92
Car [email protected], 0.50, 0.50:
bbox AP:93.1365, 88.1290, 80.6364
bev  AP:61.1731, 45.9274, 40.7388
3d   AP:55.3238, 41.9672, 35.9666
aos  AP:93.01, 87.77, 79.92
Pedestrian [email protected], 0.50, 0.50:
bbox AP:52.8775, 44.4330, 37.8220
bev  AP:6.5744, 5.0258, 4.0656
3d   AP:5.9074, 4.2190, 3.2233
aos  AP:46.37, 38.43, 32.49
Pedestrian [email protected], 0.25, 0.25:
bbox AP:52.8775, 44.4330, 37.8220
bev  AP:21.2798, 17.0815, 13.9071
3d   AP:20.5831, 16.5574, 13.3822
aos  AP:46.37, 38.43, 32.49
Cyclist [email protected], 0.50, 0.50:
bbox AP:55.2721, 36.1850, 33.6507
bev  AP:1.5678, 0.8645, 0.7968
3d   AP:1.2153, 0.3637, 0.3799
aos  AP:47.42, 30.75, 28.57
Cyclist [email protected], 0.25, 0.25:
bbox AP:55.2721, 36.1850, 33.6507
bev  AP:11.4607, 5.7936, 5.4553
3d   AP:10.6146, 5.3862, 4.8811
aos  AP:47.42, 30.75, 28.57

[2021-06-17 06:25:16,132] monoflex.trainer INFO: Total training time: 21:24:51.903272 (0.8307 s / it), best model is achieved at iteration = 91872

低于论文中Table1的val ap3d_r40,：23.64 17.51 14.83，相差4个点左右。请问是我哪里设置的不对吗？谢谢

RTX3090 only support cuda11.x，I have to install pytorch 1.7.1, but it occur error with GPU

Hey everyone, first of all thank you for your amazing work!
I have a question reagarding the depth uncertainty:
If I'm not mistaken using the depth uncertainty can lead to negative loss values, if the loss is low and the uncertainty is a large negative value e.g.:

If the depth loss is 0.0001 and the uncertainty (log(sigma^2)) is -10 this would lead to a loss of ~-7.8.

0.0001*exp(10) - 10 = 7.79...

Is this desired behaviour? If yes, could you explain the reasoning? If no, how does one prevent this? And also if I understood something wrong help is appreciated.

Best regards

why use f_u to compute the decode the depth instead of f_v here？

The obj height should correspond to the y-axis in the camera coordinate

Hello, when reading the tag data, if I change the filter conditions of the instance level, does it mean that the distance of the model I trained focus is different, and will the level filtering hyperparameter affect the evaluation process?

Hello !

Thanks for great work and congrats for your paper being accepted from CVPR21.

I'd like to re-produce the model and compare with your results in the paper.
Could you provide pre-trained models trained with KITTI train (for validation) and KITTI trainval (for testing) each ?
Since most of works only provide the pre-trained model trained on KITTI trainval, which means the user cannot re-produce the qualitative results.

Best regards.

Hi, @zhangyp15
I found the result on kitti ranking list is 14.73%. Do you change any setting?

Hello ,
I see you use gradient clipping in training code, I wonder if it's necessary ? and the gradient will be vanishing if without gradient clipping ?

when use

CUDA_VISIBLE_DEVICES=0 python tools/plain_train_net.py --config runs/monoflex.yaml --ckpt model_moderate_best_soft.pth --eval

Traceback (most recent call last):
File "tools/plain_train_net.py", line 160, in
args=(args,),
File "/MonoFlex/engine/launch.py", line 57, in launch
main_func(*args)
File "tools/plain_train_net.py", line 116, in main
model = KeypointDetector(cfg)
File "/MonoFlex/model/detector.py", line 23, in init
self.heads = bulid_head(cfg, self.backbone.out_channels)
File "/MonoFlex/model/head/detector_head.py", line 29, in bulid_head
return Detect_Head(cfg, in_channels)
File "/MonoFlex/model/head/detector_head.py", line 13, in init
self.predictor = make_predictor(cfg, in_channels)
File /MonoFlex/model/head/detector_predictor.py", line 169, in make_predictor
return func(cfg, in_channels)
File "/MonoFlex/model/head/detector_predictor.py", line 105, in init
self.edge_fusion_relu = cfg.MODEL.HEAD.EDGE_FUSION_RELU
File "/.conda/envs/monoflex/lib/python3.7/site-packages/yacs/config.py", line 141, in getattr
raise AttributeError(name)
AttributeError: EDGE_FUSION_RELU

how to solve this problem?

i prepared the environment according to the README, but still reports this error:

Traceback (most recent call last):
  File "/tmp-data/user1/code/3d_object_detection/MonoFlex-main/tools/plain_train_net.py", line 25, in <module>
    from model.detector import KeypointDetector
  File "/tmp-data/user1/code/3d_object_detection/MonoFlex-main/model/detector.py", line 6, in <module>
    from .backbone import build_backbone
  File "/tmp-data/user1/code/3d_object_detection/MonoFlex-main/model/backbone/__init__.py", line 1, in <module>
    from .dla_dcn import build_backbone
  File "/tmp-data/user1/code/3d_object_detection/MonoFlex-main/model/backbone/dla_dcn.py", line 16, in <module>
    from model.backbone.DCNv2.dcn_v2 import DCN
  File "/tmp-data/user1/code/3d_object_detection/MonoFlex-main/model/backbone/DCNv2/dcn_v2.py", line 13, in <module>
    import _ext as _backend
ImportError: dynamic module does not define module export function (PyInit__ext)

How can i convert .pth to onnx?
are there any layers not support in onnx, and is there a script?

I have run the official MonoFlex pretrained model and code on kitti test dataset , got below score:

There is a big gap between above score and MonoFlex's official score.
So, how can i reproduce the score on kitti test leadboard ? How to generate predictions on kitti test dataset?

Hi, do your know this error after running". make.sh"?
I use "torch.version=1.9.1" and "cuda version==10.2".

My self data set image resolution is too high to train the model --- "cuda out of memory".....So I want to resize images,I modify kitti.py to resize image,but when train datas, it will come the error in "if torch.isinf(v).sum() > 0: pdb.set_trace()" and "combined_MAE = torch.cat((depth_MAE.unsqueeze(1), keypoint_MAE), dim=1) "， the reason is “ num_reg_2D ” not > 0, but I can't find out the root cases.Have anyone else can share the changes for resize?,thanks!

Hello? May I ask a question? Have you tried other classes like Van or Truck? If I want to extend the class, how could I do? Thanks for reading this issue! Hope for your reply! Thanks!

Hi, I tried to run MonoFlex Code at pytorch==1.4.0 torchvision==0.5.0 cudatoolkit=10.1
but I got these error logs:

`[2021-07-26 09:43:56,528] monoflex.trainer INFO: Start training
Traceback (most recent call last):
File "tools/plain_train_net.py", line 160, in
args=(args,),
File "/home/yuhan/MonoFlex/engine/launch.py", line 54, in launch
daemon=False,
File "/home/yuhan/anaconda3/envs/monoflex/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn
while not spawn_context.join():
File "/home/yuhan/anaconda3/envs/monoflex/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 118, in join
raise Exception(msg)
Exception:

-- Process 1 terminated with the following error:
Traceback (most recent call last):
File "/home/yuhan/anaconda3/envs/monoflex/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
fn(i, *args)
File "/home/yuhan/MonoFlex/engine/launch.py", line 89, in distributed_worker
main_func(*args)
File "/home/yuhan/MonoFlex/tools/plain_train_net.py", line 139, in main
train(cfg, model, device, distributed)
File "/home/yuhan/MonoFlex/tools/plain_train_net.py", line 83, in train
arguments,
File "/home/yuhan/MonoFlex/engine/trainer.py", line 109, in do_train
loss_dict, log_loss_dict = model(images, targets)
File "/home/yuhan/anaconda3/envs/monoflex/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/home/yuhan/anaconda3/envs/monoflex/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 447, in forward
output = self.module(*inputs[0], **kwargs[0])
File "/home/yuhan/anaconda3/envs/monoflex/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/home/yuhan/MonoFlex/model/detector.py", line 34, in forward
loss_dict, log_loss_dict = self.heads(features, targets)
File "/home/yuhan/anaconda3/envs/monoflex/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/home/yuhan/MonoFlex/model/head/detector_head.py", line 18, in forward
x = self.predictor(features, targets)
File "/home/yuhan/anaconda3/envs/monoflex/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/home/yuhan/MonoFlex/model/head/detector_predictor.py", line 125, in forward
feature_cls = self.class_head:-1
File "/home/yuhan/anaconda3/envs/monoflex/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/home/yuhan/anaconda3/envs/monoflex/lib/python3.7/site-packages/torch/nn/modules/container.py", line 100, in forward
input = module(input)
File "/home/yuhan/anaconda3/envs/monoflex/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/home/yuhan/anaconda3/envs/monoflex/lib/python3.7/site-packages/inplace_abn/abn.py", line 247, in forward
self.activation_param,
File "/home/yuhan/anaconda3/envs/monoflex/lib/python3.7/site-packages/inplace_abn/functions.py", line 249, in inplace_abn
None,
File "/home/yuhan/anaconda3/envs/monoflex/lib/python3.7/site-packages/inplace_abn/functions.py", line 107, in forward
count = count.to(dtype=var.dtype)
RuntimeError: CUDA error: invalid device function`

Do you have any idea to fix this ? Thank you !

如题，有什么解决办法吗

Thank you for your excellent work and big congrats for your paper acceptation! Could you release the training log for reproducibility?

hi. i have used your pretrained model to evaluate. but get something wrong

File "/home/czx/Notebook/MonoFlex/data/datasets/evaluation/kitti_object_eval_python/eval.py", line 151, in d3_box_overlap
d3_box_overlap_kernel(boxes, qboxes, rinc, criterion)
ZeroDivisionError: division by zero

Hi @zhangyp15 , Thank you for your excellent work and big congrats for your paper acceptation! I have a question about how to predict a video ?

感觉公式好像有点问题，是不是应该这样写呀

x = (uv_depth[:, 0] - self.c_u * uv_depth[:, 2]) / self.f_u + self.b_x
y = (uv_depth[:, 1] - self.c_v * uv_depth[:, 2]) / self.f_v + self.b_y

$ python testcuda.py
torch.Size([2, 64, 128, 128])
torch.Size([20, 32, 7, 7])
torch.Size([20, 32, 7, 7])
torch.Size([20, 32, 7, 7])
0.971507, 1.943014
0.971507, 1.943014
Zero offset passed
/tmp-data/user1/soft/conda/envs/monoflex/lib/python3.7/site-packages/torch/autograd/gradcheck.py:242: UserWarning: At least one of the inputs that requires gradient is not of double precision floating point. This check will likely fail if all the inputs are not of double precision floating point.
'At least one of the inputs that requires gradient '
check_gradient_dpooling: True
Traceback (most recent call last):
File "testcuda.py", line 265, in
check_gradient_dconv()
File "testcuda.py", line 97, in check_gradient_dconv
eps=1e-2, atol=1e-4, rtol=1e-2))
File "/tmp-data/user1/soft/conda/envs/monoflex/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 289, in gradcheck
'numerical:%s\nanalytical:%s\n' % (i, j, n, a))
File "/tmp-data/user1/soft/conda/envs/monoflex/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 227, in fail_test
raise RuntimeError(msg)
RuntimeError: Jacobian mismatch for output 0 with respect to input 1,
numerical:tensor([[ 0.0000e+00, 0.0000e+00, 0.0000e+00, ..., 0.0000e+00,
0.0000e+00, 0.0000e+00],
[ 0.0000e+00, -1.1563e-03, 0.0000e+00, ..., 0.0000e+00,
0.0000e+00, 0.0000e+00],
[ 0.0000e+00, 0.0000e+00, 0.0000e+00, ..., 0.0000e+00,
0.0000e+00, 0.0000e+00],
...,
[ 0.0000e+00, 0.0000e+00, 0.0000e+00, ..., 0.0000e+00,
0.0000e+00, 0.0000e+00],
[ 0.0000e+00, 0.0000e+00, 0.0000e+00, ..., 0.0000e+00,
0.0000e+00, 0.0000e+00],
[ 0.0000e+00, 0.0000e+00, 0.0000e+00, ..., 0.0000e+00,
0.0000e+00, 2.6822e-05]])
analytical:tensor([[ 0.0000e+00, 0.0000e+00, 0.0000e+00, ..., 0.0000e+00,
0.0000e+00, 0.0000e+00],
[ 0.0000e+00, -1.1562e-03, 0.0000e+00, ..., 0.0000e+00,
0.0000e+00, 0.0000e+00],
[ 0.0000e+00, 0.0000e+00, 0.0000e+00, ..., 0.0000e+00,
0.0000e+00, 0.0000e+00],
...,
[ 0.0000e+00, 0.0000e+00, 0.0000e+00, ..., 0.0000e+00,
0.0000e+00, 0.0000e+00],
[ 0.0000e+00, 0.0000e+00, 0.0000e+00, ..., 0.0000e+00,
0.0000e+00, 0.0000e+00],
[ 0.0000e+00, 0.0000e+00, 0.0000e+00, ..., 0.0000e+00,
0.0000e+00, 2.5265e-05]])

hello, i want to use the model to infer my own picture,which file in the project should i use ?

I have trained MonoFlex, but I got weird result as below.

[The validation performance when training]
@ iter46400 (final model) –
bbox AP:98.2362, 90.4009, 82.8753
bev AP:30.7541, 22.5483, 19.4824
3d AP:22.2204, 16.2115, 14.2746
aos AP:98.07, 89.99, 82.10

[The validation performance when evaluating separately (After training had been completed)]
Car [email protected], 0.70, 0.70:
bbox AP:97.7390, 90.2411, 82.6754
bev AP:27.6806, 20.9849, 18.2696
3d AP:19.7306, 14.9945, 13.3213
aos AP:97.58, 89.87, 81.93

Though two results should show the same result, It is observed that the obvious performance gap between two results.
Is there any difference between 'validation when training' and 'validation after training' ?

Thanks.

在kitti数据集中为什么没有找到ImageSets

Dear author,

Thanks for your wonderful work. Can you kindly provide the trainval config that used for evaluate the kitti set? If it is inconvenient, could you tell me how many steps you trained and the corresponding learning rate? What's more, should I use the right images when trained the final model?

Best regards,
Qing LIAN

Hi There.

According to the repo instruction, I should add --vis flag in order to visualize the predicted heatmap and 3D bounding boxes but I don't know where the results will be saved.

Thanks in advanced

Hi, I was looking into the code and found some mismatch in Inverse_Sigmoid_Loss function.

class Inverse_Sigmoid_Loss(nn.Module):
	def __init__(self):
		super(Inverse_Sigmoid_Loss, self).__init__()

	def forward(self, prediction, target, weight=None):
		trans_prediction = 1 / torch.sigmoid(target) - 1
		loss = F.l1_loss(trans_prediction, target, reduction='none')
		
		if weight is not None:
			loss = loss * weight

		return loss

This looks odd to me. trans_prediction is calculated from target and L1 loss is calculated between trans_prediction and target which doesn't make sense. variable prediction wasn't even used in this function. Any explanation for that ? Thanks

Traceback (most recent call last):
File "plain_ train_ net.py", line 25, in
from model. detector import KeypointDetector
Fіlе " /hоmе/dуj /МоnоFlех/mоdеl/dеtесtоr .ру", lіnе б, іn <mоdulе>
from backbone import build_ backbone
File " /home /dyj/MonoFlex/model/backbone/_ init__ .py" , line 1, in from .dla_ dcn import build_ backbone
File " /home /dyj/MonoF lex/model/backbone/dla_ dcn.py", line 16, in from model. backbone . DCNv2.dcn_ v2 import DCN
File "/home/dyj/MonoFlex/nodel/backbone/DCNv2/dcn v2.py", line 13, in import_ ext as_ backend
ImportError: /home/dyj/MonoFlex/model/backbone/DCNv2/ ext.cpython- 37я-x86 64-linux-gnu. so: undefined symbol: cudaSe tupArgunent

Thanks for your work, i wonder what's the difference between two Car result?
Car [email protected], 0.70, 0.70
and
Car [email protected], 0.50, 0.50:

the program will download file from http://dl.yf.io/dla/models/imagenet/dla34-ba72cf86.pth

But this link is broken, is there any way to bypass?

Hi @zhangyp15 , I have a question regarding the definition of 2D center.

For truncated objects, how is the 2D center $x_b$ obtained? Is it the center of the 2D tight bbox (min bbox around the segmentation mask) or the 2D bbox around the partial projected 3D bbox that is inside the image?

From Fig. 4(c), it looks like the latter, as the 2D bbox is not tightly surrounding the object. It would be great if you could confirm.

sys: ubuntu 20.04
GPU: GF 1080
driver: NVIDIA-SMI 515.65.01 Driver Version: 515.65.01 CUDA Version: 11.7
cuda: Cuda compilation tools, release 10.1, V10.1.243
python: 3.7.13 haa1d7c7_1
torch: torch 1.7.0+cu110 pypi_0 pypi
torchaudio 0.4.0 py37 pytorch
torchvision 0.8.1+cu110 pypi_0 pypi
cudatoolkit cudatoolkit 10.1.243 h6bb024c_0

What can I do for it? I use 4 gpus to train data with arg"--num_gpus=4",and it can train and val,but one hours later,error occured:
`Traceback (most recent call last):
File "tools/plain_train_net.py", line 161, in
args=(args,),
File "/zft/code/MonoFlex/engine/launch.py", line 54, in launch
daemon=False,
File "/root/anaconda3/envs/monoflex/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 199, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
File "/root/anaconda3/envs/monoflex/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 157, in start_processes
while not context.join():
File "/root/anaconda3/envs/monoflex/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 118, in join
raise Exception(msg)
Exception:

-- Process 1 terminated with the following error:
Traceback (most recent call last):
File "/root/anaconda3/envs/monoflex/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
fn(i, *args)
File "/zft/code/MonoFlex/engine/launch.py", line 89, in _distributed_worker
main_func(*args)
File "/zft/code/MonoFlex/tools/plain_train_net.py", line 140, in main
train(cfg, model, device, distributed)
File "/zft/code/MonoFlex/tools/plain_train_net.py", line 84, in train
arguments,
File "/zft/code/MonoFlex/engine/trainer.py", line 109, in do_train
loss_dict, log_loss_dict = model(images, targets)
File "/root/anaconda3/envs/monoflex/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/root/anaconda3/envs/monoflex/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 619, in forward
output = self.module(*inputs[0], **kwargs[0])
File "/root/anaconda3/envs/monoflex/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/zft/code/MonoFlex/model/detector.py", line 34, in forward
loss_dict, log_loss_dict = self.heads(features, targets)
File "/root/anaconda3/envs/monoflex/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/zft/code/MonoFlex/model/head/detector_head.py", line 21, in forward
loss_dict, log_loss_dict = self.loss_evaluator(x, targets)
File "/zft/code/MonoFlex/model/head/detector_loss.py", line 271, in call
pred_targets, preds, reg_nums, weights = self.prepare_predictions(targets_variables, predictions)
File "/zft/code/MonoFlex/model/head/detector_loss.py", line 153, in prepare_predictions
target_corners_3D = self.anno_encoder.encode_box3d(target_rotys_3D, target_dimensions_3D, target_locations_3D)
File "/zft/code/MonoFlex/model/anno_encoder.py", line 108, in encode_box3d
ry = self.rad_to_matrix(rotys, N)
File "/zft/code/MonoFlex/model/anno_encoder.py", line 60, in rad_to_matrix
ry = i_temp.repeat(N, 1).view(N, -1, 3)
RuntimeError: cannot reshape tensor of 0 elements into shape [0, -1, 3] because the unspecified dimension size -1 can be any value and is ambiguous`

Dear Author
Thank you very much for your excellent and amazing work:
I have tried to replicate the task but got a flowing error during training.

Hello, thanks for your work, hope we can meet your code soon.

Hi @zhangyp15 , I have a question regarding the loss weight.
I see you use multi loss in training and set different loss weight to each loss.
In general it's better to use adaptive loss weight for multi task and I also find you add uncert_wrapper.py in project to calculate loss uncertainty. why you don't use adaptive loss weight? its effect is worse than fixed weight ? and how do you get he fixed scale loss weight ? Thanks

Thanks for your work, i wonder what's the difference between two Car result?
Car [email protected], 0.70, 0.70
and
Car [email protected], 0.50, 0.50:

Hi, when I try to evaluate in the test set, I modify the configuration file like this:

However, when executing the dataloader, the following error occurs:

I can evaluate in the val set but cannot evaluate in the test set. Do you know this reason?

What does oracle mean?

I noticed that the 3D AP of MonoFlex on the leaderboard of the KITTI website has been improved compared with the original paper, can the author tell us what changes have been made? Thanks a lot.

如题

Hi Yunpeng @zhangyp15 ,

First of all, congratulations on the great work and being accepted to CVPR 2021. This study addresses one elephant in the room for mono3D, and handles it quite elegantly.

Really looking forward to the release of the code.

Patrick

Hello, @zhangyp15 !
Thanks for your work Monoflex. I am wondering does the code only support train from scratch, but not using pretrained weights.

代码中的DEPTH REFERENCE=[26.494672, 9.075032501413093]是什么含义？

Hello !

Thanks for great work and congrats for your paper being accepted from CVPR21.
when I get the model ,I want to do the test, but don't have the lable，how to do it?

Best regards.