predict_trans error about gsnet

[lkeab]

Please ensure you used the config of kpt point number in here

[lkeab]

When you run bash run.sh, has this Line752-754 been executed?

[lkeab]

https://github.com/lkeab/gsnet/blob/master/reference_code/GSNet-release/detectron2/modeling/roi_heads/roi_heads.py#L753

[yuhanyuhang]

I have figure out that this problem is caused by the roi_head.py you provide is not used actually in my detectron2. but when i use the roi_head.py your provide in my environment(cuda11.1+pytorch1.8), i meet a lot of problem with the version of detecrton2.
i have solve most of the problem, but one:

i change the line 640: selected_keypoint_logits_heatmap = self.keypoint_head(keypoint_features)

to :selected_keypoint_logits_heatmap = self.keypoint_head(keypoint_features, pred_instances)

because missing argument.

but the output of 'self.keypoint_head' is not the same, my 'selected_keypoint_logits_heatmap' is a list, it can not put into:
(line 642)selected_keypoint_res = keypoint_rcnn_inference(selected_keypoint_logits_heatmap, pred_instances);

i find in my 'selected_keypoint_logits_heatmap' has a key named pred_keypoint_heatmaps

so i change (line 642)selected_keypoint_res = keypoint_rcnn_inference(selected_keypoint_logits_heatmap, pred_instances);
to:
selected_keypoint_res = keypoint_rcnn_inference(selected_keypoint_logits_heatmap[0].pred_keypoint_heatmaps, pred_instances);

but the selected_keypoint_logits_heatmap[0].pred_keypoint_heatmaps is shape of [9,17,56,56], i think the correct shape is [9.66,56,56]

can you print the shape of selected_keypoint_logits_heatmap in your environment, or do you have any idea about this problem.

thanks a lot

[lkeab]

Hi, please use the keypoint head in this file, which has only keypoint_features passed in.

[yuhanyuhang]

Thanks for you reply, i have use the file you provide then i can use:
(line 640)selected_keypoint_logits_heatmap = self.keypoint_head(keypoint_features)
to get tensor, but the shape of selected_keypoint_logits_heatmap is [9,17,56,56]

but i have successfully install your project on my cuda 10 environment the shape of selected_keypoint_logits_heatmap is [9,66,56,56]

i have check the shape of keypoint_features in both environments , they are[9,256,14,14]
why the input is same and function is same, but i got different output

[yuhanyuhang]

this is the warning information when run:

Skip loading parameter 'roi_heads.box_predictor.bbox_pred.weight' to the model due to incompatible shapes: (4, 1024) in the checkpoint but (316, 1024) in the model! You might want to double check if this is expected.
Skip loading parameter 'roi_heads.box_predictor.bbox_pred.bias' to the model due to incompatible shapes: (4,) in the checkpoint but (316,) in the model! You might want to double check if this is expected.
Skip loading parameter 'roi_heads.keypoint_head.score_lowres.weight' to the model due to incompatible shapes: (512, 66, 4, 4) in the checkpoint but (512, 17, 4, 4) in the model! You might want to double check if this is expected.
Skip loading parameter 'roi_heads.keypoint_head.score_lowres.bias' to the model due to incompatible shapes: (66,) in the checkpoint but (17,) in the model! You might want to double check if this is expected.
Some model parameters or buffers are not found in the checkpoint:
roi_heads.box_predictor.bbox_pred.{bias, weight}
roi_heads.keypoint_head.score_lowres.{bias, weight}
The checkpoint state_dict contains keys that are not used by the model:
proposal_generator.anchor_generator.cell_anchors.{0, 1, 2, 3, 4}

there is also the value '17' , does it mean i use the incorrect mode, but how can i change the model?

[lkeab]

Thanks for you reply, i have use the file you provide then i can use:
(line 640)selected_keypoint_logits_heatmap = self.keypoint_head(keypoint_features)
to get tensor, but the shape of selected_keypoint_logits_heatmap is [9,17,56,56]

but i have successfully install your project on my cuda 10 environment the shape of selected_keypoint_logits_heatmap is [9,66,56,56]

i have check the shape of keypoint_features in both environments , they are[9,256,14,14]
why the input is same and function is same, but i got different output

I think this 66 or 17 difference may due to config file setting.

[yuhanyuhang]

thank you very much! i have solve this problem successfully!

i meet the very similar problem then:

Skip loading parameter 'roi_heads.box_predictor.bbox_pred.weight' to the model due to incompatible shapes: (4, 1024) in the checkpoint but (316, 1024) in the model! You might want to double check if this is expected.

it is the problem about 4 and 316, i cant find this in the default file, can you help me fine how to fix the config file

[lkeab]

316/4 = 79. I think the 79 may be the number of we provided car meshes.

[yuhanyuhang]

Thank you for your guidance and patience！

Skip loading parameter 'roi_heads.box_predictor.bbox_pred.weight' to the model due to incompatible shapes: (4, 1024) in the checkpoint but (316, 1024) in the model! You might want to double check if this is expected.
Skip loading parameter 'roi_heads.box_predictor.bbox_pred.bias' to the model due to incompatible shapes: (4,) in the checkpoint but (316,) in the model! You might want to double check if this is expected.
Some model parameters or buffers are not found in the checkpoint:
roi_heads.box_predictor.bbox_pred.{bias, weight}
The checkpoint state_dict contains keys that are not used by the model:
proposal_generator.anchor_generator.cell_anchors.{0, 1, 2, 3, 4}
0%| | 0/26 [00:00<?, ?it/s]['input/']
keypoint_features_forward_box torch.Size([27, 256, 14, 14])
torch.Size([27, 66, 56, 56])
0%| | 0/26 [00:19<?, ?it/s]
Traceback (most recent call last):
File "demo2/demo.py", line 90, in
predictions, visualized_output = demo.run_on_image(img, save_name)
File "D:\ApolloCar3D\gsnet-master\reference_code\GSNet-release\demo2\predictor.py", line 54, in run_on_image
predictions = self.predictor(image)
File "d:\apollocar3d\detectron2-master\detectron2\engine\defaults.py", line 318, in call
predictions = self.model([inputs])[0]
File "C:\Users\One\anaconda3\envs\detectron2\lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "d:\apollocar3d\detectron2-master\detectron2\modeling\meta_arch\rcnn.py", line 146, in forward
return self.inference(batched_inputs)
File "d:\apollocar3d\detectron2-master\detectron2\modeling\meta_arch\rcnn.py", line 209, in inference
results, _ = self.roi_heads(images, features, proposals, None)
File "C:\Users\One\anaconda3\envs\detectron2\lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "d:\apollocar3d\detectron2-master\detectron2\modeling\roi_heads\roi_heads.py", line 568, in forward
pred_instances = self._forward_3d_pose_inference(selected_roi_features, selected_boxes, selected_kpt_pos, selected_heatmap, pred_instances)
File "d:\apollocar3d\detectron2-master\detectron2\modeling\roi_heads\roi_heads.py", line 737, in _forward_3d_pose_inference
new_box_pos = self.relu(self.fuse_box_pos(new_box_pos))
File "C:\Users\One\anaconda3\envs\detectron2\lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "C:\Users\One\anaconda3\envs\detectron2\lib\site-packages\torch\nn\modules\linear.py", line 94, in forward
return F.linear(input, self.weight, self.bias)
File "C:\Users\One\anaconda3\envs\detectron2\lib\site-packages\torch\nn\functional.py", line 1753, in linear
return torch._C._nn.linear(input, weight, bias)
RuntimeError: mat1 dim 1 must match mat2 dim 0

This is the whole error, I think is the same as the 66 or 17 difference, but I don't have any idea about it. I have check the code line you point out, i'm sure i use the 79 in here.

[yuhanyuhang]

For i successfully build your work on my 2080 environment，I also have another question. For the generated results, how to visualize them into a 3D mesh. Do you modify the render_car_instance [here]([https://github.com/ApolloScapeAuto/dataset-api/tree/master/car_instance] to achieve it.

THANK YOU VERY MUCH. I've just started my PhD. I think your work is excellent and I want to follow your work. Thanks again!

[lkeab]

yes, you could use render_car_instance.py to help visualize the projections.