Hello, I read your densefusion article before. I wanted to implement it in ROS, but it failed in the end. Can this article provide ROS code? If you can, that would be great!

Hi @j96w , i am a new HSR user. Could u provide the complete code for HSR grasping? thank you in advance

Hi, thank you for making your code publish. I am familiar with your previous work (DenseFusion) which is also a great asses to 3D pose estimation.

I have a question about 6-PACK.

Can 6-PACK train one network for multiple categories pose estimation or one category needs one network?

Hi, I am trying to run inference.py, but met with an error:

AttributeError: 'Loss' object has no attribute 'pconf2'

It comes from loss.py, line 262:

new_t = torch.sum(Kp_fr * self.pconf2, dim=1).view(1, 3).contiguous()

self.pconf2 could not be found.

I saw there is a self.pconf in __init__, dose line 262 should be change to self.pconf?

Hi, does "samller percentange" you mentioned in Evaluation-Notice mean "precision" ? while the metric used in NOCS is mAP.

Hello author, thanks for sharing your great codes, if I want to use your network to predict objects that do not exist in the data set you provide, how can I make the data set of this object? What tools should I use? Could you give me some guidance?
Looking forward to your reply.

Thank you for your work!
When I run sudo python3 train.py, the following happens

home/user/Work/6-PACK/libs/transformations.py:1912: UserWarning: failed to import module _transformations
warnings.warn('failed to import module %s' % name)
1 a1c63c463ad7fc1fb549927fbd0d12d
1 73632ddb4a5684503594b3be653e6bff
1 bd6fdeae09b3c00c3ea96ac97db63fa8
1 6444875e3217bf891f5f48a891d827bd
1 1e5abf0465d97d826118a17db9de8c0
1 6b8b2cb01c376064c8724d5673a063a6
1 109d55a137c042f5760315ac3bf2c13e
.
.
.
6 73b8b6456221f4ea20d3c05c08e26f
6 d32cd77c6630b77de47c0353c18d58e
6 b4ae56d6638d5338de671f28c83d2dcb
6 187859d3c3a2fd23f54e1b6f41fdd78a
6 b7e705de46ebdcc14af54ba5738cb1c5
6 34ae0b61b0d8aaf2d7b20fded0142d7a
6 896f1d494bac0ebcdec712af445786fe
1 bottle_starbuck_norm
1 bottle3_scene5_norm
1 bottle_blue_google_norm
2 bowl_blue_ikea_norm
2 bowl_chinese_blue_norm
2 bowl_brown_ikea_norm
3 camera_dslr_wo_len_norm
3 camera_dslr_len_norm
3 camera_anastasia_norm
4 can_tall_yellow_norm
4 can_porridge_norm
4 can_milk_wangwang_norm
5 laptop_air_0_norm
5 laptop_air_1_norm
5 laptop_dell_norm
6 mug_vignesh_norm
6 mug2_scene3_norm
6 mug_white_green_norm
1 bottle_shengjun_norm
1 bottle_red_stanford_norm
1 bottle_shampoo_norm
2 bowl_shengjun_norm
2 bowl_blue_white_chinese_norm
2 bowl_white_small_norm
3 camera_canon_len_norm
3 camera_shengjun_norm
3 camera_canon_wo_len_norm
4 can_green_norm
4 can_lotte_milk_norm
4 can_arizona_tea_norm
5 laptop_air_xin_norm
5 laptop_alienware_norm
5 laptop_mac_pro_norm
6 mug_daniel_norm
6 mug_anastasia_norm
6 mug_brown_starbucks_norm
/usr/local/lib/python3.5/dist-packages/torch/nn/functional.py:52: UserWarning: size_average and reduce args will be deprecated, please use reduction='elementwise_mean' instead.
warnings.warn(warning.format(ret))

The program has been stopped here and no error has been reported. Can you give me some suggestions? Thank you very much.

D:\Users\xsf\PycharmProjects\6-PACK-master\libs\transformations.py:1912: UserWarning: failed to import module transformations
warnings.warn('failed to import module %s' % name)
Traceback (most recent call last):
File "D:/Users/xsf/PycharmProjects/6-PACK-master/train.py", line 18, in
from libs.loss import Loss
File "D:\Users\xsf\PycharmProjects\6-PACK-master\libs\loss.py", line 11, in
from libs.knn.init import KNearestNeighbor
File "D:\Users\xsf\PycharmProjects\6-PACK-master\libs\knn_init.py", line 7, in
from libs.knn import knn_pytorch as knn_pytorch
File "D:\Users\xsf\PycharmProjects\6-PACK-master\libs\knn\knn_pytorch_init_.py", line 3, in
from ._knn_pytorch import lib as _lib, ffi as _ffi
ModuleNotFoundError: No module named 'libs.knn.knn_pytorch._knn_pytorch'

Process finished with exit code 1

Good afternoon, thank you for your work, I would very much like to conduct training, but today the link to the datasets does not work(
Maybe you saved the datasets on google drive or know some other workaround?
I also tried downloading as recommended in hughw19/NOCS_CVPR2019#59

Hi, thanks for sharing your code. I have some question about the the L_rot of Eq(4) in your paper. To calculate the error between two rotation matrix, most paper use the equation of theta = arccos((trace(R_gt^T · R_est)-1) /2) to evaluate the angular distance. I cannot find something related mathematic literature refers to this equation of Equation(4) in your paper, could you please share some details or related mathematic literature explaining this Eq(4)? I have struggled to understand this formula for a long time.

1. I followed the procedure of the datasets downloading and arrangement, but when I run data_preprocess.py, I got the problem:
   
   So is it OK to ignore the missed files?

2. I ignored the missed files, it seems some files are lost as below:

../My_NOCS/data/train/18690/0002_meta.txt
../My_NOCS/data/train/18690/0003_mask.png
../My_NOCS/data/train/18690/0009_mask.png
../My_NOCS/data/train/00469/0000_mask.png
../My_NOCS/data/train/00469/0001_mask.png
../My_NOCS/data/train/00469/0002_mask.png
../My_NOCS/data/train/00469/0004_mask.png
../My_NOCS/data/train/00469/0005_mask.png
../My_NOCS/data/train/00469/0007_depth.png
../My_NOCS/data/train/00469/0009_depth.png

3. After ignoring the missed files and preprocessing them, I run train.py in the pytorch0.4.1 environment, but the training process stuck on the 1st epoch when train_count = 5000:
   

So are there any suggestions? Thanks!

Hello, thank you for sharing this code.

I could run eval.py on NOCS real_val DB + your custom GT labels. But I got an error when running benchmark.py

No such file or directory: '/data/My_NOCS/data_obj/real_val/scene_4/0000_obj.txt'
I found only xxx.obj files in obj_models folder, not xxx_obj.txt files. what file should I use? and where can I find it?

Please give me any hint.

Thank you.

Hello.
I'm trying to get inference.py to work.

I would like a sample of the data you are using in lines 31 and 32（in inference.py）, is there somewhere I can find it?
Data used in rgb/.png and depth/.npy.

Traceback (most recent call last):
File "/home/xsf/Documents/6-PACK-master/benchmark.py", line 265, in
main()
File "/home/xsf/Documents/6-PACK-master/benchmark.py", line 238, in main
all_score[-1].append(cls_in_5_5/cls_num)
ZeroDivisionError: division by zero

Hi, thanks for your great work.
In the evaluation step, I have generated the estimated pose results of each frame by eval.py.
When I try to get the final results using benchmark.py, I got the error of IndexError. Could you tell me how to solve it?

python benchmark.py 
3
bowl_blue_white_chinese_norm
bowl_shengjun_norm
bowl_white_small_norm
Traceback (most recent call last):
  File "benchmark.py", line 271, in <module>
    main() 
  File "benchmark.py", line 206, in main
    gt_pose = result['gt_RTs'][num_idx]
IndexError: index 4 is out of bounds for axis 0 with size 4

Thanks for sharing good work.
I have some simple questions about your performance.

Q1. Can I ask how to implement the with temporal prediction??
As I understand, "eval.py" was implemented without temporal prediction.
I want to compare w/o, w temporal prediction performance.

Q2. I tested twice using pretrained weight and downloaded your onedrive folder. (https://drive.google.com/file/d/1WTarlYvObx5S6kPcGYP0k0KRvlrBCYET/view?usp=sharing)

Those performance were similar but some objects(bowl, laptop) are not close in the paper in the 5 degree and 5cm metric. Maybe random noise effect some results, but some object performance gap were high than I expected.
Could you share your opinion about why bowl and laptop are lower than paper report??

(I used the following instruments about your method)
python eval.py 
python benchmark.py

Hi, thank you for your work! When I run python benchmark.py, the following occurred:

1
bottle_red_stanford_norm
benchmark.py:102: RuntimeWarning: invalid value encountered in arccos
  theta = np.arccos(y1.dot(y2) / (np.linalg.norm(y1) * np.linalg.norm(y2)))
bottle_shampoo_norm
bottle_shengjun_norm
bowl_shengjun_norm
benchmark.py:116: RuntimeWarning: invalid value encountered in arccos
  theta = np.arccos((np.trace(R) - 1) / 2)
bowl_blue_white_chinese_norm
bowl_white_small_norm
Traceback (most recent call last):
  File "benchmark.py", line 264, in <module>
    main() 
  File "benchmark.py", line 199, in main
    gt_pose = result['gt_RTs'][num_idx]
IndexError: index 4 is out of bounds for axis 0 with size 4

Can you give me some suggestions? Thank you very much.

Hi, thanks for sharing the code.

I tried to run the benchmark.py but it requires pose information in real_val/scene_*/ dictionary. I downloaded NOCS-REAL275-additional.zip, but it seems that it only contains pose information for real_train dataset.

Could you share the pose information for real_val dataset or could you tell me how can I get the information by myself?

Thank you so much!

Hi, thank you for sharing the great code.
I want to use inference.py in a real robot, and I tried to run the inference.py, but I cannot understand what #provide the initial pose and scale of the object mean in the comment? I wonder how could I get or calculate the initial pose and scale?

Could you provide more details about how to get the initial pose and scale of the target object?

Thank you very much!

"List.txt" contains the path of "0204_meta.txt". The file (0204_meta.txt)with serial number 1 in the figure has 5 instances, while the GT file(results_real_test_scene_1_0204.pkl) with serial number 2 in the figure has only 4 instances.So this error occurred

0204_meta.txt:

print "results_real_test_scene_1_0204.pkl":

Hi, thank you for sharing the code. I am trying to run the code on my computer. I met some problem when I trying to understand the code.

In the line 112 of tracker.py, it use 'self.Kp_fr' and Kp_to[idx] to get the change of pose between keypoints of first frame and keypoints of current frame (they are new_t and new_r in the following code). self.Kp_fr is the keypoints of first frame because it haven't been changed since the first frame had been processed. Then the code use the pose of last frame (current_t and current_t in the code) and new_t and new_r to get the pose of current frame. Am I right about the code?

        self.min_dis = 1000.0
        lenggth = len(Kp_to)
        for idx in range(lenggth):
            new_r, new_t, kp_dis = self.criterion.inf(self.Kp_fr, Kp_to[idx])
            if self.min_dis > kp_dis:
                self.min_dis = kp_dis
                best_r = new_r
                best_t = new_t
        print(self.min_dis)

        current_t = current_t + np.dot(best_t, current_r.T)
        current_r = np.dot(current_r, best_r)
        current_t = self.init_t + np.dot(best_t, self.init_r.T)
        current_r = np.dot(self.init_r, best_r

Should the self.Kp_to be changed to the keypoints of last frame?

Hi, thank you for your work. When I run python train.py, the following happens

Traceback (most recent call last):
  File "train.py", line 57, in <module>
    for i, data in enumerate(dataloader, 0):
  File "/home/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 314, in __next__
    batch = self.collate_fn([self.dataset[i] for i in indices])
  File "/home/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 314, in <listcomp>
    batch = self.collate_fn([self.dataset[i] for i in indices])
  File "/home/6-PACK/dataset/dataset_nocs.py", line 438, in __getitem__
    img_fr, choose_fr, cloud_fr, r_fr, t_fr, target_fr, mesh_pts_fr, mesh_bbox_fr, mask_target = self.get_frame(choose_frame[0], choose_obj, syn_or_real)
  File "/home/6-PACK/dataset/dataset_nocs.py", line 331, in get_frame
    back_img = np.transpose(back_img, (2, 0, 1))
  File "/home/.local/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 575, in transpose
    return _wrapfunc(a, 'transpose', axes)
  File "/home/.local/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 52, in _wrapfunc
    return getattr(obj, method)(*args, **kwds)
ValueError: axes don't match array

This seems to be because the COCO dataset contains grayscale images. I fixed it by adding. convert ("RGB"). But another problem arises.

Traceback (most recent call last):
  File "train.py", line 57, in <module>
    for i, data in enumerate(dataloader, 0):
  File "/home/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 314, in __next__
    batch = self.collate_fn([self.dataset[i] for i in indices])
  File "/home/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 314, in <listcomp>
    batch = self.collate_fn([self.dataset[i] for i in indices])
  File "/home/6-PACK/dataset/dataset_nocs.py", line 441, in __getitem__
    img_to, choose_to, cloud_to, r_to, t_to, target_to, _, _, _, = self.get_frame(choose_frame[1], choose_obj, syn_or_real)
  File "/home/6-PACK/dataset/dataset_nocs.py", line 383, in get_frame
    choose = ((depth.flatten() != 0.0) * choose_temp).nonzero()[0]
ValueError: operands could not be broadcast together with shapes (128000,) (117656,) 

This problem has bothered me for a long time. Can you give me some suggestions? Thank you very much.

Thank you very much for your art work.

I'm struggling to track an object like "mug" using your inference code with my webcam + depth sensor(like real sense).
And I'm getting the initially estimated R/T value using DenseFusion.
I got runtime error because of wrong next R/T refined value during tracker.init_estimation(...) call.
I guess the cause of this error comes from the wrong inital bbox values which must have 8 x,y,z kpts(torch.size[8,3]).

Refering dateset_nocs.py and dataset/data_preprocess.py code(using model_scales/*.txt), I currently using the same pre-defined 3D model coordinates(bbox) of "mug" as one of NOCS models( model_scales/mug_brown_starbucks_norm.txt)

==================================
coord=[55.0, 48.0, 38.0],
bbox=[[coord[0], coord[1], coord[2]],
[coord[0], coord[1], -coord[2]],
[coord[0], -coord[1], coord[2]],
[coord[0], -coord[1], -coord[2]],
[-coord[0], coord[1], coord[2]],
[-coord[0], coord[1], -coord[2]],
[-coord[0], -coord[1], coord[2]],
[-coord[0], -coord[1], -coord[2]]]

to call current_r, current_t = tracker.init_estimation(current_r, current_t, "bbox",..)

current_r,current_t comes from "DenseFusion's output"

and of course I modified cam_cx,cam_cy, cam_fx, cam_fy to fit to my sensor's intrinsic parameters.

Plus, I dont understand why there is a code like

target_tmp[:, 0] *= -1.0
target_tmp[:, 1] *= -1.0

it seems like the shape should be upside down and turn opposite horizontally, but why?
I guess maybe the actual camera which is used to make NOCS dataset is set upside down...right?

Please give me any hint or example or guide for running your inference code.
Thank you very much.

ty very much, if you can provide the code that generate labels

I see 'model_scales' dir has many .txt contains 2*3 matrix, I don't know what's that mean?

There is no _bbox.txt file in the original NOCS dataset.

Is there a way to recover the camera poses for the images or are they saved somewhere?

Hi, thank you for your work! The code runs fine in training. But pause for validation. This does not seem to be caused by the "try-except" code in the testdataloader. When I on CTRL+C shutdown I get this:

Traceback (most recent call last):
  File "train.py", line 96, in <module>
    for j, data in enumerate(testdataloader, 0):
  File "/home/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 330, in __next__
    idx, batch = self._get_batch()
  File "/home/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 309, in _get_batch
    return self.data_queue.get()
  File "/usr/lib/python3.6/multiprocessing/queues.py", line 335, in get
    res = self._reader.recv_bytes()
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
KeyboardInterrupt

What might be causing this problem? Thank you very much.

Thanks for sharing good work.
I have some simple question about your implementation.

Q1. Some object meta data and pose gt not correctly matched in the dataset.
I think that the problem comes from the difference between meta data and pose label.
For example,
"nocs/real_test/scene_1/0203_color.png" there are only have 4 class, but in the "meta.txt", they have 5 class.
And in the "My_NOCS/data/gts/real_test/results_real_test_scene_1_0203.pkl", they only have 4 pose labels.
Can I ask how to handle this part??

Q2. What's difference between 5cm 5degree and Mean 5cm 5degree ??
Does Mean 5cm 5degree was used for 5 times evaluation??
And could you explain detailed whether mAP or not??, I'm still confused what is different.
#15 (comment)

[Terminal Output]

**bowl_white_small_norm**
index error 4     My_NOCS/data/gts/real_test/results_real_test_scene_1_0203.pkl
index error 4     My_NOCS/data/gts/real_test/results_real_test_scene_1_0204.pkl
**camera_canon_len_norm**
index error 4     My_NOCS/data/gts/real_test/results_real_test_scene_1_0198.pkl
index error 4     My_NOCS/data/gts/real_test/results_real_test_scene_1_0199.pkl
**mug_daniel_norm**
index error 4     My_NOCS/data/gts/real_test/results_real_test_scene_1_0200.pkl
index error 4     My_NOCS/data/gts/real_test/results_real_test_scene_1_0201.pkl
index error 4     My_NOCS/data/gts/real_test/results_real_test_scene_1_0202.pkl

********************************************************
5cm 5degree: 31.86850232958812
IoU 25:      96.40904934750537
IoU 50:      73.21566899557062
rot error:   18.560087960402228
tran error:  3.4760268421587646
********************************************************
Mean 5cm 5degree: 31.86850232958812
Mean IoU 25:      96.40904934750537
Mean IoU 50:      73.21566899557062
Mean rot error:   18.560087960402228
Mean tran error:  3.4760268421587646
********************************************************

Hi @j96w ,
This is really a nice work,
But it seems that I can't find the real_val's pose information, in other words, I can't find the directory: "My_NOCS/data_pose/real_val"

Hi.
Thanks for sharing the code.

In step 1 of the Dataset part of Read me, it said to download four datasets from the NOCS dataset.
However, when I download those four files, I don't have the "real_val" file mentioned in step 2.
Has the dataset been renamed?

CAMERA Dataset Training--------unzip-------→train/
Real Dataset Training--------unzip-------→real_train/
Real Dataset Test --------unzip-------→real_test/
Ground truth pose annotationVal&Real_test--------unzip-------→gts/

I fix the bugs following the previous ISSUEs, however, it still occurs:

FileNotFoundError: [Errno 2] No such file or directory: 'My_NOCS/data_pose/real_val/scene_3/0515_pose.txt'

I wonder how to get the pose.txt of real_val. Cause there is no data_pose/real_val in https://github.com/hughw19/NOCS_CVPR2019 or NOCS-REAL275-additional.zip.

Thanks for your kind help! @j96w @LyuJ1998

Hi, You mentioned in the readme "the ground truth of real training data generated by this method is not good enough." Is the GT pose in real_val accurate? When I run the NOCS detection code, the GT pose visualization image will be generated as shown below. It seems that the GT pose is not accurate. Have you encountered this problem?

my environment:

1. Ubuntu
2. Pytorch 0.4.1
3. Python 3.5

Does anyone encounter this problem? Thanks.