pretrained results about pointnetgpd

[lianghongzhuo]

Could you share the code and pointcloud you are using? Otherwise, I can't know what's wrong here.

[trungpham2606]

can I have your mail ? @lianghongzhuo

[trungpham2606]

@lianghongzhuo
here is the pointcloud and my python code for testing
https://drive.google.com/drive/folders/1JlKvVFy84muuC2AvBBimDjkmID7k7vi3?usp=sharing

[lianghongzhuo]

Thank you for the files, I downloaded them. However, I'm busy catching up a deadline, I will check it when I am free.

[trungpham2606]

@lianghongzhuo
Thank you. Iam also trying to figure out what is happening when I trained YCB dataset with your pretrained, the loss is quite fluctuating.
I trained 1 object only.

[trungpham2606]

@lianghongzhuo
After making several checks, I think the problem comes from the pretrained model.
I also tested the pretrained model in my data and YCB data as well.
The result for my data is above, and here is for YCB:

Then the model also outputs all generated grasp poses are bad grasp.
I also loaded the model and made a training on YCB dataset, the loss is extremely fluctuating.

[trungpham2606]

@lianghongzhuo
Can you send me the old weights : pointgpd_chann3_local.model ?

[trungpham2606]

@lianghongzhuo
Can you provide the log file of training session ?

[lianghongzhuo]

sorry, I don't save the log file.
I quick looked commit log, e2fce0f
Seems I only uploaded one model. There is no "old model"

[trungpham2606]

@lianghongzhuo
Have you checked my scripts that I had sent to you ?

There's error in PointNetGPD/model/dataset.py, in collect_pc function, you missed the line for recalculate the grasp_bottom_center (grasp_bottom_center = -self.ags.gripper.hand_depth * approach + center).

But after changing it and making some inspections, I still cant work with your pretrained model.

[trungpham2606]

@lianghongzhuo
Had you taken a look at my scripts ?

[lianghongzhuo]

Sorry for the delay. I will check it today or tomorrow.

[trungpham2606]

@lianghongzhuo
oh tks ^^
I also changed the pointnet, and retrained the dataset (eliminate the wrong objects), the results are pretty good.
But I think it'll be better if I can use your pretrained weights.
Hope to hear from you soon !

[lianghongzhuo]

I checked your code. You have downsampled the input point cloud. But by downsample it . The object point cloud are also down sampled.
The best way is to transform the point cloud into desktop coordinate. Then clip all the point cloud that belong to the table.

[trungpham2606]

@lianghongzhuo
I had transformed the point cloud into desktop coordinate already. The problem here is I cant detect good poses with your provided pretrained weights but be able to detect by using my weights (trained by myself).
So can you check the pretrained weights again to verify whether or not it's working properly ?

[lianghongzhuo]

The downsample method is not correct, as the table has a lot of points, downsample all the points together will lose some geometric information. You can first crop the points on the table. By doing that you can keep most geometric information of the object.

[trungpham2606]

@lianghongzhuo
So what happened with the YCB objects ? without any points on table. The results are the same with no good poses ?
Actually the script I sent to you is for YCB objects, with real point cloud data I had considered only the points above the table (as you can see in my commented code in main.)

[lianghongzhuo]

I currently cannot access to the robot in the lab due to the lockdown. When I come back to the office, I can record some point cloud that I use in the experiment. Maybe you can compare the difference. The networks weights work well in my experiment setup.

[trungpham2606]

@lianghongzhuo
I think you dont have to test with real point cloud, just run my script I sent to you and see what's happening. The script I wrote for testing YCB object so that this model must work well on it.

[lianghongzhuo]

as I said, the problem in your script is that you downsampling all the point cloud in the ply file. the correct way to do downsampling is to first crop the table points and then downsample the object point cloud only.

[trungpham2606]

@lianghongzhuo
Opps the scripts I sent you was not fixed.
Iam gonna send you the script for testing on YCB object.
The script and the YCB object I used for testing is in this link
https://drive.google.com/drive/folders/1aAVS01R7Z3AiM_JqQAv1kfWjiVp7WjiY?usp=sharing
Can you take a look at it.

[trungpham2606]

@lianghongzhuo
The result is as below

[trungpham2606]

@lianghongzhuo
Excuse me but have you tested my script. Iam looking forward to your help.

[trungpham2606]

@lianghongzhuo

[lianghongzhuo]

Sorry for the late reply.
This model is not trained on the ycb google point cloud. The model is trained on the Raw RGBD data.
For example this LINK.
If you test on the ycb google point cloud (the laser scan) data, it might fail.
If you have a depth camera (kinect or realsense for example) you can try our model again to see the difference. I will also upload some example point cloud in the future.

[trungpham2606]

@lianghongzhuo
Sorry but what is the LINK that you had provided ? When I clicked on it, nothing happened.
Can you double check the LINK ?

[trungpham2606]

@lianghongzhuo
I will check the pretrained model with real objects again.

[lianghongzhuo]

http://ycb-benchmarks.s3-website-us-east-1.amazonaws.com/data/berkeley/003_cracker_box/003_cracker_box_berkeley_rgbd.tgz
the link is the download link for ycb rgbd data

[trungpham2606]

@lianghongzhuo
Thank you for the link.
1 more problem is that with boxes, I always meet this failure:

it should be bad pose instead of good pose.

Do you have any suggestions to cope with this problem ?

[trungpham2606]

@lianghongzhuo

[Twilight89]

@trungpham2606 Hi~ I want to ask that in the kinect2grasp.py,
kinect_data = rospy.wait_for_message("table_top_points", PointCloud2)
the kinect_data is based on kinect coordinate, right? So I can directly use /kinect_V2/depth/points topic published by kinect?
And in
real_grasp, points, normals_cal = cal_grasp(kinect_data, cam_pos)
the points here is also based on kinect coordinate(I checked the data and in my production, it's based on kinect coordinate)? I am confused because I send cam_pos in, hope to get points based on table_top coordinate.
I don't know why, leading to my result like follow:(I use rviz, failing to import mayavi )
Hope for you reply, thanks a lot!

[trungpham2606]

@Twilight89
According to author's code, the objects point cloud are on world coordinate. I didnt use ROS for visualizing but using real camera instead.

[Twilight89]

@trungpham2606 Thanks a lot for your reply. So points in
real_grasp, points, normals_cal = cal_grasp(kinect_data, cam_pos)
are on world coordinate? But kinect_data is on camera coordinate, do I need to tranform kinect_data(camera_frame) to points(world_frame).
I used to think points are on world coordinate, but I checked the data in my repo, they are on camera coordinate. I thought I have sent cam_pos in so the tranformation was done.
Briefly, now I should get points in world frame(now they are camera frame), is that your meaning?
Hope for reply~

[trungpham2606]

@Twilight89
Yes, you should get points in world frame.

[Twilight89]

@trungpham2606 thanks again~
I want to know in your repo, is kinect_data in camera frame? because in this line
cam_pos, _ = listener.lookupTransform('/table_top', '/kinect2_ir_optical_frame', rospy.Time(0))
we can get cam_pos, and send in
real_grasp, points, normals_cal = cal_grasp(kinect_data, cam_pos)
I used to think the tranformation was done due to I sent in cam_pos. If not, why we should get cam_pos?
Hoping for reply

[trungpham2606]

@Twilight89
Sorry but we didnt run on ROS, we test on real camera(realsense) and robot(UR5) directly. In our experiment, the pointcloud and cam_pos are both on world coordinates.

[Twilight89]

@trungpham2606
Sorry I am a little confused. Did you change something about 'data transform' after git this code. I think there is no difference between simulation and reality.
In my simulation repo, the cam_pos is on world coordinate, too. And I think in real camera, the directly data it got is on camera coordinate, too.
If this, I just found my points are on carema coordinate, which is unnormal, so did you do any transformation to change points to world frame?

[trungpham2606]

@Twilight89

1. I dont use simulation.
2. Yes I had to change the pointcloud from camera (camera cooridnate) to world coordinates.
3. Sorry that I dont know much about ROS.

[Twilight89]

@trungpham2606
Thanks a lot for your patience, I'll try it later. Hope your problem can be solved soon.

[trungpham2606]

@Twilight89
Thank you Twilight89