Pretrained models about multi_part_assembly

[Wuziyi616]

Hi, thanks for your interest in our work. Since there are so many settings in the experiments (train one model per category, train one model over all categories, different subsets, etc.), it's very hard to share the pre-trained weights. But if you take a look at #1, the user confirmed that by running our code + training commands, he is able to reproduce the results reported in the paper. So I think you can just train the model by yourself.

The training time is not long! Training 1 model on 1 category usually takes less than 6h. Training on all categories usually takes less than 2 days. Hope this information helps.

[Pterosaur-Yao]

Thank you for your prompt and useful advice. In addition, I found that the point cloud of the fractured part contains not only the surface points but also the internal points. Can you suggest some ways to remove the internal points during the data generation process?

[Wuziyi616]

I'm not sure actually, cause I haven't looked at the generated point clouds. The point clouds are sampled from fracture meshes (faces) right? So if I understand correctly, that means some meshes have faces inside the shape? Is that the reason according to your observation?

[Pterosaur-Yao]

yes, there are some meshes have faces inside, you can visualize some examples, like bottle in everyday.

[Wuziyi616]

Thanks for the details. I'm pinning the authors who work on the geometry part of the paper to take a look.

[Wuziyi616]

A quick question: you are saying the fracture meshes themselves have inner faces, not the mesh assembled from fractures have inner faces (which is natural)?

At first thought I think that is impossible, because our fracture simulation algorithm (Breaking Good) generates meshes without inner faces. Can you upload a mesh file with this issue as the attachment, and point out the ID (the path to it, basically something like everyday/Bottle/xxx), so that we can localize it faster? We did go through the meshes last year, and I don't think we observe such issues. Do you observe many such cases in our dataset?

[Pterosaur-Yao]

yes, i found the fracture meshes have inner faces, for example, everyday/Bottle/fdc47f5f8dff184830eaaf40a8a562c1/mode_9/piece_0.obj. The sampled point cloud is visualized in the attached image and there are a lot of points inside the shape.
I randomly select one fracture to visualize and do not know how many issues like this are in the dataset.

[Wuziyi616]

Thanks for the information. We look into the mesh you provided, and indeed there are inner faces. This is because of the fracture simulator we use, which creates mesh cages (See Section 3.5 of Breaking Good if you are interested), will introduce inner faces to the mesh in a pre-processing step. Then, after we break the shape into fractures, we forgot to remove these inner cage faces before exporting them.

Removing inner faces should be simple, and we are working on a script to convert the current data to the inner-face-free version (e.g. we can implement this post-processing step in the decompress.py, so that when you decompress a mesh, you also remove the inner faces before saving it to file). Another solution is to sample point clouds only from surface faces (Idk if this is easy to implement in Python without using those geometry libraries). Overall, thank you for pointing this issue out. We will fix it ASAP and let you know if we have any results.

[Wuziyi616]

OK, so we have a hotfix by updating the decompress.py (see this branch). The new meshes have (nearly) no inner faces. You can try it out (requires a new dependency, please do pip install gpytoolbox).

However, the decompression speed with the new script is 100x slower. We're still looking for a way to accelerate it. Expect to see updates in the following days.

EDIT: we have fixed the speed issue as well. Can you @Pterosaur-Yao kindly run our decompression code and check if the new meshes look all good? From our side, we believe we have removed the inner faces. We are now working on re-running the benchmark results.

[Pterosaur-Yao]

Ok, I will re-run the updated code and look forward to your new benchmark results.

[Pterosaur-Yao]

Now, we find that most of the inner faces have been removed. However, in some meshes, there may still be some inner faces

[Wuziyi616]

This is the reply from my teammate:

there can still be some, but getting rid of those would either be slow or we'd need to do it at the data generation step

From my re-running benchmarks (have just finished ~20%, and there are still more to run), removing internal points doesn't seem to have a clear effect on the final performance. This could be due to the use of PointNet encoder, which may eliminate the inner point features after the max-pooling operation. That means if we use e.g. DGCNN, the inner points can possibly cause large differences. But overall, I think keeping these minimal inner points is fine.

Can you share your thoughts here? Also if you want to further improve the results, can you point me to a mesh with inner faces again (but as I said, we cannot guarantee to fix it)?

[Pterosaur-Yao]

Intuitively, the internal points do not contribute to the pose estimation, but rather reduce the sampling rate of the surface. However, the good news is that there are now very few internal triangles (I have observed at most one), which should have little effect on the results.

[Wuziyi616]

Thanks for the information. I also believe that's the case. Closing the issue for now. Feel free to re-open if you have further concerns.

[Wuziyi616]

Update decompress.py to remove inner faces in this PR.

[Wuziyi616]

@Pterosaur-Yao I've re-run some of the results and presented them here. I think the differences are minor.

[Pterosaur-Yao]

I am confused by the slight degradation of the results after the update. Why doesn't the removal of the internal points make the results better? Can you give me your insight?

[Wuziyi616]

Yeah, I'm also a bit confused TBH... My understanding is that maybe inner points cause some "errors" in metrics evaluation. For example, when calculating CD, the surface points before and after transformation usually have large distances, while the inner points usually have small distances. Thus, under the same rotation prediction error, the CD computed on surface points is (much) larger than the inner points (on the other hand, the CD computed under the same translation error should be the same).

Think of a 3D half ball as shown in the figure, the black one is the prediction, while the red one is the GT. Say we have a surface point pair (green) and a inner point pair (blue). You can see that, under the same prediction error, the distance between inner points is much smaller than the surface points. It's true that CD is not strict pairwise point distances, but from this example, you can see why the existence of inner points may favors some metrics.

For translation and rotation, I think it could be because PointNet's max-pooling already discards all features of inner points. So removing them doesn't make much difference.

[Pterosaur-Yao]

Thanks for your feedback :)

[Pterosaur-Yao]

In the paper, you say "Meshes in each category are aligned to a canonical pose as the ground-truth assembly", we want to know what is the meaning of canonical pose for different objects. BTW, how about the size of different objects, and have you re-scale them into a unit sphere?

[Wuziyi616]

This is to eliminate the symmetry ambiguity. For example, all the vases will be z-axis up, and all the cups (if having a handle) will have their handles on the same side (e.g. left).

EDIT: as a reference, you can see ShapeNet paper (the everyday subset is taken from PartNet, and the majority of PartNet is derived from ShapeNet), Section 3.2, Rigid Alignments part.

[Pterosaur-Yao]

Thank you for the valuable information. Also, do you have any plans to update the results of Table 11 in the paper?

[Wuziyi616]

Will discuss with the team for the decision. Meanwhile, I think for now you can write something like "updated results after fixing a bug in data generation after email communications with the paper authors", if you want to put the new results as baselines in your paper.