how to fit to rgbd data about 3dmm-fitting-pytorch

[ascust]

Hi, it seems like adding a depth channel would enable 3D landmark fitting. The current landmark loss only assumes 2d landmark, so you might need to use a 3d landmark detector. Furthermore, for RGB pixel fitting, apart from fitting three color channels, you could also add a depth loss for each vertex projected to the image.

Not sure if I've made myself clear.

[rlczddl]

Thanks for your reply. In fact, what I want is to fit the point-to-point error between bfm 3d face shape and point cloud converted from depth image, not just the depth error between face key points.And now, referring to https://github.com/yuhaoooo/Deep3DFaceReconstruction-Pytorch/blob/7c506b55adee55bb269f73354dd16d5327a7fb04/reconstruction_mesh.py , I have some ideas.

[rlczddl]

Hi, I'm modifying the code to fit bfm to rgbd data. Unfortunately I ran into some problems. In the rigid fitting stage, I use the real camera parameter K from the calibration, and since the scale of the bfm face model is unknown, an additional scale parameter s is optimized. Now the projection equation of the 2d face keypoint is K(sR*p+T), but the optimization result of this step is always wrong. I don't know what went wrong. Do you have any suggestions? Here are some key codes:

[rlczddl]

After some debugging, I found that the initial z value on self.camera_pose must be set to an initial value instead of 0, which indicates the distance between the camera and the bfm model, so that the rigid part can be optimized. Why is this?

[ascust]

I am not really sure what is going on here. In terms of camera_pose z, I remember I set it to (0, 0, 10) and the camera is pointing at the origin (0, 0, -1). The reason for this is that the face mesh is supposed to be at the origin and given a proper focal length we can easily project the whole face on the image plane. If you put the camera at (0, 0, 0), it would be a bit weird because the camera and the mesh target are almost at the same point. Not sure if this is what you are talking about.

[rlczddl]

Thanks for your answer. After several days of debugging, the problem has been solved. Now I use the umeyama algorithm to initialize scale, R, T. After that, I encountered the problem mentioned by the issue facebookresearch/pytorch3d#334 . Finally, I solved the problem by dividing the vertex by a fixed scale factor.