Camera Pose Optimization with Geometry Prior about bundle-adjusting-nerf

[satyajitghana]

Soft Rasterizer seems promising, saw an example of pose optimisation of cube, I'll try it out on my model !

[chenhsuanlin]

Hi @satyajitghana, I'm not sure how you're incorporating 3D meshes into the NeRF/BARF framework; this seems nontrivial to me. If you're building your work upon PyTorch3D, I would suggest contacting the PyTorch3D team for issues.

[satyajitghana]

I'm using a differentiable renderer to render the scene and then comparing with the camera image. This process didn't work.

I haven't incorporated 3D mesh into NeRF, i was hoping you could guide me on how it could be done.

[chenhsuanlin]

One possible way of utilizing a given 3D mesh in NeRF is to render the mesh depth and use it as a prior for volume rendering sampling, in which case would be (almost) equivalent to mesh rendering if the samples were fully concentrated on the mesh surfaces. Therefore, if you're working on a setup where BARF is expected to work (e.g. with sufficiently well-initialized poses), then I would expect PyTorch3D to work as well. If you're looking specifically for a coarse-to-fine solution for mesh-based inverse optimization, I would suggest looking into Soft Rasterizer, which is the baseline PyTorch3D improves upon.