How to train for the case of a real life object and blender-like camera setting about bundle-adjusting-nerf

[chenhsuanlin]

Hi @dedoogong, from your description I'm guessing there could be several issues:

1. The most critical is probably the pose initialization. BARF is still a local registration method, which means that the pose initializations have to be close enough to the underlying ground-truth poses. Since your multi-view data is object-centric and captured 360˚ spherically, I don't expect BARF to be able to make the cameras automagically "wrap" around the object from the same (all-identity) pose. Since you already know your capture configuration of the 72 viewpoints, it would be more realistic to initialize from the spherical angles you described. I would suggest using such poses to train a NeRF first to make sure it could at least get you some reasonable results, and then switch to BARF to see if it improves.
2. It sounds like you're turning the table and capturing the object every 15˚. If it's true, then the background would not be in correspondence, and BARF would have a hard time using the photometric cues for pose optimization. This wouldn't work even for the original NeRF (i.e. even when ground-truth poses are given).
3. Using camera intrinsics different from your actual sensor could have an impact, but it probably isn't the major issue. You could consider also optimizing the intrinsic parameters (e.g. focal lengths), as in NeRF-- or Self-calibrating NeRF.

Typically if you don't see signs of BARF converging in 20k steps, then it probably won't in the end either.
Hope these help!

[dedoogong]

Hi @chenhsuanlin ! thanks so much for your kind thoughtful reply!
I tried to use an estimated camera intrinsics(focal length) from colmap but still failed.
Maybe, as you pointed out, it's too hard for Barf to optimize the identical, init pose to all around, spherical posese from scratch.
I agree your opinions(1,3) and I will try to find a better initial pose manually even though is would require a lot of trial and error.
Thank you!