jiahuilei / mosca

MoSca: Dynamic Gaussian Fusion from Casual Videos via 4D Motion Scaffolds

Project Page | Video

The code will be released no later than the acceptance of the paper.

mosca's Issues

Question about optimizing camera poses

Hello,

Thanks for your fancy work. I'm interested in the dynamic 3D from a monocular video. While reading the paper, I was curious about the part about optimizing camera pose. Since the current Gaussian Splatting renderer does not support gradient propagation of camera poses, it seems that you are not optimizing camera poses with the l1 loss of the rendered image and GT, as shown in Eq. 14 and 15.

However, I'm a little confused about "Note that we will also optimize the camera poses Wt throughout the rendering phases as well with
photometric losses to further adjust the camera". I saw this was also mentioned in another issue. Does this mean that the camera pose is also optimized using Eq. 13 as loss? If so, have you implemented a renderer that can pass the camera pose gradient? If so, that's so cool.

Images split on Nvidia Dynamic datasets

Hi, thanks for your great work.

I notice that you follow roboust dynamic radiance field to split the nvidia dynamic datasets. But I found the data downloaded from roboust dynamic radiance field(https://github.com/facebookresearch/robust-dynrf) only contains 11 images for each scene. Is this true?

It seems different from the split from DynIbar (https://github.com/google/dynibar).

[Question about jointly the complete representation]

Dear authors,

Thanks for your impressive work!
In the "Full Pipeline" section, you mentioned that MoSca performs "Initialize the dynamic Gaussians G and jointly optimize the complete representation including the static background Gaussians, dynamic foreground Gaussians, the deformation motion graph, and cameras".

Will you still optimize the camera poses after the BA process in section 3.5?
Also, in section 3.5, you mention that "we jointly optimize a correction to depth Da[pa], consisting of per-frame global scaling factors and small per-pixel corrections". Can you provide more detailed descriptions for your depth alignment implementation and the camera poses training and representation?

videos with fixed viewing angle

Great work!

Is your method applicable to monocular videos with fixed viewpoint?
Thanks.

Implementation details on epipolar error map

Hi, thanks for your excellent work MoSca!

Currently I am trying to follow some techniques in MoSca to handle the monocular dynamic scene. And I meet a small problem, which I really hope that you could help me to solve it. Here is my question:

In MoSca, you use the epipolar error map to segment the dynamic region, which is really elegant. I found this part can be refered to the code in robust dynamic radiance field in CVPR2023. And when I use it to segment the dynamic part in nvidia dynamic datasets and iphone datasets, the results seem poor or instable. Here is some example in apple scene (iphone) and skating scene (nvidia). I guess that some parameters should be adjusted to adopt to these scenes. It would be so nice for you to share some experience.

[Rendering FPS]

Dear authors,

Thanks for your impressive work!
Can you provide detailed rendering FPS for both the iPhone and Nvidia datasets for comparison purposes?