change projection matrix to camera intrinsics about 2d-gaussian-splatting

[hbb1]

Hi, supporting a non-ideal-pinhole model is interesting. I will review your implementation and come back to you then.

By the way, many folks and I have made some progress toward this goal. From my understanding, we should be able to support this now, without modifying the cuda code but simply adapting a new projection matrix with shift given works have been done in b8c9128 and 21a32f1. You can also have a try.

[hbb1]

Also, ensure that your projection matrix is consistent with the original implementation when using the ideal pinhole model, which your model should be compatible with.

[CanCanZeng]

Hi @hbb1 , I just tried this commit b8c9128 today, but I found that the recovered intrinsics here is not consistent with the original parameters given in the colmap. You can take a break point here to check if the values of the intrins are completely consistent with the values given in the colmap file.

By the way, I don't intend to convince you to add my modification to your source code, because my modification will make the interface inconsistent with the original 3DGS. But the way of calculating the projection using the intrinsic matrix (K) is ​​more in line with the way of thinking of people who do computer vision, so I changed it to the form I am more accustomed to..

[hbb1]

The intrins can be recover from the projection matrix by doing so

ndc2pix = torch.tensor([
        [W / 2, 0, 0, (W) / 2],
        [0, H / 2, 0, (H) / 2],
        [0, 0, 0, 1]]).float().cuda().T
intrins = (view.projection_matrix @ ndc2pix)[:3,:3].T

I mean you should make sure you modification should work equally for an ideal pinhole model.

[hbb1]

You can also check this version that also use intrins hbb1/diff-surfel-rasterization@661d224

[CanCanZeng]

The intrins can be recover from the projection matrix by doing so

ndc2pix = torch.tensor([
        [W / 2, 0, 0, (W) / 2],
        [0, H / 2, 0, (H) / 2],
        [0, 0, 0, 1]]).float().cuda().T
intrins = (view.projection_matrix @ ndc2pix)[:3,:3].T

I mean you should make sure you modification should work equally for an ideal pinhole model.

Of course, both ideal and non-ideal pinhole camera models are supported.

You can also check this version that also use intrins hbb1/diff-surfel-rasterization@661d224

Thank you for the reminder. I am currently reviewing this part of the code.

[CanCanZeng]

I found the bug, a very inconspicuous one. Here https://github.com/CanCanZeng/diff-surfel-rasterization/blob/a463669255a6624537afeb1db1491c9ff64c21a3/cuda_rasterizer/backward.cu#L503, I should not use a local variable P, but the P declared outside. The code prompt of vscode is too weak. The variable is used without assigning a value. If it is under other IDE, there will be a warning.

[hbb1]

Ok! A very unnoticeable bug. That makes sense.