Viewing outputs in SIBR viewer about 4dgaussians

[aleatorydialogue]

any more luck getting this to work? It says connected but then immediately connection dropped, over and over...

[weihaosky]

any more luck getting this to work? It says connected but then immediately connection dropped, over and over...

Same problem. Anyone can help? Many thanks!

[leo-frank]

any more luck getting this to work? It says connected but then immediately connection dropped, over and over...

Same problem. Anyone can help? Many thanks!

I meet the same problem.
But when I training using https://github.com/graphdeco-inria/gaussian-splatting, i can use SIBR_remoteGaussian.app to inspect the training process.
It's wierd to meet connection dropped when training using 4Dgaussian

[guanjunwu]

@henrypearce4D @Zhentao-Liu
Hi, this is a local real-time viewer which can supervise training process

My command:

1. Training with lego python train.py -s data/dnerf/lego --port 6068 --expname "dnerf/lego" --configs arguments/dnerf/lego.py
2. Run viewer SIBR_viewers/install/bin/SIBR_remoteGaussian_app_rwdi.exe --path E:/code/4DGaussians/data/dnerf/lego --ip 127.0.0.1 --port 6068
   Note: I test it on Windows with RTX 3080 GPU.
   (Further complete version will update soon)
   https://github.com/hustvl/4DGaussians/assets/87054407/be80a0ff-d162-409c-af78-1ca3f8aee5fd

[EarltShirt]

Thanks a lot :) It all works now . Sorry again

[guanjunwu]

I have not time to expand the SIBR viewer and edit the rendering time locally. I only set the timestamp in the code like this:

    video_cams = scene.getVideoCameras()

    for iteration in range(first_iter, final_iter+1):        
        if network_gui.conn == None:
            network_gui.try_connect()
        while network_gui.conn != None:
            try:
                net_image_bytes = None
                custom_cam, do_training, pipe.convert_SHs_python, pipe.compute_cov3D_python, keep_alive, scaling_modifer = network_gui.receive()
                if custom_cam != None:
                    count +=1
                    viewpoint_index = (count ) % len(video_cams)
                    if (count //(len(video_cams))) % 2 == 0:
                        viewpoint_index = viewpoint_index
                    else:
                        viewpoint_index = len(video_cams) - viewpoint_index - 1
                    # print(viewpoint_index)
                    viewpoint = video_cams[viewpoint_index]
                    custom_cam.time = viewpoint.time
                    print(custom_cam.time, viewpoint_index, count)
                    net_image = render(custom_cam, gaussians, pipe, background, scaling_modifer, stage=stage)["render"]
                    net_image_bytes = memoryview((torch.clamp(net_image, min=0, max=1.0) * 255).byte().permute(1, 2, 0).contiguous().cpu().numpy())
                network_gui.send(net_image_bytes, dataset.source_path)
                if do_training and ((iteration < int(opt.iterations)) or not keep_alive) :
                    break
            except Exception as e:
                print(e)
                network_gui.conn = None

[henrypearce4D]

Could you tell me, the SIBR Remote Gaussian Viewer shown above, is that the original code from Inria or modified for playback of 4D data? Can the Viewers for Windows (60MB) work?

Also, I tested running the Remote viewer while training, it connected but did not display any window in the GUI.

[guanjunwu]

Yes. It only works in dnerf dataset. I test it in the Windows11 with RTX 3080 GPU. Other format of dataset(dynerf,hypernerf)needs add correct pose loader in the origin code of SIBR viewer. I've not finished it yet.
Besides, the rendering speed on the remote viewer may not be as fast as you imagine. Because network speed also limits the upper bound.

[henrypearce4D]

I tried this command while the bouncing balls dnerf were training;

C:\gaussian-splatting\SIBR_viewers_win\bin\SIBR_remoteGaussian_app.exe --port 6017

The GUI opens and says connected but no viewer window appears, are there any other arguments I need to use?

[guanjunwu]

maybe you can add the --path like this?
I also download datasets locally then it works..
./SIBR_viewers/install/bin/SIBR_remoteGaussian_app_rwdi.exe --path E:/code/gaussian-splatting/data/dnerf/standup --ip 127.0.0.1 --port 6017

[henrypearce4D]

Does this only work while training or can it be used to loaded pretrained dnerfs?

[henrypearce4D]

no luck, it doesnt work for me

[Zhentao-Liu]

I did it, I use the following command: SIBR_gaussianViewer_app.exe -m E:\code\4DGaussians\output\dnerf\lego --path E:\code\4DGaussians\data\dnerf\lego --iteration 20000, then I could use the viewer as in gaussian-splatting, but the rendering quality is quite poor.

[henrypearce4D]

@Zhentao-Liu thanks I was able to view the scenes I trained with that command and see the same artifacts, the models I trained were trained with the first release of code, was yours more recent?

Also, did you find a way to play the 4D sequence? is there a timeline within SIBR that you found?

[Zhentao-Liu]

@Zhentao-Liu thanks I was able to view the scenes I trained with that command and see the same artifacts, the models I trained were trained with the first release of code, was yours more recent?

Also, did you find a way to play the 4D sequence? is there a timeline within SIBR that you found?

Yes, I use the code more recent, but I think there is no core difference.
Also, I do not find a way to play the 4D sequence. Hope some day the authors will release a new viewer that suits for 4DGS.

[guanjunwu]

@Zhentao-Liu Hi, do you have any problems reproducing the results of my paper?..
I ran the code in the Ubuntu system. I never tested the real-time viewer in Ubuntu. But I also encountered the same rendering quality when training with running real-time viewer at the same time (Even CUDA Loss becomes NAN or CUDA ERROR) in Windows. But if I don't open it, the rendering quality will be similar to my paper.
Here is a viewer demo provided by yzslab, you can try it.
I apologize for the problem bothering you, and I'm fixing the BUG in 4DGS, I'll provide a new viewer suit for 4DGS soon.

[henrypearce4D]

@guanjunwu this sounds great! I will make sure to test this week!

[Huotianci]

@henrypearce4D @Zhentao-Liu Hi, this is a local real-time viewer which can supervise training process

My command:

1. Training with lego python train.py -s data/dnerf/lego --port 6068 --expname "dnerf/lego" --configs arguments/dnerf/lego.py
2. Run viewer SIBR_viewers/install/bin/SIBR_remoteGaussian_app_rwdi.exe --path E:/code/4DGaussians/data/dnerf/lego --ip 127.0.0.1 --port 6068
   Note: I test it on Windows with RTX 3080 GPU.
   (Further complete version will update soon)
   https://github.com/hustvl/4DGaussians/assets/87054407/be80a0ff-d162-409c-af78-1ca3f8aee5fd

Where is SIBR_remoteGaussian_app_rwdi.exe? Can you help me?

[Tejasdavande07]

@guanjunwu this sounds great! I will make sure to test this week!

hey you tested this?

please provide the link for downloading the viewer and also provide the commands use for run viewer.

[Huotianci]

@guanjunwu this sounds great! I will make sure to test this week!

hey you tested this?

please provide the link for downloading the viewer and also provide the commands use for run viewer.

I also want to know!
I am a newbie. Here are one material sent to me by the author. You can give it a try. But I haven't succeeded yet. If you succeed, can you share the process despite your busy schedule? Thank you so much!

#68

[EarltShirt]

Hey @guanjunwu,
Is the viewer also able to render other models during training, for example dynerf or hypernerf scenes or is it only for viewing dnerf datasets ?

[guanjunwu]

Hey @guanjunwu, Is the viewer also able to render other models during training, for example dynerf or hypernerf scenes or is it only for viewing dnerf datasets ?

Hi, the viewer is based on 3DGS's viewer. The rendering process is modified from my code.
If you want to view the result, you can following the scripts:

python train.py -s data/hypernerf/virg/virg-3dprinter --port 6017 --expname "hypernerf/3dprinter" --configs arguments/hypernerf/3dprinter.py 
./viewers/bin/SIBR_remoteGaussian_app.exe --port 6017 --path data/dnerf/lego

This path only decide the camera intrinsics and extrinsics. And it is a straightforward solution. After viewer is initialized, you can rotate the camera to find the correct position.
Sorry for that, I have no time to develop a brand new viewer. maybe it can help you.

[EarltShirt]

Thanks for your great work ! And I'm sorry for the time consumption.
It works well, i wasn't patient enough and didn't wait for some results to show up.
Thanks a a lot !

[EarltShirt]

Last quetstion, is it possible to view the trained model using you viewer ?

Running ./viewers/bin/SIBR_gaussianViewer_app.exe -m output/hypernerf/virg/3dprinter leads to the following :

[SIBR] --  INFOS  --:   Initialization of GLFW
[SIBR] --  INFOS  --:   OpenGL Version: 4.6.0 NVIDIA 536.23[major: 4, minor: 6]
[SIBR] ##  ERROR  ##:   FILE C:\projects\gauss2\SIBR_viewers\src\projects\gaussianviewer\apps\gaussianViewer\main.cpp
                        LINE 140, FUNC main
                        Could not find config file 'cfg_args' at output/hypernerf/virg/3dprinter

Even though the 'cfg_args' file is present in the given directory. (using the remoteGaussian_app.exe leads to a black screen in the viewer)

[guanjunwu]

Last quetstion, is it possible to view the trained model using you viewer ?

Running ./viewers/bin/SIBR_gaussianViewer_app.exe -m output/hypernerf/virg/3dprinter leads to the following :

[SIBR] --  INFOS  --:   Initialization of GLFW
[SIBR] --  INFOS  --:   OpenGL Version: 4.6.0 NVIDIA 536.23[major: 4, minor: 6]
[SIBR] ##  ERROR  ##:   FILE C:\projects\gauss2\SIBR_viewers\src\projects\gaussianviewer\apps\gaussianViewer\main.cpp
                        LINE 140, FUNC main
                        Could not find config file 'cfg_args' at output/hypernerf/virg/3dprinter

Even though the 'cfg_args' file is present in the given directory. (using the remoteGaussian_app.exe leads to a black screen in the viewer)

Sorry, the viewer hasn't supported it yet.
Instead, You can set the larger iterations, load the trained model's checkpoint, and disable the training process.

[EarltShirt]

Thanks for your answer. When loading the model using:
python train.py -s data/dynerf/cut_roasted_beef --port 6017 --expname "dynerf/cut_roasted_beef" --configs arguments/dynerf/cut_roasted_beef.py --start_checkpoint "output/dynerf/cut_roasted_beef/chkpnt_fine_14000.pth" , the training starts but with 0it and is completed after a few seconds. Even when trying to call the visualizer during this period only a black screen appears in the viewer ( using this for instance ./viewers/bin/SIBR_remoteGaussian_app.exe --port 6017 --path data/dnerf/lego). Those where the only lines of code I found in the readme in order to try and replicate your instructions. Maybe I'm doing it wrong... How does one disable the training process , the only argument suitable would be --skip_train when running the render.py script.

[guanjunwu]

Thanks for your answer. When loading the model using: python train.py -s data/dynerf/cut_roasted_beef --port 6017 --expname "dynerf/cut_roasted_beef" --configs arguments/dynerf/cut_roasted_beef.py --start_checkpoint "output/dynerf/cut_roasted_beef/chkpnt_fine_14000.pth" , the training starts but with 0it and is completed after a few seconds. Even when trying to call the visualizer during this period only a black screen appears in the viewer ( using this for instance ./viewers/bin/SIBR_remoteGaussian_app.exe --port 6017 --path data/dnerf/lego). Those where the only lines of code I found in the readme in order to try and replicate your instructions. Maybe I'm doing it wrong... How does one disable the training process , the only argument suitable would be --skip_train when running the render.py script.

you can open the viewer at first, and change iterations in arguments/dynerf/default.py to 14010, then launch the training process :)