jonathonluiten / dynamic3dgaussians Goto Github PK

I make my dataset with myself without background(I don't want to reconstruction background)，what's the color of background should be? black or white?is this any influence?

Hi,

I am interested in the data that you mentioned. Is it possible to release the preprocessed data anytime soon?

Thanks!

Hi, everyone!

I want to rotate my pretrained gaussian model, I have rotated the _xyz and _rotation of gaussian splats, but the rendering results got worse! The problem I guess is that the spherical harmonics (_featrues_dc and _features_rest) also need to be rotated, so how to rotate the spherical harmonics?

UnsatisfiableError: The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versions

Package _python_rc conflicts for:
pip=22.3.1 -> python[version='>=3.7'] -> _python_rc
tqdm=4.65.0 -> python[version='>=3.7'] -> _python_rc

Package openblas conflicts for:
torchaudio=0.12.1 -> numpy[version='>=1.11'] -> openblas[version='0.2.20|0.2.20.|>=0.2.20,<0.2.21.0a0|>=0.3.3,<0.3.4.0a0']
torchvision=0.13.1 -> numpy[version='>=1.11'] -> openblas[version='0.2.20|0.2.20.|>=0.2.20,<0.2.21.0a0|>=0.3.3,<0.3.4.0a0']

Package ucrt conflicts for:
pytorch=1.12.1 -> cudatoolkit[version='>=11.3,<11.4'] -> ucrt[version='>=10.0.20348.0']
pillow=9.4.0 -> ucrt[version='>=10.0.20348.0']
cudatoolkit=11.6 -> ucrt[version='>=10.0.20348.0']
torchaudio=0.12.1 -> cudatoolkit[version='>=11.6,<11.7'] -> ucrt[version='>=10.0.20348.0']
torchvision=0.13.1 -> cudatoolkit[version='>=11.6,<11.7'] -> ucrt[version='>=10.0.20348.0']
python=3.7.13 -> openssl[version='>=1.1.1q,<1.1.2a'] -> ucrt[version='>=10.0.20348.0']
tqdm=4.65.0 -> python[version='>=3.7'] -> ucrt[version='>=10.0.20348.0']
pip=22.3.1 -> python[version='>=3.7'] -> ucrt[version='>=10.0.20348.0']

Package pip conflicts for:
pytorch=1.12.1 -> python[version='>=3.7,<3.8.0a0'] -> pip
python=3.7.13 -> pip
torchaudio=0.12.1 -> python[version='>=3.9,<3.10.0a0'] -> pip
torchvision=0.13.1 -> python[version='>=3.9,<3.10.0a0'] -> pip
pillow=9.4.0 -> python[version='>=3.10,<3.11.0a0'] -> pip
tqdm=4.65.0 -> python[version='>=3.7'] -> pip
pip=22.3.1

Package vs2015_runtime conflicts for:
cudatoolkit=11.6 -> vs2015_runtime[version='>=14.16.27033|>=14.29.30139']
cudatoolkit=11.6 -> vc[version='>=14.2,<15'] -> vs2015_runtime[version='>=14.15.26706|>=14.16.27012|>=14.27.29016|>=14.28.29325|>=14.32.31332|>=14.34.31931|>=14.29.30037']

Package pypy3.8 conflicts for:
pillow=9.4.0 -> pypy3.8[version='>=7.3.11|>=7.3.9']
tqdm=4.65.0 -> python[version='>=3.7'] -> pypy3.8[version='7.3.11.|7.3.9.|7.3.8.']
torchvision=0.13.1 -> numpy[version='>=1.11'] -> pypy3.8[version='7.3.11.|>=7.3.11|>=7.3.9|>=7.3.8|7.3.9.|7.3.8.']
pillow=9.4.0 -> python[version='>=3.8,<3.9.0a0'] -> pypy3.8[version='7.3.|7.3.11.|7.3.9.|7.3.8.']
pytorch=1.12.1 -> python[version='>=3.8,<3.9.0a0'] -> pypy3.8[version='7.3.11.|7.3.9.|7.3.8.|>=7.3.9|>=7.3.8|>=7.3.11']
pip=22.3.1 -> python[version='>=3.7'] -> pypy3.8[version='7.3.11.|7.3.9.|7.3.8.|>=7.3.9|>=7.3.8']
torchaudio=0.12.1 -> numpy[version='>=1.11'] -> pypy3.8[version='7.3.11.|>=7.3.11|>=7.3.9|>=7.3.8|7.3.9.|7.3.8.*']

Package cudatoolkit conflicts for:
pytorch=1.12.1 -> cudatoolkit[version='>=11.3,<11.4|>=11.6,<11.7']
cudatoolkit=11.6
torchvision=0.13.1 -> cudatoolkit[version='>=11.3,<11.4|>=11.6,<11.7']
torchaudio=0.12.1 -> cudatoolkit[version='>=11.3,<11.4|>=11.6,<11.7']
torchvision=0.13.1 -> pytorch -> cudatoolkit[version='>=10.1,<10.2|>=10.2,<10.3|>=11.1,<11.2|>=11.5,<11.6|>=11.0,<11.1|>=9.2,<9.3']

Package packaging conflicts for:
python=3.7.13 -> pip -> packaging
pip=22.3.1 -> wheel -> packaging[version='>=20.2']

Package m2w64-gcc-libs conflicts for:
pillow=9.4.0 -> libxcb[version='>=1.13,<1.14.0a0'] -> m2w64-gcc-libs
pytorch=1.12.1 -> blas=[build=mkl] -> m2w64-gcc-libs

Package certifi conflicts for:
pip=22.3.1 -> setuptools -> certifi[version='>=2016.09|>=2016.9.26']
torchvision=0.13.1 -> requests -> certifi[version='>=2016.09|>=2016.9.26|>=2017.4.17']

Package pytorch conflicts for:
pytorch=1.12.1
torchvision=0.13.1 -> pytorch==1.12.1
torchaudio=0.12.1 -> pytorch==1.12.1

Package setuptools conflicts for:
torchvision=0.13.1 -> setuptools
python=3.7.13 -> pip -> setuptools
pip=22.3.1 -> setuptools

Package vs2008_runtime conflicts for:
tqdm=4.65.0 -> python[version='>=2.7'] -> vs2008_runtime
pytorch=1.12.1 -> ninja -> vs2008_runtime

Package expat conflicts for:
pillow=9.4.0 -> pypy3.9[version='>=7.3.11'] -> expat[version='>=2.4.8,<3.0a0|>=2.4.9,<3.0a0|>=2.5.0,<3.0a0']
tqdm=4.65.0 -> python[version='>=3.7'] -> expat[version='>=2.5.0,<3.0a0']
pip=22.3.1 -> python[version='>=3.7'] -> expat[version='>=2.5.0,<3.0a0']

Package colorama conflicts for:
tqdm=4.65.0 -> colorama
python=3.7.13 -> pip -> colorama

Package requests conflicts for:
torchvision=0.13.1 -> requests
python=3.7.13 -> pip -> requests

Package flit-core conflicts for:
pytorch=1.12.1 -> typing_extensions -> flit-core[version='>=3.6,<4']
torchvision=0.13.1 -> typing_extensions -> flit-core[version='>=3.6,<4']

Package vc14_runtime conflicts for:
cudatoolkit=11.6 -> vc14_runtime[version='>=14.29.30139']
cudatoolkit=11.6 -> vc[version='>=14.2,<15'] -> vc14_runtime[version='>=14.16.27033|>=14.28.29325|>=14.32.31332|>=14.36.32532|>=14.38.33130|>=14.34.31931']

Package pillow conflicts for:
torchvision=0.13.1 -> pillow[version='>=5.3.0,!=8.3.*']
pillow=9.4.0
Note that strict channel priority may have removed packages required for satisfiability.

Hello, thank you for the great work. I was wondering if it was possible with the current implementation to pass as input a few videos from a new scene and get a 3d animation file that could be rendered using Blender for example?
I was thinking of something little bit like the Nerfs approaches, in a way it would be like generating novel views from the ones in the video. Or do we necessarily need a 3d asset as input?

Thank you very much for the program you provide. I am very interested in Gaussian-Eye View, Compositional Dynamic Scenes, Scene-Conditional Object Insertion and Edit Propagation. Are these parts open source? When will it be open source? Looking forward to your reply.

Hey,

There's a good chance I'm doing something stupid, but when I try to clone the Repo in Conda it says"

Cloning into 'Dynamic3DGaussians'...
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Anyone have any idea how to fix?

Thanks

I managed to get past a bunch of mismatched CUDA / PyTorch version issues, but I am now getting an error that I can't seem to get past when running the setup.py script:

\.conda\envs\dynamic_gaussians\lib\site-packages\torch\include\pybind11\cast.h(1429): error: too few arguments for template template parameter "Tuple"

Has anybody encountered this before or have any ideas how to fix it?

Does the input image have to be sized at (640, 320)? I have also utilized Colmap to reconstruct the point cloud, and the data has been processed into the same format as the provided dataset. I use 13 cameras for training and the PSNR is approximately 25 at each step of the training process. However, the size of the saved params.npz file is only a few KB. What might be causing this issue? I hope to hear from you soon. Thanks!

Hi! Could this be used for street scenes with traffic? Eg, the KITTI or waymo datasets?

Hi there, I am very impressed with your results and I will say you've done quite a good job keeping your code clean and compact.

I noticed you said you were working on creating some code to help aid the creation of custom datasets, but would it be possible in the meantime to provide us with instructions to manually create one using whatever method the test dataset was created with?

I'd love to test this using some of my own test data and see how the results come out.

I'm looking forward to hearing back from you, and again, thanks for all the work!

setyp.py ran successfully but whenever I run pip install. I get the following error:

Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1

The traceback:

 File "<string>", line 13, in <module>
      ModuleNotFoundError: No module named 'torch'
      [end of output]

My CUDA version is 11.7 and I have PyTorch 1.13 installed on my environment. I tried using torch1.12.1+cu116 but it was throwing a different error: CUDA version does not match the one used to compile Pytorch.

Any ideas how to work around this issue?

Hi @JonathonLuiten, I'm eagerly following your project. Could you provide an estimated timeline for when the code might be released? Thank you for your hard work!

Hi, thanks for your great job, I would like to know when will you release the code? Thanks.

Great job!

I'm very interested in the application related to scene editing. Could you provide a brief overview of how this is achieved?

best,
chieh

I prepared a custom dataset in necessary format and ran training script. The PSNR was only around 13, also the resulting gaussian is quite odd. I used 1 for the mask segmentation for all the points in initial point cloud.

Could you point out what could be the issue here?

Do this project have the goal to add support for AMD GPU? 😃
thanks!

Thank you for your excellent work, this work is deeply inspiring to us

we sincerely waiting for 2d tracking code

I just fixed a bug where the velocity based forward projection was not applied correctly!!

a246ec6

Maybe this bug was the last one I was missing in the cleaned code? Will do a comparison train overnight and let you all know tomorrow.

Anyway, would recommend pulling latest version.

Per your invitation @JonathonLuiten, I've started to create a script for reading params.npz into Unity and animating the dynamic Gaussian Splats. For the rendering, I'm looking at this codebase:
https://github.com/aras-p/UnityGaussianSplatting

Here is the script so far. It requires NumSharp to read the npz:

using System.Collections;
using NumSharp;
using Shapes;
using UnityEngine;

public class ParamsLoader : MonoBehaviour
{
    [Header("Input")] public string NpzPath;

    Polygon[] Points;
    Frame[] Frames;

    void Start()
    {
        using (NpzDictionary<float[,]> singleContent = np.Load_Npz<float[,]>(NpzPath))
        {
            // load base point cloud and release npz file
            float[,] segColors = singleContent["seg_colors.npy"]; // [points indicies, rgb vector3]    
            float[,] logitOpacities = singleContent["logit_opacities.npy"]; // [points indicies, opacity float]
            float[,] logScales = singleContent["log_scales.npy"]; // [points indicies, size vector3]

            int pointCount = segColors.GetLength(0);
            Points = new Polygon[pointCount];

            for (int i = 0; i < pointCount; i++)
            {
                Polygon gaussianPoint = Instantiate(PolygonFactory.Instance.tri);
                gaussianPoint.gameObject.SetActive(true);
                gaussianPoint.transform.localScale = new Vector3(
                    logScales[i, 0],
                    logScales[i, 1],
                    logScales[i, 2]);
                gaussianPoint.SetColor(new Color(
                    segColors[i, 0],
                    segColors[i, 1],
                    segColors[i, 2],
                    logitOpacities[i, 0]));
                Points[i] = gaussianPoint;
            }
        }

        using (NpzDictionary<float[,,]> single3Content = np.Load_Npz<float[,,]>(NpzPath))
        {
            // load animation for all of the frames and release npz file
            float[,,] means3D = single3Content["means3D.npy"]; // [frames, points indicies, xyz vector3]
            float[,,] rgbColors = single3Content["rgb_colors.npy"]; // [frames, points indicies, rgb vector3]
            float[,,] unnormRotations =
                single3Content["unnorm_rotations.npy"]; // [frames, points indicies, rotation quaternion]

            int frameMax = means3D.GetLength(0);
            Frames = new Frame[frameMax];

            for (int i = 0; i < frameMax; i++)
            {
                Frame frame = new Frame(Points.Length);
                for (int j = 0; j < Points.Length; j++)
                {
                    frame.FramePoints[j] = new FramePoint(
                        new Vector3(
                            means3D[i, j, 0],
                            means3D[i, j, 1],
                            means3D[i, j, 2]),
                        new Color(
                            rgbColors[i, j, 0],
                            rgbColors[i, j, 1],
                            rgbColors[i, j, 2]),
                        new Quaternion(
                            unnormRotations[i, j, 0],
                            unnormRotations[i, j, 1],
                            unnormRotations[i, j, 2],
                            unnormRotations[i, j, 3])
                    );
                }

                Frames[i] = frame;
            }
        }

        // start the gaussian animation
        StartCoroutine(Animate(0));
    }

    IEnumerator Animate(int frameNumber)
    {
        Frame frame = Frames[frameNumber];
        for (int i = 0; i < Points.Length; i++)
        {
            Points[i].transform.position = frame.FramePoints[i].Position;
            Points[i].transform.rotation = frame.FramePoints[i].Rotation;
            Points[i].SetColor(frame.FramePoints[i].Color);
        }

        yield return null;

        int nextFrame = frameNumber + 1;
        if (nextFrame > Frames.Length - 1)
        {
            // repeat the animation when the end is reached
            nextFrame = 0;
        }

        StartCoroutine(Animate(nextFrame));
    }

    class Frame
    {
        public readonly FramePoint[] FramePoints;

        public Frame(int pointCount)
        {
            FramePoints = new FramePoint[pointCount];
        }
    }

    class FramePoint
    {
        public readonly Vector3 Position;
        public readonly Color Color;
        public readonly Quaternion Rotation;

        public FramePoint(Vector3 position, Color color, Quaternion rotation)
        {
            Position = position;
            Color = color;
            Rotation = rotation;
        }
    }
}

Why did I train my own data, but I can’t see my own training data in the visualization? It seems to be the author’s data display, and the characters in it are still frozen.

Hi, thanks for the amazing work. Can you please share the uncleaned processing code for the data if possible?

We have been trying to get the right camera pose from the raw CMU data (their caliberation json file) to work with Gaussian Splatting but this seems tricky.

Is the definition of the camera coordinate system in train_meta.json the same as that in Blender?

Hi, thanks for your excellent work!

I'm pretty interested in the pixel-level 2D tracking application. According to the paper, after getting the depth values, we need to perform operations including "unprojection, the highest influence Gaussian determined, tracked and then projected into a new camera frame."

I understand that the evaluation code for tracking has not been released yet, but could you please provide some sample code for this part (or to be specific, the pixel-to-3D-Gaussian process)? That will really help me a lot!

Looking forward to your response, and again, thanks for the great job!

Hi, thanks for your nice work! As you said here:#23, you are using COLMAPs Coordinate system. What I don't understand is what is the function of opengl_proj, why not use ProjectionMatrix same as the original 3D gaussian splatting?

May I ask for data you captured. I want to compare the technique of yours with other reconstruction methods.

Thought it is hard to prepare my own shooting multi-view video datasets.

Looking forward that you could share the codes/methods to 1. segment 2. get init_pt_cld.npz 3. prepare train_meta.json.

Thanks

ResolvePackageNotFound:

• open3d=0.16.0

To get this working on Windows I had to remove open3d=0.16.0 from environment.yml, run the installation, then manually do pip install open3d==0.16.0 and then everything worked as expected.

Hi, thanks for your great work，some questions about the fixed params in Gaussians, why do you fix the size, opacity, color even using rgb instead of sh, what's the insight?
are there anything other methods to solve the dynamic scenes novel view synthesis using 3d gaussians ?
@JonathonLuiten Looking forward to your reply, thanks!

First of all, thank you, and congratulations. Rarely do I pull one of these projects onto my machine, and it just WORKS. I was up and running in almost no time, with your prepared visualizations. I was shocked at how clean the code is as well. I'm also running on Windows. Thank you for making it work here too (I'm using MiniConda and MVSC x64/x86 build tools for the visualizer and CUDA 11.7).

Question about the data pipeline. I see in the readme that you are estimating a release for the data preparation code around Nov 1. A few questions ahead of this. If I have two videos of a subject shot at different angles (.mp4) and I want to create a dynamic 3d gaussian out of these videos, will the data preparation script take these videos and do the following?:

• segments them
• calculate the camera positions (similar to how Colmap would place cameras)
• generate the necessary train_meta.json and the ims and seg folders and polulate them
• create the init_pt_cld.npz file

I'm wondering if there is any additional hacking I need to do ahead of that time to prepare the data. Thank you again.

Solved

Hi, congrats on the great work. I have a query regarding initial point cloud estimate which the code expects, its being read from this file (init_pt_cld.npz) and has the shape (N, 7).

@JonathonLuiten , I have 2 questions regarding this.
1.) Could you provide any insights/suggestions on how you are constructing this from the posed images? Would colmap suffice?
2.) Especially Last col which has 'seg' label (binary), does this indicate foreground/background?
Looking forward for your response.

Hi authors, Thanks for the interesting work.

I want to ask how you authors build the first scene as described in the paper.
I tried to build the scene in the first frame but it falls whether using COLMAP or the provided point cloud and camera poses.
I guess the init point cloud and camera poses are provided from CMU Panoptic Dataset (HD videos and calibration data), which is different from 3DGS setting.
Could you kindly explain how you authors build the first scene? It seems the current code only shows how to move the point cloud, not create the GS.

Thanks for your time again!

Hi Jonathon,

Great job with the codebase here. I want to experiment with ZJU-MoCap Dataset on this code. However, I am not able to figure out the parameters and variables.

I tried to reverse engineer your datasets, however, there are largely three problems:

1. No clue about the parameters like means3D, mean3_sq_dist, cam_m, and cam_c
2. I don't understand how to calculate variables max_2D_radius, scene_radius, means2D_gradient_accum, and denom.
3. 194105 is in the shape of all vars and params, I found nothing related to the size of the image

Congrats on your excellent work!

In Gaussian-Eye View: First-Person and Gaussian-Eye View: Object View these two videos in your project page, I'm surprised that the near-field rendering from egocentric view is so bad, given promising quality in far-field rendering. Any specific reasons for this performance degradation?