charliedaveputh / comfyui-3d-pack

An extensive node suite that enables ComfyUI to process 3D inputs (Mesh & UV Texture, etc) using cutting edge algorithms (3DGS, NeRF, etc.)

ComfyUI-3D-Pack

An extensive node suite that enables ComfyUI to process 3D inputs (Mesh & UV Texture, etc) using cutting edge algorithms (3DGS, NeRF, Differentiable Rendering, SDS/VSD Optimization, etc.)

Features — Roadmap — Install — Run — Tips

Note: this project is still a WIP and not been released into ComFyUI package database yet

Currently support:

• For use case please check Example Workflows. [Last update: 09/02/2024]

  • Note: you need to put Example Inputs Files & Folders under ComfyUI Root Directory\ComfyUI\input folder before you can run the example workflow

• Large Multiview Gaussian Model: 3DTopia/LGM

  • Enable single image to 3D Gaussian in less than 30 seconds on a RTX3080 GPU, later you can also convert 3D Gaussian to mesh

    2024-02-08.23-36-31.mp4

• Triplane Gaussian Transformers: VAST-AI-Research/TriplaneGaussian

  • Enable single image to 3D Gaussian in less than 10 seconds on a RTX3080 GPU, later you can also convert 3D Gaussian to mesh

    2024-02-08.23-57-37.mp4

• Preview 3DGS and 3D Mesh: 3D Visualization inside ComfyUI:

  • Using gsplat.js and three.js for 3DGS & 3D Mesh visualization respectively

    2024-02-04.19-20-17.mp4

• Stack Orbit Camera Poses: Automatically generate all range of camera pose combinations

  • You can use it to conditioning the StableZero123 (You need to Download the checkpoint first), with full range of camera poses in one prompt pass

  • You can use it to generate the orbit camera poses and directly input to other 3D process node (e.g. GaussianSplatting and BakeTextureToMesh)

  • Example usage:

    
    

  • Coordinate system:

    • Azimuth: In top view, from angle 0 rotate 360 degree with step -90 you get (0, -90, -180/180, 90, 0), in this case camera rotates clock-wise, vice versa.
    • Elevation: 0 when camera points horizontally forward, pointing down to the ground is negitive angle, vice versa.

• 3D Gaussian Splatting

  • Improved Differential Gaussian Rasterization
  • Better Compactness-based Densification method from Gsgen,
  • Support initialize gaussians from given 3D mesh (Optional)
  • Support mini-batch optimazation
  • Multi-View images as inputs
  • Export to standard 3DGS .ply format supported

• Gaussian Splatting Orbit Renderer

  • Render 3DGS to images sequences or video, given a 3DGS file and camera poses generated by Stack Orbit Camera Poses node

• Bake Multi-View images into UVTexture of given 3D mesh using Nvdiffrast, supports:

  • Export to .obj, .ply, .glb

• Deep Marching Tetrahedrons

  • Allow convert 3DGS .ply file to 3D mesh
    Note: I didn't spent time to turn the hyperprameters yet, the result will be improved in the future!

• Save & Load 3D file

  • .obj, .ply, .glb for 3D Mesh
  • .ply for 3DGS

• Switch Axis for 3DGS & 3D Mesh

  • Since different algorithms likely use different coordinate system, so the ability to re-mapping the axis of coordinate is crucial for passing generated result between differnt nodes.

Roadmap:

• Add DMTet algorithm to allow conversion from points cloud(Gaussian/.ply) to mesh (.obj, .ply, .glb)

• Integrate Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers

• Add interactive 3D UI inside ComfuUI to visulaize training and generated results for 3D representations

• Add a new node to generate renderer image sequence given a 3D gaussians and orbit camera poses (So we can later feed it to the differentiable renderer to bake it onto a given mesh)

• Integrate LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation

• Add a general SDS/ISM Optimization algorithm to allow training 3D representations with diffusion model, The real fun starts here ;)

  • Need to do some in-depth research on Interval Score Matching (ISM), since math behind it makes perfect sense and also there are so many ways we could improve upon the result obtained from LucidDreamer

• Improve 3DGS to Mesh conversion algorithms:

  • Support to training DMTet with images(RGB, Alpha, Normal Map)
  • Find better methods to converts 3DGS or Points Cloud to Mesh (Normal maps reconstruction maybe?)

• Add Structure from motion (SfM) initialization for 3DGS (Better first guess -> Faster convergence & Better result)

• Add a few best Nerf algorithms (No idea yet, instant-ngp maybe?)

Install:

[IMPORTANT!!!]
Currently this package is only been tested in following setups:

• Windows 10/11
• Miniconda/Conda Python 3.11.7
  • I tried install this package with ComfyUI embed python env first, but I can't find a way to build CUDA related libraries, e.g. diff-gaussian-rasterization, nvdiffrast, simple-knn.
• Torch version: 2.1.2+cu121/V.2.1.2+cu118

Assume you have already downloaded ComfyUI

First download Miniconda (One of the best way to manage a clean and separated python envirments)

• Alternatively you can check this tutorial: Installing ComfyUI with Miniconda On Windows and Mac

# Go to your Your ComfyUI root directory, for my example:
cd C:\Users\reall\Softwares\ComfyUI_windows_portable 

conda create -p ./python_miniconda_env/ComfyUI python=3.11

# conda will tell what command to use to activate the env
conda activate C:\Users\reall\Softwares\ComfyUI_windows_portable\python_miniconda_env\ComfyUI

# This package also works with cu118
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121

pip install -r ./ComfyUI/requirements.txt

# Then go to ComfyUI-3D-Pack directory under the ComfyUI Root Directory\ComfyUI\custom_nodes for my example is:
cd C:\Users\reall\Softwares\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-3D-Pack
# Finally you can double click following .bat script or run it in CLI:
install.bat

• Install Plan B:

  Just in case install.bat may not working in your PC, you could also run the following commands under this package's root directory:

  # First make sure the Conda env: python_miniconda_env\ComfyUI is activated, then go to ComfyUI Root Directory\ComfyUI\custom_nodes\ComfyUI-3D-Pack and:
  pip install -r requirements.txt
  
  git clone --recursive https://github.com/ashawkey/diff-gaussian-rasterization
  pip install ./diff-gaussian-rasterization
  
  pip install ./simple-knn
  
  pip install -U xformers --index-url https://download.pytorch.org/whl/cu121
  
  git clone --recursive https://github.com/NVlabs/nvdiffrast/`
  pip install ./nvdiffrast
  
  # Install pointnet2_ops
  cd tgs/models/snowflake/pointnet2_ops_lib && python setup.py install && cd ../../../../
  
  # Install pytorch_scatter
  pip install git+https://github.com/rusty1s/pytorch_scatter.git
  
  # Install pytorch3d
  pip install git+https://github.com/facebookresearch/pytorch3d.git@stable
  
  # Install kiuikit (Computer Vision Library)
  pip install git+https://github.com/ashawkey/kiuikit.git

• Install with ComfyUI's embed python env on Windows:

  If you have already installed ComfyUI on Windows with embed python environment and you don't want to switch to Miniconda/Conda and reinstall all your packages
  (Which is ideal, but unfortunately so far it seems I'm not experienced enough in this matter, but I'll continue to looking for the better solutions and please give me some suggestions if ou know better, thanks :)

  According to @doctorpangloss, you can setup the c++/cuda build environments in windows by using chocolatey with following command:

  # using git bash for the sake of simplicity
  # enable developer mode
  # google this: allow os.symlink on windows by adding your username to the local security policy entry for it.
  # you will have to restart your computer
  # install chocolatey using powershell, then install the prereqs for compilation on Windows
  choco install -y visualstudio2022buildtools
  choco install -y visualstudio2022-workload-vctools --package-parameters "--add Microsoft.VisualStudio.Component.VC.Llvm.ClangToolset --add Microsoft.VisualStudio.Component.VC.Llvm.Clang"
  # I couldn't find cuda version 12.1, you may need to install cudav12.1 manually
  choco install -y cuda 

  Then just go to ComfyUI-3D-Pack directory under the ComfyUI Root Directory\ComfyUI\custom_nodes and running:

  pip install -r ./ComfyUI/requirements.txt
  
  install.bat

Run:

Copy the files inside folder __New_ComfyUI_Bats to your ComfyUI root directory, and double click run_nvidia_gpu_miniconda.bat to start ComfyUI!

• Alternatively you can just activate the Conda env: python_miniconda_env\ComfyUI, and go to your ComfyUI root directory then run command python ./ComfyUI/main.py

Tips

• The world & camera coordinate system is the same as OpenGL:

    World            Camera        
  
     +y              up  target                                              
     |               |  /                                            
     |               | /                                                
     |______+x       |/______right                                      
    /                /         
   /                /          
  /                /           
 +z               forward           

elevation: in (-90, 90), from +y to -y is (-90, 90)
azimuth: in (-180, 180), from +z to +x is (0, 90)

• If you encounter OpenGL errors (e.g., [F glutil.cpp:338] eglInitialize() failed), then set force_cuda_rasterize to true on corresponding node