dylanebert / gaussian-viewer

Gaussian Viewer

This repository contains a proof-of-concept web viewer for 3D Gaussian Splatting, a technique capable of rasterizing realistic scenes in real-time.

It combines the original paper diff-gaussian-rasterization implementation with hardware H264 encoding and WebRTC streaming to stream interactive views in realtime.

Live Demo

A live viewer is available at:

• Hugging Face

Prerequisites

• Python 3 or above
• NVIDIA card with display driver 525.xx.xx or above
• CUDA Toolkit 11.7

and least one of the following:

• Ubuntu 22.04 (Linux)
• WSL2 with Ubuntu 22.04 (Windows)
• Docker

Cloning this Repository

To clone this repository and its dependencies, check it out with

git clone https://github.com/dylanebert/gaussian-viewer.git --recursive

Be sure to use the --recursive flag to clone the local dependencies:

• gaussian-viewer-frontend
• diff-gaussian-rasterization
• VideoProcessingFramework

Downloading Models

This project is currently hardcoded to run the trained bicycle example from the Mip-NeRF 360 dataset.

Click here to download the data. Unzip the archive, then copy the contained bicycle folder to this repository's models folder.

Running locally

This project depends on the VideoProcessingFramework for hardware tensor-to-video encoding and diff-gaussian-rasterization for rasterization. VideoProcessingFramework in particular has very specific requirements, including Ubuntu 22.04, or WSL2 with Ubuntu 22.04 on Windows. Alternatively, you can follow the Docker instructions to run in other environments.

Without Docker

1. Create a virtual python environment

python -m venv venv

2. Install the required python packages

python -m pip install -r requirements.txt
python -m pip install diff-gaussian-rasterization/
python -m pip install VideoProcessingFramework/
python -m pip install VideoProcessingFramework/src/PytorchNvCodec/

If you encounter issues, be sure the required submodules are present

git submodule update --init --recursive

and refer to corresponding submodule documentation for troubleshooting.

3. Run the server

python -m uvicorn main:app

This should print a local web address running the demo (i.e. http://localhost:8000)

With Docker

1. Edit the Dockerfile

The provided Dockerfile is configured to serve the viewer with an SSL certificate over HTTPS. This isn't necessary for running locally.

Replace the last two lines

EXPOSE 443

CMD ["python3.10", "-m", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "443", "--ssl-keyfile", "privkey.pem", "--ssl-certfile", "fullchain.pem"]

with

EXPOSE 8000

CMD ["python3.10", "-m", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

2. Build the container

docker build -t gaussian-viewer .

3. Run the container

docker run --gpus all -p 8000:8000 gaussian-viewer

Local Configuration

Change Addresses

When running locally, you will need to update the addresses to point toward the local implementation, rather than the remote server.

• In gaussian-viewer-frontend/src/src/routes/+page.svelte, replace https://viewer.dylanebert.com/ with http://localhost:8000/.
• In main.py, replace origins = ["https://viewer.dylanebert.com", "https://dylanebert-gaussian-viewer.hf.space"] with origins = ["*"] to allow all origins.

STUN/TURN

WebRTC typically requires STUN/TURN servers to function on web. However, this shouldn't be an issue when running locally.

The provided implementation is configured to fetch ICE servers from twilio using private credentials. You can provide your own credentials, replace them with other ICE servers, or ignore them when running locally.

To ignore them when running locally, in gaussian-viewer-frontend/src/src/routes/+page.svelte in connectToPeer, remove the iceServers fetch and simply pass iceServers: [].

Project Layout

This project consists of a python backend and svelte frontend.

• Backend
  • main.py
    • src/
      • camera.py
      • gaussian_model.py
      • render.py
      • turn.py
      • utils.py
• Frontend (gaussian-viewer-frontend/)
• src/
  • src/routes/+page.svelte
• public/

The backend server entrypoint is main.py.

All functional frontend code is in +page.svelte. This is built to static HTML that is served in public/.

Community

If you're interested in community discussion on this project, join the Hugging Face discord and feel free to ping me @IndividualKex. Alternatively, email me at [email protected].

gaussian-viewer's Issues

Aren't we encoding then decoding needlessly?

It's likely I don't understand the code properly but it seems like we first encode to h264 on the GPU in render.py then decode from h264 in parse_frames(...) in main.py?

If so, isn't this a needless roundtrip?

ERROR: Could not find a version that satisfies the requirement `nvidia-cudnn-cu11==8.5.0.96`

Hi, I am using Python 3.10.12 as recommended in this repo and when running python -m pip install -r requirements.txt, I get the following error:

ERROR: Ignored the following yanked versions: 8.9.4.19
ERROR: Could not find a version that satisfies the requirement nvidia-cudnn-cu11==8.5.0.96 (from versions: 0.0.1.dev5, 8.9.4.25, 8.9.5.29)
ERROR: No matching distribution found for nvidia-cudnn-cu11==8.5.0.96

The error is mainly due to this nvidia-cudnn-cu11==8.5.0.96 library. I am using Windows 11. I would appreciate if anyone could help me with this.

HF space is down!

Just a note that your HF space is down?

https://huggingface.co/spaces/dylanebert/gaussian-viewer

Live demo has CORS issues

When opening the live demo at https://huggingface.co/spaces/dylanebert/gaussian-viewer the app fails to load due to:

Missing docs: point_cloud

Got the software running on an Ubuntu Linux server with RTX 4090 and installed the example data as described in the README but it produces this error:

FileNotFoundError: [Errno 2] No such file or directory: 'models/bicycle/point_cloud/iteration_30000/point_cloud.ply'

Looks like there are some missing steps in the documentation to get it working?

View dependent training?

Looked a lot at this repo today and realized that the input is a dense point cloud without the original images, so it is unable to capture view dependent effects like NeRF. If this feature is added it would be significantly more interesting!

Http stuck on loading

hi,
I tried using the git with docker, the installations completed successfully, after running the docker i get:
INFO: Started server process [1]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8090 (Press CTRL+C to quit)
INFO: 192.116.240.226:39292 - "GET / HTTP/1.1" 200 OK
INFO: 192.116.240.226:39292 - "GET /_app/immutable/assets/2.9877725f.css HTTP/1.1" 200 OK
INFO: 199.203.240.213:39294 - "GET /_app/immutable/entry/start.e4621793.js HTTP/1.1" 200 OK
INFO: 192.116.240.226:39292 - "GET /_app/immutable/chunks/scheduler.8b74b908.js HTTP/1.1" 200 OK
INFO: 192.116.240.226:39296 - "GET /_app/immutable/chunks/singletons.4c16d0fe.js HTTP/1.1" 200 OK
INFO: 192.116.240.226:39300 - "GET /_app/immutable/entry/app.4f5bbb5d.js HTTP/1.1" 200 OK
INFO: 199.203.240.213:39298 - "GET /_app/immutable/chunks/index.c146e4e6.js HTTP/1.1" 200 OK
INFO: 199.203.240.213:39302 - "GET /_app/immutable/nodes/0.2a6e7c35.js HTTP/1.1" 200 OK
INFO: 192.116.240.226:39292 - "GET /_app/immutable/nodes/2.06661bc9.js HTTP/1.1" 200 OK
INFO: 192.116.240.226:39296 - "GET /favicon.png HTTP/1.1" 200 OK
INFO: 199.203.240.213:39302 - "GET /_app/immutable/nodes/1.263229da.js HTTP/1.1" 200 OK

But when entering the http address all i get is:
Loading...
Do you have any idea why is that?
Thank a lot and also thank you for the GitHub!