GithubHelp home page GithubHelp logo

julienkay / mobilenerf-unity-viewer Goto Github PK

View Code? Open in Web Editor NEW
388.0 12.0 43.0 1.86 MB

An unofficial Unity port of the MobileNeRF viewer

License: MIT License

C# 100.00%
mesh nerf neural-radiance-fields unity view-synthesis mobilenerf

mobilenerf-unity-viewer's Introduction

MobileNeRF Unity Viewer

MobileNeRF Unity Viewer

This repository contains the source code for a Unity port of the web viewer from the paper MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures1

Please note, that this is an unofficial port. I am not affiliated with the original authors or their institution.

Usage

Installation

Go to the releases section, download the Unity Package, and import it into any Unity project. This is a 'Hybrid Package' that will install into your project as a local package.

Alternatives
UPM Package via OpenUPM

In Edit -> Project Settings -> Package Manager, add a new scoped registry:

Name: Doji
URL: https://package.openupm.com
Scope(s): com.doji

In the Package Manager install 'com.doji.mobilenerf either by name or via Package Manager -> My Registries

UPM Package via Git URL

In Package Manager -> Add package from git URL... paste https://github.com/julienkay/MobileNeRF-Unity-Viewer.git as described here

Importing sample scenes

After succesful installation, you can use the menu MobileNeRF -> Asset Downloads to download any of the sample scenes available. In each scene folder there will be a convenient prefab, that you can then drag into the scene and you're good to go.

Updating

Since the initial release a small number of features have been added to the automatic shader generation code. That means, that if you have already downloaded some scenes before, you'll have to regenerate the source files by going to MobileNeRF -> Asset Downloads again. (This will not actually redownload Assets unless necessary, so this shouldn't take too long)

Importing self-trained scenes

If you have successfully trained your own MobileNeRF scenes using the official code release and want to render them in Unity, you can use the menu MobileNeRF -> Import from disk. This lets you choose a folder that should contain all the output files of your training process.

More specifically, the following assets are required:

  • multiple shapes as OBJ files with naming convention: shapeX.obj
  • multiple PNG files with naming convention: shapeX.pngfeatZ.png (where Z goes from 0 to 1, so 2 .pngs per shape)
  • The weights of the MLP in a file called mlp.json

Details

The project was created with Unity 2021.3 LTS using the Built-in Render Pipeline.

The biggest deviation from the official viewer is, that this project doesn't use Deferred Rendering, but uses Forward Rendering instead. This has certain implications on performance. While the MobileNeRF representation itself greatly reduces the cost to render NeRFs, it still requires evaluating a small, view-dependent MLP (Multi Layer Perceptron) per fragment. Whenever the bottleneck is in the fragment shader, Deferred Rendering has obvious advantages, as each pixel only needs to run a single fragment shader.

Forward Rendering however gives us MSAA, which is important for VR use cases. Additionally, in VR the image has to be rendered twice, once for each eye, with a fairly large resolution. The larger the G-buffer, the smaller the benefit of Deferred Rendering. Still, MobileNeRFs mesh representations have a fairly large poly count which works against us using Forward Rendering here.

Some things to possibly look into:

  • Forward Rendering, but do a Depth Prepass to reduce overdraw (might require URP, see here and here)
  • Implement Deferred Rendering and compare performance in various scenarios

Known Issues

  • Does not work well with MSAA. For now, I recommend turning MSAA off in Project Settings -> Quality -> Anti Aliasing

References

Thanks to

  • @mrxz for providing some major performance optimizations as part of his WebXR port

Other projects exploring NeRFs and related techniques in Unity:

Footnotes

  1. Zhiqin Chen and Thomas Funkhouser and Peter Hedman and Andrea Tagliasacchi. MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures. arXiv preprint arXiv:2208.00277, 2022

mobilenerf-unity-viewer's People

Contributors

bell-one avatar julienkay avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

mobilenerf-unity-viewer's Issues

Completely not runnable in VR?

Hi, I tried running a basic scene with only the ship model from the original paper in standalone Quest 2, and it crashes my headset (~5fps). Is this normal, or am I missing some setup? I am using URP.

Questions about the prefab

Dear Julien,

very nice work!

I have a question about the available prefab, is this downloaded or also generated in your code?

Thanks in advance

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.