etherealengine / universal-volumetric

A free, fast, cross-platform volumetric codec for everyone.

Universal Volumetric (UVOL)

The open source compressed interchange format for 4D Video (streaming 3D sequences)

READ THE WHITEPAPER

UVOL 2.0

UVOL 2.0 introduces usage of Compressed Array Textures, via KTX2 supercompressed textures, allowing volumetric media to use much less texture memory than UVOL 1.0!

The geometry compression method has also been updated from Corto to Draco. UVOL 2.0 uses directories of files which represent individual frames or frame sequneces, and a manifest file which points to these files. The Encoder helps you to convert the data you have to the data the UVOL player needs. The encoder script is located at: scripts/Encoder.py.

The Encoder script requires a json configuration file. This config file is processed in a certain order that is explained below.

{
    name: string,
    draco_encoder: string,
    basisu: string,
    ABCFilePath: string,
    OBJFilesPath: string,
    DRACOFilesPath: string,
    Q_POSITION_ATTR: number,
    Q_TEXTURE_ATTR: number,
    Q_NORMAL_ATTR: number,
    Q_GENERIC_ATTR: number,
    DRACO_COMPRESSION_LEVEL: number,
    ImagesPath: string,
    KTX2_FIRST_FILE: number,
    KTX2_FILE_COUNT: number,
    KTX2_BATCH_SIZE: number,
    KTX2FilesPath: string,
    GEOMETRY_FRAME_RATE: number,
    TEXTURE_FRAME_RATE: number,
    AudioURL: string,
    OutputDirectory: string
}

Above is the type spec for the config file. It is not required to specify all the fields in the config file.

Let's start with mandatory fields.

• name: This represents the name of the manifest file (or) basically to denote a particular Volumetric video.
• GEOMETRY_FRAME_RATE: This represents how many geometry frames are rendered per second.
• TEXTURE_FRAME_RATE: This represents how many texture frames are rendered per second. It is advisable to have both frame rates factors of one another. It helps the player to avoid ambiguities in calculating frame numbers.
• KTX2_BATCH_SIZE: This represents number of frames are packed (or to be packed) in a single KTX2 video texture.
• OutputDirectory: The processed files are stored in this directory (labelled with their formats).

Now, we discuss how geometry data is processed:

Followed by texture data processing:

Below paths must have the file pattern mentioned.

• OBJFilesPath: Eg: /home/3D/export_[#####].obj
• DRACOFilesPath: Eg: /home/3D/export_[#####].drc
• ImagesPath: Eg: /home/3D/export_[#####].jpg
• KTX2FilesPath: Eg: /home/3D/export_[#####].ktx2

Frame numbers are calculated from the file names itself, Hence file names should be indexed (with padding). The manifest file also uses this notation in specifying DRCURLPattern and KTX2URLPattern. The indexing can be either 0 based indexing or 1 based indexing, but make sure it is consistent between Geometry files and Texture files. These indices are vital for the player to calculate the correct frame and render it with the right geometry/texture.

Usage

• Encoder uses bpy python package which only works with selected python versions: Python >=3.7, <3.8.
• Make sure you have draco_encoder and basisu binaries somewhere. The paths of those binaries can be either passed to the project-config, or they can be omitted if they're already in the path.
• A template project-config.json can be created with this command: python3 scripts/Encoder.py create-template.
• Fill the config file and pass it to the Encoder: python3 scripts/Encoder.py project-config.json. (Encoder raises errors if something isn't alright)

Demo

Here's a short sped up version of encoder on duty!

Collaborators Wanted!

If you are proficient in C++, python, JS, Unity/C#, or you want to support this project creatively or financially, please get in touch!

Example

Currently playback works in WebGL with three.js. Unity & Unreal support is on the roadmap.

Requirements

For encoding, you will Python 3 and blender python package installed.

For decoding, currently WebGL is supported (especially three.js), Unreal and Unity will come in a future release.

You will need a mesh and texture sequence in ABC/OBJ and PNG/JPG formats respectively.

Find us on Discord! https://discord.gg/xrf

universal-volumetric's Issues

Making draco_encoder and basisu available via PATH on windows results in runtime error within Encoder.py

Describe the bug
If I add the binary locations for draco_encoder and basisu to my PATH on windows, I get errors when Encoder.py attempts to run these executables. It looks like this is due to the shlex.split function not handling windows paths with backslashes properly.

To work around this I can of course remove these binary directories from my PATH, and add the draco_encoder and basisu paths using forward slashes directly within the project config json file, but this may not be ideal for some.

A potential fix would be to modify the check_executables function as follows to transform the backslashes into forward slashes by adding a .replace('\\', '/')

def check_executables(config):
    ok = True

    if which("draco_encoder"):
        config["draco_encoder"] = which("draco_encoder").replace('\\', '/')

    elif not config.get("draco_encoder"):
        print(
            "❌ 'draco_encoder' command doesn't exist. Please build it from https://github.com/google/draco"
        )
        ok = False

    if which("basisu"):
        config["basisu"] = which("basisu").replace('\\', '/')
    elif not config.get("basisu"):
        print(
            "❌ 'basisu' command doesn't exist. Please build it from https://github.com/BinomialLLC/basis_universal"
        )
        ok = False

    if not ok:
        exit(1)

SyntaxError: The requested module 'three' does not provide an export named 'default'

"import THREE from 'three';" is error : "default export" in es6 package language

root@opendco:/opt/UniversalVolumetric-main# node ./src/Encoder.js yifu.uvol
file:///opt/UniversalVolumetric-main/src/libs/THREECORTOLoader.js:1
import THREE from 'three';
       ^^^^^
SyntaxError: The requested module 'three' does not provide an export named 'default'
    at ModuleJob._instantiate (node:internal/modules/esm/module_job:127:21)
    at async ModuleJob.run (node:internal/modules/esm/module_job:191:5)
    at async Promise.all (index 0)
    at async ESMLoader.import (node:internal/modules/esm/loader:331:24)
    at async loadESM (node:internal/process/esm_loader:88:5)
    at async handleMainPromise (node:internal/modules/run_main:65:12)

Node.js v17.1.0

Flexibility in `targetFramesToRequest` value

Hardcoding it to 90 causes the player to stuck at buffering when the source files has few frames. (A file with 240 frames is stuck due to this issue). We probably need a better strategy to optimize this value for both small files and large files.

Trade off:

• Bigger targetFramesToRequest => Lesser requests,
• Smaller targetFramesToRequest => Support for small files

Request and render series of DRACO files

Use multi-threading within Encoder.py to speed up compression time

It would be nice to have the option to run the encoder with multiple threads in order to run the draco_encoder and basisu subprocesses in parallel. This could be accomplished with the python threading module.

Embed manifest in .uvol

The external manifest is just for debug. Can go in the front of the uvol file, which is currently headerless.

TypeError: Cannot add property CORTOLoader, object is not extensible

While I run node ./encoder_legacy/Encoder.js example.uvol
It displays the message as following

file:///C:/Users/nam/Downloads/Universal-Volumetric/encoder_legacy/libs/THREECORTOLoader.js:22
THREE.CORTOLoader = function(options, manager) {
                  ^
TypeError: Cannot add property CORTOLoader, object is not extensible
at file:///C:/Users/nam/Downloads/Universal-Volumetric/encoder_legacy/libs/THREECORTOLoader.js:22:19

• node version: v14.21.3
• windows version: 10
• Python: 3.8.5

UVOL 2.0 resolution XRUI

Add render support for requestVideoFrameCallback

https://web.dev/requestvideoframecallback-rvfc/

Start Here - driving updates in main rendering loop?
https://github.com/XRFoundation/Universal-Volumetric/blob/main/web/player.ts#L222

Essentially, in browsers that support this new API, we should be able to avoid blitting the video frames into a canvas and reading back the pixel data, which is how we are currently retrieving the current frame number. We should also be able to pass the video element directly to the GPU, and update as needed.

Add versioning

Should be able to instantiate a web player instance without providing a source

Supplying an empty paths property into the constructor of the web player throws an error, as it expects there to be at least one string in the array. This makes it hard to instantiate a player while not knowing the source ahead of time (such as creating a volumetric object from the prefab in the ethereal engine studio).

Add github publish workflow

• push to github packages like the other @EtherealEngine packages

Re-configure player for iframes

We had interframe compression and then removed it because we didn't have reliable test data. Seems like we do now, so we should reimplement ASAP. Basically means looking at non-keyframes as a list of new positions and only using those.

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• Update dependency com.android.tools.build:gradle to v7.4.2
• Update dependency gradle to v8.0.1
• Update babel monorepo (@babel/cli, @babel/core, @babel/plugin-transform-modules-commonjs)
• Update dependency @types/glob to v8.1.0
• Update dependency three to v0.150.1
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

gradle

android/gradle.properties

android/settings.gradle

android/build.gradle

• com.android.tools.build:gradle 7.4.1

android/app/build.gradle

• androidx.appcompat:appcompat 1.6.1
• com.google.android.material:material 1.8.0
• androidx.constraintlayout:constraintlayout 2.1.4
• com.jakewharton.timber:timber 5.0.1
• junit:junit 4.+
• androidx.test.ext:junit 1.1.5
• androidx.test.espresso:espresso-core 3.5.1

gradle-wrapper

android/gradle/wrapper/gradle-wrapper.properties

• gradle 8.0

npm

encoder/dev/package.json

encoder/package.json

• esm 3.2.25
• http-range-fetcher 1.5.0
• three 0.149.0
• three-obj-loader-cjs-module 1.0.0
• xmlhttprequest 1.8.0
• @babel/cli 7.20.7
• @babel/core 7.20.12
• @babel/plugin-transform-modules-commonjs 7.20.11
• @types/glob 8.0.1
• ts-loader 9.4.2
• ts-node 10.9.1
• typescript 4.9.5
• yarn >= 0.18.0

package.json

• shelljs 0.8.5
• vite ^4.0.0
• xmlhttprequest 1.8.0
• @babel/cli 7.20.7
• @babel/core 7.20.12
• @types/three 0.149.0
• @ianvs/prettier-plugin-sort-imports 3.7.1
• prettier 2.8.4
• typescript 4.9.5
• three >0.130.0

nuget

unity/Assembly-CSharp.csproj

---

• Check this box to trigger a request for Renovate to run again on this repository

[EPIC]: UVOL2

UVOL2 at a high-level

• .uvol.manifest.json - contains structured information about framerate, audio & video texture sources (ktx2 vs mp4), geometry buffer ranges, etc, compatible with gltf conventions

• .uvol.mp4 - video for texture (and audio)

• .uvol.ktx2.bin - binary buffer of compresses ktx2 texture data sequence

• .uvol.drc.bin - binary buffer of compressed drc sequence

• .uvol.gltf/glb - gltf w/ uvol extensions (pointing to uvol manifest and compressed geometry buffer)

• .uvol.usd/usda/usdc/usdz - usd w/ uvol extensions (pointing to uvol manifest and compressed geometry buffer)

• .uvol / .uvol.crt.bin - DEPRECATED (UVOL1 format) binary buffer of compressed corto sequence

• .manifest.json - DEPRECATED (UVOL1 format) - replaced with .uvol.manifest.json above

• #182

• #184

• #183

• #185

Request and apply series of KTX2 textures

Unity: support uvol on Android platform

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when I try use uvol on Android platform using Unity.
I found some problems with reading manifest and uvol file. But now I cant continue, because I got error:

Error Unity EntryPointNotFoundException: Unable to find an entry point named 'CreateDecoder' in 'corto'. Tried the following entry points: Unable to get function 'CreateDecoder' because of 'The requested function was not found (0x06000001) dlerror() = undefined symbol: CreateDecoder

And now I dont know what I should do.

Describe the solution you'd like
I want use corto on Android platform using unity. I want use corto on VR headset like Oculus Quest 2.

Describe alternatives you've considered
I consider fix reading manifest-file and uvol-file from StreamingAssets using WWW on Android.

Suggestion: 4-byte alignment for uvol meshes

A suggestion:

It looks like the Corto library requires 4-byte alignment for buffers being decoded, but since uvol files do not enforce any alignment, the current Unity code is forcing a new buffer allocation and copy for each frame mesh.

It seems that both the allocation and the copy could be avoided if the uvol files created enforced 4-byte alignment for each frame mesh...

What does one do with the "DRCS" file and Manifest? The unity Demo is also missing "Liam manifest"

This repo while good is broken (imho)
There's no example how to use the DRCS file created.

The unity demo is broken - missing a "liam manifest" and also has "uvol" no DRCS.
Can this be cleaned up please.
Thanks

Compress texture data with KTX2

Apply KTX2 compressed texture data to Draco geometry from static file server

Option to append binary encoding to bottom instead of printing on

For folks who aren't pre-scaling their UVs and baking them (which was our previous internal flow) and who don't need square textures on output for any reason, add an option to rescale the UVs and append the binary encoder on the bottom (so a 1024x texture becomes 1024x1032).

Rewrite unified UVOL player in engine with react

UVOL Uniform Solve play capability

Clean Mesh Attributes

The input PLY meshes must have clean data : only Position, UVs, Normals or Velocity.
Some input OBJ meshes have references to external textures or excess attributes, this can cause texture corruption or sync issues.

Migrate to Draco compression

Play Draco compressed geometry from static file server using current texture settings

three.js curve decoder

Encoder seems to work and motion graphs look good (still assuming linear t, will need to update with a higher order solution for fast motion) but player still needs to be implemented in webgl.

Support non-square playback

Right now there are a few calls that make assumptions that the video will be square. No need, should be any size.

three.js -- Set Draw Range

We can/should set a max buffer size and manipulate the buffer with draw ranges, so that we only render the tris we need, but we don't need to create new mesh objects ever.

Dependencies missing from requirements.txt

On a fresh install of python 3.7.9, I ran pip install -r requirements.txt and currently this only installs the bpy package. I needed to install the following additional packages to get Encoder.py to run:
pip install commentjson tqdm audioread
Ideally these would be added to requirements.txt as well.