openxrlab / xrfeitoria

OpenXRLab Synthetic Data Rendering Toolbox

Introduction

XRFeitoria is a rendering toolbox for generating synthetic data photorealistic with ground-truth annotations. It is a part of the OpenXRLab project.

XRFeitoria-Intro.mp4

Major Features

• Support rendering photorealistic images with ground-truth annotations.
• Support multiple engine backends, including Unreal Engine and Blender.
• Support assets/camera management, including import, place, export, and delete.
• Support a CLI tool to render images from a mesh file.

Installation

pip install xrfeitoria

Requirements

• Python >= 3.8
• (optional) Unreal Engine >= 5.1
  • Windows
  • Linux
  • MacOS
• (optional) Blender >= 3.0
  • Windows
  • Linux
  • MacOS

Get-Started

CLI

xf-render --help

# render a mesh file
xf-render {mesh_file}

# for example
wget https://graphics.stanford.edu/~mdfisher/Data/Meshes/bunny.obj
xf-render bunny.obj

CLI-simple.mp4

CLI-complex.mp4

Documentation

The reference documentation is available on readthedocs.

Tutorials

There are several tutorials. You can read them here.

Sample codes

There are several samples. Please follow the instructions here.

Use plugins under development

Details can be found here.

If you want to publish plugins of your own, you can use the following command:

# install xrfeitoria first
cd xrfeitoria
pip install .

# for instance, build plugins for Blender, UE 5.1, UE 5.2, and UE 5.3 on Windows.
# using powershell where backtick(`) is the line continuation character.
python -m xrfeitoria.utils.publish_plugins `
    -u "C:/Program Files/Epic Games/UE_5.1/Engine/Binaries/Win64/UnrealEditor-Cmd.exe" `
    -u "C:/Program Files/Epic Games/UE_5.2/Engine/Binaries/Win64/UnrealEditor-Cmd.exe" `
    -u "C:/Program Files/Epic Games/UE_5.3/Engine/Binaries/Win64/UnrealEditor-Cmd.exe"

Frequently Asked Questions

Please refer to FAQ.

🚀 Amazing Projects Using XRFeitoria

Project	Teaser	Engine
SynBody: Synthetic Dataset with Layered Human Models for 3D Human Perception and Modeling		Unreal Engine / Blender
Zolly: Zoom Focal Length Correctly for Perspective-Distorted Human Mesh Reconstruction		Blender
SHERF: Generalizable Human NeRF from a Single Image		Blender
MatrixCity: A Large-scale City Dataset for City-scale Neural Rendering and Beyond		Unreal Engine
HumanLiff: Layer-wise 3D Human Generation with Diffusion Model		Blender
PrimDiffusion: Volumetric Primitives Diffusion for 3D Human Generation		Blender
WHAC: World-grounded Humans and Cameras		Unreal Engine

License

The license of our codebase is Apache-2.0. Note that this license only applies to code in our library, the dependencies of which are separate and individually licensed. We would like to pay tribute to open-source implementations to which we rely on. Please be aware that using the content of dependencies may affect the license of our codebase. Refer to LICENSE to view the full license.

Citation

If you find this project useful in your research, please consider cite:

@misc{xrfeitoria,
    title={OpenXRLab Synthetic Data Rendering Toolbox},
    author={XRFeitoria Contributors},
    howpublished = {\url{https://github.com/openxrlab/xrfeitoria}},
    year={2023}
}

Projects in OpenXRLab

• XRPrimer: OpenXRLab foundational library for XR-related algorithms.
• XRSLAM: OpenXRLab Visual-inertial SLAM Toolbox and Benchmark.
• XRSfM: OpenXRLab Structure-from-Motion Toolbox and Benchmark.
• XRLocalization: OpenXRLab Visual Localization Toolbox and Server.
• XRMoCap: OpenXRLab Multi-view Motion Capture Toolbox and Benchmark.
• XRMoGen: OpenXRLab Human Motion Generation Toolbox and Benchmark.
• XRNeRF: OpenXRLab Neural Radiance Field (NeRF) Toolbox and Benchmark.
• XRFeitoria: OpenXRLab Synthetic Data Rendering Toolbox.

xrfeitoria's Issues

XRFeitoria Assets

This issue is for asset storage.

Add a thread in runner for blender to print log of engine

Like using logger.info in remote_blender()

2023-11-01 15:13:27 |   INFO   | [blender] some log

some bugs when I run 'xf-render bunny.obj'

in init │
│ │
│ 200 │ │ ) │
│ 201 │ │ self._rpc_runner.start() │
│ 202 │ │ self.utils.init_scene_and_collection(default_level_blender, self._cleanup) │
│ ❱ 203 │ │ self.utils.set_env_color(color=(1.0, 1.0, 1.0, 1.0)) │
│ 204 │ │
│ 205 │ def enter(self) -> 'init_blender': │
│ 206 │ │ return self │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
KeyError: ''\'bpy_prop_collection[key]: key "Background" not found\'' File "/Users/xxx/anaconda3/envs/synbody/lib/python3.9/site-packages/xrfeitoria/utils/functions/blender_functions.py", line 119'

A bug in blender amass

hi, when I apply the 07_amass.py, error occurs.

type object 'XRFeitoriaBlenderFactory' has no attribute 'apply_motion_data_to_actor'  
File "E:\Project\xrfeitoria\xrfeitoria\utils\functions\blender_functions.py", line 69

How to render with depth and normal as well as meshes with texture?

Also, where to set camera extrinsics and intrinsics?

How to animate static mesh for rendering

Thanks for your work! I find in 07_amass.py, skeletal .fbx can be animated. But I only have static mesh(.obj) and smpl parameter, could you tell me how to animate static mesh for rendering? The methods I found on the Internet are all through the gui operation, while I have lots of mesh.
Thanks!

Render time

Thanks for your work!
From the rendering code, I see the rendering engine you use is 'CYCLES', the realism of rendering in 'Cycles' is superior to 'Workbench', right? But the rendering speed is much slower than using 'Workbench'. So, I'd like to know what devices you use for rendering this large-scale data.

How to generate SMPL-XL?

Thanks for your great work!

As you mentioned in #11 (comment), I would like to know if the automated script generated to add the garment to SMPL-X will be released.

Unreal Engine

When will rendering without GUI be supported?

Access to asset of motions

In your paper SynBody: Synthetic Dataset with Layered Human Models for 3D Human Perception and Modeling, you provide 1,187 motions. How can I access these motions when using xrfeitoria setup_animation functions? Thanks for replying.

SMPL_BODY_BONES

Hi,

I wonder why the joint names in SMPL_BODY_BONES is different from the official SMPL (e.g., the 'left_hand', 'right_hand' is commented) ?

Thanks.

MASK and Depth are totally black

I run python -m samples.blender.05_skeletalmesh_render
but the generated mask and depth are black.

I have tested the newest code in both macos (blender4.0) and windows(unreal5.1 & blender3.6.5).

May I ask a question about SynBody motion data

Thanks for your work. I'm trying to render some synbody models in my camera setting, rendering T-pose is easy. But I only find 3 motion models in .fbx format(e.g. actor.setup_animation(animation_path=https://openxrlab-share.oss-cn-hongkong.aliyuncs.com/xrfeitoria/tutorials/assets/SMPL-XL/walking__15_01.fbx)), could you please tell me how to create more motion models, maybe from some smpl-x.npy?

best xr tool

Multiprocessing support?

Thank you for your open-source work! May I ask if this tool supports multiprocessing? I would like to render a large number of 3D models in multiprocessing. In each process, I use xf.init_blender to create a scene, and then import a model for rendering. However, I have some confusion regarding xf.init_blider(exec_path=engine_exec_path, new_process=True).
If I use new_process=True, the error is RuntimeError: RPC server in RPC_PORT=9997 already started! This is raised when calling init_blender or init_unreal with new_progress=True when an existing server (blender or unreal) is already running.
If I use new_process=False, the error is ValueError: Invalid name, '[XF]actor-001' already exists in scene. or KeyError: '\'\\\'bpy_prop_collection[key]: key "[XF]actor-001" not found\\\. There seems to be a conflict between different processes.
Could you help me? Or whether multiprocess is useful, I want to use it because I find gpu utilization is very low.