microsoft / microsoft-rocketbox

Microsoft Rocketbox is now available with MIT license! The library of 115 rigged avatars offers flexibility, enabling the easy use of animations across characters and the mixing and matching of texture elements.

MICROSOFT ROCKETBOX AVATAR LIBRARY

The Microsoft Rocketbox Avatar library consists of 115 characters and avatars fully rigged and with high definition that was developed over the course of 10 years. The diversity of the characters and the quality of the rigging together with a relatively low-poly meshes, makes this library the go-to asset among research laboratories worldwide from crowd simulation to real-time avatar embodiment and social Virtual Reality (VR). Ever since their launch, laboratories around the globe have been using the library and many of the lead authors in the VR community have extensively used these avatars during their research.

The announcement about the release here> https://www.microsoft.com/en-us/research/blog/microsoft-rocketbox-avatar-library-now-available-for-research-and-academic-use/

This release goes together with a paper that highlights and documents the creation of the library and reviews the research done with rigged avatars as well as explains the importance of having rigged avatars for Virtual Reality. If you are using this library for research you should consider citing it.

Mar Gonzalez-Franco, Eyal Ofek, Ye Pan, Angus Antley, Anthony Steed, Bernhard Spanlang, Antonella Maselli, Domna Banakou, Nuria Pelechano, Sergio Orts Escolano, Veronica Orvahlo, Laura Trutoiu, Markus Wojcik, Maria V. Sanchez-Vives, Jeremy Bailenson, Mel Slater, and Jaron Lanier "The Rocketbox library and the utility of freely available rigged avatars." Frontiers in Virtual Reality DOI: 10.3389/frvir.2020.561558

https://www.youtube.com/watch?v=43OWnUQH_p4

UPDATES

6/2022: Release of ARkit compatible blendshapes. Contribution by Fang Ma and Matias Volonte

4/2022: Release of 417 animations compatible with the Microsoft Rocketbox Avatars (in Assets/Animations)

3/2022: Release of a new library: the Headbox to do facial blendshape animations on the avatars https://github.com/openVRlab/Headbox

2/2022: Release of avatars with facial blendshapes (15 visemes, 48 FACS, 30 for the Vive facial tracker).

2/2021: Added Unreal batch importer tool. Contribution by Joachim Tesch

12/2020: Release of a new library: the Movebox for Microsoft Rocketbox to animate the avatars https://github.com/microsoft/MoveBox-for-Microsoft-Rocketbox

12/2020: Updated license to MIT.

Setup

The FixRocketboxMaxImport.cs script under “Assets/Editor” needs to go in “Assets/Editor” in the Unity project. This will fix the import of the 3dsMax materials to Unity. I.e. Max materials assume that diffuse material was set by the texture, whereas Unity multiplies the texture colour with the flat colour. Second Unity's transparent materials still show specular highlights and thus hair looks like glass sheets. The material mode "Fade" goes to full transparent. The import tool also selects the highest resolution mesh as being activated by default. By editing this file you might choose another poly level (they are "hipoly", "midpoly", "lowpoly" and "ultralowpoly") Or you could choose not to import by changing OnPreprocessMeshHierarchy.

The FixRocketboxMaxImport was first contributed by Prof. Anthony Steed from University College London. In a more recent version we have updated the import tool to set the avatars into T-pose and reorganize the bones to optimize for the Unity humanoid rig.

Running the sample

Import the desired avatar folder (including fbx files and textures of the avatar) to your unity project "Assets" folder.

Once the files are on the correct unity folders project you can open Unity and include the avatar to the scene.

Contributors

Mar Gonzalez-Franco - Microsoft Research

Markus Wojcik - Rocketbox (Original avatar creation team)

Eyal Ofek - Microsoft Research

Anthony Steed - University College London, (Visiting Researcher at Microsoft Research when this repo was conceived)

Dave Garagan - Havok

Matias Volonte - Northeastern University & Clemenson University (Creation of the blendshapes, and the headbox tool to transfer blendshapes procedurally between avatars)

Fang Ma - Goldsmith University, Creation of blendshapes for Vive Tracker and ARkit

November 2020 License Update

The library of avatars is now released under MIT License.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

microsoft-rocketbox's Issues

Skeleton Structure

The RocketBox paper mentions that the avatars have 53 body bones and 28 facial bones. Where can I find the list of the names of all the bones and a figure to map these bone-names to their location?
I require this to map the pose data from a video to the RocketBox avatars. I am aware the MoveBox repository provides this functionality for the VIBE pose detection algorithm.
I am planning on using a different pose detection method (ex: MediaPipe Pose) and hence, would need the skeleton structure of RocketBox avatars.
Thanks.

Not Unity-ready

Is there a tutorial, guide, or script to make these models Unity-ready, without having to fix hundreds of bones hierarchies? Or is there a set available that is Unity-ready?

Problems with FixRocketBoxMaxImport.cs

I'm having this issue - I placed the .cs file under Assets/Editor and edited it from Generic to Humanoid. This allowed me to set the avatar as humanoid, however it has a bunch of Max related errors and the hair and textures are not working correctly.

Thanks!

Many thanks @margonzalezfranco for this initiative !

Some years ago our research on crowd simulation and crowd rendering was based on the first generation of this dataset.

https://www.youtube.com/watch?v=xE7tK7wWinM

It was great to know Microsoft has open source this library.

Retarget to ue4 maniquin

Not really a issue but no we're else to ask really. But wondering if anyone has successfully retargeted to ue4 maniquin skelaton

Use of some animations (e.g. CMU mocap database) leads to strange corruption of face

I'm not a Unity expert so perhaps there's something I'm doing wrong when importing? The CMU mocap database animations (http://mocap.cs.cmu.edu/) don't seem compatible with these models and the results are somewhat terrifying! It would be really useful to upload a small unity project with 3/4 of these models using a couple of open source animations (e.g. Mocap from CMU & mixamo) to promote research use. I'm using Unity 2018.4.12.

Assets/Avatars/Adults/Male_Adult_18/Textures/m023_head_normal.tga is broken

https://github.com/microsoft/Microsoft-Rocketbox/blob/8046857/Assets/Avatars/Adults/Male_Adult_18/Textures/m023_head_normal.tga is not a valid TGA file:

> file Assets/Avatars/Adults/Male_Adult_18/Textures/m023_head_normal.tga
Assets/Avatars/Adults/Male_Adult_18/Textures/m023_head_normal.tga: data

In contrast, other .tga files are valid:

> file Assets/Avatars/Adults/Male_Adult_18/Textures/m023_body_normal.tga 
Assets/Avatars/Adults/Male_Adult_18/Textures/m023_body_normal.tga: Targa image data - RGB 2048 x 2048 x 24

Chef_Female_01 has inconsistent name for head material (f110_A_head insteal of f110_head)

All models except for Female_Adult_16 and Chef_Female_01 have a material with a name following the pattern <1-2 chars><3 digits>_head. Female_Adult_16 does not have a head material at all, presumably because the model is quite special.

In fact, Female_Adult_16 has only one material, named f204.

In contrast, Chef_Female_01 does have a head:

This model has two materials: f110_body and f110_A_head. Perhaps it would be better to rename f110_A_head to f110_head for consistency?

Using MoveBox with MediaPipe instead of VIBE

Reposting here the link to an issue I opened in the MoveBox repository:
microsoft/MoveBox-for-Microsoft-Rocketbox#8 (comment)

Documentation - ModelImporterAnimationType

Hello,

Thank you for providing this great package! I just wanted to let you know that it might be helpful to mention in the documentation / read.me that the attached script "FixRocketboxMaxImport.cs" overwrites the ModelImporterAnimationType (L62). This leads to the fact that you cannot manually select "Humanoid" in the ImportSettings of an avatar.fbx in Unity. A brief note on this would certainly have saved me some time and prevented some confusion.

Many thanks and best regards
David

Some file names are inconsistent

• https://github.com/microsoft/Microsoft-Rocketbox/blob/8046857/Assets/Avatars/Adults/Female_Adult_10/Export/Female_Adult_01.fbx should be Female_Adult_10.fbx.
• https://github.com/microsoft/Microsoft-Rocketbox/blob/8046857/Assets/Avatars/Professions/Security_Male_02/Export/Security_Male_01.fbx should be Security_Male_02.fbx

Facial animations files missing (e.g. "m_male_face_expressions")

Rocketbox used to come with facial animations before it was open sourced (referenced in posts dating back to 2014).

"m_male_face_expressions" (and presumably a female equivalent).

These animations are missing from the repository.

Can they be added?

T-pose issues Vicon Pegasus

Hi,

I'm very interested in using some of your avatars for my research project. I'd like to use them in Unity and Vicon Pegasus, however, within Vicon Pegasus the characters are not in a T-pose which causes issues when I want to align the Vicon objects with the avatar (see 2 photos: Imported avatar in Pegasus). The avatar's arms and shoulders are all crushed when I stream the data in Unity as I had to change the position of the avatar in Pegasus to better align the avatar towards the Vicon objects (see photo: Avatar in Unity).

Would it be possible to change the skeleton of the avatars in T-pose by default? So I can import them in Pegasus and Unity to avoid this issue. I've tried it myself by changing the skeleton in Maya before importing them in Pegasus and use that also in Unity but that didn't solve the issue.

Appreciate you're help

Many thanks,
Iris

Lipsync in Unity

Hello!

Is it possible to implement live lipsync on these models using just the bones in Unity? I saw that the models don't have blendshapes at all so I was curious if this was possible.

Regards

Exported model + animation is not compatible with Blender

I follow the instructions via https://youtu.be/gqJTjuyNfh8 and this works.

Then I export the model and animation as the .fbx format. However, when I import .fbx into the Blender, the mesh is distorted. Could anyone help me with this?

Corrupted zip file?

I've tried downloading these all as a zip but it appears to be corrupted.

Every time I try to extract the contents of the zip, I get an error saying the archive is corrupt.

I am also getting lots of errors when trying to download individual files instead; tga files that won't open, fbx files that appear to be corrupted.

Importing in Blender

I'm having trouble importing models in Blender. I know this dataset is meant for Unity, but I still hope for some help with Blender specifically.

Here is the script that I eventually came up with to import a single FBX file:

from pathlib import Path
import bpy

rocketbox_root = Path('/path/to/cloned/Microsoft-Rocketbox')
avatar_path = rocketbox_root / 'Assets/Avatars/Adults/Female_Adult_01/Export/Female_Adult_01.fbx'

bpy.ops.object.delete(use_global=False, confirm=False)  # delete cube
bpy.ops.import_scene.fbx(filepath=str(avatar_path))

# Fix paths to textures
for im in bpy.data.images:
    im.filepath = im.filepath.replace('../../../../../temp/Humans/with_opacity_version/f001/textures', '../Textures')

# Workaround: make opacity texture fully transparent
bpy.data.materials['f001_opacity'].blend_method = 'CLIP'
bpy.data.materials['f001_opacity'].alpha_threshold = 1

In order to use it, launch Blender, switch to the "Scripting" tab, create a new script via the "+ New" button, paste this code and change the rocketbox_root variable to the path of the cloned repository.

In this script, I replace paths to textures which are hardcoded in the FBX file. In addition, I disable the opacity texture by making all polygons associated with it fully transparent. Here is what the imported model looks like without that step:

If I set blend method to "alpha blend", I get this:

The result with my workaround (mode = alpha clip, clip threshold = 1) is as follows:

All of these images are rendered with the Eevee renderer, on white background. I am using Blender 2.83.

I am also not sure what the purpose is for the eyelash opacity textures. The eyelashes are already baked into the color map for the head texture, why duplicate them in the opacity texture?

Are there any guidelines for importing the models in Blender? Ideally, I would prefer a Python script that does everything, not instructions on what things need to be changed in the GUI.

Use of some animations (e.g. CMU mocap database) leads to strange corruption of face

I'm not a Unity expert so perhaps there's something I'm doing wrong when importing? The CMU mocap database animations (http://mocap.cs.cmu.edu/) don't seem compatible with these models and the results are somewhat terrifying! It would be really useful to upload a small unity project with 3/4 of these models using a couple of open source animations (e.g. Mocap from CMU & mixamo) to promote research use. I'm using Unity 2018.4.12.

Missing source files

Many of the animals are mising from the source folders as i see. There i see only the image of the model but not the .max file.
If can update it will be great. Thank you

Human rig configuration and Blend shape

It is difficult to use this as an agent character because avatars do not have a blend shape for blink.
In addition, this rig configuration does not match the humanoid rig. Then, I cannot apply humanoid animations to avatars