microsoft / mixedreality-graphicstools-unreal

Graphics tools and components for developing Mixed Reality applications in Unreal Engine.

What is Graphics Tools?

Graphics Tools for Unreal Engine is a UE game plugin with code, blueprints and example assets created to help improve the visual fidelity of Mixed Reality applications while staying within performance budgets.

NOTE: Only HoloLens 2, Windows, and Android development is tested at the moment. If you encounter issues with other platforms please file an issue.

Getting started with Graphics Tools

If you're new to MRTK or Mixed Reality development in Unreal, we recommend starting at the beginning of our Unreal development journey in the Microsoft Docs. The Unreal development journey is specifically tailored to walk new developers through the installation, core concepts, and usage of Unreal Engine for Mixed Reality.

If you're already familiar with Mixed Reality development in Unreal, you can find the installation instructions for the Graphics Tools plugin here.

Required software

Windows SDK 18362+	Unreal 5.0.0+	Visual Studio 2019
To build apps with MRTK-Unreal and Graphics Tools, you need the Windows 10 May 2019 Update SDK	The Unreal Engine provides support for building Mixed Reality projects in Windows 10	Visual Studio is used for code editing

Graphics building blocks

Lighting	Proximity Lights	Effects
Physically based lighting technique optimized for Mixed Reality	A custom light type used to demonstrate distance to surface	Common graphics effects utilized in Mixed Reality
Clipping Primitives	Profiling	Spatial Perception
Tools to dynamically slice away a mesh and peer inside the geometry	Explore profiling techniques and tools useful for Mixed Reality development	Techniques that create visually compelling materials for real world geometry
Mesh Outlines
Editor and runtime components to render an outline around a static mesh

Example levels

If you want to explore the different Graphics Tools features or want a reference for how to use them, we recommend having a look at the example levels contained in the Graphics Tools Examples plugin (/GraphicsToolsProject/Plugins/GraphicsToolsExamples). You can copy the examples plugin (along with the GraphicsTools plugin) into the 'Plugins' subfolder of your own project or you can simply try out the Graphics Tools Project (/GraphicsToolsProject) in this repository. For that you should:

1. Clone this repository.
2. Checkout the latest 'public/version.x' branch.
   • Bear in mind that this branch is alive. It's not a release, and will be updated regularly with potentially breaking changes. There will be a release tag (e.g. release/0.4.0) marked as such in GitHub.

You can now open the Graphics Tools Project (/GraphicsToolsProject) and explore individual example levels or open the Loader level to access some of the examples from a centralized hub. You can also copy the examples (/GraphicsToolsProject/Plugins/GraphicsToolsExamples) into the 'Plugins' folder in your own project and use the pre-made elements as a base for your own Mixed Reality experience.

Please note that the example content might change significantly between releases. Please make sure to always run the examples with a matching version of the Graphics Tools Plugin as there might be breaking changes in the example code.

Packaged Graphics Tools project

We also provide the Graphics Tools Project (/GraphicsToolsProject) pre-packaged for HoloLens 2 so you can try out the main features directly on device easily. To use it:

1. Obtain the packaged game from the latest release page (e.g. GTProject-HoloLens.0.4.0.zip) and unzip it to a local directory.
2. Install it in the device via the Device Portal.

Videos

	Webinar showing how to build a product configurator using Graphics Tools

Feedback and contributions

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

mixedreality-graphicstools-unreal's Issues

GTCommon.ush Syntax Error with Half Precision Floats

When compiling the project for Android the below shader compilation error is hit:

LogShaderCompilers: Display: /Plugin/GraphicsTools/Common/GTCommon.ush(29): error: syntax error, unexpected NEW_IDENTIFIER, expecting $end

This is due to min16float not being defined for Android. To fix the issue change in /Plugin/GraphicsTools/Common/GTCommon.ush change:

#define Half min16float
#define Half2 min16float2
#define Half3 min16float3
#define Half4 min16float4
#define Half3x3 min16float3x3
#define Half3x4 min16float3x4

to

#define Half float
#define Half2 float2
#define Half3 float3
#define Half4 float4
#define Half3x3 float3x3
#define Half3x4 float3x4

Or, ideally determine if a platform supports half precision and specify the right defines accordingly.

Potential leakage vector via resource monitoring APIs

Describe the bug

Summary

Your codebases (GraphicsTools-Unreal and GraphicsTools-Unnity) provide resource monitoring APIs to help developers stabilize their games’ rendering performance. However, we found that such resource monitoring APIs could be exploited by attackers to launch side-channel attacks on AR/VR systems. Our projects have demonstrated such side-channel attacks could be deployed on the Microsoft Hololens 2 and Oculus Quest 2.

We reported our findings to Microsoft Bug Bounty Program. The case submission is CRM:0461000294.
However, the issue was marked as Not a Vulnerability. The behavior is considered to be by design.

Also, the work on side-channel attack has been accepted to Usenix Security 2023 (https://www.usenix.org/conference/usenixsecurity23/presentation/zhang-yicheng). We are happy to share all our findings with you and help your product fix such vulnerabilities.

To reproduce

Directly presenting security issues here may bring in some ethical issues.
Thus, can we officially communicate via email?
Sorry, I can not find your contact information. But you can contact me via [email protected].
Completely blocking all these APIs seems unreasonable since they are important for tuning AR/VR games. We can meet to discuss some defense methods for mitigating such leakage vectors.

Expected behavior

You can check our paper for all 5 possible side-channel attacks.

Your setup (please complete the following information)

• Unity version 2020.3.16f1 and Unreal Engine version 4.27.2.
• Graphics Tools Version [v0.4.0]

Target platform (please complete the following information)

• HoloLens 2
• Meta Quest 2

Tasks

Beta Give feedback

No tasks being tracked yet.

Tasks

Beta Give feedback

No tasks being tracked yet.

Misleading Installation Guide Step

The installation guide mentions:

Download the packaged plugin zip from the latest release page (e.g. GTTools.0.1.0.zip).

GTTools.0.1.0.zip should read MixedReality-GraphicsTools-Unreal-0.1.0.zip

Visual Profiler Compile Errors in Unreal 4.27

An API used by the Visual Profiler has been changed in Unreal 4.27.

A preprocessor for engine version check might be needed together with the above changes:

"Misc/EngineVersionComparison.h"

#if !UE_VERSION_OLDER_THAN(4, 27, 0)

// code that only compiles in 4.27

#endif

Clipping Box renders masked material as translucent with MSAA

I have a simple material, set to masked with Opacity Mask Clip Value set to 0 so it doesn't feather.

I am using forward rendering with MSAA, this is the result. It works fine with other AA options

Does Hololens2 have plan to support offline TTS suport?

Does Hololens2 have plan to support offline TTS suport?

Generated outline meshes sometimes have "chamfered edges."

When generating an outline mesh using the mesh outlines feature sometimes the resulting geometry has a few artifacts. For example a flat top at the tip of the cone:

This issue is also encountered in MRTK-Unity with the same source mesh:

That hints that the issue is likely not in the port to Unreal, but the smoothing algorithm itself.

Using the Visual Profiler with the D3D12 RHI doesn't update GPU time.

OpenXR switches to the DirectX 12 RHI which doesn't support GPU time queries. Ensure this case is gracefully handled by the Visual Profiler.

ACTION REQUIRED: Microsoft needs this private repository to complete compliance info

There are open compliance tasks that need to be reviewed for your MixedReality-GraphicsTools-Unreal repo.

Action required: 4 compliance tasks

To bring this repository to the standard required for 2021, we require administrators of this and all Microsoft GitHub repositories to complete a small set of tasks within the next 60 days. This is critical work to ensure the compliance and security of your microsoft GitHub organization.

Please take a few minutes to complete the tasks at: https://repos.opensource.microsoft.com/orgs/microsoft/repos/MixedReality-GraphicsTools-Unreal/compliance

• The GitHub AE (GitHub inside Microsoft) migration survey has not been completed for this private repository
• No Service Tree mapping has been set for this repo. If this team does not use Service Tree, they can also opt-out of providing Service Tree data in the Compliance tab.
• No repository maintainers are set. The Open Source Maintainers are the decision-makers and actionable owners of the repository, irrespective of administrator permission grants on GitHub.
• Classification of the repository as production/non-production is missing in the Compliance tab.

You can close this work item once you have completed the compliance tasks, or it will automatically close within a day of taking action.

GitHub inside Microsoft program information

More information about GitHub inside Microsoft and the new GitHub AE product can be found at https://aka.ms/gim or by contacting [email protected]

FYI: current admins at Microsoft include @sw5813, @Cameron-Micka, @luis-valverde-ms, @paco-ms