caothu3d / augmentedunreality

Augmented reality for Unreal Engine 4

Augmented Unreality

Augmented reality for Unreal Engine 4

by Krzysztof Lis (Adynathos)

Features

• in-game displaying of camera video stream
• augmented reality, camera position tracking with ArUco markers
• camera calibration

Augmented reality in UE:

Camera calibration:

In-game video screen:

Download

• Augmented Unreality Project 1.00

Installation

Running the example project:

• download the latest package here
• decompress the archive - and move AugmentedUnrealityProject to the location where you store your Unreal projects
• launch Unreal Engine and open AugmentedUnrealityProject/AugmentedUnrealityPr.uproject
• optional: perform camera calibration

Augmented reality example:

• print or display the tracker marker AugmentedUnrealityProject/Saved/AugmentedUnreality/Markers/marker.png (the game needs to be run at least once to generate it)
• launch the game and point camera at the marker

In-game screen example:

• open AugmentedUnrealityProject/Content/Maps/AURExample_ARScreen.umap
• launch the game

Camera calibration

Best quality is obtained if the camera is calibrated. It is important to find the camera's field of view to match the camera image and the AR objects.

The driver first tries to load the calibration data from AugmentedUnrealityProject/Saved + Driver.CalibrationFilePath, then if it fails, AugmentedUnrealityProject/Content + Driver.CalibrationFallbackFilePath. This mechanism should allow distributing a default calibration file in the project's Content directory. A default calibration file is supplied with the example project.

To perform calibration of your camera:

• print or display on an additional screen the calibration pattern found in AugmentedUnrealityProject/Content/AugmentedUnreality/Calibration/calibration_pattern_asymmetric_circles.png
• open the example project and start the game
• in the menu in the top-right corner of the screen, click Calibrate
• point the camera at the calibration pattern from different directions - pattern is detected if a colorful overlay is drawn
• wait until the progress bar is full
• the calibration data is now saved to file AugmentedUnrealityProject/Saved/AugmentedUnreality/Calibration/camera.xml by default, or in general to AugmentedUnrealityProject/Saved + Driver.CalibrationFilePath if you changed that property of your driver
• if you use the same driver blueprint next time, this file will be automatically loaded.

Camera capture

Camera stream capture is performed using OpenCV's VideoCapture.

To choose which camera is used, change the driver blueprint's CameraIndex attribute. The driver attempts to set the camera's resolution to the desired resolution specified in the Resolution attribute, however it is not guaranteed that the camera may choose not to accept this setting. Generally, lower resolution means faster refresh rate.

Tracker markers

This plugin uses ArUco boards for camera pose estimation, specifically the implementation of ArUco in OpenCV contrib.

The parameters of the marker used can be changed in the driver blueprint's TrackerSettings.BoardDefinition attribute. Tracker markers are created and saved to file when the application starts. By default they are saved to AugmentedUnrealityProject/Saved/AugmentedUnreality/Markers/marker.png but this location can be changed in the settings.

Engine version

The pre-built package was built with UE4.11.2.

Presented solutions

The following problems have been solved in this plugin, if you want to learn about these topics, please see:

• Including external libraries in UE4
• Multi-threading in UE4 1 2 3 4
• Performing OpenCV camera calibration OpenCV tutorial integration 1 integration 2
• Drawing on dynamic textures UE tutorial (a bit old) our adaptation
• Conversion between OpenCV's and Unreal's coordinate systems