alexandregastonbellegarde / orcl_vr_eyetracking

Tobii Eye Tracking for HTC VIVE Eye Pro in Unity

ORCL_VR_EyeTracking

Introduction

This repository comes with the code for Tobii Eye Tracking integrated in HTC VIVE Eye Pro in Unity, Which is a part of projects from Omni-Reality and Cognition Lab in University of Virginia (https://engineering.virginia.edu/omni-reality-and-cognition-lab).

Prerequisite

1. HTC VIVE Eye Pro with Tobii Eye Tracking system
2. Unity version 2018.4.16 or 2018.3.14
3. Python 3.6.3 (Anaconda version recommended)
4. SteamVR
5. Finish the Set up for the HTC VIVE Eye Pro
6. Tobii Pro SDK for your platform

The HTC VIVE Eye Pro hardware (headset, controller) is from HTC VIVE, the integrated eye tracker is from Tobii, they have provided multiple ways to get access to the eye tracking data:

• Tobii Pro SDK: A general SDK for getting eye tracking data. This repository will use Python and Unity only.
• Tobii XR SDK: SDK for Unity, developed by Tobii too, to get started, follow the steps in this link. Tobii XR requires a analytical license to get the raw data, otherwise, eye tracking can only be used for interactive use.
• Vive Eye Tracking SDK : SDK for eye tracking from HTC. The forum for it can be found here.

This repository includes sample code and tutorials for Python and Unity API of Tobii Pro SDK only.

Tobii Pro SDK data collection

Website of Tobii Pro SDK: http://developer.tobiipro.com/index.html

You can either use Python API or Unity API to get the eye tracking data.

Python API

Set up Python API as http://developer.tobiipro.com/python/python-getting-started.html.

Then run the TobiiEyeTracking.py in the repository to collect the data externally (not within Unity).

If an eye tracker is successfully found, the data collecting is on going until the key 'q' is pressed (you can also change it to another key in the code). An output .csv data file (name with the start and end time like sample_output) will be exported into the out_dir defined in the code.

output_dir = 'C:/github/ORCL_VR_EyeTracking/Data/EyeTrakcing/TobiiProPython'

Unity SDK

To start with, read the document from Tobii Pro SDK (http://developer.tobiipro.com/unity.html) and download the Tobii Pro SDK for Unity.

1. Create a new project, or open an existing project, in Unity.

2. Select Assets > Import Package > Custom Package... from the main menu, or by right-clicking in the Project window.

3. Browse to the downloaded Tobii Pro SDK, named with TobiiPro.SDK.Unity.Windows.

4. In the next dialog, select to import all files.

5. In the project window, Drag and drop the "TobiiPro\VR\Prefabs[VREyeTracker]" prefab into the scene and in the inspector, select 'Subscribe To Gaze'.

6. (Not required) Drag and drop the "TobiiPro\VR\Prefabs[VRCalibration]" prefab into the scene. Select the [VRCalibration] prefab and in the inspector, select a key to be used to start a calibration.

7. Drag and drop the "TobiiPro\VR\Prefabs[VRSaveData]" prefab into the scene. Select the [VRSaveData] prefab and in the inspector, select a key to be used to start and stop saving data, select 'Save Data/Save Unity Data/Save Raw Data'.

8. Play the scene, the saved XML data can be found in the "Data" folder in the project root. Press the save data key selected earlier to stop and save data.

   More details can be found in the TobiiProVR_readme.txt in this repository.

If a XML data was created without any recorded data, check in the windows system 'Task manager' - 'Services' - 'Tobii Service' to see if it is running or not, try to restart it and collect data again.

Video Recording

For Unity 2018, in the asset store of Unity, search for "Unity Recorder", download and import. This is a free library for recording user game.

In Unity 2019 or newer, the Unity Recorder can be found in Package Manager. Go to 'Window' - 'Package Manager', click 'advance' - 'show preview packages', find the 'Unity Recorder', install.

After import or install Unity Recorder, select Window > General > Recorder > Recorder Window from the main menu,

After setting the Recorder, press 'START RECORDING', or you may press 'F10' in the keyboard for quick start. The Frame rate is suggested to 24.

Since we have already set for VR eye tracking data saving, the data collecting process will start at the same time. The saved MP4 data can be found in the "Recordings" folder in the project root.

So far, we have already set up everything for data collection. For your convenience, I also upload a sample scene for the whole process, the Google Drive Link to it is here.

Process Eye Tracking Data

Suppose we have XML data collected in "Data" folder in the project root as what I have in the '\Data\EyeTrakcing\TobiiProUnity' folder in the repository, and the videos collected in the "Recordings" folder in the project root as what we have in the '\Data\Video\1.Raw Videos' folder in the repository. The goal of this part is to map the gaze data to the videos.

The three python scripts under 'EyeTrackingProcess' folder provides a workflow of processing eye tracking data.

• 0.video2pic.py Extract frame images from the videos.
• 1.ReadingTxtFile.py reads the XML file and reshape it to a more readable .csv file.
• 2.PlotEyeTrackingOnImgs.py reads the .csv file from last step and try to map them in the corresponding video frames and write the images. For more information about the coordinate systems, please refer to 'Useful tips and hints' on the bottom of this page and this page for more details. Note: the frame rate of the Unity Recorder should be set to 24, otherwise there might be problems for this script (like 30FPS, other frame rate hasn't be tested).
• 3.Img2video.py collects all the images and convert them back to a video with gaze like the sample output video in '\Data\Video\4.Gazevideos_out\movie2020-08-2818h08m.mp4'.