illixr / illixr

ILLIXR: Illinois Extended Reality Testbed

ILLIXR

Illinois Extended Reality testbed or ILLIXR (pronounced like elixir) is the first fully open-source Extended Reality (XR) system and testbed. The modular, extensible, and OpenXR-compatible ILLIXR runtime integrates state-of-the-art XR components into a complete XR system. The testbed is part of the broader ILLIXR consortium, an industry-supported community effort to democratize XR systems research, development, and benchmarking.

You can find the complete ILLIXR system here.

ILLIXR also provides its components in standalone configurations to enable architects and system designers to research each component in isolation. The standalone components are packaged together in the as of the v3.1.0 release of ILLIXR.

ILLIXR's modular and extensible runtime allows adding new components and swapping different implementations of a given component. ILLIXR currently contains the following components:

• Perception

  • Eye Tracking
    1. RITNet **
  • Scene Reconstruction
    1. ElasticFusion **
    2. KinectFusion **
  • Simultaneous Localization and Mapping
    1. OpenVINS **
  • Cameras and IMUs
    1. ZED Mini
    2. Intel RealSense

• Visual

  • Chromatic aberration correction
  • Computational holography for adaptive multi-focal displays **
  • Lens distortion correction
  • Asynchronous Reprojection (TimeWarp)

• Aural

  • Audio encoding **
  • Audio playback **

(** Source is hosted in an external repository under the ILLIXR project.)

We continue to add more components (new components and new implementations).

Many of the current components of ILLIXR were developed by domain experts and obtained from publicly available repositories. They were modified for one or more of the following reasons: fixing compilation, adding features, or removing extraneous code or dependencies. Each component not developed by us is available as a forked github repository for proper attribution to its authors.

Papers, talks, demos, consortium

A paper with details on ILLIXR, including its components, runtime, telemetry support, and a comprehensive analysis of performance, power, and quality on desktop and embedded systems.

A talk presented at NVIDIA GTC'21 describing ILLIXR and announcing the ILLIXR consortium: Video. Slides.

A demo of an OpenXR application running with ILLIXR.

For more up-to-date list of related papers, demos, and talks, please visit illixr.org.

The ILLIXR consortium is an industry-supported community effort to democratize XR systems research, development, and benchmarking. Visit our web site for more information.

The ILLIXR consortium is also holding a biweekly consortium meeting. For past meetings, for more information, past meeting recordings, and request for presenting, please visit here. Please join our Discord for announcement.

Citation

We request that you cite our following paper when you use ILLIXR for a publication. We would also appreciate it if you send us a citation once your work has been published.

@inproceedings{HuzaifaDesai2021,
  author={Huzaifa, Muhammad and Desai, Rishi and Grayson, Samuel and Jiang, Xutao and Jing, Ying and Lee, Jae and Lu, Fang and Pang, Yihan and Ravichandran, Joseph and Sinclair, Finn and Tian, Boyuan and Yuan, Hengzhi and Zhang, Jeffrey and Adve, Sarita V.},
  booktitle={2021 IEEE International Symposium on Workload Characterization (IISWC)}, 
  title={ILLIXR: Enabling End-to-End Extended Reality Research}, 
  year={2021},
  volume={},
  number={},
  pages={24-38},
  doi={10.1109/IISWC53511.2021.00014}
}

Getting Started and Documentation

For more information, see our Getting Started page.

Acknowledgements

The ILLIXR project started in Sarita Adve’s research group, co-led by PhD candidate Muhammad Huzaifa, at the University of Illinois at Urbana-Champaign. Other major contributors include Rishi Desai, Samuel Grayson, Xutao Jiang, Ying Jing, Jae Lee, Fang Lu, Yihan Pang, Joseph Ravichandran, Giordano Salvador, Finn Sinclair, Boyuan Tian, Henghzhi Yuan, and Jeffrey Zhang.

ILLIXR came together after many consultations with researchers and practitioners in many domains: audio, graphics, optics, robotics, signal processing, and extended reality systems. We are deeply grateful for all of these discussions and specifically to the following: Wei Cu, Aleksandra Faust, Liang Gao, Matt Horsnell, Amit Jindal, Steve LaValle, Steve Lovegrove, Andrew Maimone, Vegard Øye, Martin Persson, Archontis Politis, Eric Shaffer, Paris Smaragdis, Sachin Talathi, and Chris Widdowson.

Our OpenXR implementation is derived from Monado. We are particularly thankful to Jakob Bornecrantz and Ryan Pavlik.

The development of ILLIXR was supported by the Applications Driving Architectures (ADA) Research Center (a JUMP Center co-sponsored by SRC and DARPA), the Center for Future Architectures Research (C-FAR, a STARnet research center), a Semiconductor Research Corporation program sponsored by MARCO and DARPA, and by a Google Faculty Research Award. The development of ILLIXR was also aided by generous hardware and software donations from ARM and NVIDIA. Facebook Reality Labs provided the OpenEDS Semantic Segmentation Dataset.

Wesley Darvin came up with the name for ILLIXR.

Licensing Structure

ILLIXR is available as open-source software under the permissive University of Illinois/NCSA Open Source License. As mentioned above, ILLIXR largely consists of components developed by domain experts and modified for the purposes of inclusion in ILLIXR. However, ILLIXR does contain software developed solely by us. The NCSA license is limited to only this software. The external libraries and softwares included in ILLIXR each have their own licenses and must be used according to those licenses:

• ElasticFusion \ ElasticFusion license

• KinectFusion \ MIT License

• GTSAM \ Simplified BSD License

• HOTlab \ GNU Lesser General Public License v3.0

• libspatialaudio \ GNU Lesser General Public License v2.1

• Monado \ Boost Software License 1.0

• moodycamel::ConcurrentQueue \ Simplified BSD License

• Open-VINS \ GNU General Public License v3.0

• RITnet \ MIT License

Note that ILLIXR's extensibility allows the source to be configured and compiled using only permissively licensed software.

Get in Touch

Whether you are a computer architect, a compiler writer, a systems person, work on XR related algorithms or applications, or just anyone interested in XR research, development, or products, we would love to hear from you and hope you will contribute! You can join the ILLIXR consortium, Discord, or mailing list, or send us an email, or just send us a pull request!

illixr's Issues

Translate `audio_pipeline` into HPVM

• Translate file I/O
• Create leaf nodes
• Create root node
• Figure out HPVM build process
• Ensure end-to-end run
• Utilize streaming

Push mkdocs and doxygen on merge to master

Can probably do this in GitHub actions.

Expose our pose prediction to OpenXR apps

• Figure out how Godot queries pose.
• Confirm that xrLocateViews queries pose in OpenXR apps.
• Figure out how to handle this call in ILLIXR hmd in monado_integration.

Sync timing from ILLIXR to Monado

• Make the application's rendering (ILLIXR/timewarp_gl | Monado) to wake up right after vsync (known by ILLIXR/timewarp_gl).
• Make xrWaitFrame wait until next vsync period.

Add Google Analytics to our websites

• We use three different generators for our three different sub-sites. Google Analytics tracking codes have to be added to each sub-site.

• Google Analytics will allows us to track user demographics (motivate translations), popularity over time (evaluate which marketing works), and which order users visit our pages (help lay out the documentation in a user-friendly way).

• Main website

• Doxygen site

• mkdocs site

Log M2P latency

• Use logging framework

Improve install_deps.sh

Print detailed message listing exactly what will be installed.

Research Python in Spindle

Can we support dynamic languages like Python?
Python is a bit of a special case, because the interpreter supports embedding into a larger process.

Evaluate Stereokit

Evaluate Ninja/CMake/Bazel for plugin build system

Fix documentation bugs

• default_plugins.md: Remove "TODOs"
• default_plugins.md: Remove NVIDIA GPU info from audio
• writing_your_plugin.md: Doulble-check for correctness with the new build system.

Standardize clock in every component

• In some places we use std::chrono::system_clock; in others we use std::chrono::high_resolution_clock. Which is better?
• Is there a way we could make this work on Windows?

Stylistic improvements

• Header functions should be declared static.
• Use #pragma once instead of header guard.
• Either always use _m_field and _p_method or never.

Camera/IMU Calibration Files

• Standard Camera/IMU calibration format for ILLIXR
• Have runner or runtime parse this file and give as data structure for a slam implementation

Add human-readable logging infrastructure

Our record_logger and the CPU timers in #209/#211 are good for computer-readable telemetry. However, we need an infrastructure for data intended to get to a user, such as warnings, error descriptions, and debugging messages.

Perhaps, we should use glog.

• By default, ILLIXR should not print anything except for really critical stuff (why this is considered "good UNIX style")
  • Logs in a loop can use something like glog's LOG_EVERY_N.
• Many of the assert(thing && msg) can be replaced with glog's CHECK(thing) << msg.
  • This permits the message to be a dynamically generated string, like CHECK(thing) << getErrorMessage(msg) which would not work with a raw assert.
  • CHECK is not suppressed by NDEBUG, which has caused us bugs in the past (and pesky [[maybe_unused]] annotations).
    • One can still explicitly reserve a check for debug builds by surrounding the CHECK with #ifndef NDEBUG, as we do already in many places (e.g. common/record_logger.hpp).
• Glog can dump the stacktrace on segfault.
  • We do already have catchsegv. There's not a whole lot of benefit to switching over to glog for that.
• Glog embeds the filename and line number in all logs.
• Perhaps we want to "selectively enable" debug flags for different plugins, as in LLVM. This is the only thing not readily avilable in glog.

Draft open-source community procedures

• How to deliberate over design decisions (RFC)
• How to get help
• How to file a bug
• How to suggest a feature
• How to setup dev environment
• What is the code of conduct?
• How to cite borrowed code?

Recommended reading:

• GitHub Open Source Guides
• TODO group guides
• Governance Models

Compile and run with sanitization

• This should be a new "compilation profile"
• Address sanitization and undefined behavior sanitization

Time camera "format conversion" separately

Explore OpenXR and OpenGL extensions for depth, vsync, and preemption priority

Make ZED datasets

Unlike EUROC, this would have RGB/depth data.

Use a C++ linter

• Possibly OCLint
• Possibly cppclean
• Possibly clang-tidy
• Should detect unused #include

Confirm ILLIXR works on Jetson with applications

• AR Application: Create one in Godot.
• VR Application: Sponza

Configurable ILLIXR timeout

Currently, ILLIXR timesout with 60 seconds. We need a way to configure this time or turn the timeout off. This should probably be set in the config yaml file for runner.py.

Integrate ORB-SLAM3

Rationale: full VI-SLAM system with loop closure. ORB-SLAM3 has maps that stay persistent across runs, and we will need an efficient mechanism in ILLIXR to store/retrieve them.

Direct mode conversion

Convert timewarp to use direct mode, using Monado compositor as reference.

Fix `install_deps.sh`

• Install Python3.8 (safely, with pyenv)
• Install Poetry with Python3.8
• Install clang-10
• Make clang-10 default (don't want to export every time).
• Install curl, zip.

Fail cleanly when both offline_imu_cam and ZED plugins are specified

If user specifies both offline_imu_cam and ZED plugins, the system should fault. Currently, it might assign both feeds to the same switchboard topic, resulting in a spliced feed.

Fix ZED Cmake

ZED's CMakeLists.txt currently has an additional linkage step which requires us to manually add C++ flags and a fake main(). Need to clean this up.

Set up godot repo

• Use 3.2.1 as base
• Add timing instrumentation
• Exit cleanly

Implement Switchboard bag

• Like ROS bag, but for Switchboard.
• Maybe we could convert EUROC to that. This would obviate the need for offline_imu_cam and groundtruth_slam.

Implement Pose Prediction

• See if OpenVINS pose integration will suffice.
• Rip out the Newtonian mechanics pose prediction from OpenVINS.
• Expose it through the Phonebook pose_prediction interface.
• It needs some state and some IMU readings. Inside feed_imu_cam, push the last few IMU readings + relevant state into a topic called integrated_state.
• Read async integrated_state in the pose prediction.

Evaluate Vulkan migration for postprocessing

Add a logging library

We need a logging library for human readable debugging/info. Probably want to use glog for this.

Run clang-format

• Deliberate on the format
• Also standardize naming conventions (always use _m_ for private or never)
• Figure out how to enforce (make part of CI?)

Dumping to `imu_cam` topic to disk

• Same format as EUROC dataset
• Should be able to record our own ZED datasets

Add CI/CD workflow

A workflow needs to run on branches that builds and pushes docker images to ghcr.io and runs the tests in ILLIXR.

Windows support

• Windows support for offline IMU Camera
• Document Panacea plugin translation for Windows
• Windows support for OpenVINS
• Investigate what would be necessary for MacOS/iOS?

Investigate Repo for submodules

Requirements:

• System should be able to checkout (possibly new branch) the main module and every submodule to a branch of the same name.
• System should be able to merge the main module and every submodule simultaneously.

Consider:

• Repo

Replace `std::mutex` with `std::shared_mutex`

A std::shared_mutex permits EITHER n readers OR 1 writer.

• When a thread reads a datastructure, it only needs to acquire std::shared_lock. This permits parallel reads, since they don't conflict, but locks out all writers.
• When a thread writes a datastructure, it needs to acquire std::unique_lock. This locks out all other writers and all readers.

Integrate a scene reconstruction algorithm

• See if ElasticFusion can be integrated.
• If not, find and use another.

Write per-component unittests (where possible)

Develop solutions for rendering Switchboard metrics and logging in Debug View

• In the logger, redirect some messages to a Switchboard topic.

Set up CI testing

For most components, the best we can do is ensure it builds.
However, some components can also have tests make test.
In the past, broken code (non-compiling) has been pushed to master, so automating this has real benefit.

Separate IMU/camera Switchboard topics

Or explain why we shouldn't.
This currently complicates the ZED code (where the Camera thread has to send frames to the IMU thread, which pairs them and sends them on a Switchboard topic).
This currently simplifies OpenVINS (where we need to process the camera after the IMU).

When dumping images, use timestamp at the _end of the upcoming vsync cycle_ not the _end of timewarp_

• If timewarp[i] is on time (earlier than vsync[i]), image should bear the timestamp of vsync[i].
• If timewarp[i] is late, then it should bear the timestamp of vsync[i+1].
  • (However, it may be overwritten by timewarp[i+1], if timewarp[i+1] finishes before vsync[i+1]. The latest timewarp before vsync should be used. This overwriting is how normal file semantics work, so shouldn't complicate the implementation).
• Use pose_predict to get the timestamp of the next (nearest after now()) vsync.
  • This interface is not yet present; It is being developed by @JeffreyZh4ng .

Dump eyebuffers (after timewarp) as image-files

• Should contain timestamp in filename.
• Should dump to metrics/eyes/left/{timestamp}
• Should use timestamps := time since start in ms.

Split `audio_component` into encode/decode

• Leave the submodule where it is.
• Symlink relevant parts into a new plugin for decode and a new plugin for encode.
• Adjust plugin line in Makefile.

Audit for exception-correctness

• https://docs.microsoft.com/en-us/cpp/cpp/errors-and-exception-handling-modern-cpp?view=vs-2019
• https://docs.microsoft.com/en-us/cpp/cpp/how-to-design-for-exception-safety?view=vs-2019
• https://docs.microsoft.com/en-us/cpp/cpp/exception-specifications-throw-cpp?view=vs-2019

Exit gracefully

• Have a designated way to exit the program (Ctrl+C and handle SIGINT or have a way for a plugin to signal to the runtime "game over").

Add global shutter camera

• Decide which camera to get
• Write standalone camera code
• Write plugin