microsoft / 3dtelecommunications

3D Telecommunications project utilizing Holoportation technology to provide live volumetric capture. Used in one case to increase the reach of specialized medicine.

3D Telecommunications

This is the open source repository for the 3D Telecommunications system created by Microsoft Research Special Projects Group and used in the 3D Telemedicine application: https://www.microsoft.com/en-us/research/project/3d-telemedicine/

System Requirements

The typical system layout for 3D Telemedicine is to have one machine running Fusion, and a separate machine running Render and Viewer. The pods all run their own copy of the K4AToFusion code and should be networked into the same network as Fusion. Control panel can be run on the Fusion PC, the Render PC, or a separate PC or tablet.

Minimum Operating Requirements (Fusion)

• Windows 11
• CUDA 11.7
• Nvidia GPU 2080Ti or larger
• 10 Gigabit ethernet adapter
• OpenCV 4.5.4

We use OpenCV with CUDA libraries. This means you cannot just use the binaries available directly on the website, although there are typically additional sites that provide OpenCV with CUDA built in.Currently our project requires the opencv_contrib folder also be checked out of GIT and compiled as part of OpenCV. We hope to remove this dependency in the future.

Minimum Operating Requirements (Render, Viewer)

• Windows 11
• CUDA 11.7
• Nvidia GPU 2080Ti or larger
• 10 Gigabit ethernet adapter

Minimum Operating Requirements (Control Panel)

• Windows 11
• 1 Gigabit ethernet adapter

Minimum Operating Requirements (K4AToFusion, K4ARecorder)

• Azure Kinect SDK
• Nvidia Jetson Nano

Building from Source

3rd Party Components (Windows)

This system relies on the following open-source 3rd party components:

Component	Version	Link
boost	1.72.0	https://www.boost.org/users/download/
ceres-solver	1.9.0	https://ceres-solver.googlesource.com/ceres-solver
cppzmq	4.7.1	https://github.com/zeromq/cppzmq
freeglut	3.0.0	https://freeglut.sourceforge.net/
glew	1.7.0	http://glew.sourceforge.net/
libjpeg	9c	https://github.com/winlibs/libjpeg
jsoncpp	1.8.1	https://github.com/open-source-parsers/jsoncpp
libzmq	4.2.0	https://github.com/zeromq/libzmq
LZ4	1.8.3	https://github.com/lz4/lz4
oglplus	0.72.0	https://github.com/matus-chochlik/oglplus
OpenCV	4.5.4 (CUDA)	https://github.com/opencv/opencv
qhull	2011.1	http://www.qhull.org
suitesparse-metis	1.5.0	https://github.com/jlblancoc/suitesparse-metis-for-windows.git
VXL	2.0.2	https://github.com/vxl/vxl.git
xerces	3.1.1	https://github.com/apache/xerces-c/
zlib	1.2.5	https://www.zlib.net/
CUDA	11.7	https://developer.nvidia.com/cuda-11-7-1-download-archive?
Visual Studio	2019	https://visualstudio.microsoft.com/downloads/

The default configuration expects these dependencies to be placed in a Dependencies\ folder inside the root of the repo. You can change this location by modifying the definition of Peabody_Dependency_Dir in PeabodyConfigurationMacros.props, or modifying only a specific component's location in its Peabody.x64.[componentName].props file.

3rd Party Components (Linux)

Component	Version	Link
libk4a	1.4.1	https://learn.microsoft.com/en-us/azure/kinect-dk/sensor-sdk-download
ZeroMQ	4.3.4	https://github.com/zeromq/libzmq/
CppZMQ		https://github.com/zeromq/cppzmq
OpenCV	4.5.4 (CUDA)	https://github.com/opencv/opencv
CUDA	11.x	Bundled in Jetpack OS Image

System Components

This repo includes source code to compile the following systems:

Fusion

Fusion is the main component that creates a watertight, volumetric mesh in real time from any number of RGB+D streams. The textured volume and raw (or jpeg compressed) RGB images are transmitted to any number of render clients.
It is based off of the Fusion4D algorithm https://www.microsoft.com/en-us/research/publication/fusion4d-real-time-performance-capture-challenging-scenes-2/ Fusion is a C++ and CUDA project, and is known to work with CUDA 11.4+. It uses Boost ASIO for transmmitting data.

Render

Render takes a volumetric mesh from the fusion system and textures it with the RGB images provided in the same data stream from fusion, and transmits the textured model over the network to any number of viewer clients.
It is a C# Unity project, and is known to work with Unity 2019.4.14f1 It is also dependent on the C++ Rendering Client DLL for transmitting data. It uses Boost ASIO for transmitting data.

Viewer

Viewer subscribes to a given Render stream and presents the textured 3D model to the user. It provides views of the 3D model and the 2D cameras. It also includes drawing tools and capture tools for manipulating the model. It is a C# Unity project, and is known to work with Unity 2019.4.14f1. It uses Boost ASI for transmitting data.

Control Panel

The control panel software orchestrates all of the components of the 3D Telemedicine system. It provides one interface to start a calibration session, start a broadcast (live) session, modify configuration values, transmit configuration to the various components, monitor status and version of all components in the system, and check for updated components from an Internet-accessible source. It is a C# .Net project that utilizes the zeroMQ networking library for all communication. You will find most other components in the system have a ControlPanelConnector class to handle the communication with the Control Panel.

Calibration Software

The calibration software will take a number of synchronized input mkv videos from cameras in the system (Azure Kinect) and output a calibration file that provides world coordinates of all cameras in the system. It is dependent on the PAI calibration system available at this NuGet repository: It is a C++ project. It uses ZeroMQ for transmitting progress updates to the Control Panel.

K4ARecorder

This is a port of the K4ARecorder provided in the Azure Kinect SDK that integrates it into our system and communicates with the Control Panel. It performs the same functions as K4ARecorder and is dependent on that library. Recorded videos are transmitted from the camera units to the fusion machine via scp.

K4A To Fusion

K4AToFusion is the ingest code that pulls images from an Azure Kinect and transmits them to a Fusion server. It is written to run on an Nvidia Jetson Nano (Linux) It is written in C++ and uses the ZeroMQ library for communication with the Control Panel, and Boost IO libraries for streaming image data.

Azure Kinect Launcher Daemon

Azure Kinect Launcher Daemon is the Linux system daemon that monitors the status of the Kinect-attached Jetson Nanos and communicates with the Control Panel. The Control Panel uses this daemon to launch the K4A To Fusion software, K4ARecorder software (for calibration), check version information, upload new configuration files and software versions, and receive updates about current system and application status. It is written in C++ and uses the ZeroMQ library for communication.

3DTM Launcher Service

3DTM Launcher Service is the Windows service that runs on Fusion and Render machines and communicates with the Control Panel. The Control Panel uses this service to launch Render, Fusion, and the Calibration Software. It also uses this service to check software versions, transmit configuration data and new software versions, and receive updates on current system and application status. It is written in C++ and uses the ZeroMQ library for communication.

Kinect Nano Communicator Service

In certain configurations where the camera system may be on a separate subnet from the control panel, the Kinect Nano Communicator Service can act as a router to route messages from the Azure Kinect Launcher Daemon and K4A To Fusion components of each camera unit up to the control panel on a separate subnet.

Setup [Windows components]

System requirements

The Windows components (Fusion, Render, Viewer, Control Panel, Calibration Software, 3DTM Launcher Service, and Kinect Nano Communicator Service) leverage code implemented using C#, CUDA 11.4+, and Unity. As such, a build agent containing an Nvidia GPU (RTX 2080 or later), Windows 10+ OS, and Unity 2019.4.14f1 is necessary to compile the components.

Component download

Check out the repository to a location of your choice, we'll refer to it as [3dtmroot].

Create the folder for the dependencies, [3dtmroot]\Dependencies.

Download the dependencies and install them into the Dependencies folder.

Build (Fusion)

Fusion is built using the DeformableFusion solution in the DeformableFusion folder of the repository. It should be built in Release, x64 configuration. Debug currently does not compile. The DeformableFusion solution contains a number of helper libraries that will be built first, and the Fusion executable is built as part of the LiveFusionDemo-MultiView project, which is selected as the startup project.

Setup [Linux components]

System requirements

The Linux components (Azure Kinect Launcher Daemon, K4A To Fusion, K4ARecorder) are built for the ARM64 architecture of the Nvidia Jetson Nano. Our build system uses the Ubuntu 18.04 image from the Nvidia Jetson website that includes CUDA. We have only built source code on Nvidia Jetson TX1 using the aforementioned Ubuntu image. The system should compile on newer Nvidia Jetson hardware, so long as the Ubuntu 18.04 image is used for backwards compatibility with existing PODs.

Component download

Check out the repository to a location of your choice, we'll refer to it as [3dtmroot]

Clone the Azure Kinect SDK: https://github.com/microsoft/Azure-Kinect-Sensor-SDK.git

Install the additional required build packages:

• Azure Kinect libraries - https://learn.microsoft.com/en-us/azure/kinect-dk/sensor-sdk-download#linux-installation-instructions
• Ncurses
• SoundIO (required for Azure Kinect)
• Boost - System
• Boost - IO Streams
• ZMQ
• ProcPS
• LZ4
• JSONCpp
• ZeroMQ 4.3.4
• cppzmq
• OpenCV 4.5.4

sudo apt update && sudo apt install -y libk4atools libk4a1.4  libk4a1.4-dev libncurses-dev libsoundio-dev libboost-system-dev libboost-iostreams-dev libzmq3-dev libprocps-dev liblz4-dev libjsoncpp-dev

The system requires zeromq 4.3.4 with drafts enabled, which is not available in the Jetson Ubuntu 18.04 image. So to do this:

ZMQ_VERSION=4.3.4
PREFIX=/usr/local
wget https://github.com/zeromq/libzmq/releases/download/v${ZMQ_VERSION}/zeromq-${ZMQ_VERSION}.tar.gz -O libzmq.tar.gz
tar -xzf libzmq.tar.gz
cd zeromq-${ZMQ_VERSION} 
./configure --prefix=${PREFIX} --enable-drafts
make -j && make install

You can install cppzmq from here: https://github.com/zeromq/cppzmq.

You'll also need OpenCV 4.5.5. Here's a great tutorial to get it built on the nano: https://qengineering.eu/install-opencv-4.5-on-jetson-nano.html.

Now set up the paths correctly in the cmakelists.txt.

Build (Linux)

You can build all of the linux projects at the same time by

cd [3dtmroot]
mkdir build
cd build
cmake ..
make
sudo make install

Reminder

You might be tempted to update your build environment to a newer version of Ubuntu or other libraries, but remember that most libraries will be dynamically linked, not bundled in the binary, so they won't run on a Jetson Nano, which only has Ubuntu 18.04 as the latest OS.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

Compliance with 3rd Party Components Licenses

Find below a list of messages and requirements to comply with licenses of 3rd Party Components.

libjpeg-turbo

This software is based in part on the work of the Independent JPEG Group.

The Modified (3-clause) BSD License

========================================================================

Copyright (C)2009-2021 D. R. Commander. All Rights Reserved.
Copyright (C)2015 Viktor Szathmáry. All Rights Reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

• Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
• Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
• Neither the name of the libjpeg-turbo Project nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS", AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

========================================================================

(END LICENSE TEXT)

3dtelecommunications's Issues

Azure Kinect Launcher Daemon fails to kill the Azure Kinect Nano To Fusion

Seeing this occur in the van build and in Glasgow/KBTH builds where a system shutdown is requested, and most of the system halts, but AKLD doesn't fully kill the AKNF process, causing it to not properly start on the next session. Should be fairly easy to modify AKLD to verify there are no AKNF processes, and kill any that still exist, or send a stronger kill signal when a shutdown is requested.

Action required: migrate or opt-out of migration to GitHub inside Microsoft

Migrate non-Open Source or non-External Collaboration repositories to GitHub inside Microsoft

In order to protect and secure Microsoft, private or internal repositories in GitHub for Open Source which are not related to open source projects or require collaboration with 3rd parties (customer, partners, etc.) must be migrated to GitHub inside Microsoft a.k.a GitHub Enterprise Cloud with Enterprise Managed User (GHEC EMU).

Action

✍️ Please RSVP to opt-in or opt-out of the migration to GitHub inside Microsoft.

❗Only users with admin permission in the repository are allowed to respond. Failure to provide a response will result to your repository getting automatically archived.🔒

Instructions

Reply with a comment on this issue containing one of the following optin or optout command options below.

✅ Opt-in to migrate

@gimsvc optin --date <target_migration_date in mm-dd-yyyy format>

Example: @gimsvc optin --date 03-15-2023

OR

❌ Opt-out of migration

@gimsvc optout --reason <staging|collaboration|delete|other>

Example: @gimsvc optout --reason staging

Options:

• staging : This repository will ship as Open Source or go public
• collaboration : Used for external or 3rd party collaboration with customers, partners, suppliers, etc.
• delete : This repository will be deleted because it is no longer needed.
• other : Other reasons not specified

Need more help? 🖐️

• Email [email protected]. ✉️
• Post your questions in GitHub inside Microsoft Team in Microsoft Teams. 🗨️

USB Camera - Test

Step 4 - Test
Plug in any available USB camera, doesn't have to be high quality, and test to see if it works. Once we know it works we can discuss where to obtain a camera of the quality Steven wants. We have quite a few here in MSR we can bring down if necessary.

Action required: self-attest your goal for this repository

It's time to review and renew the intent of this repository

An owner or administrator of this repository has previously indicated that this repository can not be migrated to GitHub inside Microsoft because it is going public, open source, or it is used to collaborate with external parties (customers, partners, suppliers, etc.).

Action

👀 ✍️ In order to keep Microsoft secure, we require repository owners and administrators to review this repository and regularly renew the intent to either opt-in or opt-out of migration to GitHub inside Microsoft which is specifically intended for private or internal projects.

❗Only users with admin permission in the repository are allowed to respond. Failure to provide a response will result to your repository getting automatically archived. 🔒

Instructions

❌ Opt-out of migration

If this repository can not be migrated to GitHub inside Microsoft, you can opt-out of migration by replying with a comment on this issue containing one of the following optout command options below.

@gimsvc optout --reason <staging|collaboration|delete|other>

Example: @gimsvc optout --reason staging

Options:

• staging : My project will ship as Open Source
• collaboration : Used for external or 3rd party collaboration with customers, partners, suppliers, etc.
• delete : This repository will be deleted because it is no longer needed.
• other : Other reasons not specified

✅ Opt-in to migrate

If the circumstances of this repository has changed and you decide that you need to migrate, then you can specify the optin command below. For example, the repository is no longer going public, open source or require external collaboration.

@gimsvc optin --date <target_migration_date in mm-dd-yyyy format>

Example: @gimsvc optin --date 03-15-2023

Click here for more information about optin and optout command options and examples

Opt-in

@gimsvc optin --date <target_migration_date>

When opting-in to migrate your repository, the --date option is required followed by your specified migration date using the format: mm-dd-yyyy

@gimsvc optin --date 03-15-2023

Opt-out

@gimsvc optout --reason <staging|collaboration|delete|other>

When opting-out of migration, you need to specify the --reason.

• staging
  • My project will ship as Open Source
• collaboration
  • Used for external or 3rd party collaboration with customers, partners, suppliers, etc.
• delete
  • This repository will be deleted because it is no longer needed.
• other
  • Other reasons not specified

Examples:

@gimsvc optout --reason staging

@gimsvc optout --reason collaboration

@gimsvc optout --reason delete

@gimsvc optout --reason other

Need more help? 🖐️

• Email [email protected]. ✉️
• Post your questions in GitHub inside Microsoft Team in Microsoft Teams.

USB Camera - Investigate

Step 1 - Investigate
Look for examples online of capturing still images from a camera in Unity. There should be plenty of these. Find something that seems reliable and fast.

USB Camera - Implement

Step 3 - Implement
We can compile the Unity code here at MSR if ECL doesn't have a Unity license.

USB Camera - Design

Step 2 - Design
Determine where in the viewer code this code should be added. It should only have to be added in the C# code, and not require any changes to the actual Unity project.

Evaluate legacy commented code after uploading initial source code version to Github

Evaluate if using blockDim.x / 2 instead of 80 breaks code in file /DeformableFusion/FastNonrigidMatching/EDMatchingHelperCudaImpl.cu. This will be done once the code is uploaded to Github.

Evaluate if uncommenting //#pragma omp parallel for breaks code in file /DeformableFusion/Basics/TSDF.cpp.

Expose TIMESTAMP_SCALE from BufferAggregator.cpp as a 3DTelemedicine.cfg config file parameter

The TIMESTAMP_SCALE is a Fusion parameter similar to the MaximumTimeSeparationOfSubFrames calibration parameter. Both define the number of microseconds that should be accepted between two frames of different Kinects to consider that the capture is synchronized for a given timepoint.

Ideally, both parameters should be set to 2000.0 given that, with ideal hardware configuration, up to 10 Kinects can be synchronized within that time scale. However, difference in hardware and environment configuration (e.g., sync cable length, temperature) might affect the sync signals requiring that value to be raised. IMPORTANT: the need for raising this value often indicates that something hardware-related is not ideal with the 3DTM system.

Currently, TIMESTAMP_SCALE is hardcoded to 2000.0 in the Networking/BufferAggregator.cpp file and we would like to expose it as a parameter of the [Fusion] section in the 3DTelemedicine.cfg file. This can be achieved by making TIMESTAMP_SCALE a member variable of the BufferAggregator class, and update the value in the DataCaptureNetworkMultiView subclass with the value read from the FusionConfig::current() conf object.

Return temperature values from the Kinects as part of their status updates

Kinects have a built in IMU with temperature data. This data would be valuable to determine if calibration fails or quality degrades due to temperatures outside the allowable ranges.

The kinect IMU data should be easily captured in AzureKinectNanoToFusion, and could then be transmitted as part of the CPC_STATUS packed on the control panel communication path. This would allow us to log temperature data or display warnings on the control panel.

Add ability to capture images from USB-connected cameras

When a doctor clicks the "capture" button in the viewer, it saves the current 3D model, plus a PNG of all cameras in the system. Steven has asked if we could add additional high-resolution 2D cameras to the system for captures. This should be fairly straightforward. We should be able to modify the viewer code so that when the capture button is clicked, it also checks for any connected USB cameras, and captures an image from those as well.

This issue is the main issue to track discussion. Please track progress on the below steps in their individual sub-issues.

Step 1 - Investigate
Look for examples online of capturing still images from a camera in Unity. There should be plenty of these. Find something that seems reliable and fast.

Step 2 - Design
Determine where in the viewer code this code should be added. It should only have to be added in the C# code, and not require any changes to the actual Unity project.

Step 3 - Implement
We can compile the Unity code here at MSR if ECL doesn't have a Unity license.

Step 4 - Test
Plug in any available USB camera, doesn't have to be high quality, and test to see if it works. Once we know it works we can discuss where to obtain a camera of the quality Steven wants. We have quite a few here in MSR we can bring down if necessary.

Fix control panel logging

Control panel makes a new log every time session start is pressed. Great, but it's going to fill up the hard drive. Create some kind of automated logrotation, (maybe using task scheduler?) so it removes old logs on reboot, or add code to remove old logs, or make sure when verbosity = 0 it isn't really writing anything to the log.