HOCTOPUS: Empower Remote Class Teaching with Extended Reality
๐ฅฝ + ๐ฑ = ๐
Luca Asunis*,
Andrea Cirina*,
Lorenzo Stacchio*,
Gustavo Marfia
* Authors provided equal contributions.
| ISMAR 23 - 1st Joint Workshop on Cross Reality | project page | paper | video
HOCTOPUS provides an easy tool for both teachers and students to visualize, manipulate and share 3D objects in a live-stream fashion exploiting peer-to-peer network connections. In particular, HOCTOPUS consists of two main components thought for the two main actors of our use case: teachers and students. For teachers, we provided an MR streaming experience that let the user picks the 3D model for the class she/he wants to teach and straightforward interactions to manipulate during the explanation. For students instead, we developed an AR experience to visualize the streamed 3D object in a non-situated fashion, which real-time twins all the manipulations made by the teacher in the MR experience. Students have also the possibility to interact with the object and ask questions.
In particular, we developed the MR side of the system for the Hololens 2 and the AR one for Android mobile devices.
In practice, HOCTOPUS aims at supporting remote MR class teaching by streaming the teacher's manipulations of 3D models to all the students that joined the class, to enhance and improve the learning process of complex structures and processes.
The Mixed Reality teacher application was implemented with the Hololens 2 and provides the following features:
- Loading 3D models for the objects the Teacher wants to use in its classes;
- Picking one object;
- Starting the remote class;
- Stream the manipulations made with the object;
- Manage the connected students.
The MR Teacher app exploits the Unity Relay Service by hosting the remote class session that the connecting clients will join (this aspect is used in features (3),(4), and (5)).
The Mobile Augmented Reality student one was designed for smartphones (in particular, Android-based ones). The student application provides the following features:
- Visualizing the current state of 3D models streamed by the teacher;
- Asking questions related to the object;
- Manipulating the object while doing a question to ease the comprehension between teacher and student.
The student AR app exploits the UniRS to connect to the hosted class as connected clients. We here state that all the AR-related features were developed with AR Foundation and AR core and that the system can be used with any Android smartphone with a version greater or equal to 8.0 (Oreo).
-
Create a profile on Unity Gaming Services.
-
After registering, create a new project and assign a preferred name to it.
-
In the project interface, navigate to the Multiplayer section at the bottom-left corner.
-
Select the Relay option to initiate the server relay configuration.
-
In the server relay screen, click on Get started to proceed.
-
Follow the provided instructions carefully for setup in the Unity project:
- Link Unity project:
-
Open your project in the Unity Editor and navigate to Edit > Project Settings;
-
From the Project Settings window, go to the Services tab;
-
Click on Use an existing Unity project ID and link to the following Dashboard project:
- Organization: **Name of the organization**;
- Project: **Name of the project**
-
Once you see the Project was linked successfully banner appear at the top of the window, you will be ready to move on.
-
- Link Unity project:
Feel free to skip the third step, which suggests Import a sample. Instead, just click on Finish. By following these steps, you'll be able to successfully configure the server relay within the Unity project โ .
- Go to
File -> Build Settings
. Here change the platform by selectingUniversal Windows Platform
and pressing onSwitch platform
. Once you have changed the platform, make sure you have these settings:
- Next go to
Player Settings
, make sure you are on the correct platform and scroll down until you findPublishing Settings
. Here you need to create a new certificate and select it for this project. At the end of the operation you will have a screen similar to this one:
- Now go to
Mixed Reality -> Toolkit -> Utilities -> Build Window
.
- In the window that will open, make sure you have the following settings:
Build Unity Project
.
- Once the build is finished, go to the folder that was created automatically (
Builds --> WSAPlayer
) and open the.sln
file with visual studio. In the window on the right calledSolution Explorer
, open the filePackage.appxmanifest
directly from visual studio. Click onCreate Package
, and you will see a screen similar to this one:
Here click on Choose certificate...
and select the one you created earlier.
- Now go back to
Solution Explorer
and right-click onPackage.appxmanifest
, thenOpen with... -> XML Editor
. Here you need to make sure thatPhoneProductId
andPhonePublisherId
have the same code. Also go all the way down and make sure theCapabilities
are configured like this:
๐ดImportantโ๐ด Close Visual Studio, a warning will appear and you will need to save.
- Now go back to Unity and in the
Build Window
section click onAppx Build Options
. Put in the following settings:
Finally click on Build Appx
.
- Now you'll just go to
<<NameOfTheProject>>\Builds\WSAPlayer\AppPackages\<<NameOfTheProject>>_1.0.1.0_ARM64_Test
. Here you will find the package (.appx
) to install on the Hololens 2. We recommend that you use the device portal for installation.
As a reminder, at least version 8.0 Oreo is required for deployment on Android. The settings for deploying on Android are the same as for any augmented reality application.
- Change the platform to the Android platform;
- Go to
Player Settings
; - In the
Other settings
section make sure you have the following settings:
The main unity scene is in the Assets/Scenes/MainScene.unity
.
@inproceedings{asunis2023hoctopus,
title={HOCTOPUS: An Open-Source Cross-Reality tool to Augment Live-Streaming Remote Classes},
author={Asunis, Luca and Cirina, Andrea and Stacchio, Lorenzo and Marfia, Gustavo},
booktitle={2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)},
pages={29--34},
year={2023},
organization={IEEE}
}