microsoft / mixed-reality-extension-sdk Goto Github PK
View Code? Open in Web Editor NEWThe Mixed Reality Extension SDK enables developers to build 3D world extensions for AltspaceVR, using Node.JS.
License: MIT License
The Mixed Reality Extension SDK enables developers to build 3D world extensions for AltspaceVR, using Node.JS.
License: MIT License
Currently MREs can be deployed to Macs just fine, and the logic executes, but static files are not working
Repro steps
Ability to instantly teleport (don't collide with anything between old and new positions) and interpolate (moving through space over time, and colliding with any collision geometry on the way)
When creating box collision as a parameter for CreateFromGLTF, for some scenes (Suzanne),
it creates the rigid body, but it doesn't seem to be appear to be added properly to the unity physics scene. It isn't visible in the unity physics debugger view (and therefore doesn't collide) until the owning GO (or parent GO) has moved around, or the collision component has been modified.
It seems like a potential Unity physics bug.
There is no reliable repro steps for this yet.
Currently all APIs say "do it now", and due to unpredictable latency between server and host app, it's not possible to do reliably timed API calls. We should implement a way to call things relative to some other time (ideally a timer that restarts for example every minute, but alternatively time-since-start). A couple of changes needed are:
We'd need to be very careful to make sure all clients execute all packets in the exact same order between instant and delayed commands.
App should be able to read & write sceneRoot position, rotation, and scale.
We should add something that can easily show the scene graph parent/child relationships. For example something like:
Spawn 5 primitives in a row, parent them to each other, start spinning each of them slowly, and watch the children spin around each other, after a few seconds unparent them all, and watch them spin at the point where they were detached
Currently text elements render on top of the whole scene. We need to implement a custom material to make the text element z-buffer in properly with everything else
If your baseUrl is bad, or for some other reason clients cannot download the model, there is no indication anywhere (even in the Altspace log) that the load failed. The create promise just never resolves.
In AltspaceVR, it seems keyframe animations sometimes don't stop exactly at the last frame if they are in 'once' mode.
Repro steps:
Rigid body simulations work fine with a single user. However, once a second user enters, the objects teleport away.
Repro steps:
Rigid body patch data is all in global space, but should be relative to the MRE root.
The code shouldn't assume that MRE root is the same global offset for scene 1 as for scene 2. It is possible to share a session ID across multiple different AltspaceVR scenes, for example.
Rather than starting animations at the beginning, have the option to pass in an animation start time
Sample never starts animating, because GLTFs fail to load.
Repro steps:
Observed: Celestial body names show up, but geometries never load, and don't start animating
Expected: Celestial body geometry loads and start animating
Input Behaviors don't always get piped through to MRE. it seems related to having had multiple instances of the same MRE running within a single AltspaceVR session (but not necessarily in the same scene)
Repro steps:
Note that if going back to the homespace, the first MRE instance will launch and will be interacting properly. That leads me to guess this is an AltspaceVR integration or Client Lib issue, not a Server side issue.
Sample project is pretty bare-bones right now, we may want to make more samples that developers can start from
Repro steps
Though gltf-gen exists, there's currently no way to use it in an MRE. I want to add a method to WebHost where you can get a binary blob registered with a URL for clients to load it. This would let you generate a buffer from a GLTFFactory, then load it via Actor.CreateFromGLTF
.
If an app is playing and you change the URL, it remains connected to the old app until you toggle the Is Playing off and back on.
Physics forces are in worldspace (absolute orientation and non-scaled) - but should be MRE relative (or alternatively local), both for orientation and scale.
Repro Steps
When an MRE is instantiated, and the client has errors, the errors should be sent back to the MRE, so they can be piped through to where developers expect it. By default I would expect it to show up in the Visual Studio Debug Console, and in Node.js' log files.
Examples where I would expect errors to show up
If an MRE client tries to play an animation that doesn't exist on an actor
If MRE fail to load a GLTF
If MRE tries to access an actor that doesn't exist
A user's basic performance characteristics can allow targeted assets that match the user's device's capabilities. I'm imagining a tiered system, with lower values for weaker hardware and higher values for more powerful devices. Initially, 1 for mobile, 2 for desktops with integrated graphics, and 3 for desktops with discrete GPUs.
It's confusing that spawned apps require intervention to start running.
The system supports a null sessionId, but the UI doesn't allow it.
I realize this might seem like its not a good idea, but without it people will find other much worse ways to achieve the same thing like putting avatars inside inverted boxes or moving them with planes.
Attach the avatar to a rigid body/object to move with it.
Allow the avatar to move about within a limited space - on a large vessel.
Allow multiple users to sync to the same object - maybe attachment slots.
This allows for users to be put inside things, cars, boats bikes, elevators etc.
Users should be able to disconnect at any time and it should only control the position , but possibly allow for the initial rotation to be set as an offset.
The app rose should be visible whenever the app is not running, maybe with a state transition when it's attempting and failing to connect.
When trying to have an actor listening to events (like onHover and onClick), it only works if the item is created in the home space, but not if the item is created in a new world space.
Reproducible with hello-world example: Works okay (growth/shrink on hover, sideflip on click) in Homespace, but no reaction on user interaction when created in world space (just spins)
(UPDATE): Error is more generalized: When an item is used in more than one worldspace, the first instance I encountered worked correctly, not the second one. "Re-enter space" and "Reset Space" doesn't affect that bug; one has to completely exit AltspaceVR and restart the client.
Many Shane's Editor users use planes as floors and spheres/boxes as sky boxes. If the above basic primitives at least supported the following;
Repeat X,Y
Offset X,Y
Backface/Doubleface
That would make back porting existing content much easier.
Be able to create collision geometry and assign trigger events
The functional tests seem to have a dependency when deploying to openode.io. For some reason this does not show up as a dependency in local builds.
The package exists locally as it is used as a dev dependency by tslint/typescript, so it may be because of that, combined with Lerna?
Either way, we should figure out why it's needed and why our local build don't complain about it.
Create a functional test that exercises the CreateFromLibrary API - including a number of the built-in AltspaceVR assets.
If I place a number of space components in a world, I'd want to be able to access and modify them from within my MRE.
Note that it needs to be able to restore space components when user reenters the space or stops/restarts the MRE.
One possible implementation is:
In the space editor assets are associated with the MRE (or tagged as Used By MRE).
When the MRE is in isPlaying state, the original space components are hidden from the scene, and when MRE stops playing/is removed, the original assets are unhidden.
the MRE itself could need to call CreateFromLibrary with a custom parameter, and it would clone all of the associated hidden space components.
Be able to dynamically create and apply materials to geometry.
set/update GLTF textures or colors (albedo/metalroughness/normal/occlusion/emissive)?
The code is already documented for typedoc, but we need to automate its generation, and we need to decide where to publish it. I see two possibilities:
Develop a declarative UI framework.
Investigate using AdaptiveCards: https://github.com/microsoft/AdaptiveCards
Flag needs control behavior for all child actors recursively
MRE connection fails in Android versions (GearVR etc.)
Repro steps:
ws://mre-hello-world.azurewebsites.net
.Make it possible to place an object the user can click to teleport them to another space or to another point/rotation in the existing space. This mirrors the existing functionality in the old sdk using portals.
The primitives-test functional test only shows off cubes. We should extend this to other primitive types.
Need the ability to enable behaviors for one or all users, or possibly user group/teams.
Need the ability to enable actors for one user, all users, all users except one, or possibly user group/teams.
must be dynamic (as people join/leave teams)
Must support recording+Playback.
some use cases:
first person shooter - player sees their own first person rig vs everyone else sees a real 3d model,
when players die, they can see everyone?
card games - hide cards from anyone but player - but the moment player plays a card, it becomes visible to everyone.
Currently the functional test input-test shows root motion, but doesn't actually exercise the full click behavior.
I suggest we
Right now we hardcode each functional test into the functional test level. This isn't scalable. We should instead read each test configuration from a json file and dynamically instantiate each test station. json file might look like:
{
"host": "mre-hello-world.azurewebsites.net",
"tests": {
"rigid-body-test": {
"name": "Rigid Body Test"
},
"text-test": {
...
}
}
}
Key features
URLs are too long to make good labels, and may not contain the right info to be relevant to the spawn list.
The gltf-gen library is cool, but we don't have any functional tests exercising this
In-space teleporting and movement:
World teleporting:
Need the ability to attach MRE actors to an individual user's avatar - either to the root node or relative to different body parts.
It must work regardless of any avatar model that exists now or in the future, so system can't make assumptions about different models.
This should probably be done by having pre-defined attach points on the avatars for hat, backpack etc., it would allow easy customization of avatars, and we update all avatar rigs with these attach points
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.