microsoft / mixed-reality-extension-sdk Goto Github PK

Currently MREs can be deployed to Macs just fine, and the logic executes, but static files are not working

Repro steps

1. Deploy Hello World MRE to a mac
2. instantiate it in AltspaceVR
   Observed: Hello World text shows up and spins, but cube does not
   Expected: Hello World text and cube both show up and spin.

Ability to instantly teleport (don't collide with anything between old and new positions) and interpolate (moving through space over time, and colliding with any collision geometry on the way)

When creating box collision as a parameter for CreateFromGLTF, for some scenes (Suzanne),
it creates the rigid body, but it doesn't seem to be appear to be added properly to the unity physics scene. It isn't visible in the unity physics debugger view (and therefore doesn't collide) until the owning GO (or parent GO) has moved around, or the collision component has been modified.
It seems like a potential Unity physics bug.
There is no reliable repro steps for this yet.

Currently all APIs say "do it now", and due to unpredictable latency between server and host app, it's not possible to do reliably timed API calls. We should implement a way to call things relative to some other time (ideally a timer that restarts for example every minute, but alternatively time-since-start). A couple of changes needed are:

1. Server should add a timestamp into each packet it sends out
2. Client should continually calculate its' latency, and try to delay server packages to be as smoothed out as possible to match the server timestamp. (this means added latency at the cost of smoother packages )
3. Client should have a queue of commands to execute X frames into the future, rather than apply them instantly.

We'd need to be very careful to make sure all clients execute all packets in the exact same order between instant and delayed commands.

App should be able to read & write sceneRoot position, rotation, and scale.

We should add something that can easily show the scene graph parent/child relationships. For example something like:
Spawn 5 primitives in a row, parent them to each other, start spinning each of them slowly, and watch the children spin around each other, after a few seconds unparent them all, and watch them spin at the point where they were detached

Currently text elements render on top of the whole scene. We need to implement a custom material to make the text element z-buffer in properly with everything else

If your baseUrl is bad, or for some other reason clients cannot download the model, there is no indication anywhere (even in the Altspace log) that the load failed. The create promise just never resolves.

In AltspaceVR, it seems keyframe animations sometimes don't stop exactly at the last frame if they are in 'once' mode.
Repro steps:

1. In the Hello World sample, in generateSpinKeyframes, remove the last frame.
2. During the DoAFlip animation creation, change duration from 1.0 to 0.5, so the animation executes quicker
3. Build, Launch, and instantiate the MRE
4. Click on the cube
   Observed: Sometimes the cube stops spinning a little early (or a little late?), and sometimes it stops pointing straight up.
   Expected: After the rapid spin, the cube should always stop with the top pointing straight up

Rigid body simulations work fine with a single user. However, once a second user enters, the objects teleport away.

Repro steps:

1. Instantiate an MRE that uses rigid body physics, for example the functional test named rigid-body-test
2. Have a second user enter the same space.
3. Observe rigid bodies

Rigid body patch data is all in global space, but should be relative to the MRE root.

The code shouldn't assume that MRE root is the same global offset for scene 1 as for scene 2. It is possible to share a session ID across multiple different AltspaceVR scenes, for example.

Rather than starting animations at the beginning, have the option to pass in an animation start time

Sample never starts animating, because GLTFs fail to load.

Repro steps:

1. serve solar system sample

Observed: Celestial body names show up, but geometries never load, and don't start animating
Expected: Celestial body geometry loads and start animating

Input Behaviors don't always get piped through to MRE. it seems related to having had multiple instances of the same MRE running within a single AltspaceVR session (but not necessarily in the same scene)

Repro steps:

1. Instantiate the Hello World MRE in your AltspaceVR homespace. Observe that the cube reacts when you hover, click, and unhover.
2. Enter another world in AltspaceVR, and instantiate the Hello World MRE.
   Observed: The cube does not react to hover, click, or unhover.
   Expected: The cube should react

Note that if going back to the homespace, the first MRE instance will launch and will be interacting properly. That leads me to guess this is an AltspaceVR integration or Client Lib issue, not a Server side issue.

Sample project is pretty bare-bones right now, we may want to make more samples that developers can start from

Repro steps

1. Instantiate the lookat functional test.
2. move and rotate the MRE root to not be at identity anymore
   Observed: The Monkey Head dreamily stares off into space
   Expected: The Monkey Head stares you straight in the eyes

Though gltf-gen exists, there's currently no way to use it in an MRE. I want to add a method to WebHost where you can get a binary blob registered with a URL for clients to load it. This would let you generate a buffer from a GLTFFactory, then load it via Actor.CreateFromGLTF.

If an app is playing and you change the URL, it remains connected to the old app until you toggle the Is Playing off and back on.

Physics forces are in worldspace (absolute orientation and non-scaled) - but should be MRE relative (or alternatively local), both for orientation and scale.

Repro Steps

1. Load functional test scene for rigid-body-test. If placed at identity orientation/scale it works as expected, but if MRE root has been rotated or scaled, it is off.. This can be simulated using the "change global root" function of the testbed.
   Observed: Projectile flies in different direction, and misses target, if MRE root is rotated
   Expected: Regardless of rotation/scale, projectile should hit the target.

When an MRE is instantiated, and the client has errors, the errors should be sent back to the MRE, so they can be piped through to where developers expect it. By default I would expect it to show up in the Visual Studio Debug Console, and in Node.js' log files.

Examples where I would expect errors to show up
If an MRE client tries to play an animation that doesn't exist on an actor
If MRE fail to load a GLTF
If MRE tries to access an actor that doesn't exist

A user's basic performance characteristics can allow targeted assets that match the user's device's capabilities. I'm imagining a tiered system, with lower values for weaker hardware and higher values for more powerful devices. Initially, 1 for mobile, 2 for desktops with integrated graphics, and 3 for desktops with discrete GPUs.

It's confusing that spawned apps require intervention to start running.

The system supports a null sessionId, but the UI doesn't allow it.

I realize this might seem like its not a good idea, but without it people will find other much worse ways to achieve the same thing like putting avatars inside inverted boxes or moving them with planes.

Attach the avatar to a rigid body/object to move with it.
Allow the avatar to move about within a limited space - on a large vessel.
Allow multiple users to sync to the same object - maybe attachment slots.

This allows for users to be put inside things, cars, boats bikes, elevators etc.

Users should be able to disconnect at any time and it should only control the position , but possibly allow for the initial rotation to be set as an offset.

The app rose should be visible whenever the app is not running, maybe with a state transition when it's attempting and failing to connect.

When trying to have an actor listening to events (like onHover and onClick), it only works if the item is created in the home space, but not if the item is created in a new world space.

Reproducible with hello-world example: Works okay (growth/shrink on hover, sideflip on click) in Homespace, but no reaction on user interaction when created in world space (just spins)

(UPDATE): Error is more generalized: When an item is used in more than one worldspace, the first instance I encountered worked correctly, not the second one. "Re-enter space" and "Reset Space" doesn't affect that bug; one has to completely exit AltspaceVR and restart the client.

Many Shane's Editor users use planes as floors and spheres/boxes as sky boxes. If the above basic primitives at least supported the following;

Repeat X,Y
Offset X,Y
Backface/Doubleface

That would make back porting existing content much easier.

Be able to create collision geometry and assign trigger events

The functional tests seem to have a dependency when deploying to openode.io. For some reason this does not show up as a dependency in local builds.
The package exists locally as it is used as a dev dependency by tslint/typescript, so it may be because of that, combined with Lerna?

Either way, we should figure out why it's needed and why our local build don't complain about it.

Create a functional test that exercises the CreateFromLibrary API - including a number of the built-in AltspaceVR assets.

If I place a number of space components in a world, I'd want to be able to access and modify them from within my MRE.

Note that it needs to be able to restore space components when user reenters the space or stops/restarts the MRE.
One possible implementation is:
In the space editor assets are associated with the MRE (or tagged as Used By MRE).
When the MRE is in isPlaying state, the original space components are hidden from the scene, and when MRE stops playing/is removed, the original assets are unhidden.
the MRE itself could need to call CreateFromLibrary with a custom parameter, and it would clone all of the associated hidden space components.

Be able to dynamically create and apply materials to geometry.
set/update GLTF textures or colors (albedo/metalroughness/normal/occlusion/emissive)?

The code is already documented for typedoc, but we need to automate its generation, and we need to decide where to publish it. I see two possibilities:

1. Build docs to a markdown file and host it directly from the github repo.
2. Build docs as HTML to /docs/gltf-gen, and link from the developer portal.

Develop a declarative UI framework.

Investigate using AdaptiveCards: https://github.com/microsoft/AdaptiveCards

Flag needs control behavior for all child actors recursively

MRE connection fails in Android versions (GearVR etc.)

Repro steps:

1. place an MRE in any space in AltspaceVR, and enter a valid URL, for example ws://mre-hello-world.azurewebsites.net.
2. On Android platforms, the MRE does not show up. Entering the same space on Windows, the MRE shows up fine.

Make it possible to place an object the user can click to teleport them to another space or to another point/rotation in the existing space. This mirrors the existing functionality in the old sdk using portals.

The primitives-test functional test only shows off cubes. We should extend this to other primitive types.

Need the ability to enable behaviors for one or all users, or possibly user group/teams.
Need the ability to enable actors for one user, all users, all users except one, or possibly user group/teams.

must be dynamic (as people join/leave teams)
Must support recording+Playback.

some use cases:
first person shooter - player sees their own first person rig vs everyone else sees a real 3d model,
when players die, they can see everyone?
card games - hide cards from anyone but player - but the moment player plays a card, it becomes visible to everyone.

Currently the functional test input-test shows root motion, but doesn't actually exercise the full click behavior.
I suggest we

1. take the current input-test and move into root-motion-test
2. take the hover/click/unhover example from hello world and put into input-test

Right now we hardcode each functional test into the functional test level. This isn't scalable. We should instead read each test configuration from a json file and dynamically instantiate each test station. json file might look like:

{
    "host": "mre-hello-world.azurewebsites.net",
    "tests": {
        "rigid-body-test": {
            "name": "Rigid Body Test"
        },
        "text-test": {
            ...
        }
    }
}

Key features

• play one-shot sounds at 3d position
• preload sounds
• play audio stream at 3d position
• Set/modify pitch and volume per sound instance
• Automatically audio streams on multi-user join

URLs are too long to make good labels, and may not contain the right info to be relevant to the spawn list.

The gltf-gen library is cool, but we don't have any functional tests exercising this

In-space teleporting and movement:

• The possibility to moved one user to the desired location.
• Setting a base velocity vector. (Allows for activities like downhill skiing, where the user is set in motion)
• Reparenting the user to a given actor. (Allows for the user making use of a vehicle like a mine cart)

World teleporting:

• Portals and teleporters

Need the ability to attach MRE actors to an individual user's avatar - either to the root node or relative to different body parts.
It must work regardless of any avatar model that exists now or in the future, so system can't make assumptions about different models.

This should probably be done by having pre-defined attach points on the avatars for hat, backpack etc., it would allow easy customization of avatars, and we update all avatar rigs with these attach points