vircadia / vircadia-web-sdk

Vircadia Web SDK - an SDK for the Vircadia platform that runs in your web browser.

Vircadia Web SDK

The Vircadia Web SDK (codename Ananke) is a JavaScript/TypeScript SDK for developing web-based applications for virtual worlds powered by the open source metaverse platform Vircadia. Vircadia domain servers provide the worlds (a.k.a. "domains") to visit, and the Vircadia metaverse server provides global services that connect the users and domains. See the user docs to Understand the Architecture.

This SDK provides interfaces to:

• Connect to domains.
• Use metaverse services.

The SDK is written in TypeScript.

• Project Configuration
• Coding Standard

The SDK is published at NPM: https://www.npmjs.com/package/@vircadia/web-sdk

To learn more about using Vircadia's metaverse ecosystem, see the Developer Documentation.

Prerequisites

Node.js

https://nodejs.org/en/download/

Node.js version ≥ 10.13 ; LTS version ≥ 14.16 recommended
npm version ≥ 6.4.1 ; LTS version ≥ 6.14 recommended

Jest

Jest is used for unit testing. It is included as an NPM dev dependency, however, you may also install it globally if you want to.

Project Setup

Get the source

git clone https://github.com/vircadia/vircadia-web-sdk.git

If you're working on Vircadia protocol code (that in the \src\domain directory) it is recommended that you clone the SDK as a subdirectory of the main Vircadia repo's source so that both sets of code are accessed when you open the main Vircadia repo's source in your IDE. For example, if your clone of the main Vircadia repo's source is in C:\Workspaces\vircadia then clone the SDK repo's source into C:\Workspaces\vircadia-web-sdk. Thus with the main Vircadia's repo loaded in your IDE, when you search for code then both C++ and TypeScript results are returned, helping you compare the two codebases and keep them in sync.

Note: Unix line endings (LF, 0A character) are used. If using Windows it is recommended that you configure Git's global setting core.autocrlf to be true. This makes the files use Windows line endings (CRLF, 0D0A characters) on your computer but stores the files in the Git repo with Linux line endings.

Install NPM packages

npm install

Loads all the supporting NPM packages as defined in package.json into the node_modules directory.

Development and Testing

Compile and minify for development

npm run build

Does a development compile and packages the SDK into the dist directory.

Hot-recompile for development

npm run watch

Does a development compile and enables webpack to watch the sources and recomile when source files change. This is often useful when testing SDK development using the example tool.

Compile and minify for production

npm run build-prod

Does a clean production compile and packages a releasable version of the SDK into the dist directory.

Hot-recompile for production

npm run watch-prod

Does a clean production compile and enables webpack to watch the sources and recompile when the source files change.

Clean the build directory

npm run clean

Lint files

All files:

npm run lint

A specific directory or file:

npm run lint-path <path>

Run tests

Unit tests can be run without any external dependencies but integration tests require a domain server to be running on localhost or other location specified in ./tests/test.config.json. The location and other values of the config JSON can be overridden with environment variables, using the same property names, but prefixed with "VIRCADIA_".

All tests:

npm run test

Hot retest of all tests:

npm run test-watch

Specific tests (e.g., Packet.unit.test.js, all unit tests, all integration tests):

npm run test <partial-path>

npm run test /packet.unit
npm run test .unit.
npm run test .integration.

Run tests and report open handles:

npm run test-debug [<partial-path>]

Generate docs

SDK API documentation:

npm run sdkdoc

Developer documentation (includes SDK API documentation):

npm run devdoc

Update contributors

Update contributors list in package.json:

npm run update-contributors

vircadia-web-sdk's Issues

Example app should clear its entities list upon domain disconnect.

At least user-initiated domain disconnect. Probably also unexpected disconnects - what does the native client do?

Update ESLint version and remove unnecessary disabling of no-magic-numbers rule

When setting up the project, a somewhat out of date version of the ESLint parser had to be used in order to get linting working in Visual Studio 2019. One side effect of this is ESLint erroneously complaining about magic numbers when defining default values for class fields, hence the unnecessary disabling of the no-magic-numbers rule in various places.

The ESLint parser used in package.json should be updated sometime, probably to babel-eslint, to work with the latest version of Visual Studio 2019.

AssignmentClient doesn't use SignalEmitter for onStateChange

AssignmentClient should use a SignalEmitter for onStateChange so there can be multiple subscribers.

Also check other classes for one use callbacks and change to SignalEmitters.

Needs to happen sooner than later because this effects the SDK's API.

Review and revise provision of Web SDK documentation

• Repository README and related, incl. as it is viewed on NPM.
• Clarify SDK user versus SDK developer documentation.
• Vircadia developer docs: publish SDK user doc.
• Switch to using TSDoc. See other projects for examples.

Packet details links missing from some packet descriptions in PacketType documentation

Can't disconnect and reconnect in Firefox

Using the example app with Firefox, if you connect then disconnect and then try to connect again you get errors:

You can connect, disconnect, and reconnect OK in Chromium and Edge.

I believe you used to be able to connect/disconnect/reconnect in Firefox previously. .Could be a code change or a Firefox update that has started this problem.

Note: Reloading the page doesn't work. Your have to close the browser tab and open a new one (or restart the browser, or restart the server).

AvatarIdentity.read() should probably interpret null QString values a null JavaScript values

In particular, when reading displayName and sessionDisplayName values.

Add remaining entity types.

This will enable all entity types to be read on domains.

Already implemented: Box, Sphere, Shape, Model,

Implement next:

• Image.
• Text. ... #146 #151
• Web.
• Light. ... #154
• Merge into master and release: 2022.1.7 ... #159
• Material. ... #162
• Zone. ... #165
• ParticleEffect. ... #166
• Release: 2022.1.8

Implement later:

• Grid,
• Line,
• PolyLine,
• Gizmo
• PolyVox,

Retrieve/configure ICE server used

WebRTCDataChannel.js hard codes the Vircadia ICE server: iceServers: [{ urls: "stun:ice.vircadia.com:7337" }].
The C++ equivalent is ICE_SERVER_DEFAULT_HOSTNAME in NetworkingConstants.h, though it can be overridden by a command line parameter when starting the domain server - https://docs.vircadia.com/developer/domain-server/command-line-parameters.html.

It may perhaps be necessary for Interface to get the ICE server address from the metaverse server for the chosen metaverse.

Similarly for other networking constants.

Handle packet type 21 - DomainServerAddedNode

The domain server sends when a new assignment client is started.

Reduce console log spam

The SDK logs warnings to the console for certain items, e.g., packets received but not handled or a code path not implemented.

Such warnings should be logged only once.

Get the Vircadia Web SDK packaged in a production useful way.

Resolve avatar mixer and entity mixer code's camera and view frustum overlap

And potentially expand there being a single camera to supporting multiple cameras at the same time.

PR: #100

build JS file should have SDK oriented name

Until packaging is complete, the SDK build creates the file "dist/vircadia.js". This file should, at least, be named "vircadia-web-sdk.js" to differentiate it in any library directory.

Handle DomainSettings packets received from domain server.

• The avatar min and max height values are needed for avatar scale limiting: see MyAvatar.restrictScaleFromDomainSettings(...) which sets AvatarData's _domainMinimumHeight and _domainMaximumHeight member values with values received from the domain server. ... Also check MyAvatar.clearScaleRestriction() as part of this.

Add TypeScript type file to distribution

When using components out of the SDK in a TypeScript app, one needs a .d.ts file somewhere to define the structure of the various classes, etc.

FST and default avatar model handling and support.

Address three skeleton model scenarios:

• Invalid avatar such that default FST avatar is used.
• Valid FST avatar.
• Valid glTF/FBX avatar.

For all three address:

• Own skeleton model data handling.
• Sending own skeleton model data to the avatar mixer.
• Receiving other users' skeleton model data.
• Handling other users' skeleton model data.

What should the SDK provide to support user clients for the above?

• Default avatar model and skeleton?
• FST file-related facilities?
• Avatar model file units determination?
• Anything else?

ConnectionState should be exported

Since DomainServer exports variables of type ConnectionState (onStateChangeCallback and stateToString()) the defined enum should be exported so callers can reference it.

Document that AvatarMixer.update is required for your avatar to be present to others

Move some SDK global properties and types into namespaces.

In order to reduce the amount of pollution in the namespace and tidy the use docs.
Breaking change and will improve the foundation for further work so do as soon as reasonably possible.

Entities namespace:

• EntityType
• EntityProperties, ModelEntityProperties, etc. and related.
• Shape and rename to shape. Resolve w.r.t. ShapeType.
• etc.

Skeleton namespace:

• BoneType
• SkeletonJoint
• etc.

AudioMixer:

• AudioPositionGetter.

Add blendshape support to avatars

• MyAvatar
• ScriptAvatar

Fix ESLint dot-notation rule

It should be as follows:

"@typescript-eslint/dot-notation": ["error", { "allowKeywords": true }],

Then the likes of .finally() won't generate ESLint errors.

Provide the domain's default avatar position.

Packet types:

• DomainServerPathQuery
• DomainServerPathResponse

PR: #126

Add domain-server request to fetch metaverse-server URL

Vircadia-web needs to get the metaverse-server URL that the domain server is associated with. There should be a call on the DomainServer class instance for fetching same.

There is an existing request REST API request to "/api/metaverse_info" that might be the solution.

Improve handling of Vircadia address types

Currently supported: ws[s]://127.0.0.1:40102[/path] where the address my be an IP or DNS address.

Should:

• Support hifi:// rather than ws[s]://, though also retain support for the latter.
• Automatically try both wss and ws connections, so that don't have to explicitly specify which.
• Make port optional (default 40102).

Further work to consider separately is providing support in the SDK for the other address types that the native client supports: domain paths, domain lDs, placenames,, @user, and serverless.

Implement Basic Entity Server Support

• Add EntityServer class that connects to the domain server.

  • Basic EntityServer.ts class.
  • Unit test, "Can create an EntityServer with a DomainServer".
  • Integration test, "States when connect to and disconnect from a domain".
  • Update Vircadia.ts and associated integration test.
  • Add EntityServer section to example app, with connection status.
  • PR: #93

• Request entity data from the entity server.

  • Add a PacketScribe.EntityQuery.write() method and unit test.
  • Update PacketType documentation.
  • Add an EntityServer.update() method based on the part of Application::update() which calls queryOctree(). Implement _queryExpiry but leave viewIsDifferentEnough for now.
  • Add an EntityServer.#queryOctree() method based on Application::queryOctree() with which sends EntityQuery packets to entity server.
    Include a #_physicsEnabled member hard-coded to true, and implement the if(isModifiedQuery) that uses it but log an error in the "if" branch; implement just the "else" branch.
    The #_conicalViews member value should be hard-coded for now (with a "TODO"). Use, say, a sphere at 0,0,0 with radius of 10.0m - e.g., in order to initially obtain just entities near the origin. (The #_conicalViews member value will need to be shared with the AvatarMixer via a "cameras" object or similar later on.)
    The lodManager values should be hard-coded for now (with a "TODO").
  • Implement OctreeQuery just enough to satisfy the needs of the above.
  • Confirm receipt at entity server through instrumentation.
  • PR: #98

• Receive packets from the entity server.

  • Add an empty OctreePacketProcessor class in /src/domain/octree.
  • Add a EntityServer.#_octreeProcessor member which is an OctreePacketProcessor.
  • Add a stub of OctreePacketProcessor.processPacket() and wire it up per OctreePacketProcessor::OctreePacketProcessor() so that it is called when anay entity server-related packet (message) is received. Included a comment that processPacket() is being called directly rather than being queued by a call to a handleOctreePacket() method.
  • Make OctreePacketProcessor.processPacket() log information on packets received but not handled (i.e., all packets at present), along the lines of AudioClient.#processStreamStatsPacket(). This will confirm that it is working and remind us of missing code.
  • PR: #105

• Implement OctreePacketProcessor.processPacket() for EntityData packets.

  • If an OctreeStats packet log that the packet was received but not handled and bail early because we're not interested in that packet yet or the voxel data it may contain.
  • Ignore other C++ code except for the switch statement. Include a case for EntityData but for other packets log that they were received but not handled.
  • Make the EntityData case call PacketScribe.EntityData.read().
  • Add a PacketScribe.EntityData.read() method stub.
  • Add a stub EntityServer.entityData signal called after the packet is read.
  • PR: #106

• Implement a PacketScribe.EntityData.read() method and make it handle just "Model" entity data for now (i.e., skip over other entity data).

  • A single "Model" entity as the sole entity in a domain. With a unit test.
  • Two or more "Model" entities in a domain. With a unit test.
  • A "Model" entity and other entity types in the domain. (Just the "Model" entity's data should be used.) With a unit test.
  • Editing a single "Model" entity as the sole entity in a domain. With a unit test.
  • List handled entities in a table in the example app (perhaps entity ID, entity type, entity position).
  • PR: #108

  The packet reading starts at OctreeProcessor::processDatagram() then gets rather complex.
  Note the "TO DO"s in ENTITY SERVER.md regarding signals to emit and the possible need for the entity server to maintain a copy of the entity data. The answers will depend on what the EntityData packet contains.

... Further work to plan ... See separate comment, below.

Console.log's should be centralized into a Log routine

There are "console.log"s scattered throughout the code. These should be replaced with calls to a centralized logging routine.

The centralized log routine would allow turning off diagnostic messages and potentially file or cloud message logging.

The log routine should use the common pattern of classifying messages as 'error', 'debug', 'info', etc. E.g., "Log.debug(msg)".

Export useful utility routines like Signal

There are some useful routines inside the SDK that would be nice to be available by any SDK user. In particular, vircadia-web copied and reimplemented Signal.
Please export useful internal routines. Especially Signal is useful.

SIgnalEmitter should return "slot code" so disconnect can match proper connect

When connecting a slot to the SignalEmitter, the disconnect just takes the slot again which is a handle to the original function. I have a class that makes connections for several different requestors so there would be multiple slots with the same signature (presuming the JS context doesn't make the signatures different).

It would be "cleaner" if SignalEmitter.connect returned a code (number?) for the connect that would be used to do the SignalEmitter.disconnect.

const subscriptionCode = signaler.connect(handleEvent);
...
signaler.disconnect(subscriptionCode);

Edge (and Chrome?) STATUS_BREAKPOINT crash after connecting to domain server

Using the Web SDK's example app, shortly after you connect to a domain server Edge crashes:

Edge:

• Version 97.0.1072.69 (Official build) (64-bit)
• Version 97.0.1072.62 (Official build) (64-bit) ... Browser crash originally noticed in this build.
  Tested on 3 Windows PCS, all running the same (latest) version of Edge. All crash.
  Tested connecting to domain server both on localhost and over local network. Both crash.

Chrome: Not tested yet.

Firefox: Doesn't crash.

Domain server:

• 2022.1.0
  • Edge 97.0.1072.69 doesn't crash
  • Edge 97.0.1072.62 crashes
• ...

Web SDK commits:

• dee5740 20 Jan 2022
  • Edge 97.0.1072.69 doesn't crash
  • Edge 97.0.1072.62 crashes
• 0d8b3ff 4 Dec 2021
  • Edge 97.0.1072.69 doesn't crash
  • Edge 97.0.1072.62 crashes
• ...

Possible causes may include:

• Recent domain server code changes have introduced problems.
• Recent Web SDK code changes have introduced problems.
• Browser WebRTC code has become incompatible with domain server WebRTC code.
• Browser WebRTC code has become incompatible with Web SDK audio code.

Regularize signal/slot implementation and documentation

Should both ends always be public so that they are both documented?
Should signals be be documented as properties or Signals?
Review how slots are documented.
Specify types of callback function parameters so that TypeScript performs checks.

Provide an FBX importer for Babylon.js?

Try out FBX importer - for avatars and Model entities.

Test and fix domain server restarting, network interruptions, etc.

• Domain server starting, restarting, stopping. ... #171
• Assignment client restarting. ... #172
• Network interruption (e.g., domain network disconnect and reconnect). ... #173
• Network failure (e.g., domain network disconnect). ... #173
• Connecting to valid URLs and (attempting) invalid URLs. ... No change.
• Address issue: #103 ... #172
• Clear example app's entities list upon entity server disconnect. ... #179

Read second entity type: Shape

Likely candidates (one of the following):

• Shape (skip over and ignore any pulse data - it's deprecated).
• Grid (skip over and ignore any pulse data - it's deprecated).
• Light (including adding color type).

A simple entity type so that can focus on reviewing and revising entity-reading code structure as/if necessary, in order to facilitate adding the remainder of the entity types.

As part of this:

• We'll probably want to separate the documentation of the common entity properties from the type-specific properties. We also want to provide the properties in the SDK doc, not just the dev doc.
• We'll probably want to separate out entity type specific code into their own files, e.g., ModelEntity.ts and ShapeEntity.ts, or whatever.

Allow for specifying URL to test against when entering test suite command.

Retrieve server info from config for unit tests

Address baked content

Review the current baking and output, plus industry baking practices.

Make the SDK package build with support for relevant browsers

Currently, the TypeScript is transpiled to ES2020 JavaScript in order to make use of recent feature such as BigInt literals.
Add in a Babel step to transpile that down to the target JavaScript version and browsers - ES2015 or should it be later, and what browser profile?

SDK package version number should change with feature changes

At the moment, the sdk package version is a constant "0,0,2" in the file "src/Vircadia.ts" and the NPM package version in "package.json" is "0.0.1".

The accessable sdk version should somehow be set from the NPM package version

Additionally, the sdk version should change as features change. Particularly when there are protocol changes or API additions.

Address recursion in entity reading per C++ Octree::readElementData()

Sort out what the C++ is doing and implement similar in the Web SDK.

Adopt a vector / matrix library?

Adopt a 3rd party library or continue to roll our own that is directly consistent with the Vircadia scripting API?

[ ] What does Babylon.js use?
[ ] What other well-known libraries are there?
[ ] What does the Web app use?

Send avatar bounding box to avatar mixer in AvatarData packet.

As part of this, use the bounding box as the mesh extents to implement Avatar.getUnscaledEyeHeight() which is used in the process of limiting avatar size per the domain's settings. ... Avatar.buildUnscaledEyeHeightCache() should be able to use the mesh extents, and should be called when the data are provided (instead of a non-existent Rig being ready).

Update TypeScript version and related

Should update to latest versions of all NPM packages.
This includes updating to the most recent ESLint that supports the TypeScript version, and reviewing and updating the new and changed ESLint rules.
Should update from Node 16 to Node 18 as part of this.
And remove use of blob-polyfill package which isn't needed for Node 18.

Fix handles left open after running tests

Currently, if you run all or certain tests, Jest reports the following issue after successfully completing the tests:

Jest did not exit one second after the test run has completed.
This usually means that there are asynchronous operations that weren't stopped in your tests.
Consider running Jest with `--detectOpenHandles` to troubleshoot this issue.

Address this as soon as the SDK code is complete enough with the first public API methods exposed.

Implement Basic Avatar Mixer Support

Own avatar:

• Setting own avatar URL.
• PR: #95

Others' avatar URLs:

• Camera object for use by avatar mixer and entity server code.
• Request avatar data from avatar mixer - send AvatarQuery packet to avatar mixer.
• PR: #97
• Report other users' avatar URLs.
• PR: #107

Own avatar data:

• Skeleton data. #113
• Avatar scale. #116
• Joint data. #117

Others' avatar data:

• Skeleton data. #109
• Avatar scale. #110
• Joint data. #111, #117

Default avatars
See: #133

In-line the audio worklets when built/packaged

Currently, the audio worklets are transpiled and packaged as individual files - vircadia-audio-input.js and vircadia-audio-output.js - separate to the SDK's primary Vircadia.js file. It would be preferable if the audio worklets were in-lined in Vircadia.js as Blobs to make the SDK easier to use (e.g., not having to explicitly copy the audio worklet files to the application's distribution directory and not having to potential set the relative path using AudioMixer.audioWorkletRelativePath.

Note:

• WebPack currently doesn't support audio worklets which is why the worker-url NPMJS package is used.
• If in-lining removes the usage of import.meta.url then AudioWorklets and associated Jest mock should be able to be done away with.

Obtain and handle min/max avatar height from domain server.

Packet types:

• DomainSettingsRequest
• DomainSettings

Replace the Camera API with a Cameras API.

So that more than one camera can be simultaneously used with the avatar mixer and entity server.

Entity Server: Do bare minimum reading to skip over non-Model or Shape entity types

This will enable Model and Shape entity data to be read on domains that included other types of entities.

Once this has been done on the entity-data branch, we'll merge the branch into master and do a release.

Firefox mic audio input doesn't work

Firefox mic input doesn't work because it doesn't automatically handle different sampling rates in the Web Audio context that is grabbing the mic audio input and feeding it to the 24kHz Vircadia audio pipeline. Edge/Chromium, on the other hand, automatically do the conversion.

Firefox console error message: "Uncaught (in promise) DOMException: AudioContext.createMediaStreamSource: Connecting AudioNodes from AudioContexts with different sample-rate is currently not supported."

We need to resample audio ourselves. For example, https://stackoverflow.com/questions/52787510/reducing-sample-rate-of-a-web-audio-spectrum-analyzer-using-mic-input

Probably best to implement as a new AudioWorkletProcessor alongside the existing AudioInputProcessor.ts, and insert it into the audio chain in AudioInput.#setUpAudioContext() - and perhaps insert only if Firefox or that error message is encountered.