colyseus / schema

An incremental binary state serializer with delta encoding for games.

An incremental binary state serializer with delta encoding for games.
Although it was born to be used on Colyseus, this library can be used as standalone.

Defining Schema

As Colyseus is written in TypeScript, the schema is defined as type annotations inside the state class. Additional server logic may be added to that class, but client-side generated (not implemented) files will consider only the schema itself.

import { Schema, type, ArraySchema, MapSchema } from '@colyseus/schema';

export class Player extends Schema {
  @type("string")
  name: string;

  @type("number")
  x: number;

  @type("number")
  y: number;
}

export class State extends Schema {
  @type('string')
  fieldString: string;

  @type('number') // varint
  fieldNumber: number;

  @type(Player)
  player: Player;

  @type([ Player ])
  arrayOfPlayers: ArraySchema<Player>;

  @type({ map: Player })
  mapOfPlayers: MapSchema<Player>;
}

See example.

Supported types

Primitive Types

Type	Description	Limitation
string	utf8 strings	maximum byte size of 4294967295
number	auto-detects int or float type. (extra byte on output)	0 to 18446744073709551615
boolean	true or false	0 or 1
int8	signed 8-bit integer	-128 to 127
uint8	unsigned 8-bit integer	0 to 255
int16	signed 16-bit integer	-32768 to 32767
uint16	unsigned 16-bit integer	0 to 65535
int32	signed 32-bit integer	-2147483648 to 2147483647
uint32	unsigned 32-bit integer	0 to 4294967295
int64	signed 64-bit integer	-9223372036854775808 to 9223372036854775807
uint64	unsigned 64-bit integer	0 to 18446744073709551615
float32	single-precision floating-point number	-3.40282347e+38 to 3.40282347e+38
float64	double-precision floating-point number	-1.7976931348623157e+308 to 1.7976931348623157e+308

Declaration:

Primitive types (string, number, boolean, etc)

@type("string")
name: string;

@type("int32")
name: number;

Custom Schema type

@type(Player)
player: Player;

Array of custom Schema type

@type([ Player ])
arrayOfPlayers: ArraySchema<Player>;

Array of a primitive type

You can't mix types inside arrays.

@type([ "number" ])
arrayOfNumbers: ArraySchema<number>;

@type([ "string" ])
arrayOfStrings: ArraySchema<string>;

Map of custom Schema type

@type({ map: Player })
mapOfPlayers: MapSchema<Player>;

Map of a primitive type

You can't mix types inside maps.

@type({ map: "number" })
mapOfNumbers: MapSchema<number>;

@type({ map: "string" })
mapOfStrings: MapSchema<string>;

Backwards/forwards compability

Backwards/fowards compatibility is possible by declaring new fields at the end of existing structures, and earlier declarations to not be removed, but be marked @deprecated() when needed.

This is particularly useful for native-compiled targets, such as C#, C++, Haxe, etc - where the client-side can potentially not have the most up-to-date version of the schema definitions.

Reflection

The Schema definitions can encode itself through Reflection. You can have the definition implementation in the server-side, and just send the encoded reflection to the client-side, for example:

import { Schema, type, Reflection } from "@colyseus/schema";

class MyState extends Schema {
  @type("string")
  currentTurn: string;

  // more definitions relating to more Schema types.
}

// send `encodedStateSchema` across the network
const encodedStateSchema = Reflection.encode(new MyState());

// instantiate `MyState` in the client-side, without having its definition:
const myState = Reflection.decode(encodedStateSchema);

Data filters

On the example below, considering we're making a card game, we are filtering the cards to be available only for the owner of the cards, or if the card has been flagged as "revealed".

import { Schema, type, filter } from "@colyseus/schema";

export class State extends Schema {
  @filterChildren(function(client: any, key: string, value: Card, root: State) {
      return (value.ownerId === client.sessionId) || value.revealed;
  })
  @type({ map: Card })
  cards = new MapSchema<Card>();
}

Limitations and best practices

• Each Schema structure can hold up to 64 fields. If you need more fields, use nested structures.
• NaN or null numbers are encoded as 0
• null strings are encoded as ""
• Infinity numbers are encoded as Number.MAX_SAFE_INTEGER
• Multi-dimensional arrays are not supported.
• Items inside Arrays and Maps must be all instance of the same type.
• @colyseus/schema encodes only field values in the specified order.
  • Both encoder (server) and decoder (client) must have same schema definition.
  • The order of the fields must be the same.
• Avoid manipulating indexes of an array. This result in at least 2 extra bytes for each index change. Example: If you have an array of 20 items, and remove the first item (through shift()) this means 38 extra bytes to be serialized.

Generating client-side schema files (for strictly typed languages)

If you're using JavaScript or LUA, there's no need to bother about this. Interpreted programming languages are able to re-build the Schema locally through the use of Reflection.

You can generate the client-side schema files based on the TypeScript schema definitions automatically.

# C#/Unity
schema-codegen ./schemas/State.ts --output ./unity-project/ --csharp

# C/C++
schema-codegen ./schemas/State.ts --output ./cpp-project/ --cpp

# Haxe
schema-codegen ./schemas/State.ts --output ./haxe-project/ --haxe

Benchmarks:

Scenario	@colyseus/schema	msgpack + fossil-delta
Initial state size (100 entities)	2671	3283
Updating x/y of 1 entity after initial state	9	26
Updating x/y of 50 entities after initial state	342	684
Updating x/y of 100 entities after initial state	668	1529

Decoder implementations

Decoders for each target language are located at /decoders/. They have no third party dependencies.

Why

Initial thoghts/assumptions, for Colyseus:

• little to no bottleneck for detecting state changes.
• have a schema definition on both server and client
• better experience on staticaly-typed languages (C#, C++)
• mutations should be cheap.

Practical Colyseus issues this should solve:

• Avoid decoding large objects that haven't been patched
• Allow to send different patches for each client
• Better developer experience on statically-typed languages

Inspiration:

• Protocol Buffers
• flatbuffers
• schemapack
• avro

License

MIT

schema's Issues

Allow to specify `outFile` to output all generated schemas into a single file

MapSchema onRemove issue

Hi guys!
Here I was trying to remove the old player, when a new tab is opened:

    onAuth(client, options, req) {
        if (req.sessionID) {
            if (this.state.sessions[req.sessionID]) {
                this.state.removePlayer(this.state.sessions[req.sessionID]);
            }
            this.state.sessions[req.sessionID] = client.sessionId;
            return true;
        }
        return false;
    }

    onJoin(client, options) {
        super.onJoin(client, options);
        this.state.createPlayer(client.sessionId);
    }

it works on the server side, but on the client side:

• in the old tab it correctly removes the user

• in the new tab it adds a new user, and straight away removes it

Looking at the code here (https://github.com/colyseus/schema/blob/master/src/Schema.ts#L294):
I guess this is because it takes the newly added player's index.

Client decoder (C#) should support decoding newer version of schema correctly.

Hello,
As I see that the current client decoder (C#) does not support decoding with data encoded by a new schema version (adding new properties/types), could we update the client lib to support this?

With online mobile games, we usually update our schemas when adding/updating features.
Whenever we submit a new client version for store review, we also need to deploy the new server version. The live client version (old schema) which users playing should also communicate with the new server without any problems.

Thank you.

Getting an Uncaught TypeError in the colyseus.js

I am getting this errors in the colyseus.js file

Uncaught TypeError: h is not a constructor
    at Function.t.decode (colyseus.js:formatted:3325)
    at e.handshake (colyseus.js:formatted:2906)
    at e.onMessageCallback (colyseus.js:formatted:1156)
    at t.value (colyseus.js:formatted:2267)
Uncaught TypeError: Cannot read property 'decode' of undefined
    at e.setState (colyseus.js:formatted:2892)
    at e.setState (colyseus.js:formatted:1165)
    at e.onMessageCallback (colyseus.js:formatted:1139)
    at t.value (colyseus.js:formatted:2267)

I tried to debug a little bit and found that the error is caused by an undefined "rootType" in this section var h = u[i.rootType], f = new h;

To reproduce the error just use the example as in the doc:

const schema = require('@colyseus/schema');
const Schema = schema.Schema;
const MapSchema = schema.MapSchema;
const type = schema.type;

class Player extends Schema {
}
type("number")(Map.prototype, "x");
type("number")(Map.prototype, "y");

class PlayersState extends Schema {
constructor () {
super();
this.players = new MapSchema();
}
}
type({ map: Player })(PlayersState.prototype, "players");

class PlayersRoom extends Room {
onInit (options) {
this.setState(new PlayersState());
}
/*
...
*/
}

In the client side, connect to the room and listen to the room.sate.players.

P.S: I am not using Typescript, only javascript.

Uncaught TypeError: Cannot read property '_schema' of undefined

This error can have various causes. After some days debugging mazmorra.io, I've noticed what was causing it for my game:

• The user requests to leave the room
• The user sends commands to the server to update his player's state while he's leaving
• The state of the room is now inconsistent, and any other user trying to join it will receive the Uncaught TypeError: Cannot read property '_schema' of undefined error.

The workaround I've made so far is by not allowing any actions while the user has requested to leave:

onMessage(client, message) {
    const player = this.players.get(client);

    if (player.isSwitchingDungeons) {
      console.log("player sending actions while switching dungeons or removed.");
      return;
    }

    // ...process player input
}

This is a bug though. The serializer shouldn't be considering updates from removed structures.

ArraySchema<number>: Not initializing properly with one index

when initializing an ArraySchema<number> with 1 value in the array, it initializes empty. This only appears to be an issue with Array<number>, I do not see this issue with Array<string>.

class State extends Schema {
  @type(['number']) numbers: ArraySchema<number>;
  constructor(numbers = [1]) {
    super();
    this.numbers = new ArraySchema(...numbers);
    console.log(this.numbers); // output: ArraySchema [ <1 empty item> ]
  }
}

schema-codegen: use tsconfig.json configuration

Currently, the schema-codegen tool cannot recognize imports from custom paths like @/xxx/yyy. By reading/specifying the tsconfig.json this should be fixed.

MapSchema: allow removing and re-adding item with same key

I've added a failing test case for this:

const state = new State();
state.mapOfPlayers = new MapSchema<Player>();
state.mapOfPlayers['one'] = new Player("Jake");
state.mapOfPlayers['two'] = new Player("Katarina");

const decodedState = new State();
decodedState.decode(state.encodeAll());

delete state.mapOfPlayers['one'];
state.mapOfPlayers['one'] = new Player("Jake 2");
decodedState.decode(state.encode());

delete state.mapOfPlayers['two'];
state.mapOfPlayers['two'] = new Player("Katarina 2");
decodedState.decode(state.encode());

assert.equal(decodedState.mapOfPlayers['one'].name, "Jake 2");
assert.equal(decodedState.mapOfPlayers['two'].name, "Katarina 2");

A workaround, for now, is not to call delete state.mapOfPlayers['two'].

Test result:

  1) MapSchema
       should allow to remove and set an item in the same place:

      AssertionError [ERR_ASSERTION]: 'Katarina 2' == 'Jake 2'
+       expected - actual

-      Katarina 2
+      Jake 2

      at Context.<anonymous> (test/MapSchemaTest.ts:26:16)

Support array.splice

Currently arr.splice(x) is not supported on ArraySchema.

1.0.0 Proposal: Support for JSON objects or out out the box deserialization of certain data types

One idea for a new feature would be out the box deserialization of certain data types, for example, serializing and deserializing JSON objects into strings or even better – supporting them by default via string conversion or a different method. (I am not sure which method because I don't know why it is not possible to support object in the schema by default in the first place).

Support annotating getter methods

Suggested by @Wenish

@type('boolean')
get isAlive (): boolean {
    return true
}

support polymorphism ?

my game model has an inheritance tree (Asteroid and SpaceShip both inheriting from SpaceObject), and I want to have a map of all objects in the room state(@type({map: SpaceObject})).
It would be nice if Inheritance limitations were documented (after all, there are some // support inheritance comments in the code).

schema-codegen cannot read 'arguments' of undefined

When using

schema-codegen src/schema/schema.ts --output client/schema --csharp

it gives the the error of

Cannot read property 'arguments' of undefined

I've tried it one of 100 different ways.

Currently using "@colyseus/schema": "^0.4.61",

ArraySchema: triggering `onAdd` instead of `onChange`

Reported by BRAINFUBAR on Discord:

I thought since I had initialized my array with zeros and the length never changes, it would trigger onChange instead of onAdd. But it appears that onChange is only triggered when the initial value is nonzero (checking for truthyness I assume)

This sounds like a bug to be inspected 👀

Schema deep cloning

Hello, I think we need this update:
Schema cloning should support deep cloning: property is a Schema object or collection objects (ArraySchema, MapSchema) should also be deeply cloned. This will make sure the changed states of the Schema is reseted and fully de-coupled from the original Schema object.

Thank you.

Provide a custom "LANG.ts" option in code generator cli

the inner-provided LANG.ts in code generator may not satisfy everyone's need, such as me. It will be perfect if I can implement my own LANG.ts by responding some kind of protocol (interface or methods) . BTW, I also defined some enum type in my server side schema, generate them into the client side code is a reasonable requirement, isn't it.

Wrong Decoding of Mapschema with number numeric string keys in C# client

when we create a mapschema with numeric indexes and try to decode it in the C# client the updated values are called with the wrong keys!

i created a repository to showcase this bug.

in it the server starts a mapschema with the values:

items = {"bbb":1, "aaa":1, "221":1, "021":1, "15": 1, "10":1}

it is initialised correctly in the unity client, but when we try to update a value in the client, another one is updated in the client:

for example, if we set

items["10"] = 2

in the client the value updated will be items["bbb"] with value 2

i this problem occurs because the server serialiser depends on object iteration order in javascript, and objects in javascript have a strange iteration order (link1, link2)

so in our case:
the client starts the object with correct order:

items = {"bbb": 1, "aaa": 1, "221": 1, "021": 1, "15": 1, "10": 1}

but in the server the indexes are reordered to:

items = {"10": 11, "15": 1, "221": 1, "bbb": 1, "aaa": 1, "021": 1}

so when we update the items["10"] = 2 the client receives something like:
update the object at index 0, that in the server is "10"(javascript order) but in the client is "bbb"(insertion order)

one possible solution in my opinion is tho keep the keys and values in a proper Map instead of an object, so the insertion order is guaranteed

Codegen for TypeScript should quote primitive types within array/map decorators

export class MyState extends Schema {
  @type(["int16"])
  terrainArray:ArraySchema<number>

npx schema-codegen WorldRoom.ts --output ../fe-client/src --ts

The generated MyState.ts file has the int16 without quotes:

@type([ int16 ]) public terrainArray: ArraySchema<number> = new ArraySchema<number>();

Causing "Cannot find name 'int16'." My purpose here is just to codegen my server's classes for use in my separate TypeScript client for vscode hinting.

I'm new but I'll submit a PR in a minute.

Fix autocompletion for items inside Maps

Autocompletion is not working for items inside Maps, because their types are:

schema/src/types/MapSchema.ts

Line 44 in 0f1f75f

[key: string]: T | any;

This may be fixed when Negated Types get merged into TypeScript.

Angular error ng build is error

version;
"colyseus.js": "^0.12.2",
"@colyseus/schema": "^0.5.34",

ng build
when i try to build

ERROR in node_modules/@colyseus/schema/lib/types/HelperTypes.d.ts:20:21 - error TS2536: Type '{ true: "false"; false: "true"; }[({ [K in ({ [K in keyof T[K]]: "false"; } & { [key: string]: "true"; })[keyof T[K]]]: "true"; } & { [key: string]: "false"; })["false"]]' cannot be used to index type '{ 'false': K; 'true': never; }'.

20     [K in keyof T]: {
                       ~
21         'false': K;
   ~~~~~~~~~~~~~~~~~~~
22         'true': never;
   ~~~~~~~~~~~~~~~~~~~~~~
23     }[IsFunction<T[K]>];

ng serve
when i open

  ERROR in node_modules/@colyseus/schema/lib/Schema.d.ts(54,73): error TS2536: Type 'K' cannot be used to index type 'this'.
   node_modules/@colyseus/schema/lib/Schema.d.ts(54,97): error TS2536: Type 'K' cannot be used to index type 'this'.
   node_modules/@colyseus/schema/lib/types/HelperTypes.d.ts(20,21): error TS2536: Type '{ true: "false"; false: "true"; }[({ [K in ({ [K in keyof T[K]]: "false"; } & { [key: string]: "true"; })[keyof T[K]]]: "true"; } & { [key: string]: "false"; })["false"]]' cannot be used to index type '{ 'false': K; 'true': never; }'.

1.0.0: Use Map internally for MapSchema

The MapSchema currently acts like a plain object in JavaScript, which lacks lots of useful methods that a Map would provide.

For version 1.0.0, the MapSchema should be re-structured to use a Map, and implement all methods of the Map structure: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Map

"type" field in JSON as protected keyword

Type field name look protected keyword but rise an unexpected error :

https://github.com/colyseus/schema/blob/master/src/codegen/parser.ts#L18

Reproduce :
Add any json anywhere in code extending Schema, exemple :
this.state.broadCastMessage({
type: "ACTION",
from: player.hero.properties.id
});

Code generation via NPX rise this error :

Cannot read property 'find' of undefined

onAdd should be called when replacing items

When having an ArraySchema, removing and adding an item in the same patch will result in onChange instead of onAdd.

// minimal reproduction example
player.cards.splice(0, 1);
player.cards.push(new Card());

Added a failing test case for this here: 438af2a

.listen() utility does not recognize boolean properties

TypeScript is not recognizing boolean properties when using .listen() utility

Example

class State extends Schema {
  @type("boolean")
  prop: boolean;
}

const state = new State();
state.listen("prop", () => {}) // Argument of type '"prop"' is not assignable to parameter of type ...

@filter: how to approach it?

I've been thinking about how to approach the @filter() for better performance.

Internally, @colyseus/schema uses a simple array of numbers (number[]) to encode the full state. This array of numbers is turned into a Buffer to be sent over the wire via WebSockets.

To avoid re-encoding every structure again for each client, here's my proposal:

• Encode the full state once:
  • Cache the index where each structure starts and ends within the .$changes property ($changes can be renamed to $cache)
  • Turn the full encoded state into a Buffer
• For each client:
  • Iterate through all @filter() in the structures
    • Retrieve index where filtered values are positioned at
    • Use Buffer.slice() to retrieve the portions of the state the client is going to receive
  • Use Buffer.concat() to combine all sliced portions.
  • Send it!

I've done some benchmarks and using Buffer.slice() + Buffer.concat() consumes way less CPU than operating through the number[] arrays and then turning them into a Buffer.

Support change listeners on server

Versions:

• [email protected]
• @colyseus/[email protected]

I'm experimenting with ways to synchronize simulation state of a physics library with a Schema tree. I want to subscribe to changes in Schema so I can apply them to the simulation. When change listeners are set (onAdd, onChange, etc.) on a MapSchema, I get the following error:

Error: a 'BodyState' was expected, but 'Function' was provided in PhysicsState#bodies
    at new EncodeSchemaError (/app/node_modules/@colyseus/schema/lib/Schema.js:25:42)
    at assertInstanceType (/app/node_modules/@colyseus/schema/lib/Schema.js:64:15)
    at _loop_2 (/app/node_modules/@colyseus/schema/lib/Schema.js:542:25)
    at PhysicsState.Schema.encode (/app/node_modules/@colyseus/schema/lib/Schema.js:578:13)
    at _loop_2 (/app/node_modules/@colyseus/schema/lib/Schema.js:439:48)
    at SystemState.Schema.encode (/app/node_modules/@colyseus/schema/lib/Schema.js:578:13)
    at SchemaSerializer.applyPatches (/app/node_modules/colyseus/lib/serializer/SchemaSerializer.js:28:44)
    at MainRoom.broadcastPatch (/app/node_modules/colyseus/lib/Room.js:206:33)
    at Timeout._onTimeout (/app/node_modules/colyseus/lib/Room.js:93:22)
    at listOnTimeout (internal/timers.js:531:17)

I think this behavior is currently expected since these callbacks are listed under the Client-side heading in the Colyseus docs.

support @schema class decorator

Hi
I am using a vector library that I like and I want to add it to a schema state tree. I had to resort to mixins to combine both the vector type and the Schema parent.
my code currently looks something like this:

import { vec2 } from 'tsm';
import Mixin from '@alizurchik/ts-mixin';
import { type, Schema } from '@colyseus/schema';

export class Vec2 extends Mixin(Schema, vec2) {
    public static readonly zero = Object.freeze(new Vec2([0, 0]));
    public static readonly one = Object.freeze(new Vec2([1, 1]));
}
type(['float32'])(Vec2.prototype, 'values');

I suggest providing a decorator API for schema classes themselves, so that the user can inherit ant domain classed they may have, and annotate the class and fields.

1.0.0 Proposal: generate enums during schema-codegen

I think that something useful would be to be able to map enum or initial values of the variables.
In case you have message exchanges, you don't have to duplicate a class / enum in the final language.

Schemagen does not work with indirect inheritance

Place.cs will not be generated

import { type, MapSchema, ArraySchema } from "@colyseus/schema";
import { Avatar } from "./Avatar";
import { IPlace } from "../models/Place";
import { PlaceMobi } from "./PlaceMobi";
import { VectorTriple } from "./VectorTriple";
import { Quad } from "./Quad";
import { BaseSchema } from "./BaseSchema";

export class Place extends BaseSchema<IPlace> {
  @type("string")
  id: string;
  @type("string")
  name: string;
  @type("string")
  ownerId: string;
  @type("string")
  description: string;
  @type("string")
  rating: number;
  @type("number")
  maxAvatars = 10;
  @type(VectorTriple)
  entrance: VectorTriple;
  @type("number")
  entranceDirection: number;
  @type({ map: Avatar })
  avatars: MapSchema<Avatar>;
  @type([PlaceMobi])
  mobis: ArraySchema<PlaceMobi>;
  @type([Quad])
  quads: ArraySchema<Quad>;

  setData(data: IPlace): void {
    // ..........
  }

  getData() {
    // ...................
  }
}

import { Schema } from "@colyseus/schema";
import { IParcelable } from "../patterns/IParcelable";

export abstract class BaseSchema<T> extends Schema implements IParcelable<T> {
  constructor(data?: T) {
    super();

    if (data != null) {
      this.setData(data);
    }
  }

  abstract setData(data: T): void;
  abstract getData(): T;
}

strings longer than 31 length result in "Cannot read property '_schema' of undefined"

When using a long string value with a primitive type "string" inside a Custom Schema type I got this unexpected error in the client side when trying to listen to state changes with

room.onJoin.add(function() { }) and room.state.players.onAdd = (player, key) => { } 

// Sample code to reproduce the error

const colyseus = require("colyseus")
const schema = require("@colyseus/schema")
const Schema = schema.Schema;
const MapSchema = schema.MapSchema;
//const ArraySchema = schema.ArraySchema;
const type = schema.type;

class Player extends Schema {
  constructor (id, str) {
    super();

    this.id = id;
    this.str = str;
  }
}
type('string')(Player.prototype, 'id');
type('string')(Player.prototype, 'str');


class MyState extends Schema {
    constructor () {
        super();

        this.players = new MapSchema();
    }
}
type({ map: Player })(MyState.prototype, 'players');


class MyRoom extends colyseus.Room {
    // When room is initialized
    onInit (options) {
        this.setState(new MyState());
    }

    // Checks if a new client is allowed to join. (default: `return true`)
    requestJoin (options, isNew) { }

    // Authorize client based on provided options before WebSocket handshake is complete
    onAuth (options) { }

    // When client successfully join the room
    onJoin (client, options, auth) {
        this.state.players[client.id] = new Player(1, 'test string'); //<-- This one works
        // uncomment this one bellow
        // this one rises the error
        //this.state.players[client.id] = new Player(1, 'Some Dummy And Very Long String Goes Here For Testing Purpose'); //<-- This one doesn't
    }

    // When a client sends a message
    onMessage (client, message) { }

    // When a client leaves the room
    onLeave (client, consented) { }

    // Cleanup callback, called after there are no more clients in the room. (see `autoDispose`)
    onDispose () { }
}

// in the index.js
const MyRoom = require("./MyRoom");
gameServer.register('my_room', MyRoom);

---

Good work by the way!
Thank you :)

Issue with special characters on strings

Testing a Defold/LUA project today I've found that a string containing the "гхб" characters causes decoding issue. Need to investigate what's wrong.

share state with worker thread

Hi
I'm experiencing some high CPU usage in the game loop, running physics and collision detection. I thought I'd separate that engine to a node wiorker, seeing as it's designed to take advantage of js buffers (shares the same memory instead of serializing it).

would love to hear your opinion on the most performant way to approach that.

Add codegen for Java

@doorbash has done an amazing job to port the schema de-serializer to Java (https://github.com/doorbash/colyseus-java/releases/tag/2.0.0), now it would be great that the schema-codegen tool could convert TypeScript schema files into Java.

• https://gist.github.com/doorbash/445310840227fd51d75f2da4f4e4b8a6
• https://gist.github.com/doorbash/8621303acf9375f7e295e88196a86061

Language agnostic schema definition

Would be great to have language agnostic schema definition (in yaml may be?) This would allow developers to create their own schema builders for their own client and server extern implementations.

Motivation: as I maintain haxe server externs I find it difficult to keep schemas on server and client side in sync. Having them defined in single source of truth would be a huge improvement. I would happily create schema-codegen for haxe server externs if there is a schema definition out there.

1.0.0: Implement a Set structure

A Set would be really useful: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Set

Re-using the same instance on multiple fields / .clone() automatically?

It's currently not possible to re-use the same Schema instance to be encoded on multiple fields, as whenever it is encoded once, all changes are going to be discarded for the next field.

Approaches that can be taken internally to improve this:

• Clone the instance automatically whenever it's assigned for the second time (easiest route)
• Track the instance internally, and actually re-use the same instance during decoding time (more complicated)

Add codegen for Typescript

Hi,

Is the codegen for Typescript already final? I run schema-codegen ./src/server/schema/GameState.ts --output ./src/client/schema/ --ts to generate the client schema. It seems to work properly, but won't generate anything if the class is empty.

In my last project I simply included the server side defined GameState.ts. However, this gives various unnecessary functions within my client (both TS based).
As I had to look through the source code to find this feature, I would open a Pull Request if appropriate to update the documentation:

# C#/Unity
schema-codegen ./schemas/State.ts --output ./unity-project/ --cs

# C/C++
schema-codegen ./schemas/State.ts --output ./cpp-project/ --cpp

# Haxe
schema-codegen ./schemas/State.ts --output ./haxe-project/ --hx

# Typescript
schema-codegen ./schemas/State.ts --output ./typescript-project/ --ts

Cheers

Haxe macros compatible with colyseus-hxjs

@serjek made a Colyseus Server using macros to define the Schema https://github.com/serjek/colyseus-hxjs

It would be great to have the same for the schema decoder in the Haxe client. I've started doing this here: fbf9230

The current problem is filtering fields from the parent class. Untyped fields (without @:type() annotations) shouldn't be considered when generating the decoding structure.

Problematic pieces of code are:

schema/decoders/haxe/io/colyseus/serializer/schema/Schema.hx

Lines 52 to 60 in fbf9230

	var parentFields = superClass.fields.get();
	for (f in parentFields) {
	// FIXME: check for annotated types instead of starting with "_".
	trace("EXISTING FIELD => ", f.name, f);
	if (f.name.indexOf("_") != 0) {
	index++;
	}
	}
	}

I've tried assigning metadata for declared fields, but it seems after they're evaluated, the metadata is gone:

schema/decoders/haxe/io/colyseus/serializer/schema/Schema.hx

Line 28 in fbf9230

f.meta.push({name: "typed", pos: haxe.macro.Context.currentPos()});

I've opted out of using constants for type arguments like STRING in favor of "string" because the STRING constant gets evaluated as EConst(CIdent()), which makes it difficult to select if the field is going to be inside _childSchemaTypes or _childPrimitiveTypes:

schema/decoders/haxe/io/colyseus/serializer/schema/Schema.hx

Lines 68 to 75 in fbf9230

	switch f.meta.params[1].expr {
	case EConst(CIdent(exp)):
	exprs.push(macro $p{["this", "_childSchemaTypes"]}.set($v{index}, $i{exp}));
	case EConst(CString(exp)):
	exprs.push(macro $p{["this", "_childPrimitiveTypes"]}.set($v{index}, $v{exp}));
	default:
	}

Error ocured when in Schemas use word "type" for var names

Error ocured when in Schemas use word "type" for var names

Bad "index change" encoding for Maps when using the same value

class MapWithPrimitive extends Schema {
    @type({map: "boolean"}) mapOfBool = new MapSchema<boolean>();
}

const state = new MapWithPrimitive();
state.mapOfBool['one'] = true;

const decodedState = new MapWithPrimitive();
decodedState.mapOfBool.onAdd = function(value, key) {
  console.log("ON ADD", value, key);
}
decodedState.decode(state.encodeAll());

// as both "one" and "two" are `true`, the encoder thinks it's a index change
state.mapOfBool['two'] = true;

decodedState.decode(state.encode());

console.log(decodedState.toJSON());

allow to push / shift arrays

need to ensure changes are correctly triggered when deserializing a push + shift in an array

Error on ArraySchema splicing

Hi, and thank you for your work!

I'm having issues with splicing an ArraySchema of a custom data type that extends Schema.

TypeError: Cannot read property 'deleteIndex' of undefined
    at /home/ftruzzi/projects/.../server/node_modules/@colyseus/schema/lib/types/ArraySchema.js:125:33
    at Array.map (<anonymous>)
    at ArraySchema.splice (/home/ftruzzi/projects/.../server/node_modules/@colyseus/schema/lib/types/ArraySchema.js:121:22)

Looking at ArraySchema.js:125, it looks like $changes has no parent property in my case. Adding a check and skipping the line makes the splicing work, but I have no idea of the consequences.

Thank you!

Understanding typescript and Schema

Hi,

I'm currently struggling a bit with structuring game data like this:

export class Ability extends Schema {
    @type('string') name;
    @type('number') hit;
    @type('number') value;

    constructor(abilityData: AbilityData) {
        super();
        this.name = abilityData.name;
        this.hit = abilityData.hit;
        this.value = abilityData.value;
    }
}

export class Char extends Schema {
    @type('string') name;
    @type('string') type;
    @type('string') category;
    @type(Ability) slot1;
    @type(Ability) slot2;

    constructor(charData: CharData) {
        super();
        this.name = charData.name;
        this.type = charData.type;
        this.category = charData.category;
        this.slot1 = new Ability(charData.slot1);
        this.slot2 = new Ability(charData.slot2);
    }
}

export class Team extends Schema {
    @type(Char) char1;
    @type(Char) char2;
    @type('number') activeChar;

    constructor(teamData: TeamData) {
        super();
        this.char1 = new Char(teamData.char1);
        this.char2 = new Char(teamData.char2);
        this.activeChar = teamData.activeChar;
    }
}

when I do this:

var myTeam = {
        char1: {
            name: '',
            type: '',
            category: '',
            slot1: {
                name: '',
                hit: 0,
                value: 0,
            },
            slot2: {
                name: '',
                hit: 0,
                value: 0,
            },
        },
        char2: {
            name: '',
            type: '',
            category: '',
            slot1: {
                name: '',
                hit: 0,
                value: 0,
            },
            slot2: {
                name: '',
                hit: 0,
                value: 0,
            },
        },
        activeChar: 1,
    };
var client = new Colyseus.Client('ws://localhost:2567');
var room = client.join('battle', myTeam);

I get this error

TypeError: Cannot read property 'name' of undefined
    at new Ability (/Volumes/WORK/SERVER/models.ts:38:33)

ArraySchema: support `map` and `filter`

v0.10 : value validation in schema 0.4.14

"5c9964165e8afb70f44bc762" look like string, isn't ?

        return _super !== null && _super.apply(this, arguments) || this;
                                         ^
Error: a 'string' was expected, but a '"5c9964165e8afb70f44bc762"' was provided.

Impossible to create an array of arrays type

Imagine you have a board in game with x and y axis. You want a 2d array in your state.

Right now the only way to achieve this is this:

class Row extends Schema {
  @type(['number'])
  items
}
class Board extends Schema {
  @type([ Row ])
}

Which means you would then have to access each cell like: board[12].items[3] instead of just board[12][3].

Or am I missing something obvious?

1.0.0 Proposal: Officially supported data filters

One feature that I would like to see in 1.0.0 is officially supported data filters.

fix code generation with inheritance

colyseus/colyseus-unity-sdk#60

toJSON() should be rescursive for nested Schema properties.

Could you please implement the toJSON() method to have recursion call for other nested Schema properties inside a Schema?

Map decode will fail for some stored values

The decode logic for a map will test for nil values (192) to determine if a map entry was removed. However, a map which has a int32 type, for example, can have the first byte with a value of 192. For example, the number 3520 will be encoded as 192, 13, 0, 0 but will be decoded as a nil value (i.e. the entry was removed). This will happen for any encoded value where 192 is a valid first byte (uint8, int8, uint16, etc.)

This logic tests for the nil values

schema/src/Schema.ts

Line 356 in 76a8cac

if (decode.nilCheck(bytes, it)) {

and calls the onRemove event. Removed values should be encoded in a different way, maybe we could add a byte to indicate the removal or not of the key before the new value, such that this byte can then have a magic value and the actual value of the field can be anything?