kettle11 / tangle

Radically simple multiplayer / networked WebAssembly

WARNING

This library is incredibly new. I value building in public so I'm making Tangle available very early. As a result Tangle has major bugs, TODOs, and known issues. The library is not stable: it will change constantly! Give it a try, be patient, and consider contributing!

Tangle

Tangle is a library that aims to make multiplayer apps and games far easier to build.

Tangle 'magically' wraps WebAssembly so you can write programs without worrying about message passing, serialization, or consensus.

Play with a live demo:

Check out the examples to get started!

Questions

What is this for?

Tangle is a great fit for small games and apps that want to add multiplayer. In the future it may be useful for all sorts of things including (but not limited to): syncing programs between servers / clients, untrusted plugins in networked software, 'metaverse'-like scripting, and backend-less collaborative software.

Does this use WebSockets or WebRTC?

Under the hood Tangle uses peer to peer WebRTC connections. This may change!

How is this hosted?

Connections are peer to peer but the initial connection needs to be facilitated by a central server. Right now I run a free instance of that server but if it gets serious traffic I may offer a paid version.

How does it work?

Tangle syncs initial state and networks all subsequent inputs that could cause simulations to diverge. Tangle keeps every bit identical between all connected Wasm instances. 'Snapshots' are taken at each simulation step to enable 'rewinding' and 'replaying' events to ensure all peers apply events in the same order, without adding latency to user input.

TODO: Write a more detailed explainer.

What are its limitations?

Due to how rollback works frequently updated programs will have increased input latency if they do not keep CPU and memory usage low. This is an area that will see future improvement.

Tangle does nothing to merge long stretches of offline editing. It is not a CRDT.

What languages does this support?

In theory: Any language that targets WebAssembly. In practice I've only tested AssemblyScript and Rust.

Is this web only?

Right now Tangle is implemented as a TypeScript library for web but its fundamental architecture could eventually be made to work on native platforms.

tangle's Issues

Consider using unreliable RTCDataChannels for better latency

Traditionally games use UDP-like packets for things like position updates and reliable-ordered protocols for important events or streaming assets. Right now Tangle uses reliable-ordered RTCDataChannels for everything, which means that all events are received by Tangle in order. But if message 12 arrives before message 10 it needs to wait for message 10 to arrive before Tangle processes it.

Tangle could instead rely on its rollback-based architecture to allow events to be processed in any order. This may dramatically improve latency in some cases in exchange for more internal code complexity.

Harden Tangle to Malicious Peers

If a peer is malicious they should only be able to cause a bad experience for themselves. Tangle should be audited to find any potential ways a malicious peer could intentionally desync another peer. In scenarios with only one other peer this is likely impossible to do perfectly, but in rooms with 2+ peers it's should be possible to verify and reach consensus. In some cases a peer could be designated as a source of authority that all peers should defer to, like when running an authoritative server.

Consider giving JavaScript host hooks to respond to rollbacks

Tangle only syncs WebAssembly but to do anything useful it has to integrate with the host environment.

Tangle could expose some way for the host environment to respond to rollbacks.

A pontential solution could look something like this:

let imports = {
  createBuffer: () => {
    return context.createBuffer(); 
  }
};

let rollbacks = {
  createBuffer: (args, returnValue) => {
    context.deleteBuffer(returnValue);
  }
};

const result = await Tangle.instantiateStreaming(fetch("my_wasm.wasm"), imports, { rollbacks });

Then Tangle would have to track internally which import calls have rollbacks and when Tangle rolls back and resimulates it will call the appropriate rollback callbacks.

---

A bigger-picture solution might be some sort of meta-Tangle layer that connects individual Tangle nodes and standardizes how rollback events are passed between them. The JavaScript host could be seen as a 'node' that must be manually networked but it still abides by the same interface. Probably something like that is too big-picture to do soon but it's worth keeping in mind.

In-Module 'Dirty Pages' Array

Firstly, what a clever idea, I'm fascinated by it! Lockstep has been done by snapshotting runtimes (LUA of note) before, but this is a whole new level of flexible. It would fit a game I've been toying with for years quite well, which requires untrusted networked user code to run alongside the game in a browser. Many kudos to you.

Let me test some understanding:

• You have wasm_guardian which rewrites the WASM module to call a host (JS) function on each memory mutation. That is an immense amount of overhead I'm guessing?
• Which I assume is why it was removed in this commit. But wasm_guardian is still being applied to the WASM module.
• Both of those were in an effort to not have to snapshot the entire WASM module memory each time, which can very quickly grow into gigabytes.
• gzip was introduced as a workaround for snapshot memory size. It's computationally expensive.

Taking a page out of hardware TLBs (sorry for the pun) I could see tracking dirty pages in WASM memory itself. Allocate a static array, probably 1 byte per flag, say 64KiB pages, so 64k entry dirty array. That reduces the runtime overhead to dirty_flags[mem_addr >> 16] = 1. It's still a lot for each and every memory write, but without access to the hardware TLB (don't see that happening from a browser any time soon) I can't see any way to avoid that cost. CPUs all use barrel-shifters, so that whole thing is likely to be 3 or 4 cycles and cache coherency should be really good. I saw notes in your code somewhere about this too, but excluding the stack from this tracking would be a huge win.

Again, awesome work, I love it.

Reduce size of `rust_utilities.wasm`

The important part of Tangle is a very small distributable, but the rust_utilities.wasm file is ~500kb! Most of that is due to the use of Walrus, a library for working with Wasm. By rewriting / replacing Tangle's use of that library it should be possible to significantly reduce the final size to use Tangle.

Take less frequent snapshots

Right now Tangle takes a snapshot every single time a new event comes in. This incurs a lot of overhead. Snapshots should instead be taken after certain amounts of time have passed.

Throw better errors when the matchmaking server can't be reached

Right now it's really unclear what's happening if the matchmaking server is unavailable for some reason.

Cannot read property: `_in_call_that_will_be_reverted`

Description

tangle/tangle_ts/src/tangle.ts

Lines 92 to 97 in 9abd443

	moduleImports[importName] = function (...args: any) {
	const r = importValue(...args);
	// This call will be reverted so it's OK if it causes a temporary desync.
	if (this._in_call_that_will_be_reverted) {
	return r;
	}

When I call imported JS functions, this becomes undefined.
So, this._in_call_that_will_be_reverted cannot be read.

What I did
I ran the example counter but in Zig instead of AssemblyScript.

my_wasm.zig

var num: f64 = 0.0;

extern fn report_number_change(n: f64) void;

export fn increment( val: f64) void{
    num += val;
    report_number_change(num);
}

export fn multiply( val: f64) void{
    num *= val;
    report_number_change(num);
}

However, I doubt that is a Zig-related problem.
As I know, the value of the this keyword depends on its context.
Then, what is the value of this, if I call a function in a WASM?

To check that, I modified tangle.js:

              const r = importValue(...args);
              console.warn(`${importName}():${this}`)
              if (this?._in_call_that_will_be_reverted) {
                return r;
              }

Then it says:

The first 4 calls are exports.increment(1)s, and others are exports.multiply.callAndRevert(3)s.

This shows that wrapping a function call did not help!

tangle/tangle_ts/src/tangle.ts

Lines 328 to 331 in 9abd443

	wrapped_function.callAndRevert = (...args: any) => {
	this._in_call_that_will_be_reverted = true;
	this.call_and_revert(key, ...args);
	this._in_call_that_will_be_reverted = false;

Additional note
I tried an arrow function, but it made this._in_call_that_will_be_reverted undefined, and this was not a Tangle instance. (it was a function.)
moduleImports[importName] = (...args) => {
I want to figure out why this happens, but the keyword this was nearly impossible to search.

Consider moving network code to Rust (?)

taking a quick look through the code, and it seems as though the code is being run through a custom rollback netcode implementation written in typescript. However, wouldn't typescript be a bit slow? It feels like it would be better to use a network library written in rust, like GGRS

Is there a way to distinguish players?

I am impressed by this work.
It is working magically. (I've only tried the demo
and examples, though)
However, I could not find how to distinguish players.
For example, I need this feature to prohibit other players to control a player's character.
I have considered doing it on the JS side, but I could not get my identification from Tangle.

Document Signaling / Room Server

Add mechanism to read / write from WebAssembly memory

Right now a big missing feature in Tangle is the ability to pass in large data or read data from Tangle.

Libraries like wasm-bindgen directly write / read Wasm memory like this new Uint8Array(wasm.memory.buffer) but unfortunately Tangle can't polyfill that directly without creating desyncs.

Tangle should expose an alternative way of reading / writing to Wasm memory in bulk.

---

I wonder if Tangle could have an optional mode that replaces WebAssembly.instantiate completely with Tangle.instantiate and new Uint8Array with something that checks if the value passed in is a special Tangle object?

This could make using Tangle with something like wasm-bindgen as easy as importing the library.