streamich / crowd.hyoo.ru

CROWD - like CRDT but better

Conflict-free Reinterpretable Ordered Washed Data (CROWD)

Key Properties

Conflict-free

• Any states can be merged without conflicts.
• Strong Eventual Consistency.
• Merge result is independent of merge order on different peers.
• Branch merge is semilattice.

Reinterpretable

• Same state can be reinterpreted as any CROWD Storage.
• CROWD Storage type can be changed dynamicaly without data migration.
• Cross-merge is available between different CROWD Storages.

Ordered

• Changes from same peer are always ordered and can't be reordered.
• Deltas from same peer aren't commutative.
• All deltas are idempotent.

Washed

• Historical data isn't stored (except tombstones).
• Small footprint. Metadata size ~= user data size.
• Past state can't be reproduced.
• Garbage collection isn't required.

Data

• Closest to user data as more as possible. Just list of values and list of stamps.
• Deltas are simple slices of full state.
• Deltas can be merged together to reduce transmit size.

Comparison of Approaches

With CRDT

• CRDT has stronger guarantees for events commutativity. It gives a strong restriction for deleting old data. CROWD slightly weakens the guarantees, which gives more compact data representation without garbage collection and compactification.
• Some CROWD storages are accidentally dCRDT too.
• Stored CROWD State can be reinterpredeted by different CROWD Storages. Different CROWD Storages may be cross merged. CRDT structures are incompatible in general.

With OT

• OT stores full edit history which is redundant. CROWD competely erases history. For history navigation purposes periodically snapshots is better solution for both.
• OT requires history rebase for convergence. This is too slow and complex. CROWD merge is very simple and fast.

Available Stores

CROWD	CRDT
CROWD Counter	Is equal to dCRDT PN-Counter
CROWD Register	Is same as CvRDT LWW-Register
CROWD Unordered Set	Is equal to dCRDT LWW-Element-Set
CROWD Ordered Set	No equal type
CROWD Tagged Union	No equal type
CROWD Dictionary	No equal type
CROWD Text	No equal type
CROWD JSON	No equal type
CROWD Graph	No equal type

Utilites

• CROWD Store - Base store class with common CROWD API.
• CROWD Clock - Manages stamps for composed CROWD stores.

Common API

• delta( clock ) Returns delta between past clock and now.
• apply( delta ) Merges delta to current state.
• toJSON() Returns full state dump.
• fork( peer: number ) Makes independent clone with fixed peer id for testing purposes.

State/Delta Format

{
	"values": ( string | number | boolean | null )[]
	"stamps": number[] // ints
}

Reinterpretations

From \ To	Counter	Register	Tagged Union	Unordered Set	Ordered Set	Dictionary	Text
Counter	✅ Same	❌	❌	⭕ Set of summands	⭕ Set of summands	❌	❌
Register	✅ As first summand	✅ Same	⭕ As first type	✅ As key	✅ As key	❌	❌
Tagged Union	✅ Value as first summand	✅ Value	✅ Same	⭕ Set of type and value	⭕ Set of type and value	❌	❌
Unordered Set	❌	⭕ Last added key	❌	✅ Same	✅ Accidental order	❌	❌
Ordered Set	❌	⭕ Last inserted key	❌	✅ Remain order	✅ Same	❌	❌
Dictionary	❌	⭕ Last changed value	❌	⭕ Set of values	⭕ Set of values	✅ Same	❌
Text	❌	❌	❌	❌	❌	⭕ With keys: flow, token	✅ Same

• ✅ Expected behaviour.
• ⭕ Unexpected but acceptable behaviour.
• ❌ Unacceptable behaviour in most cases.

Usage Example

// // Usage from NPM. Isn't required in MAM.
// import {
//   $hyoo_crowd_numb,
//   $hyoo_crowd_reg,
//   $hyoo_crowd_union,
//   $hyoo_crowd_set
//   $hyoo_crowd_list,
//   $hyoo_crowd_dict,
//   $hyoo_crowd_text,
// } from 'hyoo_crowd_lib'

// Dynamic typing in custom store
const MyStore = $hyoo_crowd_dict.of({
  val: $hyoo_crowd_union.of({
    count: $hyoo_crowd_numb,
    bool: $hyoo_crowd_reg,
    numb: $hyoo_crowd_reg,
    str: $hyoo_crowd_reg,
    seq: $hyoo_crowd_list
  })
});

// Normal store creation
const base = MyStore.make();

// Make independent forks for testng
const alice = base.fork(1);
const bob = base.fork(2);
const carol = base.fork(3);

// Twice change register named "foo"
alice.for("foo").to("str").str = "A1";
alice.for("foo").to("str").str = "A2";

// Change register named "foo" then converts it to sequence and insert value
bob.for("foo").to("str").str = "B1";
bob.for("foo").to("seq").insert("B2").insert("B3");

// Serial insert to sequence named "foo"
carol.for("foo").to("seq").insert("C1").insert("C2");

// Make deltas
const alice_delta = alice.delta(base.clock);
const bob_delta = bob.delta(base.clock);
const carol_delta = carol.delta(base.clock);

// Cross merge all of them
alice.apply(bob_delta).apply(carol_delta);
bob.apply(alice_delta).apply(carol_delta);
carol.apply(bob_delta).apply(alice_delta);

// ["A2","C1","C2","B1","B2","B3"]
console.log(
  alice.for("foo").as("seq").items,
  bob.for("foo").as("seq").items,
  carol.for("foo").as("seq").items
);

Sandbox

Comparison of Libraries

	$hyoo_crowd	Automerge	YJS	delta-crdt
Approach	CROWD	CRDT	CRDT	CRDT
Garbage Collection	Doesn't required	Stores full history	Enabled by default	❓
Gzipped Bundle Size	4 KB	60 KB	23 KB	43 KB
Sequence: Push + Shift	2 µs	400 µs	50 µs
Text: Append + Crop	16 µs	1050 µs	72 µs

Benchmarks

Sequence: Push + Shift

Chrome 89

FireFox 86

Text: Append + Crop

Chrome 89

FireFox 86

crdt-benchmarks

Community

• Leave us a feedback in duscussions section.
• Support Hyoo Guild to drive open source to the future.