y-crdt / yrb-actioncable

An ActionCable companion for Y.js clients.

Y CRDT

A collection of Rust libraries oriented around implementing Yjs algorithm and protocol with cross-language and cross-platform support in mind. It aims to maintain behavior and binary protocol compatibility with Yjs, therefore projects using Yjs/Yrs should be able to interoperate with each other.

Project organization:

• lib0 is a serialization library used for efficient (and fairly fast) data exchange.
• yrs (read: wires) is a core Rust library, a foundation stone for other projects.
• yffi (read: wifi) is a wrapper around yrs used to provide a native C foreign function interface. See also: C header file.
• ywasm is a wrapper around yrs that targets WebAssembly and JavaScript API.

Other projects using yrs:

• ypy - Python bindings.
• yrb - Ruby bindings.

Feature parity among projects

	yjs (13.6)	yrs (0.18)	ywasm (0.18)	yffi (0.18)	y-rb (0.5)	y-py (0.6)	ydotnet (0.4)	yswift (0.2)
YText: insert/delete	✅	✅	✅	✅	✅	✅	✅	✅
YText: formatting attributes and deltas	✅	✅	✅	✅	✅	✅	✅	✅
YText: embeded elements	✅	✅	✅	✅	✅	✅	✅	✅
YMap: update/delete	✅	✅	✅	✅	✅	✅	✅	✅
YMap: weak links	✅  (weak-links branch)	✅	✅	✅	❌	❌	❌	❌
YArray: insert/delete	✅	✅	✅	✅	✅	✅	✅	✅
YArray & YText quotations	✅  (weak links branch)	✅	✅	✅	❌	❌	❌	❌
YArray: move	✅  (move branch)	✅	✅	✅	❌	✅	✅	❌
XML Element, Fragment and Text	✅	✅	✅	✅	✅	✅	✅	❌
Sub-documents	✅	✅	✅	✅	❌	❌	✅	❌
Shared collections: observers	✅	✅	✅  (incompatible with yjs)	✅	✅	✅	✅	✅
Shared collections: recursive nesting	✅	✅	✅	✅	❌	❌	❌	❌
Document observers	✅	✅	✅  (incompatible with yjs)	✅	✅	✅	✅
Transaction: origins	✅	✅	✅	✅	❌	❌	✅	✅
Snapshots	✅	✅	✅	✅	❌	❌	❌	❌
Sticky indexes	✅	✅	✅	✅	❌	❌	✅	❌
Undo Manager	✅	✅	✅	✅	❌	❌	✅	✅
Awareness	✅	✅	❌	❌	✅	❌	✅	❌
Network provider: WebSockets	✅  (y-websocket)	✅  (yrs-warp)	❌	❌	✅  (y-rb_actioncable)	✅  (ypy-websocket)	✅	❌
Network provider: WebRTC	✅  (y-webrtc)	✅  (yrs-webrtc)	❌	❌	❌	❌	❌	❌

Maintainers

• Bartosz Sypytkowski
• Kevin Jahns
• John Waidhofer

Sponsors

yrb-actioncable's Issues

Y::Lib0::Decoding.read_var_uint8_array should increment decoder's `pos`

Problem

I encountered an issue while using Y::Lib0::Decoding.read_var_uint8_array, where pos does not change, preventing successful sequential reading of strings.

I suspect the solution might involve adapting and fixing the part decoder.pos += len from https://github.com/dmonad/lib0/blob/616c7cc61c0bc03d1419bb347977259673ca8aa3/decoding.js#L104.

js:

https://github.com/dmonad/lib0/blob/616c7cc61c0bc03d1419bb347977259673ca8aa3/decoding.js#L104

export const readUint8Array = (decoder, len) => {
  const view = new Uint8Array(decoder.arr.buffer, decoder.pos + decoder.arr.byteOffset, len)
  decoder.pos += len
  return view
}

ruby:

https://github.com/y-crdt/yrb-actioncable/blob/358efdce316042d4261ced4caaed29e8a4b85252/gems/yrb-actioncable/lib/y/lib0/decoding.rb#L29-L31

      def self.read_uint8_array(decoder, size)
        view = Buffer.create_uint8_array_view_from_buffer(decoder.arr, decoder.pos + 0, size)
      end

---

demo

doc = Y::Doc.new
text = doc.get_text("text")
text << "content"
p doc.diff
# => [1, 1, 219, 249, 254, 144, 12, 0, 4, 1, 4, 116, 101, 120, 116, 7, 99, 111, 110, 116, 101, 110, 116, 0]

decoder = Y::Lib0::Decoding::Decoder.new(doc.diff)

Y::Lib0::Decoding.read_var_uint(decoder) #=> 1 (numOfStateUpdates)
Y::Lib0::Decoding.read_var_uint(decoder) #=> 1 (numberOfStructs)
Y::Lib0::Decoding.read_var_uint(decoder) #=> 3256859867 (client)
Y::Lib0::Decoding.read_var_uint(decoder) #=> 0 (clock)
Y::Lib0::Decoding.read_var_uint(decoder) #=> 4 (info(String))
Y::Lib0::Decoding.read_var_uint(decoder) #=> 1 (parent info)

# first, read "text"
array = Y::Lib0::Decoding.read_var_uint8_array(decoder)

p array
#=> [116, 101, 120, 116]
p array.pack("C*")
#=> "text"



# **Problem**
# **Now, we should increment `pos` manually**
decoder.pos += array.size



# Now, we can read next content  "content"
array = Y::Lib0::Decoding.read_var_uint8_array(decoder)

p array
#=> [99, 111, 110, 116, 101, 110, 116]
p array.pack("C*")
#=> "content"

When will v0.1.6 or patch version be released?

@eliias

First of all, thank you for the great gem.

We are trying to use this gem in a closed beta product. We will actually offer this functionality to our users on a trial basis, although the number of users using it is small.

So I have a request. I would like you to release a new version (0.1.6 or patch version) including the following commits Without this change, we will get errors in name resolution on CI.
#41

yrb-actioncable sync awareness

The awareness development is only left as a TODO, so it is impossible to track when the client side disconnects from the socket. I was wondering if you are aware of this issue and if so, are you planning to develop it, and if not, can I raise a pull request for it?

How to get editor full text value?

We are currently storing document data with Redis as the destination, using the EXAMPLE example as a guide.

Is there any way to extract only the input string information from the binary data exchanged by y.js?

I am planning to store the data in mysql as well as Redis, using batches and such. However, the data used in this library is binary information that follows the y.js protocol, so I would like to handle it in a simple markdown-like format.

Also, I'm assuming that when I return values from mysql to editor, I need to re-convert them to a form that follows the y.js protocol. If you know how to do that, please let me know.

Add ReliableWebsocketProvider to JavaScript module

Problem

A reliable channel ensures at-least-once guarantees via acknolwedge messages sent by both, client and server. Therefore, the client, and the server must reliably store messages that weren't acknowledged by all recipients yet. In case of the client, this means the server must respond with a last-ack-id response when messages are sent, and the client must always send the full message queue to workaround missing “in-between” updates.

Implementation

The default WebsocketProvider does not support message buffers and acknowledgment of messages. The ReliableWebsocketProvider is therefore a drop-in replacement that takes on the additional responsibilities of transparently managing a send buffer, and acknowledging message retrieval.

Due to Y.js supporting staging of messages (messages are not integrated until all necessary messages are available), and applying updates being idempotent, this isn't an issue.

sequenceDiagram
Client->>+Server: Send message buffer to server
alt no response
  loop Retry (w/ exponential backoff)
        Client-->>Server: Send message buffer
  end
else
  Server-->>-Client: Acknowledge message retrieval
end

Caveats

We aim for eventual consistency (at-least-once delivery), and similar transport latency as with the regular WebsocketProvider on the happy path (exactly-once delivery). There is a chance of transmitting messages twice, when an “older” message wasn't transmitted, but the client is producing “new” messages.

How to load initial `@editor_content` in `PagesController#index`

Hi, thanks for making this gem!

I'm working on a collaborative IDE of sorts and I'm running into a problem loading initial content from the server.

I'm proactively checking in pages#index if content exists in redis (in the example, it just returns if full_state nil), and if it's not present, I set it, retrieve it and return it – but that hasn't worked. I noticed two issues.

1. The session name in the controller has to match the name of the sync channel session in order to retrieve the content. Right now, as the example is, it won't ever match as the session name in the index controller will be something like sessions:id where id is the path, and the id of the session for the sync channel is something like sync:channel-syncchannel:id-hello2 so the index controller will always have full_state as nil and @editor_content will always be "" – is this supposed to be the case? If so why?
2. Is there a way to use SyncChannel from the PagesController to set initial content? Is that a best practice?

Thanks so much for your help! 🙏

Add receive/transmit hooks

Problem

To allow the extension of the Sync module with e.g., Reliable messaging, there need to be defined hooks/callbacks. We can use them to transform behavior (e.g., store sent messages in buffer, move client tracker, send acknowledge messages to client, etc.).

Implementation

• Add ActionCable like callbacks to Sync (like subscribed, unsubscribed).
• Add a chain of responsibility implementation for extensions (e.g., Reliable for Sync)

Multiple editors on one page

Hi - thank you for sharing this code.

I have multiple editors on one page.

If _proto.connectBc = function connectBc() runs, then they both end up broadcasting to each other, so they end up identical. But I want two separate ydocs with different contents.

I think that because it's only using the channel name inside that function, so it can't differentiate between different paths in the publish call.

Not sure - I don't really understand any of this. However, changing this fixed my issue.

Support non Redis adapter in Y::Actioncable

The only adapter supported is Redis, but we should have a configurable test adapter like Rails ActionCable itself, to make it easier to test sync (and reliable sync), and also reduce a test dependency on a running Redis instance.

SSR fails with LocalStorage issue from BroadcastChannel

When Node.js cannot be detected, but the app is server-side rendered, the following import statement fails due to initializing an instance of LocalStorage. We should introduce an SSR flag (mode) and re-implement the broadcastchannel with some lazy initialization for the LocalStorage, so that it will only be created when disableBc = false.

https://github.com/y-crdt/yrb-actioncable/blob/main/packages/yrb-actioncable/src/websocket-provider.ts#L2

import { publish, subscribe, unsubscribe } from 'lib0/broadcastchannel';

Messages are lost after a large amount of input

Thanks for developing a great gem.
And thanks for answering some of my previous questions.

We have been using this gem for the past few months as a beta version and have found cases where messages are lost under certain conditions.

First, some information about our environment:
Ruby on Rails: 7.0.4
Ruby: 3.2.1
Backend Data Source: Redis
Frontend Frame Work: Next.js (We use y-actioncable npm package in app.)

Our scripts. For debug, I put some loggers.

class DocumentChannel < ApplicationCable::Channel
  include Y::Actioncable::Sync

  def subscribed
    reject unless document

    sync_for(document) do |id, update|
      redis.save(id, update)
    end
  end

  def unsubscribed
    stop_stream_from canonical_channel_key
  end

  def receive(message)
    sync_to(document, message)
  end

  def doc
    @doc ||= load do |id|
      redis.load(id)
    end
  end

  def document
    @document ||= organization.documents.find(params[:document_id])
  end

  def redis
    @redis ||= Yrb::Editor::Redis.new
  end
end

module Yrb
  module Editor
    class Redis
      attr_reader :client, :logger

      def initialize
        @client ||= ::Redis.new(url: Rails.application.config.x.redis_url)
        @logger = SemanticLogger[self.class.to_s]
      end

      def save(id, update)
        begin
          data = encode(update)
          doc = client.set(id, data)
          logger.info("Yrb::Editor::Redis Finish Save", name: self.class.to_s, id:)
          doc
        rescue ::Redis::BaseError => e
          logger.error("Yrb::Editor::Redis Failed to Save", name: self.class.to_s, inspect: e.inspect, message: e.message)
        end
      end

      def load(id)
        begin
          data = client.get(id)
          if data.present?
            logger.info("Yrb::Editor::Redis Finish Load", name: self.class.to_s, id:)
            decode(data) unless data.nil?
          else
            logger.error("Yrb::Editor::Redis Not Found", name: self.class.to_s, id:)
            nil
          end
        rescue ::Redis::BaseError => e
          logger.error("Yrb::Editor::Redis Failed to Load", name: self.class.to_s, inspect: e.inspect, message: e.message)
        end
      end

      def encode(update)
        update.pack("C*")
      end

      def decode(data)
        data.unpack("C*") unless data.nil?
      end
    end
  end
end

---

Kapture.2023-09-04.at.15.32.41.mp4

In the video above, the message is saved correctly for normal input, but after a large amount of input, it is lost after the page is reloaded. (Lines of text beginning with "e" are missing.)

Could #25 be related to this issue?

If you want any information to understand the issue, please let us know.
Thank for reading.

Compatibility with import maps

I'm looking into using yrb-actioncable with my rails app, but am blocked on the requirement to use yarn… Is there a way to use this project with import maps, which is the standard way to load JavaScript in a rails app since 7.0?