yjs / y-indexeddb

IndexedDB database adapter for Yjs

y-indexeddb

IndexedDB database provider for Yjs. Documentation

Use the IndexedDB database adapter to store your shared data persistently in the browser. The next time you join the session, your changes will still be there.

• Minimizes the amount of data exchanged between server and client
• Makes offline editing possible

Getting Started

You find the complete documentation published online: API documentation.

npm i --save y-indexeddb

const provider = new IndexeddbPersistence(docName, ydoc)

provider.on('synced', () => {
  console.log('content from the database is loaded')
})

API

provider = new IndexeddbPersistence(docName: string, ydoc: Y.Doc)

Create a y-indexeddb persistence provider. Specify docName as a unique string that identifies this document. In most cases, you want to use the same identifier that is used as the room-name in the connection provider.

provider.on('synced', function(idbPersistence: IndexeddbPersistence))

The "synced" event is fired when the connection to the database has been established and all available content has been loaded. The event is also fired when no content is available yet.

provider.set(key: any, value: any): Promise<any>

Set a custom property on the provider instance. You can use this to store relevant meta-information for the persisted document. However, the content will not be synced with other peers.

provider.get(key: any): Promise>any<

Retrieve a stored value.

provider.del(key: any): Promise>undefined<

Delete a stored value.

provider.destroy(): Promise

Close the connection to the database and stop syncing the document. This method is automatically called when the Yjs document is destroyed (e.g. ydoc.destroy()).

provider.clearData(): Promise

Destroy this database and remove the stored document and all related meta-information from the database.

License

Yjs is licensed under the MIT License.

[email protected]

y-indexeddb's Issues

reliable 'synced' or loaded event

Is your feature request related to a problem? Please describe.

The current synced event has the following description in the readme

The "synced" event is fired when the connection to the database has been established and all available content has been loaded. The event is also fired when no content is available yet.

Describe the solution you'd like
I would like to be able to have an event that is fired once the content is guaranteed to be available. It could be 'synced' or another event.

Knowing when a ydoc has been 'loaded' is important to state external to this plugin, an example use case is: handling empty docs. From my testing this is not currently possible.

Describe alternatives you've considered
I've considered a few different approaches (timeouts, update event from ydoc) but none seem to work 100%

Additional context
I've tried looking for similar issues or discussions related to this but couldn't find any. Perhaps this is due to an implementation detail that I'm not aware of since I'm not familiar with the inner workings of this plugin. Something similar to the state provided in the WS plugin would be ideal.

Some guidance or clarity on this subject would be greatly appreciated 🙏

CommonJS require doesn't work

This is not really a blocking issue, but for quick prototyping I often still use CommonJS instead of ESM imports. For all the other yjs modules I've used so far this works fine, but with this one Browserify is giving me the dreaded ParseError: 'import' and 'export' may appear only with 'sourceType: module'.

Can we use IPFS Connector?

Can we use IPFS Connector?

Blocked IDB event forces a page reload

[deleted]

Performance on `IndexeddbPersistence`

I'm investigating the benchmark between y-indexeddb and our @toverything/y-indexeddb (not published yet) because of Our application's requirements for the first screen speed.

I run some basic bench(not accurate), but it looks like the boost speed when creating multiple persistence is much slower than the package we maintained

create many persistence
[
  PerformanceMeasure {
    name: 'yjs',
    entryType: 'measure',
    startTime: 1138.3797089457512,
    duration: 3805.7039580345154,
    detail: null
  },
  PerformanceMeasure {
    name: 'toeverything',
    entryType: 'measure',
    startTime: 4944.275416970253,
    duration: 1129.9851250052452,
    detail: null
  }
]

async function yjs_create_persistence(n = 1e3) {
  for (let i = 0; i < n; i++) {
    const yDoc = new Y.Doc();
    const persistence = new IndexeddbPersistence('test', yDoc);
    await persistence.whenSynced;
    persistence.destroy();
  }
}
async function toeverything_create_provider(n = 1e3) {
  for (let i = 0; i < n; i++) {
    const yDoc = new Y.Doc();
    const provider = createIndexedDBProvider('test', yDoc);
    provider.connect();
    await provider.whenSynced;
    provider.disconnect();
  }
}

Source Code

• https://github.com/toeverything/AFFiNE/blob/master/packages/y-indexeddb/benchmark/index.ts
• https://github.com/toeverything/AFFiNE/blob/master/packages/y-indexeddb/src/index.ts

y-indexeddb does not set the correct origin on initial load

Describe the bug
On initial load using y-indexeddb, event.transaction.origin is null and event.transaction.local is true.

To Reproduce
Steps to reproduce the behavior:

1. Create a basic example that uses y-indexeddb provider and ymap.observe(event => { console.log(event) }) to see the event on initial load.

Expected behavior
The expected behavior is that event.transaction.origin should be the provider and event.transaction.local is set to false (because it is coming from a provider? not sure about the expected behavior here)

Environment Information

• Chrome
• yjs v13.4.9, y-indexeddb v9.0.6

duplicated update for multiple IndexeddbPersistence and single Y.Doc

Upstream: toeverything/AFFiNE#1651

In our project case, we will create multiple IndexeddbPersistence for one Y.Doc. And we realized that it will apply duplicate data into indexeddb because this line

y-indexeddb/src/y-indexeddb.js

Line 101 in 217fd6d

if (this.db && origin !== this) {

Filter by transaction-origin instead of mutexes

Note to myself

ClearData does not return a Promise

I was purging an ever-growing IndexedDB database occasionally - trying to use await indexeddbProvider.clearData(); as the documentation says provider.clearData(): Promise.

Typescript rightfully reports:

'await' has no effect on the type of this expression.ts(80007)

/**
   * Destroys this instance and removes all data from indexeddb.
   */
  clearData () {
    this.destroy().then(() => {
      idb.deleteDB(this.name)
    })
  }

Maybe this would fix it:

/**
   * Destroys this instance and removes all data from indexeddb.
   */
  clearData () {
    return this.destroy().then(() => {
      idb.deleteDB(this.name)
    })
  }

The specified object store was not found

Hello there,

Tried using the IndexDB adaptor after using y-memory for some time and this error occurred.

Uncaught NotFoundError: Failed to execute 'objectStore' on 'IDBTransaction': The specified object store was not found.

Store @ y-indexeddb.es6:9
SmallLookupBuffer @ y.js:2899
Transaction @ y-indexeddb.es6:98
handleTransactions @ y-indexeddb.es6:250
request.onsuccess @ y-indexeddb.es6:279

It is initiated as follows:

Y({
    db: {
        name: 'indexeddb'
    },
    connector: {
        name: 'websockets-client',
        room: 'y-' + ID,
        socket: io,
        type: 'n'
    },
    sourceDir: '/javascripts/Plugins/yjs',
    share: {
        richtext: 'Richtext',
        map: 'Map'
    }
}).then(function (y) {
    doStuff()
});

I checked the repo to see if I should be doing something in preparation but couldn't see anything. Let me know if you require more information.

Chrome 50.0
OSX 10.11.3

indexdb not defined

Please save me some time and use the following template. In 90% of all issues I can't reproduce the problem because I don't know what exactly you are doing, in which environment, or which y-* version is responsible. Just use the following template even if you think the problem is obvious.

Checklist

• Are you reporting a bug? Use github issues for bug reports and feature requests. For general questions, please use https://discuss.yjs.dev/
• Try to report your issue in the correct repository. Yjs consists of many modules. When in doubt, report it to https://github.com/yjs/yjs/issues/

Describe the bug
Thanks for providing this, but when trying to install this, I get the error 'indexdb not defined'.
Maybe the documentation/readme is outdated, as whenSynced also seems not available anymore.

To Reproduce
Steps to reproduce the behavior:

1. install latest version
2. Follow readme
3. check console

Environment Information

• Chrome, Firefox, Node.js

Additional context

  System:
    OS: macOS 10.15.7
    CPU: (6) x64 Intel(R) Core(TM) i5-8500 CPU @ 3.00GHz
    Memory: 2.87 GB / 40.00 GB
  Binaries:
    Node: 20.11.0 - /usr/local/bin/node
    Yarn: 1.22.17 - /usr/local/bin/yarn
    npm: 7.10.0 - ~/node_modules/.bin/npm
  npmPackages:
    @playwright/test: ^1.41.2 => 1.41.2 
    @sinclair/typebox: ^0.32.14 => 0.32.14 
    @sveltejs/adapter-auto: ^3.1.1 => 3.1.1 
    @sveltejs/adapter-static: ^3.0.1 => 3.0.1 
    @sveltejs/kit: ^2.5.0 => 2.5.0 
    @sveltejs/vite-plugin-svelte: ^3.0.2 => 3.0.2 
    @types/eslint: 8.56.0 => 8.56.0 
    @typescript-eslint/eslint-plugin: ^6.21.0 => 6.21.0 
    @typescript-eslint/parser: ^6.21.0 => 6.21.0 
    autoprefixer: ^10.4.16 => 10.4.16 
    bits-ui: ^0.18.1 => 0.18.1 
    clsx: ^2.1.0 => 2.1.0 
    eslint: ^8.56.0 => 8.56.0 
    eslint-config-prettier: ^9.1.0 => 9.1.0 
    eslint-plugin-svelte: 2.36.0-next.5 => 2.36.0-next.5 
    formsnap: ^0.5.0 => 0.5.0 
    lucide-svelte: ^0.336.0 => 0.336.0 
    mode-watcher: ^0.2.1 => 0.2.1 
    postcss: ^8.4.32 => 8.4.35 
    postcss-load-config: ^5.0.2 => 5.0.2 
    prettier: ^3.2.5 => 3.2.5 
    prettier-plugin-svelte: ^3.2.1 => 3.2.1 
    prettier-plugin-tailwindcss: ^0.5.9 => 0.5.9 
    sass: ^1.71.1 => 1.71.1 
    svelte: 5.0.0-next.69 => 5.0.0-next.69 
    svelte-check: ^3.6.4 => 3.6.4 
    sveltekit-superforms: ^2.5.0 => 2.5.0 
    tailwind-merge: ^2.2.1 => 2.2.1 
    tailwind-variants: ^0.2.0 => 0.2.0 
    tailwindcss: ^3.3.6 => 3.3.6 
    tslib: ^2.6.2 => 2.6.2 
    typescript: ^5.3.3 => 5.3.3 
    vaul-svelte: ^0.2.3 => 0.2.3 
    vite: ^5.1.1 => 5.1.1 
    vitest: ^1.2.2 => 1.2.2 
    y-indexeddb: ^9.0.12 => 9.0.12 
    yjs: ^13.6.14 => 13.6.14 
    zod: ^3.22.4 => 3.22.4 

Remove memory buffer

Shouldn't be necessary anymore since yjs@11 (contents upgrade)

Syncing problems

Hello,

We've got some problems with resyncing the local indexDB storage with the server. When we turn off the network connection and after a few seconds reconnect, the client shows the local offline generated text, but don't (re)sync the data to the other clients.

Also after CMD + SHIFT + R the data shows in the reconnected client, so that part works well.

See our prototype files too see whats happening.

Thanks Dennis

y-test.zip

Doc grows without any changes

Checklist

• Are you reporting a bug? Use github issues for bug reports and feature requests. For general questions, please use https://discuss.yjs.dev/
• Try to report your issue in the correct repository. Yjs consists of many modules. When in doubt, report it to https://github.com/yjs/yjs/issues/

Describe the bug

The size of the object store grows every time the page is refreshed. This occurs because the doc is encoded as an update and appended to the object store every time IndexeddbPersistence is instantiated.

The db is compacted after reaching the PREFERRED_TRIM_SIZE whenever an update is stored. However, this assumes 1) a small number of Docs, and 2) Docs will be modified. While reasonable on the surface, neither of these assumptions hold in all cases. I currently instantiate thousands of Docs for an offline-first graph database built on YJS. Many hundreds are instantiated and destroyed dynamically as the user navigates the graph, and some may never be modified.

To Reproduce

Minimal example:

https://codesandbox.io/p/sandbox/pedantic-euler-jje92f?file=%2Fsrc%2FApp.tsx%3A1%2C1-2%2C1

Notice that no modifications are made to the Doc, yet it grows in size.

Expected behavior

The persisted Doc should not grow indefinitely when no changes are made.

Environment Information

• Browser: Chrome
• Yjs: v13.6.2
• y-indexeddb: v9.0.11

Additional context

I can see that the behavior was introduced in this commit.

One possible solution is adding the condition if (updates.length === 0) to beforeApplyUpdatesCallback, as it seems to only be needed on a new Doc. That stops the infinite growth, and the tests pass, however I don't know the full implications of this change. If beforeApplyUpdatesCallback needs to be invoked in other cases that are not covered by the tests that would be important to know.

Is there a condition that detects when beforeApplyUpdatesCallback does not need to be called? I have tried skipping it when the Doc is empty, but that does not work.

Failed to execute 'put' on 'IDBObjectStore': The parameter is not a valid key

Describe the bug
IndexeddbPersistence.set() method has problems.

Checkout this line of code

y-indexeddb/src/y-indexeddb.js

Line 152 in 2d21bdf

return idb.put(custom, key, value)

and then check this definition of put method from lib0:

export const put = (store, item, key) =>
  rtop(store.put(item, key))

https://github.com/dmonad/lib0/blob/8a89d8d369c9dccd1187a75c04232e2a1ce39b7b/indexeddb.js#L126

The lib0 is expecting item followed by key as parameters, while the set call in this package is sending key first and then value next, which is leading to the error:

Failed to execute 'put' on 'IDBObjectStore': The parameter is not a valid key

array insert happens twice on receiving end

Steps to reproduce : Set up a Y object in two browsers, using the following configuration, and open the page twice, with two windows of the same browser.

Y({
  db: {
    name: 'indexeddb'
  },
  connector: {
    name: 'websockets-client',
    room: 'test'
  },
  share: {
    state : 'Map'
  }
}).then((y) => { window.y = y })

In window A, type var a = y.set('list', Y.Array);a.push([1])

Now, in window B, inspect the result with console.log(y.get('list').toArray(). The result is [ 1, 1 ], rather than the expected [1]. This doesn't occur when the receiving window is in a separate browser, or when the two parts of setting the list var a = y.set('list', Y.Array) and populating it a.push([1]) are executed with a pause inbetween.

Failed to execute 'transaction' on 'IDBDatabase': One of the specified object stores was not found.

Hello!
I am on version 9.0.11 of y-indexeddb and am getting this error occasionally NotFoundError: Failed to execute 'transaction' on 'IDBDatabase': One of the specified object stores was not found. in my monitoring.

Looking at the stack trace, the issue starts in the constructor of IndexeddbPersistence where fetchUpdates is called https://github.com/yjs/y-indexeddb/blob/master/src/y-indexeddb.js#L93
fetchUpdates then calls idb.transact with the updates store name to retrieve the "updates" store https://github.com/yjs/y-indexeddb/blob/master/src/y-indexeddb.js#L17
idb.transact then throws the error because the updates store cannot be found.

It seems there may be an issue where it tries to fetch the updates store before the db is fully initialised. i.e. before this code is run https://github.com/yjs/y-indexeddb/blob/master/src/y-indexeddb.js#L72 which creates the updates store. (Perhaps some kind of race condition?)

Steps to reproduce the behavior:
I have been unable to reproduce this, I am just seeing error logs coming through my monitoring from other users.

Expected behavior
The updates store should be found

Environment Information

• Browser: the error has been reported from Chrome 114.0.0, Chrome 115.0.0 and Edge 114.0.1823
• yjs v13.5.50, y-indexeddb v9.011,

Extra info:
Here is how I am using Indexeddb
const indexedDb = new IndexeddbPersistence("test name", new Y.Doc());
indexedDb?.on?.('synced', () => { // send performance mark for synced db });

I am hoping you can help!

provider.destroy() may cause race condition

Checklist

• Are you reporting a bug? Use github issues for bug reports and feature requests. For general questions, please use https://discuss.yjs.dev/
• Try to report your issue in the correct repository. Yjs consists of many modules. When in doubt, report it to https://github.com/yjs/yjs/issues/

Describe the bug

Calling indexdbProvider.destroy() before sync-Event still triggers sync (async race condition).

To Reproduce
To reproduce the behavior, run the following code:

const ydoc = new Y.Doc();
const indexDBProvider = new IndexeddbPersistence("somedoc", ydoc);
indexDBProvider.on("synced", () => {
  console.log("should not be called");
});
console.log("destroy");
indexDBProvider.destroy();

Expected behavior

Calling indexdbProvider.destroy() before sync event never triggers sync.

Environment Information

• All Envs (Node, Browser)
• "y-indexeddb": "^9.0.8", (all versions afaik)
• "yjs": "^13.5.39"

Additional context

I am trying to create react 18 bindings. When using react 18 in strict mode effects may be called twice. (More Info why react 18 is doing this)

useEffect(() => {
  const ydoc = new Y.Doc();

  new IndexeddbPersistence("docName", ydoc);

  return () => {
    ydoc.destroy();
  };
}, []);

Since the provider->sync is still being triggered, all entries inside the doc will be duplicated.

Under the hood the following happens here:

const ydoc = new Y.Doc();
(new IndexeddbPersistence("somedoc", ydoc)).destroy()
new IndexeddbPersistence("somedoc", ydoc)

Afaik: A similar issue applies to y-webrtc.

Possible Solutions:

    this.whenSynced = this._db.then(db => {
+     if (this._isDestroyed) return this
      this.db = db
      const currState = Y.encodeStateAsUpdate(doc)
      return fetchUpdates(this).then(updatesStore => idb.addAutoKey(updatesStore, currState)).then(() => {
        this.emit('synced', [this])
        this.synced = true
        return this
      })
    })
    
    ...
    
  destroy () {
    if (this._storeTimeoutId) {
      clearTimeout(this._storeTimeoutId)
    }
    this.doc.off('update', this._storeUpdate)
    this.doc.off('destroy', this.destroy)
+   this._isDestroyed = true
    return this._db.then(db => {
      db.close()
    })
  }

---

@dmonad I really love you work 🙏! Thank you so much providing such a cool package. If you are willing to change this behaviour I am willing to create a PR.

init ydoc from indexeddb failed

Describe the bug
I'm using this project to add offline support for our preject, everything looks fine, but sometimes the initialization from local indexeddb will fail, it looks like that the initial gc process failed, but I'm not sure, and I haven't look deep into the codebase, any information will be very appreciated.

To Reproduce
it happens occasionally

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots

Environment Information

• Browser Chrome
• y-indexeddb9.0.7, [email protected]

A way to know if an update has been stored (feedback)

Checklist

[x] Are you reporting a bug? Use github issues for bug reports and feature requests. For general questions, please use https://discuss.yjs.dev/
[x] Try to report your issue in the correct repository. Yjs consists of many modules. When in doubt, report it to https://github.com/yjs/yjs/issues/

Is your feature request related to a problem? Please describe.
Apparently not all updates are guaranteed to be stored (is it itself a bug? This needs to be documented at least). For example, if you do an update and close/reload the tab immediately, it won't show up on the next load.

There's no conventional way to know in what state the document has been stored in indexedDB. Has the update that I just made been stored? Is it safe to close the tab now?

Describe the solution you'd like
Expose some getter, or method, or add an event (or all at the same time), that lets you determine if the document is stored in indexedDB in the same state as it is in JS right now.

Also I guess this should apply to other persistence adapters (leveldb).

Describe alternatives you've considered
Perhaps additionally a method that upon calling stores the latest update

Additional context
My case:
I'm developing an app that builds the state from a list of updates (like this one). The list is an external thing that persists between page reloads. For performance, I want to store the state of the Y.Doc in browser storage in order to not rebuild it from the list of updates each time. For this I need to remember the index of the last update in the list that has been cached in indexedDB. But I don't know the index because it's not guaranteed that every update made on the doc with the y-indexeddb adapter attached will be saved.

sub-doc syncing support

Checklist

• Support sub doc syncing

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.