local-first-web / state Goto Github PK

Possibilities:

• Go to the next row, first column if it exists (current, default)
• Create a new column
• Create a new row and go to the first column

I'm thinking of the grid data generator, which completes pretty quickly but then still has a lot of work to do in the background. In general, keep track of a queue of tasks to complete - whether they're changes that have been dispatched, or messages to process from peers; and add information to state about whether we're busy or not, and how much is left to do.

We're currently using my fork of idb.

https://github.com/DevResults/cevitxe/blob/8929b0347ef6d477366cfd1fcfb1ca22dc6fa509/packages/cevitxe/package.json#L29

My fork just adds a file called with-async-ittr-cjs.d.ts that is a copy of the existing with-async-ittr-es.d.ts. We then import that file in the IndexedDb adapter.

https://github.com/DevResults/cevitxe/blob/8929b0347ef6d477366cfd1fcfb1ca22dc6fa509/packages/cevitxe-storage-indexeddb/index.ts#L3

IIRC everything works fine with the es version of that file, except Jest is unhappy.

https://github.com/DevResults/cevitxe/blob/f1626ccf91fca32a1c89a7cc20c8ef6d56b5f93a/packages/core/src/Repo.ts#L108-L116

https://github.com/DevResults/cevitxe/blob/c8f9e8c6676987c5fed42131fd7115a22c9fd12e/packages/core/src/StoreManager.ts#L97-L102

This is obviously the right thing to do, but it breaks tests. For some reason RepoSync
continues to respond to messages from the peer after the repo is closed, and then tries to
access the closed database.

We've manually gotten our signal server up and running on Heroku. We'd like to script this and/or include configuration files to make it possible to deploy instances of the signal server to multiple providers such as Heroku, Zeit Now, and Netlify.

I observed this with the known document IDs dropdown on the grid example.

1. Run yarn dev:grid

2. Generate 10 rows

3. Notice the order of the rows

   

4. Reload

5. Notice that the order has changed

   

On subsequent reloads, the sort order won't change any further. I'm guessing the initial sort order reflects the order in which items were added, whereas the subsequent sort order is maybe the lexical order of the UUIDs?

Probably the same issue:

1. Generate 10 rows

2. Add 5 rows at the bottom containing the numbers 1 through 5. The rows are where you put them.

   

3. Reload. The numbered rows are now randomly scattered among the other rows.

   

so we don't have to iterate each document when returning to a large repo

We currently only provide browser-based persistent storage using IndexedDb. A superpeer running in a serverless function would need access to a third-party hosted datastore like Redis, MongoDB or DynamoDB. An installable Electron app might prefer to work with self-contained, portable documents in the file system; or with a database like Sqlite.

• IndexedDb
• Sqlite
• File system
• Redis
• MongoDB
• AWS DynamoDB

Hypercore provided this for us, needs to be reimplemented in Repo. We already have the infrastructure in place for storing keypairs associated with a discovery key:

https://github.com/DevResults/cevitxe/blob/f1626ccf91fca32a1c89a7cc20c8ef6d56b5f93a/packages/core/src/keys.ts#L34-L51

so should just be a matter of encrypting/decrypting just before & after db write/read, respectively.

https://github.com/DevResults/cevitxe/blob/f1626ccf91fca32a1c89a7cc20c8ef6d56b5f93a/packages/core/src/Repo.ts#L186-L196

https://github.com/DevResults/cevitxe/blob/f1626ccf91fca32a1c89a7cc20c8ef6d56b5f93a/packages/core/src/RepoSync.ts#L24-L40

Store collections as multiple Automerge documents: One per item, plus one for an index.

I'm really interested in this one, because I think the idea of having a local file - that you could email to someone, or put on a USB drive, or whatever - will be comforting to some potential users.

e.g.

• Empty
• Contact list
• Different sized datasets for load testing (M, L, XL)

Possible behaviors:
Just go down, add row if needed (kinda-default, current)
Go to the first column (in addition to the above)
Guess the right column to go to (true Excel-like behavior)

randomly select a URL if more than one is provided? select best based on ping?

https://github.com/DevResults/cevitxe/blob/c8f9e8c6676987c5fed42131fd7115a22c9fd12e/packages/core/src/Client.ts#L23-L28

This comment was generated by todo based on a TODO comment in c8f9e8c in #26. cc @DevResults.

Originally posted by @todo in #26 (comment)

From #49 (comment) :

we need some additional markers on the provided state to indicate which things should be persisted/synced to the remote instances. For instance, you might want to exclude the ui section of the state in this example from the redux docs.

A potential solution could look like the whitelist/blacklist config used in redux-persist.

Something to think about, I think we'll want to allow the flexibility of consumers putting things in their redux store but not being transmitted to all replicas (e.g. UI filter selections, etc.).

waiting on #26

Would be nice to have some indication in the toolbar of whether or not we're connected to a signal server.

Add core.js import

for large datasets, send in batches

https://github.com/DevResults/cevitxe/blob/c8f9e8c6676987c5fed42131fd7115a22c9fd12e/packages/core/src/Repo.ts#L288-L293

This comment was generated by todo based on a TODO comment in c8f9e8c in #26. cc @DevResults.

Originally posted by @todo in #26 (comment)

In a peer-to-peer network where clients are only online intermittently and updates can only happen when two peers happen to be online at the same time, it can take a while for updates to propagate. One solution would be to leave a client running on a computer that's never turned off. We'd like to provide a "superpeer" (don't call it a server!!) that can be inexpensively deployed using the command line to cloud hosting providers.

Currently the StoreManager constructor takes an array of middlewares.

export const storeManager = new StoreManager({
  databaseName: 'todo',
  proxyReducer,
  initialState,
  urls,
  middlewares: [logger],
})

To maintain consistency with the Redux createStore interface, this should accept a StoreEnhacer:

export const storeManager = new StoreManager({
  databaseName: 'todo',
  proxyReducer,
  initialState,
  urls,
  middlewares: applyMiddleware([logger]),
})

I think the difficulty had to do with composing an existing enhancer with the Redux Dev Tools browser extension here:

https://github.com/DevResults/cevitxe/blob/dffe4ac90008f0777e16fb6170d2d1b88f0c0ef2/packages/core/src/StoreManager.ts#L72-L79

We haven't really used the Redux Dev Tools, so we could just not wire it up. Or we could figure out how to make it work. See http://github.com/zalmoxisus/redux-devtools-extension#1-with-redux

To reproduce

1. Run the todo example yarn dev:todo
2. Create two tasks, 'a' and 'b' and mark them both complete.

3. Press 'Clear completed.' Task list will be empty.

4. Reload. One of the tasks is still there.

5. Add a task 'c'. After a moment the app crashes:

After generating or uploading data, adjust column widths to fit data

We don't have any way for DocSetSync to communicate to peers that DocSet.removeDoc has been called. For now we're just removing the removed doc's ID from the collection index so it's effectively ignored but this doesn't help reduce the size of the state and it probably doesn't work at all for non-collection documents.

I suspect we might need to update DocSet.removeDoc to call the handlers with undefined as the new doc so they can appropriately handle the removal of the document. We might get more clarity on this if we hear anything back on my automerge PR: #210

It should keep the alignment and font (and hopefully not move things by a pixel or so)

In the Todo example, our entire state fits easily in a single document, and the Repo API feels really cumbersome.

• You have to put your state in a root-level document: instead of https://github.com/DevResults/cevitxe/blob/b34789110b10bcc1c83d4952b9a9a360550a64df/packages/examples/todo/src/redux/store.ts#L6-L10 you have to do this https://github.com/DevResults/cevitxe/blob/117ca3b0161dfcbe1a55849242afcac20b7ea1f0/packages/examples/todo/src/redux/store.ts#L6-L12

• Reducers have to return a dictionary of change functions that work on individual documents, instead of a single function that works on the entire repo state: this https://github.com/DevResults/cevitxe/blob/117ca3b0161dfcbe1a55849242afcac20b7ea1f0/packages/examples/todo/src/redux/reducers.ts#L11-L19 instead of this https://github.com/DevResults/cevitxe/blob/b34789110b10bcc1c83d4952b9a9a360550a64df/packages/examples/todo/src/redux/reducers.ts#L18-L24

• Same deal with selectors and anything else that cares about the shape of the overall state: You have to file everything inside this bogus root property.

Do we care? If so, I wonder how we can simplify this use case?

One idea I had was to accept reducers that return a single function instead of a dictionary of functions. When initializing the StoreManager, we could then check the shape of the proxyreducer we're passed, and if it returns single functions then go into "single-document mode" where we take care of putting things in the root document, and just exposing that document as the store's state, so the developer can deal with this as if there was just one document.

To reproduce:

• Run yarn dev:grid
• Visit localhost:3000
• You should be able to edit the grid normally
• Press NEW
• When you edit the grid now, changes aren't visible because they are going to the previous store
• Reload
• Now you should be able to edit the grid normally again

For now, besides introducing two peers to each other, the signal server also serves as a relay, piping their two WebSocket connections together. In the future, Cevitxe will establish a direct connection between the two peers, using browser-to-browser communication technologies like WebRTC. For now, we've found WebRTC too unreliable to be worth the trouble.

Loading from storage currently ties up the UI while it's reading and rehydrating; this should be done asynchronously.

To reproduce:

• Run dev:grid
• Create a new document
• Generate 10K rows
• Reload the browser

The window remains blank until the stored feed entries have been read, parsed, and applied. I think at some point we're awaiting when we should just be firing an async action?

Enable running Cevitxe's StoreManager entirely in a web worker process. See HerbCaudill/react-redux-worker for a starting point.

All the Automerge, local storage and sync components would live in the worker, thus freeing the UI context of all the expensive overhead of the system.

We'll need a coarse-grained query API to query and retrieve a view of a set of rows, aggregate information, etc. In order of priority:

• order + range / count
• filter
• aggregation, projection, etc. (might be entirely out of scope)