multi-node about nstore

[creationix]

No, nStore only allows a single process accessing a single db file at once. You can have a master process and cause it to make it's services available over network or some other IPC.

[creationix]

Though, this isn't a bad idea, it would just require a change in how the program is architected.

[dvv]

Couldn't you outline the needed changes?

[creationix]

I don't know of a production database in the world that can have multiple processes accessing and modifying the same data file on disk safely without killing performance. sqlite sortof supports this, but it's highly discouraged. I still think the best solution is to have nStore be a single dedicated process and have it talk to the other processes via sockets.

If your reason for wanting this is to have the dedicated nStore core have multiple processes itself, then you would have to implement locks so that two processes can't write at the same time to the file. Also, the other processes would need to know somehow when a new record was added to the file on disk. I guess if there were no in-memory caches, then every read could start with the end of the file and work backwards.

I'm not sure the overhead of all this syncing and IPC would outweigh the benefits of having more CPU power available.

Better would be to shard your data into separate nStore processes and have those all available via sockets to the multiple processes needing the data.

The largest bottleneck (other than drive seek time) is serialization and deserialization of data. The more processes you have, the more data to serialize and pass around. and the more CPU you need.

[dvv]

I see. I came to the same conclusions. Sharding...

Thanks for extended answer.