serve multiple dat repositories from a single server about dat

[swizzard]

I think #2 shouldn't be too bad to implement with node's routing
infrastructure (+ maybe some rpc-type situation?) You could even store all
the repos in 1 directory to make it a little easier (so it's a call to
dat-repos/my-repo/somechecksum or whatever.)
On Feb 14, 2014 6:48 PM, "Kyle Corbitt" [email protected] wrote:

I'm working on an open source Github-for-dat project called DatRepohttps://github.com/kcorbitt/datrepo.
The idea is that users will be able to upload, download, search and share
datasets easily.

My question is whether there is a vision/roadmap for serving multiple dat
repositories from a single machine. I can think of at least three different
implementations.

1. The most trivial approach would be to run a separate instance of
   the dat server on a unique port for each repository. This has the big
   advantage of simplicity. However, for a server with hundreds or thousands
   of repositories I'm not sure that this will scale very well, despite node's
   light footprint.
2. A more complex solution would be to have a single dat server per
   machine, and calls to "dat serve" would simply register the current dat
   directory with the server in a unique namespace. So "dat serve --namespace
   my_data" could activate the routes "localhost:6461/my_data/_package", etc.
   This approach seems scalable and in the ethos of dat to me, and it's my
   preferred solution.
3. Alternatively, dat could mimic git's paradigm and not mount a
   server on the remote at all, instead just piggybacking on SSH and having
   the local dat client connect to the remote filesystem and synchronize that
   way. I don't think that this is a good idea.

Are any of these, or an alternative, likely to become part of the dat
protocol? Is there a reason why supporting multiple repositories on a
single machine is a bad idea?

Reply to this email directly or view it on GitHubhttps://github.com//issues/41
.

[corbt]

I also think 2 shouldn't be that bad. I'll take a look at it and try to get an implementation. @maxogden, if I work on this and create a pull request are you interested in using it?

[max-mapper]

I haven't quite decided what the best way to manage multiple dat databases is going to be... but I like the ideas proposed here and would love to see a third party sort of "multiple dat manager" command line tool or something along those lines as a way to prototype the idea.

There's no reason why we couldn't just store multiple leveldb folders inside the .dat folder and then make each one have an ID, I just didn't want to add that extra complexity so early on when the simple workaround is to just create another .dat repo. I'm scared that if I put it in prematurely then I'll end up having something horrible like git submodules.

[max-mapper]

oh yeah and re: the 3 options presented in the first comment I think 2 is the most interesting. E.g. something that can mount multiple dats in a higher level rest API (kind of like what couchdb does with it's _dbs API)

1 is implemented now, 3 is also a valid use case that I'd like to support but I also recognize that it's a little more advanced

[corbt]

Nice. I'll work on creating a lightweight module then that uses dat's API and builds a higher-level REST API for accessing multiple dat repositories with a single server. I'll also try to come up with a scheme for creating and administering new repositories over HTTP.

[max-mapper]

@kcorbitt awesome. It might require making this more pluggable from an external API https://github.com/maxogden/dat/blob/master/lib/commands.js#L179-L199, e.g. how this is a separate requireable file https://github.com/maxogden/dat/blob/master/lib/rest-handler.js

let me know if you need assistance there, feel free to ping me in #hyperdata on freenode for chat

[corbt]

Just an update: unfortunately as the semester ends at the uni I haven't had time to work on this, so I'll have to bow out for now. If I have more time over the summer I'll try to work on it.

[acbart]

Have there been any updates on this? It seems like a very important feature to be able to support many different datasets without spawning new instances.

[corbt]

I'm not currently working on this feature.

[sethvincent]

I started work on a module called multi-dat. It's a too-simple prototype, and uses approach 1 for now, but I'd be interested in working on approach 2. https://github.com/sethvincent/multi-dat

[junosuarez]

Can we call this Dat-a-plex? Possibly without the hyphens? 👻

[sethvincent]

also considered octodat

[okdistribute]

Just had a conversation today with @maxogden and @olgabot where @olgabot was interested in having multiple "datasets" (multiple 'dats'), where they could be a blob or a table. A hosted dat, in this view, then is more akin to a "project" or "data catalog."

I like number 2 -- exposing routes through http for each "dataset" (blob or table) makes sense. A nice benefit is that it is easy to reference a particular dataset in a paper this way like http://ebola.datplace.org/occurrences#ab3f42 or http://ebola.datplace.org/model#b3423

Exposing the REST api nicely with the dataset and commit/version number will need some thinking

[okdistribute]

We now support multiple datasets in dat beta. We also piggyback on ssh/http now, although there's still a simple dat server in core that allows for syncing and replication. I think we can close this now, thanks for all the discussion and input!

There are a bunch of open issues so if you think you want to keep chugging on this, we could use more input and hands! :)