Server-less p2p database built on libp2p
Join us on our public Slack channel for news, discussions, and status updates. Check out our blog for the latest posts and announcements.
- Table of Contents
- Security
- Background
- Install
- Getting Started
- Developing
- Contributing
- Changelog
- License
ThreadDB is still under heavy development and no part of it should be used before a thorough review of the underlying code and an understanding APIs and protocols may change rapidly. There may be coding mistakes, and the underlying protocols may contain design flaws. Please let us know immediately if you have discovered a security vulnerability.
Please also read the security note for go-ipfs.
ThreadDB is an implementation of the database described in the paper entitled A protocol & event-sourced database for decentralized user-siloed data.
Go to the docs for more about the motivations behind ThreadDB and Textile.
ThreadDB has two distinct layers:
db: The database layer is a document store, which internally leverages thenetAPI. Most applications will only interface with this layer.net: The network layer maintains and orchestrates append-only event logs between network participants. Some applications, like event logging, may choose to rely on this layer directly.
This repo contains a daemon and client for interacting with these layers as a remote service. Depending on the application, Golang projects may choose to import the internal db and net packages directly.
- Prebuilt package: See release assets
- Docker image: See the
latesttag on Docker Hub - Build from the source:
git clone https://github.com/textileio/go-threads
cd go-threads
go get ./threadsdimport "github.com/textileio/go-threads/api/client"You can think of the DB client as a gRPC client wrapper around the internal db package API, and the Network client as a gRPC client wrapper around the internal net package API. This section will only focus on getting started with the gRPC clients, but Golang apps may choose to interact directly with db and/or net.
The threadsd daemon can be run as a server or alongside desktop apps or command-line tools. The easiest way to run threadsd is by using the provided Docker Compose files. If you're new to Docker and/or Docker Compose, get started here. Once you are setup, you should have docker-compose in your PATH.
Create an .env file and add the following values:
THRDS_REPO=~/myrepo
THRDS_DEBUG=trueCopy this compose file and run it with the following command.
docker-compose -f docker-compose.yml up You should see some console output:
threads_1 | 2020-09-19T16:34:06.420Z DEBUG threadsd repo: /data/threads
threads_1 | 2020-09-19T16:34:06.420Z DEBUG threadsd hostAddr: /ip4/0.0.0.0/tcp/4006
threads_1 | 2020-09-19T16:34:06.421Z DEBUG threadsd apiAddr: /ip4/0.0.0.0/tcp/6006
threads_1 | 2020-09-19T16:34:06.421Z DEBUG threadsd apiProxyAddr: /ip4/0.0.0.0/tcp/6007
threads_1 | 2020-09-19T16:34:06.421Z DEBUG threadsd connLowWater: 100
threads_1 | 2020-09-19T16:34:06.421Z DEBUG threadsd connHighWater: 400
threads_1 | 2020-09-19T16:34:06.422Z DEBUG threadsd connGracePeriod: 20s
threads_1 | 2020-09-19T16:34:06.423Z DEBUG threadsd keepAliveInterval: 5s
threads_1 | 2020-09-19T16:34:06.423Z DEBUG threadsd enableNetPubsub: false
threads_1 | 2020-09-19T16:34:06.424Z DEBUG threadsd debug: true
threads_1 | Welcome to Threads!
threads_1 | Your peer ID is 12D3KooWFCXqmQTwvpfYFWK3DjXChEc4NoPt8pp5jjC8REZ3g6NZCongrats! Now you have ThreadDB running locally.
Note the various configuration values shown in the output above. These can be modified with environment variables show below.
THRDS_REPO: Repo location. Mandatory when launching from docker compose.THRDS_HOSTADDR: Libp2p host bind address./ip4/0.0.0.0/tcp/4006by default.THRDS_APIADDR: gRPC API bind address./ip4/0.0.0.0/tcp/6006by default.THRDS_APIPROXYADDR: gRPC API web proxy bind address./ip4/0.0.0.0/tcp/6007by default.THRDS_CONNLOWWATER: Low watermark of libp2p connections that'll be maintained.100by default.THRDS_CONNHIGHWATER: High watermark of libp2p connections that'll be maintained.400by default.THRDS_CONNGRACEPERIOD: Duration a new opened connection is not subject to pruning.20seconds by default.THRDS_KEEPALIVEINTERVAL: Websocket keepalive interval (must be >= 1s).5seconds by default.THRDS_ENABLENETPUBSUB: Enables thread networking over libp2p pubsub.falseby default.THRDS_DEBUG: Enables debug logging.falseby default.
The database layer is a document store, which internally leverages the net API. Most applications will only interface with this layer.
The full API spec is available here.
As described in the paper, ThreadDB's network layer orchestrates groups of event logs, or threads. In the current implementation, a single database leverages a single network-layer thread for state orchestration.
import "github.com/textileio/go-threads/api/client"
...
db, err := client.NewClient("127.0.0.1:6006", grpc.WithInsecure())Thread tokens (JWTs) are used by the daemon to determine the identity of the caller. Most APIs take a thread token as an optional argument, since whether or not they are needed usually depends on how the target collection is configured (see Write Validation and Read Filtering). These tokens are obtained by performing a signing challenge with the daemon using a libp2p private key.
privateKey, _, err := crypto.GenerateEd25519Key(rand.Reader) // Private key is kept locally
myIdentity := thread.NewLibp2pIdentity(privateKey)
threadToken, err := db.GetToken(context.Background(), myIdentity)threadID := thread.NewIDV1(thread.Raw, 32)
err := db.NewDB(context.Background(), threadID)An existing DB can be added to a different daemon by providing a valid host address and thread key.
threadID := thread.NewIDV1(thread.Raw, 32)
err := db1.NewDB(context.Background(), threadID)
dbInfo, err := db1.GetDBInfo(context.Background(), threadID)
...
// db2 is a different client (this would normally be done on a different machine)
err := db2.NewDBFromAddr(context.Background(), dbInfo.Addrs[0], dbInfo.Key)Collections are groups of documents or instances and are analogous to tables in relational databases. Creating a collection involves defining the following configuration parameters:
Name: The name of the collection, e.g, "Animals" (must be unique per DB).Schema: A JSON Schema), which is used for instance validation.Indexes: An optional list of index configurations, which define how instances are indexed.WriteValidator: An optional JavaScript (ECMAScript 5.1) function that is used to validate instances on write.ReadFilter: An optional JavaScript (ECMAScript 5.1) function that is used to filter instances on read.
The WriteValidator function receives three arguments:
writer: The multibase-encoded public key identity of the writer.event: An object describing the update event (seecore.db.Event).instance: The current instance as a JavaScript object before the update event is applied.
A falsy return value indicates a failed validation.
Having access to writer, event, and instance opens the door to a variety of app-specific logic. Textile Buckets file-level access roles are implemented in part with a write validator.
The function receives three arguments:
reader: The multibase-encoded public key identity of the reader.instance: The current instance as a JavaScript object.
The function must return a JavaScript object. Most implementation will modify and return the current instance.
Like write validation, read filtering opens the door to a variety of app-specific logic. Textile Buckets file-level access roles are implemented in part with a read filter.
import "github.com/alecthomas/jsonschema"
...
// We can use a struct to define a collection schema
type Person struct {
ID string `json:"_id"`
Name string `json:"name"`
Age int `json:"age"`
CreatedAt int `json:"created_at"`
}
reflector := jsonschema.Reflector{}
mySchema = reflector.Reflect(&Person{}) // Generate a JSON Schema from a struct
err := db.NewCollection(context.Background(), myThreadID, db.CollectionConfig{
Name: "Persons",
Schema: mySchema,
Indexes: []db.Index{{
Path: "name", // Value matches json tags
Unique: true, // Create a unique index on "name"
}},
})
...
// We can use the same schema to create more collections.
err := db.NewCollection(context.Background(), myThreadID, db.CollectionConfig{
Name: "Persons",
Schema: mySchema,
Indexes: []db.Index{{
Path: "name",
Unique: true,
}},
WriteValidator:
var type = event.patch.type
var patch = event.patch.json_patch
switch (type) {
case "delete":
if (writer != "the_boss") {
return false // Not the boss? No deletes for you.
}
default:
return true
}
,
ReadFilter:
if (instance.Age > 50) {
delete instance.Age // Getting old, let's hide just _how_ old hehe
}
return instance
,
})Each of the collection configuration parameters above can be updated.
...
err := db.UpdateCollection(context.Background(), myThreadID, db.CollectionConfig{
Name: "Persons",
Schema: mySchema,
Indexes: []db.Index{{
Path: "name",
Unique: true,
},
{
Path: "created_at", // Add an additional index on "created_at"
}},
})Creating a collection instance is analogous to inserting a row in a relational database table.
...
// ID is autogenerated when omitted
alice := &Person{
ID: "",
Name: "Alice",
Age: 30,
CreatedAt: time.Now().UnixNano(),
}
ids, err := db.Create(context.Background(), threadID, "Persons", Instances{alice})
alice.ID = ids[0] // ids contains autogenerated instance identifiers
// We can also define a custom ID, it just has to be a collection-wide unique string
bob := &Person{
ID: "123",
Name: "Bob",
Age: 30,
CreatedAt: time.Now().UnixNano(),
}
ids, err := db.Create(context.Background(), threadID, "Persons", Instances{bob})Similarly, we can update an instance with new values.
...
alice.Age = 31
err = db.Save(context.Background(), threadID, "Persons", Instances{alice})There are three methods to query for collection instances: Find, FindByID, and Has. As usual, queries are enhanced by indexes.
Check out db.Query and db.Criterion for more about constructing queries and ordering results.
...
// Find instances with a query
query := db.Where("name").Eq("Alice")
results, err := db.Find(context.Background(), threadID, "Persons", query, &Person{})
alice := results[0].(*Person)
...
// Find an instance by ID
alice := &Person{}
err = db.FindByID(context.Background(), threadID, "Persons", aliceID, alice)
...
// Determine if an instance exists by ID
exists, err := db.Has(context.Background(), threadID, "Persons", []string{aliceID})ThreadDB transactions come in two flavors: WriteTransaction and ReadTransaction.
...
txn, err := db.WriteTransaction(context.Background(), threadID, "Persons")
end, err := txn.Start()
alice.Age = 32
err = txn.Save(alice)
err = txn.Create(&Person{
Name: "Bob",
Age: 30,
CreatedAt: time.Now().UnixNano(),
})
end() // Done writing, commit transaction updates...
txn, err := db.ReadTransaction(context.Background(), threadID, "Persons")
end, err := txn.Start()
hasAlice, err := txn.Has(alice.ID)
results, err := txn.Find(db.Where("name").Eq("Bob"), &Person{})
bob := results[0].(*Person)
end() // Done readingWe can listen for DB changes on three levels: DB, collection, or instance.
Check out ListenOption for more.
...
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
events, err := db.Listen(ctx, threadID, []db.ListenOption{{
Type: client.ListenAll,
Collection: "Persons", // Omit to receive events from all collections
InstanceID: bob.ID, // Omit to receive events from all instances
}})
for event := range events {
// Handle event
}The network layer maintains and orchestrates append-only event logs between network participants and is used internally by the database layer. Some applications, like event logging, may choose to rely on this layer directly.
The full API spec is available here.
import "github.com/textileio/go-threads/net/api/client"
...
net, err := client.NewClient("127.0.0.1:6006", grpc.WithInsecure())Thread tokens (JWTs) are used by the daemon to determine the identity of the caller. Most APIs take a thread token as an optional argument.
privateKey, _, err := crypto.GenerateEd25519Key(rand.Reader) // Private key is kept locally
myIdentity := thread.NewLibp2pIdentity(privateKey)
threadToken, err := net.GetToken(context.Background(), myIdentity)threadID := thread.NewIDV1(thread.Raw, 32)
threadInfo, err := net.CreateThread(context.Background(), threadID)An existing thread can be added to a different daemon by providing a valid host address and thread key.
threadID := thread.NewIDV1(thread.Raw, 32)
threadInfo1, err := net1.CreateThread(context.Background(), threadID)
...
// net2 is a different client (this would normally be done on a different machine)
threadInfo2, err := net2.AddThread(context.Background(), threadInfo1.Addrs[0], core.WithThreadKey(threadInfo1.Key))We can replicate a thread on a different host. All logs and records are pushed to the new host. However, it will not be able to read them since it won't receive read portion of the thread key.
threadID := thread.NewIDV1(thread.Raw, 32)
threadInfo, err := net1.CreateThread(context.Background(), threadID)
replicatorAddr, err := multiaddr.NewMultiaddr("/ip4/<REPLICATOR_IP_ADDRESS>/tcp/4006/p2p/<REPLICATOR_PEER_ID>")
replicatorID, err := net.AddReplicator(context.Background(), threadID, replicatorAddr)A thread record can have any body.
import ipldcbor "github.com/ipfs/go-ipld-cbor"
...
body, err := ipldcbor.WrapObject(map[string]interface{}{
"foo": "bar",
"baz": []byte("howdy"),
}, multihash.SHA2_256, -1)
record, err := net.CreateRecord(context.Background(), threadID, body)We can also retain control over the read portion of the thread key and the log private key and create records locally.
import ipldcbor "github.com/ipfs/go-ipld-cbor"
...
privateKey, _, err := crypto.GenerateEd25519Key(rand.Reader)
myIdentity := thread.NewLibp2pIdentity(privateKey)
threadToken, err := net.GetToken(context.Background(), myIdentity)
threadID := thread.NewIDV1(thread.Raw, 32)
threadKey := thread.NewRandomKey()
logPrivateKey, logPublicKey, err := crypto.GenerateEd25519Key(rand.Reader)
logID, err := peer.IDFromPublicKey(logPublicKey)
threadInfo, err := net.CreateThread(
context.Background(),
threadID,
core.WithThreadKey(thread.NewServiceKey(threadKey.Service())), // Read key is kept locally
core.WithLogKey(logPublicKey), // Private key is kept locally
core.WithNewThreadToken(threadToken)) // Thread token for identity is needed to verify records
body, err := ipldcbor.WrapObject(map[string]interface{}{
"foo": "bar",
"baz": []byte("howdy"),
}, mh.SHA2_256, -1)
// Create the event locally
event, err := cbor.CreateEvent(context.Background(), nil, body, threadKey.Read())
// Create the record locally
record, err := cbor.CreateRecord(context.Background(), nil, cbor.CreateRecordConfig{
Block: event,
Prev: cid.Undef, // No previous records because this is the first
Key: logPrivateKey,
PubKey: myIdentity.GetPublic(),
ServiceKey: threadKey.Service(),
})
err = net.AddRecord(context.Background(), threadID, logID, record)Although all known hosts of a particular thread are internally polled for new records (as part of the orchestration protocol), doing so manually can often be useful.
err := net.PullThread(context.Background(), info.ID)We can listen for new thread records across all or a subset of known threads.
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
records, err := net.Subscribe(ctx, core.WithSubFilter(threadID)) // Only receive new records from this thread
for record := range records {
// Handle record
}The easiest way to develop against threadsd is to use the Docker Compose files. The -dev flavored file doesn't persist a repo via Docker Volumes, which may be desirable in some cases.
Pull requests and bug reports are very welcome ❤️
This repository falls under the Textile Code of Conduct.
Feel free to get in touch by:
- Opening an issue
- Joining the public Slack channel
- Sending an email to [email protected]
A changelog is published along with each release.
![dependabot-preview[bot] avatar](https://avatars.githubusercontent.com/in/2141?v=4 "dependabot-preview[bot]")
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent