jgarzik / kvdbd

API service exporting embedded key/value databases to remote users

kvdbd

Daemon that enables reading/writing of flat-file key/value databases available via HTTP API, using REST/JSON or Protobufs.

Goals

• "NoSQL databases go from filesystem to docker microservice"
• Written in safe rust
• Can be queried by stock HTTP clients
• Modern HTTP service with threads, HTTP/2 etc.
• Beyond Get/Put/Delete, expose db-specific operations such as transactions or batch-update.
• Access multiple databases simultaneously from a single HTTP endpoint.
• Support multiple database back-ends (sled, lmdb, more soon)
• Docker-friendly Zero configuration mode

Documentation

• Requirements: Rust 2018+ and protoc compiler.
• Quick Start: see below
• Configuration: config.md
• Remote HTTP API: api.md

Using kvdbd

From cargo

Standard rust cargo multi-binary setup:

$ cargo build
$ cargo run --bin kvdbd

From docker

Zeroconf docker example, with sled database stored on docker volume dbdata:

$ docker volume create dbdata
$ docker run --rm -p 8080:8080 -v dbdata:/data jgarzik/kvdbd \
	kvdbd --bind-addr 0.0.0.0 --sled /data/sled.db
$ curl http://127.0.0.1:8080/ | json_pp

Testing

Integration testing is performed via a separate binary, tester.

$ cargo run --bin tester

Other projects

Unrelated projects with similar names:

• Another "kvdb" exists as a Go library: https://github.com/portworx/kvdb
• An online cloud service https://kvdb.io/

kvdbd's Issues

TLS support

Enable TLS support in actix. Example: https://github.com/actix/actix-website/blob/master/examples/server/src/ssl.rs

Improve error propagation

Propagate db-specific errors up the stack, giving the end user/devop improved diagnostics for any error condition (which are rare, in production).

This may imply providing our own generic Error and Result.

Protobuf helper binary

Small helper binary to encode/decode our wire protocol via stdin/stdout, e.g.

$ kvdb-pb --encode --op get --key FirstName > /tmp/get-pb.bin
$ curl -X POST --data-binary @/tmp/get-pb.bin http://localhost:8080/api/db/get

API: Prefix filtering, for db iteration

Related to #17

Limit the query to only keys that match a certain prefix.

For b-tree databases, this is easy.
This hash databases, this requires us to filter out non-matches during a full db walk.

Support multiple db backends (sled, leveldb, lmdb, gdbm ...)

• Design rust modular database driver construct
• Call generic db driver API from web request handlers
• Add 2nd db backend
• Test continually with multiple database back-ends

Key size limits

Keys for each type of database tend to have size limits, e.g. 4096 bytes.

Validate and check these limits.

Improve API reference documentation

The current API reference leaves a lot to be desired. Use modern tools to create something better.

Debian/ubuntu packaging

Add control files and other setup necessary to create an OS package for Linux.

• Debian control files
• systemd configuration

Publish on Docker Hub

Enable users to discover and use kvdbd directly from Docker Hub.

e.g.

$ docker run -v /path/to/host/dbdir:/data jgarzik/kvdbd kvdbd --sled /data

API: DB Statistics (record count)

Expose ability to query total record count (primarily), and any other useful db metadata.

Record count is often an expensive operation, walking the entire database tree / hash buckets.

JSON and zeroconf ServerConfig structs differ

In zeroconf mode, ServerConfig struct is used.

In JSON config mode, the following is used,

server_cfg = serde_json::from_str(&cfg_text)?;

This creates problems such as the unintended requirement that the ssl section of the config be present, even if using unencrypted HTTP mode.

API: Clear database (delete all records)

Most databases expose a delete-all-records call; export it via API.

GRPC support

Support a pure GRPC mode.

Berkeley DB4 driver

Dockerize

Create and test project Dockerfile

API: Ability to list/iterate keys

All key/value databases offer some method of listing keys, or prefix-matched subsets of keys.

GDBM driver

Preferably using https://github.com/jgarzik/gdbm-rs

Fix caching docker build

For some as-yet-not-investigated reason, the double-build technique that enables docker caching, currently commented-out in the Dockerfile, causes the docker build to fail thusly:

   Compiling actix-files v0.1.4
   Compiling kvdbd v0.3.1 (/usr/src/kvdbd)
error: couldn't read src/kvdb-pb.rs: No such file or directory (os error 2)

error: aborting due to previous error

error: Could not compile `kvdbd`.
warning: build failed, waiting for other jobs to finish...
error: couldn't read src/tester.rs: No such file or directory (os error 2)

error: aborting due to previous error

error: Could not compile `kvdbd`.
warning: build failed, waiting for other jobs to finish...
error: build failed
The command '/bin/sh -c cargo build --release' returned a non-zero code: 101

Go async: reqwest

Convert tester to async, which uses reqwest.

Support non-UTF8 keys

Update protoc version

First noted in #16 - protoc link has disappeared, because we're referencing an older protoc version. Update in travis etc. scripts.

kvdb-pb util: encode "batch" operation

The kvdb-pb util should --encode the BatchRequest operation.

This requires a more complicated command line parse, than for get/put/del.

Document kvdb-pb

Add some examples of kvdb-pb use to docs.

Zero-configuration support

Users should be able to run a fully working server from the command line, with no configuration files, e.g.

$ kvdb --gdbm /path/to/database.dat &
$ sleep 1
$ curl http://127.0.0.1:8080/

Uploading Large values

Large values are likely to present at least two problems:

1. actix may not buffer more than a chunk, during uploads.
2. Payload buffered entirely in memory.

Use rust native-tls

rather than openssl