jakekgrog / ghostdb

GhostDB is a distributed, in-memory, general purpose key-value data store that delivers microsecond performance at any scale.

Hi there 👋!

My name is Jake Grogan. I'm interested in databases and distributed systems. Currently working on GhostDB. I like to use Go, Python, and NodeJS.

Connect with me!

ghostdb's Issues

Tidb CDC example integration

GhostDB would be a great match for databases that support CDC

You could use GhostDB to hold your data in a cdb network like cloudflare as a materialised view that is fed from tidb.

Here is an example that feeds data from tidb to Kafka to get an idea :
https://github.com/pingcap/ticdc/tree/master/kafka_consumer.

GhostDB would then act as the read only dB and TiDb cluster as the write dB.

This would be amazingly efficient I think.

Add support for Queue data type

Adding support for queues will allow for GhostDB to function as a message queue.

How does GhostDB compare to ...

There are many KV stores, and many caches, and many in-memory things.

How does GhostDB compare? Can you add comparisons to memcache, redis, annakv.. etc to the website? Seems like that would be a FAQ.

Data type metadata should be included in cache node objects

Now that we support more than 1 data type, we should include the values data type in cache nodes. This give us a way of easily identifying what type of data a key points to e.g. string, queue (and more types in the future)

Currently a cache node object consists of a Key, a Value and a TTL. We should also add a field specifying the type of the Value

Add improve README.md

Please add a license to this repo

First, thank you for sharing this project with us!

Could you please add an explicit LICENSE file to the repo so that it's clear
under what terms the content is provided, and under what terms user
contributions are licensed?

Per GitHub docs on licensing:

[...] without a license, the default copyright laws apply, meaning that you
retain all rights to your source code and no one may reproduce, distribute,
or create derivative works from your work. If you're creating an open source
project, we strongly encourage you to include an open source license.

Thanks!

Add linting phase to workflow

Adding a linting phase to our workflow will ensure all future changes conform to the same style standard.

Installer rework

Creating a user via the installer doesn't create the user dir. Requires manual creation in order for the service to run.

Add issue and PR templates in /docs

Add issue and PR templates in /docs

Keyspace analysis

We would like a way for our users to be able to perform analysis of the keyspace of a node. This would require keeping track of keys that are frequently used (keys with high number of hits; we need to define what constitutes a high number of hits, perhaps just the top x keys can be returned), the longest keys, shortest keys etc.

We would also like to be able to analyse the largest items in the cache. For example, a user might want to know the largest queue on the node, the longest string, largest set etc.

Again, this is up for discussion and I'm interested in hearing your thoughts!

Benchmark result?

Compared to similar product: Redis, Aerospike, Tarantool, etc

Request history

We would like to be able to create a log of recent request history for the node (the last hour for example). These logs would contain information about what commands were executed, who the request came from (what client). Whether to request failed or succeeded etc. It is open for discussion here so please contribute if you've any good ideas!

Use YAML as default config format

We and our users currently write our configs in JSON. We want to move to the YAML format for writing configs.

To help you get started, the following files will need updating (some other files may require changes):

• config_reader.go
• config_reader_test.go
• INSTALLATION.md

When service is stopped it is shown as failed

● ghostdb.service loaded failed failed ghostdb daemon
Should show inactive - no exit code

keyspaceSize definition

Found this project via Hacker News. Looks awesome. Quick question:

“keyspaceSize”: 65536

Is that the number of items or number of bytes that can be stored? Typically you allocate X number of bytes of memory that can be used. Allocating the number of items seems strange as items will be variable length right?

Add get number of client connections

For the GhostDB CLI we need a way of counting the number of active connections to each node in a cluster.