zond / god

A Go database

god

god is a scalable, performant, persistent, in-memory data structure system. It allows massively distributed applications to update and fetch common data in a structured and sorted format.

Its main inspirations are Redis and Chord/DHash. Like Redis it focuses on performance, ease of use, and a small, simple yet powerful feature set, while from the Chord/DHash projects it inherits scalability, redundancy, and transparent failover behaviour.

Try it out

Install Go, git, Mercurial and gcc, go get github.com/zond/god/god_server, run god_server, browse to http://localhost:9192/.

Embed it in your Go application

import "github.com/zond/god/dhash"
s := dhash.NewNodeDir(fmt.Sprintf("%v:%v", listenIp, listenPort), fmt.Sprintf("%v:%v", broadcastIp, broadcastPort), dataDir)
s.MustStart()
s.MustJoin(fmt.Sprintf("%v:%v", joinIp, joinPort))

Documents

HTML documentation: http://zond.github.com/god/

godoc documentation: http://godoc.org/github.com/zond/god

TODO

• Docs
• Add illustrations to the usage manual
• Benchmark
• Consecutively start 1-20 instances on equally powerful machines and benchmark against each size
• Need 20 machines of equal and constant performance. Is anyone willing to lend me this for few days of benchmarking?
• Add benchmark results to docs

god's Issues

god_server panics due to invalid memory referencing on windows

After changing god server's dir name from ip:port to ip_port, the server panics as below:

D:\sources\gopath\src\github.com\zond\god\god_server>god_server
panic: runtime error: invalid memory address or nil pointer dereference
panic: sync: unlock of unlocked mutex
[signal 0xc0000005 code=0x0 addr=0x0 pc=0x49ce6b]

goroutine 1 [running]:
sync.(_Mutex).Unlock(0x12c87f60)
D:/sources/golang/src/pkg/sync/mutex.go:91 +0x79
sync.(_RWMutex).Unlock(0x12c87f60)
D:/sources/golang/src/pkg/sync/rwmutex.go:113 +0x77
sync/atomic.StoreUint64()
D:/sources/golang/src/pkg/sync/atomic/asm_386.s:147 +0xb
github.com/zond/god/dhash.func-+016(0x12d0cc40, 0x12d0cc60)
D:/sources/gopath/src/github.com/zond/god/dhash/dhash.go:104 +0x9b
github.com/zond/god/common.(_Ring).sendChanges(0x12c87f40, 0x12c5bc00, 0x10, 0x10)
D:/sources/gopath/src/github.com/zond/god/common/ring.go:103 +0x141
github.com/zond/god/common.(_Ring).Add(0x12c87f40, 0x12c5bb80, 0x10, 0x10, 0x12c5b270, ...)
D:/sources/gopath/src/github.com/zond/god/common/ring.go:182 +0x49f
github.com/zond/god/discord.(_Node).Start(0x12c5cf40, 0x0, 0x0)
D:/sources/gopath/src/github.com/zond/god/discord/node.go:208 +0x36b
github.com/zond/god/dhash.(_Node).Start(0x12c61840, 0x0, 0x0)
D:/sources/gopath/src/github.com/zond/god/dhash/dhash.go:198 +0xcd
github.com/zond/god/dhash.(*Node).MustStart(0x12c61840, 0xe)
D:/sources/gopath/src/github.com/zond/god/dhash/dhash.go:393 +0x28
main.main()
D:/sources/gopath/src/github.com/zond/god/god_server/god_server.go:48 +0x333

goroutine 2 [syscall]:

goroutine 17 [select]:
github.com/zond/god/persistence.(_Logger).record(0x12c5e390, 0x12d0d240)
D:/sources/gopath/src/github.com/zond/god/persistence/persistence.go:402 +0x414
created by github.com/zond/god/persistence.(_Logger).Record
D:/sources/gopath/src/github.com/zond/god/persistence/persistence.go:382 +0x10d

goroutine 18 [syscall]:
syscall.Syscall6(0x7549d3ab, 0x5, 0xd0, 0x12c5bcb0, 0x12bf0a98, ...)
D:/sources/golang/src/pkg/runtime/zsyscall_windows_windows_386.c:97 +0x49
syscall.GetQueuedCompletionStatus(0xd0, 0x12c5bcb0, 0x12bf0a98, 0x12bf0a90, 0xffffffff, ...)
D:/sources/golang/src/pkg/syscall/zsyscall_windows_386.go:507 +0x7e
net.(*resultSrv).Run(0x12bf09a0)
D:/sources/golang/src/pkg/net/fd_windows.go:150 +0x11a
created by net.startServer
D:/sources/golang/src/pkg/net/fd_windows.go:285 +0xde

Your code is not worthy of being called "God"

By the way, my "God" is an Edward Witten inspired, Supersymmetric learning algorithm.

1. Gods ought to be learning algorithms, not basic code as you currently have in your repo.
2. Notably I am an atheist, so I observe that Gods as defined within the realm of artificial general intelligence ought not to be worshipped and ought to be atheism compatible!!

Question about latency/garbage collection

Is the hashtable stored in the garbage collected heap? If there's eg. 16GB worth of ~1kb key-value-pairs, what is the expected maximum latency if the GC happens to run?

Docs not updated?

First of all thanks for the great project. Very interesting indeed. However when I was learning about this DB, i noticed some inconsistencies with API and docs on http://zond.github.io/god/#friends, as it is importing " github.com/zond/god/setop ", however it looks like the actual API comes somewhere from " github.com/zond/god/dhash ". Does the doc need updating?

Thanks again.

"../web/web.go:4:2: no Go source files in" god/templates

I built a week or two ago without a problem, but fails now ...

I am not sure what it is you were trying to do with the main() in the template/compiler folder and then no template package but importing it as an absolute path.

Good luck, looks great - my interest is mostly in how you are handling the on disk storage.

Thanks

newb Issue

chuck:server$ go run server.go

github.com/zond/god/murmur

/usr/lib/go/src/pkg/github.com/zond/god/murmur/murmur.c:27:23: warning: always_inline function might not be inlinable [-Wattributes]
/usr/lib/go/src/pkg/github.com/zond/god/murmur/murmur.c:19:23: warning: always_inline function might not be inlinable [-Wattributes]
/usr/lib/go/src/pkg/github.com/zond/god/murmur/murmur.c: In function ‘MurmurHash3_x64_128’:
/usr/lib/go/src/pkg/github.com/zond/god/murmur/murmur.c:19:23: error: inlining failed in call to always_inline ‘getblock’: function body can be overwritten at link time
/usr/lib/go/src/pkg/github.com/zond/god/murmur/murmur.c:59:14: error: called from here
/usr/lib/go/src/pkg/github.com/zond/god/murmur/murmur.c:19:23: error: inlining failed in call to always_inline ‘getblock’: function body can be overwritten at link time
/usr/lib/go/src/pkg/github.com/zond/god/murmur/murmur.c:60:14: error: called from here
/usr/lib/go/src/pkg/github.com/zond/god/murmur/murmur.c:27:23: error: inlining failed in call to always_inline ‘fmix’: function body can be overwritten at link time
/usr/lib/go/src/pkg/github.com/zond/god/murmur/murmur.c:109:6: error: called from here
/usr/lib/go/src/pkg/github.com/zond/god/murmur/murmur.c:27:23: error: inlining failed in call to always_inline ‘fmix’: function body can be overwritten at link time
/usr/lib/go/src/pkg/github.com/zond/god/murmur/murmur.c:110:6: error: called from here

god_server fails to start on windows due to mkdir of invalid directory

The go_server fails to start while attempting to create a invalid directory on windows.
Directory name = ip:port

Hence fails to start

D:\sources\gopath\src\github.com\zond\god\god_server>god_server
panic: mkdir 127.0.0.1:9191: The directory name is invalid.

stoped the development?

Well it's a pity that...

Cockroachdb is very good..

But I need a distributed cache database is used to replace redis database.

and "god" Have many good features.

panic

when i start with:
god_server -listenIp="10.1.252.12"

then:
panic: <[email protected]:9191> is trying to remove itself from the routing!

10.1.252.12 is my ip

License?

This project looks very interesting. Wondering what the license is?

Detailed explanation for SetExpression and Operations?

Last two examples are quite overwhelming and I kinda was lost in translation of how some of the operations and expressions really work? For example

From lines 22, 36 on similarity and revelance example resp.

op := &setop.SetOp{ Merge: setop.IntegerSum, Type: setop.Union, } ......... op := &setop.SetOp{ Merge: setop.First, Type: setop.Union, }

I am not familiar with set operations in context with programming. So I was wondering if you have some detailed explanation on how to interpret such operations?

Thanks a lot.

Test test

Warning / Error?

I was running one of the examples from your documentations and I was running on my local computer. Everytime I build and run it gives me this

Any idea?

The DB works perfectly, only concerned about this warning!

Thanks

how to maintain a radix tree in god?

we want to maintain a radix tree in god db. is it possible?
rather than putting it as a sub-tree under a node, we want to use the internal radix tree of god. so we just insert key/value into god and god maintains the radix tree for us. also prefix based search is desired.

Support different bind and announce IPs

Suppose I have an EC2 instance in us-west-2 and another in us-east-1, both running a god instance; I would want them both to bind on 0.0.0.0 but to each other, they would be accessible via the public external IPs (e.g. 50.100.100.100), so in communication with other god instances, they should use 50.100.100.100 rather than 0.0.0.0.

Currently, if I run:

god_server -ip="0.0.0.0"

this enters the node into the DHT as "0.0.0.0", which prevents any further nodes from properly discovering this node.

To solve this issue, I'd propose an additional option:

god_server -ip="0.0.0.0" -announceIp="50.100.100.100"

The -ip option would continue to be the IP to bind to (and, if announceIp isn't specified, the IP to announce), but -announceIp would be the IP to publish to other nodes and store in the DHT.

DB is not progress?

Hello, @zond.
Why DB is stopped in development?

Set Intersection Theoretical Question.

I'm interested in repetitively executing a matching operation on a K x K (K = ~8000) matrix of Sets to determine best matches by Set intersection. The Sets may each have on the order 100-2000 elements each. And would change regularly. And potentially the number of matrices would grow and be processed in parallel.

At first I thought to do this using Redis Set Intersection SINT with a O(N^2) Lua script,
then I started to think I should look into keeping track of the sets with Redis and piping them in groups to a GPU for batches of Set Compare ops.

Now I'm trying to figure out if radix-tree / mirror arrangement is better suited. With some simple straight forward paper scratching I'm thinking that it does not.

It seems that you are addressing a problem of Set operations with your design, or is this simply a noted side benefit?

Do you think the radix tree / mirror arrangement would have an advantage for operations for my use case?