houzuoguo / tiedot

A rudimentary implementation of a basic document (NoSQL) database in Go

2021-09-05: thanks for evaluating tiedot! The project started as a short programming exercise and I am very grateful for the small user base it has received. I am afraid the project has been dormant since 2016, the feature set is minimal and lacking for a modern application. Please consider other embedded DB engine alternatives. In the meanwhile, check out this actively maintained open source project of mine: laitos - Top geek's chindogu.

tiedot

Documentation

Keywords: Golang, go, document database, NoSQL, JSON

tiedot - Your NoSQL database powered by Golang

tiedot is a document database engine that uses JSON as document notation; it has a powerful query processor that supports advanced set operations; it can be embedded into your program, or run a stand-alone server using HTTP for an API. It runs on *nix and Windows operating systems.

tiedot has fault-tolerant data structures that put your data safety first, while easily scales to 4+ CPU cores.

tiedot has very stable performance, even with millions of records! It consistently achieves high throughput - swallow more than 120k records or 80k complicated queries per second with confidence.

Get tiedot!

tiedot is distributed under the Simplified BSD license.

The newest version 3.4 comes with general performance and compatibility improvements. Find out more in releases.

Running in Docker

Run tiedot with help from docker and docker compose:

$ docker-compose build
$ docker-compose up -d

To view the logs:

$ docker-compose logs

References

• Tutorial: tiedot in 10 minutes
• API reference and embedded usage
• Limitations

... and more

• Performance tuning and benchmarks
• Data structures
• Query processor and index
• Concurrency and networking

Contributions welcome!

tiedot is a very small project in the large open source community - it is growing fast thanks to the 800+ stars and watchers, as well as many contributors for their feedback, comments, ideas and code. Your contribution matters a lot!

Pull requests/forks all welcome, and please share your thoughts, questions and feature requests in Issues section.

Let me know what you think about tiedot, I'd love to hear from you! Please Email me, follow my Twitter and blog.

The Go gopher was designed by Renee French. (http://reneefrench.blogspot.com/). The Go gopher is covered by the Creative Commons Attribution 3.0 license.

tiedot's Issues

Parameterized queries

Using tiedot in embedded mode, one often wants to create a query based on user input. There isn't really a safe / easy way to do this at the moment. Concatenating user-supplied strings could embed things that muck up your JSON (analogous to SQL injection). Substituting a string in your final marshaled JSON object requires you to write a decent amount of specific code & casts for each parameterization as you traverse what the compiler thinks is a map[string]interface{}.

My preferred solution would be to have a function EvalParameterizedQuery(q string, src *Col, result *map[uint64]struct{}, params ...string) (err error) that interpolates ? to the parameters (considering them as strings) and then marshals the JSON internally. Since a raw ? embedded in JSON, without wrapping in a string, is a syntax error, this should be unambiguous.

One would have to do quote-counting to identify the un-string-wrapped ?'s, mindful of the possibility of escaped " characters.

My understanding is that given you're able to identify the insertion point as valid, it's sufficient to wrap the user-supplied string in double-quotes and escape the characters:

"
\
control characters (<U+0020)

If this is an acceptable solution, I'm willing to write this up (although it may take me some time).

Unmarshal from collection to struct

If you change the signature of the Read-Method, then it would be easier to directly unmarshal into go-structs. Now, Read() always returns a Map, because the return value is an interface{}.

Let's take a struct and insert a record:
type Person struct {
Firstname string
Lastname string
}

People := myDB.Use("People")
doc := Person{"Rob", "Pike"}
docID, err := People.Insert(doc)

Currently the signature of Read() is:
func (col *Col) Read(id uint64) (doc interface{}, size int)

and Read() returns this:
res, s := People.Read(docID)
-> map[string]interface {}{"Firstname":"Rob", "Lastname":"Pike"}

If you change the signature of Read() to this (and maybe also returning err):
func (col *Col) Read2(id uint64, doc interface{}) (size int)

then it would automatically unmarshal into the struct we passed into the Read-Method:
s := People.Read(docID, &doc)
-> main.Person{Firstname:"Rob", Lastname:"Pike"}

Correct or did I miss something?! If so, do you want to have a pull request from me? :)

Upsert

Sometimes, it's silly to have to check if the object is there first (i.e. update err). An upsert call would be useful.

Embedded - Scaling out?

Are there concerns with running multiple instances of the embedded engine across multiple web servers/VMs? What about multiple instances sharing the same file system?

For example, I can setup and run an instance of Go on Windows Azure Web Sites as well as Rackspace Website Cloud - both of which uses shared file systems, most likely attached via network.

Are their concurrency issues with multiple embedded engines running against a single shared file system? For example, Lucene can have multiple embedded engines running in READ mode, but only one instance can have a Write-lock.

I'm attempting to avoid running dedicated DB instances externally from the main web apps - by embedding tiedot directly in my web apps. This keeps costs down by only running two instances for redundancy, while also making the overall cloud hosting elastic by spinning up more VMs as server load increases (e.g. front page of NY Times).

Though, I am aware that scaling would be easier to tune by separating them. Just trying to jeep initial costs down. Which, brings up the same question: are there any issues with running multiple "Service" instances against a single shared file system?

to type more easily

recommend
change "int-from" and "int-to" to "int from" "int to"
:)

FOSDEM

Saw your talk ... Well done. It would be nice if you add more time. 👍

Range query support

Currently tiedot only supports hash table index, it will be very helpful to add another index type to support range queries.

Make more embedded usage examples

EvalQuery problem

according to example.go,

    // Execute query
    result := make(map[uint64]struct{})
    var query interface{}
    json.Unmarshal([]byte(`["all"]`), &query)
    if err := db.EvalQuery(query, F, &result); err != nil {
        panic(err)
    }

EvalQuery returns result, which has key and value. but I printed value, it always be nil.

Fix two embedded usage pitfalls

Pitfall : User may forget to initialize random number generator
Proposed fix: Initialize the Go pseudo random number generator by feeding seed from crypto/rand upon opening a database

Pitfall 2: Type conversion failure caused missing index content.
Proposed fix: Log an error message when such type conversion failure occurs.

Why no JSON query response option?

If using tiedot in stand-alone http server mode, it seems odd that a query request is structured as JSON but the response is not JSON.

If you want to write a quick JavaScript browser app to work with data in tiedot, you would always need to wrap the query response before parsing it to JSON. Also, you can pass a query to tiedot in the address bar of a browser, but if you have an extension like JSONView it will choke on the response because it is not valid JSON.

For a "query" command, the documents could be wrapped in an array.
Example:
{"results":[
{"doc": 1},
{"doc": 7),
...
{"doc": n)
]}

For a "queryID commad, the ID's could be wrapped in an array.
Example:
{"resultIDs": [1,7, ... ,n]}

I realize that in embedded mode this is not useful and you are not trying to compete with CouchDB, MongoDB, etc. However, if a standalone HTTP API is being offered, which uses JSON for documents and queries, why not use it for query results?

Also, the "query" results do not include the document ID, so if you chose to edit or delete one of the result documents, how do you know which ID to pass to the the document "update" or "delete" methods?

Error with go 1.2

I know your docs say go 1.1.2

Just letting you know

/github.com/HouzuoGuo/tiedot/gommap.a: object is [linux amd64 go1.1.2 X:none] expected [linux amd64 go1.2 X:none]

Give each document a unique and persistent ID

Currently database does not assign a persistent and unique ID to collection documents, but it is very helpful to have such unique ID in each document.

Send content type in HTTP response header

Server response should contain "content-type" header.

Support indexing and querying objects in JSON array

This will likely require:

• Write new methods for adding/removing array element index
• Refactor collection "GetIn" function
• Rewrite certain parts in query processor

Gopher is cut in half by line break

Github appears to put a line break after heading 1, and the line break unfortunately cuts gopher bunny into halves...

Greater than, Lesser than Query?

Is >, < query possible? If so, how can I do it?

Example Documents :

{"Age": "8", "Name":"Pat"}
{"Age": "9", "Name":"Kat"}
{"Age": "10", "Name":"John"}
{"Age": "11", "Name":"George"}
{"Age": "12", "Name":"Matt"}
{"Age": "13", "Name":"Richard"}
{"Age": "14", "Name":"Doe"}

Example Query : [">", {"eq": "9", "limit": 5, "in": ["Age"]}]

Result :

{"Age": "10", "Name":"John"}
{"Age": "11", "Name":"George"}
{"Age": "12", "Name":"Matt"}
{"Age": "13", "Name":"Richard"}
{"Age": "14", "Name":"Doe"}

Thanks

Why not use UUIDs?

Is there any reason apart from space requirements that
the generated ids are no UUIDs. That would make it easier to
synchronized different tiedot databases.

Tiedot has memory issues with many collections and <256MB of RAM

fatal error: runtime: out of memory

goroutine 1 [running]:
runtime.throw(0x71e777)
/usr/local/go/src/pkg/runtime/panic.c:464 +0x69 fp=0x7f2c829f5768
runtime.SysMap(0xc216100000, 0x8000000, 0x72c538)
/usr/local/go/src/pkg/runtime/mem_linux.c:131 +0xfe fp=0x7f2c829f5798
runtime.MHeap_SysAlloc(0x736480, 0x8000000)
/usr/local/go/src/pkg/runtime/malloc.goc:473 +0x10a fp=0x7f2c829f57d8
MHeap_Grow(0x736480, 0x8000)
/usr/local/go/src/pkg/runtime/mheap.c:241 +0x5d fp=0x7f2c829f5818
MHeap_AllocLocked(0x736480, 0x8000, 0xc200000000)
/usr/local/go/src/pkg/runtime/mheap.c:126 +0x305 fp=0x7f2c829f5858
runtime.MHeap_Alloc(0x736480, 0x8000, 0x100000000, 0x1)
/usr/local/go/src/pkg/runtime/mheap.c:95 +0x7b fp=0x7f2c829f5880
runtime.mallocgc(0x8000000, 0x525e81, 0x1)
/usr/local/go/src/pkg/runtime/malloc.goc:89 +0x484 fp=0x7f2c829f58f0
cnew(0x525e80, 0x8000000, 0xc200000001)
/usr/local/go/src/pkg/runtime/malloc.goc:718 +0xc1 fp=0x7f2c829f5910
runtime.cnewarray(0x525e80, 0x8000000)
/usr/local/go/src/pkg/runtime/malloc.goc:731 +0x3a fp=0x7f2c829f5930
makeslice1(0x51daa0, 0x8000000, 0x8000000, 0x7f2c829f5990)
/usr/local/go/src/pkg/runtime/slice.c:57 +0x4d fp=0x7f2c829f5948
runtime.makeslice(0x51daa0, 0x8000000, 0x8000000, 0x0, 0x8000000, ...)
/usr/local/go/src/pkg/runtime/slice.c:38 +0x98 fp=0x7f2c829f5978
loveoneanother.at/tiedot/file.(*File).CheckSizeAndEnsure(0xc21001eeb0, 0x8000000)
/tmp/selfgz6496/src/loveoneanother.at/tiedot/file/file.go:87 +0x132 fp=0x7f2c829f5a00
loveoneanother.at/tiedot/file.Open(0xc210059980, 0x30, 0x8000000, 0xc21001eeb0, 0x0, ...)
/tmp/selfgz6496/src/loveoneanother.at/tiedot/file/file.go:36 +0x2dd fp=0x7f2c829f5aa8
loveoneanother.at/tiedot/file.OpenCol(0xc210059980, 0x30, 0x2, 0xc210059980, 0x30)
/tmp/selfgz6496/src/loveoneanother.at/tiedot/file/col.go:30 +0x3a fp=0x7f2c829f5b18
loveoneanother.at/tiedot/db.OpenCol(0xc210036ba0, 0x2b, 0xc21001ee60, 0x0, 0x0)
/tmp/selfgz6496/src/loveoneanother.at/tiedot/db/col.go:56 +0x30c fp=0x7f2c829f5bf8
loveoneanother.at/tiedot/db.OpenDB(0xc210036450, 0x21, 0xc21004c180, 0x0, 0x0)
/tmp/selfgz6496/src/loveoneanother.at/tiedot/db/db.go:30 +0x2bb fp=0x7f2c829f5d10
main.main()
/tmp/selfgz6496/databank.go:355 +0x149 fp=0x7f2c829f5f48
runtime.main()
/usr/local/go/src/pkg/runtime/proc.c:215 +0x11f fp=0x7f2c829f5fa0
runtime.goexit()
/usr/local/go/src/pkg/runtime/proc.c:1389 fp=0x7f2c829f5fa8

goroutine 3 [syscall]:
os/signal.loop()
/usr/local/go/src/pkg/os/signal/signal_unix.go:21 +0x1e
created by os/signal.init·1
/usr/local/go/src/pkg/os/signal/signal_unix.go:27 +0x31

The problem function is CheckSizeAndEnsure(). Perhaps if the call to make
([]byte, file.Growth)

Was split up into smaller chunks and written separately? That shouldn't have a runtime performance hit.

I split the write call into 64 different interations to reduce the impact on memory. This solved the above problem, but a new one appeared after 5 collections have been opened:

fatal error: runtime: out of memory

goroutine 1 [running]:
runtime.throw(0x71e777)
/usr/local/go/src/pkg/runtime/panic.c:464 +0x69 fp=0x7fc88b846898
runtime.SysMap(0xc21a100000, 0x2000000, 0x72c538)
/usr/local/go/src/pkg/runtime/mem_linux.c:131 +0xfe fp=0x7fc88b8468c8
runtime.MHeap_SysAlloc(0x736480, 0x2000000)
/usr/local/go/src/pkg/runtime/malloc.goc:473 +0x10a fp=0x7fc88b846908
MHeap_Grow(0x736480, 0x2000)
/usr/local/go/src/pkg/runtime/mheap.c:241 +0x5d fp=0x7fc88b846948
MHeap_AllocLocked(0x736480, 0x2000, 0x0)
/usr/local/go/src/pkg/runtime/mheap.c:126 +0x305 fp=0x7fc88b846988
runtime.MHeap_Alloc(0x736480, 0x2000, 0x100000000, 0x1)
/usr/local/go/src/pkg/runtime/mheap.c:95 +0x7b fp=0x7fc88b8469b0
runtime.mallocgc(0x2000000, 0x525e81, 0x1)
/usr/local/go/src/pkg/runtime/malloc.goc:89 +0x484 fp=0x7fc88b846a20
cnew(0x525e80, 0x2000000, 0xc200000001)
/usr/local/go/src/pkg/runtime/malloc.goc:718 +0xc1 fp=0x7fc88b846a40
runtime.cnewarray(0x525e80, 0x2000000)
/usr/local/go/src/pkg/runtime/malloc.goc:731 +0x3a fp=0x7fc88b846a60
makeslice1(0x51daa0, 0x2000000, 0x2000000, 0x7fc88b846ac0)
/usr/local/go/src/pkg/runtime/slice.c:57 +0x4d fp=0x7fc88b846a78
runtime.makeslice(0x51daa0, 0x2000000, 0x2000000, 0xc21001ec80, 0x2000000, ...)
/usr/local/go/src/pkg/runtime/slice.c:38 +0x98 fp=0x7fc88b846aa8
loveoneanother.at/tiedot/file.OpenCol(0xc2100368a0, 0x2f, 0x2, 0xc2100368a0, 0x2f)
/root/shockwave/src/loveoneanother.at/tiedot/file/col.go:32 +0x77 fp=0x7fc88b846b18
loveoneanother.at/tiedot/db.OpenCol(0xc210036780, 0x2a, 0xc21001ec30, 0x0, 0x0)
/root/shockwave/src/loveoneanother.at/tiedot/db/col.go:56 +0x30c fp=0x7fc88b846bf8
loveoneanother.at/tiedot/db.OpenDB(0xc210036450, 0x21, 0xc21004c180, 0x0, 0x0)
/root/shockwave/src/loveoneanother.at/tiedot/db/db.go:30 +0x2bb fp=0x7fc88b846d10
main.main()
/root/shockwave/databank.go:355 +0x149 fp=0x7fc88b846f48
runtime.main()
/usr/local/go/src/pkg/runtime/proc.c:215 +0x11f fp=0x7fc88b846fa0
runtime.goexit()
/usr/local/go/src/pkg/runtime/proc.c:1389 fp=0x7fc88b846fa8

goroutine 3 [syscall]:
os/signal.loop()
/usr/local/go/src/pkg/os/signal/signal_unix.go:21 +0x1e
created by os/signal.init·1
/usr/local/go/src/pkg/os/signal/signal_unix.go:27 +0x31

Can we work on reducing the memory footprint of tiedot per collection?

upstart question

Hi,

I use tiedot (happly) in embed mode, all works fine, but If I try to run the app proxed by ngnix and runned by upstart tiedot crashes with a "slice bounds out of range" probably due to a data race. Any ideas/clues?

thanks

Teo.

logs:

panic: runtime error: slice bounds out of range

goroutine 1 [running]:
runtime.panic(0x2c5378, 0x68ca72)
        /home/pi/go/src/pkg/runtime/panic.c:264 +0x134
github.com/HouzuoGuo/tiedot/file.Open(0x109dc780, 0x10, 0x8000000, 0x0, 0x109c59c0, ...)
        /home/pi/gopath/parmi.go/src/github.com/HouzuoGuo/tiedot/file/file.go:60 +0x4a4
github.com/HouzuoGuo/tiedot/file.OpenHash(0x109dc780, 0x10, 0xe, 0x0, 0xc8, ...)
        /home/pi/gopath/parmi.go/src/github.com/HouzuoGuo/tiedot/file/hash.go:35 +0xa8
github.com/HouzuoGuo/tiedot/db.(*Col).LoadConf(0x109c58d0, 0x0, 0x0)
        /home/pi/gopath/parmi.go/src/github.com/HouzuoGuo/tiedot/db/col.go:115 +0x40c
github.com/HouzuoGuo/tiedot/db.OpenCol(0x109c2840, 0xb, 0x109c58d0, 0x0, 0x0)
        /home/pi/gopath/parmi.go/src/github.com/HouzuoGuo/tiedot/db/col.go:66 +0x394
github.com/HouzuoGuo/tiedot/db.OpenDB(0x2f43b0, 0x2, 0x109c27e0, 0x0, 0x0)
        /home/pi/gopath/parmi.go/src/github.com/HouzuoGuo/tiedot/db/db.go:32 +0x288
github.com/tejo/blog/comments.(*CommentStore).Init(0x109cb7e0, 0x0, 0x0)
        /home/pi/gopath/parmi.go/src/github.com/tejo/blog/comments/comments.go:80 +0x54
main.main()
        /home/pi/gopath/parmi.go/src/github.com/tejo/blog/blog.go:52 +0x19c

Mapped file buffers are not periodically synchronized with storage device

tiedot currently let operating system decide timing of file buffer synchronization.

Golang is not quite good at mmap, its standard library does not handle memory mapped files in a cross platform manner. msync is not even available in the standard library. But the common practice for a document database is to periodically (every 30 seconds or minute) flush file buffers, usually via msync call.

package loveoneanother.at/tiedot: unrecognized import path "loveoneanother.at/ti edot"

When using go get loveoneanother.at/tiedot, the cmd just finished with message:
package loveoneanother.at/tiedot: unrecognized import path "loveoneanother.at/ti edot"

And the code could not be downloaded, so what's wrong?

Investigate IO queuing and its benefits

Investigate how introducing IO queues can benefit tiedot's operation.

The IO queues will operate on:

• document CRUD
• hash bucket CRU

Difficulty storing id's in data

I have the following code:

package main

import (
"fmt"
"github.com/HouzuoGuo/tiedot/db"
"math/rand"
"os"
"time"
)

func main() {
fmt.Println("Hello World!")
rand.Seed(time.Now().UTC().UnixNano())
database, err := db.OpenDB("\tmp\sandbox1")
if err != nil {
fmt.Println("Database open error", err)
os.Exit(1)
}
// database.Create("names", 1)
// database.Create("ids", 1)
names := database.Use("names")
ids := database.Use("ids")
name := map[string]interface{}{"Name": "Ross Albertson", "Age": 45}
nameId, err := names.Insert(name)
if err != nil {
fmt.Println("Database insert error", err)
os.Exit(1)
}
fmt.Println("Created name id", nameId)
idDoc := map[string]interface{}{"NameId": nameId}
idDocId, err := ids.Insert(idDoc)
if err != nil {
fmt.Println("Database insert error", err)
os.Exit(1)
}
var doc map[string]interface{}
_, err = ids.Read(idDocId, &doc)
if err != nil {
fmt.Println("Database id read error", err)
os.Exit(1)
}
readId := uint64(doc["NameId"].(float64))
fmt.Println("Found id", readId, "in ids")
var doc2 map[string]interface{}
_, err = names.Read(readId, &doc2)
if err != nil {
fmt.Println("Database name read error", err)
os.Exit(1)
}
fmt.Println("Retrieved name", doc2)
}

I would think I would get the same "names" record that I stored. Instead, here is my output:

Hello World!
2014/02/12 09:57:26 Opening chunk \tmp\sandbox1/ids/chunk_0
2014/02/12 09:57:26 Opening collection data file _data
2014/02/12 09:57:26 Opening PK hash table file _pk
2014/02/12 09:57:26 \tmp\sandbox1/ids/chunk_0/_pk has 16384 buckets, and 5324800
bytes out of 8388608 bytes in-use
2014/02/12 09:57:26 Successfully opened collection ids
2014/02/12 09:57:26 Opening chunk \tmp\sandbox1/names/chunk_0
2014/02/12 09:57:26 Opening collection data file _data
2014/02/12 09:57:26 Opening PK hash table file _pk
2014/02/12 09:57:26 \tmp\sandbox1/names/chunk_0/_pk has 16384 buckets, and 53248
00 bytes out of 8388608 bytes in-use
2014/02/12 09:57:26 Successfully opened collection names
Created name id 9170366255560425888
Found id 9170366255560425472 in ids
Database name read error Cannot find physical ID of 9170366255560425472
exit status 1

Notice the uint64 I got back isn't the same as what I stored. Is this my fault, a Go compiler issue, or a Tiedot issue?

Support RPC as API

For some usages, the additional overhead of HTTP protocol can be a waste of bandwidth and CPU cycles. Let's support RPC as well.

Provide better redundancy by continuous data replication

Continuous data replication seems to be industrial best practice at improving both scalability and availability of NoSQL database solutions.

And this should be a wonderful programming exercise.

Make a web control panel

Go has excellent built-in support for web development, therefore it makes a lot of sense to develop a tiedot web control panel using Go.

Proposed features:

• Overview of server performance and databases
• Document management
• Index management
• Query executor

Out of memory error running bench3 with 500000 records

Hi,

I was looking into tiedot and was running the benchmark with UID's with 500000 as benchsize (./tiedot --mode=bench3 --benchsize=500000). I get this error:

runtime: memory allocated by OS (0xb0514000) not in usable range [0x11407000,0x91407000)
fatal error: out of memory

Is this a bug or a known limitation due to the use of memory mapped files?

No Windows??? BAD CODER!!!

Hi there,
I tried to run this on window... to no avail. Basically, Go for windows does not support mmap and its siblings.

Not being the type that backs up that easily, I set to try to solve it (Bear in mind, I had Go installed locally because I found a nice tutorial few days ago, but haven't actually used it and have zero experience with the language).

I found online a Go package that is supposed to make mmap work in windows:

https://github.com/edsrzf/mmap-go

It didn't work out right out of the oven, so I tinkered with it and - several hours later - I had a working server running under windows :)

Still haven't used it so can't tell it it really works (I went as far as creating a collection, and the benchmark doesn't work). I haven't got any ide of how the prot flags work in Windows compared to *nix, and same for the flags, so I just put there what I thought was correct. Find below the patch for file.go

P.S.: HN brought me here.

diff --git a/src/loveoneanother.at/tiedot/file/file.go b/src/loveoneanother.at/tiedot/file/file.go
index ccb5a83..3170314 100644
--- a/src/loveoneanother.at/tiedot/file/file.go
+++ b/src/loveoneanother.at/tiedot/file/file.go
@@ -7,17 +7,24 @@ import (
    "log"
    "os"
    "sync"
-   "syscall"
+    "loveoneanother.at/tiedot/mmap"
 )

 type File struct {
    Name                 string
    Fh                   *os.File
    Append, Size, Growth uint64
-   Buf                  []byte
+   Buf                  mmap.MMap
    Sync                 *sync.RWMutex
 }

+//  void *mmap(void *addr, size_t length, int prot, int flags, int fd, off_t offset);
+
+// func Mmap(fd int, offset int64, length int, prot int, flags int) (data []byte, err error)
+
+// func mmap(len int, prot, flags, hfile uintptr, off int64)
+
+//mmap.Mmap(int(file.Size), mmap.RDWR, 0, int(file.Fh.Fd()), 0 )
 // Open (create if non-exist) the file.
 func Open(name string, growth uint64) (file *File, err error) {
    if growth < 1 {
@@ -38,7 +45,7 @@ func Open(name string, growth uint64) (file *File, err error) {
    if file.Size == 0 {
        return file, file.Ensure(file.Growth)
    }
-   if file.Buf, err = syscall.Mmap(int(file.Fh.Fd()), 0, int(file.Size), syscall.PROT_READ|syscall.PROT_WRITE, syscall.MAP_SHARED); err != nil {
+   if file.Buf, err = mmap.MapoRegion(file.Fh, int(file.Size), mmap.RDWR, 0, 0); err != nil {
        return
    }
    // find append position
@@ -72,7 +79,7 @@ func (file *File) Ensure(more uint64) (err error) {
        return
    }
    if file.Buf != nil {
-       if err = syscall.Munmap(file.Buf); err != nil {
+       if err = file.Buf.Unmap(); err != nil {
            return
        }
    }
@@ -85,9 +92,9 @@ func (file *File) Ensure(more uint64) (err error) {
    if err = file.Fh.Sync(); err != nil {
        return
    }
-   if newSize := int(file.Size + file.Growth); newSize < 0 {
+    if newSize := int(file.Size + file.Growth); newSize < 0 {
        log.Panicf("File %s is getting too large", file.Name)
-   } else if file.Buf, err = syscall.Mmap(int(file.Fh.Fd()), 0, newSize, syscall.PROT_READ|syscall.PROT_WRITE, syscall.MAP_SHARED); err != nil {
+   } else if file.Buf, err = mmap.MapoRegion(file.Fh, newSize, mmap.RDWR, 0, 0); err != nil {
        return
    }
    file.Size += file.Growth
@@ -97,7 +104,7 @@ func (file *File) Ensure(more uint64) (err error) {

 // Close the file.
 func (file *File) Close() (err error) {
-   if err = syscall.Munmap(file.Buf); err != nil {
+   if err = file.Buf.Unmap(); err != nil {
        return
    }
    return file.Fh.Close()

I think we should show more example for query.

I am a new guy for document db.So it's really difficult for me to run a query.So I think we should give more for new guys
thanks

Show more benchmark results in wiki

farmdawgnation suggested (and I whole heartedly agree) that the following benchmark results will be interesting to look at (and useful to know):

• how tiedot performs when mapped region size is greater than available memory size
• performance comparison with mainstream NoSQL solutions.

Make a -verbose flag to control display of verbose log messages

Less important log messages will be silenced by -verbose=false.

By default, -verbose is true.

struct (un)marshal

Hey, tiedot's great.

It would be good if tiedot supported direct struct marshal/unmarshal. There's no reason it shouldn't, as encoding/json supports it, and that's what tiedot's using.

I see that the reason the interface has map[string]interface{} is to add the id field. Is this necessary? Or, could reflection be used? (structs could include the id field).

Move DB dump implementation out of HTTP API

To keep logics separated from API, DB dump implementation should be moved out of API implementation.

Create a func db.Dump(dest string) and invoke it from HTTP API.

Document tiedot's stance on ACID properties

Currently the wiki pages make no mention about the tiedot's ACID properties.

The answer will be similar to MongoDB VS ACID, but should be documented.

Regular Expression Operator

I'm new to document databases and I wondered whether regular expression support would be useful. You could always do this on the client side, but a large result set could be slow (large amount of data being returned to client over network connection).

If the regex was implemented on the server side, wouldn't that be more efficient?

I'm enjoying tiedot, thanks!

Online database dump (backup)

Just another nice-to-have feature.

Why 130MB Initial Bucket on windows ?

Just curios why tiedot need to create initial 130MB file for _uid & data on windows. That is 260MB wasted without a inserting any documents.

is this performance ok on my virtualbox?

my laptop is i7 2.8G, 8G memory(3G for virtualbox), host OS win7, client OS linux mint 3.2.0-23-generic #36-Ubuntu SMP.

this is my test log:

$ tiedot -mode=bench
2013/08/14 12:36:01 GOMAXPROCS is set to 8
2013/08/14 12:36:13 File /tmp/tiedot_bench/data has grown 134217728 bytes
2013/08/14 12:36:15 File /tmp/tiedot_bench/a,b,c1662212681 has grown 67108864 bytes
2013/08/14 12:36:17 File /tmp/tiedot_bench/a,b,c1662212681 has grown 67108864 bytes
2013/08/14 12:36:19 File /tmp/tiedot_bench/c,d2035745710 has grown 67108864 bytes
2013/08/14 12:36:22 File /tmp/tiedot_bench/c,d2035745710 has grown 67108864 bytes
insert 400000: 746327 ns/iter, 1339 iter/sec
read 400000: 9181 ns/iter, 108916 iter/sec
lookup 400000: 32909 ns/iter, 30386 iter/sec
update 400000: 1775335 ns/iter, 563 iter/sec
delete 400000: 1009520 ns/iter, 990 iter/sec

is this normal? seems very poor.. and CPU seems not reach 100%, is this caused by some I/O bandwidth issue?

Query for latest articles by date/time

If I have 1000 objects (articles) and every article has an "Created" date property, which is UTC serialized date/time string (e.g. "2014-01-11T12:16:11.38920392Z")

1.) How can I create query that will return 10 most recent articles?
2.) How can I return all articles from certain month (e.g. November 2014)?

What would be the best approach to use to be able to do 1.) and 2.) ?

Properly replicate the repository under the domain name, to support "go get"

package is currently hosted on github only, which means "go get" will not work.

I better set it up properly on my domain host "loveoneanother.at"

I can't Install, please can someone help me?

I have a virtualbox with o.s arch linux, but I don't know where I have to install, I did it:

1. created tiedot folder on Desktop, I did export GOPATH='pwd' and then go get...
   when I did go get I looked two msj: open pws/src/loveoneanother.at/tiedot/gommap/mmap.go: No such file or directory,
   the same messaje for /pws/src/loveoneanother.at/tiedot/uid/uid.go
2. when I wrote ./bin/tiedot ... don´t work

Can Someone help me?

Thanks

does tiedot support concurrnet access？

like insert two doc in two goroutines at same time?

What license are you releasing this as?

Are you releasing this as your choice of Public Domain or BSD?

Is anybody attending FOSDEM 14 in Brussels?

Hello!

Is anybody attending FOSDEM 14 in Brussels? I will talk about tiedot as a lightning talk topic.

Check this out:

https://fosdem.org/2014/schedule/track/go/

How to contribute to tiedot

Hi.
I'm beginner in Go programming, but I'd like to contribute to this awesome project.
I don't really know how to manage project bigger than few files in Go.
I've confront a problem. I've cloned tiedot, but in main.go there are imports pointing to your repository.
What I have to do to manage this (I think that changing these import to my repository isn't good idea - when committing and merging these lines will override good ones).
How to control that?

dependence error of loveoneanther.at

package loveoneanther.at/tiedot/db: unrecognized import path "loveoneanther.at/tiedot/db"

Order?

Currently, there is no way to order queries. This would be great.

It probably requires a different type of index (as is described for better range queries).

Support collection join and aggregation methods

tiedot could become much more useful when join and aggregation are introduced!

How about having a join query like this:

{"join": "collection 1", "with": "collection 2", "left": "name", "right": "person_name", "limit": 12}

And an aggregation like this:

{"aggregate": "collection 1", "by": ["field1", "field2"], "fun": "sum"}

Limited scalability on Intel Xeon CPUs

tiedot seems to scale well enough on conventional desktop/workstation/laptop CPUs, but it shows very limited scalability on Intel Xeon CPUs.