GithubHelp home page GithubHelp logo

erigontech / mdbx-go Goto Github PK

View Code? Open in Web Editor NEW
60.0 5.0 20.0 6.34 MB

Bindings for https://gitflic.ru/project/erthink/libmdbx/

License: Apache License 2.0

Makefile 0.37% Go 3.29% C 74.50% CMake 1.95% Roff 0.30% C++ 19.60%

mdbx-go's Introduction

mdbx-go

Go bindings to the libmdbx: https://libmdbx.dqdkfa.ru

Most of articles in internet about LMDB are applicable to MDBX. But mdbx has more features.

For deeper DB understanding please read through mdbx.h

Min Requirements

C language Compilers compatible with GCC or CLANG (mingw 10 on windows) Golang: 1.15

Packages

Functionality is logically divided into several packages. Applications will usually need to import mdbx but may import other packages on an as needed basis.

Packages in the exp/ directory are not stable and may change without warning. That said, they are generally usable if application dependencies are managed and pinned by tag/commit.

Developers concerned with package stability should consult the documentation.

mdbx GoDoc stable

import "github.com/torquem-ch/mdbx-go/mdbx"

Core bindings allowing low-level access to MDBX.

exp/mdbxpool GoDoc experimental

import "github.com/torquem-ch/mdbx-go/exp/mdbxpool"

A utility package which facilitates reuse of mdbx.Txn objects using a sync.Pool. Naively storing mdbx.Txn objects in sync.Pool can be troublesome. And the mdbxpool.TxnPool type has been defined as a complete pooling solution and as reference for applications attempting to write their own pooling implementation.

The mdbxpool package is relatively new. But it has a lot of potential utility. And once the mdbxpool API has been ironed out, and the implementation hardened through use by real applications it can be integrated directly into the mdbx package for more transparent integration. Please test this package and provide feedback to speed this process up.

Key Features

Idiomatic API

API inspired by BoltDB with automatic commit/rollback of transactions. The goal of mdbx-go is to provide idiomatic database interactions without compromising the flexibility of the C API.

NOTE: While the mdbx package tries hard to make MDBX as easy to use as possible there are compromises, gotchas, and caveats that application developers must be aware of when relying on MDBX to store their data. All users are encouraged to fully read the documentation so they are aware of these caveats. And even better if read through mdbx.h

High Performance notices

Applications with high performance requirements can opt-in to fast, zero-copy reads at the cost of runtime safety. Zero-copy behavior is specified at the transaction level to reduce instrumentation overhead.

err := mdbx.View(func(txn *mdbx.Txn) error {
    // RawRead enables zero-copy behavior with some serious caveats.
    // Read the documentation carefully before using.
    txn.RawRead = true

    val, err := txn.Get(dbi, []byte("largevalue"), 0)
    // ...
})

Use NoReadahead if Data > RAM

Advantages of BoltDB

  • Nested databases allow for hierarchical data organization.

  • Far more databases can be accessed concurrently.

  • No Bucket object - means less allocations and higher performance

  • Operating systems that do not support sparse files do not use up excessive space due to a large pre-allocation of file space.

  • As a pure Go package bolt can be easily cross-compiled using the go toolchain and GOOS/GOARCH variables.

  • Its simpler design and implementation in pure Go mean it is free of many caveats and gotchas which are present using the MDBX package. For more information about caveats with the MDBX package, consult its documentation so they are aware of these caveats. And even better if read through mdbx.h.

Advantages of LMDB over BoltDB

  • Keys can contain multiple values using the DupSort flag.

  • Updates can have sub-updates for atomic batching of changes.

  • Databases typically remain open for the application lifetime. This limits the number of concurrently accessible databases. But, this minimizes the overhead of database accesses and typically produces cleaner code than an equivalent BoltDB implementation.

  • Significantly faster than BoltDB. The raw speed of MDBX easily surpasses BoltDB. Additionally, MDBX provides optimizations ranging from safe, feature-specific optimizations to generally unsafe, extremely situational ones. Applications are free to enable any optimizations that fit their data, access, and reliability models.

  • MDBX allows multiple applications to access a database simultaneously. Updates from concurrent processes are synchronized using a database lock file.

  • As a C library, applications in any language can interact with MDBX databases. Mission critical Go applications can use a database while Python scripts perform analysis on the side.

Advantages of MDBX over LMDB

See in mdbx's readme.md

Build

On FreeBSD 10, you must explicitly set CC (otherwise it will fail with a cryptic error), for example:

CC=clang go test -v ./...

Maintainance, update .c code

In MDBX repo: make dist && cp -R ./dist/* ./../mdbx-go/mdbxdist/. Then in mdbx-go repo: make cp

Build binaries

In mdbx-go repo: MDBX_BUILD_TIMESTAMP=unknown make tools

Or if use mdbx-go as a library:

go mod vendor && cd vendor/github.com/torquem-ch/mdbx-go && make tools 
rm -rf vendor

Documentation

Versioning and Stability

The mdbx-go project makes regular releases with IDs X.Y.Z. All packages outside of the exp/ directory are considered stable and adhere to the guidelines of semantic versioning.

Experimental packages (those packages in exp/) are not required to adhere to semantic versioning. However packages specifically declared to merely be "unstable" can be relied on more for long term use with less concern.

The API of an unstable package may change in subtle ways between minor release versions. But deprecations will be indicated at least one release in advance and all functionality will remain available through some method.

mdbx-go's People

Contributors

alexeyakhunov avatar askalexsharov avatar battlmonstr avatar canepat avatar du5 avatar elee1766 avatar jackmeng1985 avatar lgtm-com[bot] avatar missinglink avatar racytech avatar yihuang avatar yperbasis avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

mdbx-go's Issues

Sequence returns value prior to increment?

Hi,

Apologies if I'm reading this wrong, it seems to me that the Sequence() return value is equal to the md_seq before the increment and not after?

Shouldn't we be reading the value of new?

func (txn *Txn) Sequence(dbi DBI, increment uint64) (uint64, error) {
	var res C.uint64_t
	ret := C.mdbx_dbi_sequence(txn._txn, C.MDBX_dbi(dbi), &res, C.uint64_t(increment))
	if ret != 0 {
		return uint64(res), operrno("mdbx_dbi_sequence", ret)
	}
	return uint64(res), nil
}

Screenshot 2022-04-29 at 10 42 53

Great work! appreciate if can emphasize this section....

  1. what's the cost of runtime safety?
  2. how much faster? 2x? 3x? ( just wondering if worth while to implement over safety issues)

Zero-copy reads
Applications with high performance requirements can opt-in to fast, zero-copy reads at the cost of runtime safety. Zero-copy behavior is specified at the transaction level to reduce instrumentation overhead.

mdbx_env_open: MDBX_BUSY

opening the same path twice results in an MDBX_BUSY error despite not passing the MDBX_EXCLUSIVE flag 🤷‍♂️

panic: mdbx_env_open: MDBX_BUSY: Another write transaction is running, 
or environment is already used while opening with MDBX_EXCLUSIVE flag

in the example below there is neither a write transaction running, nor were either opened with the MDBX_EXCLUSIVE flag:

package main

import (
	"fmt"
	"io/ioutil"

	"github.com/torquem-ch/mdbx-go/mdbx"
)

func check(err error) {
	if err != nil {
		panic(err)
	}
}

func main() {
	// create a temporary directory
	tmpdir, err := ioutil.TempDir("", "mdbx")
	check(err)
	datpath := fmt.Sprintf("%s/foobar.db", tmpdir)
	fmt.Println("datpath:", datpath)

	// open environment
	env1, err := mdbx.NewEnv()
	check(err)
	check(env1.Open(datpath, mdbx.NoSubdir, 444))
	flag1, err := env1.Flags()
	check(err)
	fmt.Println("exclusive:", flag1&mdbx.Exclusive)

	// open new environment
	env2, err := mdbx.NewEnv()
	check(err)
	check(env2.Open(datpath, mdbx.NoSubdir, 444))
}
cat go.sum
github.com/torquem-ch/mdbx-go v0.23.2 h1:7axXl0leix2v8No+mRzeTV32hJrV1817aKhh+hTEpC8=
github.com/torquem-ch/mdbx-go v0.23.2/go.mod h1:T2fsoJDVppxfAPTLd1svUgH1kpPmeXdPESmroSHcL1E=

mdbx-go crashes under windows

System information

which the exact version of Windows you're on ?
I have tried multiple versions of windows and it will reproduce,This crash has been in the version starting in July
Windows 11 21H2 22000.120 1000.22000.120.0

May I ask which version of GO are you on ?
go version go1.16.7 windows/amd64

how much free disk space you have on C: ?
My free disk space have on C: is 1.9T (PM9A1 2T PCIE4.0 NVME m.2)

how much RAM on your system ?
64.0 GB

are you running on a virtualized machine or physical ?
Physical machine

Expected behaviour
run

Actual behaviour
panic

Steps to reproduce the behaviour
just run erigon.exe at windows

Backtrace
image
Microsoft Windows [版本 10.0.22000.120]
(c) Microsoft Corporation。保留所有权利。

c:\weup\src\github.com\ledgerwatch\mdbx-go\mdbx>go test
Exception 0xc0000005 0x8 0x0 0x0
PC=0x0

runtime: unknown pc 0x356f55
stack: frame={sp:0xb21f1ffc80, fp:0x0} stack=[0x0,0xb21f1ffda0)
000000b21f1ffb80: 0000000000000330 0000000000000000
000000b21f1ffb90: 0000000b00000040 0000000ffa66b000
000000b21f1ffba0: 0000000e1220a000 000000125a66b000
000000b21f1ffbb0: 0000000fcf78a000 00007ffffffe0000
000000b21f1ffbc0: 00007ffed3c6f000 0000000000000000
000000b21f1ffbd0: 0000000002030000 0000000000000000
000000b21f1ffbe0: 0000000000000000 00000007fffffff0
000000b21f1ffbf0: 00007ffd827c8e50 000000c0000ac510
000000b21f1ffc00: 0000026cd04e9510 00007ffd827c8ee9
000000b21f1ffc10: 000000c000089de0 00000000b4bbbdff
000000b21f1ffc20: 0002625a00000000 00ffa66b00001000
000000b21f1ffc30: 0102f2ff00000001 0000000000010000
000000b21f1ffc40: 0000000000ffa66b 0000000000e1220a
000000b21f1ffc50: 00000000ffffffff 0000026cd04e93d0
000000b21f1ffc60: 0000b79ce51fd316 000000c0000ac510
000000b21f1ffc70: 000000c0000ac510 0000000000356f55
000000b21f1ffc80: <00000000001f776f <runtime.semawakeup+47> 0000026cd04e93d0
000000b21f1ffc90: 0000000000000000 0000000000000000
000000b21f1ffca0: 00000000004a01c8 0000000000379b80
000000b21f1ffcb0: 0000100000000009 0000000000010000
000000b21f1ffcc0: 00007ffffffeffff 00000000ffffffff
000000b21f1ffcd0: 000021d800000020 2100001900010000
000000b21f1ffce0: 000000c000089e20 00000000004d65c0
000000b21f1ffcf0: 000000c00008a000 0000000000314aea
000000b21f1ffd00: 000000000022a88a <runtime.callers.func1+138> 00000000002ad51f <testing.callerName+95>
000000b21f1ffd10: 000000c000089f40 0000000000000000
000000b21f1ffd20: 000000c000089e20 00000000004d65c0
000000b21f1ffd30: 000000c000089e20 0000000000230e93 <runtime.asmcgocall+115>
000000b21f1ffd40: 0000000000000000 0000000000000000
000000b21f1ffd50: 0000000000000000 0000000000000001
000000b21f1ffd60: 000000c000089ee0 0000000000000250
000000b21f1ffd70: 000000c000075380 0000000000201440 <runtime.mstart+0>
runtime: unknown pc 0x356f55
stack: frame={sp:0xb21f1ffc80, fp:0x0} stack=[0x0,0xb21f1ffda0)
000000b21f1ffb80: 0000000000000330 0000000000000000
000000b21f1ffb90: 0000000b00000040 0000000ffa66b000
000000b21f1ffba0: 0000000e1220a000 000000125a66b000
000000b21f1ffbb0: 0000000fcf78a000 00007ffffffe0000
000000b21f1ffbc0: 00007ffed3c6f000 0000000000000000
000000b21f1ffbd0: 0000000002030000 0000000000000000
000000b21f1ffbe0: 0000000000000000 00000007fffffff0
000000b21f1ffbf0: 00007ffd827c8e50 000000c0000ac510
000000b21f1ffc00: 0000026cd04e9510 00007ffd827c8ee9
000000b21f1ffc10: 000000c000089de0 00000000b4bbbdff
000000b21f1ffc20: 0002625a00000000 00ffa66b00001000
000000b21f1ffc30: 0102f2ff00000001 0000000000010000
000000b21f1ffc40: 0000000000ffa66b 0000000000e1220a
000000b21f1ffc50: 00000000ffffffff 0000026cd04e93d0
000000b21f1ffc60: 0000b79ce51fd316 000000c0000ac510
000000b21f1ffc70: 000000c0000ac510 0000000000356f55
000000b21f1ffc80: <00000000001f776f <runtime.semawakeup+47> 0000026cd04e93d0
000000b21f1ffc90: 0000000000000000 0000000000000000
000000b21f1ffca0: 00000000004a01c8 0000000000379b80
000000b21f1ffcb0: 0000100000000009 0000000000010000
000000b21f1ffcc0: 00007ffffffeffff 00000000ffffffff
000000b21f1ffcd0: 000021d800000020 2100001900010000
000000b21f1ffce0: 000000c000089e20 00000000004d65c0
000000b21f1ffcf0: 000000c00008a000 0000000000314aea
000000b21f1ffd00: 000000000022a88a <runtime.callers.func1+138> 00000000002ad51f <testing.callerName+95>
000000b21f1ffd10: 000000c000089f40 0000000000000000
000000b21f1ffd20: 000000c000089e20 00000000004d65c0
000000b21f1ffd30: 000000c000089e20 0000000000230e93 <runtime.asmcgocall+115>
000000b21f1ffd40: 0000000000000000 0000000000000000
000000b21f1ffd50: 0000000000000000 0000000000000001
000000b21f1ffd60: 000000c000089ee0 0000000000000250
000000b21f1ffd70: 000000c000075380 0000000000201440 <runtime.mstart+0>

goroutine 6 [syscall]:
github.com/torquem-ch/mdbx-go/mdbx._Cfunc_mdbx_env_create(0xc0000ac510, 0xc000000000)
_cgo_gotypes.go:620 +0x4b
github.com/torquem-ch/mdbx-go/mdbx.NewEnv.func1(0xc0000ac510, 0xc0000ac510)
c:/weup/src/github.com/ledgerwatch/mdbx-go/mdbx/env.go:137 +0x66
github.com/torquem-ch/mdbx-go/mdbx.NewEnv(0x2ad766, 0x4b1970, 0x41f37d)
c:/weup/src/github.com/ledgerwatch/mdbx-go/mdbx/env.go:137 +0x4b
github.com/torquem-ch/mdbx-go/mdbx.setupFlags(0x40ba30, 0xc000075200, 0x0, 0x379b80)
c:/weup/src/github.com/ledgerwatch/mdbx-go/mdbx/env_test.go:496 +0x2d
github.com/torquem-ch/mdbx-go/mdbx.setup(...)
c:/weup/src/github.com/ledgerwatch/mdbx-go/mdbx/env_test.go:492
github.com/torquem-ch/mdbx-go/mdbx.TestCursor_Txn(0xc000075200)
c:/weup/src/github.com/ledgerwatch/mdbx-go/mdbx/cursor_test.go:15 +0x4b
testing.tRunner(0xc000075200, 0x3dec60)
C:/Program Files/Go/src/testing/testing.go:1193 +0xef
created by testing.(*T).Run
C:/Program Files/Go/src/testing/testing.go:1238 +0x2b3

goroutine 1 [chan receive]:
testing.(*T).Run(0xc000075200, 0x3d4375, 0xe, 0x3dec60, 0x251bed)
C:/Program Files/Go/src/testing/testing.go:1239 +0x2da
testing.runTests.func1(0xc000075080)
C:/Program Files/Go/src/testing/testing.go:1511 +0x7f
testing.tRunner(0xc000075080, 0xc0000bdde0)
C:/Program Files/Go/src/testing/testing.go:1193 +0xef
testing.runTests(0xc000004078, 0x38d460, 0x3d, 0x3d, 0xc03d0ad69a213fa8, 0x8bb3012681, 0x4d6300, 0x3d4109)
C:/Program Files/Go/src/testing/testing.go:1509 +0x310
testing.(*M).Run(0xc0000dc000, 0x0)
C:/Program Files/Go/src/testing/testing.go:1417 +0x1f5
main.main()
_testmain.go:187 +0x145
rax 0x0
rbx 0x26cd04e93d0
rcx 0x26cd04e9558
rdi 0xc0000ac510
rsi 0x7ffd827c8e50
rbp 0xc000089de0
rsp 0xb21f1ffc78
r8 0x2d
r9 0x3e
r10 0x0
r11 0x3ef6e5f10e4be44
r12 0x404ed0
r13 0x0
r14 0x0
r15 0x2030000
rip 0x0
rflags 0x10202
cs 0x33
fs 0x53
gs 0x2b
exit status 2
FAIL github.com/torquem-ch/mdbx-go/mdbx 0.523s

c:\weup\src\github.com\ledgerwatch\mdbx-go\mdbx>

more examples of the api would be great, error in a simple demo

package main

import (
	"fmt"
	"github.com/torquem-ch/mdbx-go/mdbx"
	"os"
)

func main() {
	env, err := mdbx.NewEnv()
	if err != nil {
		fmt.Fprintf(os.Stderr, "failed to create mdbx env: %s\n", err)
		os.Exit(1)
	}

	env.SetOption(mdbx.OptMaxDB, 20)

	err = env.Open("/tmp/test-mdbx", mdbx.Coalesce|mdbx.LifoReclaim|mdbx.WriteMap, 0664)
	if err != nil {
		fmt.Fprintf(os.Stderr, "failed to open the env: %s\n", err)
		os.Exit(1)
	}

	idx := 0

	updateDbFunc := func(txn *mdbx.Txn) error {
		dbi, err := txn.CreateDBI("hello")
		if err != nil {
			fmt.Fprintf(os.Stderr, "failed to create dbi: %s\n", err)
			os.Exit(1)
		}

		for i := 0; i < 1024*1024*2; i++ {
			key := fmt.Sprintf("abcdef%d", idx)
			idx += 1

			err = txn.Put(dbi, []byte(key), []byte("xyzxyz"), 0)
			if err != nil {
				// some errors
				fmt.Fprintf(os.Stderr, "failed to write to db: %s\n", err)
				os.Exit(1)
			}
		}

		env.CloseDBI(dbi)

		return nil
	}
	env.Update(updateDbFunc)

	env.Close()
}

In this simple demo, I got errors:

failed to write to db: mdbx_put: MDBX_MAP_FULL: Environment mapsize limit reached

Would someone help illuminate which parat I'm missing?

problem importing

go mod tidy
go: finding module for package github.com/torquem-ch/mdbx-go/mdbx
go: found github.com/torquem-ch/mdbx-go/mdbx in github.com/torquem-ch/mdbx-go v0.36.2
go: github.com/blockinterface/prospect/database imports
        github.com/torquem-ch/mdbx-go/mdbx: github.com/torquem-ch/[email protected]: parsing go.mod:
        module declares its path as: github.com/erigontech/mdbx-go
                but was required as: github.com/torquem-ch/mdbx-go
xxx@node1:~/projects/psp$ go get github.com/torquem-ch/mdbx-go/mdbx

What is the best way for store long values?

when I want to store the values which abount 100K ~ 1000K, on my ssd disk the benchmark put function is less than 1000 per second.

this is my mdbx env.Open flag :

mdbx.Accede | mdbx.UtterlyNoSync | mdbx.Coalesce | mdbx.NoReadahead

key, val = 20 bytes:

BenchmarkMdbxStorage_Set-8   	 1152529	      1884 ns/op	     144 B/op	       6 allocs/op

key = 20bytes, val = 512K bytes

BenchmarkMdbxStorage_Set-8   	     720	   1521572 ns/op	     141 B/op	       6 allocs/op

So, when I store 1 large item, it will take more than 1ms, what is the best way to use mdbx store 100K ~ 1000K item?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.