salortiz / lmdb_file Goto Github PK
View Code? Open in Web Editor NEWPerl wrapper around the OpenLDAP's LMDB
License: Other
Perl wrapper around the OpenLDAP's LMDB
License: Other
Usage: LMDB_File::_del(txn, dbi, key, data)
Usage scenario
# Results in: Usage: LMDB_File::_del(txn, dbi, key, data)
$txn->del($db->{dbi}, $pos);
# Same behavior
LMDB_File::_del($txn, $db->{dbi}, $pos);
If the database does not support sorted duplicate data items, (MDB_DUPSORT) the $data parameter is optional and is ignored.
say "mdb notls is " .MDB_NOTLS . ' and nometaasync ' .MDB_NOMETASYNC;
$Env->set_flags(
MDB_NOMETASYNC | MDB_NOTLS,
1
);
my $flags;
p $envs->{$name}->get_flags($flags);
say "flags are ";
p $flags;
mdb notls is 2097152 and nometaasync 262144
0
flags are
0
Works fine when setting in constructor:
my $Env = LMDB::Env->new($dbPath, {
mapsize => 1024 * 1024 * 1024 * 1024, # Plenty space, don't worry
#maxdbs => 20, # Some databases
mode => 0755,
flags => MDB_NOMETASYNC | MDB_NOTLS,
});
flags are
2359296
Interestingly, MDB_NOMETASYNC works in set_flags:
$Env->set_flags(
MDB_NOMETASYNC ,
1
);
$Env->get_flags($flags);
flags are
262144
Same issue with MDB_RDONLY
use strict;
use warnings;
use LMDB_File qw(:envflags :cursor_op);
use File::Temp;
my $dir = File::Temp::tempdir( CLEANUP => 1 );
my $d = "é";
my $u = "ü";
utf8::upgrade $u;
my $path = "$dir/file";
{
my %hash;
my $db = tie %hash, 'LMDB_File', "$dir/file", MDB_NOSUBDIR;
$hash{$d} = $d;
$hash{$u} = $u;
}
use Data::Dumper;
{
my %hash;
my $db = tie %hash, 'LMDB_File', "$dir/file", MDB_NOSUBDIR;
print Dumper \%hash;
}
… prints:
$VAR1 = {
'ü' => 'ü',
'é' => 'é'
};
The UTF8 option toggles between SvPV and SvPVutf8; it should actually toggle between SvPVbyte and SvPVutf8 to avoid a spurious dependency on Perl’s internal string storage logic.
My use case below. "Transaction Terminated" returned when $txn->renew() called. I don't think this is the expected behavior, because transaction termination should only occur if there are no remaining references to the transaction, according to LMDB_File.pm, but there clearly are, below.
There may be a better way of doing what I aim: Reuse a read-only transaction, in a read-only environment, across multiple dbRead function calls.
Thanks!
sub dbRead {
my $db = $_[0]-> _getDbi($_[1]);
my $txn;
if(defined $db->{rdOnlyTxn} ) {
$txn = $db->{rdOnlyTxn};
## Results in "Transaction Terminated"
$txn->renew();
}
}
sub _getDbi {
my $env = LMDB::Env->new($dbPath, {
...
});
if(! $env ) {
# handle
}
my $txn = $env->BeginTxn();
my $DB = $txn->OpenDB();
my $err = $txn->commit();
# Check error
$envs->{$name} = {env => $env, dbi => $DB->dbi};
if($dbReadOnly) {
$envs->{$name}{rdOnlyTxn} = $env->BeginTxn(MDB_RDONLY);
}
return $envs->{$name};
}
Same thing occurs in a version where the new transaction is given to $DB->Txn instead, and the entire $DB reference is stored in $envs.
$envs->{$name} = {env => $env, dbi => $DB->dbi, DB => $DB};
if($dbReadOnly) {
$DB->Txn = $env->BeginTxn(MDB_RDONLY);
}
Line 383 of LMDB_File.pm shows the expected behavior, no croak:
This is a bug because it subverts the consistency expected from get operations. Either both the high-level and low-level interface should croak, or neither. I vote neither, because in my experience dying for MDB_NOTFOUND is never the behavior I want; when data is not found I know it by less destructive means.
sub get {
warn "get: '$_[1]'\n" if $DEBUG > 2;
my($txn, $dbi) = _chkalive($_[0]);
return _get($txn, $dbi, $_[1], $_[2]) if @_ > 2;
my($res, $data);
{
local $die_on_err = 0;
$res = _get($txn, $dbi, $_[1], $data);
}
croak($@) if $res && $die_on_err && $res != MDB_NOTFOUND() or undef $@;
$data;
}
If making many databases, get deadlock (linux error 35), without NOTLS.
Example case attached.
Run on Mac OSX Mojave ( 10.14.0 ).
Could you help me understand whether this is an LMDB or LMDB_File issue? Related issue here: bmatsuo/lmdb-go#94
Hello
https://www.cpantesters.org/cpan/report/d0083274-3128-11ee-adf2-4199e865f37b
/usr/bin/ld: LMDB.o: in function `XS_LMDB__Txn__dbi_open':
LMDB.c:(.text+0x7c0a): undefined reference to `Perl_do_vecget'
/usr/bin/ld: LMDB.o: in function `XS_LMDB__Cursor__del':
LMDB.c:(.text+0x9127): undefined reference to `Perl_do_vecget'
/usr/bin/ld: LMDB.o: in function `XS_LMDB_File__put':
LMDB.c:(.text+0x9c2d): undefined reference to `Perl_do_vecget'
/usr/bin/ld: LMDB.o: in function `XS_LMDB_File__get':
LMDB.c:(.text+0xab65): undefined reference to `Perl_do_vecget'
/usr/bin/ld: LMDB.o: in function `XS_LMDB_File__dcmp':
LMDB.c:(.text+0xb766): undefined reference to `Perl_do_vecget'
/usr/bin/ld: LMDB.o:LMDB.c:(.text+0xc256): more undefined references to `Perl_do_vecget' follow
/usr/bin/ld: blib/arch/auto/LMDB_File/LMDB_File.so: hidden symbol `Perl_do_vecget' isn't defined
/usr/bin/ld: final link failed: bad value
collect2: error: ld returned 1 exit status
Hello, I'm trying to package your library to Fedora but under ppc64, ppc64le tests failed. Do you support these architectures?
+ make test
"/usr/bin/perl" -MExtUtils::Command::MM -e 'cp_nonempty' -- LMDB_File.bs blib/arch/auto/LMDB_File/LMDB_File.bs 644
PERL_DL_NONLAZY=1 "/usr/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/00-basic.t ........ ok
# Failed test 'Default psize'
# at t/01-environment.t line 58.
# got: '32768'
# expected: '4096'
# Looks like you failed 1 test of 175.
t/01-environment.t ..
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/175 subtests
t/02-named.t ........ ok
MDB_MAP_FULL: Environment mapsize limit reached at /builddir/build/BUILD/LMDB_File-0.12/blib/lib/LMDB_File.pm line 370.
# Looks like your test exited with 255 just after 3.
t/03-fastmode.t .....
Dubious, test returned 255 (wstat 65280, 0xff00)
Failed 50/53 subtests
t/04-bigone.t ....... ok
Test Summary Report
-------------------
t/01-environment.t (Wstat: 256 Tests: 175 Failed: 1)
Failed test: 29
Non-zero exit status: 1
t/03-fastmode.t (Wstat: 65280 Tests: 3 Failed: 0)
Non-zero exit status: 255
Parse errors: Bad plan. You planned 53 tests but ran 3.
Files=5, Tests=245, 7 wallclock secs ( 0.05 usr 0.01 sys + 5.71 cusr 0.06 csys = 5.83 CPU)
Result: FAIL
Failed 2/5 test programs. 1/245 subtests failed.
make: *** [Makefile:963: test_dynamic] Error 255
You can look at https://koji.fedoraproject.org/koji/taskinfo?taskID=22677624 and https://koji.fedoraproject.org/koji/taskinfo?taskID=22677626
Thanks for reply.
I get the errors:
LMDB.o:LMDB.c:(.text+0xb98): undefined reference to `MIN'
LMDB.o:LMDB.c:(.text+0xba4): undefined reference to `MIN'
LMDB.o:LMDB.c:(.text+0x2c46): undefined reference to `__imp_Perl_do_vecget'
...
LMDB.o:LMDB.c:(.text+0xaaaf): more undefined references to `__imp_Perl_do_vecget' follow
Perl environment below. I did a little playing/googling and I couldn't see why Perl_do_vecget wouldn't exist. LMDB Seemed to build correctly under Alien.
Summary of my perl5 (revision 5 version 24 subversion 1) configuration:
Platform:
osname=MSWin32, osvers=6.3, archname=MSWin32-x64-multi-thread
uname='Win32 strawberry-perl 5.24.1.1 #1 Mon Jan 16 02:00:29 2017 x64'
config_args='undef'
hint=recommended, useposix=true, d_sigaction=undef
useithreads=define, usemultiplicity=define
use64bitint=define, use64bitall=undef, uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='gcc', ccflags =' -s -O2 -DWIN32 -DWIN64 -DCONSERVATIVE -DPERL_TEXTMODE_SCRIPTS -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -fwrapv -fno-strict-aliasing -mms-bitfields',
optimize='-s -O2',
cppflags='-DWIN32'
ccversion='', gccversion='4.9.2', gccosandvers=''
intsize=4, longsize=4, ptrsize=8, doublesize=8, byteorder=12345678, doublekind=3
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16, longdblkind=3
ivtype='long long', ivsize=8, nvtype='double', nvsize=8, Off_t='long long', lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries:
ld='g++', ldflags ='-s -L"C:\STRAWB~1\perl\lib\CORE" -L"C:\STRAWB~1\c\lib"'
libpth=C:\STRAWB~1\c\lib C:\STRAWB~1\c\x86_64-w64-mingw32\lib C:\STRAWB~1\c\lib\gcc\x86_64-w64-mingw32\4.9.2
libs=-lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32 -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32 -lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32
perllibs=-lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32 -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32 -lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32
libc=, so=dll, useshrplib=true, libperl=libperl524.a
gnulibc_version=''
Dynamic Linking:
dlsrc=dl_win32.xs, dlext=xs.dll, d_dlsymun=undef, ccdlflags=' '
cccdlflags=' ', lddlflags='-mdll -s -L"C:\STRAWB~1\perl\lib\CORE" -L"C:\STRAWB~1\c\lib"'
Characteristics of this binary (from libperl):
Compile-time options: HAS_TIMES HAVE_INTERP_INTERN MULTIPLICITY
PERLIO_LAYERS PERL_COPY_ON_WRITE
PERL_DONT_CREATE_GVSV
PERL_HASH_FUNC_ONE_AT_A_TIME_HARD
PERL_IMPLICIT_CONTEXT PERL_IMPLICIT_SYS
PERL_MALLOC_WRAP PERL_PRESERVE_IVUV USE_64_BIT_INT
USE_ITHREADS USE_LARGE_FILES USE_LOCALE
USE_LOCALE_COLLATE USE_LOCALE_CTYPE
USE_LOCALE_NUMERIC USE_LOCALE_TIME USE_PERLIO
USE_PERL_ATOF
Built under MSWin32
Compiled at Jan 16 2017 02:12:13
@INC:
C:/Strawberry/perl/site/lib
C:/Strawberry/perl/vendor/lib
C:/Strawberry/perl/lib
.
I especially need 73a4bef. Thanks!
Makefile.PL contains this:
if($Config{archname} =~ /686/) {
warn "liblmdb isn't supported in your platform, sorry.\n";
exit 0;
}
But liblmdb evidently builds fine on 32-bit archs including i686: https://buildd.debian.org/status/package.php?p=liblmdb-file-perl
Please drop that check - or at least adapt it to only apply for locally compiling library, not for pre-compiled system-shared library which if it exists obviously is possible.
Behavior: Simply calling get brings RES and SHR up to the size of the database on disk.
Example: My database is 3.6G. I read every key. Somewhere during that process, I/O wait time drops to 0% and stays there (read ahead). At that point RES = 3.6g SHR = 3.6g. Over the course of the rest of the run, RES grows to 3.7G. This could be a rounding issue.
This is a simple case with no chance of a closed over variable maintaining a reference. This behavior is seen with the database with flags MDB_NOTLS | MDB_NOMETASYNC | MDB_NOLOCK | MDB_NOSYNC | MDB_RDONLY ( I think NOLOCK pretty much obviates all others), and with no flags set.
This occurs both with MDB_RDONLY and not.
Code:
for my $posKey (0 .. $lastPosKey) {
$db->dbReadOne('databaseName', $posKey);
}
sub dbReadOne {
my ($self, $dbName, $key) = @_;
my $db = $self->_getDbi($dbName);
my $txn = $db->{env}->BeginTxn(MDB_RDONLY);
$txn->get($db->{dbi}, $key, my $data);
return undef;
}
Using the speedy transaction interface seems to lead to a memory leak. Iterating over millions of positions leads to gigabytes of reserved memory, even when no hanging reference to the data is possible.
my $db = $self->_getDbi($chr);
my $dbi = $db->{dbi};
my $txn = $db->{env}->BeginTxn(MDB_RDONLY);
my $json;
for my $pos ( @allPositions ) {
# Undefined values allowed
#zero-copy
$txn->get($dbi, $pos, undef);
}
$txn->commit();
In linux kernels where fdatasync is broken the 'Flags setted' test in 01-environment.t will fail.
This is because there is an additional private flag set 'MDB_FSYNCONLY' not just MDB_ENV_ACTIVE | MDB_ENV_TXKEY.
Issue:
If you create a parent transaction using LMDB::Txn->new(), then a child transaction using Env->BeginTxn(), no problem.
The reverse order utterly breaks: using the parent transaction causes MDB_BAD_TXN error.
Not certain if this is an issue with my understanding, or LMDB_File.
I've attached a test case, and below
Edit: Similar issues arise with read_only transactions, except it becomes impossible to have two simultaneous transactions. This seems strange to me: as LMDB transactions are isolated there should be no issue with N transactions per thread.
use 5.10.0;
use strict;
use warnings;
package DBManager;
use LMDB_File qw/:all/;
my %envs;
sub new {
my $class = shift;
my $self = { };
bless $self, $class;
}
sub dbPut {
my ($self,$dbName, $key, $data, $skipCommit) = @_;
# 0 to create database if not found
my $db = $self->_getDbi($dbName);
if(!$db->{db}->Alive) {
$db->{db}->Txn = $db->{env}->BeginTxn();
# not strictly necessary, but I am concerned about hard to trace abort bugs related to scope
$db->{db}->Txn->AutoCommit(1);
}
$db->{db}->Txn->put($db->{dbi}, $key, $data);
$db->{db}->Txn->commit() unless $skipCommit;
if($LMDB_File::last_err) {
if($LMDB_File::last_err != MDB_KEYEXIST) {
die $LMDB_File::last_err;
}
#reset the class error variable, to avoid crazy error reporting later
$LMDB_File::last_err = 0;
}
return 0;
}
sub dbReadOne {
my ($self, $dbName, $key, $skipCommit) = @_;
my $db = $self->_getDbi($dbName) or return undef;
if(!$db->{db}->Alive) {
$db->{db}->Txn = $db->{env}->BeginTxn();
# not strictly necessary, but I am concerned about hard to trace abort bugs related to scope
$db->{db}->Txn->AutoCommit(1);
}
$db->{db}->Txn->get($db->{dbi}, $key, my $data);
# Commit unless the user specifically asks not to
#if(!$skipCommit) {
$db->{db}->Txn->commit() unless $skipCommit;
if($LMDB_File::last_err) {
if($LMDB_File::last_err != MDB_NOTFOUND ) {
die $LMDB_File::last_err;
}
$LMDB_File::last_err = 0;
}
return $data;
}
sub dbStartCursorTxn {
my ($self, $dbName) = @_;
my $db = $self->_getDbi($dbName) or return;
my $txn = $db->{env}->BeginTxn();
# Help LMDB_File track our cursor
LMDB::Cursor::open($txn, $db->{dbi}, my $cursor);
# Unsafe, private LMDB_File method access but Cursor::open does not track cursors
$LMDB::Txn::Txns{$$txn}{Cursors}{$$cursor} = 1;
return [$txn, $cursor];
}
sub _getDbi {
# Exists and not defined, because in read only database we may discover
# that some chromosomes don't have any data (example: hg38 refSeq chrM)
# $_[0] $_[1], $_[2]
# Don't create used by dbGetNumberOfEntries
my ($self, $dbPath) = @_;
if ($envs{$dbPath}) {
return $envs{$dbPath};
}
my $env = LMDB::Env->new($dbPath, {
mapsize => 128 * 1024 * 1024 * 1024, # Plenty space, don't worry
#maxdbs => 20, # Some databases
mode => 0600,
maxdbs => 0, # Some databases; else we get a MDB_DBS_FULL error (max db limit reached)
});
if(! $env ) {
die 'No env';
}
my $txn = $env->BeginTxn();
my $dbFlags;
my $DB = $txn->OpenDB(undef, MDB_INTEGERKEY);
# ReadMode 1 gives memory pointer for perf reasons, not safe
$DB->ReadMode(1);
if($LMDB_File::last_err) {
die $LMDB_File::last_err;
}
# Now db is open
my $err = $txn->commit();
if($err) {
die $err;
}
$envs{$dbPath} = {env => $env, dbi => $DB->dbi, db => $DB};
return $envs{$dbPath};
}
1;
use Test::More;
use DDP;
my $db = DBManager->new();
my $dbIdx = 1;
my $pos = 99;
my $val = "HELLO WORLD";
system('rm -rf ./test && mkdir ./test');
#### WORKS GREAT ####
my $cursor;
$cursor = $db->dbStartCursorTxn('test');
### Test Unsafe Transactions (Manually Managed) ##########
$db->dbPut('test', $pos, [], 1);
$db->dbReadOne('test', $pos);
p %LMDB::Env::Envs;
$db->dbReadOne('test', $pos);
undef $db;
undef $cursor;
system('rm -rf ./test && mkdir ./test');
$db = DBManager->new();
#### DIES MISERABLE DEATH ####
say "The reverse order doesn't work";
$db->dbPut('test', $pos, [], 1);
$cursor = $db->dbStartCursorTxn('test');
$db->dbReadOne('test', $pos);
say "We will never see this";
There is no documentation for a way to use LMDB::Cursor directly (not via LMDB_File object, aka $DB->Cursor). It would be nice to have this feature exposed.
Would invoking LMDB::Cursor::new($txn,$dbi,my $cursor) be the way to do this? However, how would one call mdb_cursor_close, renew, etc ?
I suppose in general it would be nice to have the ability to access basic LMDB methods directly, because this would avoid the need to document all of the syntactic sugar (use LMDB C documentation).
with the stabilization of 5.24 this bug fix should probably be deployed: 3f3858d
Documentation states $DB->Txn can be used as lvalue. This works when assigning $Env->BeginTxn() but not when assigning LMDB::Txn->new(). $DB->Txn is undef in the latter case.
I think this is part of the larger general issue of documentation (#22). We have low-level methods, but there are few examples (and few tests) for their use. The low-level API is also incomplete, with no direct access to Cursor (./t/04-bigone.t shows how to use the underlying _get method, which is undocumented, but no ability to use Cursor outside of LMDB_File objects).
I think it would be more idiomatic to have access to mdb_* methods. They are already present in the XS package. Is there any way to use them directly? If not, would it be hard to expose them?
As tflags
are documented as optional in the costructor of LMDB::Txn ($Txn = LMDB::Txn->new ( $Env [, $tflags ] )
) the $tflags
constructor argument should be defaulted to 0 to avoid the tedious warning about uninitialized value in bitwise.
After the v0.11 release that includes @rouzier's work some CPAN testers are failing, see:
http://matrix.cpantesters.org/?dist=LMDB_File+0.11
I'm suspecting we needs to test for the availability of the symbol in lmdb library so that code can be conditionally compiled.
@rouzier, can you take a look, please?
I think being able to do $DB->set_compare(sub {})
is a really neat feature, thanks for getting that working!
However, some of my comparison functions could be implemented a lot more efficiently with a simple C function instead of a perl callback. I could make an XS module return a bare function pointer in an SV, is there an easy way to install that with mdb_set_compare, or some other way to accomplish this?
Thanks!
Would be useful to have a flag to set the default auto commit mode to commit instead of abort.
I could see this being done in several places, such as BeginTxn(), LMDB::Env->new, or as a property on $DB.
This saves the overhead of having to set the autocommit mode every time a new transaction is created, which in our case happens billions of times per database.
Hi Salvador,
I just tried to install 0.09 and I got this error from cpanm:
Configuring LMDB_File-0.09 Running Makefile.PL Warning (mostly harmless): No library found for -llmdb Clone lmdb from its repo and put a copy/link in 'liblmdb' directory at Makefile.PL line 34. -> N/A -> FAIL Configure failed for LMDB_File-0.09. See /home/doug/.cpanm/work/1455036782.31765/build.log for details.
It looks like we're not bundling lmdb with the cpan distribution anymore:
$ tar tf LMDB_File-0.07.tar.gz | grep lmdb.h LMDB_File-0.07/liblmdb/lmdb.h $ tar tf LMDB_File-0.09.tar.gz | grep lmdb.h $
Is this intentional? After I did apt-get install liblmdb-dev
it built and tested fine.
Thanks,
Doug
When trying to create more than one transaction, I receive either undef (if using $env->BeginTxn()), or "Transaction active, should be sub transaction" when using LMDB::Txn->new.
my @txns;
foreach (0 .. $maxReaders - 1) {
my $txn = LMDB::Txn->new ( $env, MDB_RDONLY );
$txn->reset();
push @txns, $txna;
}
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.