salortiz / lmdb_file Goto Github PK

Usage: LMDB_File::_del(txn, dbi, key, data)

Usage scenario

# Results in: Usage: LMDB_File::_del(txn, dbi, key, data)
$txn->del($db->{dbi}, $pos);

# Same behavior
LMDB_File::_del($txn, $db->{dbi}, $pos);

Expected behavior:

If the database does not support sorted duplicate data items, (MDB_DUPSORT) the $data parameter is optional and is ignored.

say "mdb notls is " .MDB_NOTLS . ' and nometaasync ' .MDB_NOMETASYNC;
$Env->set_flags(
    MDB_NOMETASYNC | MDB_NOTLS,
    1
  );

  my $flags;
  p $envs->{$name}->get_flags($flags);
  say "flags are ";
  p $flags;

mdb notls is 2097152 and nometaasync 262144
0
flags are 
0

Works fine when setting in constructor:

my $Env =  LMDB::Env->new($dbPath, {
      mapsize => 1024 * 1024 * 1024 * 1024, # Plenty space, don't worry
      #maxdbs => 20, # Some databases
      mode   => 0755,
      flags => MDB_NOMETASYNC | MDB_NOTLS,
  });

flags are 
2359296

Interestingly, MDB_NOMETASYNC works in set_flags:

$Env->set_flags(
    MDB_NOMETASYNC ,
    1
  );
$Env->get_flags($flags);

flags are 
262144

Same issue with MDB_RDONLY

use strict;
use warnings;

use LMDB_File qw(:envflags :cursor_op);

use File::Temp;
my $dir = File::Temp::tempdir( CLEANUP => 1 );

my $d = "é";
my $u = "ü";
utf8::upgrade $u;

my $path = "$dir/file";

{
    my %hash;
    my $db = tie %hash, 'LMDB_File', "$dir/file", MDB_NOSUBDIR;

    $hash{$d} = $d;
    $hash{$u} = $u;
}

use Data::Dumper;
{
    my %hash;
    my $db = tie %hash, 'LMDB_File', "$dir/file", MDB_NOSUBDIR;

    print Dumper \%hash;
}

… prints:

$VAR1 = {
          'ü' => 'ü',
          'é' => 'é'
        };

The UTF8 option toggles between SvPV and SvPVutf8; it should actually toggle between SvPVbyte and SvPVutf8 to avoid a spurious dependency on Perl’s internal string storage logic.

My use case below. "Transaction Terminated" returned when $txn->renew() called. I don't think this is the expected behavior, because transaction termination should only occur if there are no remaining references to the transaction, according to LMDB_File.pm, but there clearly are, below.

There may be a better way of doing what I aim: Reuse a read-only transaction, in a read-only environment, across multiple dbRead function calls.

Thanks!

sub dbRead {
 my $db = $_[0]-> _getDbi($_[1]);

  my $txn;

  if(defined $db->{rdOnlyTxn} ) {
    $txn = $db->{rdOnlyTxn};

   ## Results in "Transaction Terminated"
    $txn->renew();
  }
}

sub _getDbi {
 my $env = LMDB::Env->new($dbPath, {
    ...
 });

 if(! $env ) {
   # handle
 }

 my $txn = $env->BeginTxn();

 my $DB = $txn->OpenDB();

 my $err = $txn->commit();
 # Check error

 $envs->{$name} = {env => $env, dbi => $DB->dbi};

 if($dbReadOnly) {
     $envs->{$name}{rdOnlyTxn} = $env->BeginTxn(MDB_RDONLY);
 }

 return  $envs->{$name};
}

Same thing occurs in a version where the new transaction is given to $DB->Txn instead, and the entire $DB reference is stored in $envs.

$envs->{$name} = {env => $env, dbi => $DB->dbi, DB => $DB};

 if($dbReadOnly) {
     $DB->Txn = $env->BeginTxn(MDB_RDONLY);
 }

Line 383 of LMDB_File.pm shows the expected behavior, no croak:

This is a bug because it subverts the consistency expected from get operations. Either both the high-level and low-level interface should croak, or neither. I vote neither, because in my experience dying for MDB_NOTFOUND is never the behavior I want; when data is not found I know it by less destructive means.

sub get {
    warn "get: '$_[1]'\n" if $DEBUG > 2;
    my($txn, $dbi) = _chkalive($_[0]);
    return _get($txn, $dbi, $_[1], $_[2]) if @_ > 2;
    my($res, $data);
    {
    local $die_on_err = 0;
    $res = _get($txn, $dbi, $_[1], $data);
    }
    croak($@) if $res && $die_on_err && $res != MDB_NOTFOUND() or undef $@;
    $data;
}

If making many databases, get deadlock (linux error 35), without NOTLS.

Example case attached.

Run on Mac OSX Mojave ( 10.14.0 ).

Could you help me understand whether this is an LMDB or LMDB_File issue? Related issue here: bmatsuo/lmdb-go#94

deadlock.pl.zip

Hello

https://www.cpantesters.org/cpan/report/d0083274-3128-11ee-adf2-4199e865f37b

/usr/bin/ld: LMDB.o: in function `XS_LMDB__Txn__dbi_open':
LMDB.c:(.text+0x7c0a): undefined reference to `Perl_do_vecget'
/usr/bin/ld: LMDB.o: in function `XS_LMDB__Cursor__del':
LMDB.c:(.text+0x9127): undefined reference to `Perl_do_vecget'
/usr/bin/ld: LMDB.o: in function `XS_LMDB_File__put':
LMDB.c:(.text+0x9c2d): undefined reference to `Perl_do_vecget'
/usr/bin/ld: LMDB.o: in function `XS_LMDB_File__get':
LMDB.c:(.text+0xab65): undefined reference to `Perl_do_vecget'
/usr/bin/ld: LMDB.o: in function `XS_LMDB_File__dcmp':
LMDB.c:(.text+0xb766): undefined reference to `Perl_do_vecget'
/usr/bin/ld: LMDB.o:LMDB.c:(.text+0xc256): more undefined references to `Perl_do_vecget' follow
/usr/bin/ld: blib/arch/auto/LMDB_File/LMDB_File.so: hidden symbol `Perl_do_vecget' isn't defined
/usr/bin/ld: final link failed: bad value
collect2: error: ld returned 1 exit status

Hello, I'm trying to package your library to Fedora but under ppc64, ppc64le tests failed. Do you support these architectures?

+ make test
"/usr/bin/perl" -MExtUtils::Command::MM -e 'cp_nonempty' -- LMDB_File.bs blib/arch/auto/LMDB_File/LMDB_File.bs 644
PERL_DL_NONLAZY=1 "/usr/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/00-basic.t ........ ok
#   Failed test 'Default psize'
#   at t/01-environment.t line 58.
#          got: '32768'
#     expected: '4096'
# Looks like you failed 1 test of 175.
t/01-environment.t .. 
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/175 subtests 
t/02-named.t ........ ok
MDB_MAP_FULL: Environment mapsize limit reached at /builddir/build/BUILD/LMDB_File-0.12/blib/lib/LMDB_File.pm line 370.
# Looks like your test exited with 255 just after 3.
t/03-fastmode.t ..... 
Dubious, test returned 255 (wstat 65280, 0xff00)
Failed 50/53 subtests 
t/04-bigone.t ....... ok
Test Summary Report
-------------------
t/01-environment.t (Wstat: 256 Tests: 175 Failed: 1)
  Failed test:  29
  Non-zero exit status: 1
t/03-fastmode.t   (Wstat: 65280 Tests: 3 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 53 tests but ran 3.
Files=5, Tests=245,  7 wallclock secs ( 0.05 usr  0.01 sys +  5.71 cusr  0.06 csys =  5.83 CPU)
Result: FAIL
Failed 2/5 test programs. 1/245 subtests failed.
make: *** [Makefile:963: test_dynamic] Error 255

You can look at https://koji.fedoraproject.org/koji/taskinfo?taskID=22677624 and https://koji.fedoraproject.org/koji/taskinfo?taskID=22677626

Thanks for reply.

I get the errors:

LMDB.o:LMDB.c:(.text+0xb98): undefined reference to `MIN'
LMDB.o:LMDB.c:(.text+0xba4): undefined reference to `MIN'
LMDB.o:LMDB.c:(.text+0x2c46): undefined reference to `__imp_Perl_do_vecget'
...
LMDB.o:LMDB.c:(.text+0xaaaf): more undefined references to `__imp_Perl_do_vecget' follow

Perl environment below. I did a little playing/googling and I couldn't see why Perl_do_vecget wouldn't exist. LMDB Seemed to build correctly under Alien.

Summary of my perl5 (revision 5 version 24 subversion 1) configuration:

  Platform:
    osname=MSWin32, osvers=6.3, archname=MSWin32-x64-multi-thread
    uname='Win32 strawberry-perl 5.24.1.1 #1 Mon Jan 16 02:00:29 2017 x64'
    config_args='undef'
    hint=recommended, useposix=true, d_sigaction=undef
    useithreads=define, usemultiplicity=define
    use64bitint=define, use64bitall=undef, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='gcc', ccflags =' -s -O2 -DWIN32 -DWIN64 -DCONSERVATIVE  -DPERL_TEXTMODE_SCRIPTS -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -fwrapv -fno-strict-aliasing -mms-bitfields',
    optimize='-s -O2',
    cppflags='-DWIN32'
    ccversion='', gccversion='4.9.2', gccosandvers=''
    intsize=4, longsize=4, ptrsize=8, doublesize=8, byteorder=12345678, doublekind=3
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16, longdblkind=3
    ivtype='long long', ivsize=8, nvtype='double', nvsize=8, Off_t='long long', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='g++', ldflags ='-s -L"C:\STRAWB~1\perl\lib\CORE" -L"C:\STRAWB~1\c\lib"'
    libpth=C:\STRAWB~1\c\lib C:\STRAWB~1\c\x86_64-w64-mingw32\lib C:\STRAWB~1\c\lib\gcc\x86_64-w64-mingw32\4.9.2
    libs=-lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32 -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32 -lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32
    perllibs=-lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32 -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32 -lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32
    libc=, so=dll, useshrplib=true, libperl=libperl524.a
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_win32.xs, dlext=xs.dll, d_dlsymun=undef, ccdlflags=' '
    cccdlflags=' ', lddlflags='-mdll -s -L"C:\STRAWB~1\perl\lib\CORE" -L"C:\STRAWB~1\c\lib"'


Characteristics of this binary (from libperl):
  Compile-time options: HAS_TIMES HAVE_INTERP_INTERN MULTIPLICITY
                        PERLIO_LAYERS PERL_COPY_ON_WRITE
                        PERL_DONT_CREATE_GVSV
                        PERL_HASH_FUNC_ONE_AT_A_TIME_HARD
                        PERL_IMPLICIT_CONTEXT PERL_IMPLICIT_SYS
                        PERL_MALLOC_WRAP PERL_PRESERVE_IVUV USE_64_BIT_INT
                        USE_ITHREADS USE_LARGE_FILES USE_LOCALE
                        USE_LOCALE_COLLATE USE_LOCALE_CTYPE
                        USE_LOCALE_NUMERIC USE_LOCALE_TIME USE_PERLIO
                        USE_PERL_ATOF
  Built under MSWin32
  Compiled at Jan 16 2017 02:12:13
  @INC:
    C:/Strawberry/perl/site/lib
    C:/Strawberry/perl/vendor/lib
    C:/Strawberry/perl/lib
    .

I especially need 73a4bef. Thanks!

Makefile.PL contains this:

if($Config{archname} =~ /686/) {
    warn "liblmdb isn't supported in your platform, sorry.\n";
    exit 0;
}                                   

But liblmdb evidently builds fine on 32-bit archs including i686: https://buildd.debian.org/status/package.php?p=liblmdb-file-perl

Please drop that check - or at least adapt it to only apply for locally compiling library, not for pre-compiled system-shared library which if it exists obviously is possible.

Behavior: Simply calling get brings RES and SHR up to the size of the database on disk.

Example: My database is 3.6G. I read every key. Somewhere during that process, I/O wait time drops to 0% and stays there (read ahead). At that point RES = 3.6g SHR = 3.6g. Over the course of the rest of the run, RES grows to 3.7G. This could be a rounding issue.

This is a simple case with no chance of a closed over variable maintaining a reference. This behavior is seen with the database with flags MDB_NOTLS | MDB_NOMETASYNC | MDB_NOLOCK | MDB_NOSYNC | MDB_RDONLY ( I think NOLOCK pretty much obviates all others), and with no flags set.

This occurs both with MDB_RDONLY and not.

Code:

for my $posKey (0 .. $lastPosKey) {
  $db->dbReadOne('databaseName', $posKey);
}

sub dbReadOne {
  my ($self, $dbName, $key) = @_;
  my $db = $self->_getDbi($dbName);

  my $txn = $db->{env}->BeginTxn(MDB_RDONLY);

  $txn->get($db->{dbi}, $key, my $data);
  
  return undef;
}

Using the speedy transaction interface seems to lead to a memory leak. Iterating over millions of positions leads to gigabytes of reserved memory, even when no hanging reference to the data is possible.

  my $db = $self->_getDbi($chr);
  my $dbi = $db->{dbi};

  my $txn = $db->{env}->BeginTxn(MDB_RDONLY);

  my $json;

  for my $pos ( @allPositions ) {

    # Undefined values allowed

    #zero-copy
    $txn->get($dbi, $pos, undef);
}
$txn->commit();

In linux kernels where fdatasync is broken the 'Flags setted' test in 01-environment.t will fail.
This is because there is an additional private flag set 'MDB_FSYNCONLY' not just MDB_ENV_ACTIVE | MDB_ENV_TXKEY.

Issue:

If you create a parent transaction using LMDB::Txn->new(), then a child transaction using Env->BeginTxn(), no problem.

The reverse order utterly breaks: using the parent transaction causes MDB_BAD_TXN error.

Not certain if this is an issue with my understanding, or LMDB_File.

I've attached a test case, and below

Edit: Similar issues arise with read_only transactions, except it becomes impossible to have two simultaneous transactions. This seems strange to me: as LMDB transactions are isolated there should be no issue with N transactions per thread.

• I think this is actually correct: I believe LMDB does not allow any read-only sub transactions.
  • NO_TLS should remove this restriction
• Is there a way to issue several, non-sub transactions, without NO_TLS.
• Is the problem that only one transaction per thread can have access to one dbi (like a file handle) at a time?

use 5.10.0;
use strict;
use warnings;
package DBManager;
use LMDB_File qw/:all/;
my %envs;

sub new { 
  my $class = shift; 
  my $self = { }; 
  bless $self, $class; 
} 

sub dbPut {
  my ($self,$dbName, $key, $data, $skipCommit) = @_;

  # 0 to create database if not found
  my $db = $self->_getDbi($dbName);

  if(!$db->{db}->Alive) {
    $db->{db}->Txn = $db->{env}->BeginTxn();
    # not strictly necessary, but I am concerned about hard to trace abort bugs related to scope
    $db->{db}->Txn->AutoCommit(1);
  }

  $db->{db}->Txn->put($db->{dbi}, $key, $data);

  $db->{db}->Txn->commit() unless $skipCommit;

  if($LMDB_File::last_err) {
    if($LMDB_File::last_err != MDB_KEYEXIST) {
      die $LMDB_File::last_err;
    }

    #reset the class error variable, to avoid crazy error reporting later
    $LMDB_File::last_err = 0;
  }

  return 0;
}

sub dbReadOne {
  my ($self, $dbName, $key, $skipCommit) = @_;

  my $db = $self->_getDbi($dbName) or return undef;

  if(!$db->{db}->Alive) {
    $db->{db}->Txn = $db->{env}->BeginTxn();
    # not strictly necessary, but I am concerned about hard to trace abort bugs related to scope
    $db->{db}->Txn->AutoCommit(1);
  }

  $db->{db}->Txn->get($db->{dbi}, $key, my $data);

  # Commit unless the user specifically asks not to
  #if(!$skipCommit) {
  $db->{db}->Txn->commit() unless $skipCommit;

  if($LMDB_File::last_err) {
    if($LMDB_File::last_err != MDB_NOTFOUND ) {
      die $LMDB_File::last_err;
    }

    $LMDB_File::last_err = 0;
  }

  return $data;
}


sub dbStartCursorTxn {
  my ($self, $dbName) = @_;

  my $db = $self->_getDbi($dbName) or return;

  my $txn = $db->{env}->BeginTxn();

  # Help LMDB_File track our cursor
  LMDB::Cursor::open($txn, $db->{dbi}, my $cursor);

  # Unsafe, private LMDB_File method access but Cursor::open does not track cursors
  $LMDB::Txn::Txns{$$txn}{Cursors}{$$cursor} = 1;

  return [$txn, $cursor];
}

sub _getDbi {
  # Exists and not defined, because in read only database we may discover
  # that some chromosomes don't have any data (example: hg38 refSeq chrM)
  

  #   $_[0]  $_[1], $_[2]
  # Don't create used by dbGetNumberOfEntries
  my ($self, $dbPath) = @_;

  if ($envs{$dbPath}) {
    return $envs{$dbPath};
  }

  my $env = LMDB::Env->new($dbPath, {
    mapsize => 128 * 1024 * 1024 * 1024, # Plenty space, don't worry
    #maxdbs => 20, # Some databases
    mode   => 0600,
    maxdbs => 0, # Some databases; else we get a MDB_DBS_FULL error (max db limit reached)
  });

  if(! $env ) {
    die 'No env';
  }

  my $txn = $env->BeginTxn();

  my $dbFlags;

  my $DB = $txn->OpenDB(undef, MDB_INTEGERKEY);

  # ReadMode 1 gives memory pointer for perf reasons, not safe
  $DB->ReadMode(1);

  if($LMDB_File::last_err) {
    die $LMDB_File::last_err;
  }

  # Now db is open
  my $err = $txn->commit();

  if($err) {
    die $err;
  }

  $envs{$dbPath} = {env => $env, dbi => $DB->dbi, db => $DB};

  return $envs{$dbPath};
}

1;

use Test::More;
use DDP;
my $db = DBManager->new();

my $dbIdx = 1;
my $pos = 99;
my $val = "HELLO WORLD";

system('rm -rf ./test && mkdir ./test');

#### WORKS GREAT ####
my $cursor;
$cursor = $db->dbStartCursorTxn('test');

### Test Unsafe Transactions (Manually Managed) ##########
$db->dbPut('test', $pos, [], 1);

$db->dbReadOne('test', $pos);

p %LMDB::Env::Envs;

$db->dbReadOne('test', $pos);
undef $db;
undef $cursor;

system('rm -rf ./test && mkdir ./test');

$db = DBManager->new();
#### DIES MISERABLE DEATH ####
say "The reverse order doesn't work";

$db->dbPut('test', $pos, [], 1);
$cursor = $db->dbStartCursorTxn('test');
$db->dbReadOne('test', $pos);

say "We will never see this";

subtxn_bug.pl.zip

There is no documentation for a way to use LMDB::Cursor directly (not via LMDB_File object, aka $DB->Cursor). It would be nice to have this feature exposed.

Would invoking LMDB::Cursor::new($txn,$dbi,my $cursor) be the way to do this? However, how would one call mdb_cursor_close, renew, etc ?

I suppose in general it would be nice to have the ability to access basic LMDB methods directly, because this would avoid the need to document all of the syntactic sugar (use LMDB C documentation).

with the stabilization of 5.24 this bug fix should probably be deployed: 3f3858d

Documentation states $DB->Txn can be used as lvalue. This works when assigning $Env->BeginTxn() but not when assigning LMDB::Txn->new(). $DB->Txn is undef in the latter case.

I think this is part of the larger general issue of documentation (#22). We have low-level methods, but there are few examples (and few tests) for their use. The low-level API is also incomplete, with no direct access to Cursor (./t/04-bigone.t shows how to use the underlying _get method, which is undocumented, but no ability to use Cursor outside of LMDB_File objects).

I think it would be more idiomatic to have access to mdb_* methods. They are already present in the XS package. Is there any way to use them directly? If not, would it be hard to expose them?

https://github.com/salortiz/LMDB_File/blob/328a572c5db9c8edfa40e7269bd8d1fdf738b377/lib/LMDB_File.pm#L148C27-L148C27

As tflags are documented as optional in the costructor of LMDB::Txn ($Txn = LMDB::Txn->new ( $Env [, $tflags ] )) the $tflags constructor argument should be defaulted to 0 to avoid the tedious warning about uninitialized value in bitwise.

After the v0.11 release that includes @rouzier's work some CPAN testers are failing, see:
http://matrix.cpantesters.org/?dist=LMDB_File+0.11

I'm suspecting we needs to test for the availability of the symbol in lmdb library so that code can be conditionally compiled.

@rouzier, can you take a look, please?

I think being able to do $DB->set_compare(sub {}) is a really neat feature, thanks for getting that working!

However, some of my comparison functions could be implemented a lot more efficiently with a simple C function instead of a perl callback. I could make an XS module return a bare function pointer in an SV, is there an easy way to install that with mdb_set_compare, or some other way to accomplish this?

Thanks!

Would be useful to have a flag to set the default auto commit mode to commit instead of abort.

I could see this being done in several places, such as BeginTxn(), LMDB::Env->new, or as a property on $DB.

This saves the overhead of having to set the autocommit mode every time a new transaction is created, which in our case happens billions of times per database.

Hi Salvador,

I just tried to install 0.09 and I got this error from cpanm:

Configuring LMDB_File-0.09
Running Makefile.PL
Warning (mostly harmless): No library found for -llmdb
Clone lmdb from its repo and put a copy/link in 'liblmdb' directory at Makefile.PL line 34.
-> N/A
-> FAIL Configure failed for LMDB_File-0.09. See /home/doug/.cpanm/work/1455036782.31765/build.log for details.

It looks like we're not bundling lmdb with the cpan distribution anymore:

$ tar tf LMDB_File-0.07.tar.gz | grep lmdb.h
LMDB_File-0.07/liblmdb/lmdb.h
$ tar tf LMDB_File-0.09.tar.gz | grep lmdb.h
$

Is this intentional? After I did apt-get install liblmdb-dev it built and tested fine.

Thanks,

Doug

When trying to create more than one transaction, I receive either undef (if using $env->BeginTxn()), or "Transaction active, should be sub transaction" when using LMDB::Txn->new.

my @txns;
  foreach (0 .. $maxReaders - 1) {
    my $txn = LMDB::Txn->new ( $env, MDB_RDONLY );

    $txn->reset();

    push @txns, $txna;
  }