yahoo / mdbm Goto Github PK
View Code? Open in Web Editor NEWThis project forked from timrc-git/mdbm
MDBM a very fast memory-mapped key/value store.
License: BSD 3-Clause "New" or "Revised" License
This project forked from timrc-git/mdbm
MDBM a very fast memory-mapped key/value store.
License: BSD 3-Clause "New" or "Revised" License
hugo@noib4h6jac:~/inst/mdbm$ make
make -C src default-make-target && make -C include default-make-target && /bin/true
make[1]: Entering directory `/home/hugo/inst/mdbm/src'
make -C lib default-make-target && make -C tools default-make-target && make -C test default-make-target && make -C scripts default-make-target && /bin/true
make[2]: Entering directory `/home/hugo/inst/mdbm/src/lib'
/bin/true
cc -D_FILE_OFFSET_BITS=64 -DUSE_OPENSSL -DALLOW_MLOCK_RESET -DHAVE_ROBUST_PTHREADS -g -O2 -I/usr/include -I/usr/local/include -I/home/hugo/inst/mdbm/include -I. -Wall -fPIC -DDISABLE_TSC -Wall -pedantic -Wno-variadic-macros -Wno-long-long -Wno-overlength-strings -Wno-format-security -Wno-unused-result -Werror -c mdbm.c -o object/mdbm.o
mdbm.c: In function ‘mdbm_sparsify_file’:
mdbm.c:9659:7: error: format ‘%lu’ expects argument of type ‘long unsigned int’, but argument 6 has type ‘off_t’ [-Werror=format]
mdbm.c:9666:9: error: format ‘%lu’ expects argument of type ‘long unsigned int’, but argument 6 has type ‘off_t’ [-Werror=format]
cc1: all warnings being treated as errors
make[2]: *** [object/mdbm.o] Error 1
make[2]: Leaving directory `/home/hugo/inst/mdbm/src/lib'
make[1]: *** [make-recursive-default-make-target] Error 2
make[1]: Leaving directory `/home/hugo/inst/mdbm/src'
make: *** [make-recursive-default-make-target] Error 2
removing -Werror from Makefile.base allows building mdbm sucessfull (after installing readline and cunit)
Curious, have you thought about running clang-format over the source? I find it refreshingly useful to automate indentation, since it takes the subjective aspect out of it (just agree on a .clang-format config).
I tried it quickly, and I have to say it looks a lot nicer :-).
Hi there.
I'm trying to make N threads write to the same DB, and I keep getting some strange errors.
I'm using latest MDBM from github on Ubuntu 14 x86_64.
I use following code:
#include <string.h>
#include <sys/fcntl.h>
#include <mdbm/mdbm.h>
#include <stdlib.h>
#include <pthread.h>
#include <thread>
#include <iostream>
#include <vector>
struct mdbm_key_t {
uint32_t k1;
};
pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;
int nthreads = 2;
int niterations = 100000;
void threadfunc(MDBM* db, int idx) {
std::thread::id tid = std::this_thread::get_id();
srand(time(NULL));
// main processing here
mdbm_key_t key;
uint32_t data1, data2;
datum d_key, d_val;
d_key.dptr = (char*)&key;
d_key.dsize = sizeof(key);
d_val.dptr = (char*)&key;
d_val.dsize = sizeof(key);
for(int i=0; i<niterations; i++) {
key.k1 = rand(); //%10000;
data2++;
//std::cout << "thread " << tid << " locking key " << key.k1 << " .." << std::endl;
if( mdbm_plock(db, &d_key, 0) != 1 ){
perror("mdbm_plock");
exit( EXIT_FAILURE );
}
//std::cout << "thread " << tid << " storing key " << key.k1 << " .." << std::endl;
if( mdbm_store(db,d_key,d_val,MDBM_INSERT) == -1 ) {
perror("mdbm_store");
exit(1);
}
//std::cout << "thread " << tid << " unlocking key " << key.k1 << " .." << std::endl;
if( mdbm_punlock(db, &d_key, 0) != 1 ) {
perror("mdbm_punlock");
exit( EXIT_FAILURE );
}
}
// end main processing
std::cout << "thread " << tid << " done" << std::endl;
}
int main(int argc, char **argv)
{
char fn[2048];
char buf[2048];
int ndups;
std::vector <std::thread> threads;
std::vector <MDBM*> pdbms;
if( argc == 3 ){
nthreads = atoi(argv[1]);
niterations = atoi(argv[2]);
}
std::cout << "nthreads = " << nthreads << ", niterations = " << niterations << std::endl;
snprintf(fn,sizeof(fn),"%s","example.mdbm");
MDBM* db = mdbm_open(fn,MDBM_O_RDWR|MDBM_O_CREAT|MDBM_SINGLE_ARCH|MDBM_PARTITIONED_LOCKS,0666,0,0);
if (!db) {
perror("Unable to create database");
exit( EXIT_FAILURE );
}
pthread_mutex_lock(&mutex);
for( int i=0; i<nthreads; i++ ){
MDBM* clone = mdbm_dup_handle(db, 0);
if( clone == NULL ){
perror("mdbm_dup_handle");
exit( EXIT_FAILURE );
}
pdbms.push_back( clone );
}
pthread_mutex_unlock(&mutex);
for( int i=0; i<nthreads; i++ ){
threads.emplace_back( threadfunc, pdbms[i], i );
//threads.emplace_back( threadfunc, db );
}
for( auto& thread : threads )
thread.join();
std::cout << "all threads done" << std::endl;
mdbm_close(db);
return 0;
and in 9 out of 10 runs I get errors like:
1:
3:55708863:528fc:07df8 mdbm_lock.cc:201 /home/adfox/adfox/mdbm/example.mdbm: another partition locking conflict (locked=? want=1) excl_nest=0, part_nest=1, part_count=128 : Operation not permitted
Begin Lock state (for non-zero locks) (pid:32244, tid:32248 self:7ffff65b5700 uuid:32248 ):
BaseLocks:1 Core:1 Shared/Partitioned:128 mode:MLOCK_INDEX (2)
End Lock state
mdbm_store: Operation not permitted
2: SEGFAULT in MDBM_PAGE_PTR (pagenum=268439516, db=0x60dee0)
Am I doing anything wrong? Or is it a bug?
Thanks!
I saw that people keep submitting PRs to fix the same things and you didn't accept any of them in more than 6 months.
Should we use timrc-git/mdbm instead?
Thanks,
Luiz.
Line 1189 in 87c20e2
I have looked at a few source files for your current software. I have noticed that some checks for return codes are missing.
Would you like to add more error handling for return values from functions like the following?
It's the fastest hash, and it's pretty good. Every intel 64 bit CPU should have it, and the new arm's also. See https://github.com/rurban/smhasher#smhasher
You just need to probe for it. Best try the one in -lz at first, and if it's the HW assisted one, not the SW, use that. Otherwise the one at https://github.com/rurban/smhasher/blob/master/crc32_hw.c
I've installed MDBM using make install
, but mvn verify -DnativeDir=../lib/object/ -DlibDir=object/
gives me the following error. I'm running this command with the config change specified in #81.
I guess -Djava.library.path=../lib/object/:object/
has some problem.
Configuring TestNG with: TestNG652Configurator
Cannot load mdbm_java native library, -Djava.library.path=../lib/object/:object/
start testDontCloseFromPool
finally testDontCloseFromPool
start testInvalidSize
start testInvalidSize
start testPool
opening /home/qubuntu/Documents/mdbm/src/java/test/resources/testv3.mdbm
Tests run: 49, Failures: 4, Errors: 0, Skipped: 19, Time elapsed: 0.379 sec <<< FAILURE! - in TestSuite
initEmptyMdbm(com.yahoo.db.mdbm.TestV4) Time elapsed: 0.014 sec <<< FAILURE!
java.lang.UnsatisfiedLinkError: no mdbm_java in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1122)
at com.yahoo.db.mdbm.internal.NativeMdbmAccess.<clinit>(NativeMdbmAccess.java:15)
at com.yahoo.db.mdbm.internal.NativeMdbmImplementation.mdbm_open(NativeMdbmImplementation.java:444)
at com.yahoo.db.mdbm.MdbmProvider.open(MdbmProvider.java:140)
at com.yahoo.db.mdbm.TestSimpleMdbm.initEmptyMdbm(TestSimpleMdbm.java:42)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84)
at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:564)
at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:213)
at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:138)
at org.testng.internal.TestMethodWorker.invokeBeforeClassMethods(TestMethodWorker.java:175)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:107)
at org.testng.TestRunner.privateRun(TestRunner.java:767)
at org.testng.TestRunner.run(TestRunner.java:617)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:329)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:291)
at org.testng.SuiteRunner.run(SuiteRunner.java:240)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1224)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1149)
at org.testng.TestNG.run(TestNG.java:1057)
at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:115)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeMulti(TestNGDirectoryTestSuite.java:204)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:107)
at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:112)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
testDontCloseFromPool(com.yahoo.db.mdbm.internal.TestMdbmPool) Time elapsed: 0 sec <<< FAILURE!
java.lang.NoClassDefFoundError: Could not initialize class com.yahoo.db.mdbm.internal.NativeMdbmAccess
at com.yahoo.db.mdbm.internal.NativeMdbmImplementation.mdbm_open(NativeMdbmImplementation.java:444)
at com.yahoo.db.mdbm.MdbmProvider.open(MdbmProvider.java:140)
at com.yahoo.db.mdbm.MdbmProvider.openPool(MdbmProvider.java:83)
at com.yahoo.db.mdbm.internal.TestMdbmPool.testDontCloseFromPool(TestMdbmPool.java:93)
testPool(com.yahoo.db.mdbm.internal.TestMdbmPool) Time elapsed: 0.001 sec <<< FAILURE!
java.lang.NoClassDefFoundError: Could not initialize class com.yahoo.db.mdbm.internal.NativeMdbmAccess
at com.yahoo.db.mdbm.internal.NativeMdbmImplementation.mdbm_open(NativeMdbmImplementation.java:444)
at com.yahoo.db.mdbm.MdbmProvider.open(MdbmProvider.java:140)
at com.yahoo.db.mdbm.MdbmProvider.openPool(MdbmProvider.java:83)
at com.yahoo.db.mdbm.internal.TestMdbmPool.testPool(TestMdbmPool.java:45)
testEmptyFirst(com.yahoo.db.mdbm.TestPool) Time elapsed: 0.001 sec <<< FAILURE!
java.lang.NoClassDefFoundError: Could not initialize class com.yahoo.db.mdbm.internal.NativeMdbmAccess
at com.yahoo.db.mdbm.internal.NativeMdbmImplementation.mdbm_open(NativeMdbmImplementation.java:444)
at com.yahoo.db.mdbm.MdbmProvider.open(MdbmProvider.java:140)
at com.yahoo.db.mdbm.MdbmProvider.openPool(MdbmProvider.java:83)
at com.yahoo.db.mdbm.TestPool.testEmptyFirst(TestPool.java:18)
Results :
Failed tests:
TestV4>TestSimpleMdbm.initEmptyMdbm:42 » UnsatisfiedLink no mdbm_java in java....
TestMdbmPool.testDontCloseFromPool:93 » NoClassDefFound Could not initialize c...
TestMdbmPool.testPool:45 » NoClassDefFound Could not initialize class com.yaho...
TestPool.testEmptyFirst:18 » NoClassDefFound Could not initialize class com.ya...
Tests run: 49, Failures: 4, Errors: 0, Skipped: 19
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
The docs for MDBM are extensive, but only available in source form. Please build the docs and publish them somewhere, perhaps with a pretty 2014-era landing page.
Running make
at top of repo on macOs High Sierria 10.13.3 generates this error:
mash.cc:1885:36: error: ordered comparison between pointer and zero
('char *' and 'int')
if (getcwd(pwdbuf, MAXPATHLEN) > 0) {
~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~
Looks like incorrect path for BIN_BASE and invalid variable substitution:
/bin/mdbm_reset_all_locks: line 4: BIN_BASE: command not found
/bin/mdbm_reset_all_locks: line 5: BIN_BASE: command not found
/bin/mdbm_reset_all_locks: line 19: mlock64: command not found
/bin/mdbm_reset_all_locks: line 19: -r: command not found
The Makefile contains references to the RTFM doc publishing tool. Scrub the docs for this & any other Y!-isms.
Testing with latest master branch, on macOS 10.13, I see
Test [ImportTestV3::TestImportNoDelete]: 0.0036 sec elapsed
.Test [ImportTestV3::TestSmallDbPageSize]: 0.0071 sec elapsed
.TESTWITHLOCKINGMODE /var/tmp/mdbm/leif-22613/mdbm-00006outexclusive
TESTWITHLOCKINGMODE /var/tmp/mdbm/leif-22613/mdbm-00007outpartition
Partition locking requires a fixed size DB.
Consider using the -d or -y options to specify the size
TESTWITHLOCKINGMODE /var/tmp/mdbm/leif-22613/mdbm-00008outshared
TESTWITHLOCKINGMODE /var/tmp/mdbm/leif-22613/mdbm-00009outnone
3:59dfd791:b6236:05855 multi_lock.cc:607 PLockFile: [/tmp/.mlock-named/var/tmp/mdbm/leif-22613/mdbm-00000input._int_] is too small 249 vs 9392 bytes
ERROR (77 No locks available) in mdbm_open_inner() mdbm.c:4188
/var/tmp/mdbm/leif-22613/mdbm-00000input: No locks available
Test [ImportTestV3::TestAllLocking]: FAILED!
FTest [ImportTestV3::TestAllLocking]: 1.0530 sec elapsed
TestSuite [ImportTestV3]: 1.0789 sec elapsed
TestSuite [All Tests]: 1.0789 sec elapsed
!!!FAILURES!!!
Test Results:
Run: 5 Failures: 1 Errors: 0
I have not investigated further into this issue, wanted to see if this is an expected failure at this time.
make rpm
does not build RPM package for perl code eventhough perl and its deps are installed
make rpm
Requires(rpmlib): rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1 rpmlib(CompressedFileNames) <= 3.0.4-1
Checking for unpackaged file(s): /usr/lib/rpm/check-files /home/centos/rpmbuild/BUILDROOT/mdbm-4.12.0.0-1.el7.centos.x86_64
warning: Could not canonicalize hostname: centos7
Wrote: /home/centos/rpmbuild/SRPMS/mdbm-4.12.0.0-1.el7.centos.src.rpm
Wrote: /home/centos/rpmbuild/RPMS/x86_64/mdbm-4.12.0.0-1.el7.centos.x86_64.rpm
Wrote: /home/centos/rpmbuild/RPMS/x86_64/mdbm-devel-4.12.0.0-1.el7.centos.x86_64.rpm
Wrote: /home/centos/rpmbuild/RPMS/x86_64/mdbm-debuginfo-4.12.0.0-1.el7.centos.x86_64.rpm
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.PmKfj5
+ umask 022
+ cd /home/centos/rpmbuild/BUILD
+ cd mdbm-4.12.0.0-0
+ /usr/bin/rm -rf /home/centos/rpmbuild/BUILDROOT/mdbm-4.12.0.0-1.el7.centos.x86_64
+ exit 0
rm -rf ./build/* ./dist/* 2>/dev/null || :
rm -rf mdbm.spec
rmdir ./build/ ./dist/ 2>/dev/null || :
Installed perl packages
[centos@centos7 redhat]$ rpm -qpl /home/centos/rpmbuild/RPMS/x86_64/mdbm-4.12.0.0-1.el7.centos.x86_64.rpm | grep -i perl
[centos@centos7 redhat]$ rpm -qpl /home/centos/rpmbuild/RPMS/x86_64/mdbm-devel-4.12.0.0-1.el7.centos.x86_64.rpm | grep -i perl
[centos@centos7 redhat]$ rpm -qpl /home/centos/rpmbuild/RPMS/x86_64/mdbm-debuginfo-4.12.0.0-1.el7.centos.x86_64.rpm | grep -i perl
[centos@centos7 redhat]$ rpm -qa perl\* | grep -i ext
perl-Text-ParseWords-3.29-4.el7.noarch
perl-ExtUtils-Manifest-1.61-244.el7.noarch
perl-ExtUtils-ParseXS-3.18-2.el7.noarch
perl-ExtUtils-Install-1.58-285.el7.noarch
perl-ExtUtils-MakeMaker-6.68-3.el7.noarch
[centos@centos7 redhat]$ rpm -q perl-devel
perl-devel-5.16.3-285.el7.x86_64
[centos@centos7 redhat]$
I would like to point out that identifiers like "__MDBM_H_
" and "__Y_MDBM3_INTERNAL_H__
" do not fit to the expected naming convention of the C language standard.
Would you like to adjust your selection for unique names?
https://github.com/yahoo/mdbm/blob/master/src/java/Makefile#L18-L19 causes Java wrapper compile error on machine that has a different Java installation like
/usr/lib/jvm/java-8-openjdk-amd64/include/
instead of /usr/lib/jvm/java/include
and
/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64
instead of /usr/lib/jvm/java/jre/lib/amd64/
User have to manually replace/fix this
I can work on fixing it if needed. 😉
Hello,
are there any plans for Windows support? I have tried to compile with VS 2015, but there are some linux specific includes, like fcntl.h, sys/types.h. Apart from that Windows will need mmap port.
Hi, How to I choose between mdbm and redis? Could you give some examples?
Hello, I know that this is not the place to make this question, but I don't find a play to ask, and also I don't find nothing in Internet.
I'm looking for information to use MDBM with php, but I don't find nothing, someone can tell me to look for it.
Thanks and sorry.
Oskar
[fedora@fedora22 mdbm]$ git remote -v
origin https://github.com/yahoo/mdbm.git (fetch)
origin https://github.com/yahoo/mdbm.git (push)
[fedora@fedora22 mdbm]$
[fedora@fedora22 mdbm]$ make
make -C src default-make-target && make -C include default-make-target && true
make[1]: Entering directory '/home/fedora/mdbm/src'
make -C lib default-make-target && make -C tools default-make-target && make -C test default-make-target && make -C scripts default-make-target && true
make[2]: Entering directory '/home/fedora/mdbm/src/lib'
true
clang -D_FILE_OFFSET_BITS=64 -DUSE_OPENSSL -DALLOW_MLOCK_RESET -DHAVE_ROBUST_PTHREADS -DHAVE_WINDOWED_MODE -DHAVE_PROC_FILESYS -g -O2 -I/usr/include -I/usr/local/include -I/home/fedora/mdbm/include -I. -Wall -fPIC -DDISABLE_TSC -Wall -pedantic -Wno-variadic-macros -Wno-long-long -Wno-overlength-strings -Wno-format-security -Wno-unused-result -Werror -Wunreachable-code -c hash.c -o object/hash.o
hash.c:25:1: error: unused variable 'what' [-Werror,-Wunused-const-variable]
WHATSTR("@(#)hash.c 1.4");
^
/home/fedora/mdbm/include/mdbm.h:2813:51: note: expanded from macro 'WHATSTR'
# define WHATSTR(X) static const char what[] = X
^
1 error generated.
/home/fedora/mdbm/Makefile.base:168: recipe for target 'object/hash.o' failed
make[2]: *** [object/hash.o] Error 1
make[2]: Leaving directory '/home/fedora/mdbm/src/lib'
/home/fedora/mdbm/Makefile.base:99: recipe for target 'make-recursive-default-make-target' failed
make[1]: *** [make-recursive-default-make-target] Error 2
make[1]: Leaving directory '/home/fedora/mdbm/src'
/home/fedora/mdbm/Makefile.base:99: recipe for target 'make-recursive-default-make-target' failed
I tried to build mdbm in OS X, but some linux related headers does not exist in OS X.
such as syscall.h
, and some open file flags.
Will it support OS X in the future? Or if it's difficult to port it to OS X?
make -C unit-test default-make-target && make -C func-test default-make-target && make -C smoke-test default-make-target && make -C pool_test default-make-target && /bin/true make[3]: Entering directory `/home/user/code/mdbm/src/test/unit-test' /bin/true g++ -D_FILE_OFFSET_BITS=64 -DUSE_OPENSSL -DALLOW_MLOCK_RESET -DHAVE_ROBUST_PTHREADS -g -O2 -I/usr/include -I/usr/local/include -I/home/user/code/mdbm/include -I. -Wall -fPIC -DDISABLE_TSC -I/home/user/code/mdbm/src/lib -Wall -pedantic -Wno-variadic-macros -Wno-long-long -Wno-overlength-strings -Wno-format-security -Wno-unused-result -Werror -Wunreachable-code CppUnitTestRunnerLocal.cc TestBase.cc test_lockbase.cc test_align.cc -lm -lpthread -lcrypto -L/home/user/code/mdbm/src/lib/object -lmdbm -lpthread -lrt -lstdc++ -lreadline -lcppunit -lcppunit -L/home/user/code/mdbm/src/lib/object -fPIC -o object/test_align CppUnitTestRunnerLocal.cc:15:40: fatal error: cppunit/ui/text/TestRunner.h: No such file or directory #include ^ compilation terminated. TestBase.cc:10:32: fatal error: cppunit/TestAssert.h: No such file or directory #include ^ compilation terminated. test_lockbase.cc:24:32: fatal error: cppunit/TestAssert.h: No such file or directory #include ^ compilation terminated. test_align.cc:17:32: fatal error: cppunit/TestAssert.h: No such file or directory #include ^ compilation terminated. make[3]: *** [object/test_align] Error 1 make[3]: Leaving directory`/home/user/code/mdbm/src/test/unit-test' make[2]: **\* [make-recursive-default-make-target] Error 2 make[2]: Leaving directory `/home/user/code/mdbm/src/test' make[1]: *** [make-recursive-default-make-target] Error 2 make[1]: Leaving directory`/home/user/code/mdbm/src' make: **\* [make-recursive-default-make-target] Error 2git clone https://github.com/yahoo/mdbm.git
cd mdbm
make
uname -a
Linux hostname 3.13.0-43-generic #72-Ubuntu SMP Mon Dec 8 19:35:06 UTC 2014 x86_64 x86_64 x86_64 GNU/Linuxcat /etc/issue
Ubuntu 14.04.1 LTS \n \lgcc -v
Using built-in specs.
COLLECT_GCC=/usr/bin/gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.8/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 4.8.2-19ubuntu1' --with-bugurl=file:///usr/share/doc/gcc-4.8/README.Bugs --enable-languages=c,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.8 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.8 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --disable-libmudflap --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.8-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.8-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.8-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.8.2 (Ubuntu 4.8.2-19ubuntu1)
It is hard coded in Makefile.base
:
https://github.com/yahoo/mdbm/blob/master/Makefile.base#L34
It would be useful if PREFIX is overridable when user run make install
something like:
PREFIX=$(pwd)/install make install
I guess that a different program design will be needed for your function "sig_handler".
Hi,
It's very nice if we can install the perl mdbm binding MDBM_File from cpan.
So could you upload MDBM_File to cpan?
I tried preparing a repository so that we can install MDBM_File via:
$ cpanm git://github.com/shoichikaji/perl-mdbm-install.git
I bet there are some folks around who can fast-track this into FreeBSD ports. Would love to have one for mdbm. https://www.freebsd.org/cgi/ports.cgi?query=mdbm&stype=all
If I pass a fragment of mdbm to mdbm_check
, then it core dumps.
How to reproduce the issue:
#0. set LD_LIBRARY_PATH
$ export LD_LIBRARY_PATH=$PWD/src/lib/object
#1. create test.mdbm
$ src/tools/object/mdbm_create test.mdbm
#2. save its first 1000bytes to head.mdbm
$ head -c 1000 test.mdbm > head.mdbm
#3. execute mdbm_check with head.mdbm
$ src/tools/object/mdbm_check head.mdbm
zsh: bus error (core dumped) src/tools/object/mdbm_check head.mdbm
gdb
$ gdb src/tools/object/mdbm_check core
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from src/tools/object/mdbm_check...done.
[New LWP 7790]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `src/tools/object/mdbm_check head.mdbm'.
Program terminated with signal SIGBUS, Bus error.
#0 check_db_chunks (verbose=1, db=0x19f4060) at mdbm.c:758
758 if (MDBM_PAGE_PTR(db,pg+page->p_num_pages)->p_prev_num_pages != page->p_num_pages) {
Traceback (most recent call last):
File "/usr/share/gdb/auto-load/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19-gdb.py", line 63, in <module>
from libstdcxx.v6.printers import register_libstdcxx_printers
ImportError: No module named 'libstdcxx'
(gdb) set pagination off
(gdb) bt full
#0 check_db_chunks (verbose=1, db=0x19f4060) at mdbm.c:758
pg = 0
cur_free = 0
nerr = 0
h = 0x7fcfd76ca010
nchunks = 1
#1 check_db (db=db@entry=0x19f4060, level=level@entry=3, maxlevel=maxlevel@entry=10, verbose=verbose@entry=1) at mdbm.c:1212
nerr = 0
#2 0x00007fcfd706e83c in mdbm_check (db=db@entry=0x19f4060, level=level@entry=3, verbose=verbose@entry=1) at mdbm.c:7353
No locals.
#3 0x000000000040114b in main (argc=<optimized out>, argv=<optimized out>) at mdbm_check.c:191
db = 0x19f4060
ret = 0
opt = <optimized out>
opt_version = <optimized out>
opt_nonversion = 0
opt_verify_version = 0
pno = -1
fn = 0x7ffdc5e3aa9f "head.mdbm"
oflags = 131072
verbose = 1
dbcheck = 3
lock = 1
winsize = 0
I know head.mdbm is invalid, but I think mdbm_check should not core dump.
Hi,
ENV: Ubuntu 14.04 and Debian 7.6. (run in docker)
I can't make test done.
~/mdbm# make test
make -C src test && make -C include test && /bin/true
make[1]: Entering directory /root/mdbm/src' make -C lib test && make -C tools test && make -C test test && make -C scripts test && /bin/true make[2]: Entering directory
/root/mdbm/src/lib'
/bin/true
make[2]: Leaving directory /root/mdbm/src/lib' make[2]: Entering directory
/root/mdbm/src/tools'
/bin/true
make[2]: Leaving directory /root/mdbm/src/tools' make[2]: Entering directory
/root/mdbm/src/test'
make -C unit-test test && make -C func-test test && make -C smoke-test test && make -C pool_test test && /bin/true
make[3]: Entering directory /root/mdbm/src/test/unit-test' /bin/true RUNNING UNIT TESTS MODE=, FAST= !!!! make run-test_align && make run-test_backstore && make run-test_cache && make run-test_close_sync_repl && make run-test_compare && make run-test_delete && make run-test_dibase && make run-test_dmbase && make run-test_export_api && make run-test_fetch && make run-test_getlimit && make run-test_getsize && make run-test_hash && make run-test_import && make run-test_iter && make run-test_lego_large_objects && make run-test_limitdir && make run-test_limitsize && make run-test_mag_vers && make run-test_mash && make run-test_other && make run-test_open && make run-test_pagesize && make run-test_presplit && make run-test_signals && make run-test_spillsize && make run-test_stats && make run-test_store && make run-test_tsc && make run-test_util && make run-test_dup_replace && make run-test_lockv3 && /bin/true make[4]: Entering directory
/root/mdbm/src/test/unit-test'
LD_LIBRARY_PATH=/root/mdbm/src/lib/object: object/test_align -tall -x test_align.xml
Writing unit test results to XML file (test_align.xml).
.
DB Alignment Test Suite Beginning...
Test [AlignTestSuiteV3::AllValidMasks]: 0.0129 sec elapsed
.Test [AlignTestSuiteV3::InvalidMasks]: 0.0249 sec elapsed
.Test [AlignTestSuiteV3::ResetValidMasks]: 0.0033 sec elapsed
.Test [AlignTestSuiteV3::DefaultAlignPerPagesize]: 0.1395 sec elapsed
.Test [AlignTestSuiteV3::VerifyAlignAfterTrunc]: 0.0021 sec elapsed
.Test [AlignTestSuiteV3::SetAlignForNumType]: 0.0022 sec elapsed
.Test [AlignTestSuiteV3::SetAlignPerMultiDatatype]: 0.0020 sec elapsed
.Test [AlignTestSuiteV3::SetAlignErrorCases]: 0.0021 sec elapsed
TestSuite [AlignTestSuiteV3]: 0.1893 sec elapsed
TestSuite [All Tests]: 0.1893 sec elapsed
OK (8 tests)
make[4]: Leaving directory /root/mdbm/src/test/unit-test' make[4]: Entering directory
/root/mdbm/src/test/unit-test'
LD_LIBRARY_PATH=/root/mdbm/src/lib/object: object/test_backstore -tall -x test_backstore.xml
Writing unit test results to XML file (test_backstore.xml).
.
Backing Store TestSuite Beginning...
Testing PAGE_SIZE 4096
Testing PAGE_SIZE 8192
Testing PAGE_SIZE 12288
Testing PAGE_SIZE 16384
Testing PAGE_SIZE 20480
Test [BackStoreTsV3::TestWindowedMode]: 0.1635 sec elapsed
.Test [BackStoreTsV3::BsSetWithNullParamA2]: 0.0000 sec elapsed
.Test [BackStoreTsV3::BsSetMdbmMultipleTimesThenCloseA3]: 0.0053 sec elapsed
.Test [BackStoreTsV3::BsSetBsWithoutCacheModeA5]: 0.0046 sec elapsed
.Test [BackStoreTsV3::BsUserDefinedBsFunctionsA6]: 0.0028 sec elapsed
.Test [BackStoreTsV3::BsStoreFetchLargeObjectA7]: 0.0088 sec elapsed
.3:542e5056:5240b:024f7 mdbm_lock.cc:201 /var/tmp/mdbm/root-9463/mdbm-00013tcbackstoreA8: backing store part/rw lock failed; dumping cache lock state: Invalid argument
Begin Lock state (for non-zero locks) (pid:9463, tid:9463 self:2ab7f3ce9780 uuid:9463 ):
BaseLocks:1 Core:1 Shared/Partitioned:1 mode:MLOCK_SINGLE (-1)
core[] local:1 any:1, owner:9463
End Lock state
Test [BackStoreTsV3::BsUseFileForBsA8]: FAILED!
FTest [BackStoreTsV3::BsUseFileForBsA8]: 0.0080 sec elapsed
.3:542e5056:5493d:024f7 mdbm_lock.cc:201 /var/tmp/mdbm/root-9463/mdbm-00014tcbackstoreA8: backing store part/rw lock failed; dumping cache lock state: Invalid argument
Begin Lock state (for non-zero locks) (pid:9463, tid:9463 self:2ab7f3ce9780 uuid:9463 ):
BaseLocks:1 Core:1 Shared/Partitioned:1 mode:MLOCK_SINGLE (-1)
core[] local:1 any:1, owner:9463
End Lock state
Test [BackStoreTsV3::BsUseNullForBsA8]: FAILED!
FTest [BackStoreTsV3::BsUseNullForBsA8]: 0.0093 sec elapsed
.Test [BackStoreTsV3::BsUseInvalidForBsA8]: 0.0043 sec elapsed
.Test [BackStoreTsV3::BsUseInvalid2ForBsA8]: 0.0035 sec elapsed
.Test [BackStoreTsV3::BsCacheModeLruStoreCacheOnlyA9]: 0.0089 sec elapsed
.Test [BackStoreTsV3::BsCacheModeNoneStoreCacheOnlyA10]: 0.0069 sec elapsed
.Test [BackStoreTsV3::BsCacheModeLruStoreModifyNewKeyA11]: 0.0113 sec elapsed
.Test [BackStoreTsV3::BsCacheModeLruStoreModifyOldKeyA12]: 0.0065 sec elapsed
.Test [BackStoreTsV3::BsSameWindowSizeCacheAndBsA13]: 0.0055 sec elapsed
.Test [BackStoreTsV3::BsSmallWindSizeCacheAndBigWindSizeBsA14]: 0.0057 sec elapsed
.Test [BackStoreTsV3::BsBigWindSizeCacheAndSmallWindSizeBsA15]: 0.0096 sec elapsed
.Test [BackStoreTsV3::BsLimitSizeSameCacheAndBsA16]: 0.0062 sec elapsed
.Test [BackStoreTsV3::BsLimitSizeBigCacheAndSmallBsA17]: 0.0055 sec elapsed
.Test [BackStoreTsV3::BsForEachPSizeOpenCacheForEachPSizeOpenBsLimitAndFillA18]: 0.0923 sec elapsed
.Test [BackStoreTsV3::BsUseFileBsFillDbCloseDbNewDbUseFileBsA19]: 0.0000 sec elapsed
.Test [BackStoreTsV3::BsUseMdbmBsFillDbCloseDbNewDbUseMdbmBsA20]: 0.0053 sec elapsed
.Test [BackStoreTsV3::BsUseMdbmBsFillDbCloseDbSameDbUseMdbmBsA21]: 0.0061 sec elapsed
.Test [BackStoreTsV3::BsStoreDataSetBsWithNewDbStoreNewA22]: 0.0116 sec elapsed
.Test [BackStoreTsV3::BsStoreDataSetReadOnlyBsWithNewDbStoreNewA23]: 0.0087 sec elapsed
.Test [BackStoreTsV3::BsTestReplaceA24]: 0.0987 sec elapsed
.3:542e5056:a00ae:024f7 mdbm.c:8781 /var/tmp/mdbm/root-9463/mdbm-00069bs_replace_cache: mdbm_replace_backing_store(): 'cache' must be a cache with MDBM backing-store: No such file or directory
Test [BackStoreTsV3::BsTestReplaceWithCache]: 0.0118 sec elapsed
.Test [BackStoreTsV3::BsUseMdbmBsInsertDataDeleteDataA25]: 0.0086 sec elapsed
.ERROR (2 No such file or directory) in mdbm_open_inner() mdbm.c:4124
3:542e5056:a3125:024f7 mdbm.c:4125 /var/tmp/mdbm/root-9463/mdbm-00072reopen-win-too-small: Page size 512 must be a multiple of system page size 4096: No such file or directory
3:542e5056:a3205:024f7 mdbm.c:4293 /var/tmp/mdbm/root-9463/mdbm-00072reopen-win-too-small: Page size 0 must be a multiple of system page size 4096: Invalid argument
ERROR (22 Invalid argument) in mdbm_open_inner() mdbm.c:4294
Test [BackStoreTsV3::OpenTooSmallReopenWindowed]: 0.0035 sec elapsed
.Test [BackStoreTsV3::BsTestMisc]: 0.0086 sec elapsed
.Test [BackStoreTsV3::BsTestInvalid]: 0.0030 sec elapsed
.3:542e5056:a6be4:024f7 mdbm.c:7862 /var/tmp/mdbm/root-9463/mdbm-00076tcbackstoreB1: mdbm_set_window_size() wsize should be at least 2 pages
Test [BackStoreTsV3::BsCreateDbNoWindowFlagSetWindowSizeZeroB1]: 0.0029 sec elapsed
.Test [BackStoreTsV3::BsCreateDbNoWindowFlagSetWindowSizeB2]: 0.0035 sec elapsed
.Test [BackStoreTsV3::BsSetWindowSizeMaxIntB3]: 0.0097 sec elapsed
.Test [BackStoreTsV3::BsSetWindowSizeMinusOneB4]: 0.0000 sec elapsed
.3:542e5056:aaa5b:024f7 mdbm.c:7862 /var/tmp/mdbm/root-9463/mdbm-00079tcbackstoreB5: mdbm_set_window_size() wsize should be at least 2 pages
Test [BackStoreTsV3::BsSetWindowSizeZeroB5]: 0.0025 sec elapsed
.Test [BackStoreTsV3::BsCheckSeveralDiffPageSizesSetWinSizeB6]: 0.0000 sec elapsed
.Test [BackStoreTsV3::BsCheckDiffPageSizesAgainstDiffWinSizesB7]: 0.0000 sec elapsed
.Test [BackStoreTsV3::BsNPageDbSetWinSizeStoreDataSetWinSize0B8]: 0.0000 sec elapsed
.Test [BackStoreTsV3::BsNPageDbSetWinSize3timesPsizeStoreDataSetWinSize2timesPsizeB9]: 0.0025 sec elapsed
.Test [BackStoreTsV3::BsNPageDbSetWinSize3timesPsizeStoreDataSetWinSize4timesPsizeB10]: 0.0024 sec elapsed
.Test [BackStoreTsV3::BsCreateDbLargeObjFlagSetWinSizeNinsertObjSizeLessNB11]: 0.0024 sec elapsed
.3:542e5056:ad0ad:024f7 mdbm.c:4836 /var/tmp/mdbm/root-9463/mdbm-00083tcbackstoreB12: window size too small for large object (need at least 28672 bytes)
Test [BackStoreTsV3::BsCreateDbLargeObjFlagSetWinSizeNinsertObjSizeEqualNB12]: 0.0023 sec elapsed
.3:542e5056:adb21:024f7 mdbm.c:4836 /var/tmp/mdbm/root-9463/mdbm-00084tcbackstoreB13: window size too small for large object (need at least 32768 bytes)
Test [BackStoreTsV3::BsCreateDbLargeObjFlagSetWinSizeNinsertObjSizeGreaterNB13]: 0.0027 sec elapsed
.Test [BackStoreTsV3::BsSetWindowIncreaseNumPagesFillDbB14]: 0.0000 sec elapsed
.Test [BackStoreTsV3::BsUseNullStatsPointerC1]: 0.0031 sec elapsed
.Test [BackStoreTsV3::BsUseTooSmallSizeC2]: 0.0029 sec elapsed
.Test [BackStoreTsV3::BsUseBigSizeC3]: 0.0029 sec elapsed
.Test [BackStoreTsV3::BsUseInBetweenSizeC4]: 0.0049 sec elapsed
.Test [BackStoreTsV3::BsCheckVariousWinSizesInLoopC5]: 0.0077 sec elapsed
.Test [BackStoreTsV3::BsNoWinModeFillDbGetStatsC6]: 0.0053 sec elapsed
.Test [BackStoreTsV3::BsWinModeNpagesFillDbNplusNpagesGetStatsC7]: 0.0105 sec elapsed
.Test [BackStoreTsV3::BsWinModeNpagesFillDbNpagesSetWinSize0GetStatsC8]: 0.0000 sec elapsed
TestSuite [BackStoreTsV3]: 0.6050 sec elapsed
TestSuite [All Tests]: 0.6050 sec elapsed
!!!FAILURES!!!
Test Results:
Run: 53 Failures: 2 Errors: 0
make[4]: *** [run-test_backstore] Error 1
make[4]: Leaving directory /root/mdbm/src/test/unit-test' make[3]: *** [test] Error 2 make[3]: Leaving directory
/root/mdbm/src/test/unit-test'
make[2]: *** [make-recursive-test] Error 2
make[2]: Leaving directory `/root/mdbm/src/test'
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.