Plyvel, a fast and feature-rich Python interface to LevelDB
Home Page: https://plyvel.readthedocs.io/
License: Other
Plyvel is a fast and feature-rich Python interface to LevelDB.
Plyvel has a rich feature set, high performance, and a friendly Pythonic API. See the documentation and project page for more information:
Note that using a released version is recommended over a checkout from version control. See the installation docs for more information.
I installed 'VCForPython27.msi' for building plyvel on windows 7.
But I got the following message.
Could you please consider this issue for windows user?
E:\work_python>pip install plyvel
Collecting plyvel
Using cached plyvel-0.9.tar.gz
Building wheels for collected packages: plyvel
Running setup.py bdist_wheel for plyvel
Complete output from command c:\python27\python.exe -c "import setuptools;file='c:\users\ybaik\appdata\local\temp\pip-build-hljopj\plyvel\setup.py';exec(compile(open(file).read().replace('\r\n', '\n'), f
p-wheel-:
running bdist_wheel
running build
running build_py
creating build
creating build\lib.win32-2.7
creating build\lib.win32-2.7\plyvel
copying plyvel_version.py -> build\lib.win32-2.7\plyvel
copying plyvel__init.py -> build\lib.win32-2.7\plyvel
running build_ext
building 'plyvel._plyvel' extension
creating build\temp.win32-2.7
creating build\temp.win32-2.7\Release
creating build\temp.win32-2.7\Release\plyvel
plyvel/_plyvel.cpp(352) : fatal error C1083: Cannot open include file: 'stdint.h': No such file or directory
error: command 'C:\\Users\\YBaik\\AppData\\Local\\Programs\\Common\\Microsoft\\Visual C++ for Python\\9.0\\VC\\Bin\\cl.exe' failed with exit status 2
=============================================================================
Seems like the lib misses Snappy symbols.
Installed via:
$ make
$ python setup.py installand then tried to use it like:
>>> import leveldb
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: ./leveldb.so: undefined symbol: _ZN6snappy13RawUncompressEPKcmPcProblem appears though I've got python-snappy(virtualenv) and libsnappy-dev(os) installed. Tried to add python-snappy to the setup.py: install_requires to enforce it to build with it. Same problem.
Hi there,
First of all thanks for creating Plyvel, it's been incredibly useful in my experimentation with LevelDB.
I'm currently facing an issue where I can't seem to reverse iterate reliably from the end of the store, without knowing the key I requested was past the end. and would like your thoughts.
The options are either to invoke iterator(reverse=True, include_stop=True, stop=k) , or .iterator(reverse=True) then .seek(k).
In the former case, this succeeds if the key exists (and yields the key correctly), otherwise if the key is greater than any in the store, it fails by yielding nothing.
With the second approach, I can reliably iterate from the end of the store when providing a nonexistent key, except if the key I provide actually exists, then it is skipped during the first .next() call.
My workaround for the second approach is to append a single '\x00' to the key if iterating in reverse, which seems to produce my desired result.
My issue is whether the skipping behaviour in the latter case is sensible, and if not, whether my workaround is actually exploiting a bug that might be fixed later. :)
In any case the first approach seems totally wrong. If .Seek() results in an invalid iterator, then I think calling SeekToEnd() should occur.
Thanks again for an excellent library, I use it all the time!
See #35 (comment) for a bit more details.
Trying to get Plyvel working on Mavericks. I have LevelDB 1.15 installed via homebrew. In a Python 3.3 virtualenv, after pip install plyvel, I get the following:
Python 3.3.3 (default, Dec 4 2013, 09:48:34)
[GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.28)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import plyvel
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "./plyvel/__init__.py", line 6, in <module>
from ._plyvel import ( # noqa
ImportError: dlopen(./plyvel/_plyvel.so, 2): Symbol not found: __ZN7leveldb2DB4OpenERKNS_7OptionsERKSsPPS0_
Referenced from: ./plyvel/_plyvel.so
Expected in: flat namespace
in ./plyvel/_plyvel.so
The docs say to ensure the linker can find the shared library, but I am not sure how to do that with pip. When grepping for that symbol, I find something similar, but not exact. Is this a version mismatch or I am just not looking at the right things?
nm /usr/local/lib/libleveldb.1.dylib | grep __ZN7leveldb2DB4Open
0000000000009456 T __ZN7leveldb2DB4OpenERKNS_7OptionsERKNSt3__112basic_stringIcNS4_11char_traitsIcEENS4_9allocatorIcEEEEPPS0_
Here is a simple test to reproduce:
https://gist.github.com/aglyzov/5643563
I tested with Python 3.3.1 and Python 3.3.2 on x86-64 and ARM. Plyvel 0.2, latest leveldb + snappy.
Getting this error with sudo pip install plyvel
OS X 10.9
Thanks!
building 'plyvel._plyvel' extension
creating build/temp.macosx-10.9-intel-2.7
creating build/temp.macosx-10.9-intel-2.7/plyvel
cc -fno-strict-aliasing -fno-common -dynamic -arch x86_64 -arch i386 -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -mno-fused-madd -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch x86_64 -arch i386 -pipe -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c plyvel/_plyvel.cpp -o build/temp.macosx-10.9-intel-2.7/plyvel/_plyvel.o -Wall -g
clang: error: unknown argument: '-mno-fused-madd' [-Wunused-command-line-argument-hard-error-in-future]
clang: note: this will be a hard error (cannot be downgraded to a warning) in the future
error: command 'cc' failed with exit status 1
I passed a memoryview to db.put() as the value, and I got an error
TypeError: Argument 'value' has incorrect type (expected bytes, got memoryview)
I doubt there is a good reason for this. Can you accept memoryviews too please in all relevant APIs? They enable me to avoid redundant copying of big binary objects when taking slices.
I've only used Plyvel on Linux before, and I'm having the following issues when trying to install on OSX.
plyvel/_plyvel.cpp:321:10: fatal error: 'leveldb/db.h' file not found
#include "leveldb/db.h"
I've installed LevelDB from source, and installing "pip install leveldb" gets me a working py-leveldb install, but I'd much rather use plyvel if I can get it working.
I've tried Googling every aspect of the issue possible, but at this point I've been on a wild goose chase for over a day now and am no closer to being able to use plyvel on OSX.
Thanks,
Nick
The standard git way to package is to create a packaging branch that contains all of the packaging-specific files and data, then rebase or cherry-pick when building a package for a specific version. I've put together the packaging materials here:
It's been reviewed by one of Debian's maintainers (Paul Cannon).
To be able to submit a proper merge request, the branch has to exist before the merge request is created. Hence this issue. If you could create a new branch called "debian" (or similar), I'll submit a pull request to get the packaging back in core - then we can submit plyvel to Debian for inclusion.
Hi,
I ran into an issue with installing plyvel on my server running Ubuntu 12.04, I received this error.
$python -c 'import plyvel'
Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python2.7/dist-packages/plyvel/init.py", line 6, in from ._plyvel import ( ImportError: /usr/local/lib/python2.7/dist-packages/plyvel/_plyvel.so: undefined symbol: _ZN6snappy13RawUncompressEPKcmPc
I have these shared libraries install but it is still not working for me.
libleveldb1 (.deb for 12.10) libsnappy1 (apt-get), libleveldb-dev (apt-get), leveldb-doc (apt-get)
I am not installing plyvel on virtualenv on my server. Any know issue with plyvel install on the native python 2.7.3?
Hi,
During the installation, Plyvel could try to compile LevelDB if it's not already installed. That's what PyZMQ is doing :
When compiling pyzmq (e.g. installing with pip on Linux), it is generally recommended that zeromq be installed separately, via homebrew, apt, yum, etc. If this is not available, pyzmq will try to build libzmq as a Python Extension, though this is not guaranteed to work.
I think it would require quite a bit of work in order to work on a maximum of platform, but it would be a really great advantage for Plyvel. I'll try do to it, but I won't have much time in the next few days, so it might be in a while.
Hey there I tried to pip install plyvel on OSX (python 3.3) and there seems to be a problem with the compilation.
leveldb version 1.19.0
cc -Wno-unused-result -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python3/3.3.2/Frameworks/Python.framework/Versions/3.3/include/python3.3m -c plyvel/_plyvel.cpp -o build/temp.macosx-10.9-x86_64-3.3/plyvel/_plyvel.o -Wall -g
plyvel/_plyvel.cpp:13845:125: warning: offset of on non-POD type 'struct __pyx_obj_6plyvel_7_plyvel_Iterator' [-Winvalid-offsetof]
if (__pyx_type_6plyvel_7_plyvel_Iterator.tp_weaklistoffset == 0) __pyx_type_6plyvel_7_plyvel_Iterator.tp_weaklistoffset = offsetof(struct __pyx_obj_6plyvel_7_plyvel_Iterator, weakref);
^ ~~~~~~~~~~~
/Library/Developer/CommandLineTools/usr/bin/../lib/clang/5.0/include/stddef.h:84:24: note: expanded from macro 'offsetof'
#define offsetof(t, d) __builtin_offsetof(t, d)
^
1 warning generated.
cc -Wno-unused-result -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python3/3.3.2/Frameworks/Python.framework/Versions/3.3/include/python3.3m -c plyvel/comparator.cpp -o build/temp.macosx-10.9-x86_64-3.3/plyvel/comparator.o -Wall -g
c++ -bundle -undefined dynamic_lookup -L/usr/local/lib -L/usr/local/opt/sqlite/lib build/temp.macosx-10.9-x86_64-3.3/plyvel/_plyvel.o build/temp.macosx-10.9-x86_64-3.3/plyvel/comparator.o -lleveldb -o build/lib.macosx-10.9-x86_64-3.3/plyvel/_plyvel.so
warning: no files found matching 'plyvel/*.pxi'
Any clue what goes wrong?
Thank you!
@wbolster what's the latest version of levelDB you have tried with plyvel? I am still on 1.12. Wondering if I should upgrade.
I've just encounter this problem when inserting a stream of data using a custom comparator. Code is something like below:
def str_int_comparator(a, b):
a = int(a)
b = int(b)
if a < b:
return -1
if a > b:
return 1
return 0
db = plyvel.DB(db_path, create_if_missing=True,
comparator=str_int_comparator,
comparator_name='str_int_comparator')
stream = get_the_stream(...)
before = time.time()
wb = db.write_batch()
# generate a kv stream
for i, line in enumerate(stream):
if i > 0 and i % 50000 == 0:
wb.write()
wb = db.write_batch()
print "Inserted upto {} ...".format(i)
wb.put(str(i), json.dumps(line))
after = time.time()
used = after - beforeI got this after 100k elements are inserted:
Inserted upto 50000 ...
Inserted upto 100000 ...
Fatal Python error: This thread state must be current when releasing
Aborted
If I use the default comparator, everything goes well.
I have no idea the implementation but it seems to me that when LevelDB begins to compact files, the error occurs.
I'm using plyvel 0.8, LevelDB 1.15.0 and python 2.7.8.
OS kali-linux
version 0.8 work`s fine
and in 0.9 i have this issue
Starting program: venv/bin/python index_db_test.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff662c18d in leveldb::InternalFilterPolicy::CreateFilter(leveldb::Slice const*, int, std::__cxx11::basic_string<char, std::char_traits, std::allocator >*) const ()
from /usr/lib/x86_64-linux-gnu/libleveldb.so.1
Hi there,
I'm trying to open an existing LevelDB (A) with plyvel, create a new LevelDB (B), and write the first X values of A to B. I'm doing this in batch writes of 100, but for some reason, the first few batches write extremely quickly, and then as the process continues, the writing slows down tremendously. I'm talking from < 1 second for the first batch write to ~10 seconds for the 70th batch.
I'd like to know if this is expected (as B grows in size?), or if there's possibly a small mistake I'm making that's slowing the process down.
Also, if anyone has any suggestions for a better way to subset part of A into a new LevelDB, please let me know.
Thanks!
I'm getting an error when loading plyvel.
darky:code medecau$ python
Python 2.7.6 (default, May 14 2014, 21:37:36)
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.1.00)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import plyvel
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/site-packages/plyvel/__init__.py", line 6, in <module>
from ._plyvel import ( # noqa
ImportError: dlopen(/usr/local/lib/python2.7/site-packages/plyvel/_plyvel.so, 2): Library not loaded: /usr/local/opt/leveldb/lib/libleveldb.dylib.1
Referenced from: /usr/local/lib/python2.7/site-packages/plyvel/_plyvel.so
Reason: image not found
>>>
I have installed leveldb using brew and these are the available libleveldb files.
darky:levelzero medecau$ ls /usr/local/opt/leveldb/lib/
libleveldb.1.15.dylib libleveldb.1.dylib libleveldb.a libleveldb.dylib
If I manually create a link from /usr/local/opt/leveldb/lib/libleveldb.dylib to /usr/local/opt/leveldb/lib/libleveldb.dylib.1 the import error goes away.
I tried to add another link to the brew formula for leveldb but it was as I agree not accepted. (See: Homebrew/legacy-homebrew#29263)
When needs to be done in order to change plyvel from using /usr/local/opt/leveldb/lib/libleveldb.dylib.1 to /usr/local/opt/leveldb/lib/libleveldb.dylib or even /usr/local/opt/leveldb/lib/libleveldb.1.dylib?
Add a dict-like optional 'default' argument to the .get() calls on DB, Snapshot, and PrefixedDB.
It is common that I want to copy a leveldb. I finally wrote a function to do it. Would it make sense to include such a function in plyvel? If yes, I might add it.
The function would look a bit like this, but not as complicated. The second half of this function creates an archive of the leveldb so you can download it in one file, because that's what I wanted in my case; if this is to be included in plyvel, I think the function should return either a new plyvel object or the str name of the directory that contains the new directory.
def archive_level_db(db):
'''
Copy a leveldb to a new directory, and put it in a gzipped tarball.
:param plyvel.DB db: The source database
:param directory: The directory to put the new database in
:returns: The gzipped tarball as bytes
'''
arcname = datetime.datetime.now().strftime('leveldb-%Y-%m-%d')
with tempfile.TemporaryDirectory() as directory:
new_db = plyvel.DB(directory, create_if_missing=True)
with new_db.write_batch() as b:
for k,v in db.snapshot().iterator():
b.put(k, v)
new_db.close()
with tempfile.NamedTemporaryFile() as tmp:
with tarfile.open(tmp.name, "w:gz") as tar:
tar.add(directory, arcname=arcname)
tmp.file.seek(0)
out = tmp.file.read()
return out
In case I update it, the updated version will probably be in this file.
Much like #19:
Snapshot.close() methodSnapshot.release() alias to mimic LevelDB terminology.Hi everyone,
I am trying to install plyvel on my system but when I import the library in my code, I get the following error:
Traceback (most recent call last):
File "", line 1, in
File "/home/aniss/anaconda3/lib/python3.6/site-packages/plyvel/init.py", line 6, in
from ._plyvel import ( # noqa
ImportError: /home/aniss/anaconda3/lib/python3.6/site-packages/plyvel/_plyvel.cpython-36m-x86_64-linux-gnu.so: undefined symbol: ZN7leveldb2DB4OpenERKNS_7OptionsERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPPS0
and following is the command that I used to install it:
pip install --global-option=build_ext --global-option="-I/usr/local/include" --global-option="-L/usr/local/lib" --no-cache-dir --no-deps Plyvel
Can anyone help me with this issue please, thank you in advance.
Hi,
I am wondering if you can help me with this one.
Traceback (most recent call last):
File "release_lock.py", line 4, in <module>
db = plyvel.DB('/root/.leveldb/db/', create_if_missing=True)
File "_plyvel.pyx", line 229, in plyvel._plyvel.DB.__init__ (plyvel/_plyvel.cpp:2601)
File "_plyvel.pyx", line 75, in plyvel._plyvel.raise_for_status (plyvel/_plyvel.cpp:1242)
plyvel._plyvel.IOError: IO error: lock /root/.leveldb/db//LOCK: Resource temporarily unavailable
I have a small project that reads rss feeds and save it to levelDB. I have bump into the issue a couple of times and hope db.close() will solve the issue. I will like to confirm if it is a DB close issue. My guess is it could be an unhandled exception in the code that does not close the DB properly.
Thanks in advance.
I have a ceph instance that recently had a power failure. All of the monitors use leveldb to store their data and all 3 of them seem to have a corrupt leveldb database in /var/lib/ceph/mon/ceph-$(hostname)/store.db. I found your library and tried running a repair in ipython which results in a crash::
In [4]: plyvel.repair_db('ceph-kh09-8/store.db/')
terminate called after throwing an instance of 'std::length_error'
what(): basic_string::_M_create
[1] 13259 abort (core dumped) ipython
I was wondering if I am doing something wrong or if this is an error in the plyvel handler? It is my understanding that leveldb should skip keys with corruption and return any valid data from the remaining keys but I can't seem to open the databases at all.
Hey,
I used the python leveldb wrapper before, but your's seems to have some nice features like prefix. The project doesn't compile for me with missing file.
x86_64-linux-gnu-gcc: error: plyvel/_plyvel.cpp: No such file or directory
I can't find this file in your repository. Can you add it, so I check again?
Cheers
Jens.
running install
running bdist_egg
running egg_info
writing dependency_links to plyvel.egg-info\dependency_links.txt
writing plyvel.egg-info\PKG-INFO
writing top-level names to plyvel.egg-info\top_level.txt
reading manifest file 'plyvel.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '_' under directory 'doc\build\html'
warning: no files found matching 'plyvel_.pxi'
writing manifest file 'plyvel.egg-info\SOURCES.txt'
installing library code to build\bdist.win-amd64\egg
running install_lib
running build_py
running build_ext
building 'plyvel._plyvel' extension
D:\Program Files\vs2015\VC\BIN\amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -IC:\Python35\include -IC:\Python35\include "-ID:\Program Files\vs2015\VC\INCLUDE" "-ID:\Program Files\vs2015\VC\ATLMFC\INCLUDE" "-IC:\Program Files
(x86)\Windows Kits\10\include\10.0.10150.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6\include\um" "-IC:\Program Files (x86)\Windows Kits\8.1\include\shared" "-IC:\Program Files (x86)\Windows Kits\8.1\include\um" "-
IC:\Program Files (x86)\Windows Kits\8.1\include\winrt" /EHsc /Tpplyvel/_plyvel.cpp /Fobuild\temp.win-amd64-3.5\Release\plyvel/_plyvel.obj -Wall -g
cl: ??? warning D9002 :??????"-g"
_plyvel.cpp
c1xx: fatal error C1083: ???????: "plyvel/_plyvel.cpp": No such file or directory
error: command 'D:\Program Files\vs2015\VC\BIN\amd64\cl.exe' failed with exit status 2
Python:3.5
OS:Windows 10 (64bit)
Setup: download(py setup.py) and pip install plyvel
Specifically:
This is a completely different model of iteration from standard python. As mentioned here:
... we have an existing code base whose logic depends on the outlined features. It may not be appropriate for the general Plyvel consumer.
Let me know what you think.
The __repr__() should include the database name, the state (open/closed) and (if applicable) the prefix.
leveldb installs fine, plyvel errors with:
plyvel/_plyvel.cpp:269:35: fatal error: leveldb/filter_policy.h: No such file or directory
I would like to use plyvel in a notebook development environment (ipython). Supporting a graceful close (i.e. lock release) between code evaluations using the __exit__ interface would be great for dynamic evaluation environments. I would make a pull request but I think this change is trivial and you'd have it done before breakfast :)
While trying to port plyvel into Elevator, I noticed that DB.del method does not actually release the LOCK.
>>> db = plyvel.DB('default')
>>> del db
>>> new_db = plyvel.DB('default')
---------------------------------------------------------------------------
IOError Traceback (most recent call last)
<ipython-input-10-7174fcd34511> in <module>()
----> 1 new_db = plyvel.DB('default')
/Users/oleiade/.python-eggs/plyvel-0.1dev-py2.7-macosx-10.8-intel.egg-tmp/plyvel/_plyvel.so in plyvel._plyvel.DB.__init__ (plyvel/_plyvel.cpp:2264)()
/Users/oleiade/.python-eggs/plyvel-0.1dev-py2.7-macosx-10.8-intel.egg-tmp/plyvel/_plyvel.so in plyvel._plyvel.raise_for_status (plyvel/_plyvel.cpp:1061)()
IOError: IO error: lock default/LOCK: already held by processgcc: error: plyvel/_plyvel.cpp: No such file or directory
Hi,
I'm trying to implement a function checking a given key exists in the database without getting the value.
With the native API it can be done by creating an iterator, seeking to the key and checking the return of valid(). However, I don't manage to do it with Plyvel, as I'm getting this strange behavior :
>>> import plyvel
>>> db = plyvel.DB('test', create_if_missing=True)
>>> db.put('foo', 'bar')
>>> list(db.iterator())
[('foo', 'bar')]
>>> it = db.raw_iterator()
>>> it.seek('foo')
>>> it.valid()
True
>>> it = db.raw_iterator()
>>> it.seek('toto')
>>> it.valid()
False
>>> it = db.raw_iterator()
>>> it.seek('aaa')
>>> it.valid()
True
>>> it.key()
'foo'
>>> it.value()
'bar'If I seek to a key < to an existing one, it actually seeks to the first existing key. If the given key is > to an existing one, the behavior is ok.
I'm using Plyvel 0.8 with LevelDB 1.15.0. I didn't check if I get the same kind of result in C++, but I don't think it'll be the case.
Do you plan on exposing the slice interface in leveldb?
The reason I ask is that I plan on storing large values and would prefer to stream them in and out of a key instead of having the interpreter allocate large chunks of memory.
BTW... thanks for an excellent library!
On arrmv7h (Raspberry Pi) with archlinux, I get a segmentation fault when executing db.close() or del db.
Steps to reproduce the issue:
import plyvel
db = plyvel.DB('/tmp/testdb/', create_if_missing=True)
db.close() # segfault
Plyvel version --> 0.9-4
LevelDB version --> 1.20-1
I've submitted leveldb-cython to few benchmark tests using Hurdles; in order to see if it would be a good replacement for py-leveldb in Elevator.
Here are the results, and the bench case:
Nota : Benchmarks are ran into a virtual machine run on one 3.5Ghz core and 4G ram, and virtual disk. So they don't mean to be a real world usage benchmark, but rather to give a comparison between the two libs on same testing environment.
Plus, hurdles has been run 3 times for each lib in order to have aricher sampling (leveldb performance greatly relies on current disk usage), but as a default hurdles runs every bench 10times (sampling).
With py-leveldb
BenchLevelDB.bench_atomic_get ... 13.742 ms
BenchLevelDB.bench_atomic_put ... 28.224 ms
BenchLevelDB.bench_batch_mixed_put_delete ... 177.609 ms
BenchLevelDB.bench_batch_only_put ... 180.283 ms
BenchLevelDB.bench_range ... 33.658 ms
------------------------------------------------------------
Ran 5 benchmarks
Done.
BenchLevelDB.bench_atomic_get ... 13.974 ms
BenchLevelDB.bench_atomic_put ... 25.984 ms
BenchLevelDB.bench_batch_mixed_put_delete ... 180.751 ms
BenchLevelDB.bench_batch_only_put ... 187.625 ms
BenchLevelDB.bench_range ... 34.347 ms
------------------------------------------------------------
Ran 5 benchmarks
Done.
BenchLevelDB.bench_atomic_get ... 13.945 ms
BenchLevelDB.bench_atomic_put ... 26.036 ms
BenchLevelDB.bench_batch_mixed_put_delete ... 178.113 ms
BenchLevelDB.bench_batch_only_put ... 181.373 ms
BenchLevelDB.bench_range ... 33.568 ms
------------------------------------------------------------
Ran 5 benchmarks
Done.% with leveldb-cython
BenchLevelDB.bench_atomic_get ... 14.436 ms
BenchLevelDB.bench_atomic_put ... 25.468 ms
BenchLevelDB.bench_batch_mixed_put_delete ... 139.5 ms
BenchLevelDB.bench_batch_only_put ... 141.825 ms
BenchLevelDB.bench_range ... 116.309 ms
------------------------------------------------------------
Ran 5 benchmarks
Done.
BenchLevelDB.bench_atomic_get ... 13.188 ms
BenchLevelDB.bench_atomic_put ... 25.35 ms
BenchLevelDB.bench_batch_mixed_put_delete ... 140.97 ms
BenchLevelDB.bench_batch_only_put ... 139.18 ms
BenchLevelDB.bench_range ... 114.635 ms
------------------------------------------------------------
Ran 5 benchmarks
Done
BenchLevelDB.bench_atomic_get ... 12.902 ms
BenchLevelDB.bench_atomic_put ... 24.773 ms
BenchLevelDB.bench_batch_mixed_put_delete ... 140.039 ms
BenchLevelDB.bench_batch_only_put ... 143.598 ms
BenchLevelDB.bench_range ... 119.553 ms
------------------------------------------------------------
Ran 5 benchmarks
Done.Results interpretation:
Let me know if you'd like other benchmarks to be added, or any recommandation on how I test each cases.
I've just tried to install plyvel in pypy but got the following error:
Downloading/unpacking plyvel
http://206524166f7446e3946dfb16b510c1e6:[email protected]/simple/plyvel/ uses an insecure transport scheme (http). Consider using https if 206524166f7446e3946dfb16b510c1e6:[email protected] has it available
Downloading plyvel-0.9.tar.gz (121kB): 121kB downloaded
Running setup.py (path:/home/darkjh/projects/python/pypy_env/build/plyvel/setup.py) egg_info for package plyvel
warning: no files found matching '*' under directory 'doc/build/html'
warning: no files found matching 'plyvel/*.pxi'
Installing collected packages: plyvel
Running setup.py install for plyvel
building 'plyvel._plyvel' extension
cc -O2 -fPIC -Wimplicit -I/home/darkjh/projects/python/pypy_env/include -c plyvel/_plyvel.cpp -o build/temp.linux-x86_64-2.7/plyvel/_plyvel.o -Wall -g
cc1plus: warning: command line option ‘-Wimplicit’ is valid for C/ObjC but not for C++
plyvel/_plyvel.cpp: In function ‘int __Pyx_GetItemInt_ByteArray_Fast(PyObject*, Py_ssize_t, int, int)’:
plyvel/_plyvel.cpp:18127:45: error: ‘PyByteArray_GET_SIZE’ was not declared in this scope
length = PyByteArray_GET_SIZE(string);
^
plyvel/_plyvel.cpp:18130:65: error: ‘PyByteArray_AS_STRING’ was not declared in this scope
return (unsigned char) (PyByteArray_AS_STRING(string)[i]);
^
plyvel/_plyvel.cpp:18136:61: error: ‘PyByteArray_AS_STRING’ was not declared in this scope
return (unsigned char) (PyByteArray_AS_STRING(string)[i]);
^
plyvel/_plyvel.cpp: In function ‘int __Pyx_SetItemInt_ByteArray_Fast(PyObject*, Py_ssize_t, unsigned char, int, int)’:
plyvel/_plyvel.cpp:18144:45: error: ‘PyByteArray_GET_SIZE’ was not declared in this scope
length = PyByteArray_GET_SIZE(string);
^
plyvel/_plyvel.cpp:18147:41: error: ‘PyByteArray_AS_STRING’ was not declared in this scope
PyByteArray_AS_STRING(string)[i] = (char) v;
^
plyvel/_plyvel.cpp:18154:37: error: ‘PyByteArray_AS_STRING’ was not declared in this scope
PyByteArray_AS_STRING(string)[i] = (char) v;
^
plyvel/_plyvel.cpp: In function ‘int __Pyx_GetItemInt_ByteArray_Fast(PyObject*, Py_ssize_t, int, int)’:
plyvel/_plyvel.cpp:18138:1: warning: control reaches end of non-void function [-Wreturn-type]
}
^
error: command 'cc' failed with exit status 1
Complete output from command /home/darkjh/projects/python/pypy_env/bin/pypy -c "import setuptools, tokenize;__file__='/home/darkjh/projects/python/pypy_env/build/plyvel/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-iLSs8H-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/darkjh/projects/python/pypy_env/include/site/python2.7:
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/plyvel
copying plyvel/__init__.py -> build/lib.linux-x86_64-2.7/plyvel
copying plyvel/_version.py -> build/lib.linux-x86_64-2.7/plyvel
running build_ext
building 'plyvel._plyvel' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/plyvel
cc -O2 -fPIC -Wimplicit -I/home/darkjh/projects/python/pypy_env/include -c plyvel/_plyvel.cpp -o build/temp.linux-x86_64-2.7/plyvel/_plyvel.o -Wall -g
cc1plus: warning: command line option ‘-Wimplicit’ is valid for C/ObjC but not for C++
plyvel/_plyvel.cpp: In function ‘int __Pyx_GetItemInt_ByteArray_Fast(PyObject*, Py_ssize_t, int, int)’:
plyvel/_plyvel.cpp:18127:45: error: ‘PyByteArray_GET_SIZE’ was not declared in this scope
length = PyByteArray_GET_SIZE(string);
^
plyvel/_plyvel.cpp:18130:65: error: ‘PyByteArray_AS_STRING’ was not declared in this scope
return (unsigned char) (PyByteArray_AS_STRING(string)[i]);
^
plyvel/_plyvel.cpp:18136:61: error: ‘PyByteArray_AS_STRING’ was not declared in this scope
return (unsigned char) (PyByteArray_AS_STRING(string)[i]);
^
plyvel/_plyvel.cpp: In function ‘int __Pyx_SetItemInt_ByteArray_Fast(PyObject*, Py_ssize_t, unsigned char, int, int)’:
plyvel/_plyvel.cpp:18144:45: error: ‘PyByteArray_GET_SIZE’ was not declared in this scope
length = PyByteArray_GET_SIZE(string);
^
plyvel/_plyvel.cpp:18147:41: error: ‘PyByteArray_AS_STRING’ was not declared in this scope
PyByteArray_AS_STRING(string)[i] = (char) v;
^
plyvel/_plyvel.cpp:18154:37: error: ‘PyByteArray_AS_STRING’ was not declared in this scope
PyByteArray_AS_STRING(string)[i] = (char) v;
^
plyvel/_plyvel.cpp: In function ‘int __Pyx_GetItemInt_ByteArray_Fast(PyObject*, Py_ssize_t, int, int)’:
plyvel/_plyvel.cpp:18138:1: warning: control reaches end of non-void function [-Wreturn-type]
}
^
error: command 'cc' failed with exit status 1
----------------------------------------
Cleaning up...
Command /home/darkjh/projects/python/pypy_env/bin/pypy -c "import setuptools, tokenize;__file__='/home/darkjh/projects/python/pypy_env/build/plyvel/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-iLSs8H-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/darkjh/projects/python/pypy_env/include/site/python2.7 failed with error code 1 in /home/darkjh/projects/python/pypy_env/build/plyvel
Traceback (most recent call last):
File "app_main.py", line 75, in run_toplevel
File "/home/darkjh/projects/python/pypy_env/bin/pip", line 11, in <module>
sys.exit(main())
File "/home/darkjh/projects/python/pypy_env/site-packages/pip/__init__.py", line 185, in main
return command.main(cmd_args)
File "/home/darkjh/projects/python/pypy_env/site-packages/pip/basecommand.py", line 161, in main
text = '\n'.join(complete_log)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 42: ordinal not in range(128)
I know that plyvel does not target Pypy but just want to know if it's possible to make it work with reasonable effort? It seems Cython has some support to generate Pypy compat code.
The iterator API should allow explicit closing. Ideas:
.iterator(autoclose=True) or similar flag to automatically cleanup an iterator whenever the first StopIteration is raised.@contextlib.closing would be the same if .close() was implemented).This issue was inspired by @jtolds in #17:
also, we need to support closing iterators, so that we don't have to wait for garbage collection to kick in to free the leveldb snapshot backing the iterator
In the docs for Iterator and RawIterator it says they have methods "seek_to_start" and "seek_to_end". The actual methods are "seek_to_first" and "seek_to_last".
What time can support python3.5?
the database formats are different.
it is not possible to use the same db with both.
is there a reason for that?
Assuming db is instance of plyvel.DBInterface, it would be useful to do something like:
db['key']
or
if 'key' in db
Hello,
I am trying to use plyvel 0.9 on a Ubuntu 16.04.2 LTS machine with Python 3.5.2-2ubuntu0~16.04.1.
I expected it to work, but I get an ImportError.
I did the following:
I installed leveldb with apt install libleveldb-dev which installed the following packages:
libleveldb-dev:amd64 1.18-5
libleveldb1v5:amd64 1.18-5
I installed plyvel with pip3 install plyvel:
$ sudo pip3 install plyvel
Collecting plyvel
Installing collected packages: plyvel
Successfully installed plyvel-0.9
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ python3 -c 'import plyvel'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/local/lib/python3.5/dist-packages/plyvel/__init__.py", line 6, in <module>
from ._plyvel import ( # noqa
ImportError: /usr/local/lib/python3.5/dist-packages/plyvel/_plyvel.cpython-35m-x86_64-linux-gnu.so: undefined symbol: _ZN7leveldb9DestroyDBERKSsRKNS_7OptionsE
The packages mentioned in https://plyvel.readthedocs.io/en/latest/installation.html#build-and-install-plyvel are installed and the shared objects of libleveldb can be found using locate:
$ locate leveldb
/usr/lib/x86_64-linux-gnu/libleveldb.a
/usr/lib/x86_64-linux-gnu/libleveldb.so
/usr/lib/x86_64-linux-gnu/libleveldb.so.1
/usr/lib/x86_64-linux-gnu/libleveldb.so.1.18
And the plyvel extension library appears to link against it correctly:
$ readelf -d /usr/local/lib/python3.5/dist-packages/plyvel/_plyvel.cpython-35m-x86_64-linux-gnu.so
Dynamic section at offset 0x27cc0 contains 27 entries:
Tag Type Name/Value
0x0000000000000001 (NEEDED) Shared library: [libleveldb.so.1]
...
However, curiously I found in the libleveldb.so.1.18 the searched symbol only with a slightly different variation:
$ objdump -T libleveldb.so.1.18 | grep leveldb9Destroy
0000000000016c40 g DF .text 0000000000000a94 Base _ZN7leveldb9DestroyDBERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_7OptionsE
This looks like the search thing, but with extra __cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS inside. Any help will be appreciated.
The .seek() method on a PrefixedDB iterator does not take the db prefix into account.
Python(5404,0x7fffe58773c0) malloc: *** malloc_zone_unregister() failed for 0x7fffe586d000
Don't know if it is leveldb's problem or cython's problem, seems like there are some changes in regards to malloc_default_zone, but plyvel only did used malloc cimport from clib, so don't know if any idea I could get this fixed.
Some additional issues linked to this issue.
gperftools/gperftools#827
leveldb versio is 1.15.0
plyvel version is 0.9
centos 64bit python2.7
When run with LANG=POSIX, python assumes the filesystem encoding is 'ascii' so plyvel's test for db name encoding fails:
$ LANG=POSIX python setup.py nosetests
running nosetests
running egg_info
writing plyvel.egg-info/PKG-INFO
writing top-level names to plyvel.egg-info/top_level.txt
writing dependency_links to plyvel.egg-info/dependency_links.txt
reading manifest file 'plyvel.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'plyvel/*.pxi'
writing manifest file 'plyvel.egg-info/SOURCES.txt'
running build_ext
copying build/lib.linux-x86_64-2.7/plyvel/_plyvel.so -> plyvel
.E..............................
======================================================================
ERROR: test.test_plyvel.test_open
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/home/dan/fedora/python-plyvel/plyvel-0.2/test/test_plyvel.py", line 100, in test_open
with tmp_db('úñîçøđê_name') as db:
File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/home/dan/fedora/python-plyvel/plyvel-0.2/test/test_plyvel.py", line 46, in tmp_db
name = tempfile.mkdtemp(prefix=name_prefix + '-', dir=TEST_DB_DIR)
File "/usr/lib64/python2.7/tempfile.py", line 325, in mkdtemp
_os.mkdir(file, 0700)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 7-13: ordinal not in range(128)
The reason this matters is because mock (the tool used for building Fedora packages) runs the build with POSIX locale so the test suite fails during package build. I'm not sure of any way around it apart from skipping the test if sys.getfilesystemencoding() != 'UTF-8'.
To avoid messing with iterators, the DB could have .get_or_next(...) and .get_or_prev(...) methods. In addition to the args already supported by the .get() method, it would also have include_key and include_value args.
Support for snapshots and prefixed databases sounds doable as well.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.