Comments (18)
attached.
Original comment by [email protected]
on 9 Jul 2012 at 7:11
Attachments:
from py-leveldb.
FWIW, I don't know much about the Python C API, so I'm guessing I'm messing up
the object ownership here by holding onto the name in the comparator. I'm
providing this patch merely as a starting point.
Original comment by [email protected]
on 9 Jul 2012 at 7:12
from py-leveldb.
This is something along the lines I was hoping for.
I'll just have the comparator default to "bytewise", perhaps add a custom
example, and include instructions on how to create more.
Original comment by [email protected]
on 9 Jul 2012 at 8:06
- Changed state: Accepted
- Added labels: Type-Enhancement
- Removed labels: Type-Defect
from py-leveldb.
Implemented comparator name for opening databases. Will port this to RepairDB()
as well.
Original comment by [email protected]
on 10 Jul 2012 at 12:50
from py-leveldb.
Thanks for starting this - I'm still hoping to be able to specify a comparator
name, even if I get the bytewise comparator as a result - the problem is
leveldb will refuse to open the database if the names don't match exactly - I'm
trying to use pyleveldb to examine a leveldb created by a c++ program, and
until I fully port the comparator over (it has some gnarly dependencies) I'd
like to at least let pyleveldb open it (that's what I did with the patch I have
here)
Original comment by [email protected]
on 10 Jul 2012 at 8:25
from py-leveldb.
I may have misunderstood you. You just wanted a way to override the Name()
function for a given comparator (bytewise in your patch), right?
If you´re into that territory, why not just use the bytewise comparator, and
disable the name check? Hopefully you´re not interested in mutating the
database, or getting keys in any sensible order.
And I looked the the comparator from chrome IndexedDB. Good luck with that :)
Original comment by [email protected]
on 11 Jul 2012 at 12:04
from py-leveldb.
there's no way to disable the name check in leveldb (as best as I can tell
anyway) - at least not without patching it.. I'm not mutating the database,
just doing one big scan.
I could actually write comparator fairly trivially in Python if it were
possible to actually implement a comparator in pyleveldb .. I know that's a lot
more work though.
Original comment by [email protected]
on 11 Jul 2012 at 12:09
from py-leveldb.
that´s not a bed test case.
there are a couple of implementation details that need to be worked out:
comparator exception propogation and conditional GIL releasing. nothing too
serious. you´d just pass a callable object, and a name.
GIL can´t be released for mutation or reading, that´s about the only
difference.
Original comment by [email protected]
on 11 Jul 2012 at 12:26
from py-leveldb.
I´m close to a prototype of this, but have hit a snag.
The Compare() function can be called from pretty much any thread, including the
compaction thread. Calling the Python callback from there is not a problem.
The problem comes when the Python code raises an exception, and we´re left
with no comparison result. The obvious solution would be to raise a C++
exception, but I'm worried that it would break some invariants, e.g. not
releasing mutexes.
Returning an arbitrary comparison value, and just logging the error, might
leave the database in an inconsistent state, so that's not an option.
I'm gonna try the C++ exception solution, but I'm not optimistic that it will
work.
A solution of last resort would simply be to call abort(), after printing out
the Python stacktrace.
If anybody has other ideas, I'd very much like to hear them.
Original comment by [email protected]
on 23 Jul 2012 at 11:30
from py-leveldb.
I just committed a preliminary version supporting Python based comparators.
Any feedback appreciated.
I haven´t done extensive testing on this, but I´ll close the issue once I´m
confident the code works as advertised.
Original comment by [email protected]
on 27 Jul 2012 at 12:03
from py-leveldb.
great news! I'll try to get to this today. I've rewritten a lot of the
WebKit/IndexedDB comparator code in Python for other purposes, so it shouldn't
be too hard to hook up.
Original comment by [email protected]
on 27 Jul 2012 at 4:56
from py-leveldb.
ok, finally got to try this out. A few issues:
1) just passing the comparator name, means the code complains that you should
just pass a comparator name or a tuple. But a tuple works
2) I eventually get a segfault, after lots of successful comparisons - no stack
trace, just a hard crash. I verified that it wasn't my comparator throwing an
exception, either.
Original comment by [email protected]
on 31 Jul 2012 at 5:22
from py-leveldb.
Interesting. in my particular dataset, it very consistently crashes after 253
comparisons. (using db.RangeIter())
I sprinkled some printfs around, and the crash seems to be in the releasing of
the GIL inside PythonComparatorWrapper::Compare()
Original comment by [email protected]
on 31 Jul 2012 at 5:43
from py-leveldb.
Alright. I wrote a stress test to verify this, and succeeded in crashing it,
but it happens more randomly than for you test case.
Original comment by [email protected]
on 31 Jul 2012 at 11:20
from py-leveldb.
Think I figured this one out. I'm running a test-suite on a dual core machine,
but will get the chance to run it on a bigger machine tomrrow. Can you please
see if can break this?
Original comment by [email protected]
on 1 Aug 2012 at 12:34
from py-leveldb.
I just ran my code against this, wasn't able to make it crash at all, hitting a
ldb with 2800 rows... I'm guessing that was the fix.
Original comment by [email protected]
on 1 Aug 2012 at 1:29
from py-leveldb.
FWIW I'm on a 24 core machine... my script just goes through the ldb once
though, so it's just running for a few seconds.
Original comment by [email protected]
on 1 Aug 2012 at 1:30
from py-leveldb.
Original comment by [email protected]
on 6 Aug 2012 at 11:24
- Changed state: Fixed
from py-leveldb.
Related Issues (20)
- add setuptools support HOT 3
- Return None from LevelDB.Get rather than raising KeyError (performance) HOT 5
- build fails due to __init__.py not found HOT 3
- RangeIter doesn't accept None for key_from or key_to HOT 2
- RepairDB segfaults HOT 9
- RepairDB does not accept comparators HOT 1
- Support for CompactRange() HOT 1
- 求增加连接python-leveldb 32位 for windows
- Does not build due to missing snappy header HOT 2
- Build fails with latest version of snappy and pyleveldb HOT 2
- Segmentation fault ? It was py-leveldb bug ? or levelDB bug? HOT 2
- Need a method to close/unlock the database HOT 3
- Memory leak when using WriteBatch HOT 1
- Request to update setup.py/pypi HOT 4
- Installation requires ez_setup HOT 3
- leveldb cannot be installed HOT 4
- ..
- can't iterate over the db keys & values
- Fails to build on OS X 10.9.5 with Xcode 6.1.1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from py-leveldb.