Comments (4)
You can add a BlockCache in open options.
options.BlockCache = rosedb.MB * 64
because you iterate all data in wal, block cache fits for the case(sequential reading).
from rosedb.
@taroim it makes sense that Ascend will take more time since it also fetches values from disk where the iterator solely iterate over the in-memory index (i didnt really read the code so i assume it). some possible optimizations that i can think of is that it worth checking the implementation of the BTREE and see if it all leaf nodes (the ones that contain the KV pairs) are chained so it easy to iterate over them and not going back to their parents. also reading from disk already optimized by the kernel's page cache where it fetches disk pages and by that we gain alot of cache hits when reading sequentially. and as @roseduan it wont harm if we add another applicative layer of caching that is doing the same. BTW in the current version the functionallity to solely iterate over keys still exists?
EDIT: ok so after giving it a second thought this is not actually the case. There may be a lot of random io going on when ascending the btree which is a performance penalty. I'll explain, so basically the keys are ordered by bytes.compare function in the btree and not by their insertion time. This means that fresh keys can be placed adjacent to old keys which their values are deep in the wal files. So when ascending the btree each key may point to a different block in the wal file thus creating a lot of random io. So I'm not sure enabling the block cache will help since it will cause trashing as well. @roseduan please correct me if I'm wrong
from rosedb.
@roseduan Add the BlockCache option will lead to increased memory overhead. Could you consider implementing a method similar to Redis' "KEYS," which allows searching for keys matching a provided pattern?
from rosedb.
@roseduan Add the BlockCache option will lead to increased memory overhead. Could you consider implementing a method similar to Redis' "KEYS," which allows searching for keys matching a provided pattern?
OK, search keys are easy to support. I will open an issue for that.
from rosedb.
Related Issues (20)
- V2.3.2 Release Plan
- add example code(in examples folder) for TTL (good first issue)
- add example code(in examples folder) for iterating (good first issue)
- Question: Hint file usage HOT 3
- panic: send on closed channel HOT 1
- Suggestion: add a Command for Removing Key Expire HOT 9
- Expired Keys Found in AscendKeys/DescendKeys HOT 4
- Optimize memory usage
- Proposal: keep key and value into memory HOT 3
- Proposal: allow key and value to store per bucket HOT 1
- Proposal: raft protocol support HOT 2
- Possible project use HOT 1
- about replicator HOT 1
- 有个问题,如果key过期了,不执行merge就永远不会删除文件是吗 HOT 9
- is hash ops still supported? HOT 1
- rose.Get panil HOT 10
- Bug: Is it a bug in *Batch.Put? HOT 2
- Proposal: kv db base on remote file system HOT 4
- [BUG] unknown field BlockCache in struct literal of type wal.Options HOT 2
- runtime error: slice bounds out of range [:54721] with capacity 32768 HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rosedb.