bnclabs / gostore Goto Github PK
View Code? Open in Web Editor NEWStorage algorithms.
License: MIT License
Storage algorithms.
License: MIT License
Right now Clone() implementation acquires a write while cloning the
llrb tree. I think this can be converted into read-lock.
For Arena, Calculate the size of mpools
map.
Full table scan on active tree will have following issues:
To avoid this, implement a PiecewiseIterator that scans the tree
part by part and stitches them together to simulate a full-table scan.
Here are some implementation guidelines.
metadata.bornseqno
, metadata.deadseqno
tillseqno
as a form of timestamp tonil
and endkey as nil
.high
.tillseqno
. ElseThe key difference between Iterator and PiecewiseIterator is that
PiecewiseIterator does not give a point time view of the tree snapshot.
For example, if an entry that is not yet read by the piecewise-iterator
is updated after the iteration has started, then the entry might be skipped
and this entry won't be part of the final output.
This implies:
Hopefully it can still be used with LSM reads.
Statistics are vital for debugging and characterisation. Adequate stats are implemented for malloc/ bubt/ and llrb/ packages. Create a page with a short overview of storage and memory stats and detailed description on how to interpret then and relationship between stats.
build a bubt index with empty iterator.
build a bubt index with empty metadata.
go_writer.go exports several API that internally uses unbuffered
chan []interface{}
as response channel that are purely used for
synchronisation.
May be this can be changed to chan struct{}
.
Should provide two variant.
Log-Structured-Merge (LSM) is available off-the-shelf with LLRB store.
Just enable lsm
via settings while creating the LLRB tree. Enabling
LSM will have the following effects:
NOTE: DeleteMin and DeleteMax is not useful when LLRB index is only
holding a subset, called working-set, of the full index.
Similar to llrb. Add settings parameters:
Add validation logic to check and fail.
Package malloc/ has global variable MEMUtilization used for
Blocksize
calculation. Once Slab struct
is created and
Blocksizes
and SuitableSize
are localized to Slab, we can
configure MEMUtilization per slab instance instead of keeping
it global.
Package llrb/ has config-parameter called memutilization
used
while validation. There is also an API ExpectedUtilization
that
makes the config-parameter redundant. This parameter is used while
validating llrb tree. I suppose we can can avoid both the
config-paramter and the API, instead add an argument to Validate
for expected-memory-utilization ratio.
Sizing requirements need to be presented for.
Get a clean chit from goreportcard.
In some places chan bool
is used to synchronise between go-routines. This can be replace with chan struct{}
which consume ZERO memory footprint.
Create a local data-structure and a list of the same to test LSM logic. This purpose of this test case , when compared to existing LLRB based merge index, is to make corner case testing easer to create and maintain.
Before flushing the z-blocks and m-blocks to disk, pad them to align with zblocksize and mblocksize.
When opening the snapshot, one of the header fields points to the rootblock, this should match with rootblock position after header, stats, setts and metadata.
Stats and Fullstats return a map of useful statistics about the index and operations on index.
At present malloc uses a fair-model to allocate a pool from OS. There
are some considerations while allocating an entire pool from OS:
Right now, allocator assumes that number of chunks, allocated by
application, in each slab will be same for all slab-size. This is
probably a good way to start a new arena instance, but for every
new allocation we get to know the histogram of chunk-size for
each slab and with that information we can pick an optimal size
while allocating the next pool from OS.
Add a test case that has deleted entries and entries that has not suffered deletes. Add another case where deleted entries are created once again.
Picture can say thousand words. Create a block diagram of go-routines, how the interact with application logic and underlying socket. Mostly it is important to trace the execution path and its exceptional cases.
Returned statistics map should include all statistics that are relevant to the btree index.
Tree validation. This would involve walking the entire tree.
When Opening a well-formed bubt-index cache the intermediate nodes (m-blocks) in memory. Either using golang map
or using llrb
or some other fast lookup mechanism. It should typically be a map of fpos->mblock buffer.
Implement this as a configurable feature.
LSM merges pre-sorted input. There might be a better way than using sort.Sort().
FailsafeRequest and FailsafePost can be localized as gen-server methods. That way we can be specific about the channel type instead of typing it as chan []interface{}
.
There might be imports of the form
import _ "fmt"
or
import "fmt"
var _ = fmt.Sprintf(...)
Remove them. Search the net ask community for alternate ideas.
command line program to check the bubt file and its options value file.
Involves the following APIs on bubt.Snapshot{} - GetClock().
Returned statistics map should include only pre-computed statistics. Avoid walking over the tree.
This must be a cheap call. Some of the statistics should include:
All of them can be part of the infoblock.
Stats and Fullstats return a map of useful statistics about the index and operations on index.
Sizing information need to be presented for:
Start a slack channel for the following projects - golog, gosettings, gson, gofast, gostore. If possible start it under gophers.slack.com
Reverse link them with golog, gosettings, gson, gofast, gostore.
When there are no active snapshots, MVCC operations like Upsert, UpsertCAS, DeleteMin, DeleteMax and Delete can be fallback to non-mvcc operations. This can give 2x improvement.
Create a command line tool, may be outside gostore repository, to:
Let this be easily repeatable. Add cmd-line options to integrate the tool with gostore CI.
Create a test case for UpsertCAS with CAS value as ZERO. This is same as CREATE operation.
Picture can say thousand words. Create a block diagram of go-routines, how the interact with application logic and underlying socket. Mostly it is important to trace the execution path and its exceptional cases.
Start a proper README page for gostore. Every sub-package should have a README package. Let the README content have the following template.
For sub-pkg README, keep the original details with sub-pkg's doc.go. Include introduction to basic concepts, ideas, and getting-started guid.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.