antidotedb / antidote Goto Github PK
View Code? Open in Web Editor NEWA planet scale, highly available, transactional database built on CRDT technology
Home Page: https://www.antidotedb.eu
License: Apache License 2.0
A planet scale, highly available, transactional database built on CRDT technology
Home Page: https://www.antidotedb.eu
License: Apache License 2.0
Tools?
Demo for meeting in October
Responsible: Manuel
Collaborator: Alejandro
Read latency increases when more writes are happening as each read is processing all log entries for the respective object.
So far we have just tried with dets. We should try with something else and see what performs better. I have abstracted the operations to the persistent store in the logging_vnode to ease the shifting to other persistent storage systems.
Apache 2.0 was to be used for the SyncFree github
Dialyzer produces this error message:
floppy_rep_vnode.erl:51: Expression produces a value of type {'error',} | {'ok','undefined' | pid()} | {'ok','undefined' | pid(),}, but this value is unmatched
Similar for
floppy_rep_vnode.erl:67
floppy_rep_vnode.erl:100
This is due to calls such as:
floppy_coord_sup:start_fsm([self(), append, Key, Op])
receive
{ok,{_, Result}} ->
lager:info("Append completed!wn",[Result]),
{ok, Result}
after 5000 ->
lager:info("Append failed!~n"),
{error, timeout}
end.
It seems that the receive is trying to match another value?
The root of the cause seems to lie with starting child processes in floppy_coord_sup:start_fsm.
I've found an issue in the clock-SI + partitioned log branch (as I have merged them into clock-SI to support the "one log per preflist" feature).
In this branch, there are currently 4 files opened "per vnode" (as there is a logging_vnode that opens N; a log per partition, and a clock-SI vnode that opens one).
When starting a single node cluster, all of these files are opened by a single unix process. I was getting the error:
[error] Supervisor riak_core_vnode_sup had child undefined started with {riak_core_vnode,start_link,undefined} at <0.1453.0> exit with reason bad return value: {stop,{file_error,"./data/1301649895747835411525156804137939564381064921088_log3",emfile}} in context child_terminated
Unix allows, by default, a maximum of 256 file descriptors opened per process, and this limit was exceeded when running a single node cluster. you can check that in your machine with "ulimit -n".
change (only for the current session) the maximum number of file descriptors allowed per process with:
ulimit -n new_limit
Finish this.
We should remove the branches that have already been merged to master and the ones that we don't expect to work on any longer.
Please, each branch creator report which branches he/she knows that we can delete now.
Reads and writes should return operation id
It needs info on:
It would be nice to also have documentation for:
Break up log in different logs per partition -> N logs per vnode
Ensure that log is correctly sorted based on insertion order
Maybe choose an alternative to DETS for better performance, e.g. LevelDB?
Responsable: Manuel
Participating: Alejandro
Another test case for causality across partitions
Currently, the type for a CRDT is specified only when reading.
However, generating downstream operations requires the type of CRDT, which is now hardcoded to riak_dt_pncounter.
Therefore, when using types other than counter, downstream op generator will throw exceptions.
Modify wallet code to not use RPC's, but instead use the API.
log_utilities:get_preflist_from_key(Key)
and log_utilities:get_preflist_from_logid( get_logid_from_key(Key)) does not return same value.
Similarly, get_logid_from_partition(P) is not same as get_logid_from_key(Key), where
vnode with partition number P is head of preflist of Key. In many cases, we need to reach vnodes using partition number, without knowing which keys they are responsible for. How to fix this?
floppy_coord_fsm.erl:81: The call
floppy_rep_vnode:handleOp(IndexNode::integer(),pid(),Op::atom(),Key::any(),Param::any())
will never return since it differs in the 1st argument from the
success typing arguments: (maybe_improper_list({integer(),_},[] |
{integer(),_}) | {integer(),_},any(),any(),any(),any())
floppy_coord_fsm.erl:100: The call
floppy_rep_vnode:handleOp(IndexNode::integer(),pid(),Op::atom(),Key::any(),Param::any())
will never return since it differs in the 1st argument from the
success typing arguments: (maybe_improper_list({integer(),_},[] |
{integer(),_}) | {integer(),_},any(),any(),any(),any())
Replacing the rpc calls
Hello, after discussing with Deepthi, we agreed that we can remove the following bramches as they are inactive:
The clock si functionallity is now provided in the causality branch, that has already a pull request to be merged to master. Then, in master, we should create some configuration mechanism in order to enable/disable the use of the protocol.
Restructure the log entries to generate down-stream operations for inter-DC replication
Check correctness of protocols:
Are the data structures used the right ones?
Are messages all handled correctly?
Is fault-tolerance achieved? At which level?
Responsible: Alejandro
The signature of floppy_rep_vnode:handleOP differs from its usage in:
floppy_coord_fsm.erl:81
floppy_coord_fsm.erl:100
Add timeouts and error message handle for states waitread, waitappend. repairrest.
Today, I've realised that the following error occurs when trying to build any branch (including master):
riak_test/floppystore/.rebar_plugins/rebar_test_plugin.erl:96: variable 'OutputDir' is unused
ERROR: riak_test_compile failed while processing /Users/alek/Dropbox/PhD/code/floppystore/floppystore: rebar_abort
make: *** [compile-riak-test] Error 1
Add test
Setup demo
Should be an additional causality check
Neither of these tests is actually testing anything; append_list generates a partition after writing to the partitioned node, and is not exercising the behavior correctly.
Participating: Alejandro
Now if rep_vnode restarts, opid is restarted from 1 which assigns duplicate opid to new updates which makes these updates invisible.
Hello,
As we had agreed on the mumble meetings, branches that develop features like interDC replication or clockSI will not be merged to master in order to keep different "flavours" of floppystore, and make those features optional.
I wanted to ask when or how do we ask for code review for those branches. e.g. the "clock-SI_v2" branch and the "causality" branch have not been reviewed and it would be nice to have a review at this stage.
walletapp.erl:13: Expression produces a value of type {'error',_} | {'ok','undefined' | pid()} | {'ok','undefined' | pid(),_}, but this value is unmatched
walletclient.erl:10: Function start/1 has no local return
walletclient.erl:14: Function run/1 has no local return
walletclient.erl:19: The call walletclient:testvoucher(Key_voucher::any(),10,[]) will never return since it differs in the 2nd argument from the success typing arguments: (any(),0,[any()])
walletclient.erl:26: Expression produces a value of type 'undefined' | {integer(),integer(),integer()}, but this value is unmatched
walletclient.erl:34: Expression produces a value of type 'ok' | {'error',string()}, but this value is unmatched
walletclient.erl:36: The pattern {'error', Reason} can never match the type 'error' | number()
walletclient.erl:39: The variable _ can never match since previous clauses completely covered the type 'false'
walletclient.erl:44: Expression produces a value of type 'ok' | {'error',string()}, but this value is unmatched
walletclient.erl:46: The pattern {'error', Reason} can never match the type 'error' | number()
walletclient.erl:49: The variable _ can never match since previous clauses completely covered the type 'false'
walletclient.erl:52: Function testvoucher/3 has no local return
walletclient.erl:52: The pattern <_, 0, Result> can never match the type <_,10,[]>
walletclient.erl:56: Expression produces a value of type 'undefined' | {integer(),integer(),integer()}, but this value is unmatched
walletclient.erl:61: Function testadd/2 has no local return
walletclient.erl:63: Expression produces a value of type 'ok' | {'error',string()}, but this value is unmatched
walletclient.erl:65: The created fun has no local return
walletclient.erl:65: The call lists:usort(nonempty_improper_list(pos_integer(),number() | {'error',string()})) will never return since it differs in the 1st argument from the success typing arguments: ([any()])
walletclient.erl:65: Cons will produce an improper list since its 2nd argument is number() | {'error',string()}
Reading from the log in the logging vnode does not ensure operations are ordered given the op_id. This needs to be done to ensure the correct ordering of update operations.
The merge function should fail and raise an error, even / especially if it is not implemented.
How is it related to merge_to? This looks like an implementation of merge.
Setup a benchmark for (unbounded) CRDT types, e.g. Counters
Do debugging and improving of Floppystore
Show throughput
Figure out why.
so people can build something on top
Change all CamelCase to snake_case, and while you're in the files, fix the tab indentation!
Compare performance, message size, log size, ...
Re-use implementation from riak_DT/ riak_KV
There is bug caused by network partitions. The replication fsm only sends to the primary vnodes that are up. The preflist used to send to the vnodes is the one used for identifying the log in the logging_vnode. Since some vnodes might not be available, the preflist will not contain the down vnode; therefore, the preflist (composed by less than N ids) will not much any of the preflist that the vnode expects. In consequence the operation will fail.
When the replication coordinator generates a quorum_unreachable message, the coordinator for the request returns a normal timeout message, because the FSM fails to forward it back. Fix this.
The materializer has O(n) performance on operations, which needs to be addressed as a scalability concern.
Want to have tests the same location as code.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.