vrv / fawn-kv Goto Github PK

When the system is run with R=1 (no replicas), then there is a potential consistency bug (as verified by dBug: http://www.cs.cmu.edu/~jsimsa/dbug/ )

Specifically, when a node joins the system, it takes over for a range from another node. Get() requests during a particular time in this process must be served by an interim tail (the node before the old tail) of a chain. With R > 1, this interim tail exists, but when R=1, there is no interim tail. Therefore, these requests need to be buffered and sent later, or alternatively should return an error of the form "try again" so that the node join can complete fully.

Currently, I'm trying to use the FAWN-KV. but after setup, it can't work.

the compile
autoreconf -is
./configure
make

then,

cd fawnkv; ./manager 127.0.0.1 -j
cd fawnkv; ./backend -m 127.0.0.1 -i 127.0.0.1 -p 8000 -j
cd lib/cpp; ./tester -c 127.0.0.1 -p 10000 127.0.0.1

But it enter to the loop of "connecting to 127.0.0.1:8000"

Would you mind give me some suggestion?

These got inserted during some of Wyatt's WideKV work, but they need to be pulled out of the master branch until it officially is added as a feature to FawnKV (if it ever does :)).

Following temporary thrift client creations (e.g., communicating between a joining node and an old tail to get updates), the current code does not delete the thrift client object used to communicate. It used to be done as follows

client->rpc_call(...);
delete client;

But this would result in segmentation faults, presumably because the rpc_call() code has some degree of asynchrony with respect to network sockets, etc. To ensure the system doesn't crash, we don't delete these temporary objects, but we should figure out 1) whether this is a thrift issue or 2) what we can do in our code to perform the cleanup.

Taking a cue from CRAQ (http://www.usenix.org/event/usenix09/tech/full_papers/terrace/terrace.pdf), we should implement a form of query-anywhere chain replication in FAWN-KV to load balance keys across our chain replicas. When using a front-end, this can be simply done by keeping track of 'outstanding key updates' at the frontend: on a lookup, you check to see if the key is being updated, and if so, send the request to the tail. Otherwise the key is safe to be requested from any of the backends.

When not using a frontend, we would have to directly implement CRAQ: there's no frontend to keep track of pending updates in the chain and so it would require communication between the nodes in the chain.

There are some locking issues in FawnKVFrontendHandler.cpp and Manager.cpp (according to TODOs in the code).

Also, chain_repair_mid almost certainly does not work properly. At some point we consolidated the put() and remove() functions to just use a put() + a boolean flag indicating if it's a remove(). The chain_repair code requires state maintenance of pending put requests, but it has not been updated to also include the remove flag as part of the state. Also, it appears the 'flush' boolean isn't being kept track of either (though this may be intentional). None of our standard tests test for this particular condition (it requires a flush/remove to be inserted at a precise time during chain repair).

The problem will likely manifest in a consistency error or in an empty object insertion instead of removal. The wrong flush flag will send an update to the temporary datastore and overwrite a new value with an old one, and the incorrect remove flag will try to insert an empty valued object instead of a 'remove()' request.

Right now our merge code does not lock the datastore it inserts into; this assumes that the file being merged into is not "live" or actively being inserted into. At some point we changed the fawnkv code to make some of the distributed part of the code more simple and thus broke this assumption.

Two possible solutions:

1. Acquire a lock on the datastore during this insertion ( fawnds.cc line 891 ) and release it right after.

2. Change FawnKV code to merge differently: it can create a new datastore and then sequentially merge two contiguous keyranges into that file. Since the ranges are disjoint there are no ordering/consistency issues to worry about.

Alternatively, await Reinhard's changes to FAWNDS to solve most of these issues.

Current version has only partial support for multiple vnodes. You can "statically" join with multiple vnodes, but adding a new server "dynamically" only results in the first vnode joining.

fawnkv/FawnKVBackendHandler.cpp:292 shows where the next vnode join request should be sent