willemt / raft Goto Github PK

version 0.7.0
The code below，when "raft_send_appendentries" return not 0，the cycle will return. So if raft_node A is offline， B need send appendentries to A and C ，send A callback return -1，here return， C will not receive heatbeat？C will request vote？

int raft_send_appendentries_all(raft_server_t* me_)
{
raft_server_private_t* me = (raft_server_private_t*)me_;
int i, e;

__log(me_, NULL, "%s:%d num node=%d",__FILE__, __LINE__, me->num_nodes);
me->timeout_elapsed = 0;
for (i = 0; i < me->num_nodes; i++)
{
    if (me->node == me->nodes[i] || !raft_node_is_active(me->nodes[i]))
        continue;

    e = raft_send_appendentries(me_, me->nodes[i]);
    if (0 != e)
        return e;
}

return 0;

}

The current implementation unconditionally sends empty an AppendEntries request every time the heartbeat interval (request_timeout) expires.

However sending empty/heartbeat AppendEntries is only required if no "actual" non-AppendEntries request was sent recently (where here "recently" essentially means within request_timeout milliseconds).

Suppose that the circular log buffer is of size 3 and has the following layout:

[<entry>, NULL, <entry>]
    ^              ^
   back          front

base index is 3 (so <entry> at front has index 3 and <entry> at back has index 4)

then it seems that a call to log_get_from_idx(3) instead of returning an array of 2 entries which includes entries 3 and 4, returns an array of just 1 entry which includes only entry 3.

This can be reproduced with the following (failing) test:

void TestLog_get_from_idx_with_wrapping(CuTest * tc)
{
    void* queue = llqueue_new();
    void *r = raft_new();
    raft_cbs_t funcs = {
        .log_pop = __log_pop,
        .log_get_node_id = __logentry_get_node_id
    };
    raft_set_callbacks(r, &funcs, queue);

    void *l;
    raft_entry_t e1, e2, e3, e4;

    memset(&e1, 0, sizeof(raft_entry_t));
    memset(&e2, 0, sizeof(raft_entry_t));
    memset(&e3, 0, sizeof(raft_entry_t));
    memset(&e4, 0, sizeof(raft_entry_t));

    e1.id = 1;
    e2.id = 2;
    e3.id = 3;
    e4.id = 4;

    l = log_alloc(3);
    log_set_callbacks(l, &funcs, r);

    raft_entry_t* ety;

    /* append append append */
    CuAssertIntEquals(tc, 0, log_append_entry(l, &e1));
    CuAssertIntEquals(tc, 0, log_append_entry(l, &e2));
    CuAssertIntEquals(tc, 0, log_append_entry(l, &e3));
    CuAssertIntEquals(tc, 3, log_count(l));

    /* poll poll */
    CuAssertIntEquals(tc, log_poll(l, (void*)&ety), 0);
    CuAssertIntEquals(tc, ety->id, 1);
    CuAssertIntEquals(tc, log_poll(l, (void*)&ety), 0);
    CuAssertIntEquals(tc, ety->id, 2);
    CuAssertIntEquals(tc, 1, log_count(l));

    /* append */
    CuAssertIntEquals(tc, 0, log_append_entry(l, &e4));
    CuAssertIntEquals(tc, 2, log_count(l));

    /* get from index 3 */
    int n_etys;
    raft_entry_t* etys;
    etys = log_get_from_idx(l, 3, &n_etys);

    CuAssertPtrNotNull(tc, etys);
    CuAssertIntEquals(tc, n_etys, 2);

    CuAssertIntEquals(tc, etys[0].id, 3);
    CuAssertIntEquals(tc, etys[1].id, 4);
}

It seems the bug is log_get_from_idx() when it calculates the length of the array to be returned:

    if (i < me->back)
        logs_till_end_of_log = me->back - i;
    else
        logs_till_end_of_log = me->size - i;

In the else case the length should actually be me->size - i + me->back, to include the most recent entries that have wrapped around. However it's not clear how to return a pointer without performing some allocation, because of course pointers don't wrap around.

Does it make sense? Is this a bug?

raft_recv_entry -> raft_recv_entries
raft_append_entry -> raft_append_entries
applylog -> apply_logs
log_offer -> offer_logs
log_poll -> poll_logs
log_pop -> pop_logs

Hi,

As described in another issue, changing #if 0 /* debugging */ to #if 1 /* debugging */ fails to compile due to duplicate definitions and undefined functions. The line is:

https://github.com/willemt/raft/blob/master/src/raft_server.c#L36

Cheers,

Pete

it's about 1.5 year since the 0.5.
So, do you plan to make 0.6?

Hi,

Our implementation is pretty complete now - few more hurdles!

We are running raft in embedded terminals that communicate locally on a wired/wireless network. typically there would be 2-10 terminals participating in a cluster.

Each terminal applies entries from the log via applylog() to their local sqlite db but there is no mechanism at present to ever truncate that log. As the system is used it will become unmanageably large - the log is loaded on startup so we can provide log entries to followers if necessary, but loading this log is already becoming slow.

In this application there is a concept of agreeing on a time period. This is a point at which all earlier log entries are redundant; they might be applied to the local db but they are no longer relevant anyway. For example, entries from yesterday might be unimportant and candidates for removing.

My plan to compact the log is to append a log entry when this period is reached, this log entry will include the entry_idx. The other terminals commit this, when they apply they truncate all entries up to and including this entry.

Thoughts?

Neil

https://pypi.python.org/pypi/blockade

in Test_server.c , function TestRaft_server_recv_requestvote_response_increase_votes_for_me may have some error.
void TestRaft_server_recv_requestvote_response_increase_votes_for_me(
CuTest * tc
)
{
void *r = raft_new();
raft_add_node(r, NULL, 1, 1);
raft_add_node(r, NULL, 2, 0);

this action will increase term id to 2;

raft_set_current_term(r, 1);   
CuAssertTrue(tc, 0 == raft_get_nvotes_for_me(r));

this action will increase term id to 2;

raft_become_candidate(r);   
msg_requestvote_response_t rvr;
memset(&rvr, 0, sizeof(msg_requestvote_response_t));
rvr.term = 1;
rvr.vote_granted = 1;
raft_recv_requestvote_response(r, raft_get_node(r, 2), &rvr);   

this action wont increase vote because rcr.termid =1 < current term id;

CuAssertTrue(tc, 1 == raft_get_nvotes_for_me(r));

}
wonder is this a mistake or my understanding error?

Hi:
Thank you for this wonderful project!
I have a few questions about the entry log I don't quite understand.

typedef struct
{
    void *buf;

    unsigned int len;
} raft_entry_data_t;

int the raft.h use raft_entry_data_t store entry.when to release the buf ? ?
when to call the log_pop and log_poll (not have conflicting entries) ??
And if we use variable-length entry how to handle it ??

When preparing to send AppendEntries, followers that lag behind last_snapshot_idx will not have a snapshot sent instead of AE. However, this state will persist as not sending AE means there's also no way to track their current index.

Currently the way to deal with this is for the application to manually call raft_node_set_next_idx() after snapshot has been installed. I think ideally the library should deal with it, although this may be good enough if documented.

In raft_log.c

int log_load_from_snapshot(log_t *me_, raft_index_t idx, raft_term_t term)
{
    log_private_t* me = (log_private_t*)me_;

    log_clear_entries(me_);
    log_clear(me_);
    me->base = idx;

    return 0;
}

The term is unused.

So could be safely deleted from source code?

Here's a major refactoring step to consider.

Currently the server logic and log implementation are quite coupled, to the point where the log implementation in many cases is responsible to trigger operations on the Raft side.

Decoupling would not just clean things up, but also make it possible to replace the log implementation. For example, it may be useful to implement a disk based log that does not require all entries to be in memory at all time and can fetch them from disk on demand.

Hi,

In my code I set election_timeout and request_timeout to say 5,000ms. I then call raft_periodic every 1,000ms. In my own callback ( the one that calls raft_periodic) it prints PING! every time it is executed.

Previously, I would see five of these pings before the timeout was hit and raft started sending out vote requests. However in the latest version, every time I call raft_periodic it also calls the log function.

So two questions:

1. What does the log function actually do? It's not documented as far as I can tell...
2. Why is it being called each time?

Thanks!

Cheers,

Pete

raft is a dependency of our app, when installing the major version of libraft is missing:
Error: Package: daos-server-0.9.0-2.2819.gcdecd456.el7.x86_64 (/daos-server-0.9.0-2.2819.gcdecd456.el7.x86_64)
Requires: libraft.so()(64bit)

The version of libraft.so is missed in this message, it should be: libraft.so.$n

How can we add the raft SOVERSION to it, probably in the Makefile?
.PHONY: shared
shared: $(OBJECTS)
$(CC) $(OBJECTS) $(LDFLAGS) $(CFLAGS) -fPIC $(SHAREDFLAGS) -o $(BUILDDIR)/libraft.$(SHAREDEXT)

Thx.

See Ongaro, D. (2014). Consensus: bridging theory and practice

Struct field raft_server_private_t.connected is set in the only place into RAFT_NODE_STATUS_CONNECTED state. But this field is compared in the other place with the const RAFT_NODE_STATUS_DISCONNECTING.
Moreover, connection enum has 2 more states.
So, I suppose that there's some missed functionality which is set RAFT_NODE_STATUS_DISCONNECTING and RAFT_NODE_STATUS_CONNECTING values.

Functions raft_send_requestvote and raft_send_appendentries return 0 in any case.
At the same time they call callbacks which return errorcodes.
May be the output from callbacks should be transmitted to the functions output?

synchronously writing log to disk can blocks leader's main loop, and causes it to timeout, then the cluster has to reelect a new leader, this impacts cluster availability, I think in leader, there should have a thread to flush log to disk, an index field indicating at which point the log is flushed, the the committed index should be based on this, for follower, log is synchronously written.

--
update:

also it could be better to apply log in asynchronous manner to not block main loop.

You mentioned some of the callbacks that need to be implemented in the README. Sadly in our case we must also implement the log_clear callback for the case we load a snapshot with log_load_from_snapshot. Because in this case the whole log should be cleared (is empty afterwards) before applying the snapshot. Initially I thought that this will be done by using the callbacks log_pop and/or log_poll but it isn't. Only log_clear is implicitly called by using log_clear_entries internally of log_load_from_snapshot.

In our case we still had 1 entry in our own log where the raft internal log was empty after loading a snapshot.

Either this is a bug in the documentation (because this callback is needed if working with callbacks) or the implementation of log_load_from_snapshot should handle the case where the callback is not available.

In addition the description of the callback is not very clear to me:

    /** Callback called for every existing log entry when clearing the log.
     * If memory was malloc'd in log_offer and the entry doesn't get a chance
     * to go through log_poll or log_pop, this is the last chance to free it.
     */

I do not see the requirement to individually handle single items. In my implementation I simply clear my whole log at once. I am using an std::queue with smart enough element class that manages its internal memory. Therefore I simply call: log_.clear();.

I have done the following testing:

1. I applied one log item to three nodes, and persisted.
2. delete persist data from one node (not leader) and start raft (other nodes keep alive).
3. raft log will not be applied for this node, there is some log on this node:
   raft: AE no log at prev_idx 1
   send: {"myid": 1, "message_type": "append_entries_response", "term": 1, "success": 0, "current_idx": 0, "first_idx": 2}

Then it will be loopback forever.

Hi,

Okay, sorry for raising all these issues :)

This one I'm also a little unsure of. Basically I've got the nodes communicating but I have not wired up the ReceiveAppendEntries. So what I expect to see is that they have a vote, one of them wins the vote and becomes the leader and then they start sending AppendEntries. However the other node can't see this so it should timeout and restart the voting process - which it does.

When the first node gets the first vote request, it votes no because it is already in the term. It then gets another vote request for the the next term which it then correctly votes yes to.

But here's the weird thing - after it votes yes, it goes back to sending AppendEntries. This means after a few terms, all my nodes are sending them as they all think they're the leader. Here's some of the output:

Node1 is requesting a vote on term 1
Response:  Term [1] Vote Granted [0]

Timer hit: Send append entries!

Node1 is requesting a vote on term 2
Response:  Term [1] Vote Granted [1]

Timer hit: Send append entries!

I would expect that after a Leader grants a vote, it should step down as the Leader right? I may actually have this totally wrong, and maybe this is not the behaviour I should expect.

If this is the right behaviour, then I guess we need to update:

What do you think?

Cheers,

Pete

PS How do you enable the built in logging that I just noticed? :)

in raft_recv_appendentries_response(raft_server_t* me_, raft_node_t* node, msg_appendentries_response_t* r)

int votes = 1; 
int point = r->current_idx;
int i;
for (i = 0; i < me->num_nodes; i++)
{
    if (me->node == me->nodes[i] || !raft_node_is_voting(me->nodes[i]))
        continue;

    int match_idx = raft_node_get_match_idx(me->nodes[i]);

    if (0 < match_idx)
    {
        raft_entry_t* ety = raft_get_entry_from_idx(me_, match_idx);
        if (ety->term == me->current_term && point <= match_idx)
            votes++;
    }
}
if (raft_get_num_voting_nodes(me_) / 2 < votes && raft_get_commit_idx(me_) < point)
    raft_set_commit_idx(me_, point);

I add an if before the loop in the code based on two reasons:

• we first check if commit index < point to avoid unnecessary loop
• the raft paper said if we are going to change leader's commit index to point, we only need to make sure that log[point].term == current_term

int point = r->current_idx;
raft_entry_t* ety = raft_get_entry_from_idx(me_, point)
if (raft_get_commit_idx(me_) < point && ety->term == me->current_term)
{
    int votes = 1;
    int i;
    for (i = 0; i < me->num_nodes; i++)
    {
        if (me->node == me->nodes[i] || !raft_node_is_voting(me->nodes[i]))
            continue;
        int match_idx = raft_node_get_match_idx(me->nodes[i]);
        if (match_idx >= point)
            vote++;
    }
    if (raft_get_num_voting_nodes(me_) / 2 < votes)
        raft_set_commit_idx(me_, point);
}

This happens if you call raft_get_current_leader() before the node has a chance to work out its state. In my case, it claimed the leader was something like 14424808. After the first round of voting, this was sorted out.

I was assuming that the log_offer callback always appends to the log. However, the callback has an entry_idx parameter. Does this mean that log_offer might require overwriting an old entry in the log, and writing beyond the current head of the log?

On a related note: are the newest/oldest log entries for log_pop/log_poll determined by entry_idx, or the order in which log_offer is called for entries?

Hi, I was wondering whether these two functions are considered obsolete or conversely prepared for the future use.

/**
 * @return this server's node ID */
int raft_get_my_id(raft_server_t* me) { return 0; }

/**
 * @return 1 if node is leader; 0 otherwise */
int raft_node_is_leader(raft_node_t* node) { return 0; }

They have no implementation as for now.

in test_server.c ,function TestRaft_leader_recv_entry_resets_election_timeout may exists some error .

void TestRaft_leader_recv_entry_resets_election_timeout(
CuTest * tc)
{
void *r = raft_new();
raft_set_election_timeout(r, 1000);
raft_set_state(r, RAFT_STATE_LEADER);

if we change 900 to 1, this test wont pass,because raft_periodic function will call raft_send_append_entries_all function when msec_since_last_period > request_timeout (default 200), which will reset the timeout_elapsed

raft_periodic(r, 900);

/* entry message */
msg_entry_t mety = {};
mety.id = 1;
mety.data.buf = "entry";
mety.data.len = strlen("entry");

add a new assert,this is true and still can pass the tests

CuAssertTrue(tc, 0 == raft_get_timeout_elapsed(r));

/* receive entry */
msg_entry_response_t cr;

this does not change the timeout_elapsed at all

raft_recv_entry(r, &mety, &cr);
CuAssertTrue(tc, 0 == raft_get_timeout_elapsed(r));

}

I would like to point out that an identifier like “__raft_malloc” does eventually not fit to the expected naming convention of the C language standard.
Would you like to adjust your selection for unique names?

Hi Willemt,
I like your work so much. I have a question here:

void raft_set_current_term(raft_server_t* me_, int term)
{
raft_server_private_t* me = (void*)me_;
me->current_term = term;
}

Like this,
why not just use raft_server_t, since it is already void*.
I don't quite understand. Could you please tell me?

Thank you

This might be considered a issue or simply a design choice, but the fact raft_recv_appendentries_response does not invoke raft_apply_all after updating the commit index and rather relies on the next call of raft_periodic to do that, means that a client call to raft_msg_entry_response_committed might return 1 even if the FSM hasn't yet applied the log entry.

It feels this might cause confusion to users or timing-related issues, see for example the scenario I described in issue #12 of ticketd.

Please let me know if my reading is correct. Thanks!

Hi Willem,

Your implementation on Raft is pretty cool for solving the replicated finite state machine problem. If I only focus on the leader election, no log needs to maintain (e.g., no raft_recv_entry() call from clients), which means we only depend on the term parameter for the election, could I still use it as a leader election?

Thanks,
Shifeng

I notice that there is a branch names 'membership changes', so does this lib supports membership changes now ?

consider this situation, leader sends a few logs(first rpc) to follower, before the response return to the leader, a heartbeat rpc(second rpc) arrives this follower, the follower will delete first rpc's logs because the second rpc's prev_log_idx is the same as in first rpc, when the first rpc returns to the leader, it says logs replicated successfully, but in fact the logs have been deleted, so i think maybe we should not delete the logs when it is heartbeat.

in raft_server.c : raft_recv_appendentries: 385-389

if (ae->n_entries == 0 && 0 < ae->prev_log_idx && ae->prev_log_idx + 1 < raft_get_current_idx(me_)) 
    { 
        assert(me->commit_idx < ae->prev_log_idx + 1); 
        raft_delete_entry_from_idx(me_, ae->prev_log_idx + 1); 
    }  

in Test_server.c, function TestRaft_follower_recv_appendentries_delete_entries_if_conflict_with_new_entries my be some problem.

void TestRaft_follower_recv_appendentries_delete_entries_if_conflict_with_new_entries(
CuTest * tc)
{
msg_appendentries_t ae;
msg_appendentries_response_t aer;

void *r = raft_new();
raft_add_node(r, NULL, 1, 1);
raft_add_node(r, NULL, 2, 0);

raft_set_current_term(r, 1);

char* strs[] = {"111", "222", "333"};

termid of entries in log are 1

raft_entry_t *ety_appended = __entries_for_conflict_tests(tc, r, strs);

pass a appendentry that is newer  */
msg_entry_t mety = {};

memset(&ae, 0, sizeof(msg_appendentries_t));
ae.term = 2;

prev_log_idx is 1 and prev_termid is 1 ,so there is no conflict with entries in log. maybe ae.prev_log_term should change to 2

ae.prev_log_idx = 1;
ae.prev_log_term = 1;

this entry termid is 0, so conflict with previous one

/* include one entry */
memset(&mety, 0, sizeof(msg_entry_t));
char *str4 = "444";
mety.data.buf = str4;
mety.data.len = 3;
mety.id = 4;
ae.entries = &mety;
ae.n_entries = 1;

raft_recv_appendentries(r, raft_get_node(r, 2), &ae, &aer);
CuAssertTrue(tc, 1 == aer.success);
CuAssertTrue(tc, 2 == raft_get_log_count(r));
CuAssertTrue(tc, NULL != (ety_appended = raft_get_entry_from_idx(r, 1)));
CuAssertTrue(tc, !strncmp(ety_appended->data.buf, strs[0], 3));

}

wonder if this is a mistake or my understanding error? thanks !

It would be nice if you create a dist folder per release with the amalgamated source file included so that when you're upgrading between releases you simply need to add $(RaftDir)\dist as include directory and off you go 🍻

in test_server.c

void TestRaft_server_recv_requestvote_response_increase_votes_for_me(
CuTest * tc
)
{
void *r = raft_new();
raft_add_node(r, NULL, 1, 1);
raft_add_node(r, NULL, 2, 0);
raft_set_current_term(r, 1);
CuAssertTrue(tc, 0 == raft_get_nvotes_for_me(r));

vote for myself ,so vote nums is 1

raft_become_candidate(r);

msg_requestvote_response_t rvr;
memset(&rvr, 0, sizeof(msg_requestvote_response_t));
rvr.term = 1;
rvr.vote_granted = 1;

recv vote response, so vote nums increases to 2

raft_recv_requestvote_response(r, raft_get_node(r, 2), &rvr);

maybe this should be 2

CuAssertTrue(tc, 1 == raft_get_nvotes_for_me(r));

}

wonder is my understanding error, or a mistake ,thanks!

Hi,
I'm new to the library, so please feel free to close this issue if this is not relevant. When I was sending a vote request, I noticed that last log term was always a large random number. I was expecting it to be zero at this stage.

In this piece of code:

https://github.com/willemt/raft/blob/master/src/raft_server.c#L422

int raft_send_requestvote(raft_server_t* me_, int node)
{
    raft_server_private_t* me = (raft_server_private_t*)me_;
    msg_requestvote_t rv;

    __log(me_, "sending requestvote to: %d", node);

    rv.term = me->current_term;
    rv.last_log_idx = raft_get_current_idx(me_);
    if (me->cb.send_requestvote)
        me->cb.send_requestvote(me_, me->udata, node, &rv);
    return 0;
}

rv isn't zero'd and rv.last_log_term is never explicitly set.

Thanks in advance!

Cheers,

Pete

msg_appendentries_response_t.first_idx field is not used in any algorithm's actions, so could be safely deleted from sources

Hi,

When called raft_recv_requestvote, the msg_requestvote_response_t that I get back has a garbled term. For example I had a long running app on term 180. I started up the test app and it started at term 1. The test app received a RequestVote message from the long running app. However then the test app replied with a No vote and with term 428825 (or similar - I don't have it in front of me right now).

I haven't yet been able to track down where this number is coming from...

Cheers,

Pete

PS I sent you an email a week or so go but didn't get a reply - not sure if it got stuck in a spam filter somewhere...

Hi,

I've got my implementation to the point where the cluster is now stable, in that a leader can be elected and maintain its authority over the cluster. So far so good :)

However now that I'm implementing raft_recv_appendentries_response I've hit something of a snag. There are two fields (highlighted in the header file as being non-raft) that I don't know how to provide:

/* Non-Raft fields follow: */
/* Having the following fields allows us to do less book keeping in
 * regards to full fledged RPC */
/* This is the highest log IDX we've received and appended to our log */
int current_idx;
/* The first idx that we received within the appendentries message */
int first_idx;

What should I be putting in current_idx and first_idx and where should I get it from?

call of raft_periodic with the only non-self node added causes the library to crash there:

    if (1 == raft_get_num_voting_nodes(me_) &&
        raft_node_is_voting(raft_get_my_node((void*)me)) &&
        !raft_is_leader(me_))
        raft_become_leader(me_);

because raft_get_my_node return null.
null pointer check should be added.

function raft_recv_appendentries_response contains 2 checks about null pointer node with different error codes but it's not clear witch one should be used:
`
if (!node)
return -1;

...
/* stop processing, this is a node we don't have in our configuration */
if (!node)
return 0;
`

May be the second check is redundant?

TxestRaft_full_snapshot -> TestRaft_full_snapshot

There are some new breaking changes in the current version of raft which is very different than the one used in ticketd. For example, things like log_get_node_id changed the way how users should interact with the library.

Can you update the documents or update the raft code in ticketd?

first, thanks for your excellent work. I wonder under what situation function raft_recv_entry will be called? this function seems not mentioned in raft paper?

ie.
RAFT_ERR_NOT_LEADER
RAFT_ERR_ONE_VOTING_CHANGE_ONLY