disk write and leader timeout about raft

[vgfree]

I suggest that the library need to provide asynchronous interfaces to write operations.
It is also best to provide batch write interfaces, because one by one is too slow.
such as:

typedef struct
{
/** number of entries within this batch */
int n_entries;

/** array of entries within this batch */
raft_entry_t* entries[0];

} raft_batch_t;

int log_append_entry(log_t* me_, raft_batch_t* bat)
{
log_private_t* me = (log_private_t*)me_;
int e;

e = __ensurecapacity(me, bat->n_entries);
if (e != 0)
    return e;

for (int i = 0; i < bat->n_entries; i ++) {
        int back = (me->back + i) % me->size;
        memcpy(&me->entries[back], bat->entries[i], sizeof(raft_entry_t));
}

void* ud = raft_get_udata(me->raft);
if (me->cb) {
        if (me->cb->log_offer_batch) {
                int idx = me->base + me->count + 1;
                e = me->cb->log_offer_batch(me->raft, ud, bat, idx);
                if (0 != e)
                        return e;
                for (int i = 0; i < bat->n_entries; i ++) {
                        idx = me->base + me->count + 1 + i;
                        raft_offer_log(me->raft, bat->entries[i], idx);
                }
        } else if (me->cb->log_offer) {
                for (int i = 0; i < bat->n_entries; i ++) {
                        int idx = me->base + me->count + 1 + i;
                        e = me->cb->log_offer(me->raft, ud, bat->entries[i], idx);
                        if (0 != e)
                                return e;
                        raft_offer_log(me->raft, bat->entries[i], idx);
                }
        }
}

me->count += bat->n_entries;
me->back += bat->n_entries;
me->back = me->back % me->size;

return 0;

}

[freeekanayaka]

I'm not entirely sure @skypexu and @vgfree intended it as well, but it'd be nice to also have the leader not wait for its own log to be written and synced to disk, effectively performing its own local AppendEntries in parallel with the network AppendEntries RPCs. The matchedIndex array should include the leader itself, and the leader should update it when it's done with its local disk write. This eliminates a disk write from raft's critical path. Note that an entry can be then considered committed even before the leader finishes writing it to its own local disk, if a majority of followers respond that they have appended it.

See "10.2.1 Writing to the leader’s disk in parallel" in Ongaro's Consenus: bridging theory and practice dissertation. His LogCabin implementation of raft uses this optimization.

[freeekanayaka]

@vgfree regarding batch interfaces feature request, I'd say it's essentially tracked by #16.

[skypexu]

I'm not entirely sure @skypexu and @vgfree intended it as well, but it'd be nice to also have the leader not wait for its own log to be written and synced to disk, effectively performing its own local AppendEntries in parallel with the network AppendEntries RPCs. The matchedIndex array should include the leader itself, and the leader should update it when it's done with its local disk write. This eliminates a disk write from raft's critical path. Note that an entry can be then considered committed even before the leader finishes writing it to its own local disk, if a majority of followers respond that they have appended it.

See "10.2.1 Writing to the leader’s disk in parallel" in Ongaro's Consenus: bridging theory and practice dissertation. His LogCabin implementation of raft uses this optimization.

Yes, I agree it should write to leader's disk in parallel.