Async append_entries handler sometimes not called about nuraft

[costasgambit]

I have tried your branch and it appears to have fixed the problem. Thanks!

[costasgambit]

This cheat solves the problem, which strongly suggests that the problem is in nuraft:

void PeshkaCtrl::asyncAppendEntries(
    const std::shared_ptr<nuraft::buffer> &logs,
    cutils::handler_t<void(cutils::ResponseCode &rc)> handler) const
{
    LOGTRACE(m_Logger, "asyncAppendEntries start");

    boost::asio::post(
        [this, logs, handler=std::move(handler)]()
        {
            auto res = m_RaftServer->append_entries({logs});

            if (res->get_accepted())
            {
                LOGTRACE(m_Logger, "asyncAppendEntries accepted");
                cutils::post(handler, cutils::ResponseCode());
            }
            else
            {
                std::string message = "ERROR???";
                LOGTRACE(m_Logger, "asyncAppendEntries error ", message);
                cutils::post(handler, cutils::ResponseCode(enums::Response::ERROR_PERMANENT, message));
            }
        }
    );
}

This runs the blocking version of append_entries, but in a background thread.

[greensky00]

Hi @costasgambit

Thanks for bringing up this issue. We will take a look at it in detail, but I'd like to ask a few questions to narrow down the problem:

• You don't use auto_forwarding_ option, right?

• "never receives a matching "done" or "error" message." does this mean

  1. The callback function passed through when_ready has never been invoked, OR
  2. The callback function has been invoked, but it is neither accepted nor contains an error message.
  • I'm assuming it is the first case.

• You mentioned the data is committed in the state machine, but only the callback is missing. Is it right? Please note that the callback will not be invoked until the corresponding log entry is committed in the state machine. If the state machine commit is lagging far behind the latest one, it may take a long time and there is no automatic notification regarding this long-running task.

[greensky00]

If the answer is 1) auto_forwarding_ is disabled, 2) the first case, 3) only the callback is missing, I believe the below one will be the fix to your issue.
#215

[costasgambit]

Thank you very much for your reply.

Your assumptions are correct: we do not use auto_forwarding_, the callback is never invoked, and the data is committed.

I will give your patch a try today and let you know the result. From reading it, it does look like it should be the solution.