Recommended way to handle server downtime about elixir-thrift

[sarat1669]

After a bit of a reading, a cicuit breaker seems like a good solution

[fishcakez]

Hi @sarat1669, the Thrift connection processes are very simple and represent the lifetime of a connection over the network. Usually you do not want to allow a network failure to propagate failures through your supervision tree. The supervision tree should be used to handle errors with internal state and logic because those should be locally recoverable errors. Therefore you would need a slightly different pattern.

If you want to leverage OTP to its fullest you could start the connection as temporary worker from another process that monitors and restarts it when needed, perhaps it would do some backoff if start_link returns a (connect) error.

If you want to use a circuit breaker, you may wish to put that around the thrift service calls in the processes making requests as the connection error is not the only type of error you may wish to handle. For example timeouts, tapplicationexceptions or even specific service exceptions.

[sarat1669]

@fishcakez, The issue is when the connection is dropped, the process kills itself.
And that bubbles up the supervision tree.
I had to do some defensive programming to avoid this.

try do 
   Client.get_element(client, params)
catch 
   :exit, _ -> {:error, :internal_server_error}
end

And I had to trap exits:

def handle_info({:EXIT, _pid, {:error, :econnrefused}}, {breaker, _client}) do
  BreakerBox.increment_error(breaker)
  {:noreply, {breaker, nil}}
 end

Which is not recommended in beam ecosystem.

[fishcakez]

The first one:

try do 
   Client.get_element(client, params)
catch 
   :exit, _ -> {:error, :internal_server_error}
end

is a valid use of a catch and not too defensive because you are expecting to handle this error and preventing it bubbling up. Likely you want to differentiate between timeouts and other errors. You can do this by matching on the _ portion.

For the second example:

def handle_info({:EXIT, _pid, {:error, :econnrefused}}, {breaker, _client}) do
  BreakerBox.increment_error(breaker)
  {:noreply, {breaker, nil}}
 end

This seems like you are not using the directly supervisor anymore and starting the process underneath another one instead of using the temporary child under a supervisor I mentioned above.

[sarat1669]

@fishcakez, Makes sense. Yeah, I have wrapped it in a genserver.
And will add other parameters to the circuit breaking logic. Thanks for the info.