No way to set `inet6` option for the TCP transport (and by extension, to SSL transport) about thousand_island

[mtrudel]

As I'm sure you have figured out Thousand Island just passes parameters down into :gen_tcp / :ssl without altering them at all; literally anything that's possible to do directly should transparently be possible within Thousand Island as well (at least, that's the intent).

Let me dig into this for a moment; it's possible that I've somehow gotten in the way of that being the case.

[hauleth]

Yes, it technically is possible, but the problem is that Keyword.merge/2 will fail if one of the lists is non-keyword list [:inet6, fd: 1] is not keyword list.

[mtrudel]

Yep, I see the issue now!

Should be an easy fix! Stay tuned.

[hauleth]

My quick and dirty solution was to do something like:

  @impl Transport
  def listen(port, user_options) do
    default_options = [
      backlog: 1024,
      nodelay: true,
      linger: {true, 30},
      send_timeout: 30_000,
      send_timeout_close: true,
      reuseaddr: true
    ]

    {inet, user_options} = Keyword.pop(user_options, :net, :inet)

    resolved_options =
      default_options |> Keyword.merge(user_options) |> Keyword.merge(@hardcoded_options)

    :gen_tcp.listen(port, [inet | resolved_options])
  end

But it is still limited to supporting only inet6/inet option, and it will not support other options that do not conform to "keyword list definition" (for example raw option).

[mtrudel]

Fix is up at #26 - is it possible for you to test this branch & verify it fixes your issue? I've tested locally with a bare :inet6 parameter and it works, but I didn't go the whole way of tying with an open FD.

(I'd love a PR describing how to do that, btw!)

[hauleth]

I have updated Plug example in systemd library with Bandit support - hauleth/erlang-systemd@81e98d3. It still has some problem with draining from what I see. I cannot find a way to make it work better though and the amount of dropped requests is small enough that it shouldn't be a problem.

[mtrudel]

That's awesome to hear! I'll merge the changes in #26 and tag a TI / Bandit release later today so you can ensure your dependency has that fix included.

Thousand Island's drain support is based entirely on OTP shutdown semantics within the Thousand Island process tree, and should generally be opaque (the parent TI supervisor process will wait up to 5s* for connections to close, at which point they'll be brutally killed). I'd be curious to see a repro case for drain failures; it's entirely possible there are bugs.

* This isn't yet configurable, but I'll be adding in a configuration point to Thousand Island for it as part of the Bandit 0.7 train later this year.

[hauleth]

The reproducing code is in the repo I have linked. Details can be found there hauleth/erlang-systemd#24 (comment). In short if you run the service via systemd make in the example/plug directory on systemd-enabled Linux (beware, that it installs example app globally in /opt and uses sudo, so it is best done in throwaway VM rather on host). And then when you run any HTTP benmcharking tool and issue systemctl stop plug.service then there should be no connection errors. Currently there are few of them. Not enough IMHO to worry, but ideally I would like to see 0 of them. The problem is that I do not know whether it is problem on systemd or Elixir/Erlang side.

[mtrudel]

Any connections which are still open after 5 seconds will be brutally killed, which is likely what's happening here. The issue comes from the fact that 'connection' in this sense is at the network layer, not the application layer.

Specifically, HTTP connections that make use of keep-alive will happily continue accepting subsequent requests all the way until they're brutally killed (likely mid-way through a request).

I'll make a note to look at improving the drain process' integration with the upper level protocol handler (Bandit, in this case).

[mtrudel]

The fix in #26 has been published in Thousand Island 0.5.13, which is now a dependency of of Bandit 0.6.1. You may want to update the mix.lock dep in hauleth/erlang-systemd@81e98d3 to reflect that, as it's a bit out of date.

Thanks for the great suggestion!

[mtrudel]

Specifically, HTTP connections that make use of keep-alive will happily continue accepting subsequent requests all the way until they're brutally killed (likely mid-way through a request).

I'll make a note to look at improving the drain process' integration with the upper level protocol handler (Bandit, in this case).

This is not, in fact, the case. I've verified that (at least for HTTP/1.1):

• Connections which are in a keep-alive & waiting for a subsequent request to come in will be killed cleanly & immediately at server shutdown. They DO NOT stick around waiting.
• Connections which are mid-way through generating a response will complete the response (ie: they will complete the Bandit.HTTP1.Handler.handle_data/3 call that they're in) before being cleanly killed. That is, assuming that the response generation completes before the shutdown_timeout expires & they're brutally killed.

I'll also note that Thousand Island 0.6.3 (released a few days ago) supports configurability of the shutdown_timeout, and also ups its value to 15 seconds by default (up from 5 seconds previously).

[hauleth]

I will check that out today.