problems with tcpkali about thousand_island

[mtrudel]

My first guess is that tcpkali may not see a connection as being completed since you're keeping the connection open (you're using {:continue, state} instead of {:close, state}, which is correct by the Echo RFC but may be not what tcp kali expects). That won't be an issue on the Thousand Island side (at least not until you're up in the ten / hundreds of thousands of concurrent connections), but may be something to look at.

[ruslandoga]

tcpkali doesn't expect Echo RFC, but rather just an open connection. If connection is opened and then closed (which is what seems to be happening in this example, as pids returned by connection_pids seem increasing (<0.1234.0>, then <0.3456.0>, etc.) but their count stays roughly the same, ~5), then tcpkali doesn't start the benchmark.

---

Also I'm using very similar logic in https://github.com/ruslandoga/serv and tcpkali seems fine with it:

Mix.install [{:serv, github: "ruslandoga/serv"}]

defmodule Echo do
  use Serv.Handler

  def handle_data(data, socket, state) do
    with :ok <- :socket.send(socket, data) do
      {:continue, state}
    end
  end
end

Serv.start_link(port: 8000, handler: Echo)

$ tcpkali --message '$' --connections=100 --duration=15s 127.0.0.1:8000
Destination: [127.0.0.1]:8000
Interface lo0 address [127.0.0.1]:0
Using interface lo0 to connect to [127.0.0.1]:8000
Ramped up to 100 connections.
Total data sent:     7148.6 MiB (7495850022 bytes)
Total data received: 7147.1 MiB (7494276090 bytes)
Bandwidth per channel: 79.900⇅ Mbps (9987.5 kBps)
Aggregate bandwidth: 3994.595↓, 3995.434↑ Mbps
Packet rate estimate: 379531.7↓, 350009.2↑ (8↓, 10↑ TCP MSS/op)
Test duration: 15.0088 s.

[ruslandoga]

tcpkali + ThousandIsland over 3 connections also succeeds:

$ tcpkali --message '$' --connections=3 --duration=15s 127.0.0.1:8080
Destination: [127.0.0.1]:8080
Interface lo0 address [127.0.0.1]:0
Using interface lo0 to connect to [127.0.0.1]:8080
Ramped up to 3 connections.
Total data sent:     5388.0 MiB (5649694055 bytes)
Total data received: 5358.3 MiB (5618539542 bytes)
Bandwidth per channel: 2001.875⇅ Mbps (250234.4 kBps)
Aggregate bandwidth: 2994.511↓, 3011.115↑ Mbps
Packet rate estimate: 339287.1↓, 260048.5↑ (4↓, 15↑ TCP MSS/op)
Test duration: 15.0102 s.

But "in-progress" tcpkali output showed that some connections had been reopened ("seen" 24 connections over 15 seconds).

Here's this "in-progress" view

$ tcpkali --message '$' --connections=3 --duration=60s 127.0.0.1:8080
Destination: [127.0.0.1]:8080
Interface lo0 address [127.0.0.1]:0
Using interface lo0 to connect to [127.0.0.1]:8080
Ramped up to 3 connections.
Traffic 3183.256↓, 3183.162↑ Mbps (conns 0↓ 3↑ 0⇡; seen 31)

And here are the connections:

iex(10)> spawn(fn -> Enum.each(1..1000, fn _ ->
...(10)>   :timer.sleep(:timer.seconds(1))
...(10)>    IO.inspect(ThousandIsland.connection_pids(pid), label: to_string(Time.utc_now()))
...(10)> end) end)
14:30:14.846764: {:ok, []}
14:30:15.854497: {:ok, []}
14:30:16.857685: {:ok, []}
14:30:17.860868: {:ok, []}
14:30:18.863119: {:ok, [#PID<0.557.0>, #PID<0.556.0>, #PID<0.551.0>]}
14:30:19.865058: {:ok, [#PID<0.560.0>, #PID<0.568.0>, #PID<0.565.0>]}
14:30:20.866956: {:ok, [#PID<0.575.0>, #PID<0.570.0>, #PID<0.574.0>]}
14:30:21.870869: {:ok, [#PID<0.579.0>, #PID<0.570.0>, #PID<0.576.0>]}
14:30:22.875142: {:ok, [#PID<0.579.0>, #PID<0.570.0>, #PID<0.576.0>]}
14:30:23.878833: {:ok, [#PID<0.579.0>, #PID<0.570.0>, #PID<0.576.0>]}
14:30:24.882544: {:ok, [#PID<0.579.0>, #PID<0.570.0>, #PID<0.576.0>]}
14:30:25.885478: {:ok, [#PID<0.579.0>, #PID<0.570.0>, #PID<0.576.0>]}
14:30:26.888937: {:ok, [#PID<0.579.0>, #PID<0.570.0>, #PID<0.576.0>]}

Note that they became "stable" after a few seconds.

[ruslandoga]

This doesn't seem to be specific to ThousandIsland and affects all apps using gen_tcp + tcpkali on mac.

Moved to https://erlangforums.com/t/connection-closed-on-gen-tcp-but-not-socket-when-using-tcpkali