bug(gateway): Heartbeats are not sent at `INTERVAL` about firezone

[thomaseizinger]

This isn't failing on heartbeats, it fails because we receive an empty message which means EOF.

You can see that in the two lines that are empty after from = "portal".

If that is expected, we can just skip completely empty messages but that is a bit odd i think?

[jamilbk]

I think what's happening is:

• The gateway's eventloop isn't sending heartbeats every 30s
• After 60s it tries to send one but the connection is closed by GCP loadbalancer, so we get EOF

[jamilbk]

If I set INTERVAL in phoenix_channel to 61s so that client's eventloop is also affected by this bug, we see the same behavior in the client:

connlib	14:25:37.252423-0700	FirezoneNetworkExtensionmacOS	dev.firezone.firezone	Flushed websocket
connlib	14:25:37.397634-0700	FirezoneNetworkExtensionmacOS	dev.firezone.firezone	  from="portal"
connlib	14:25:37.400763-0700	FirezoneNetworkExtensionmacOS	dev.firezone.firezone	Reconnecting to portal on transient client error: failed to deserialize message: EOF while parsing a value at line 1 column 0  backoff=660.454294ms max_elapsed_time=Some(86400s)
connlib	14:25:37.505007-0700	FirezoneNetworkExtensionmacOS	dev.firezone.firezone	bytes=64 dst=149.112.112.112 from="device" src=100.78.41.157
connlib	14:25:38.201637-0700	FirezoneNetworkExtensionmacOS	dev.firezone.firezone	Connected to portal  host=api.firez.one
connlib	14:25:38.201905-0700	FirezoneNetworkExtensionmacOS	dev.firezone.firezone	{"topic":"client","event":"phx_join","payload":null,"ref":2}  to="portal"
connlib	14:25:38.202153-0700	FirezoneNetworkExtensionmacOS	dev.firezone.firezone	{"topic":"phoenix","event":"heartbeat","payload":{},"ref":3}  to="portal"
connlib	14:25:38.202444-0700	FirezoneNetworkExtensionmacOS	dev.firezone.firezone	Flushed websocket
connlib	14:25:38.253050-0700	FirezoneNetworkExtensionmacOS	dev.firezone.firezone	bytes=64 dst=9.9.9.9 from="device" src=100.78.41.157
connlib	14:25:38.253427-0700	FirezoneNetworkExtensionmacOS	dev.firezone.firezone	Flushed websocket
connlib	14:25:38.278330-0700	FirezoneNetworkExtensionmacOS	dev.firezone.firezone	{"event":"phx_reply","ref":2,"topic":"client","payload":{"status":"ok","response":{}}}  from="portal"
connlib	14:25:38.278595-0700	FirezoneNetworkExtensionmacOS	dev.firezone.firezone	Received empty reply for request OutboundRequestId(2)
connlib	14:25:38.278764-0700	FirezoneNetworkExtensionmacOS	dev.firezone.firezone	{"event":"phx_reply","ref":3,"topic":"phoenix","payload":{"status":"ok","response":{}}}  from="portal"
connlib	14:25:38.279027-0700	FirezoneNetworkExtensionmacOS	dev.firezone.firezone	Flushed websocket
connlib	14:25:38.295420-0700	FirezoneNetworkExtensionmacOS	dev.firezone.firezone	{"event":"init","ref":null,"topic":"client","payload":{"interface":{"ipv6":"fd00:2021:1111::18:95aa","ipv4":"100.78.41.157","upstream_dns":[{"protocol":"ip_port","address":"9.9.9.9:53"}]},"resources":[{"id":"5377d0c2-0668-43c0-aebe-2f1daefabcb3","name":"155.138.222.187","type":"cidr","address":"155.138.222.187/32","gateway_groups":[{"id":"d1570e1e-dc01-4781-8e31-19274d755059","name":"Vultr Performance Tests | NO auto-upgrades","routing":"managed"}],"address_description":"http://155.138.222.187/"},{"id":"2c27e800-0277-49a7-987c-de6a8028d14d","name":"5.4.6.7","type":"cidr","address":"5.4.6.7/32","gateway_groups":[{"id":"7a868483-0e15-4255-a9b2-fe4c631c5ffa","name":"AWS Dev (Gateways track `main`)","routing":"managed"}],"address_description":"http://5.4.6.7/"},{"id":"f7ea6dba-564c-4c85-9fb9-01fb920a678e","name":"speed.cloudflare.com","type":"dns","address":"speed.cloudflare.com","gateway_groups":[{"id":"d1570e1e-dc01-4781-8e31-19274d755059","name":"Vultr Performance Tests | NO auto-upgrades","routing":"<…>

[jamilbk]

Actually, it's not GCP. It's our WebSocket in Elixir which times out after 60s of no data which is being triggered by this bug.

https://hexdocs.pm/phoenix/Phoenix.Endpoint.html#socket/3-websocket-configuration

cc @AndrewDryga

[thomaseizinger]

Fixed in #4486.