Frame too Large quirk about emqx

[SlothCroissant]

Update - Didn't even think about the fact that when running replicants, MQTT requests hit the replicants, not the core :)

Adding the same env var to the replicantsTemplate seems to have resolved my problem. Appreciate your support @zmstone!

---
apiVersion: apps.emqx.io/v2beta1
kind: EMQX
metadata:
  name: emqx
spec:
  image: emqx:5.5.0
  coreTemplate:
    spec:
      replicas: 3
      env:
      - name: EMQX_MQTT__MAX_PACKET_SIZE
        value: 256MB
  bootstrapAPIKeys:
    - key: some_api_key
      secret: some_api_secret
  replicantTemplate:
    spec:
      replicas: 3
      env:
      - name: EMQX_MQTT__MAX_PACKET_SIZE
        value: 256MB
  listenersServiceTemplate:
    spec:
      type: LoadBalancer
      loadBalancerIP: 10.1.50.13
  dashboardServiceTemplate:
    spec:
      type: LoadBalancer
      loadBalancerIP: 10.1.50.14

[zmstone]

Hi @SlothCroissant

To inspect the config value in effect, you can execute this command:

$ emqx eval 'emqx_config:get_zone_conf(default, [mqtt, max_packet_size])'
268435456

If it's indeed 256MB in your case, it's likely that the client has sent some bytes which were parsed into a frame length greater than this.

[SlothCroissant]

If it's indeed 256MB in your case, it's likely that the client has sent some bytes which were parsed into a frame length greater than this.

emqx@emqx-core-bfd48bd6c-0:/opt/emqx$ emqx eval 'emqx_config:get_zone_conf(default, [mqtt, max_packet_size])'
268435456

Thanks for your reply, @zmstone. Indeed it is set properly, it seems. Is there a trace log that will show the payload size? From what I can tell, the client is simply sending some JSON (which is not nearly 256MB in size). It's also strange that I can fix the issue with 100% success rate simply by pressing "save" in the EMQX UI.

[SlothCroissant]

Did some more digging to find out if the message was indeed going rogue and sending 256MB+

Packet capture taken on the client side is below.

Client: 10.42.2.183
EMQX: 10.43.247.45

The summary is that no single packet is larger than a couple dozen Kilobytes (Largest single packert is roughly 20KB). Entire TCP conversation is only 100-200KB total.

Thoughts on how I can debug further?

[zmstone]

The trace logs do not include the actual length received (this is something we can improve, we will work on a fix).

Could you share the pcap file so we can have a closer look?

Another question, are you running EMQX nodes in a cluster?
The changes made from UI applies to all nodes in the cluster, but environment variable is not shared.
You mentioned that you are setting EMQX_MQTT__MAX_PACKET_SIZE environment variable, but also changing settings from the UI.

[SlothCroissant]

I deploy EMQX in a clustered environment via the emqx-operator:

---
apiVersion: apps.emqx.io/v2beta1
kind: EMQX
metadata:
  name: emqx
spec:
  image: emqx:5.5.0
  coreTemplate:
    spec:
      replicas: 3
      env:
      - name: EMQX_MQTT__MAX_PACKET_SIZE
        value: 256MB

Does this deploy the env var to all nodes (I assume so, but sanity check)? Is there a better way to set this via the operator?

Pcap attached.

12527.pcap.zip

[zmstone]

Thanks @SlothCroissant
I have sent a logging enhancement here: #12530 to help troubleshooting.