Can't reconfigure mqtts listener about docker-vernemq

[maelp]

@ioolkos I tried adding the variables you sent above, now I have those logs

17:26:37.432 [error] can't reconfigure mqtt listener({127,0,0,1}, 1883) with Options [{max_connections,10000},{nr_of_acceptors,10},{mountpoint,[]},{proxy_protocol,false},{allowed_protocol_versions,[3,4,5,131]},{allow_anonymous_override,false}] due to {already_started,<0.435.0>}
17:26:37.514 [error] can't reconfigure mqtts listener({127,0,0,1}, 1883) with Options [{max_connections,10000},{nr_of_acceptors,10},{mountpoint,[]},{cafile,"/etc/ssl/vernemq/ca.crt"},{depth,1},{certfile,"/etc/ssl/vernemq/tls.crt"},{eccs,[secp521r1,brainpoolP512r1,brainpoolP384r1,secp384r1,brainpoolP256r1,secp256k1,secp256r1,secp224k1,secp224r1,secp192k1,secp192r1,secp160k1,secp160r1,secp160r2]},{keyfile,"/etc/ssl/vernemq/tls.key"},{require_certificate,false},{tls_version,'tlsv1.2'},{use_identity_as_username,false},{allowed_protocol_versions,[3,4,131]},{allow_anonymous_override,false}] due to {already_started,<0.435.0>}

[maelp]

08:14:21.018 [error] can't reconfigure mqtts listener({127,0,0,1}, 1883) with Options [{max_connections,10000},{nr_of_acceptors,10},{mountpoint,[]},{cafile,"/etc/ssl/vernemq/..data/ca.crt"},{depth,1},{certfile,"/etc/ssl/vernemq/tls.crt"},{eccs,[secp521r1,brainpoolP512r1,brainpoolP384r1,secp384r1,brainpoolP256r1,secp256k1,secp256r1,secp224k1,secp224r1,secp192k1,secp192r1,secp160k1,secp160r1,secp160r2]},{keyfile,"/etc/ssl/vernemq/tls.key"},{require_certificate,true},{tls_version,'tlsv1.2'},{use_identity_as_username,true},{allowed_protocol_versions,[3,4,131]},{allow_anonymous_override,false}] due to {already_started,<0.460.0>}
08:16:23.322 [warning] session stopped abnormally due to '{cant_parse_connect_fixed_header,<<22,3,3,0,240,1,0,0,236,3,3,207,93,101,109,171,15,73,140,246,37,247,70,91,86,25,191,97,71,59,244,209,226,97,191,153,221,242,36,107,217,30,127,0,0,18,192,44,192,43,192,47,192,48,192,19,192,20,0,156,0,47,0,53,1,0,0,177,0,0,0,25,0,23,0,0,20,119,101,98,45,97,112,112,45,49,46,103,111,117,97,99,104,46,99,111,109,0,5,0,5,1,0,0,0,0,0,10,0,12,0,10,0,29,0,23,0,24,0,25,0,30,0,11,0,2,1,0,0,13,0,40,0,38,4,3,5,3,6,3,8,4,8,5,8,6,8,9,8,10,8,11,4,1,5,1,6,1,4,2,3,3,3,1,3,2,2,3,2,1,2,2,0,50,0,40,0,38,4,3,5,3,6,3,8,4,8,5,8,6,8,9,8,10,8,11,4,1,5,1,6,1,4,2,3,3,3,1,3,2,2,3,2,1,2,2,0,17,0,9,0,7,2,0,4,0,0,0,0,0,23,0,0,0,43,0,3,2,3,3,255,1,0,1,0>>}'

[maelp]

Okay, got it working, I'll document the minimal changes that are required

[maelp]

When connecting with hivemq mqtt-cli I get this log

Restriction request problem information was set but is unused in MQTT Version MQTT_3_1_1
Client '[email protected]' sending CONNECT MqttConnect{keepAlive=60, cleanSession=true, restrictions=MqttConnectRestrictions{receiveMaximum=65535, sendMaximum=65535, maximumPacketSize=268435460, sendMaximumPacketSize=268435460, topicAliasMaximum=0, sendTopicAliasMaximum=16, requestProblemInformation=true, requestResponseInformation=false}}
Client '[email protected]' DISCONNECTED null: com.hivemq.client.mqtt.exceptions.ConnectionFailedException: java.nio.channels.ClosedChannelException
Caused by: java.nio.channels.ClosedChannelException
	at io.netty.handler.ssl.SslHandler.channelInactive(SslHandler.java:1065)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)

[maelp]

I checked that the certificates are correct, and the 8883 port is correctly opened, so I guess the error is from the configuration?

Here is the config:

import pulumi
from pulumi_kubernetes.meta.v1 import ObjectMetaArgs
from pulumi_kubernetes.core.v1 import Namespace, Secret
from pulumi_kubernetes.helm.v3 import Chart, ChartOpts
from .env import k8s_provider


# Install the vernemq controller on the cluster
chart_ns = Namespace("vernemq", metadata=ObjectMetaArgs(name="vernemq"))
certificates_secret = Secret(
    "vernemq-certificates-secret",
    metadata=ObjectMetaArgs(
        name="vernemq-certificates-secret", namespace=chart_ns.metadata.name
    ),
    string_data={
        "tls.crt": open(
            "secrets-data/staging/certs/gsc/gsc.crt", "r", encoding="utf-8"
        ).read(),
        "tls.key": open(
            "secrets-data/staging/certs/gsc/private.pem", "r", encoding="utf-8"
        ).read(),
        "ca.crt": open(
            "secrets-data/staging/certs/rk/root.crt", "r", encoding="utf-8"
        ).read(),
    },
)
chart = Chart(
    "vernemq",
    config=ChartOpts(
        fetch_opts={"repo": "https://vernemq.github.io/docker-vernemq"},
        chart="vernemq",
        version="1.8.1",
        namespace=chart_ns.metadata.name,
        values={
            "additionalEnv": [
                {"name": "DOCKER_VERNEMQ_ACCEPT_EULA", "value": "yes"},
                {"name": "DOCKER_VERNEMQ_ALLOW_ANONYMOUS", "value": "on"},
                {
                    "name": "DOCKER_VERNEMQ_LISTENER__SSL__CAFILE",
                    "value": "/etc/ssl/vernemq/ca.crt",
                },
                {
                    "name": "DOCKER_VERNEMQ_LISTENER__SSL__CERTFILE",
                    "value": "/etc/ssl/vernemq/tls.crt",
                },
                {
                    "name": "DOCKER_VERNEMQ_LISTENER__SSL__KEYFILE",
                    "value": "/etc/ssl/vernemq/tls.key",
                },
            ],
            "persistentVolume": {
                "enabled": "true",
                "size": "5Gi",
                # "storageClass": "..."
            },
            "service": {
                "mqtt": {"enabled": "true", "port": "1883"},
                "mqtts": {"enabled": "true", "port": "8883"},
            },
            "secretMounts": [
                {
                    "name": "vernemq-certificates",
                    "secretName": "vernemq-certificates-secret",
                    "path": "/etc/ssl/vernemq",
                }
            ],
        },
    ),
    opts=pulumi.ResourceOptions(
        provider=k8s_provider, depends_on=[chart_ns, certificates_secret]
    ),
)

[ioolkos]

@maelp there's currently a minor issue coming from configuring default listeners in the release package. Will be fixed in next release.
Disable as much default listeners configs as you can. Then set the MQTTS listener like the one below.

listener.ssl.default=0.0.0.0:8883
listener.ssl.default.keyfile=/etc/ssl/vernemq/tls.key
listener.ssl.default.cafile=/etc/ssl/vernemq/ca.crt
listener.ssl.default.certfile=/etc/ssl/vernemq/tls.crt

if the reconfigure error bothers you, try setting an MQTT default listener too

listener.tcp.default = 127.0.0.1:1883

To arrive at those, you have to translate to the ENV variable format here.

I'll look at that client side error later, maybe related to allowed protocol version?

---

👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq
👉 Using the binary VerneMQ packages commercially (.deb/.rpm/Docker) requires a paid subscription.

[maelp]

(the error seems to be that the server closes the socket when trying to do a SSL connection)

[maelp]

I'm using a client which handles MQTT 3,4,5 I think (hive mqtt-cli), I'm not sure what version the VerneMQ server is handling, is that configurable?

[codeadict]

I have seen this error @ioolkos , might be fixed on my currently open PR in VerneMQ but we have to double check

[maelp]

Here's what I added

                {
                    "name": "DOCKER_VERNEMQ_LISTENER__SSL__DEFAULT",
                    "value": "0.0.0.0:8883",
                },
                {
                    "name": "DOCKER_VERNEMQ_LISTENER__TCP__DEFAULT",
                    "value": "127.0.0.1:1883",
                },

[maelp]

But for now it seems I can't connect to VerneMQ with SSL, but perhaps I can do it using a Nginx in front, but this defeats the purpose of using VerneMQ TLS setting which can be convenient (to auto-set the user with the CommonName)

[codeadict]

I'll do some testing on the branch I've opened with these settings. What does vmq-admin listener show is telling you @maelp ?

[ioolkos]

@maelp MQTT protocol versions can be allowed per listener.

listener.ssl.default.allowed_protocol_versions = 3,4,5

Terminating TLS in HAProxy or similar component is often done, yes. Of course, Verne can also terminate TLS without problems; so we'll find out what's happing here.

---

👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq
👉 Using the binary VerneMQ packages commercially (.deb/.rpm/Docker) requires a paid subscription.

[maelp]

@codeadict this is the result

+--------+---------+-----------+-------+------------+-----------+
| type   | status  | ip        | port  | mountpoint | max_conns |
+--------+---------+-----------+-------+------------+-----------+
| http   | running | 127.0.0.1 | 8888  |            | 10000     |
+--------+---------+-----------+-------+------------+-----------+
| http   | running | 10.4.0.8  | 8888  |            | 10000     |
+--------+---------+-----------+-------+------------+-----------+
| vmq    | running | 10.4.0.8  | 44053 |            | 10000     |
+--------+---------+-----------+-------+------------+-----------+
| mqttws | running | 10.4.0.8  | 8080  |            | 10000     |
+--------+---------+-----------+-------+------------+-----------+
| mqtt   | running | 127.0.0.1 | 1883  |            | 10000     |
+--------+---------+-----------+-------+------------+-----------+
| mqtt   | running | 10.4.0.8  | 1883  |            | 10000     |
+--------+---------+-----------+-------+------------+-----------+

[maelp]

(weird that there's no mqtts, is that because of the error above?)

[maelp]

also not sure why there is mqttws although I haven't enabled it?

[maelp]

If this helps, when I set the log level to debug, I have a lot of these:

08:47:33.345 [debug] Replica meta1: Can't initialize AE exchange due to no peer available
08:47:33.429 [debug] Replica meta9: Can't initialize AE exchange due to no peer available
08:47:33.618 [debug] Replica meta4: Can't initialize AE exchange due to no peer available
08:47:34.507 [debug] Replica meta3: Can't initialize AE exchange due to no peer available
08:47:34.689 [debug] Replica meta8: Can't initialize AE exchange due to no peer available
08:47:34.755 [debug] Replica meta5: Can't initialize AE exchange due to no peer available
08:47:35.127 [debug] Replica meta6: Can't initialize AE exchange due to no peer available
08:47:35.155 [debug] Replica meta10: Can't initialize AE exchange due to no peer available
08:47:35.267 [debug] Replica meta7: Can't initialize AE exchange due to no peer available

[maelp]

@ioolkos @codeadict when I disable "mqtt" and only enable "mqtts" in the config, I still have the "can't reconfigure mqtts" error

08:54:06.132 [error] can't reconfigure mqtts listener({127,0,0,1}, 1883) with Options [{max_connections,10000},{nr_of_acceptors,10},{mountpoint,[]},{cafile,"/etc/ssl/vernemq/ca.crt"},{depth,1},{certfile,"/etc/ssl/vernemq/tls.crt"},{eccs,[secp521r1,brainpoolP512r1,brainpoolP384r1,secp384r1,brainpoolP256r1,secp256k1,secp256r1,secp224k1,secp224r1,secp192k1,secp192r1,secp160k1,secp160r1,secp160r2]},{keyfile,"/etc/ssl/vernemq/tls.key"},{require_certificate,false},{tls_version,'tlsv1.2'},{use_identity_as_username,false},{allowed_protocol_versions,[3,4,131]},{allow_anonymous_override,false}] due to {already_started,<0.458.0>}

but then each time I try to connect with SSL I now have this (before I didn't even have a log) and the connection still closes:

09:04:43.371 [warning] session stopped abnormally due to '{cant_parse_connect_fixed_header,<<22,3,3,0,240,1,0,0,236,3,3,114,105,135,159,176,108,221,9,230,83,91,123,59,137,214,40,86,81,117,114,74,34,165,44,40,75,196,182,67,195,180,244,0,0,18,192,44,192,43,192,47,192,48,192,19,192,20,0,156,0,47,0,53,1,0,0,177,0,0,0,25,0,23,0,0,20,119,101,98,45,97,112,112,45,49,46,103,111,117,97,99,104,46,99,111,109,0,5,0,5,1,0,0,0,0,0,10,0,12,0,10,0,29,0,23,0,24,0,25,0,30,0,11,0,2,1,0,0,13,0,40,0,38,4,3,5,3,6,3,8,4,8,5,8,6,8,9,8,10,8,11,4,1,5,1,6,1,4,2,3,3,3,1,3,2,2,3,2,1,2,2,0,50,0,40,0,38,4,3,5,3,6,3,8,4,8,5,8,6,8,9,8,10,8,11,4,1,5,1,6,1,4,2,3,3,3,1,3,2,2,3,2,1,2,2,0,17,0,9,0,7,2,0,4,0,0,0,0,0,23,0,0,0,43,0,3,2,3,3,255,1,0,1,0>>}'

[ioolkos]

@maelp the AE exchange logs above are unrelated and harmless (coming from the sync protocol).

There are other users having troubles with MQTTS listeners in the Docker image, apparently: https://erlangforums.com/t/latest-vernemq-not-allowing-tls-traffic-with-self-signed-ca-certificate/2394
I'll need to check that.

Have you enabled listener.ssl.default.allowed_protocol_versions = 3,4,5?

[maelp]

@ioolkos I added this, but same error

                {
                    "name": "DOCKER_VERNEMQ_LISTENER__SSL__ALLOWED_PROTOCOL_VERSIONS",
                    "value": "3,4,5",
                },

[ioolkos]

Btw, the WS listener comes from the fact that we seem to start one via start script, in case the DOCKER_VERNEMQ variable is not set.

docker-vernemq/bin/vernemq.sh

Line 214 in 148c27a

if [ -z "$DOCKER_VERNEMQ_LISTENER__WS__DEFAULT" ]; then

[ioolkos]

Can you check on the Pod, what the exact config injected into etc/vernemq/vernemq.conf is and post here?
All values injected are at the end of the conf file between ###START### and ###END###.

---

👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq
👉 Using the binary VerneMQ packages commercially (.deb/.rpm/Docker) requires a paid subscription.

[maelp]

@ioolkos this is what I get

########## Start ##########
listener.ssl.use_identity_as_username=on
listener.ssl.allowed_protocol_versions=3,4,5
accept_eula=yes
listener.tcp.default=127.0.0.1:1883
listener.ssl.default=0.0.0.0:8883
listener.ssl.require_certificate=on
listener.ssl.keyfile=/etc/ssl/vernemq/tls.key
listener.ssl.cafile=/etc/ssl/vernemq/ca.crt
log.console=console
listener.tcp.localhost=127.0.0.1:1883
allow_anonymous=on
listener.ssl.certfile=/etc/ssl/vernemq/tls.crt
erlang.distribution.port_range.minimum = 9100
erlang.distribution.port_range.maximum = 9109
listener.ws.default = 10.4.0.13:8080
listener.vmq.clustering = 10.4.0.13:44053
listener.http.metrics = 10.4.0.13:8888
########## End ##########

[ioolkos]

Can you also add:
listener.ssl.default.depth = 10 to ensure we're not limited by chain length.

You could also attach to VerneMQ in the Pod by using vernemq attach.
And then get info with something similar to:

rp(ranch:info({{127,0,0,1},8883})). (hit Enter)

Detach from Verne with ctrl+g, then q.

I'm currently unclear whether you manage to setup an SSL listener or not. (check with vmq-admin listener show)

---

👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq
👉 Using the binary VerneMQ packages commercially (.deb/.rpm/Docker) requires a paid subscription.

[maelp]

Adding the default depth of 10 did not change anything

for the command: if I include the final . (not sure if that was intended), I get the following error:

if I don't include it, I get an empty response

([email protected])1> rp(ranch:info({{127,0,0,1},8883})).
** exception error: bad argument
     in function  ets:lookup_element/3
        called as ets:lookup_element(ranch_server,{listener_sup,{{127,0,0,1},8883}},2)
        *** argument 2: not a key that exists in the table
     in call from ranch_server:get_listener_sup/1 (/opt/vernemq/_build/default/lib/ranch/src/ranch_server.erl, line 122)
     in call from ranch:info/1 (/opt/vernemq/_build/default/lib/ranch/src/ranch.erl, line 434)
([email protected])2> rp(ranch:info({{127,0,0,1},8883}))
([email protected])2>

[maelp]

If I do it with "1883" instead of "8883" I get this

([email protected])2> rp(ranch:info({{127,0,0,1},1883})).
#{active_connections => 0,all_connections => 0,
  ip => {127,0,0,1},
  max_connections => 10000,
  metrics =>
      #{{conns_sup,1,accept} => 0,
        {conns_sup,1,terminate} => 0,
        {conns_sup,2,accept} => 0,
        {conns_sup,2,terminate} => 0,
        {conns_sup,3,accept} => 0,
        {conns_sup,3,terminate} => 0,
        {conns_sup,4,accept} => 0,
        {conns_sup,4,terminate} => 0,
        {conns_sup,5,accept} => 0,
        {conns_sup,5,terminate} => 0,
        {conns_sup,6,accept} => 0,
        {conns_sup,6,terminate} => 0,
        {conns_sup,7,accept} => 0,
        {conns_sup,7,terminate} => 0,
        {conns_sup,8,accept} => 0,
        {conns_sup,8,terminate} => 0,
        {conns_sup,9,accept} => 0,
        {conns_sup,9,terminate} => 0,
        {conns_sup,10,accept} => 0,
        {conns_sup,10,terminate} => 0},
  pid => <0.458.0>,port => 1883,protocol => vmq_ranch,
  protocol_options =>
      [{mountpoint,[]},
       {allowed_protocol_versions,[3,4,5,131]},
       {allow_anonymous_override,false},
       {buffer_sizes,undefined}],
  status => running,transport => ranch_tcp,
  transport_options =>
      #{max_connections => 10000,num_acceptors => 10,
        socket_opts =>
            [{ip,{127,0,0,1}},
             {port,1883},
             {nodelay,true},
             {linger,{true,0}},
             {send_timeout,30000},
             {send_timeout_close,true}]}}
ok

so I guess for some reason the mqtts handler is not launched, although I think I configured everything? is that related to the error above where it complains that it cannot reconfigure the listener (not sure why it tries to reconfigure the mqtts listener on 1883 instead of 8883?)

[maelp]

It seems though that the 8883 port is correctly opened (and correctly proxied-through by the ingress-nginx TCP proxy in front), because when I try to make a MQTTS request on 8883 I get this log to appear on the service logs

14:29:10.285 [warning] session stopped abnormally due to '{cant_parse_connect_fixed_header,<<22,3,3,0,240,1,0,0,236,3,3,95,237,223,125,180,210,123,10,168,11,41,16,254,209,7,51,234,66,84,195,188,130,8,2,82,22,116,40,207,75,203,201,0,0,18,192,44,192,43,192,47,192,48,192,19,192,20,0,156,0,47,0,53,1,0,0,177,0,0,0,25,0,23,0,0,20,119,101,98,45,97,112,112,45,49,46,103,111,117,97,99,104,46,99,111,109,0,5,0,5,1,0,0,0,0,0,10,0,12,0,10,0,29,0,23,0,24,0,25,0,30,0,11,0,2,1,0,0,13,0,40,0,38,4,3,5,3,6,3,8,4,8,5,8,6,8,9,8,10,8,11,4,1,5,1,6,1,4,2,3,3,3,1,3,2,2,3,2,1,2,2,0,50,0,40,0,38,4,3,5,3,6,3,8,4,8,5,8,6,8,9,8,10,8,11,4,1,5,1,6,1,4,2,3,3,3,1,3,2,2,3,2,1,2,2,0,17,0,9,0,7,2,0,4,0,0,0,0,0,23,0,0,0,43,0,3,2,3,3,255,1,0,1,0>>}'

[maelp]

and if I try to send a non-SSL MQTT message on port 8883 (to try), I don't see a log on the server, and the socket is closed immediately

[ioolkos]

Yeah, it's weird. Can you add the TLS listener not with 0.0.0.0 but that actual internal IP like the other listeners?
You can add as many TLS listeners on different ports as you want... that 1883 reconfigure thing should not matter here.

---

👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq
👉 Using the binary VerneMQ packages commercially (.deb/.rpm/Docker) requires a paid subscription.

[maelp]

I'm not sure how to do this, since I guess with Kubernetes each time I deploy I might be on a different node IP no?

[maelp]

for the "1883 reconfigure", the thing which surprises me is that it says both "mqtts" and "1883". Shouldn't it be "8883" that it tries to reconfigure for mqtts?

[codeadict]

The bug with the default listener should be fixed in my branch vernemq/vernemq#2078, if you can build Docker pointing to that it would be great. Although I believe there is still some bug that might have been caused by our latest ranch upgrade, I'm still researching the changes in ranch. But this is what I see:

^C%
codeadict@andromeda vernemq % _build/default/rel/vernemq/bin/vmq-admin listener stop address="127.0.0.1" port=8883
Done
codeadict@andromeda vernemq % _build/default/rel/vernemq/bin/vmq-admin listener show
+-------+---------+-----------+-------+------------+-----------+
| type  | status  | address   | port  | mountpoint | max_conns |
+-------+---------+-----------+-------+------------+-----------+
| http  | running | 127.0.0.1 | 8888  |            | 10000     |
+-------+---------+-----------+-------+------------+-----------+
| vmq   | running | 0.0.0.0   | 44053 |            | 10000     |
+-------+---------+-----------+-------+------------+-----------+
| mqtts | stopped | 127.0.0.1 | 8883  |            | 10000     |
+-------+---------+-----------+-------+------------+-----------+
| mqtt  | running | 127.0.0.1 | 1883  |            | 10000     |
+-------+---------+-----------+-------+------------+-----------+
codeadict@andromeda vernemq % _build/default/rel/vernemq/bin/vmq-admin listener start address="127.0.0.1" port=8883
can't start listener due to '{already_started,<0.20463.0>}'
codeadict@andromeda vernemq % _build/default/rel/vernemq/bin/vmq-admin listener show
+-------+---------+-----------+-------+------------+-----------+
| type  | status  | address   | port  | mountpoint | max_conns |
+-------+---------+-----------+-------+------------+-----------+
| http  | running | 127.0.0.1 | 8888  |            | 10000     |
+-------+---------+-----------+-------+------------+-----------+
| vmq   | running | 0.0.0.0   | 44053 |            | 10000     |
+-------+---------+-----------+-------+------------+-----------+
| mqtts | stopped | 127.0.0.1 | 8883  |            | 10000     |
+-------+---------+-----------+-------+------------+-----------+
| mqtt  | running | 127.0.0.1 | 1883  |            | 10000     |
+-------+---------+-----------+-------+------------+-----------+

Notice when I try to start the listener again, it believes it's already started and remains stopped.

[maelp]

I don't have a build script for the repo now, do you think you will publish an update soon of the package if this fixes the bug?

[ioolkos]

@codeadict start and restart in vmq-admin listener have different purposes. Does restart work for you?

What I do not understand in the open issue, is why we do not see a MQTTS listener running at all on 8883. (using vmq-admin listener show)
@maelp to me it looks like the service port is running but not the internal targetport (which is the MQTTS listener).

Maybe you can configure a different port for your MQTTS listener and targetport to decouple things from that 1883/8883 question.

---

👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq
👉 Using the binary VerneMQ packages commercially (.deb/.rpm/Docker) requires a paid subscription.

[maelp]

So you mean I should configure VerneMQ to listen to eg 8884? And have my ingress-nginx do the 8883 -> 8884?

[maelp]

@ioolkos could the issue come from the Helm template?

[ioolkos]

Yeah, that's what I meant.
Helm templates, yes, could also be. Or that secretCert mounts thing. Or maybe special Pulumi magic?
We'll find out :)
Apologies for the friction. Some days Kubernetes just hates us...

---

👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq
👉 Using the binary VerneMQ packages commercially (.deb/.rpm/Docker) requires a paid subscription.

[maelp]

🤣 yes kubernetes is sometimes a bit complicated

I'll try changing the port.

The secretCert mount seems to be working, because before I did it a different way (like mentioned in your documentation, I was using the certificates.{cert,key,ca} options in values.yamlwhich are perhaps deprecated?) and it wasn't working, there was an error log, which has now disappeared when using the secretMount way of adding certificates

[maelp]

I doubt it's due to Pulumi, the only thing Pulumi does is translating some DSL to actual yaml files for Kubernetes and applying them

[maelp]

Although this file seems to be handling the certificates config, so not sure why it wasn't working? https://github.com/vernemq/docker-vernemq/blob/master/helm/vernemq/templates/certificate-secret.yaml

[maelp]

changing the port to 8884 doesn't change the issue

[maelp]

For the "certificates" variables issue, could it be because in the certicicates-secret template there is a "$root" which is not in the statefulset file?

certificate-secret:

statefulset:

[ioolkos]

There seems to be an issue in the Docker images for 1.12.6.1 and 1.12.6.2. The issue is not present in the images for 1.12.3 (the previous official release).

The issue does not seem to be in the Linux packages.

In the 1.12.3 Pod, you can even test a connection from within a VerneMQ shell to its own TLS port:

Eshell V11.2.1  (abort with ^G)
([e-mail])1> {ok, S} = ssl:connect("localhost", 8884, [{cacerts, "/etc/ssl/vernemq/ca.crt"}, {verify, verify_none}]).
{ok,{sslsocket,{gen_tcp,#Port<0.24>,tls_connection,
                        undefined},
               [<0.626.0>,<0.625.0>]}}

This will get {error, close} on 1.12.6.2.

Nothing logged in the Verne log, this is a bummer. Investigating. This seems not related to configuration issues. (maybe OpenSSL version?)

---

👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq
👉 Using the binary VerneMQ packages commercially (.deb/.rpm/Docker) requires a paid subscription.

[ioolkos]

I think I found the issue. I traced the SSL statemachine and when a connection comes in it gets an enoent error for the ca.crt file.
The reason for this is that it links to ..data/ca.crt from /etc/ssl/vernemq/. It only seems to happen for the ca.crt file.

vernemq@maverne-vernemq-0:~$ ls -al /etc/ssl/vernemq/
total 4
drwxrwsrwt 3 root vernemq  140 Feb 26 20:05 .
drwxr-xr-x 1 root root    4096 Feb 26 20:05 ..
drwxr-sr-x 2 root vernemq  100 Feb 26 20:05 ..2023_02_26_20_05_39.2948119764
lrwxrwxrwx 1 root vernemq   32 Feb 26 20:05 ..data -> ..2023_02_26_20_05_39.2948119764
lrwxrwxrwx 1 root vernemq   13 Feb 26 20:05 ca.crt -> ..data/ca.crt
lrwxrwxrwx 1 root vernemq   14 Feb 26 20:05 tls.crt -> ..data/tls.crt
lrwxrwxrwx 1 root vernemq   14 Feb 26 20:05 tls.key -> ..data/tls.key

Now, I can manually do a mkdir ..data and copy ca.crt there, and voilà, the TLS connection is accepted.
Of course, this might be specific to how I set up the certificates (using the secret mount method), so your picture might vary.

But exec into the pod and do a ls -al /etc/ssl/vernemq/
Hopefully we can find a configuration based fix for this.

---

👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq
👉 Using the binary VerneMQ packages commercially (.deb/.rpm/Docker) requires a paid subscription.

[maelp]

I get this indeed:

bash-5.1$ ls -al /etc/ssl/vernemq/
total 4
drwxrwsrwt    3 root     vernemq        140 Feb 24 18:02 .
drwxr-xr-x    1 root     root          4096 Feb 24 18:02 ..
drwxr-sr-x    2 root     vernemq        100 Feb 24 18:02 ..2023_02_24_18_02_56.1832396605
lrwxrwxrwx    1 root     vernemq         32 Feb 24 18:02 ..data -> ..2023_02_24_18_02_56.1832396605
lrwxrwxrwx    1 root     vernemq         13 Feb 24 18:02 ca.crt -> ..data/ca.crt
lrwxrwxrwx    1 root     vernemq         14 Feb 24 18:02 tls.crt -> ..data/tls.crt
lrwxrwxrwx    1 root     vernemq         14 Feb 24 18:02 tls.key -> ..data/tls.key

[ioolkos]

1.12.3 has the same links, though.

---

👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq
👉 Using the binary VerneMQ packages commercially (.deb/.rpm/Docker) requires a paid subscription.

[ioolkos]

I found an easy fix for this. Use the full linked path for the CAFILE in your values.yaml/ENV variables.

  - name: DOCKER_VERNEMQ_LISTENER__SSL__CAFILE
    value: "/etc/ssl/vernemq/..data/ca.crt"
  - name: DOCKER_VERNEMQ_LISTENER__SSL__CERTFILE
    value: "/etc/ssl/vernemq/tls.crt"
  - name: DOCKER_VERNEMQ_LISTENER__SSL__KEYFILE
    value: "/etc/ssl/vernemq/tls.key"

---

👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq
👉 Using the binary VerneMQ packages commercially (.deb/.rpm/Docker) requires a paid subscription.

[maelp]

It might also be interesting to add info / error logs in the pod if the certificates couldn't be loaded correctly, because for now there was no such logs

[maelp]

Setting the ENV variables explicitly does not seem to solve the issue on my Helm deployment

[maelp]

it seems to be due to the "cant_parse_connect_fixed_header", is there a way to debug this? Perhaps check whether the bytes do indeed correspond to some valid message?

[ioolkos]

Please add listener.ssl.allowed_protocol_versions = 3,4,5 to your config. You can also use a named version:
listener.ssl.default.allowed_protocol_versions = 3,4,5

Also, note that the can't reconfigure mqtts listener is NOT the central issue at hand here. The issue is that we have an MQTT listener running that runs into trouble loading the CA chain when an actual TLS connection comes in. The reconfigure error log will be fixed separately.

The CA issue is unfortunately not captured by the Verne boot validators. We have 3 simple validators on those cert files: file-exists, file-is-readable, and file-is-pem. They pass & the issue only shows in the Erlang SSL statemachine.

---

👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq
👉 Using the binary VerneMQ packages commercially (.deb/.rpm/Docker) requires a paid subscription.

[ioolkos]

I guess the reason is that for a self-signed situation the client also needs your specific CAfile with the cert chain.
You probably also need to look into the question of hostname verification.

---

👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq
👉 Using the binary VerneMQ packages commercially (.deb/.rpm/Docker) requires a paid subscription.

[maelp]

Seems to work with those options

                {"name": "DOCKER_VERNEMQ_ACCEPT_EULA", "value": "yes"},
                {"name": "DOCKER_VERNEMQ_ALLOW_ANONYMOUS", "value": "on"},
                {
                    "name": "DOCKER_VERNEMQ_LISTENER__SSL__CAFILE",
                    "value": "/etc/ssl/vernemq/..data/ca.crt",
                },
                {
                    "name": "DOCKER_VERNEMQ_LISTENER__SSL__CERTFILE",
                    "value": "/etc/ssl/vernemq/tls.crt",
                },
                {
                    "name": "DOCKER_VERNEMQ_LISTENER__SSL__KEYFILE",
                    "value": "/etc/ssl/vernemq/tls.key",
                },

for some reason, it doesn't work if I don't set explictly DOCKER_VERNEMQ_LISTENER__SSL__CERTFILE and DOCKER_VERNEMQ_LISTENER__SSL__KEYFILE although their values are the same as the defaults?

[maelp]

Ugh, seems like an Heisenbug, I tried deleting the VerneMQ service and recreating it with the same options, and suddenly it no longer works

[ioolkos]

Verne only picks up the settings injected into etc/vernemq.conf in your pod. If in doubt, always look at the last lines in the file.
Yes, I think the SSL_KEYFILE and SSL_CERTFILE need to be set if you want to have an SSL listener.

Be aware that the additional method to configure certs that you see in the main branch was only added 3 weeks ago:

docker-vernemq/helm/vernemq/templates/statefulset.yaml

Line 88 in 148c27a

{{- if .Values.certificates }}

---

👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq
👉 Using the binary VerneMQ packages commercially (.deb/.rpm/Docker) requires a paid subscription.

[ManuelMueller1st]

@maelp We are facing the same issue. Could you let me know if you managed to fix it?

[ioolkos]

@ManuelMueller1st the underlying cause in the packages was fixed but it seems with Helm it can still be an issue.

Please determine what listeners you want to have running.
Then determine which listeners are actually running after a start (vmq-admin listener show).

Then we can determine which listeners to start manually (by adding to ENV), and which listeners are started implicitly by the service definitions. Possibly we can also just solve it by the naming of the listeners.

---

👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq
👉 Using the binary VerneMQ packages commercially (.deb/.rpm/Docker) requires a paid subscription.