[error] failed_to_check_schema: emqx_conf_schema about emqx

[thalesmg]

Instead of using env vars, maybe you could use file-based secrets for the bridges. This was added to 5.4.0.

• #11896 Introduced an enhancement for configuring sensitive authentication fields in bridges, such as passwords, tokens, and secret keys. This improvement allows the use of secrets stored as files in the file system. These secrets can be securely referenced in configuration files using the special file:// prefix, enhancing the security of sensitive data handling in bridge configurations.

Since the original problem is resolved, I'll close this.

[thalesmg]

Hi, are you using persistence in k8s (PVC)?

If you made changes to the bridge configuration using the dashboard UI, then those changes would have been saved to data/configs/cluster.hocon. In that case, you would need to either delete the bridge using the dashboard, or edit the persisted cluster.hocon files as well.

[N0tronic]

Hi, exactly k8s with pvc. I´ve tested like you subscribe. Delete the old bridge over the emqx dashboard. After that I´d delete the config lines from the old bridge in the values.yml and upgrade over helm.

But same failure:
[error] failed_to_check_schema: emqx_conf_schema [error] #{kind => validation_error,path => "bridges.mqtt.bridge_name.server",reason => required_field}

[thalesmg]

Are you upgrading from some older version to 5.4.1, or from 5.4.1 to 5.5.0?

How many nodes are in your cluster?

Could you please share the contents of your data/configs directory? i.e.: ls data/configs.
I'm checking if that directory contains a cluster-overrides.conf file by any chance.

Also, after deleting the bridge via the dashboard UI, could you please confirm that cluster.hocon does not contain any traces of said bridge?

[N0tronic]

There´re 3 nodes in the cluster. The problem is not only even, if I try to upgrade the version. Also if I did a helm upgrade only with deleted bridge values in the values.yml

There´re no more bridge options in the cluster.hocon file.

[N0tronic]

Should I sync the cluster config after delete the bridge over emqx dashboard? For example with the emqx_ctl tool?

It´s possible that this is the same problem? #12311

[thalesmg]

Should I sync the cluster config after delete the bridge over emqx dashboard? For example with the emqx_ctl tool?

Shouldn't be needed. Using the dashboard already takes care of that.

It´s possible that this is the same problem? #12311

Only if you are using environment variables like in that issue. If something like EMQX_BRIDGES__MQTT__... is also being set, then it's likely the cause.

Could you share your values.yaml file (with any sensitive data censored)?

[N0tronic]

Damn. Off course values from the values.yml file are set as environment variables.

[thalesmg]

@N0tronic was the problem solved after removing the bridge-related environment variables from values.yaml?

[N0tronic]

@thalesmg

I've solved the problem, but it was a bit more complicated. First the failure was still there, after I deleted all bridge variables from the values.yml

Then I realized, that I outsourced the bridge user password into a secret, that loaded as environment variable during deployment. Deleting the secret solved the problem.

But I'm not very satisfied with this restricted behavior of the system. One forgotten bridge variable prevents the whole deployment 😩

[zmstone]

But I'm not very satisfied with this restricted behavior of the system. One forgotten bridge variable prevents the whole deployment 😩

Hi @N0tronic
We understand the frustration.
Though I consider this a necessary trade-off for strictly type-checked config.
Maybe we can improve error logging to help locating the issue faster, we are open to suggestions in this regard.

[N0tronic]

@zmstone I understand, it's a narrow line. Some systems gives the user more flexibility, cause they can decide to turn the strict mode on or off. It would be nice, if u can think about such possibilities 🙂

But more details in the logs is also great as a first step 👍🏼