Persisting web hooks across broker restarts about vmq_webhooks

[larshesel]

Hi @matthiasfeurer in the next release (already available in nightly) everything gets configured directly in the vernemq.conf file. Another nice thing is that vmq_webhooks will be distributed with VerneMQ itself, so there'll be no need to compile and enable the plugin manually anymore.

If you can, I'd check out the nightly and look at the vernemq.conf file shipped as hopefully everything should be more or less self-explanatory there (that was our aim, at least). We'd love to hear your feedback.

[larshesel]

Let me know if you need more detailed information on what you're trying to achieve with 0.15.3 - but that way will be deprecated with the next release so I don't think it would make sense to spend too much effort configuring everything that way.

[matthiasfeurer]

Thanks for the quick feedback @larshesel ! We'll check out the nightly build then and hope to find the information on how to configure the web hooks as well as persistently disable the acl and pwd plugins.
We have been trying to use the vernemq.conf file for configuring the web hooks, but vernemq refused to start after the change and no logs were written...
The lines we have added to the vernemq.conf are the following:

# register web hooks
vmq_webhooks.webhook1.hook = auth_on_register
vmq_webhooks.webhook1.endpoint = http://localhost:28/on_register
vmq_webhooks.webhook2.hook = auth_on_publish
vmq_webhooks.webhook2.endpoint = http://localhost:28/on_publish
vmq_webhooks.webhook3.hook = auth_on_subscribe
vmq_webhooks.webhook3.endpoint = http://localhost:28/on_subscribe

Will let you know if we succeeded.

[larshesel]

That looks pretty correct to me. I'd be happy to try and help if I can - if there's a bug or unclear docs it would be great to get it fixed before we release 1.0. Which package did you install and can you show me the complete vernemq.conf that's causing you trouble?

And before I was on the way out and completely forgot to mention that in the new format the way to enable/disable a plugin is plugins.plugin = on | off. So to disable the vmq_acl you should find the line with plugins.vmq_acl = on and change it to plugins.vmq_acl = off and the same of course goes for vmq_passwd.

Also remember to enable webhooks: plugins.vmq_webhooks = on.

Again, please let me know if you're stuck!

[der-schuessel]

Thanks for your help @larshesel ! I checked out the nightly build and followed your instructions and add the needed commands to the vernemq.conf. After container creation the end of the file looks like this:

## 
## Default: 70
## 
## Acceptable values:
##   - an integer
leveldb.maximum_memory.percent = 70

vmq_webhooks.webhook1.hook = auth_on_register
vmq_webhooks.webhook1.endpoint = http://localhost:28/on_register
vmq_webhooks.webhook2.hook = auth_on_publish
vmq_webhooks.webhook2.endpoint = http://localhost:28/on_publish
vmq_webhooks.webhook3.hook = auth_on_subscribe
vmq_webhooks.webhook3.endpoint = http://localhost:28/on_subscribe
plugins.vmq_acl = off 
plugins.vmq_passwd = off 
plugins.vmq_webhooks = on
erlang.distribution.port_range.minimum = 9100
erlang.distribution.port_range.maximum = 9109
listener.tcp.default = 10.60.3.141:1883
listener.ws.default = 10.60.3.141:8080
listener.vmq.clustering = 10.60.3.141:44053

Unfortunately this ends in this error when trying to check the logs:
tail: cannot open '/var/log/vernemq/console.log' for reading: No such file or directory tail: no files remaining

If I only add plugins.vmq_acl = off though it works. Is there something I am missing?

[larshesel]

@der-schuessel hi - this looks strange. Can you show/outline exactly the steps leading to this error? I only know of one place where we do a tail on the console log and that's in our docker images, but we don't provide those for nightly builds.

[larshesel]

I missed that you wrote that you created a container out of it. Could you try and start vernemq outside of a container to see if it can start properly? I suppose the last thing the container does is to try to tail the console log to make it easy to see what's going on, but if vernemq never starts then that file won't exist.

[der-schuessel]

I tried starting a node with web-hooks activated on a simple vm and this has seemed to have worked. We are using this as a base for the docker image. Of course we changed the vernemq version to the nightly build and added the changes to the vernemq.conf file.

[larshesel]

Regarding docker, have you looked at: https://github.com/erlio/docker-vernemq ?

In any case a list of steps from start to finish needed to reproduce the issue would be great to help understand what's going on.

[der-schuessel]

After some further testing it turned out, that somehow a unicode (200b: zero width space) got pasted into the file leading to a syntax error. Thus, as you said vernemq doesn't start and the container can't do a tail on the logs. Right now everything seems to work fine. Again, thanks a lot for your support!

[larshesel]

hah, good one! Great you found it!

Sometimes I long for simpler days when only ascii whitespace existed. Unicode has tons of advantages but all the new whitespace chars are IMHO not one of them ;)

Let us know if you have other issues!