astarte-platform / astarte_vmq_plugin

Astarte VerneMQ plugin publishes incoming messages on an AMQP exchange with some additional metadata, so they can be processed by other components.

Astarte VerneMQ Plugin

Astarte platform plugin for VerneMQ.

This plugin publishes incoming messages on an AMQP exchange with some additional metadata, so they can be processed by other Astarte components.

astarte_vmq_plugin's Issues

Queue astarte_vmq_plugin_rpc disappearing from RabbitMQ

We found an odd crash on Data Updater Plant. It crashed with a timeout after trying to push a force_disconnection message in the VerneMQ RPC queue.

Aug  7 12:48:27 |ERRO| GenServer {Registry.DataUpdater, {"test", <<110, 174, 96, 120, 180, 191, 67, 214, 138, 198, 238, 112, 135, 193, 218, 234>>}} terminating
** (stop) exited in: GenServer.call(Astarte.RPC.AMQP.Client, {:rpc, <<26, 33, 10, 29, 108, 117, 116, 101, 115, 116, 47, 98, 113, 53, 103, 101, 76, 83, 95, 81, 57, 97, 75, 120, 117, 53, 119, 104, 56, 72, 97, 54, 103, 16, 1>>, "vmq_plugin_rpc"}, 5000)
    ** (EXIT) time out
    (elixir 1.14.5) lib/gen_server.ex:1038: GenServer.call/3
    (astarte_data_updater_plant 1.1.0) lib/astarte_data_updater_plant/rpc/vmq_plugin.ex:63: Astarte.DataUpdaterPlant.RPC.VMQPlugin.disconnect/2
    (astarte_data_updater_plant 1.1.0) lib/astarte_data_updater_plant/data_updater/impl.ex:2152: Astarte.DataUpdaterPlant.DataUpdater.Impl.force_disconnection/2
    (astarte_data_updater_plant 1.1.0) lib/astarte_data_updater_plant/data_updater/impl.ex:2131: Astarte.DataUpdaterPlant.DataUpdater.Impl.ask_clean_session/2
    (astarte_data_updater_plant 1.1.0) lib/astarte_data_updater_plant/data_updater/impl.ex:875: Astarte.DataUpdaterPlant.DataUpdater.Impl.handle_data/6
    (astarte_data_updater_plant 1.1.0) lib/astarte_data_updater_plant/data_updater/server.ex:83: Astarte.DataUpdaterPlant.DataUpdater.Server.handle_cast/2
    (stdlib 4.3.1) gen_server.erl:1123: :gen_server.try_dispatch/4
    (stdlib 4.3.1) gen_server.erl:1200: :gen_server.handle_msg/6

After some debugging we found that the astarte_vmq_plugin_rpc queue was not present in RabbitMQ and even killing Astarte.RPC.AMQP.Server (which is responsible for declaring said queue) did not restore it.
in the end we opted for restarting VerneMQ which solved the issue and the queue was declared as expected.

Environment:

• Astarte v1.1.0
• RabbitMQ 3.8.34 (with 3 replicas)
• Astarte VerneMQ 1.1.0

Send a periodic heartbeat for every device to Data Updater Plant

There are some corner cases (e.g. VerneMQ restarts and, while restarting, a device disconnects) where Astarte devices are stuck in the connected: true state.

To avoid this, the VerneMQ plugin should send a periodic "heartbeat" to indicate that a device is still connected. DUP will then handle the interruption of the heartbeat as a sign that the device is not connected anymore, and will act accordingly.

Docker and CI cleanup

1. Travis CI (release-0.10, on Github)

• Add Travis CI

2. Gitlab CI cleanup (release-0.10, on Gitlab)

• Cache deps directory using $CI_COMMIT_REF_SLUG cache key
• Use RabbitMQ 3.7.15
• test stage, consisting of 2 jobs:
• mix:test-coverage: runs mix coveralls.details
• docker:test-container: builds the docker image without pushing it
• deploy stage, consisting of 1 job:
• docker:snapshot-prod-container: builds the docker image with env prod and pushes it to Gitlab Registry
• Replace DOCKER_REGISTRY with CI_REGISTRY

3. Dockerfile + Gitlab CI cleanup (master, on Gitlab)

• Add mix format --check-formatted to test-coverage
• Add config/ci.exs with just use Mix.Config to be able to use ci as env.
• Add dialyzex dep and configure it to use local directory for PLTs in ci env
• Add the PLTs directory to cached paths
• Add mix:test-dialyzer job to test stage, runs mix dialyzer in ci env.

4. Travis CI cleanup (master, on Github)

• Use elixir 1.8.1 and otp 21.3
• Update RabbitMQ to 3.7.21
• Add mix format --check-formatted so that it runs for external contributions

Do not authorize non-devices blindly in auth_on_publish and auth_on_subscribe

Right now the Plugin is returning :ok in auth_on_publish and auth_on_subscribe when the username doesn't contain a / (and so it is not considered a device).

Instead, :next should be returned to let another plugin down the chain decide what to do.

Docker image does not work on Apple M1 machine

As the title says. The error appears both when running the image alone and with the Astarte docker-compose.
Here's the log:

m1@5c5e690b-19ac-4f24-9aee-2925b86df2e8 astarte % docker run astarte/vernemq:1.0.0
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
request send failed: Operation not supported
15:03:14.906 [error] Error generating configuration in phase transform_datatypes
15:03:14.906 [error] Error transforming datatype for: listener.http.metrics
15:03:14.906 [error] ":8888" cannot be converted to a(n) IP
15:03:14.907 [error] Error transforming datatype for: listener.ws.default
15:03:14.907 [error] ":8080" cannot be converted to a(n) IP
15:03:14.907 [error] Error transforming datatype for: listener.tcp.revproxy
15:03:14.907 [error] ":1885" cannot be converted to a(n) IP
15:03:14.907 [error] Error transforming datatype for: listener.tcp.default
15:03:14.907 [error] ":1883" cannot be converted to a(n) IP
15:03:14.907 [error] Error transforming datatype for: listener.vmq.clustering
15:03:14.907 [error] ":44053" cannot be converted to a(n) IP
configuration error, exit
15:03:14.906 [error] Error generating configuration in phase transform_datatypes
15:03:14.906 [error] Error transforming datatype for: listener.http.metrics
15:03:14.906 [error] ":8888" cannot be converted to a(n) IP
15:03:14.907 [error] Error transforming datatype for: listener.ws.default
15:03:14.907 [error] ":8080" cannot be converted to a(n) IP
15:03:14.907 [error] Error transforming datatype for: listener.tcp.revproxy
15:03:14.907 [error] ":1885" cannot be converted to a(n) IP
15:03:14.907 [error] Error transforming datatype for: listener.tcp.default
15:03:14.907 [error] ":1883" cannot be converted to a(n) IP
15:03:14.907 [error] Error transforming datatype for: listener.vmq.clustering
15:03:14.907 [error] ":44053" cannot be converted to a(n) IP

Logging is not consistent with the rest of Astarte components

We should standardize VerneMQ logging to use logfmt format like all other Astarte services.

Moreover, right now the logger does not print a date, just a timestamp, so it's hard to understand when something was logged if a problem is being debugged some days later.

Default data queue count to 128

Default data queue count to 128 to make it consistent with both Astarte and Astarte Operator.

Configuration script runs despite `CFSSL_URL` being not set when SSL listener is enabled

Using VerneMQ with SSL listener enabled, in the script used to write /opt/vernemq/etc/vernemq.conf, the following line should stop and exit when the environment variable CFSSL_URL is not set.

astarte_vmq_plugin/docker/bin/vernemq.sh

Line 104 in d34d65a

if ! curl -s -d '{"label": "primary"}' -X POST $CFSSL_URL/api/v1/cfssl/info | jq -e -r ".result.certificate" > /etc/ssl/cfssl-ca-cert.crt; then

However, it doesn't. This allows for a broken SSL listener setup where CFSSL's CA is missing.