erlio / vmq_server Goto Github PK

08:49:00.326 [error] Error generating configuration in phase apply_translations
08:49:00.326 [error] Translation for 'vmq_server.persistent_client_expiration' found invalid configuration: 1d must be an integer followed by one of 'h', 'd', 'w', 'm', 'y' for hour, day, week, month, and year, or the string 'never'
Error generating config with cuttlefish

The commit doesn't exist and so the project fails to build.

Limit sessions displayed with vmq-admin session list

is its omission from stats intentional or an oversight?

Currently the delivery of offline messages to a reconnected subscriber is done one by one. This means that every message must go through the whole ack cycle before the next message is sent to the client.
This is efficient in terms of copying messages around in case of a multi node setup. However this is quite (very) slow and should be improved. Multiple solutions exist:

• Introducing a batch mode
• only do one-by-one for offline messages coming from different node
• combination of both

We fold the ETS based retain cache every second to search non-persisted Retain Items... This works great if the number of items isn't huge.. Optimize this to use ets:select and continuations to allow a 'huge' amount of Retain Items.

I have a setup where I dynamically discover hosts and they dynamically try to form a cluster. It might be that a host goes down (i.e. the container stops) only to join again later on with the same hostname but a clean state. The hosts seem to continuously try to not only join one another (which works) but also to join themselves, as depicted below.

12:54:29.440 [info] Application vmq_server started on node '[email protected]'
12:54:29.868 [debug] got connect request from unknown cluster node '[email protected]'
12:54:29.868 [warning] terminate due to remote_node_not_available

(The last two log entries keep on being repeated)

I suspect that the host tries to join all other cluster nodes, and doesn't check whether the host equals the current node.

...I suspect that this in turn leads to my connections being refused with 'not_ready'...

I was load testing verne with 10 clients concurrently sending 20000 messages each. Sometimes the I get reset error and then the client doesn't receive any more messages after that.

Hi,

I need a serious dataplane that support millions of connections and for this reason I would like to add dpdk or netmap support for vernemq. Any "best practice" advice from you ?
Basically I thought to use edpdk on top of a dpdk kubernetes network manager.

However I'm new in Erlang. I love it but I don't have experience so any advice from you would be great. Thank you.

vmq-admin cluster leave doesn't work

Using the paho android client library...I am getting this error in the broker and the client disconnects

2015-06-01 15:37:40.840 [debug] <0.2493.0>@vmq_session_proxy:handle_info:87 ['[email protected]'][{[],"mqttjs334389048b872a533002b34d73f8c29fd09efc50"}] stopped message replication for subscriber '[email protected]' due to deliver_from_store proc stopped normally
2015-06-01 15:37:41.564 [warning] <0.2490.0>@vmq_ranch:teardown:121 [<0.2491.0>] session stopped
abnormally due to parse_error

This sound like a useful option, where QoS 0 messages are optionally persisted for later delivery...

currently scaling down a VerneMQ cluster won't handoff the persisted messages to another cluster node. When taking a node out of the cluster it should be possible to distribute the persisted messages to one or more nodes of the cluster.

Hi there,

thanks for the fantastic project!
Have a quick question (currently stuck).
Is there any way to pass client state between authentication and authorization hooks? Lets say there is some logic in register hook that extract \ inits some security data for the client session. This info needs to be persisted somewhere to verify on each user subscribe(), publish() call later. Is there any way on how to do this properly without touching VerneMQ core client session objects and maintaining dedicated for this ets table in plugin (where CliendId is kind of a key and value - security data)?

Thanks!

After running VMQ for a while and having several hundreds of (re)-connects, it keeps logging crashes in a metrics counter.

I'm not sure what it is related to, but I can try to investigate if you can give me hint of what to look for.

09:30:13.974 [error] gen_server vmq_metrics terminated with reason: bad argument in call to mzmetrics:update_resource_counter(<<>>, 0, 341606371714284795) in lists:foreach/2 line 1337
09:30:13.974 [error] CRASH REPORT Process vmq_metrics with 0 neighbours exited with reason: bad argument in call to mzmetrics:update_resource_counter(<<>>, 0, 341606371714284795) in lists:foreach/2 line 1337
09:30:13.976 [error] Supervisor vmq_metrics_sup had child undefined started with vmq_metrics:start_link() at <0.25047.25> exit with reason bad argument in call to mzmetrics:update_resource_counter(<<>>, 0, 341606371714284795) in lists:foreach/2 line 1337 in context child_terminated
09:30:13.976 [debug] Supervisor vmq_metrics_sup started vmq_metrics:start_link() at pid <0.25029.25>
09:30:14.976 [error] GenServer :vmq_metrics terminating
** (ArgumentError) argument error
    (mzmetrics) :mzmetrics.update_resource_counter("", 0, 341606371714284803)
    (stdlib) lists.erl:1337: :lists.foreach/2
    (vmq_server) /home/vagrant/vernemq/distdir/0.13.1/_build/default/lib/vmq_server/src/vmq_metrics.erl:486: :vmq_metrics.handle_info/2
    (stdlib) gen_server.erl:615: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:681: :gen_server.handle_msg/5
    (stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
Last message: :calc_rates
State: {:state}

"bin/vmq-admin config reset" should show documentation that the command is not complete, but fails instead with:

RPC to '[email protected]' failed: {'EXIT',
{{badmatch,false},
[{vmq_config_cli,
'-vmq_config_reset_cmd/0-fun-5-',2,
[{file,
"/home/afa/Erlang/verne/vernemq/_build/default/lib/vmq_server/src/vmq_config_cli.erl"},
{line,180}]},
{clique_command,run,1,
[{file,
"/home/afa/Erlang/verne/vernemq/_build/default/lib/clique/src/clique_command.erl"},
{line,59}]},
{clique,run,1,
[{file,
"/home/afa/Erlang/verne/vernemq/_build/default/lib/clique/src/clique.erl"},
{line,117}]},
{rpc,'-handle_call_call/6-fun-0-',5,
[{file,"rpc.erl"},{line,205}]}]}}

We don't need to log "[warning] <0.97.0> OS_MON (cpu_sup), util/1 unavailable for this OS" (which is true for OS X) every second to console.log.
This is of minor importance.

Hi there, I'm trying to run the project on CentOS 7 but whatever I do, VerneMQ always fails to start with vernemq start. Here is the process showing what's happening:

[root@ip-10-239-182-15 centos]# ps aux | grep verne
vernemq   9652  0.0  0.0  11580   336 ?        S    17:35   0:00 /usr/lib64/vernemq/erts-7.3/bin/epmd -daemon
root     10706  0.0  0.0 112648   972 pts/0    S+   17:39   0:00 grep --color=auto verne
[root@ip-10-239-182-15 centos]#
[root@ip-10-239-182-15 centos]# vernemq console
config is OK
-config /var/lib/vernemq/generated.configs/app.2016.08.16.17.39.43.config -args_file /var/lib/vernemq/generated.configs/vm.2016.08.16.17.39.43.args -vm_args /var/lib/vernemq/generated.configs/vm.2016.08.16.17.39.43.args
Exec:  /usr/lib64/vernemq/erts-7.3/bin/erlexec -boot /usr/lib64/vernemq/releases/0.13.1/vernemq               -config /var/lib/vernemq/generated.configs/app.2016.08.16.17.39.43.config -args_file /var/lib/vernemq/generated.configs/vm.2016.08.16.17.39.43.args -vm_args /var/lib/vernemq/generated.configs/vm.2016.08.16.17.39.43.args              -pa /usr/lib64/vernemq/lib/erlio-patches -- console
Root: /usr/lib64/vernemq
Erlang/OTP 18 [erts-7.3] [source] [64-bit] [smp:1:1] [async-threads:64] [hipe] [kernel-poll:true]

Eshell V7.3  (abort with ^G)
(VerneMQ@vernemq-0)1> q().
ok
(VerneMQ@vernemq-0)2> [os_mon] memory supervisor port (memsup): Erlang has closed
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed
[root@ip-10-239-182-15 centos]# ps aux | grep verne
vernemq   9652  0.0  0.0  11580   336 ?        S    17:35   0:00 /usr/lib64/vernemq/erts-7.3/bin/epmd -daemon
root     11028  0.0  0.0 112648   972 pts/0    R+   17:39   0:00 grep --color=auto verne
[root@ip-10-239-182-15 centos]#
[root@ip-10-239-182-15 centos]# vernemq start
vernemq failed to start within 15 seconds,
see the output of 'vernemq console' for more information.
If you want to wait longer, set the environment variable
WAIT_FOR_ERLANG to the number of seconds to wait.
[root@ip-10-239-182-15 centos]# ps aux | grep verne
vernemq   9652  0.0  0.0  11580   336 ?        S    17:35   0:00 /usr/lib64/vernemq/erts-7.3/bin/epmd -daemon
vernemq  11151  0.4  0.0  13676   744 ?        S    17:40   0:00 /usr/lib64/vernemq/erts-7.3/bin/run_erl -daemon /var/run/vernemq// /var/log/vernemq exec /usr/sbin/vernemq console
vernemq  11153  5.3  2.4 1595940 91348 pts/1   Ssl+ 17:40   0:01 /usr/lib64/vernemq/erts-7.3/bin/beam.smp -P 256000 -A 64 -K true -W w -- -root /usr/lib64/vernemq -progname vernemq -- -home /var/lib/vernemq -- -boot /usr/lib64/vernemq/releases/0.13.1/vernemq -config /var/lib/vernemq/generated.configs/app.2016.08.16.17.40.07.config -setcookie vmq -name VerneMQ@vernemq-0 -smp enable -vm_args /var/lib/vernemq/generated.configs/vm.2016.08.16.17.40.07.args -pa /usr/lib64/vernemq/lib/erlio-patches -- console
vernemq  11487  0.6  0.0 113124  1204 ?        Ss   17:40   0:00 sh -s disksup
vernemq  11488  0.4  0.0   4300   568 ?        Ss   17:40   0:00 /usr/lib64/vernemq/lib/os_mon-2.4/priv/bin/memsup
vernemq  11489  0.5  0.0   4300   568 ?        Ss   17:40   0:00 /usr/lib64/vernemq/lib/os_mon-2.4/priv/bin/cpu_sup
root     11823  0.0  0.0 112648   972 pts/0    R+   17:40   0:00 grep --color=auto verne

To me, it seems that the process is running, however:

[root@ip-10-239-182-15 vernemq]# vernemq ping
Node 'VerneMQ@vernemq-0' not responding to pings.

Okay, maybe it's the timeout. Let's try:

[root@ip-10-239-182-15 centos]# kill 11151
[root@ip-10-239-182-15 centos]# ps aux | grep verne
vernemq   9652  0.0  0.0  11580   336 ?        S    17:35   0:00 /usr/lib64/vernemq/erts-7.3/bin/epmd -daemon
root     10706  0.0  0.0 112648   972 pts/0    S+   17:39   0:00 grep --color=auto verne
[root@ip-10-239-182-15 vernemq]# export WAIT_FOR_ERLANG=60
[root@ip-10-239-182-15 vernemq]# vernemq start
vernemq failed to start within 15 seconds,
see the output of 'vernemq console' for more information.
If you want to wait longer, set the environment variable
WAIT_FOR_ERLANG to the number of seconds to wait.
[root@ip-10-239-182-15 centos]#

My CentOS:

[root@ip-10-239-182-15 vernemq]# uname -a
Linux ip-10-239-182-15 3.10.0-327.28.2.el7.x86_64 #1 SMP Wed Aug 3 11:11:39 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Nothing in /var/log/vernemq indicates any errors. How do I debug?

Setting -env ERL_CRASH_DUMP /erl_crash.dump possibly prevents erl_crash.dump from being written.

HI,

I am new to common test framework.I tried with all the commands but I am not able to run them.Could you please help in running the test suites in vmq_server/test directory?

Thank you...

Best Regards,
shahid.

Improve Logging of Parse Errors