vmq_mzbench workers crashed with message "no function clause matching gen_emqtt:handle_frame(waiting_for_connack....)" about vmq_mzbench

[ioolkos]

Hi @akdj thanks for opening the issue.
I can review your BDL script, but I need you to show me a runnable complete version of it. (replacing your Broker IP with "127.0.0.1" to not publicly announce your IP).

[a-wf]

Hi @ioolkos

Thank you for your response! here is my full BDL script:
In fact my broker IP is my load balancer(HAproxy)'s IP

#!benchDL

make_install(rsync = "/worker/vmq_mzbench/",
             exclude = "deps")

defaults("username" = "user1")
defaults("password" = "passwd1")

defaults("broker" = "127.0.0.1")
defaults("port" = 1883)

defaults("topic" = "test/user1/loadtest/")

defaults("poolsize" = 200)

# wait time for consumers in seconds
defaults("wait_time" = 15)
# publisher loop duration in minutes
defaults("loop_time" = 10)
# publisher loop rate in rpm
defaults("loop_rate" = 1250)

# Nb rps = (nb pool * poolsize * loop_rate) / 60

pool(size = numvar("poolsize"),
     worker_type = mqtt_worker,
     worker_start = poisson(25 rps)):
        connect([t(host, var("broker")),
                 t(port, var("port")),
                 t(username, var("username")),
                 t(password, var("password")),
                 t(client, fixed_client_id("loadtests-mqttbroker-id0", worker_id())),
                 t(clean_session, true),
                 t(keepalive_interval, 60),
                 t(proto_version, 4),
                 t(reconnect_timeout, 45)
                ])
        wait(var("wait_time") sec)
        # wait for consumer to be setup
        subscribe(var("topic"), 0)
        set_signal(connect1, 1)
        wait_signal(connect1, numvar("poolsize"))
        wait(var("wait_time") sec)
        loop(time = var("loop_time") min, rate = var("loop_rate") rpm):
            publish(var("topic"), random_binary(150), 0)
        wait(30 sec)
        disconnect()

[ioolkos]

@akdj you have to get numvars as numvars:

#!benchDL

make_install(git = "https://github.com/erlio/vmq_mzbench.git",
             branch = "master")

defaults("username" = "user1")
defaults("password" = "passwd1")

defaults("broker" = "127.0.0.1")
defaults("port" = 1883)

defaults("topic" = "test/user1/loadtest/")

defaults("poolsize" = 200)

# wait time for consumers in seconds
defaults("wait_time" = 15)
# publisher loop duration in minutes
defaults("loop_time" = 10)
# publisher loop rate in rpm
defaults("loop_rate" = 1250)

# Nb rps = (nb pool * poolsize * loop_rate) / 60

pool(size = numvar("poolsize"),
     worker_type = mqtt_worker,
     worker_start = poisson(25 rps)):
        connect([t(host, var("broker")),
                 t(port, numvar("port")),
                 t(username, var("username")),
                 t(password, var("password")),
                 t(client, fixed_client_id("loadtests-mqttbroker-id0", worker_id())),
                 t(clean_session, true),
                 t(keepalive_interval, 60),
                 t(proto_version, 4),
                 t(reconnect_timeout, 45)
                ])
        wait(var("wait_time") sec)
        # wait for consumer to be setup
        subscribe(var("topic"), 0)
        set_signal(connect1, 1)
        wait_signal(connect1, numvar("poolsize"))
        wait(var("wait_time") sec)
        loop(time = numvar("loop_time") min, rate = numvar("loop_rate") rpm):
            publish(var("topic"), random_binary(150), 0)
        wait(30 sec)
        disconnect()

[a-wf]

thanks @ioolkos

I changed it, and retried. But I still have the same error messages:
user:

12:00:27.210 [error] [Undefined] <0.632.0> gen_fsm <0.632.0> in state waiting_for_connack terminated with reason: no function clause matching gen_emqtt:handle_frame(waiting_for_connack, {mqtt_publish,undefined,[<<"test">>,<<"user1">>,<<"loadtest">>,<<>>],0,0,0,<<131,104,2,104,3,98,...>>}, {state,"127.0.0.1",1883,#Port<0.3507>,6268,"user1","passwd1",["loadtest...",...],...}) line 354

12:00:27.823 [error] [Undefined] <0.632.0> CRASH REPORT Process <0.632.0> with 1 neighbours exited with reason: no function clause matching gen_emqtt:handle_frame(waiting_for_connack, {mqtt_publish,undefined,[<<"test">>,<<"user1">>,<<"loadtest">>,<<>>],0,0,0,<<131,104,2,104,3,98,...>>}, {state,"127.0.0.1",1883,#Port<0.3507>,6268,"user1","passwd1",["loadtest...",...],...}) line 354 in gen_fsm:terminate/7 line 559

12:03:28.856 [error] [Undefined] <0.426.0> gen_fsm <0.426.0> in state waiting_for_connack terminated with reason: no function clause matching gen_emqtt:handle_frame(waiting_for_connack, {mqtt_publish,undefined,[<<"test">>,<<"user1">>,<<"loadtest">>,<<>>],0,0,0,<<131,104,2,104,3,98,...>>}, {state,"127.0.0.1",1883,#Port<0.3627>,10067,"user1","passwd1",["loadtes...",...],...}) line 354

12:03:28.890 [error] [Undefined] <0.426.0> CRASH REPORT Process <0.426.0> with 1 neighbours exited with reason: no function clause matching gen_emqtt:handle_frame(waiting_for_connack, {mqtt_publish,undefined,[<<"test">>,<<"user1">>,<<"loadtest">>,<<>>],0,0,0,<<131,104,2,104,3,98,...>>}, {state,"127.0.0.1",1883,#Port<0.3627>,10067,"user1","passwd1",["loadtes...",...],...}) line 354 in gen_fsm:terminate/7 line 559

system:

Raw logScroll to end

12:00:28.235 [error] [[email protected]] <0.260.0> [ "pool1" ] Received DOWN from worker <0.631.0> with reason {function_clause,[{gen_emqtt,handle_frame,[waiting_for_connack,{mqtt_publish,undefined,[<<"test">>,<<"user1">>,<<"loadtest">>,<<>>],0,0,0,<<131,104,2,104,3,98,0,0,5,240,98,0,14,96,252,98,0,4,23,172,109,0,0,0,150,91,248,233,66,113,226,9,195,213,219,236,230,56,122,83,70,33,112,124,186,81,94,241,240,153,84,180,214,22,190,242,70,155,28,221,240,83,31,208,17,10,67,115,234,204,249,176,208,60,248,17,226,67,174,65,15,108,196,113,245,87,127,118,181,47,93,106,45,181,94,116,91,113,18,225,26,142,216,20,67,208,170,240,54,181,51,187,195,54,194,21,145,159,40,192,57,5,190,222,216,30,223,136,95,10,36,9,72,83,145,114,250,79,110,76,102,174,243,239,16,190,139,240,84,83,154,167,105,178,110,94,242,227,189,226,68,68,149,165,253,207,198,220,55,120,94,32,2,0,0>>},{state,"127.0.0.1",1883,#Port<0.3507>,6268,"user1","passwd1",["loadtests-mqttbroker-id0","-","183"],true,undefined,undefined,0,<<>>,45000,60000,10000,4,mqtt_worker,[],{gen_tcp,[]},<<48,196,1,0,19,115,97,104,47,117,115,101,114,49,47,108,111,97,100,116,101,115,116,47,131,104,2,104,3,98,0,0,5,240,98,0,14,96,252,98,0,4,23,172,109,0,0,0,150,91,248,233,66,113,226,9,195,213,219,236,230,56,122,83,70,33,112,124,186,81,94,241,240,153,84,180,214,22,190,242,70,155,28,221,240,83,31,208,17,10,67,115,234,204,249,176,208,60,248,17,226,67,174,65,15,108,196,113,245,87,127,118,181,47,93,106,45,181,94,116,91,113,18,225,26,142,216,20,67,208,170,240,54,181,51,187,195,54,194,21,145,159,40,192,57,5,190,222,216,30,223,136,95,10,36,9,72,83,145,114,250,79,110,76,102,174,243,239,16,190,139,240,84,83,154,167,105,178,110,94,242,227,189,226,68,68,149,165,253,207,198,220,55,120,94>>,{dict,1,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[[ping|#Ref<0.0.1.16683>]],[],[],[],[],[],[],[],[]}}},{#Fun<mqtt_worker.0.79002790>,#{462 => {1520,942101,982539},704 => {1520,942113,597196},5961 => {1520,942366,68532},1328 => {1520,942143,553137},4351 => {1520,942288,867302},6074 => {1520,942371,574790},3590 => {1520,942252,200437},5934 => {1520,942364,847254},4631 => {1520,942302,426103},1935 => {1520,942172,698344},3742 => {1520,942259,609753},4969 => {1520,942318,595079},3216 => {1520,942234,199446},2992 => {1520,942223,558682},4395 => {1520,942291,24464},1792 => {1520,942165,819071},2339 => {1520,942192,81111},4380 => {1520,942290,213154},3212 => {1520,942234,90250},3091 => {1520,942228,308347},3521 => {1520,942248,955128},3079 => {1520,942227,725682},3184 => {1520,942232,751563},5075 => {1520,942323,606150},417 => {1520,942099,820135},3052 => {1520,942226,354613},513 => {1520,942104,432225},2633 => {1520,942206,188030},263 => {1520,942092,426222},525 => {1520,942105,7165},3951 => {1520,942269,767137},5572 => {1520,942347,396139},2380 => {1520,942194,42732},2528 => {1520,942201,146981},3883 => {1520,942266,514326},1013 => {1520,942128,427243},4193 => {1520,942281,300813},475 => {1520,942102,606635},3424 => {1520,942244,394978},500 => {1520,942103,802673},5673 => {1520,942352,275520},105 => {1520,942084,843150},4498 => {1520,942295,913692},3510 => {1520,942248,455217},5569 => {1520,942347,396090},5637 => {1520,942350,461783},3908 => {1520,942267,615460},878 => {1520,942121,946751},2468 => {1520,942198,271816},4606 => {1520,942301,2632},1513 => {1520,942152,429124},313 => {1520,942094,840574},2497 => {1520,942199,658693},2637 => {1520,942206,379051},3301 => {1520,942238,256630},1514 => {1520,942152,473697},3405 => {1520,942243,399601},3219 => {1520,942234,414471},5757 => {1520,942356,341841},4561 => {1520,942298,935858},1892 => {1520,942170,622661},3200 => {1520,942233,423342},3838 => {1520,942264,209740},1851 => {1520,942168,653950},2754 => {1520,942212,33640},1070 => {1520,942131,162561},2007 => {1520,942176,139948},5684 => {1520,942352,809401},3236 => {1520,942235,178768},1784 => {1520,942165,434971},5493 => {1520,942343,692198},296 => {1520,942094,34340},3800 => {1520,942262,504984},6105 => {1520,942372,991976},947 => {1520,942125,264393},3917 => {1520,942268,...},...}}}],...},...]}

12:03:28.945 [error] [[email protected]] <0.260.0> [ "pool1" ] Received DOWN from worker <0.425.0> with reason {function_clause,[{gen_emqtt,handle_frame,[waiting_for_connack,{mqtt_publish,undefined,[<<"test">>,<<"user1">>,<<"loadtest">>,<<>>],0,0,0,<<131,104,2,104,3,98,0,0,5,240,98,0,14,97,184,98,0,3,117,10,109,0,0,0,150,149,34,42,205,204,7,113,29,116,209,233,26,177,42,102,115,67,13,72,126,183,186,55,156,244,32,244,196,96,120,213,207,100,198,167,154,184,1,199,77,152,25,172,234,41,170,43,219,224,82,123,166,95,46,77,193,66,232,95,84,208,50,234,169,251,89,208,49,37,18,112,173,209,254,149,225,27,52,225,123,151,120,22,171,107,50,147,66,1,11,43,199,58,96,214,32,170,146,26,238,87,134,241,30,82,158,192,78,160,44,157,89,56,41,116,132,243,175,241,91,66,214,101,170,252,68,221,10,56,123,153,184,128,244,222,118,71,136,97,251,134,190,123,198,64,234,32,2,0,0>>},{state,"127.0.0.1",1883,#Port<0.3627>,10067,"user1","passwd1",["loadtests-mqttbroker-id0","-","80"],true,undefined,undefined,0,<<>>,45000,60000,10000,4,mqtt_worker,[],{gen_tcp,[]},<<48,196,1,0,19,115,97,104,47,117,115,101,114,49,47,108,111,97,100,116,101,115,116,47,131,104,2,104,3,98,0,0,5,240,98,0,14,97,184,98,0,3,117,10,109,0,0,0,150,149,34,42,205,204,7,113,29,116,209,233,26,177,42,102,115,67,13,72,126,183,186,55,156,244,32,244,196,96,120,213,207,100,198,167,154,184,1,199,77,152,25,172,234,41,170,43,219,224,82,123,166,95,46,77,193,66,232,95,84,208,50,234,169,251,89,208,49,37,18,112,173,209,254,149,225,27,52,225,123,151,120,22,171,107,50,147,66,1,11,43,199,58,96,214,32,170,146,26,238,87,134,241,30,82,158,192,78,160,44,157,89,56,41,116,132,243,175,241,91,66,214,101,170,252,68,221,10,56,123,153,184,128,244,222,118,71,136,97,251,134,190,123,198,64,234>>,{dict,1,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[[ping|#Ref<0.0.2.14420>]],[],[],[],[],[],[],[],[]}}},{#Fun<mqtt_worker.0.79002790>,#{462 => {1520,942101,956633},704 => {1520,942113,570512},7994 => {1520,942463,858228},6850 => {1520,942408,989992},6836 => {1520,942408,308244},9765 => {1520,942548,817651},6653 => {1520,942399,546924},5961 => {1520,942366,249057},1328 => {1520,942143,523409},4351 => {1520,942288,891724},8373 => {1520,942481,993425},6749 => {1520,942404,163171},9406 => {1520,942538,183956},8441 => {1520,942485,263561},6570 => {1520,942395,514631},6074 => {1520,942371,739816},3590 => {1520,942252,140778},7636 => {1520,942446,668665},5934 => {1520,942365,40088},4631 => {1520,942302,324540},1935 => {1520,942172,674835},8406 => {1520,942483,578224},9860 => {1520,942553,379085},3742 => {1520,942259,659751},9221 => {1520,942522,697152},4969 => {1520,942318,685657},9846 => {1520,942552,700548},3216 => {1520,942234,342081},2992 => {1520,942223,515047},4395 => {1520,942291,51294},6546 => {1520,942394,376340},7109 => {1520,942421,359672},7738 => {1520,942451,591555},1792 => {1520,942165,797703},2339 => {1520,942192,54738},4380 => {1520,942290,345783},9697 => {1520,942545,544640},3212 => {1520,942234,98755},3091 => {1520,942228,212973},3521 => {1520,942248,925658},9091 => {1520,942516,458144},6330 => {1520,942383,991435},6415 => {1520,942388,22716},3079 => {1520,942227,630712},3184 => {1520,942232,810609},10024 => {1520,942561,271812},8243 => {1520,942475,752820},5075 => {1520,942323,888801},417 => {1520,942099,793803},3052 => {1520,942226,508555},513 => {1520,942104,417971},2633 => {1520,942206,161624},263 => {1520,942092,402318},8854 => {1520,942505,82268},6459 => {1520,942390,137703},525 => {1520,942104,978434},3951 => {1520,942269,806157},5572 => {1520,942347,629286},2380 => {1520,942194,17538},2528 => {1520,942201,122073},9362 => {1520,942538,177908},3883 => {1520,942266,443636},1013 => {1520,942128,401671},4193 => {1520,942281,212155},475 => {1520,942102,576783},3424 => {1520,942244,248090},500 => {1520,942103,780009},5673 => {1520,942352,520073},105 => {1520,942084,817880},4498 => {1520,942295,965698},8832 => {1520,942504,26568},8468 => {1520,942486,552593},8527 => {1520,942489,385659},9704 => {1520,942545,883011},3510 => {1520,942248,378666},8596 => {1520,...},...}}}],...},...]}

12:05:05.297 [error] [[email protected]] <0.260.0> [ "pool1" ] Received DOWN from worker <0.513.0> with reason {function_clause,[{gen_emqtt,handle_frame,[waiting_for_connack,{mqtt_publish,undefined,[<<"test">>,<<"user1">>,<<"loadtest">>,<<>>],0,0,0,<<131,104,2,104,3,98,0,0,5,240,98,0,14,97,9,98,0,4,59,200,109,0,0,0,150,93,158,59,126,85,182,34,181,21,192,92,139,108,105,203,134,39,114,220,224,137,139,127,40,191,135,12,110,132,163,114,42,10,100,63,84,71,117,126,72,112,185,73,142,24,115,31,51,103,12,3,167,18,181,118,166,6,120,57,155,170,140,103,67,5,68,193,210,103,105,56,14,208,178,175,210,244,250,28,1,235,223,15,74,30,125,205,187,133,26,165,155,81,63,27,207,80,37,229,20,226,148,92,244,84,220,26,182,180,122,7,12,230,210,13,181,195,194,31,116,197,146,246,251,53,28,170,171,169,85,36,48,246,179,159,220,96,31,32,2,0,0,32,2,0,0,32,2,0,0>>},{state,"127.0.0.1",1883,#Port<0.3699>,6451,"user1","passwd1",["loadtests-mqttbroker-id0","-","124"],true,undefined,undefined,0,<<>>,45000,60000,10000,4,mqtt_worker,[],{gen_tcp,[]},<<48,196,1,0,19,115,97,104,47,117,115,101,114,49,47,108,111,97,100,116,101,115,116,47,131,104,2,104,3,98,0,0,5,240,98,0,14,97,9,98,0,4,59,200,109,0,0,0,150,93,158,59,126,85,182,34,181,21,192,92,139,108,105,203,134,39,114,220,224,137,139,127,40,191,135,12,110,132,163,114,42,10,100,63,84,71,117,126,72,112,185,73,142,24,115,31,51,103,12,3,167,18,181,118,166,6,120,57,155,170,140,103,67,5,68,193,210,103,105,56,14,208,178,175,210,244,250,28,1,235,223,15,74,30,125,205,187,133,26,165,155,81,63,27,207,80,37,229,20,226,148,92,244,84,220,26,182,180,122,7,12,230,210,13,181,195,194,31,116,197,146,246,251,53,28,170,171,169,85,36,48,246,179,159,220,96,31,32,2,0,0,32,2,0,0>>,{dict,1,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[[ping|#Ref<0.0.1.16645>]],[],[],[],[],[],[],[],[]}}},{#Fun<mqtt_worker.0.79002790>,#{462 => {1520,942101,945436},704 => {1520,942113,554809},5961 => {1520,942365,947238},1328 => {1520,942143,513997},4351 => {1520,942288,624393},6074 => {1520,942371,387964},3590 => {1520,942252,157462},5934 => {1520,942364,721347},4631 => {1520,942302,210870},1935 => {1520,942172,644515},3742 => {1520,942259,397856},4969 => {1520,942318,469563},3216 => {1520,942234,244311},2992 => {1520,942223,550355},4395 => {1520,942290,877796},1792 => {1520,942165,781111},2339 => {1520,942192,37587},4380 => {1520,942290,160289},3212 => {1520,942233,991521},3091 => {1520,942228,241001},3521 => {1520,942248,940417},6330 => {1520,942383,626718},6415 => {1520,942387,813813},3079 => {1520,942227,657293},3184 => {1520,942232,713815},5075 => {1520,942323,540822},417 => {1520,942099,780624},3052 => {1520,942226,254856},513 => {1520,942104,418075},2633 => {1520,942206,147009},263 => {1520,942092,385837},525 => {1520,942104,963018},3951 => {1520,942269,547860},5572 => {1520,942347,356334},2380 => {1520,942194,2746},2528 => {1520,942201,108985},3883 => {1520,942266,260389},1013 => {1520,942128,386846},4193 => {1520,942281,104443},475 => {1520,942102,563414},3424 => {1520,942244,88588},500 => {1520,942103,762606},5673 => {1520,942352,124179},105 => {1520,942084,802956},4498 => {1520,942295,796972},3510 => {1520,942248,308227},5569 => {1520,942347,137163},5637 => {1520,942350,424729},3908 => {1520,942267,458418},878 => {1520,942121,907353},2468 => {1520,942198,237351},4606 => {1520,942301,58986},1513 => {1520,942152,390658},313 => {1520,942094,795573},2497 => {1520,942199,619367},2637 => {1520,942206,338795},3301 => {1520,942238,305514},1514 => {1520,942152,437772},3405 => {1520,942243,349390},3219 => {1520,942234,364551},6350 => {1520,942384,647730},5757 => {1520,942356,200530},4561 => {1520,942298,725093},1892 => {1520,942170,586056},3200 => {1520,942233,470976},3838 => {1520,942263,959536},1851 => {1520,942168,615041},2754 => {1520,942212,99201},1070 => {1520,942131,125664},2007 => {1520,942176,103821},5684 => {1520,942352,748979},6314 => {1520,942382,885190},6426 => {1520,942388,238540},3236 => {1520,942235,130914},1784 => {1520,942165,393875},5493 => {1520,942343,488330},296 => {1520,942093,...},...}}}],...},...]}

12:05:09.934 [error] [ API ] <0.10399.0> Benchmark result: FAILED
3 of 200 workers failed

12:05:10.069 [error] [ API ] <0.10358.0> Stage 'pipeline - running': failed
Benchmark has failed on running with reason:
{benchmark_failed,{workers_failed,3}}

Stacktrace: [{mzb_pipeline,error,2,
                           [{file,"/mzbench/server/_build/default/deps/mzbench_api/src/mzb_pipeline.erl"},
                            {line,90}]},
             {mzb_pipeline,'-handle_cast/2-fun-0-',6,
                           [{file,"/mzbench/server/_build/default/deps/mzbench_api/src/mzb_pipeline.erl"},
                            {line,172}]}]

12:05:40.194 [error] [ API ] <0.10909.0> [ REMOTE EXEC ] Command execution failed:
Cmd: bash -c -l "export PATH='/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'; kill -9 "stdin: is not a tty\n11606"; true "
Exit code: 1
Output: stdin: is not a tty
is: line 0: kill: stdin:: arguments must be process or job IDs

12:05:40.195 [error] [ API ] <0.10358.0> Stage 'finalize - cleaning_nodes': failed
Command returned 1:
 bash -c -l "export PATH='/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'; kill -9 "stdin: is not a tty\n11606"; true "
Command output: stdin: is not a tty
is: line 0: kill: stdin:: arguments must be process or job IDs

[ioolkos]

Your script is working for me. The only difference is that i set allow_anonymous to 'on' as I don't have your username and passwords configured.
Can you connect to the broker with a MQTT client? do the provided credentials work?
Do you see any kind of timeouts from the Loadbalancer?
EDIT: and can you run the test by connecting directly to the broker IP?

[a-wf]

I can connect with the provided credentials
I tried before without the loadbalancer, and there is no error message.
However, I have a mqtt broker cluster (3 brokers) for having high availability. That why I am using the loadbalancer.

I will try again, and check the logs of loadbalancer.

Thank you @ioolkos , I will come back later :)

[a-wf]

@ioolkos Hi, I increased the max connection value in my Haproxy config, and mzbench's error messages disappear.
Thank you for your help
ps: I have another question, will open a new issue