gen_rpc error on ipv6 only environment about emqx

[zmstone]

@zhupeijun did such logs repeat until the service restart ?

[zhupeijun]

@zmstone yes, it repeats infinitely. but the service seems still alive

uptime    : 1 days, 1 hours, 42 minutes, 1 seconds

[zmstone]

@zhupeijun thank you for your reply.
How many nodes are you trying to cluster, could you share logs from the other nodes which are trying to connect this node?

Although I cannot reproduce the enotconn error code, I am able to get a similar error of code einval.
This seems to be a race condition.

[zhupeijun]

@zmstone we have 2 nodes, both printed similar logs.

Node1

2024-01-17T10:29:21.788662+08:00 [error] crasher: initial call: gen_rpc_server:init/1, pid: <0.21589.23>, registered_name: gen_rpc_server_tcp, error: {{badmatch,{error,enotconn}},[{gen_rpc_driver_tcp,get_peer,1,[{file,"gen_rpc_driver_tcp.erl"},{line,134}]},{gen_rpc_server,waiting_for_connection,3,[{file,"gen_rpc_server.erl"},{line,74}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1426}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}, ancestors: [gen_rpc_sup,<0.2185.0>], message_queue_len: 0, messages: [], links: [#Port<0.991414>,#Port<0.991418>,<0.2186.0>], dictionary: [], trap_exit: false, status: running, heap_size: 4185, stack_size: 28, reductions: 8015; neighbours:
2024-01-17T10:29:21.789082+08:00 [error] Supervisor: {local,gen_rpc_sup}. Context: child_terminated. Reason: {{badmatch,{error,enotconn}},[{gen_rpc_driver_tcp,get_peer,1,[{file,"gen_rpc_driver_tcp.erl"},{line,134}]},{gen_rpc_server,waiting_for_connection,3,[{file,"gen_rpc_server.erl"},{line,74}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1426}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}. Offender: id=gen_rpc_server_tcp,pid=<0.21589.23>.
2024-01-17T10:29:22.060430+08:00 [error] State machine gen_rpc_server_tcp terminating. Reason: {badmatch,{error,enotconn}}. Stack: [{gen_rpc_driver_tcp,get_peer,1,[{file,"gen_rpc_driver_tcp.erl"},{line,134}]},{gen_rpc_server,waiting_for_connection,3,[{file,"gen_rpc_server.erl"},{line,74}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1426}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]. Last event: {internal,accept}. State: {waiting_for_connection,{state,#Port<0.991419>,tcp,gen_rpc_driver_tcp}}.
2024-01-17T10:29:22.060794+08:00 [error] crasher: initial call: gen_rpc_server:init/1, pid: <0.21591.23>, registered_name: gen_rpc_server_tcp, error: {{badmatch,{error,enotconn}},[{gen_rpc_driver_tcp,get_peer,1,[{file,"gen_rpc_driver_tcp.erl"},{line,134}]},{gen_rpc_server,waiting_for_connection,3,[{file,"gen_rpc_server.erl"},{line,74}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1426}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}, ancestors: [gen_rpc_sup,<0.2185.0>], message_queue_len: 0, messages: [], links: [#Port<0.991419>,#Port<0.991422>,<0.2186.0>], dictionary: [], trap_exit: false, status: running, heap_size: 4185, stack_size: 28, reductions: 8017; neighbours:
2024-01-17T10:29:22.061251+08:00 [error] Supervisor: {local,gen_rpc_sup}. Context: child_terminated. Reason: {{badmatch,{error,enotconn}},[{gen_rpc_driver_tcp,get_peer,1,[{file,"gen_rpc_driver_tcp.erl"},{line,134}]},{gen_rpc_server,waiting_for_connection,3,[{file,"gen_rpc_server.erl"},{line,74}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1426}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}. Offender: id=gen_rpc_server_tcp,pid=<0.21591.23>.
2024-01-17T10:29:23.438408+08:00 [error] State machine gen_rpc_server_tcp terminating. Reason: {badmatch,{error,enotconn}}. Stack: [{gen_rpc_driver_tcp,get_peer,1,[{file,"gen_rpc_driver_tcp.erl"},{line,134}]},{gen_rpc_server,waiting_for_connection,3,[{file,"gen_rpc_server.erl"},{line,74}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1426}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]. Last event: {internal,accept}. State: {waiting_for_connection,{state,#Port<0.991423>,tcp,gen_rpc_driver_tcp}}.

Node2

2024-01-17T10:31:24.015710+08:00 [error] crasher: initial call: gen_rpc_server:init/1, pid: <0.18306.23>, registered_name: gen_rpc_server_tcp, error: {{badmatch,{error,enotconn}},[{gen_rpc_driver_tcp,get_peer,1,[{file,"gen_rpc_driver_tcp.erl"},{line,134}]},{gen_rpc_server,waiting_for_connection,3,[{file,"gen_rpc_server.erl"},{line,74}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1426}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}, ancestors: [gen_rpc_sup,<0.2241.0>], message_queue_len: 0, messages: [], links: [#Port<0.985692>,#Port<0.985696>,<0.2242.0>], dictionary: [], trap_exit: false, status: running, heap_size: 4185, stack_size: 28, reductions: 8017; neighbours:
2024-01-17T10:31:24.016086+08:00 [error] Supervisor: {local,gen_rpc_sup}. Context: child_terminated. Reason: {{badmatch,{error,enotconn}},[{gen_rpc_driver_tcp,get_peer,1,[{file,"gen_rpc_driver_tcp.erl"},{line,134}]},{gen_rpc_server,waiting_for_connection,3,[{file,"gen_rpc_server.erl"},{line,74}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1426}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}. Offender: id=gen_rpc_server_tcp,pid=<0.18306.23>.
2024-01-17T10:31:24.792451+08:00 [error] State machine gen_rpc_server_tcp terminating. Reason: {badmatch,{error,enotconn}}. Stack: [{gen_rpc_driver_tcp,get_peer,1,[{file,"gen_rpc_driver_tcp.erl"},{line,134}]},{gen_rpc_server,waiting_for_connection,3,[{file,"gen_rpc_server.erl"},{line,74}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1426}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]. Last event: {internal,accept}. State: {waiting_for_connection,{state,#Port<0.985697>,tcp,gen_rpc_driver_tcp}}.
2024-01-17T10:31:24.792745+08:00 [error] crasher: initial call: gen_rpc_server:init/1, pid: <0.18307.23>, registered_name: gen_rpc_server_tcp, error: {{badmatch,{error,enotconn}},[{gen_rpc_driver_tcp,get_peer,1,[{file,"gen_rpc_driver_tcp.erl"},{line,134}]},{gen_rpc_server,waiting_for_connection,3,[{file,"gen_rpc_server.erl"},{line,74}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1426}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}, ancestors: [gen_rpc_sup,<0.2241.0>], message_queue_len: 0, messages: [], links: [#Port<0.985697>,#Port<0.985699>,<0.2242.0>], dictionary: [], trap_exit: false, status: running, heap_size: 4185, stack_size: 28, reductions: 8015; neighbours:
2024-01-17T10:31:24.793063+08:00 [error] Supervisor: {local,gen_rpc_sup}. Context: child_terminated. Reason: {{badmatch,{error,enotconn}},[{gen_rpc_driver_tcp,get_peer,1,[{file,"gen_rpc_driver_tcp.erl"},{line,134}]},{gen_rpc_server,waiting_for_connection,3,[{file,"gen_rpc_server.erl"},{line,74}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1426}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}. Offender: id=gen_rpc_server_tcp,pid=<0.18307.23>.
2024-01-17T10:31:25.118951+08:00 [error] State machine gen_rpc_server_tcp terminating. Reason: {badmatch,{error,enotconn}}. Stack: [{gen_rpc_driver_tcp,get_peer,1,[{file,"gen_rpc_driver_tcp.erl"},{line,134}]},{gen_rpc_server,waiting_for_connection,3,[{file,"gen_rpc_server.erl"},{line,74}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1426}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]. Last event: {internal,accept}. State: {waiting_for_connection,{state,#Port<0.985700>,tcp,gen_rpc_driver_tcp}}.

Our environment has no v4 ip and routing

$ ifcofnig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet6 < ipv6 addr #### >   prefixlen 64  scopeid 0x0<global>
        ether <##### > txqueuelen 0  (Ethernet)