Exception in the log when using HTTP-Auth-Backend about discussions

[klpanagi]

Dear @lukebakken and @michaelklishin, you can find the example here.

I have created a compose file which builds a network and includes:

• RabbitMQ message broker
• Authentication backend server
• webstomp client app

You can execute the testcase using the below command (build process must be enabled):

docker-compose -p rabbitmq-test up --remove-orphans --build

By default, rabbitmq is configured to enable reverse_dns_lookups, which produces the exception. By disabling reverse_dns_lookups everything works as expected

[klpanagi]

@lukebakken I get this error also for MQTT, with reverse_dns_lookups=False.

See below the log report:

r4a-rabbitmq-broker        | 2020-02-16 11:41:10.012 [info] <0.909.0> MQTT vhost picked using plugin configuration or default
r4a-broker-auth-backend    | 172.18.0.6 - - [16/Feb/2020 11:41:10] "POST /auth/user HTTP/1.1" 200 -
r4a-rabbitmq-broker        | 2020-02-16 11:41:10.025 [error] <0.912.0> ** Generic server <0.912.0> terminating 
r4a-rabbitmq-broker        | ** Last message in was connect
r4a-rabbitmq-broker        | ** When Server state == {<0.911.0>,{amqp_params_direct,<<"skata">>,<<"ZoKEQAlhG+COwrnFQctOKlFT4pg9Ra3LvhKMW/7bRzPYhoPtdRgds4dAzDp1BM3A">>,<<"/">>,rabbit@2ad4426d1750,{amqp_adapter_info,{172,18,0,6},1883,{155,207,33,187},47885,<<"155.207.33.187:47885 -> 172.18.0.6:1883">>,{'MQTT',"3.1.1"},[{variable_map,#{<<"client_id">> => <<"UPahzyw0tnNofE0FBEpzwA">>}},{channels,1},{channel_max,1},{frame_max,0},{client_properties,[{<<"product">>,longstr,<<"MQTT client">>},{client_id,longstr,<<"UPahzyw0tnNofE0FBEpzwA">>}]},{ssl,false}]},[]}}
r4a-rabbitmq-broker        | ** Reason for termination ==
r4a-rabbitmq-broker        | ** {function_clause,[{amqp_gen_connection,terminate,[{{case_clause,{badrpc,{'EXIT',{{badmatch,false},[{rabbit_auth_backend_http,do_http_req,2,[{file,"src/rabbit_auth_backend_http.erl"},{line,131}]},{rabbit_auth_backend_http,http_req,3,[{file,"src/rabbit_auth_backend_http.erl"},{line,118}]},{rabbit_auth_backend_http,bool_req,2,[{file,"src/rabbit_auth_backend_http.erl"},{line,109}]},{rabbit_auth_backend_cache,with_cache,3,[{file,"src/rabbit_auth_backend_cache.erl"},{line,88}]},{rabbit_access_control,check_access,5,[{file,"src/rabbit_access_control.erl"},{line,211}]},{lists,foldl,3,[{file,"lists.erl"},{line,1263}]},{rabbit_direct,connect1,5,[{file,"src/rabbit_direct.erl"},{line,191}]},{rpc,local_call,3,[{file,"rpc.erl"},{line,321}]}]}}}},[{amqp_direct_connection,connect,4,[{file,"src/amqp_direct_connection.erl"},{line,153}]},{amqp_gen_connection,handle_call,3,[{file,"src/amqp_gen_connection.erl"},{line,174}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,661}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,690}]},{proc_lib,init_p_do_apply,3,[{file,"proc_li..."},...]}]},...],...},...]}
r4a-rabbitmq-broker        | ** Client <0.909.0> stacktrace
r4a-rabbitmq-broker        | ** [{gen,do_call,4,[{file,"gen.erl"},{line,167}]},{gen_server,call,3,[{file,"gen_server.erl"},{line,219}]},{rabbit_mqtt_processor,process_login,4,[{file,"src/rabbit_mqtt_processor.erl"},{line,559}]},{rabbit_mqtt_processor,process_request,3,[{file,"src/rabbit_mqtt_processor.erl"},{line,137}]},{rabbit_mqtt_processor,process_frame,2,[{file,"src/rabbit_mqtt_processor.erl"},{line,77}]},{rabbit_mqtt_reader,process_received_bytes,2,[{file,"src/rabbit_mqtt_reader.erl"},{line,295}]},{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1067}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]
r4a-rabbitmq-broker        | 2020-02-16 11:41:10.026 [error] <0.912.0> CRASH REPORT Process <0.912.0> with 0 neighbours crashed with reason: no function clause matching amqp_gen_connection:terminate({{case_clause,{badrpc,{'EXIT',{{badmatch,false},[{rabbit_auth_backend_http,do_http_req,2,[{file,...},...]},...]}}}},...}, {<0.911.0>,{amqp_params_direct,<<"skata">>,<<"ZoKEQAlhG+COwrnFQctOKlFT4pg9Ra3LvhKMW/7bRzPYhoPtdRgd...">>,...}}) line 242
r4a-rabbitmq-broker        | 2020-02-16 11:41:10.026 [error] <0.910.0> Supervisor {<0.910.0>,amqp_connection_sup} had child connection started with amqp_gen_connection:start_link(<0.911.0>, {amqp_params_direct,<<"skata">>,<<"ZoKEQAlhG+COwrnFQctOKlFT4pg9Ra3LvhKMW/7bRzPYhoPtdRgds4dAzDp1BM3...">>,...}) at <0.912.0> exit with reason no function clause matching amqp_gen_connection:terminate({{case_clause,{badrpc,{'EXIT',{{badmatch,false},[{rabbit_auth_backend_http,do_http_req,2,[{file,...},...]},...]}}}},...}, {<0.911.0>,{amqp_params_direct,<<"skata">>,<<"ZoKEQAlhG+COwrnFQctOKlFT4pg9Ra3LvhKMW/7bRzPYhoPtdRgd...">>,...}}) line 242 in context child_terminated
r4a-rabbitmq-broker        | 2020-02-16 11:41:10.026 [error] <0.910.0> Supervisor {<0.910.0>,amqp_connection_sup} had child connection started with amqp_gen_connection:start_link(<0.911.0>, {amqp_params_direct,<<"skata">>,<<"ZoKEQAlhG+COwrnFQctOKlFT4pg9Ra3LvhKMW/7bRzPYhoPtdRgds4dAzDp1BM3...">>,...}) at <0.912.0> exit with reason reached_max_restart_intensity in context shutdown
r4a-rabbitmq-broker        | 2020-02-16 11:41:10.027 [error] <0.909.0> ** Generic server <0.909.0> terminating
r4a-rabbitmq-broker        | ** Last message in was {tcp,#Port<0.49>,<<16,24,0,4,77,81,84,84,4,194,0,60,0,0,0,5,115,107,97,116,97,0,3,98,48,116>>}
r4a-rabbitmq-broker        | ** When Server state == {state,#Port<0.49>,"155.207.33.187:47885 -> 172.18.0.6:1883",true,undefined,false,running,{none,none},<0.908.0>,false,none,{proc_state,#Port<0.49>,#{},{undefined,undefined},{0,nil},{0,nil},undefined,1,undefined,undefined,undefined,{undefined,undefined},undefined,<<"amq.topic">>,{amqp_adapter_info,{172,18,0,6},1883,{155,207,33,187},47885,<<"155.207.33.187:47885 -> 172.18.0.6:1883">>,{'MQTT',"N/A"},[{channels,1},{channel_max,1},{frame_max,0},{client_properties,[{<<"product">>,longstr,<<"MQTT client">>}]},{ssl,false}]},none,undefined,undefined,#Fun<rabbit_mqtt_processor.0.41775190>,{155,207,33,187},#Fun<rabbit_mqtt_util.4.116266148>,#Fun<rabbit_mqtt_util.5.116266148>},undefined,{state,fine,5000,undefined}}
r4a-rabbitmq-broker        | ** Reason for termination == 
r4a-rabbitmq-broker        | ** {{function_clause,[{amqp_gen_connection,terminate,[{{case_clause,{badrpc,{'EXIT',{{badmatch,false},[{rabbit_auth_backend_http,do_http_req,2,[{file,"src/rabbit_auth_backend_http.erl"},{line,131}]},{rabbit_auth_backend_http,http_req,3,[{file,"src/rabbit_auth_backend_http.erl"},{line,118}]},{rabbit_auth_backend_http,bool_req,2,[{file,"src/rabbit_auth_backend_http.erl"},{line,109}]},{rabbit_auth_backend_cache,with_cache,3,[{file,"src/rabbit_auth_backend_cache.erl"},{line,88}]},{rabbit_access_control,check_access,5,[{file,"src/rabbit_access_control.erl"},{line,211}]},{lists,foldl,3,[{file,"lists.erl"},{line,1263}]},{rabbit_direct,connect1,5,[{file,"src/rabbit_direct.erl"},{line,191}]},{rpc,local_call,3,[{file,"rpc.erl"},{line,321}]}]}}}},[{amqp_direct_connection,connect,4,[{file,"src/amqp_direct_connection.erl"},{line,153}]},{amqp_gen_connection,handle_call,3,[{file,"src/amqp_gen_connection.erl"},{line,174}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,661}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,690}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]},{<0.911.0>,{amqp_params_direct,<<"skata">>,<<"ZoKEQAlhG+COwrnFQctOKlFT4pg9Ra3LvhKMW/7bRzPYhoPtdRgds4dAzDp1BM3A">>,<<"/">>,rabbit@2ad4426d1750,{amqp_adapter_info,{172,18,0,6},1883,{155,207,33,187},47885,<<"155.207.33.187:47885 -> 172.18.0.6:1883">>,{'MQTT',"3.1.1"},[{variable_map,#{<<"client_id">> => <<"UPahzyw0tnNofE0FBEpzwA">>}},{channels,1},{channel_max,1},{frame_max,0},{client_properties,[{<<"product">>,longstr,<<"MQTT client">>},{client_id,longstr,<<"UPahzyw0tnNofE0FBEpzwA">>}]},{ssl,false}]},[]}}],[{file,"src/amqp_gen_connection.erl"},{line,242}]},{gen_server,try_terminate,3,[{file,"gen_server.erl"},{line,673}]},{gen_server,terminate,10,[{file,"gen_server.erl"},{line,858}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]},{gen_server,call,[<0.912.0>,connect,60000]}}
r4a-rabbitmq-broker        | 2020-02-16 11:41:10.027 [error] <0.909.0> CRASH REPORT Process <0.909.0> with 0 neighbours exited with reason: {{function_clause,[{amqp_gen_connection,terminate,[{{case_clause,{badrpc,{'EXIT',{{badmatch,false},[{rabbit_auth_backend_http,do_http_req,2,[{file,"src/rabbit_auth_backend_http.erl"},{line,131}]},{rabbit_auth_backend_http,http_req,3,[{file,"src/rabbit_auth_backend_http.erl"},{line,118}]},{rabbit_auth_backend_http,bool_req,2,[{file,"src/rabbit_auth_backend_http.erl"},{line,109}]},{rabbit_auth_backend_cache,with_cache,3,[{file,"src/rabbit_auth_backend_cache.erl"},{line,88}]},{rabbit_access_control,...},...]}}}},...},...],...},...]},...} in gen_server2:terminate/3 line 1183
r4a-rabbitmq-broker        | 2020-02-16 11:41:10.027 [error] <0.907.0> Supervisor {<0.907.0>,rabbit_mqtt_connection_sup} had child rabbit_mqtt_reader started with rabbit_mqtt_reader:start_link(<0.908.0>, {acceptor,{0,0,0,0},1883}) at <0.909.0> exit with reason {{function_clause,[{amqp_gen_connection,terminate,[{{case_clause,{badrpc,{'EXIT',{{badmatch,false},[{rabbit_auth_backend_http,do_http_req,2,[{file,"src/rabbit_auth_backend_http.erl"},{line,131}]},{rabbit_auth_backend_http,http_req,3,[{file,"src/rabbit_auth_backend_http.erl"},{line,118}]},{rabbit_auth_backend_http,bool_req,2,[{file,"src/rabbit_auth_backend_http.erl"},{line,109}]},{rabbit_auth_backend_cache,with_cache,3,[{file,"src/rabbit_auth_backend_cache.erl"},{line,88}]},{rabbit_access_control,...},...]}}}},...},...],...},...]},...} in context child_terminated
r4a-rabbitmq-broker        | 2020-02-16 11:41:10.027 [error] <0.907.0> Supervisor {<0.907.0>,rabbit_mqtt_connection_sup} had child rabbit_mqtt_reader started with rabbit_mqtt_reader:start_link(<0.908.0>, {acceptor,{0,0,0,0},1883}) at <0.909.0> exit with reason reached_max_restart_intensity in context shutdown

As you can see on the second line, the backend is properly called. Again, a deny response in body does not produce the crash and the user is not authenticated as expected.

Any updates on the subject?

[lukebakken]

rabbit_http_util,quote_plus,[{error,einval} suggests that an invalid URL is being used to contact your HTTP auth server.

[klpanagi]

Dear @lukebakken the URL seems to be correct and the /auth/user endpoint of the HTTP auth server is been properly called. Furthermore, AMQP connections work properly with the current setup of the HTTP auth server.

UPDATE
I also did try with the django example backend and still getting exception logs. As you can see below, the endpoint is properly called and the response is an HTTP 200 OK status with a body string "allow" (as described here).

INFO <QueryDict: {'username': ['skata'], 'password': ['skata'], 'vhost': ['/']}>                                                                                                                                      
INFO "GET /auth/user?username=skata&password=skata&vhost=%2F HTTP/1.1" 200 18                                                                                                                                         
2019-11-28 21:32:04.993 [error] <0.706.0> STOMP error frame sent:                                                                                                                                                     
Message: "Processing error"                                                                                                                                                                                           
Detail: "Processing error" 

I did play around with parameters and responses and noticed that the exception happens when sending an "allow" string in the response body. If the body is set to "deny" then the behaviour is as expected (as specified here), without causing any exception.

INFO <QueryDict: {'username': ['skata'], 'password': ['skata'], 'vhost': ['/']}>
INFO "GET /auth/user?username=skata&password=skata&vhost=%2F HTTP/1.1" 200 4
2019-11-28 21:40:03.026 [warning] <0.706.0> STOMP login failed for user "skata"
2019-11-28 21:40:03.026 [error] <0.706.0> STOMP error frame sent:
Message: "Bad CONNECT"
Detail: "Access refused for user 'skata'\n"
Server private detail: none

[klpanagi]

This exception is seems to be caused when the reverse_dns_lookups config is set to true.

@lukebakken Maybe you should include this as a Warning in the documentation of the plugin (or the rabbitmq-auth-backend-http).

[michaelklishin]

@klpanagi can you put together an executable example we can use to reproduce? E.g. using Docker Compose?

[klpanagi]

Hey @michaelklishin !
Of course, I am going to prepare a containerized example (via docker-compose) in the next few days and come back to you when ready.

[lukebakken]

@klpanagi thanks. If we can reproduce it, it will be very easy to fix.

[michaelklishin]

There are no updates and likely won't be for a while. This issue is not observed commonly enough to become a priority.

RabbitMQ is open source software, however, and users who hit less common issues are welcome to contribute a solution for the problems they face.

According to the stack trace, MQTT connection closure hits an exception because this line in the HTTP authN/authZ backend has a badmatch, a false is returned where a tuple with a hostname extracted from the URI is expected.

Which to me suggests that something is off with the URI used. I see no evidence of reverse DNS lookups being relevant here. I do not see how a deny response could be relevant here beyond the fact that for denied requests, certain code paths would never be executed (e.g. the connection termination won't have to close an underlying "internal" connection MQTT plugin maintains since there never would be one).

You are welcome to introduce additional logging to the function in question to see what the URI is. A plugin built from source can be dropped in place of the one that ships with RabbitMQ if there's no easy way to run the test against a node running from source.

[klpanagi]

I think it is about time to learn Erlang!...
Thank you @michaelklishin for your time.

[lukebakken]

@klpanagi - I don't have access to Docker, but I did add some logging to the function that crashes in this branch and produced a plugin archive from that branch:

rabbitmq_auth_backend_http-3.8.0+rc.1.14.gebd3632.zip

Here's how to use it:

• Rename .zip to .ez
• In your Docker image after it starts up but before RabbitMQ starts, copy the file to the current location of rabbitmq_auth_backend_http-*.ez and remove the file that exists there.
• Enable the plugin as usual in the Docker image
• Re-run your test case to trigger the error. There will be a couple new [info] lines logged
• Attach the log file here

[eflukx]

Similar problem here. Written a simple http auth server.
When using the auth backend for logging in to the management UI all works as expected, when using an MQTT client however, the server crashes with a similar log output:

amqp_gen_connection,terminate,[{{case_clause,{badrpc,{'EXIT',{{case_clause,"\"allow\""}
(full log output)

@lukebakken
Tried running with your plugin version, log output here.

edit add AMQP case:

Tried with an AMQP client as well, also results in a (different) error:

2020-04-21 23:53:53.861 [info] <0.692.0> accepting AMQP connection <0.692.0> (10.135.10.147:49550 -> 10.135.10.6:5672)
2020-04-21 23:53:53.865 [info] <0.692.0> rabbit_auth_backend_http:do_http_req Path0 "http://10.30.10.54:6980/api/v1/user"
2020-04-21 23:53:53.866 [info] <0.692.0> rabbit_auth_backend_http:do_http_req URI [{fragment,[]},{host,"10.30.10.54"},{path,"/api/v1/user"},{port,6980},{query,[]},{scheme,"http"},{userinfo,[]}]
2020-04-21 23:53:53.997 [info] <0.692.0> rabbit_auth_backend_http:do_http_req Path0 "http://10.30.10.54:6980/api/v1/vhost"
2020-04-21 23:53:53.998 [info] <0.692.0> rabbit_auth_backend_http:do_http_req URI [{fragment,[]},{host,"10.30.10.54"},{path,"/api/v1/vhost"},{port,6980},{query,[]},{scheme,"http"},{userinfo,[]}]
2020-04-21 23:53:57.236 [error] <0.692.0> closing AMQP connection <0.692.0> (10.135.10.147:49550 -> 10.135.10.6:5672):
{handshake_error,opening,0,{error,{case_clause,"\"allow\""},'connection.open',[{rabbit_access_control,check_access,5,[{file,"src/rabbit_access_control.erl"},{line,211}]},{lists,foldl,3,[{file,"lists.erl"},{line,1263}]},{rabbit_reader,handle_method0,2,[{file,"src/rabbit_reader.erl"},{line,1231}]},{rabbit_reader,handle_method0,3,[{file,"src/rabbit_reader.erl"},{line,1135}]},{rabbit_reader,handle_input,3,[{file,"src/rabbit_reader.erl"},{line,1049}]},{rabbit_reader,recvloop,4,[{file,"src/rabbit_reader.erl"},{line,489}]},{rabbit_reader,run,1,[{file,"src/rabbit_reader.erl"},{line,471}]},{rabbit_reader,start_connection,4,[{file,"src/rabbit_reader.erl"},{line,370}]}]}}

[michaelklishin]

This function expected a true or false but got an "allow" (with quotes) instead from one of the functions it depends on. Consider sharing your server's code or at least double check that the returned value does not have any extra quotes.

A traffic capture would help immensely as well. Without knowing what your server returns we can only guess what may be going on and guessing is very time consuming.

[michaelklishin]

At least as of 3.8.3 (rabbitmq/rabbitmq-auth-backend-http@10491ca), rabbitmq-auth-backend-http does log backing server's response code and body at debug level.

There are also several example services available in Django, Spring Boot, PHP and ASP.NET MVC. Given how small the expected API is it should be reasonably easy to find the delta between your service and those examples.

[eflukx]

First of all, @michaelklishin thanks very much for your quick response on this. Very much appreciated!

I think your analysis is right (will verify later today), I'm sending quoted repsonses over the line. The RabbitMQ http auth is part of a bigger API, which is JSON based. I even fondled with the Content-Type header of this endpoint as I suspected this might be causing problems. It seems the API framework (Ruby Grape) is JSON-ifying my string output (by quoting it). Silly mistake, maybe the sheer simplicity of the API made it easier for me to gloss over such thing... (sigh) I even noticed the \-escaped quotes in the Erlang back trace, but thought of it as a rendering/representation quirk.

What also confused me: the management UI login worked fine with my API. Probably other code get hit internally in RabbitMQ for Authenticating users to the management UI.

Will confirm -most probably- a working solution later today!

As a final note: IMHO the HTTP-auth plugin should not panic on a malformed input from an external API. Maybe the title of the issue can be changed to "HTTP-Auth-Backend throws exception on malformed input" (and reopened with medium/minor importance)

Ps. I'm on 3.8.3, so good to know about the improved logging facilities.

[eflukx]

Fixed formatter, can confirm it works like a charm now. Thanks for your help!
As said above, please consider improving the handling of malformed API responses. 😄

[klpanagi]

The problem with MQTT was caused due to malformed URL of one of the authorization endpoints.
Though, the initial issue with STOMP + reverse_dns_lookups still is unresolved.
I will agree with @eflukx about improving error reporting.

[lukebakken]

@klpanagi -

Though, the initial issue with STOMP + reverse_dns_lookups still is unresolved.

I tried reproducing your issue outside of docker but couldn't. My guess is that reverse DNS lookups do not work correctly in your docker environment. You can verify by using dig or nslookup to do a reverse DNS lookup inside of docker to see what is returned.

[michaelklishin]

We can make rabbitmq-auth-backend-http deny all operations for which it could not issue a request or did not receive an expected response. It would not be meaningfully different in practice to your applications, however: things will still fail and will still require troubleshooting by inspecting logs and network traffic. Applications would be able to keep their connections open, though, although that's of questionable value if every operation invoked on said connection would be guaranteed to fail if unless the root cause is addressed.

Where I'd focus is improving debug logging in that plugin as it is a really common source of questions, nearly all of which come down to "the plugin could not contact your service or your service does not produce an acceptable response".