3.12.x: AMQP 1.0 connection authentication fails with an external HTTP auth service about rabbitmq-server

[lukebakken]

@motmot80 - Everything works fine when I use my test project, which is what I used to reproduce the problem last time.

• Are you using the cache backend as is indicated in this comment
• Could you please attach your complete RabbitMQ configuration file?

[lukebakken]

@motmot80 everything works fine when I use the cache backend -

https://github.com/lukebakken/rabbitmq-server-9031/blob/main/rabbitmq.conf

We will need more information! If you could provide REST-Mock and RabbitMQ debug logs that would be great.

[lukebakken]

I also confirmed the following, using my test project:

• RMQ 3.11.10 - fails
• RMQ 3.11.21 - works
• RMQ 3.13.2 - works

[motmot80]

@lukebakken I am on it!

[motmot80]

Sorry for the delay. I needed to remove project specific information.

Here are the k8s cluster definitions. As you can see the only difference is changing the RMQ image version.
3.11 is working. 3.12 not.

k8s-rmq-cluster-definition-3-12-fail.yml.txt
k8s-rmq-cluster-definition-3-11-ok.yml.txt

[MarcialRosales]

@motmot80
I cannot reproduce it against 3.12.13 docker image (more specifically rabbitmq:3.12.13-management). Can you try that image? and also, could you please send us your logs after enabling debug level?

[michaelklishin]

@MarcialRosales @lukebakken note that the most recent 3.12 release is 3.12.14.

[lukebakken]

Yep, I have been testing using the rabbitmq:3.12-management image, which is 3.12.14

[MarcialRosales]

@michaelklishin I have even tested against v3.12.x and v3.13.x branches. I picked 3.12.13 because that is what the customer is using.. well, not exactly, they are using rabbitmq:3.12.13-debian-12-r2-main.. so i suggest them to use 3.12.13.

[motmot80]

@lukebakken

https://hub.docker.com/layers/bitnami/rabbitmq/3.12.13-debian-12-r2/images/sha256-87c42f599e2319fcf510b32e18a266668102f246d5fddd3d9e1c219d8067a2b1?context=explore

is the image used in the example.

We will change the image to 3.12.14-management on an isolated environment and extract the debug logs.

https://hub.docker.com/layers/library/rabbitmq/3.12.14-management/images/sha256-8f3b358689236940d351710e396da89555f3a5c646bb46a00f9d71b9d4fede7f?context=explore

But I think that will take until Tuesday, May 28th.

[motmot80]

@lukebakken

Good news - as you already found out - the issue does not occur on the 3.12.14 RabbitMQ from RabbitMQ.
We tested several setups by changing just the docker image.

We were probably very unlucky that exactly this error pattern reappeared in 3.12.+-bitnami debian version.

Because in this case it's related to the base image and not RabbitMQ we can close the issue.
So I think we need to change to image base.

Maybe vmware(tanzu/pivotal) can interally address this issue to the vmware(bitnami) colleagues.

vmware/bitnami 3.11.21-debian-11-r0

OK

2024-05-27 17:27:13.394747+00:00 [info] <0.2126.0> accepting AMQP connection <0.2126.0> (127.0.0.1:33584 -> 127.0.0.1:5672)
2024-05-27 17:27:13.547123+00:00 [debug] <0.2126.0> User 'TESTUSER' failed authentication by backend rabbit_auth_backend_internal
2024-05-27 17:27:13.547440+00:00 [debug] <0.2126.0> auth_backend_http: GET https://AUTHSERVICE/auth/user?username=USER&password=PWD
2024-05-27 17:27:13.547541+00:00 [debug] <0.2126.0> auth_backend_http: request timeout: 15000, connection timeout: 15000
2024-05-27 17:27:13.708774+00:00 [debug] <0.2126.0> auth_backend_http: response code is 200, body: "allow"
2024-05-27 17:27:13.708909+00:00 [debug] <0.2126.0> User 'TESTUSER' authenticated successfully by backend rabbit_auth_backend_cache
2024-05-27 17:27:13.709293+00:00 [debug] <0.2126.0> auth_backend_http: GET https://AUTHSERVICE/auth/user?username=USER&password=PWD
2024-05-27 17:27:13.709390+00:00 [debug] <0.2126.0> auth_backend_http: request timeout: 15000, connection timeout: 15000
2024-05-27 17:27:13.791276+00:00 [debug] <0.2126.0> auth_backend_http: response code is 200, body: "allow"
2024-05-27 17:27:13.791475+00:00 [info] <0.2126.0> AMQP 1.0 connection <0.2126.0>: user 'TESTUSER' authenticated

vmware/bitnami 3.12.13-debian-12-r2

NOT OK

2024-05-27 17:38:54.347834+00:00 [info] <0.1111.0> accepting AMQP connection <0.1111.0> (127.0.0.1:43424 -> 127.0.0.1:5672)
2024-05-27 17:38:54.522839+00:00 [debug] <0.1111.0> User 'TESTUSER' failed authentication by backend rabbit_auth_backend_internal
2024-05-27 17:38:54.523212+00:00 [debug] <0.1111.0> auth_backend_http: GET https://AUTHSERVICE/auth/user?username=USER&password=PWD
2024-05-27 17:38:54.523378+00:00 [debug] <0.1111.0> auth_backend_http: request timeout: 15000, connection timeout: 15000
2024-05-27 17:39:00.525899+00:00 [error] <0.1111.0> closing AMQP connection <0.1111.0> (127.0.0.1:43424 -> 127.0.0.1:5672):
2024-05-27 17:39:00.525899+00:00 [error] <0.1111.0> {handshake_error,starting,0,
2024-05-27 17:39:00.525899+00:00 [error] <0.1111.0>                  {'v1_0.error',{symbol,<<"amqp:unauthorized-access">>},
2024-05-27 17:39:00.525899+00:00 [error] <0.1111.0>                                {utf8,<<"PLAIN login refused: rabbit_auth_backend_cache failed authenticating TESTUSER: {failed_connect,\n                                                                                                       [{to_address,\n                                                                                                         {\"AUTHSERVICE\",\n                                                                                                          443}},\n                                                                                                        {inet,\n                                                                                                         [inet],\n                                                                                                         {option,\n                                                                                                          server_only,\n                                                                                                          fail_if_no_peer_cert}}]}">>},
2024-05-27 17:39:00.525899+00:00 [error] <0.1111.0>                                undefined}}

vmware/bitnami 3.12.14-debian-12-r1

NOT OK

2024-05-27 18:16:34.323787+00:00 [info] <0.1418.0> accepting AMQP connection <0.1418.0> (127.0.0.1:53436 -> 127.0.0.1:5672)
2024-05-27 18:16:34.496925+00:00 [debug] <0.1418.0> User 'TESTUSER' failed authentication by backend rabbit_auth_backend_internal
2024-05-27 18:16:34.497238+00:00 [debug] <0.1418.0> auth_backend_http: GET https://AUTHSERVICE/auth/user?username=USER&password=PWD
2024-05-27 18:16:34.497348+00:00 [debug] <0.1418.0> auth_backend_http: request timeout: 15000, connection timeout: 15000
2024-05-27 18:16:37.467018+00:00 [debug] <0.1422.0> accepting AMQP connection <0.1422.0> (127.0.0.6:51729 -> 192.168.2.56:5672)
2024-05-27 18:16:37.467182+00:00 [debug] <0.1422.0> closing AMQP connection <0.1422.0> (127.0.0.6:51729 -> 192.168.2.56:5672):
2024-05-27 18:16:37.467182+00:00 [debug] <0.1422.0> connection_closed_with_no_data_received
2024-05-27 18:16:40.499074+00:00 [error] <0.1418.0> closing AMQP connection <0.1418.0> (127.0.0.1:53436 -> 127.0.0.1:5672):
2024-05-27 18:16:40.499074+00:00 [error] <0.1418.0> {handshake_error,starting,0,
2024-05-27 18:16:40.499074+00:00 [error] <0.1418.0>                  {'v1_0.error',{symbol,<<"amqp:unauthorized-access">>},
2024-05-27 18:16:40.499074+00:00 [error] <0.1418.0>                                {utf8,<<"PLAIN login refused: rabbit_auth_backend_cache failed authenticating TESTUSER: {failed_connect,\n                                                                                                       [{to_address,\n                                                                                                         {\"AUTHSERVICE\",\n                                                                                                          443}},\n                                                                                                        {inet,\n                                                                                                         [inet],\n                                                                                                         {option,\n                                                                                                          server_only,\n                                                                                                          fail_if_no_peer_cert}}]}">>},
2024-05-27 18:16:40.499074+00:00 [error] <0.1418.0>                                undefined}}

vmware/rabbitmq 3.12.14-management

OK

2024-05-27 17:27:13.394747+00:00 [info] <0.2126.0> accepting AMQP connection <0.2126.0> (127.0.0.1:33584 -> 127.0.0.1:5672)
2024-05-27 17:27:13.547123+00:00 [debug] <0.2126.0> User 'TESTUSER' failed authentication by backend rabbit_auth_backend_internal
2024-05-27 17:27:13.547440+00:00 [debug] <0.2126.0> auth_backend_http: GET https://AUTHSERVICE/auth/user?username=USER&password=PWD
2024-05-27 17:27:13.547541+00:00 [debug] <0.2126.0> auth_backend_http: request timeout: 15000, connection timeout: 15000
2024-05-27 17:27:13.708774+00:00 [debug] <0.2126.0> auth_backend_http: response code is 200, body: "allow"
2024-05-27 17:27:13.708909+00:00 [debug] <0.2126.0> User 'TESTUSER' authenticated successfully by backend rabbit_auth_backend_cache
2024-05-27 17:27:13.709293+00:00 [debug] <0.2126.0> auth_backend_http: GET https://AUTHSERVICE/auth/user?username=USER&password=PWD
2024-05-27 17:27:13.709390+00:00 [debug] <0.2126.0> auth_backend_http: request timeout: 15000, connection timeout: 15000
2024-05-27 17:27:13.791276+00:00 [debug] <0.2126.0> auth_backend_http: response code is 200, body: "allow"
2024-05-27 17:27:13.791475+00:00 [info] <0.2126.0> AMQP 1.0 connection <0.2126.0>: user 'TESTUSER' authenticated

Thank you very much for your support.

Best regards
Thomas