cloudamqp / amqproxy

An intelligent AMQP proxy, with connection and channel pooling/reusing

AMQProxy

An intelligent AMQP proxy with AMQP connection and channel pooling/reusing. Allows e.g. PHP clients to keep long lived connections to upstream servers, increasing publishing speed with a magnitude or more.

In the AMQP protocol, if you open a connection the client and the server has to exchange 7 TCP packages. If you then want to publish a message you have to open a channel which requires 2 more, and then to do the publish you need at least one more, and then to gracefully close the connection you need 4 more packages. In total 15 TCP packages, or 18 if you use AMQPS (TLS). For clients that can't for whatever reason keep long-lived connections to the server this has a considerable latency impact.

This proxy server, if run on the same machine as the client can save all that latency. When a connection is made to the proxy the proxy opens a connection to the upstream server, using the credentials the client provided. AMQP traffic is then forwarded between the client and the server but when the client disconnects the proxy intercepts the Channel Close command and instead keeps it open on the upstream server (if deemed safe). Next time a client connects (with the same credentials) the connection to the upstream server is reused so no TCP packages for opening and negotiating the AMQP connection or opening and waiting for the channel to be opened has to be made.

Only "safe" channels are reused, that is channels where only Basic Publish or Basic Get (with no_ack) has occurred. Any channels who has subscribed to a queue will be closed when the client disconnects. However, the connection to the upstream AMQP server are always kept open and can be reused.

In our benchmarks publishing one message per connection to a server (using TLS) with a round-trip latency of 50ms, takes on avarage 10ms using the proxy and 500ms without. You can read more about the proxy here Maintaining long-lived connections with AMQProxy

As of version 2.0.0 connections to the server can be shared by multiple client connections. When a client opens a channel it will get a channel on a shared upstream connection, the proxy will remap the channel numbers between the two. Many client connections can therefor share a single upstream connection. The benefit is that way fewer connections are needed to the upstream server. For instance, establihsing 10.000 connections after a server reboot might normally take several minutes, but with this proxy it can happen in seconds.

Installation

Debian/Ubuntu

Packages are available at Packagecloud. Install the latest version with:

curl -fsSL https://packagecloud.io/cloudamqp/amqproxy/gpgkey | gpg --dearmor | sudo tee /usr/share/keyrings/amqproxy.gpg > /dev/null
. /etc/os-release
echo "deb [signed-by=/usr/share/keyrings/amqproxy.gpg] https://packagecloud.io/cloudamqp/amqproxy/$ID $VERSION_CODENAME main" | sudo tee /etc/apt/sources.list.d/amqproxy.list
sudo apt-get update
sudo apt-get install amqproxy

If you need to install a specific version, do so using the following command: sudo apt install lavinmq=<version>. This works for both upgrades and downgrades.

Docker/Podman

Docker images are published at Docker Hub. Fetch and run the latest version with:

docker run --rm -it -p 5673:5673 cloudamqp/amqproxy amqp://SERVER:5672

Note: If you are running the upstream server on localhost then you will have to add the --network host flag to the docker run command.

Then from your AMQP client connect to localhost:5673, it will resuse connections made to the upstream. The AMQP_URL should only include protocol, hostname and port (only if non default, 5672 for AMQP and 5671 for AMQPS). Any username, password or vhost will be ignored, and it's up to the client to provide them.

Installation (from source)

Install Crystal

shards build --release --production
cp bin/amqproxy /usr/bin
cp extras/amqproxy.service /etc/systemd/system/
systemctl enable amqproxy
systemctl start amqproxy

You probably want to modify /etc/systemd/system/amqproxy.service and configure another upstream host.

Configuration

Available settings

Setting	Description	Command line	Environment variable	Config file setting	Default value
Listen address	Address to listen on. This is the hostname/IP address which will be the target of clients	--listen / -l	LISTEN_ADDRESS	[listen] > address or [listen] > bind	localhost
Listen port	Port to listen on. This is the port which will be the target of clients	--port / -p	LISTEN_PORT	[listen] > port	5673
Log level	Controls log verbosity. Available levels (see 84codes/logger.cr): - DEBUG: Low-level information for developers - INFO: Generic (useful) information about system operation - WARN: Warnings - ERROR: Handleable error conditions - FATAL: Unhandleable errors that results in a program crash	--debug / -d: Sets the level to DEBUG	-	[main] > log_level	INFO
Idle connection timeout	Maximum time in seconds an unused pooled connection stays open	--idle-connection-timeout / -t	IDLE_CONNECTION_TIMEOUT	[main] > idle_connection_timeout	5
Upstream	AMQP URL that points to the upstream RabbitMQ server to which the proxy should connect to. May only contain scheme, host & port (optional). Example: amqps://rabbitmq.example.com	Pass as argument after all options	AMQP_URL	[main] > upstream

How to configure

There are three ways to configure the AMQProxy.

• Environment variables
• Passing options & argument via the command line
  • Usage: amqproxy [options] [amqp upstream url]
  • Additional options, that are not mentioned in the table above:
    • --config / -c: Load config file at given path
    • --help / -h: Shows help
    • --version / -v: Displays AMQProxy version
• Config file
  • You can find an example at config/example.ini

Precedence

1. Config file
2. Command line options & argument
3. Environment variables

Settings that are avilable in the config file will override the corresponding command line options. A command line option will override the corresponding environment variable. And so on. The different configuration approaches can also be mixed.

amqproxy's Issues

Not reuse connection between clients.

I faced with problem that AMQProxy not save connections between different clients request. In RabbitMq server`=INFO REPORT==== 3-Mar-2021::18:36:23 ===
accepting AMQP connection <0.1626.0> (192.168.240.2:39874 -> 192.168.240.8:5672)

=INFO REPORT==== 3-Mar-2021::18:36:23 ===
connection <0.1626.0> (192.168.240.2:39874 -> 192.168.240.8:5672): user 'guest' authenticated and granted access to vhost '/'

=INFO REPORT==== 3-Mar-2021::18:36:25 ===
closing AMQP connection <0.1626.0> (192.168.240.2:39874 -> 192.168.240.8:5672, vhost: '/', user: 'guest')` I just see that on every client request create and close new connection instead of reuse previous one. Any thought about that?

Help, my connections are not being reused.

Hello,

My application is running on the php-amqplib library. I start my amqproxy instance via the command below in ubuntu terminal.

amqproxy -l 127.0.0.1 -p 5674 amqp://127.0.0.1 -d

my code in php establishes the connection like this,

$this->connection = new AMQPStreamConnection($connection, 5674, $this->user, $this->password);

I then proceed to create 2 instances of the code above on 2 separate terminal sessions.
Upon viewing the management UI connection tab, I see 2 connections. I am expecting 1 only.

Please help.

OpenSSL VerifyMode

I'm facing a problem that amqproxy is falling when verifying the upstream certificate

#<OpenSSL::SSL::Error:SSL_connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed>

is there a way to go around this? (disabling the verification would work well in my case)

INFO log level too verbose

With 10000 non-persistent connections connecting and disconnecting all the time, there is too much noise in the logs. I understand this approach is to replicate RabbitMQ logs behaviour but this tool is mainly for producers/consumers that can't handle long lived connections.

Install Crystal link has changed

Hi, Install Crystal link in readme was broken. It had changed from https://crystal-lang.org/reference/installation/ to https://crystal-lang.org/install/

docker build failed: Error: Errno is not a subclass of Exception

docker build -t amqproxy:v0.4.4 https://github.com/cloudamqp/amqproxy.git#v0.4.4
Sending build context to Docker daemon  185.3kB
Step 1/8 : FROM crystallang/crystal
latest: Pulling from crystallang/crystal
5bed26d33875: Pull complete 
f11b29a9c730: Pull complete 
930bda195c84: Pull complete 
78bf9a5ad49e: Pull complete 
e27d61588c18: Pull complete 
Digest: sha256:b3f86e938a3de7454d466736c013cf25ed37e378f64bb0583555627b6d2ea90f
Status: Downloaded newer image for crystallang/crystal:latest
 ---> 89062caa0bb5
Step 2/8 : WORKDIR /app
 ---> Running in b3153b37c2be
Removing intermediate container b3153b37c2be
 ---> f6afa181fbf2
Step 3/8 : COPY . .
 ---> 4404ad8c69d8
Step 4/8 : RUN shards build --release --production
 ---> Running in ba4ee679d3f9
Resolving dependencies
Fetching https://github.com/cloudamqp/amq-protocol.cr.git
Installing amq-protocol (0.3.4)
Building: amqproxy
Error target amqproxy failed to compile:
Showing last frame. Use --error-trace for full trace.

In src/amqproxy/upstream.cr:118:17

 118 | rescue ex : Errno | IO::Error | OpenSSL::SSL::Error
                   ^----
Error: Errno is not a subclass of Exception

The command '/bin/sh -c shards build --release --production' returned a non-zero code: 1

I have tried multiple releases, master - same error.

AMQPLAIN not supported

I am getting from Celery(4.2.1)/Kombu(4.2.1)/PyAMQP(2.3.2):
ConnectionError: Couldn't find appropriate auth mechanism (can offer: AMQPLAIN; available: PLAIN) but it looks like AMQPLAIN should be supported

amqproxy/src/amqproxy/client.cr

Line 67 in be42d09

when "AMQPLAIN"

From AMQProxy side:

Unhandled exception in spawn: AMQ::Protocol::Frame::Body-end was 10, expected 206 (AMQ::Protocol::Error::InvalidFrameEnd)
  from src/amqproxy/client.cr:42:15 in '->'
  from /usr/share/crystal/src/fiber.cr:255:3 in '???'
  from ???

Heartbeat from proxy to rabbitmq server

We are try to use amqproxy in our php stack to avoid performance lack and avoid of ghost connections on rabbitmq server.
But if we have the network issue and rabbitmq is not reachable for proxy then there would be ghost connections from our consumers. That is caused impossibility to send heartbeat by proxy to rabbitmq.
Enabling possibility to send heartbeat to amqp proxy will solve this issue.
Thanks

Not able to recreate upstream connection

I'm running the proxy on a docker container and found that if the upstream rabbitmq cluster closes the connection or is not available, the proxy is not able to reconnect. The client connection is constantly being accepted and closed while the proxy errors trying to read from the upstream.

Is there something I'm doing wrong here?

See the log below, thanks

Proxy upstream: rabbitmq:5672
Proxy listening on [::]:5673
Client connection accepted from [::ffff:172.23.0.18]:38426
Connected to upstream 172.23.0.2:5672
Client connection closed from [::ffff:172.23.0.18]:38426
Client connection accepted from [::ffff:172.23.0.18]:38434
Client connection accepted from [::ffff:172.23.0.16]:56310
Connected to upstream 172.23.0.2:5672
Client connection accepted from [::ffff:172.23.0.18]:38468
Connected to upstream 172.23.0.2:5672
Client connection accepted from [::ffff:172.23.0.16]:56338
Connected to upstream 172.23.0.2:5672
Client connection accepted from [::ffff:172.23.0.16]:56368
Connected to upstream 172.23.0.2:5672
Client connection accepted from [::ffff:172.23.0.18]:38510
Connected to upstream 172.23.0.2:5672
Unhandled exception in spawn:
cast from AMQProxy::AMQP::Connection::Close to AMQProxy::AMQP::Channel::CloseOk failed, at /var/amqpproxy/src/amqproxy/upstream.cr:101:22:101 (TypeCastError)
Client connection closed from [::ffff:172.23.0.18]:38434
from var/amqpproxy/src/amqproxy/upstream.cr:100:11 in '->'
from var/amqpproxy/src/amqproxy/pool.cr:255:3 in 'handle_connection'
from var/amqpproxy/src/amqproxy/server.cr:41:13 in '->'
from usr/share/crystal/src/fiber.cr:255:3 in '???'
from ???
Error reading from upstream: #<Errno:Error reading socket: Connection reset by peer>
Unhandled exception in spawn:
cast from AMQProxy::AMQP::Connection::Close to AMQProxy::AMQP::Channel::CloseOk failed, at /var/amqpproxy/src/amqproxy/upstream.cr:101:22:101 (TypeCastError)
Client connection closed from [::ffff:172.23.0.16]:56310
from var/amqpproxy/src/amqproxy/upstream.cr:100:11 in '->'
from var/amqpproxy/src/amqproxy/pool.cr:255:3 in 'handle_connection'
from var/amqpproxy/src/amqproxy/server.cr:41:13 in '->'
from usr/share/crystal/src/fiber.cr:255:3 in '???'
from ???
Error reading from upstream: #<Errno:Error reading socket: Connection reset by peer>
Error reading from upstream: #<Errno:Error reading socket: Connection reset by peer>
Client connection closed from [::ffff:172.23.0.18]:38468
Error reading from upstream: #<Errno:Error reading socket: Connection reset by peer>
Client connection closed from [::ffff:172.23.0.16]:56338
Error reading from upstream: #<Errno:Error reading socket: Connection reset by peer>
Client connection closed from [::ffff:172.23.0.16]:56368
Error reading from upstream: #<Errno:Error reading socket: Connection reset by peer>
Client connection closed from [::ffff:172.23.0.18]:38510
Client connection accepted from [::ffff:172.23.0.18]:38520
Client connection closed from [::ffff:172.23.0.18]:38520
Client connection accepted from [::ffff:172.23.0.18]:38524
Client connection closed from [::ffff:172.23.0.18]:38524
Client connection accepted from [::ffff:172.23.0.18]:38536
Client connection closed from [::ffff:172.23.0.18]:38536
Client connection accepted from [::ffff:172.23.0.16]:56406
Client connection closed from [::ffff:172.23.0.16]:56406
Client connection accepted from [::ffff:172.23.0.18]:38546
Client connection closed from [::ffff:172.23.0.18]:38546
Client connection accepted from [::ffff:172.23.0.16]:56418
Client connection closed from [::ffff:172.23.0.16]:56418
Client connection accepted from [::ffff:172.23.0.18]:38558
Error sending to upstream: #<Errno:Error writing to socket: Broken pipe>
Client connection closed from [::ffff:172.23.0.18]:38558
Client connection accepted from [::ffff:172.23.0.18]:38564
Error sending to upstream: #<Errno:Error writing to socket: Broken pipe>
Client connection closed from [::ffff:172.23.0.18]:38564
Client connection accepted from [::ffff:172.23.0.16]:56432
Error sending to upstream: #<Errno:Error writing to socket: Broken pipe>
Client connection closed from [::ffff:172.23.0.16]:56432
Client connection accepted from [::ffff:172.23.0.18]:38574
Error sending to upstream: #<Errno:Error writing to socket: Broken pipe>
Client connection closed from [::ffff:172.23.0.18]:38574
Client connection accepted from [::ffff:172.23.0.18]:38580
Error sending to upstream: #<Errno:Error writing to socket: Broken pipe>
Client connection closed from [::ffff:172.23.0.18]:38580
Client connection accepted from [::ffff:172.23.0.18]:38584
Error sending to upstream: #<Errno:Error writing to socket: Broken pipe>
Client connection closed from [::ffff:172.23.0.18]:38584
Client connection accepted from [::ffff:172.23.0.18]:38588
Error sending to upstream: #<Errno:Error writing to socket: Broken pipe>
Client connection closed from [::ffff:172.23.0.18]:38588
Client connection accepted from [::ffff:172.23.0.18]:38592
Error sending to upstream: #<Errno:Error writing to socket: Broken pipe>

seconds out of range

I'm having the following log error when a php client connects (php-amqplib), and not sure the reason

Connected to upstream x.x.x.x:5672
Unhandled exception in spawn: Invalid time: seconds out of range (ArgumentError)
  from ???
  from ???
  from ???
  from ???
  from ???
  from ???
Connected to upstream x.x.x.x:5672
Unhandled exception in spawn: Invalid time: seconds out of range (ArgumentError)
  from ???
  from ???
  from ???
  from ???
  from ???
  from ???

AMQProxy keeps opening new connections

Since two weeks ago, we experience a problem where the connections opened by our amqproxy instance won't be closed, resulting in more and more open connections, up to the point where our CloudAMQP instance runs out of memory, effectively shutting our application down:

The sharp drops are us deploying a new amqproxy version to Kubernetes, killing all connections during deployment (we're now on 0.4.0).

At this point, I'm not sure what exactly is happening. I suspect it's an issue with one of our services, but I've traced down every single commit in the relevant time frame and I couldn't find anything related to the lower-level queue components.
Our setup looks like the following:

CloudAMQP <----> Google Kubernetes
                        |
                     amqproxy
                        |
            +-----+-----+-----+-----+
          srv1   srv2  srv3  srv4  srvN

The services all use the PHP AMQP Library with a super lightweight abstraction on top. After we've noticed the problem, we even implemented a shutdown handler to explicitly close the connection under all circumstances, but to no avail.

The connections all look like the following in the RabbitMQ manager:

Virtual host	Name	Node	State	SSL / TLS	Protocol	Channels	Channel max	Client	From Client	To Client	Heartbeat	Connected at
XXX	35.xxx.xxx.xxx:2113	rabbit@XXX-01	running	○	AMQP 0-9-1	0	0	AMQProxy0.4.0	0iB/s	0iB/s		22:06:372019-09-03
XXX	35.xxx.xxx.xxx:32902	rabbit@XXX-01	running	○	AMQP 0-9-1	0	0	AMQProxy0.4.0	0iB/s	0iB/s		08:37:022019-09-04
XXX	35.xxx.xxx.xxx:33072	rabbit@XXX-01	running	○	AMQP 0-9-1	0	0	AMQProxy0.4.0	0iB/s	0iB/s		02:28:202019-09-04
XXX	35.xxx.xxx.xxx:33196	rabbit@XXX-02	running	○	AMQP 0-9-1	0	0	AMQProxy0.4.0	0iB/s	0iB/s		00:38:142019-09-04
XXX	35.xxx.xxx.xxx:33248	rabbit@XXX-01	running	○	AMQP 0-9-1	0	0	AMQProxy0.4.0	0iB/s	0iB/s		03:26:462019-09-04
XXX	35.xxx.xxx.xxx:33528	rabbit@XXX-01	running	○	AMQP 0-9-1	0	0	AMQProxy0.4.0	0iB/s	0iB/s		17:20:152019-09-03

The AMQProxy logs contain lots and lots of entries like the following:

Error reading from upstream: #<Errno:Error reading socket: Bad file descriptor>

Is there anything we can do to prevent this excessive connection spawning?

Connection not kept open?

I'm experimenting with amqproxy a bit (using a kubernetes cluster). I have:

• a pod running a rabbitmq 3.8.13 with no tuning, just a test user and test vhost
• a pod with two co-located containers:
  • one with nodejs
  • one with amqproxy 0.5.3 from official docker image started with --debug amqp://rabbit:5672

Then on the nodejs container I run this very simple script:

require('amqplib')
  .connect('amqp://test:test@localhost:5673/test')  // that's the amqproxy url
  .catch(console.error);

And I can see on the rabbitmq logs that the connection was successful (so far so good). After a few seconds I stop the previous script, but as soon as I do this the connection is closed on rabbitmq side.

I expected the amqproxy to keep the connection open at least for a few seconds, but it is immediately closed. Was my assumption wrong ? It looks like #46 but if my assumption is correct I'm willing to get to the bottom of this so I started a fresh one. I will provide a tcpdump soon if that helps.

Relevant logs:

2021-03-31 11:30:17.366 [info] <0.17676.9> accepting AMQP connection <0.17676.9> (10.233.98.89:48262 -> 10.233.125.85:5672)
2021-03-31 11:30:17.373 [info] <0.17676.9> connection <0.17676.9> (10.233.98.89:48262 -> 10.233.125.85:5672): user 'test' authenticated and granted access to vhost 'test'
2021-03-31 11:30:34.875 [info] <0.17676.9> closing AMQP connection <0.17676.9> (10.233.98.89:48262 -> 10.233.125.85:5672, vhost: 'test', user: 'test')
2021-03-31 11:30:34.876 [info] <0.17691.9> Closing all channels from connection '10.233.98.89:48262 -> 10.233.125.85:5672' because it has been closed

2021-03-31 11:29:34 UTC: Proxy upstream: rabbit:5672 
2021-03-31 11:29:34 UTC: Proxy listening on 0.0.0.0:5673
2021-03-31 11:30:17 UTC: Client connected: 127.0.0.1:35470
2021-03-31 11:30:34 UTC: Client disconnected: 127.0.0.1:35470: #<AMQProxy::Client::Error:Client disconnected>
2021-03-31 11:30:34 UTC: Client disconnected: 127.0.0.1:35470

Packagecloud repository doesn't support debian buster

I couldn't set up amqproxy under debian buster even after looking up debian bash script. It seems to only account for debian stretch.

possibility to also use environments instead of arguments

sometimes it is better or even necessary to use environment variables instead of arguments. the crytsal image in the dokerfile is good to build the binary, but for a running system you have to use a multistage build, so you can shrink your running image to a size of 12.4MB instead of 633MB. but mostly without bash or something else (scratch image) you can't use CMD, where arguments can be passed. So you wish to use ENTRYPOINT with environment variables. i created a feature #18 so the environment variables take place before defaults but after arguments.

Docker build failed

`docker build -t amqproxy:v0.4.5 https://github.com/cloudamqp/amqproxy.git#v0.4.5
Sending build context to Docker daemon 70.14kB
Step 1/8 : FROM crystallang/crystal
---> 4687b017edf3
Step 2/8 : WORKDIR /app
---> Using cache
---> 6ab8562c6b50
Step 3/8 : COPY . .
---> Using cache
---> 15a7f1d11032
Step 4/8 : RUN shards build --release --production
---> Running in e2c4d603cac8
Resolving dependencies
Fetching https://github.com/cloudamqp/amq-protocol.cr.git
Installing amq-protocol (0.3.6)
Building: amqproxy
Error target amqproxy failed to compile:
Showing last frame. Use --error-trace for full trace.

In lib/amq-protocol/src/amq/protocol/table.cr:227:32

227 | private def skip_field : Int32
^
Error: method must return Int32 but it is returning Int64
`

Setup in Ubuntu

Hi,

I need help for setting this up in Ubuntu. Followed installation instruction in here https://github.com/cloudamqp/amqproxy
I'm using pika python as client library.

I'm getting this error at journalctl of amqproxy.service

amqproxy[9360]: Upstream error for user '' to vhost '*': #<AMQProxy::Upstream::Error:Cannot establish connection to upstream>

I edited the upstream of amqproxy.service to point to my CloudAMQP URL by running this command

sudo systemctl edit --full amqproxy.service

then I am pointing my client to connect to localhost:5673 and providing the username, password and vhost in the upstream server.

What am I doing wrong? Thanks in advance.

Test multiple threads with Lua client

"...multiple threads connecting... the number of connections increases"
https://github.com/mengz0/amqp

End of file reached (IO::EOFError)

I have issue #33 reproduced in the latest version (0.4.5)

Unhandled exception in spawn: End of file reached (IO::EOFError)
  from ???
  from /usr/share/crystal/src/gc/boehm.cr:110:5 in 'read_loop'
  from /usr/share/crystal/src/fiber.cr:255:3 in 'run'
  from ???

Crystal version:

Crystal 0.34.0 [4401e90f0] (2020-04-06)

LLVM: 8.0.0
Default target: x86_64-unknown-linux-gnu

This problem is reproduced only for consumer connections. The producers work great.

Invalid protocol start

Hello,

I'm getting this error when trying to connect through the proxy using Celery.

Proxy upstream: __removed__.rmq.cloudamqp.com:5672 
Proxy listening on [::]:5673
Client connection error from [::ffff:172.17.0.1]:36944: #<IO::EOFError:Invalid protocol start>
Client connection closed from [::ffff:172.17.0.1]:36944
Client connection error from [::ffff:172.17.0.1]:36946: #<IO::EOFError:Invalid protocol start>
Client connection closed from [::ffff:172.17.0.1]:36946

And from celery:

[2018-07-26 17:03:43,610: DEBUG/MainProcess] | Worker: Starting Hub
[2018-07-26 17:03:43,610: DEBUG/MainProcess] ^-- substep ok
[2018-07-26 17:03:43,610: DEBUG/MainProcess] | Worker: Starting Pool
[2018-07-26 17:03:44,179: DEBUG/MainProcess] ^-- substep ok
[2018-07-26 17:03:44,180: DEBUG/MainProcess] | Worker: Starting Consumer
[2018-07-26 17:03:44,180: DEBUG/MainProcess] | Consumer: Starting Connection
[2018-07-26 17:03:44,192: ERROR/MainProcess] consumer: Cannot connect to amqp://__removed__:**@127.0.0.1:5673/__removed__: Socket closed.
Trying again in 2.00 seconds...

[2018-07-26 17:03:46,208: ERROR/MainProcess] consumer: Cannot connect to amqp://__removed__:**@127.0.0.1:5673/__removed__: Socket closed.
Trying again in 4.00 seconds...

^C
worker: Hitting Ctrl+C again will terminate all running tasks!

worker: Warm shutdown (MainProcess)
[2018-07-26 17:03:46,313: DEBUG/MainProcess] | Worker: Closing Hub...
[2018-07-26 17:03:46,313: DEBUG/MainProcess] | Worker: Closing Pool...
[2018-07-26 17:03:46,314: DEBUG/MainProcess] | Worker: Closing Consumer...
[2018-07-26 17:03:46,314: DEBUG/MainProcess] | Worker: Stopping Consumer...
[2018-07-26 17:03:46,314: DEBUG/MainProcess] | Worker: Stopping Pool...
[2018-07-26 17:03:47,326: DEBUG/MainProcess] | Worker: Stopping Hub...
[2018-07-26 17:03:47,326: DEBUG/MainProcess] | Consumer: Shutdown Heart...
[2018-07-26 17:03:47,327: DEBUG/MainProcess] | Consumer: Shutdown Gossip...
[2018-07-26 17:03:47,327: DEBUG/MainProcess] | Consumer: Shutdown Control...
[2018-07-26 17:03:47,327: DEBUG/MainProcess] | Consumer: Shutdown Tasks...
[2018-07-26 17:03:47,327: DEBUG/MainProcess] | Consumer: Shutdown Events...
[2018-07-26 17:03:47,327: DEBUG/MainProcess] | Consumer: Shutdown Connection...
[2018-07-26 17:03:47,329: DEBUG/MainProcess] removing tasks from inqueue until task handler finished

Connection directly to cloudamqp works fine tho:

[2018-07-26 17:05:22,180: DEBUG/MainProcess] | Worker: Starting Hub
[2018-07-26 17:05:22,180: DEBUG/MainProcess] ^-- substep ok
[2018-07-26 17:05:22,180: DEBUG/MainProcess] | Worker: Starting Pool
[2018-07-26 17:05:22,768: DEBUG/MainProcess] ^-- substep ok
[2018-07-26 17:05:22,769: DEBUG/MainProcess] | Worker: Starting Consumer
[2018-07-26 17:05:22,769: DEBUG/MainProcess] | Consumer: Starting Connection
[2018-07-26 17:05:23,076: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': '__removed__', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'information': 'Licensed under the MPL.  See http://www.rabbitmq.com/', 'platform': 'Erlang/OTP 20.1', 'product': 'RabbitMQ', 'version': '3.7.7'}, mechanisms: [b'AMQPLAIN', b'PLAIN'], locales: ['en_US']
[2018-07-26 17:05:23,267: INFO/MainProcess] Connected to amqp://__removed__:**@__removed__.rmq.cloudamqp.com:5672/__removed__

Newline in logfile

I noticed that AMQProxy logs have \r as newline. This is very annoying when trying to grep, count lines and whatnot on linux. It means that the newlines have to be converted to \n first. Actually, I just checked again and I'm also seeing some \n\r scattered around, which is really exotic. Anyone running AMQProxy on BBC Micro?

I don't know anything about Crystal and I don't wanna, but PHP has a builtin constant for a platform-specific newline: PHP_EOL. This will be \n on linux and \r\n on windows. Although to be fair, does AMQProxy even target windows at all? If not, search/replace \r with \n in the codebase is a solution.

Question: How are channels handled by the proxy?

If I have a PHP page:

<?php

echo 'Hello, welcome to my lovely website.';

$connection = new \AMQPConnection(...);
$channel = new \AMQPChannel($connection);
$exchange = new \AMQPExchange($channel);
$exchange->publish(...);

?>

Will the channel get reused on the second execution of this code?

Could I get the proxy to reuse the channel by setting the channel ID when I open the channel? (I don't believe the PECL version allows this but other PHP libraries do).

how can i determine if im actually getting cached connections?

I installed the service and configured it to hit my rabbitmq server, and my code is configured to hit the amqproxy (and working)

however i dont see any performance increase and see a bunch of this in the system.log:

Client connection closed from [::ffff:127.0.0.1]:36414
Client connection accepted from [::ffff:127.0.0.1]:36422

So can I confirm 1 way or another that its actually reusing a connection?

Login refused: invalid credentials

I'm trying to get this working on my local development environment. I've installed amqproxy_0.2.0-1.deb and it's running as:

/usr/bin/amqproxy -p 5673 amqp://127.0.0.1:5672

I've updated my app config to connect to port 5673 instead of the usual 5672 for RabbitMQ. Everything else remains the same. Here's the relevant config values for reference:

RABBITMQ_HOST="localhost"
RABBITMQ_PORT="5673"
RABBITMQ_LOGIN="guest"
RABBITMQ_PASSWORD="guest"
RABBITMQ_VHOST="/"

Here's our connection code using php-amqplib:

                $streamConnect = new AMQPStreamConnection(
                    getenv('RABBITMQ_HOST'),
                    getenv('RABBITMQ_PORT'),
                    getenv('RABBITMQ_LOGIN'),
                    getenv('RABBITMQ_PASSWORD'),
                    getenv('RABBITMQ_VHOST')
                );

Here's the error I'm seeing in /var/log/rabbitmq/[email protected]:

=INFO REPORT==== 3-Apr-2018::15:33:11 ===
accepting AMQP connection <0.877.0> (127.0.0.1:45566 -> 127.0.0.1:5672)

=ERROR REPORT==== 3-Apr-2018::15:33:14 ===
closing AMQP connection <0.877.0> (127.0.0.1:45566 -> 127.0.0.1:5672):
{handshake_error,starting,0,
                 {amqp_error,access_refused,
                             "PLAIN login refused: user '' - invalid credentials",
                             'connection.start_ok'}}

It appears that the username isn't being forwarded via the proxy. Is there anything else I should be doing to debug this?

TLS Support

Hello,

I did some testing and it does not seem like the proxy supports TLS?
My RabbitMQ server is configured to verify peer certificates and I get the following error:

read tcp 127.0.0.1:47624->127.0.0.1:5673: read: connection reset by peer

which is expected if the peer is not presenting a certificate.

Name resolution issue

We recently upgraded from version 0.3.6 to 0.4.4 and started receiving the following error in production:

2 clients		 2 upstreams
Upstream connection could not be established
Unhandled exception in spawn: getaddrinfo: Temporary failure in name resolution (Socket::Addrinfo::Error)
from usr/share/crystal/src/gc/boehm.cr:110:5 in 'handle_connection'
from vagrant/src/amqproxy/server.cr:38:11 in '->'
from usr/share/crystal/src/fiber.cr:255:3 in 'run'
from ???

We have several proxies in a k8s deployment which directs traffic to CloudAMQP. The pod is a ubuntu:18.04 image if that helps. Let me know if you need any more details.

Memory leak in 0.5.4

We are running AmqProxy in a mostly RPC style setup and are experiencing some annoying memory leaks.
It looks like the proces is leaking 8188k sized page tables, which causes the server to run out of memory:

We figured the servers could use a nice upgrade due to recent load issues and steady growth, but it blew throug 4gb of extra ram within a day:

me@api-server-02:~$ ./getfreemem.sh
  PID USER     %MEM COMMAND
  861 censored+  4.8 /home/censored/amqproxy/amqproxy -l 127.0.0.1 -p 5673 -d amqps://censored.rmq.cloudamqp.com
  430 root      0.5 php-fpm: master process (/etc/php/7.4/fpm/php-fpm.conf)
 9746 censored+  0.4 php-fpm: pool censored
 9745 censored+  0.4 php-fpm: pool censored
  973 censored+  0.4 php-fpm: pool censored
  972 censored+  0.4 php-fpm: pool censored
  971 censored+  0.4 php-fpm: pool censored
 9615 censored+  0.4 php-fpm: pool censored
 9313 censored+  0.4 php-fpm: pool censored
 9299 censored+  0.4 php-fpm: pool censored

me@api-server-02:~$ sudo pmap 861
861:   /home/censored/amqproxy/amqproxy -l 127.0.0.1 -p 5673 -d amqps://censored.rmq.cloudamqp.com
0000563f03e5a000    896K r-x-- amqproxy
0000563f0413a000      4K r---- amqproxy
0000563f0413b000      4K rw--- amqproxy
0000563f0413c000   2384K rw---   [ anon ]
0000563f054b0000  12132K rw---   [ anon ]
00007fae56907000      4K -----   [ anon ]
00007fae56908000   8188K rw---   [ anon ]
00007fae57107000      4K -----   [ anon ]
00007fae57108000   8188K rw---   [ anon ]
00007fae58907000      4K -----   [ anon ]
00007fae58908000   8188K rw---   [ anon ]
00007fae59107000      4K -----   [ anon ]
00007fae59108000   8188K rw---   [ anon ]
00007fae59907000      4K -----   [ anon ]
00007fae59908000   8188K rw---   [ anon ]
00007fae5a907000      4K -----   [ anon ]
00007fae5a908000   8188K rw---   [ anon ]
00007fae5b107000      4K -----   [ anon ]
00007fae5b108000   8188K rw---   [ anon ]
00007fae5b907000      4K -----   [ anon ]
00007fae5b908000   8188K rw---   [ anon ]
00007fae5c107000      4K -----   [ anon ]
00007fae5c108000   8188K rw---   [ anon ]
00007fae5c907000      4K -----   [ anon ]
00007fae5c908000   8188K rw---   [ anon ]
00007fae5d107000      4K -----   [ anon ]
00007fae5d108000   8188K rw---   [ anon ]
00007fae5d907000      4K -----   [ anon ]
00007fae5d908000   8188K rw---   [ anon ]
00007fae5e107000      4K -----   [ anon ]
00007fae5e108000   8188K rw---   [ anon ]
00007fae5e907000      4K -----   [ anon ]
00007fae5e908000   8188K rw---   [ anon ]
00007fae5f107000      4K -----   [ anon ]
00007fae5f108000   8188K rw---   [ anon ]
00007fae5f907000      4K -----   [ anon ]
00007fae5f908000   8188K rw---   [ anon ]
00007fae60107000      4K -----   [ anon ]
00007fae60108000   8188K rw---   [ anon ]
00007fae60907000      4K -----   [ anon ]
00007fae60908000   8188K rw---   [ anon ]
00007fae61907000      4K -----   [ anon ]
00007fae61908000   8188K rw---   [ anon ]
00007fae62107000      4K -----   [ anon ]
00007fae62108000   8188K rw---   [ anon ]
00007fae62907000      4K -----   [ anon ]
00007fae62908000   8188K rw---   [ anon ]
00007fae63907000      4K -----   [ anon ]
00007fae63908000   8188K rw---   [ anon ]
00007fae64107000      4K -----   [ anon ]
00007fae64108000   8188K rw---   [ anon ]
00007fae64907000      4K -----   [ anon ]
00007fae64908000   8188K rw---   [ anon ]
00007fae65107000      4K -----   [ anon ]
00007fae65108000   8188K rw---   [ anon ]
00007fae65907000      4K -----   [ anon ]
00007fae65908000   8188K rw---   [ anon ]
00007fae66107000      4K -----   [ anon ]
00007fae66108000   8188K rw---   [ anon ]
00007fae66907000      4K -----   [ anon ]
00007fae66908000   8188K rw---   [ anon ]
00007fae67907000      4K -----   [ anon ]
00007fae67908000   8188K rw---   [ anon ]
00007fae68107000      4K -----   [ anon ]
00007fae68108000   8188K rw---   [ anon ]
00007fae68907000      4K -----   [ anon ]
00007fae68908000   8188K rw---   [ anon ]
00007fae69107000      4K -----   [ anon ]
00007fae69108000   8188K rw---   [ anon ]
00007fae69907000      4K -----   [ anon ]
00007fae69908000   8188K rw---   [ anon ]
00007fae6a107000      4K -----   [ anon ]
00007fae6a108000   8188K rw---   [ anon ]
00007fae6a907000      4K -----   [ anon ]
00007fae6a908000   8188K rw---   [ anon ]
00007fae6b107000      4K -----   [ anon ]
00007fae6b108000   8188K rw---   [ anon ]
00007fae6b907000      4K -----   [ anon ]
00007fae6b908000   8188K rw---   [ anon ]
00007fae6c107000      4K -----   [ anon ]
00007fae6c108000   8188K rw---   [ anon ]
00007fae6c907000      4K -----   [ anon ]
00007fae6c908000   8188K rw---   [ anon ]
00007fae6d107000      4K -----   [ anon ]
00007fae6d108000   8188K rw---   [ anon ]
00007fae6d907000      4K -----   [ anon ]
00007fae6d908000   8188K rw---   [ anon ]
00007fae6e107000      4K -----   [ anon ]
00007fae6e108000   8188K rw---   [ anon ]
00007fae6e907000      4K -----   [ anon ]
00007fae6e908000   8188K rw---   [ anon ]
00007fae6f107000      4K -----   [ anon ]
00007fae6f108000   8188K rw---   [ anon ]
00007fae6f907000      4K -----   [ anon ]
00007fae6f908000   8188K rw---   [ anon ]
00007fae70107000      4K -----   [ anon ]
00007fae70108000   8188K rw---   [ anon ]
00007fae70907000      4K -----   [ anon ]
00007fae70908000   8188K rw---   [ anon ]
00007fae71107000      4K -----   [ anon ]
00007fae71108000   8188K rw---   [ anon ]
00007fae71907000      4K -----   [ anon ]
00007fae71908000   8188K rw---   [ anon ]
00007fae72107000      4K -----   [ anon ]
00007fae72108000   8188K rw---   [ anon ]
00007fae72907000      4K -----   [ anon ]
00007fae72908000   8188K rw---   [ anon ]
00007fae73107000      4K -----   [ anon ]
00007fae73108000   8188K rw---   [ anon ]
00007fae73907000      4K -----   [ anon ]
00007fae73908000   8188K rw---   [ anon ]
00007fae74107000      4K -----   [ anon ]
00007fae74108000   8188K rw---   [ anon ]
00007fae74907000      4K -----   [ anon ]
00007fae74908000   8188K rw---   [ anon ]
00007fae75107000      4K -----   [ anon ]
00007fae75108000   8188K rw---   [ anon ]
00007fae75907000      4K -----   [ anon ]
00007fae75908000   8188K rw---   [ anon ]
00007fae76107000      4K -----   [ anon ]
00007fae76108000   8188K rw---   [ anon ]
00007fae76907000      4K -----   [ anon ]
00007fae76908000   8188K rw---   [ anon ]
00007fae77107000      4K -----   [ anon ]
00007fae77108000   8188K rw---   [ anon ]
00007fae77907000      4K -----   [ anon ]
00007fae77908000   8188K rw---   [ anon ]
00007fae78107000      4K -----   [ anon ]
00007fae78108000   8188K rw---   [ anon ]
00007fae78907000      4K -----   [ anon ]
00007fae78908000   8188K rw---   [ anon ]
00007fae79107000      4K -----   [ anon ]
00007fae79108000   8188K rw---   [ anon ]
00007fae79907000      4K -----   [ anon ]
00007fae79908000   8188K rw---   [ anon ]
00007fae7a107000      4K -----   [ anon ]
00007fae7a108000   8188K rw---   [ anon ]
00007fae7a907000      4K -----   [ anon ]
00007fae7a908000   8188K rw---   [ anon ]
00007fae7b107000      4K -----   [ anon ]
00007fae7b108000   8188K rw---   [ anon ]
00007fae7b907000      4K -----   [ anon ]
00007fae7b908000   8188K rw---   [ anon ]
00007fae7c107000      4K -----   [ anon ]
00007fae7c108000   8188K rw---   [ anon ]
00007fae7c907000      4K -----   [ anon ]
00007fae7c908000   8188K rw---   [ anon ]
00007fae7d107000      4K -----   [ anon ]
00007fae7d108000   8188K rw---   [ anon ]
00007fae7d907000      4K -----   [ anon ]
00007fae7d908000   8188K rw---   [ anon ]
00007fae7e107000      4K -----   [ anon ]
00007fae7e108000   8188K rw---   [ anon ]
00007fae7e907000      4K -----   [ anon ]
00007fae7e908000   8188K rw---   [ anon ]
00007fae7f107000      4K -----   [ anon ]
00007fae7f108000   8188K rw---   [ anon ]
00007fae7f907000      4K -----   [ anon ]
00007fae7f908000   8188K rw---   [ anon ]
00007fae80107000      4K -----   [ anon ]
00007fae80108000   8188K rw---   [ anon ]
00007fae80907000      4K -----   [ anon ]
00007fae80908000   8188K rw---   [ anon ]
00007fae81107000      4K -----   [ anon ]
00007fae81108000   8188K rw---   [ anon ]
00007fae81907000      4K -----   [ anon ]
00007fae81908000   8188K rw---   [ anon ]
00007fae82107000      4K -----   [ anon ]
00007fae82108000   8188K rw---   [ anon ]
00007fae82907000      4K -----   [ anon ]
00007fae82908000   8188K rw---   [ anon ]
00007fae83107000      4K -----   [ anon ]
00007fae83108000   8188K rw---   [ anon ]
00007fae83907000      4K -----   [ anon ]
00007fae83908000   8188K rw---   [ anon ]
00007fae84107000      4K -----   [ anon ]
00007fae84108000   8188K rw---   [ anon ]
00007fae84907000      4K -----   [ anon ]
00007fae84908000   8188K rw---   [ anon ]
00007fae85107000      4K -----   [ anon ]
00007fae85108000   8188K rw---   [ anon ]
00007fae85907000      4K -----   [ anon ]
00007fae85908000   8188K rw---   [ anon ]
00007fae86107000      4K -----   [ anon ]
00007fae86108000   8188K rw---   [ anon ]
00007fae86907000      4K -----   [ anon ]
00007fae86908000   8188K rw---   [ anon ]
00007fae87107000      4K -----   [ anon ]
00007fae87108000   8188K rw---   [ anon ]
00007fae87907000      4K -----   [ anon ]
00007fae87908000   8188K rw---   [ anon ]
00007fae88107000      4K -----   [ anon ]
00007fae88108000   8188K rw---   [ anon ]
00007fae88907000      4K -----   [ anon ]
00007fae88908000   8188K rw---   [ anon ]
00007fae89107000      4K -----   [ anon ]
00007fae89108000   8188K rw---   [ anon ]
00007fae89907000      4K -----   [ anon ]
00007fae89908000   8188K rw---   [ anon ]
00007fae8a107000      4K -----   [ anon ]
00007fae8a108000   8188K rw---   [ anon ]
00007fae8a907000      4K -----   [ anon ]
00007fae8a908000   8188K rw---   [ anon ]
00007fae8b107000      4K -----   [ anon ]
00007fae8b108000   8188K rw---   [ anon ]
00007fae8b907000      4K -----   [ anon ]
00007fae8b908000   8188K rw---   [ anon ]
00007fae8c107000      4K -----   [ anon ]
00007fae8c108000   8188K rw---   [ anon ]
00007fae8c907000      4K -----   [ anon ]
00007fae8c908000   8188K rw---   [ anon ]
00007fae8d107000      4K -----   [ anon ]
00007fae8d108000   8188K rw---   [ anon ]
00007fae8d907000      4K -----   [ anon ]
00007fae8d908000   8188K rw---   [ anon ]
00007fae8e107000      4K -----   [ anon ]
00007fae8e108000   8188K rw---   [ anon ]
00007fae8e907000      4K -----   [ anon ]
00007fae8e908000   8188K rw---   [ anon ]
00007fae8f107000      4K -----   [ anon ]
00007fae8f108000   8188K rw---   [ anon ]
00007fae8f907000      4K -----   [ anon ]
00007fae8f908000   8188K rw---   [ anon ]
00007fae90107000      4K -----   [ anon ]
00007fae90108000   8188K rw---   [ anon ]
00007fae90907000      4K -----   [ anon ]
00007fae90908000   8188K rw---   [ anon ]
00007fae91107000      4K -----   [ anon ]
00007fae91108000   8188K rw---   [ anon ]
00007fae91907000      4K -----   [ anon ]
00007fae91908000   8188K rw---   [ anon ]
00007fae92107000      4K -----   [ anon ]
00007fae92108000   8188K rw---   [ anon ]
00007fae92907000      4K -----   [ anon ]
00007fae92908000   8188K rw---   [ anon ]
00007fae93107000      4K -----   [ anon ]
00007fae93108000   8188K rw---   [ anon ]
00007fae93907000      4K -----   [ anon ]
00007fae93908000   8188K rw---   [ anon ]
00007fae94107000      4K -----   [ anon ]
00007fae94108000   8188K rw---   [ anon ]
00007fae94907000      4K -----   [ anon ]
00007fae94908000   8188K rw---   [ anon ]
00007fae95107000      4K -----   [ anon ]
00007fae95108000   8188K rw---   [ anon ]
00007fae95907000      4K -----   [ anon ]
00007fae95908000   8188K rw---   [ anon ]
00007fae96107000      4K -----   [ anon ]
00007fae96108000   8188K rw---   [ anon ]
00007fae96907000      4K -----   [ anon ]
00007fae96908000   8188K rw---   [ anon ]
00007fae97107000      4K -----   [ anon ]
00007fae97108000   8188K rw---   [ anon ]
00007fae97907000      4K -----   [ anon ]
00007fae97908000   8188K rw---   [ anon ]
00007fae98107000      4K -----   [ anon ]
00007fae98108000   8188K rw---   [ anon ]
00007fae98907000      4K -----   [ anon ]
00007fae98908000   8188K rw---   [ anon ]
00007fae99107000      4K -----   [ anon ]
00007fae99108000   8188K rw---   [ anon ]
00007fae99907000      4K -----   [ anon ]
00007fae99908000   8188K rw---   [ anon ]
00007fae9a107000      4K -----   [ anon ]
00007fae9a108000   8188K rw---   [ anon ]
00007fae9a907000      4K -----   [ anon ]
00007fae9a908000   8188K rw---   [ anon ]
00007fae9b107000      4K -----   [ anon ]
00007fae9b108000   8188K rw---   [ anon ]
00007fae9b907000      4K -----   [ anon ]
00007fae9b908000   8188K rw---   [ anon ]
00007fae9c107000      4K -----   [ anon ]
00007fae9c108000   8188K rw---   [ anon ]
00007fae9c907000      4K -----   [ anon ]
00007fae9c908000   8188K rw---   [ anon ]
00007fae9d107000      4K -----   [ anon ]
00007fae9d108000   8188K rw---   [ anon ]
00007fae9d907000      4K -----   [ anon ]
00007fae9d908000   8188K rw---   [ anon ]
00007fae9e107000      4K -----   [ anon ]
00007fae9e108000   8188K rw---   [ anon ]
00007fae9e907000      4K -----   [ anon ]
00007fae9e908000   8188K rw---   [ anon ]
00007fae9f107000      4K -----   [ anon ]
00007fae9f108000   8188K rw---   [ anon ]
00007fae9f907000      4K -----   [ anon ]
00007fae9f908000   8188K rw---   [ anon ]
00007faea0107000      4K -----   [ anon ]
00007faea0108000   8188K rw---   [ anon ]
00007faea0907000      4K -----   [ anon ]
00007faea0908000   8188K rw---   [ anon ]
00007faea1107000      4K -----   [ anon ]
00007faea1108000   8188K rw---   [ anon ]
00007faea1907000      4K -----   [ anon ]
00007faea1908000   8188K rw---   [ anon ]
00007faea2107000      4K -----   [ anon ]
00007faea2108000   8188K rw---   [ anon ]
00007faea2907000      4K -----   [ anon ]
00007faea2908000   8188K rw---   [ anon ]
00007faea3107000      4K -----   [ anon ]
00007faea3108000   8188K rw---   [ anon ]
00007faea3907000      4K -----   [ anon ]
00007faea3908000   8188K rw---   [ anon ]
00007faea4107000      4K -----   [ anon ]
00007faea4108000   8188K rw---   [ anon ]
00007faea4907000      4K -----   [ anon ]
00007faea4908000   8188K rw---   [ anon ]
00007faea5107000      4K -----   [ anon ]
00007faea5108000   8188K rw---   [ anon ]
00007faea5907000      4K -----   [ anon ]
00007faea5908000   8188K rw---   [ anon ]
00007faea6107000      4K -----   [ anon ]
00007faea6108000   8188K rw---   [ anon ]
00007faea6907000      4K -----   [ anon ]
00007faea6908000   8188K rw---   [ anon ]
00007faea7107000      4K -----   [ anon ]
00007faea7108000   8188K rw---   [ anon ]
00007faea7907000      4K -----   [ anon ]
00007faea7908000   8188K rw---   [ anon ]
00007faea8107000      4K -----   [ anon ]
00007faea8108000   8188K rw---   [ anon ]
00007faea8907000      4K -----   [ anon ]
00007faea8908000   8188K rw---   [ anon ]
00007faea9107000      4K -----   [ anon ]
00007faea9108000   8188K rw---   [ anon ]
00007faea9907000      4K -----   [ anon ]
00007faea9908000   8188K rw---   [ anon ]
00007faeaa107000      4K -----   [ anon ]
00007faeaa108000   8188K rw---   [ anon ]
00007faeaa907000      4K -----   [ anon ]
00007faeaa908000   8188K rw---   [ anon ]
00007faeab107000      4K -----   [ anon ]
00007faeab108000   8188K rw---   [ anon ]
00007faeab907000      4K -----   [ anon ]
00007faeab908000   8188K rw---   [ anon ]
00007faeac107000      4K -----   [ anon ]
00007faeac108000   8188K rw---   [ anon ]
00007faeac907000      4K -----   [ anon ]
00007faeac908000   8188K rw---   [ anon ]
00007faead107000      4K -----   [ anon ]
00007faead108000   8188K rw---   [ anon ]
00007faead907000      4K -----   [ anon ]
00007faead908000   8188K rw---   [ anon ]
00007faeae107000      4K -----   [ anon ]
00007faeae108000   8188K rw---   [ anon ]
00007faeae907000      4K -----   [ anon ]
00007faeae908000   8188K rw---   [ anon ]
00007faeaf107000      4K -----   [ anon ]
00007faeaf108000   8188K rw---   [ anon ]
00007faeaf907000      4K -----   [ anon ]
00007faeaf908000   8188K rw---   [ anon ]
00007faeb0107000      4K -----   [ anon ]
00007faeb0108000   8188K rw---   [ anon ]
00007faeb0907000      4K -----   [ anon ]
00007faeb0908000   8188K rw---   [ anon ]
00007faeb1107000      4K -----   [ anon ]
00007faeb1108000   8188K rw---   [ anon ]
00007faeb1907000      4K -----   [ anon ]
00007faeb1908000   8188K rw---   [ anon ]
00007faeb2107000      4K -----   [ anon ]
00007faeb2108000   8188K rw---   [ anon ]
00007faeb2907000      4K -----   [ anon ]
00007faeb2908000   8188K rw---   [ anon ]
00007faeb3107000      4K -----   [ anon ]
00007faeb3108000   8188K rw---   [ anon ]
00007faeb3907000      4K -----   [ anon ]
00007faeb3908000   8188K rw---   [ anon ]
00007faeb4107000      4K -----   [ anon ]
00007faeb4108000   8188K rw---   [ anon ]
00007faeb4907000      4K -----   [ anon ]
00007faeb4908000   8188K rw---   [ anon ]
00007faeb5107000      4K -----   [ anon ]
00007faeb5108000   8188K rw---   [ anon ]
00007faeb5907000      4K -----   [ anon ]
00007faeb5908000   8188K rw---   [ anon ]
00007faeb6107000      4K -----   [ anon ]
00007faeb6108000   8188K rw---   [ anon ]
00007faeb6907000      4K -----   [ anon ]
00007faeb6908000   8188K rw---   [ anon ]
00007faeb7107000      4K -----   [ anon ]
00007faeb7108000   8188K rw---   [ anon ]
00007faeb7907000      4K -----   [ anon ]
00007faeb7908000   8188K rw---   [ anon ]
00007faeb8107000      4K -----   [ anon ]
00007faeb8108000   8188K rw---   [ anon ]
00007faeb8907000      4K -----   [ anon ]
00007faeb8908000   8188K rw---   [ anon ]
00007faeb9107000      4K -----   [ anon ]
00007faeb9108000   8188K rw---   [ anon ]
00007faeb9907000      4K -----   [ anon ]
00007faeb9908000   8188K rw---   [ anon ]
00007faeba107000      4K -----   [ anon ]
00007faeba108000   8188K rw---   [ anon ]
00007faeba907000      4K -----   [ anon ]
00007faeba908000   8188K rw---   [ anon ]
00007faebb107000      4K -----   [ anon ]
00007faebb108000   8188K rw---   [ anon ]
00007faebb907000      4K -----   [ anon ]
00007faebb908000   8188K rw---   [ anon ]
00007faebc107000      4K -----   [ anon ]
00007faebc108000   8188K rw---   [ anon ]
00007faebc907000      4K -----   [ anon ]
00007faebc908000   8188K rw---   [ anon ]
00007faebd107000      4K -----   [ anon ]
00007faebd108000   8188K rw---   [ anon ]
00007faebd907000      4K -----   [ anon ]
00007faebd908000   8188K rw---   [ anon ]
00007faebe107000      4K -----   [ anon ]
00007faebe108000   8188K rw---   [ anon ]
00007faebe907000      4K -----   [ anon ]
00007faebe908000   8188K rw---   [ anon ]
00007faebf107000      4K -----   [ anon ]
00007faebf108000   8188K rw---   [ anon ]
00007faebf907000      4K -----   [ anon ]
00007faebf908000   8188K rw---   [ anon ]
00007faec0107000      4K -----   [ anon ]
00007faec0108000   8188K rw---   [ anon ]
00007faec0907000      4K -----   [ anon ]
00007faec0908000   8188K rw---   [ anon ]
00007faec1107000      4K -----   [ anon ]
00007faec1108000   8188K rw---   [ anon ]
00007faec1907000      4K -----   [ anon ]
00007faec1908000   8188K rw---   [ anon ]
00007faec2107000      4K -----   [ anon ]
00007faec2108000   8188K rw---   [ anon ]
00007faec2907000      4K -----   [ anon ]
00007faec2908000   8188K rw---   [ anon ]
00007faec3107000      4K -----   [ anon ]
00007faec3108000   8188K rw---   [ anon ]
00007faec3907000      4K -----   [ anon ]
00007faec3908000   8188K rw---   [ anon ]
00007faec4107000      4K -----   [ anon ]
00007faec4108000   8188K rw---   [ anon ]
00007faec4907000      4K -----   [ anon ]
00007faec4908000   8188K rw---   [ anon ]
00007faec5107000      4K -----   [ anon ]
00007faec5108000   8188K rw---   [ anon ]
00007faec5907000      4K -----   [ anon ]
00007faec5908000   8188K rw---   [ anon ]
00007faec6107000      4K -----   [ anon ]
00007faec6108000   8188K rw---   [ anon ]
00007faec6907000      4K -----   [ anon ]
00007faec6908000   8188K rw---   [ anon ]
00007faec7107000      4K -----   [ anon ]
00007faec7108000   8188K rw---   [ anon ]
00007faec7907000      4K -----   [ anon ]
00007faec7908000   8188K rw---   [ anon ]
00007faec8107000      4K -----   [ anon ]
00007faec8108000   8188K rw---   [ anon ]
00007faec8907000      4K -----   [ anon ]
00007faec8908000   8188K rw---   [ anon ]
00007faec9107000      4K -----   [ anon ]
00007faec9108000   8188K rw---   [ anon ]
00007faec9907000      4K -----   [ anon ]
00007faec9908000   8188K rw---   [ anon ]
00007faeca107000      4K -----   [ anon ]
00007faeca108000   8188K rw---   [ anon ]
00007faeca907000      4K -----   [ anon ]
00007faeca908000   8188K rw---   [ anon ]
00007faecb107000      4K -----   [ anon ]
00007faecb108000   8188K rw---   [ anon ]
00007faecb907000      4K -----   [ anon ]
00007faecb908000   8188K rw---   [ anon ]
00007faecc107000      4K -----   [ anon ]
00007faecc108000   8188K rw---   [ anon ]
00007faecc907000      4K -----   [ anon ]
00007faecc908000   8188K rw---   [ anon ]
00007faecd107000      4K -----   [ anon ]
00007faecd108000   8188K rw---   [ anon ]
00007faecd907000      4K -----   [ anon ]
00007faecd908000   8188K rw---   [ anon ]
00007faece107000      4K -----   [ anon ]
00007faece108000   8188K rw---   [ anon ]
00007faece907000      4K -----   [ anon ]
00007faece908000   8188K rw---   [ anon ]
00007faecf107000      4K -----   [ anon ]
00007faecf108000   8188K rw---   [ anon ]
00007faecf907000      4K -----   [ anon ]
00007faecf908000   8188K rw---   [ anon ]
00007faed0107000      4K -----   [ anon ]
00007faed0108000   8188K rw---   [ anon ]
00007faed0907000      4K -----   [ anon ]
00007faed0908000   8188K rw---   [ anon ]
00007faed1107000      4K -----   [ anon ]
00007faed1108000   8188K rw---   [ anon ]
00007faed1907000      4K -----   [ anon ]
00007faed1908000   8188K rw---   [ anon ]
00007faed2107000  32776K rw---   [ anon ]
00007faed4109000      4K -----   [ anon ]
00007faed410a000   8188K rw---   [ anon ]
00007faed4909000      4K -----   [ anon ]
00007faed490a000   8188K rw---   [ anon ]
00007faed5109000      4K -----   [ anon ]
00007faed510a000   8188K rw---   [ anon ]
00007faed5909000      4K -----   [ anon ]
00007faed590a000   8188K rw---   [ anon ]
00007faed6109000      4K -----   [ anon ]
00007faed610a000   8188K rw---   [ anon ]
00007faed6909000      4K -----   [ anon ]
00007faed690a000   8188K rw---   [ anon ]
00007faed7109000      4K -----   [ anon ]
00007faed710a000   8188K rw---   [ anon ]
00007faed7909000      4K -----   [ anon ]
00007faed790a000   8188K rw---   [ anon ]
00007faed8109000      4K -----   [ anon ]
00007faed810a000   8188K rw---   [ anon ]
00007faed8909000      4K -----   [ anon ]
00007faed890a000   8188K rw---   [ anon ]
00007faed9109000      4K -----   [ anon ]
00007faed910a000   8188K rw---   [ anon ]
00007faed9909000      4K -----   [ anon ]
00007faed990a000   8188K rw---   [ anon ]
00007faeda109000      4K -----   [ anon ]
00007faeda10a000   8188K rw---   [ anon ]
00007faeda909000      4K -----   [ anon ]
00007faeda90a000   8188K rw---   [ anon ]
00007faedb109000      4K -----   [ anon ]
00007faedb10a000   8188K rw---   [ anon ]
00007faedb909000      4K -----   [ anon ]
00007faedb90a000   8188K rw---   [ anon ]
00007faedc109000      4K -----   [ anon ]
00007faedc10a000   8188K rw---   [ anon ]
00007faedc909000      4K -----   [ anon ]
00007faedc90a000   8188K rw---   [ anon ]
00007faedd109000      4K -----   [ anon ]
00007faedd10a000   8188K rw---   [ anon ]
00007faede108000      4K -----   [ anon ]
00007faede109000   8188K rw---   [ anon ]
00007faede908000  65540K rw---   [ anon ]
00007faee2909000      4K -----   [ anon ]
00007faee290a000   8188K rw---   [ anon ]
00007faee3109000      4K -----   [ anon ]
00007faee310a000   8188K rw---   [ anon ]
00007faee3909000      4K -----   [ anon ]
00007faee390a000   8188K rw---   [ anon ]
00007faee4109000      4K -----   [ anon ]
00007faee410a000   8188K rw---   [ anon ]
00007faee4909000      4K -----   [ anon ]
00007faee490a000   8188K rw---   [ anon ]
00007faee5109000      4K -----   [ anon ]
00007faee510a000   8188K rw---   [ anon ]
00007faee5909000      4K -----   [ anon ]
00007faee590a000   8188K rw---   [ anon ]
00007faee6109000      4K -----   [ anon ]
00007faee610a000   8188K rw---   [ anon ]
00007faee6909000  16384K rw---   [ anon ]
00007faee7909000      4K -----   [ anon ]
00007faee790a000   8188K rw---   [ anon ]
00007faee8109000      4K -----   [ anon ]
00007faee810a000   8188K rw---   [ anon ]
00007faee8909000      4K -----   [ anon ]
00007faee890a000   8188K rw---   [ anon ]
00007faee9109000      4K -----   [ anon ]
00007faee910a000   8188K rw---   [ anon ]
00007faee9909000  16384K rw---   [ anon ]
00007faeea909000      4K -----   [ anon ]
00007faeea90a000   8188K rw---   [ anon ]
00007faeeb109000      4K -----   [ anon ]
00007faeeb10a000   8188K rw---   [ anon ]
00007faeeb909000      4K -----   [ anon ]
00007faeeb90a000   8188K rw---   [ anon ]
00007faeec109000      4K -----   [ anon ]
00007faeec10a000   8188K rw---   [ anon ]
00007faeec909000      4K -----   [ anon ]
00007faeec90a000   8188K rw---   [ anon ]
00007faeed109000      4K -----   [ anon ]
00007faeed10a000   8188K rw---   [ anon ]
00007faeed909000      4K -----   [ anon ]
00007faeed90a000   8188K rw---   [ anon ]
00007faeee109000  16392K rw---   [ anon ]
00007faeef10b000      4K -----   [ anon ]
00007faeef10c000   8188K rw---   [ anon ]
00007faef010a000      4K -----   [ anon ]
00007faef010b000   8188K rw---   [ anon ]
00007faef090a000      4K -----   [ anon ]
00007faef090b000   8188K rw---   [ anon ]
00007faef110a000      4K -----   [ anon ]
00007faef110b000   8188K rw---   [ anon ]
00007faef190a000      4K -----   [ anon ]
00007faef190b000   8188K rw---   [ anon ]
00007faef210a000      4K -----   [ anon ]
00007faef210b000   8188K rw---   [ anon ]
00007faef290a000  65540K rw---   [ anon ]
00007faef690b000      4K -----   [ anon ]
00007faef690c000   8188K rw---   [ anon ]
00007faef710b000      4K -----   [ anon ]
00007faef710c000   8188K rw---   [ anon ]
00007faef790b000      4K -----   [ anon ]
00007faef790c000   8188K rw---   [ anon ]
00007faef810b000      4K -----   [ anon ]
00007faef810c000   8188K rw---   [ anon ]
00007faef890b000      4K -----   [ anon ]
00007faef890c000   8188K rw---   [ anon ]
00007faef910b000      4K -----   [ anon ]
00007faef910c000   8188K rw---   [ anon ]
00007faef990b000      4K -----   [ anon ]
00007faef990c000   8188K rw---   [ anon ]
00007faefa10b000      4K -----   [ anon ]
00007faefa10c000   8188K rw---   [ anon ]
00007faefa90b000      4K -----   [ anon ]
00007faefa90c000   8188K rw---   [ anon ]
00007faefb10b000      4K -----   [ anon ]
00007faefb10c000   8188K rw---   [ anon ]
00007faefb90b000      4K -----   [ anon ]
00007faefb90c000   8188K rw---   [ anon ]
00007faefc10b000      4K -----   [ anon ]
00007faefc10c000   8188K rw---   [ anon ]
00007faefc90b000      4K -----   [ anon ]
00007faefc90c000   8188K rw---   [ anon ]
00007faefd10b000      4K -----   [ anon ]
00007faefd10c000   8188K rw---   [ anon ]
00007faefd90b000      4K -----   [ anon ]
00007faefd90c000   8188K rw---   [ anon ]
00007faefe10b000      4K -----   [ anon ]
00007faefe10c000   8188K rw---   [ anon ]
00007faefe90b000      4K -----   [ anon ]
00007faefe90c000   8188K rw---   [ anon ]
00007faeff10b000      4K -----   [ anon ]
00007faeff10c000   8188K rw---   [ anon ]
00007faeff90b000      4K -----   [ anon ]
00007faeff90c000   8188K rw---   [ anon ]
00007faf0010b000      4K -----   [ anon ]
00007faf0010c000   8188K rw---   [ anon ]
00007faf0090b000      4K -----   [ anon ]
00007faf0090c000   8188K rw---   [ anon ]
00007faf0110b000      4K -----   [ anon ]
00007faf0110c000   8188K rw---   [ anon ]
00007faf0190b000      4K -----   [ anon ]
00007faf0190c000   8188K rw---   [ anon ]
00007faf0210b000      4K -----   [ anon ]
00007faf0210c000   8188K rw---   [ anon ]
00007faf0290b000      4K -----   [ anon ]
00007faf0290c000   8188K rw---   [ anon ]
00007faf0310b000      4K -----   [ anon ]
00007faf0310c000   8188K rw---   [ anon ]
00007faf0390b000      4K -----   [ anon ]
00007faf0390c000   8188K rw---   [ anon ]
00007faf0410b000      4K -----   [ anon ]
00007faf0410c000   8188K rw---   [ anon ]
00007faf0490b000      4K -----   [ anon ]
00007faf0490c000   8188K rw---   [ anon ]
00007faf0510b000      4K -----   [ anon ]
00007faf0510c000   8188K rw---   [ anon ]
00007faf0590b000      4K -----   [ anon ]
00007faf0590c000   8188K rw---   [ anon ]
00007faf0610b000      4K -----   [ anon ]
00007faf0610c000   8188K rw---   [ anon ]
00007faf0690b000      4K -----   [ anon ]
00007faf0690c000   8188K rw---   [ anon ]
00007faf0710b000      4K -----   [ anon ]
00007faf0710c000   8188K rw---   [ anon ]
00007faf0790b000      4K -----   [ anon ]
00007faf0790c000   8188K rw---   [ anon ]
00007faf0810b000      4K -----   [ anon ]
00007faf0810c000   8188K rw---   [ anon ]
00007faf0890b000      4K -----   [ anon ]
00007faf0890c000   8188K rw---   [ anon ]
00007faf0910b000      4K -----   [ anon ]
00007faf0910c000   8188K rw---   [ anon ]
00007faf0990b000      4K -----   [ anon ]
00007faf0990c000   8188K rw---   [ anon ]
00007faf0a10b000      4K -----   [ anon ]
00007faf0a10c000   8188K rw---   [ anon ]
00007faf0a90b000      4K -----   [ anon ]
00007faf0a90c000   8188K rw---   [ anon ]
00007faf0b10b000      4K -----   [ anon ]
00007faf0b10c000   8188K rw---   [ anon ]
00007faf0b90b000  16384K rw---   [ anon ]
00007faf0ce01000      4K -----   [ anon ]
00007faf0ce02000   8188K rw---   [ anon ]
00007faf0d601000      4K -----   [ anon ]
00007faf0d602000   8188K rw---   [ anon ]
00007faf0de01000      4K -----   [ anon ]
00007faf0de02000   8188K rw---   [ anon ]
00007faf0e601000  11304K rw---   [ anon ]
00007faf0f10b000      4K -----   [ anon ]
00007faf0f10c000   8188K rw---   [ anon ]
00007faf0f90b000      4K -----   [ anon ]
00007faf0f90c000   8188K rw---   [ anon ]
00007faf1010b000      4K -----   [ anon ]
00007faf1010c000   8188K rw---   [ anon ]
00007faf1090b000      4K -----   [ anon ]
00007faf1090c000   8188K rw---   [ anon ]
00007faf1110b000      4K -----   [ anon ]
00007faf1110c000   8188K rw---   [ anon ]
00007faf1190b000      4K -----   [ anon ]
00007faf1190c000   8188K rw---   [ anon ]
00007faf1210b000  16384K rw---   [ anon ]
00007faf13451000      4K -----   [ anon ]
00007faf13452000   8188K rw---   [ anon ]
00007faf13c51000      4K -----   [ anon ]
00007faf13c52000   8188K rw---   [ anon ]
00007faf14451000      4K -----   [ anon ]
00007faf14452000   8188K rw---   [ anon ]
00007faf14c51000      4K -----   [ anon ]
00007faf14c52000   8188K rw---   [ anon ]
00007faf15451000  13032K rw---   [ anon ]
00007faf1610b000      4K -----   [ anon ]
00007faf1610c000   8188K rw---   [ anon ]
00007faf1690b000      4K -----   [ anon ]
00007faf1690c000   8188K rw---   [ anon ]
00007faf1710b000      4K -----   [ anon ]
00007faf1710c000   8188K rw---   [ anon ]
00007faf1790b000      4K -----   [ anon ]
00007faf1790c000   8188K rw---   [ anon ]
00007faf1810b000      4K -----   [ anon ]
00007faf1810c000   8188K rw---   [ anon ]
00007faf1890b000      4K -----   [ anon ]
00007faf1890c000   8188K rw---   [ anon ]
00007faf1910b000      4K -----   [ anon ]
00007faf1910c000   8188K rw---   [ anon ]
00007faf1990b000      4K -----   [ anon ]
00007faf1990c000   8188K rw---   [ anon ]
00007faf1a10b000      4K -----   [ anon ]
00007faf1a10c000   8188K rw---   [ anon ]
00007faf1a90b000      4K -----   [ anon ]
00007faf1a90c000   8188K rw---   [ anon ]
00007faf1b10b000      4K -----   [ anon ]
00007faf1b10c000   8188K rw---   [ anon ]
00007faf1b90b000      4K -----   [ anon ]
00007faf1b90c000   8188K rw---   [ anon ]
00007faf1c10b000      4K -----   [ anon ]
00007faf1c10c000   8188K rw---   [ anon ]
00007faf1c90b000      4K -----   [ anon ]
00007faf1c90c000   8188K rw---   [ anon ]
00007faf1d10b000      4K -----   [ anon ]
00007faf1d10c000   8188K rw---   [ anon ]
00007faf1d90b000      4K -----   [ anon ]
00007faf1d90c000   8188K rw---   [ anon ]
00007faf1e10b000      4K -----   [ anon ]
00007faf1e10c000   8188K rw---   [ anon ]
00007faf1e90b000      4K -----   [ anon ]
00007faf1e90c000   8188K rw---   [ anon ]
00007faf1f10b000      4K -----   [ anon ]
00007faf1f10c000   8188K rw---   [ anon ]
00007faf1f90b000      4K -----   [ anon ]
00007faf1f90c000   8188K rw---   [ anon ]
00007faf2010b000      4K -----   [ anon ]
00007faf2010c000   8188K rw---   [ anon ]
00007faf2090b000      4K -----   [ anon ]
00007faf2090c000   8188K rw---   [ anon ]
00007faf2110b000      4K -----   [ anon ]
00007faf2110c000   8188K rw---   [ anon ]
00007faf21984000  40476K rw---   [ anon ]
00007faf2410b000   2880K rw---   [ anon ]
00007faf243db000      4K -----   [ anon ]
00007faf243dc000   8188K rw---   [ anon ]
00007faf24bdb000      4K -----   [ anon ]
00007faf24bdc000   8188K rw---   [ anon ]
00007faf253db000      4K -----   [ anon ]
00007faf253dc000   8188K rw---   [ anon ]
00007faf25bdb000      4K -----   [ anon ]
00007faf25bdc000   8188K rw---   [ anon ]
00007faf263db000      4K -----   [ anon ]
00007faf263dc000   8188K rw---   [ anon ]
00007faf26bdb000      4K -----   [ anon ]
00007faf26bdc000   8188K rw---   [ anon ]
00007faf2764f000   1536K rw---   [ anon ]
00007faf277cf000      4K -----   [ anon ]
00007faf277d0000   8188K rw---   [ anon ]
00007faf27fcf000   6096K rw---   [ anon ]
00007faf285c3000      4K -----   [ anon ]
00007faf285c4000   8188K rw---   [ anon ]
00007faf28dc3000  16384K rw---   [ anon ]
00007faf29dc3000      4K -----   [ anon ]
00007faf29dc4000   8188K rw---   [ anon ]
00007faf2a5c3000    624K rw---   [ anon ]
00007faf2a65f000      4K -----   [ anon ]
00007faf2a660000   8188K rw---   [ anon ]
00007faf2ae5f000      4K -----   [ anon ]
00007faf2ae60000   8188K rw---   [ anon ]
00007faf2b65f000    256K rw---   [ anon ]
00007faf2bd9f000    512K rw---   [ anon ]
00007faf2c45f000   1536K rw---   [ anon ]
00007faf2c5df000      4K -----   [ anon ]
00007faf2c5e0000   8188K rw---   [ anon ]
00007faf2cddf000    256K rw---   [ anon ]
00007faf2ce1f000      4K -----   [ anon ]
00007faf2ce20000   8188K rw---   [ anon ]
00007faf2d61f000      4K -----   [ anon ]
00007faf2d620000   8188K rw---   [ anon ]
00007faf2de1f000    768K rw---   [ anon ]
00007faf2dedf000     12K r---- libnss_files-2.28.so
00007faf2dee2000     28K r-x-- libnss_files-2.28.so
00007faf2dee9000      8K r---- libnss_files-2.28.so
00007faf2deeb000      4K ----- libnss_files-2.28.so
00007faf2deec000      4K r---- libnss_files-2.28.so
00007faf2deed000      4K rw--- libnss_files-2.28.so
00007faf2deee000     24K rw---   [ anon ]
00007faf2df27000   5988K rw---   [ anon ]
00007faf2e519000  10348K rw---   [ anon ]
00007faf2ef34000   7936K rw---   [ anon ]
00007faf2f6f4000      4K -----   [ anon ]
00007faf2f6f5000   8188K rw---   [ anon ]
00007faf2fef4000      4K -----   [ anon ]
00007faf2fef5000   8188K rw---   [ anon ]
00007faf306f4000      4K -----   [ anon ]
00007faf306f5000   8188K rw---   [ anon ]
00007faf30ef4000     64K rw---   [ anon ]
00007faf30f04000      4K -----   [ anon ]
00007faf30f05000   8192K rw---   [ anon ]
00007faf31705000      4K -----   [ anon ]
00007faf31706000   8192K rw---   [ anon ]
00007faf31f06000      4K -----   [ anon ]
00007faf31f07000   9232K rw---   [ anon ]
00007faf3280b000    136K r---- libc-2.28.so
00007faf3282d000   1312K r-x-- libc-2.28.so
00007faf32975000    304K r---- libc-2.28.so
00007faf329c1000      4K ----- libc-2.28.so
00007faf329c2000     16K r---- libc-2.28.so
00007faf329c6000      8K rw--- libc-2.28.so
00007faf329c8000     16K rw---   [ anon ]
00007faf329cc000     12K r---- libgcc_s.so.1
00007faf329cf000     68K r-x-- libgcc_s.so.1
00007faf329e0000     12K r---- libgcc_s.so.1
00007faf329e3000      4K ----- libgcc_s.so.1
00007faf329e4000      4K r---- libgcc_s.so.1
00007faf329e5000      4K rw--- libgcc_s.so.1
00007faf329e6000      4K r---- libdl-2.28.so
00007faf329e7000      4K r-x-- libdl-2.28.so
00007faf329e8000      4K r---- libdl-2.28.so
00007faf329e9000      4K r---- libdl-2.28.so
00007faf329ea000      4K rw--- libdl-2.28.so
00007faf329eb000      8K r---- librt-2.28.so
00007faf329ed000     16K r-x-- librt-2.28.so
00007faf329f1000      8K r---- librt-2.28.so
00007faf329f3000      4K r---- librt-2.28.so
00007faf329f4000      4K rw--- librt-2.28.so
00007faf329f5000      8K rw---   [ anon ]
00007faf329f7000    336K r-x-- libevent-2.1.so.6.0.2
00007faf32a4b000   2044K ----- libevent-2.1.so.6.0.2
00007faf32c4a000      8K r---- libevent-2.1.so.6.0.2
00007faf32c4c000      4K rw--- libevent-2.1.so.6.0.2
00007faf32c4d000     24K r---- libpthread-2.28.so
00007faf32c53000     60K r-x-- libpthread-2.28.so
00007faf32c62000     24K r---- libpthread-2.28.so
00007faf32c68000      4K r---- libpthread-2.28.so
00007faf32c69000      4K rw--- libpthread-2.28.so
00007faf32c6a000     16K rw---   [ anon ]
00007faf32c6e000     52K r---- libm-2.28.so
00007faf32c7b000    636K r-x-- libm-2.28.so
00007faf32d1a000    852K r---- libm-2.28.so
00007faf32def000      4K r---- libm-2.28.so
00007faf32df0000      4K rw--- libm-2.28.so
00007faf32df1000      8K r---- libpcre.so.3.13.3
00007faf32df3000    328K r-x-- libpcre.so.3.13.3
00007faf32e45000    120K r---- libpcre.so.3.13.3
00007faf32e63000      4K r---- libpcre.so.3.13.3
00007faf32e64000      4K rw--- libpcre.so.3.13.3
00007faf32e65000    532K r---- libcrypto.so.1.1
00007faf32eea000   1656K r-x-- libcrypto.so.1.1
00007faf33088000    572K r---- libcrypto.so.1.1
00007faf33117000      4K ----- libcrypto.so.1.1
00007faf33118000    192K r---- libcrypto.so.1.1
00007faf33148000      8K rw--- libcrypto.so.1.1
00007faf3314a000     16K rw---   [ anon ]
00007faf3314e000    116K r---- libssl.so.1.1
00007faf3316b000    308K r-x-- libssl.so.1.1
00007faf331b8000    104K r---- libssl.so.1.1
00007faf331d2000      4K ----- libssl.so.1.1
00007faf331d3000     36K r---- libssl.so.1.1
00007faf331dc000     16K rw--- libssl.so.1.1
00007faf331e0000      8K rw---   [ anon ]
00007faf331e9000      4K r---- ld-2.28.so
00007faf331ea000    120K r-x-- ld-2.28.so
00007faf33208000     32K r---- ld-2.28.so
00007faf33210000      4K r---- ld-2.28.so
00007faf33211000      4K rw--- ld-2.28.so
00007faf33212000      4K rw---   [ anon ]
00007ffd5fe7a000    132K rw---   [ stack ]
00007ffd5ff3d000     12K r----   [ anon ]
00007ffd5ff40000      8K r-x--   [ anon ]
 total          3538460K

Procinfo shows use of some 3.6gb of RAM

me@api-server-02:~$ cat /proc/861/status
Name:   amqproxy
Umask:  0022
State:  S (sleeping)
Tgid:   861
Ngid:   0
Pid:    861
PPid:   496
TracerPid:      0
Uid:    1013    1013    1013    1013
Gid:    1018    1018    1018    1018
FDSize: 512
Groups: 33 1017 1018
NStgid: 861
NSpid:  861
NSpgid: 861
NSsid:  496
VmPeak:  3620496 kB
VmSize:  3595920 kB
VmLck:         0 kB
VmPin:         0 kB
VmHWM:    394108 kB
VmRSS:    394060 kB
RssAnon:          387452 kB
RssFile:            6608 kB
RssShmem:              0 kB
VmData:  3583136 kB
VmStk:       132 kB
VmExe:       896 kB
VmLib:      4880 kB
VmPTE:      2412 kB
VmSwap:        0 kB
HugetlbPages:          0 kB
CoreDumping:    0
Threads:        4
SigQ:   0/31797
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000001001000
SigCgt: 00000001a0814442
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 0000003fffffffff
CapAmb: 0000000000000000
NoNewPrivs:     0
Seccomp:        0
Speculation_Store_Bypass:       vulnerable
Cpus_allowed:   f
Cpus_allowed_list:      0-3
Mems_allowed:   00000000,00000001
Mems_allowed_list:      0
voluntary_ctxt_switches:        24003617
nonvoluntary_ctxt_switches:     139748

Installing Crystal failed during Vagrant machine provisioning

Reproducing steps:

vagrant up debian9

Output

Warning: apt-key output should not be parsed (stdout is not a terminal)
Executing: /tmp/apt-key-gpghome.Al6wzHKRRU/gpg.1.sh --keyserver keys.gnupg.net --recv-keys 09617FD37CC06B54
gpg: keyserver receive failed: No name
GPG error: http://dist.crystal-lang.org/apt crystal InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 09617FD37CC06B54
The repository 'http://dist.crystal-lang.org/apt crystal InRelease' is not signed.

Is this still supported?
Should the vagrant file follow the installation instruction on crystal-lang.org? https://crystal-lang.org/install/on_debian/#official-crystal-deb-repository

problem with DNS load balansing

Hello

We got a error with upstream domain name in GCP.

installed on the server:
Consul client with DNS on the localhost: 8600
dnsmask on localhost:53, which redirects all requests of the consul domain to localhost:8600

configuration /etc/resolve.conf:

cat /etc/resolv.conf
domain us-central1-a.c.production-***.internal
search us-central1-a.c.production-***.internal. c.production-***.internal. google.internal. consul. node.consul.
nameserver 127.0.0.1
nameserver 169.254.169.254

in consul we have service name witch resolved each node in rabbit cluster:

host rabbitmq-analytics.service.consul
rabbitmq-analytics.service.consul has address 10.0.15.223
rabbitmq-analytics.service.consul has address 10.0.15.224
rabbitmq-analytics.service.consul has address 10.0.15.249

Installed from relaeses amqproxy

amqproxy -v
0.3.6

then we run amqproxy and got errors:

/usr/bin/amqproxy -d -l 127.0.0.1 -p 5673 amqp://rabbitmq-analytics.service.consul
Proxy upstream: rabbitmq-analytics.service.consul:5672
Proxy listening on 127.0.0.1:5673
Client connection accepted from 127.0.0.1:57532
Upstream connection could not be established
Client connection closed from 127.0.0.1:57532
Unhandled exception in spawn: No address found for rabbitmq-analytics.service.consul:5672 over TCP (Socket::Error)
  from ???
  from ???
  from ???
  from ???
  from ???
Client connection accepted from 127.0.0.1:57534
Upstream connection could not be established
Client connection closed from 127.0.0.1:57534
Unhandled exception in spawn: No address found for rabbitmq-analytics.service.consul:5672 over TCP (Socket::Error)
  from ???
  from ???
  from ???
  from ???
  from ???
Client connection accepted from 127.0.0.1:57538
Upstream connection could not be established
Client connection closed from 127.0.0.1:57538
Unhandled exception in spawn: No address found for rabbitmq-analytics.service.consul:5672 over TCP (Socket::Error)
  from ???
  from ???
  from ???
  from ???
  from ???
Client connection accepted from 127.0.0.1:57540
Client connection accepted from 127.0.0.1:57542
Upstream connection could not be established
Client connection closed from 127.0.0.1:57542
Unhandled exception in spawn: No address found for rabbitmq-analytics.service.consul:5672 over TCP (Socket::Error)
  from ???
  from ???
  from ???
  from ???
  from ???

We try build from source amqproxy with new version Crystal and it worked fine.

Please make a release built on the new version of the crystal.
Please indicate the version of the crystal on which release is assembled

Problem using rabbitmq transaction

Hi, I'm using RabbitMQ v3.8.9 with AMQProxy v0.4.7 and it seems that AMQProxy can't handle transactions.

Here is a portion of AMQProxy's logs showing the error when one of my service tries to use a transaction:

10 clients		 16 upstreams
11 clients		 16 upstreams
Upstream closed connection: FRAME_ERROR - cannot decode <<>>
2020-10-16T18:27:13.270997652Z 
Upstream connection closed when returned
2020-10-16T18:28:19.077851087Z 
10 clients		 15 upstreams
9 clients		 15 upstreams

When the service is directly connected to the broker, transactions are working fine.

Is this a known limitation/bug of AMQProxy? If so, is this something that can be supported in future versions? If I can provide any additional informations, let me know.

Thank you in advance.

Just a bit of context why I use transaction in this service: I know that transactions are generally not recommended since they can significantly decrease the throughput (according to this). However, I specifically need them in this service because I'm sending a large amount of messages (few thousands) on a work queue. If I don't use a transaction, the consumers will start sending processed messages back even though the publishing service is not done dispatching the messages, thus causing weird race conditions.

Memory Leak in 0.4.2

In our QA environment, over the course of two hours the proxy process rose to over 2GB of memory consumed before crashing.

0.4.0 does not seem to exhibit the issue. 0.4.1 does not work with my application, so I'm not able to confirm that the leak was introduced there or not.

Steps to reproduce:

1. Produce messages on a loop through amqproxy v0.4.2 using the produce.php file in the attached tar.
   while : ; do php producer.php proxy ; done
2. View the process in top, noting that the amount of memory consumed always increases over time, and memory is never released.

I performed the above steps simultaneously to both 0.4.0 and 0.4.2 for 30 minutes or so, resulting in the below screenshot of top, note the VIRT column.

This screenshot is a consumer host that has been running the proxy for about 2 hours.

bernard_test.tar.gz

Systemctl log Error

amqproxy[1367]: 2021-03-22 15:58:49 +08:00: Receiving AMQ::Protocol::Frame::Channel::Close(@channel=1, @bytesize=96, @reply_code=404, @reply_text="ExchangeNotExist, ReqId:60584E39433839BF6765A9DE, ErrorHelp[60584E39433839BF6765A9D6]", @classid=60, @MethodID=40) but no client to delivery to
amqproxy[1367]: 2021-03-22 16:00:50 +08:00: Receiving AMQ::Protocol::Frame::Channel::Close(@channel=1, @bytesize=96, @reply_code=404, @reply_text="ExchangeNotExist, ReqId:60584EB2433839BF67671EC5, ErrorHelp[60584EB2433839BF67671EBB]", @classid=60, @MethodID=40) but no client to delivery to
amqproxy[1367]: 2021-03-22 16:00:57 +08:00: Receiving AMQ::Protocol::Frame::Channel::Close(@channel=1, @bytesize=96, @reply_code=404, @reply_text="ExchangeNotExist, ReqId:60584EB93937334A54B4D7D8, ErrorHelp[60584EB93937334A54B4D7D3]", @classid=60, @MethodID=40) but no client to delivery to
amqproxy[1367]: 2021-03-22 16:01:33 +08:00: Receiving AMQ::Protocol::Frame::Basic::QosOk(@channel=1, @bytesize=4) but no client to delivery to
amqproxy[1367]: 2021-03-22 16:01:33 +08:00: Upstream closed connection: ChannelNotFind
amqproxy[1367]: 2021-03-22 16:01:33 +08:00: Upstream connection closed when returned
amqproxy[1367]: 2021-03-22 16:02:24 +08:00: Receiving AMQ::Protocol::Frame::Channel::Close(@channel=1, @bytesize=96, @reply_code=404, @reply_text="ExchangeNotExist, ReqId:60584F10344230DF117686FC, ErrorHelp[60584F10344230DF117686F4]", @classid=60, @MethodID=40) but no client to delivery to
amqproxy[1367]: 2021-03-22 16:03:00 +08:00: Receiving AMQ::Protocol::Frame::Channel::Close(@channel=1, @bytesize=96, @reply_code=404, @reply_text="ExchangeNotExist, ReqId:60584F34383038457786EFA4, ErrorHelp[60584F34383038457786EF9B]", @classid=60, @MethodID=40) but no client to delivery to
amqproxy[1367]: 2021-03-22 16:03:26 +08:00: Receiving AMQ::Protocol::Frame::Channel::Close(@channel=1, @bytesize=96, @reply_code=404, @reply_text="ExchangeNotExist, ReqId:60584F4E433839BF6768CDB2, ErrorHelp[60584F4E433839BF6768CDAC]", @classid=60, @MethodID=40) but no client to delivery to
amqproxy[1367]: 2021-03-22 16:05:49 +08:00: Receiving AMQ::Protocol::Frame::Channel::Close(@channel=1, @bytesize=96, @reply_code=404, @reply_text="ExchangeNotExist, ReqId:60584FDD433745A76B0C6DAE, ErrorHelp[60584FDD433745A76B0C6D9F]", @classid=60, @MethodID=40) but no client to delivery to

Such As this

Strange timing

48core*200G server with zero LA :

via proxy

require_once __DIR__ . '/vendor/autoload.php';

use PhpAmqpLib\Connection\AMQPStreamConnection;
use PhpAmqpLib\Message\AMQPMessage;

$t1 = microtime(true);
$connection = new AMQPStreamConnection('localhost', 5673, 'guest', 'guest');

$channel = $connection->channel();

$channel->queue_declare('hello', false, true, false, false);

$msg = new AMQPMessage('Hello World!');
$channel->basic_publish($msg, '', 'hello');

echo " [x] Sent 'Hello World!'\n";

$channel->close();
$connection->close();

$t2 = microtime(true);

echo ($t2-$t1);

[x] Sent 'Hello World!'
0.35578203201294

direct to rabbitmq:

 [x] Sent 'Hello World!'
0.0085749626159668

355ms vs 8ms, what I'm doing wrong?

ClientProvidedName

Hi, I configurated ClientProvidedName, but when I checked at rabbit the connection_name is empty or null and doesn't appear at client property, in attachment the problem. Can anyone help me?

Missing new releases

Last release is 0.5.0

Last tag is 0.5.7

AMQP proxy closes connection if message contains timestamp in ms

We were consuming messages from RabbitMQ queues through the amqpproxy.

We noticed that we had one specific queue which caused our consuming application to loose the connection to RabbitMQ.

After some analysing we found that the messages in that particular queue contained a timestamp in ms as shown in the screenshot.
We played around with the timestamp and the connection did not die if the timestamp was in seconds instead of milliseconds.

Guess it would be nice to not close the connection if the timestamp is in milliseconds.

This is the error message the amqpproxy gave us:

amqp-proxy    | Unhandled exception in spawn: Invalid time: seconds out of range (ArgumentError)
amqp-proxy    |   from ???
amqp-proxy    |   from ???
amqp-proxy    |   from ???
amqp-proxy    |   from ???
amqp-proxy    |   from ???

AMQProxy should re-use connections for RPC

Publishing to non-existent exchange using confirm select not cleaning up channel

Server is closing channel when confirm fails from exchange not existing when publishing. Think the channel should be cleaned up when receiving confirm errors.

To get around this is there some way to force close the channel in the proxy?

Retrying connection to AMQP broker

Looks like I can authenticate now but connection is lost after a while, most likely during send or receive.

Unhandled exception in spawn: AMQ::Protocol::Frame::Body-end was 123, expected 206 (AMQ::Protocol::Error::InvalidFrameEnd)
  from /usr/share/crystal/src/gc/boehm.cr:71:5 in '->'
  from /usr/share/crystal/src/fiber.cr:255:3 in '???'
  from ???

Unhandled exception in spawn: AMQ::Protocol::Frame::Body-end was 10, expected 206 (AMQ::Protocol::Error::InvalidFrameEnd)
  from /usr/share/crystal/src/gc/boehm.cr:71:5 in '->'
  from /usr/share/crystal/src/fiber.cr:255:3 in '???'
  from ???

Unhandled exception in spawn: AMQ::Protocol::Frame::Body-end was 116, expected 206 (AMQ::Protocol::Error::InvalidFrameEnd)
  from /usr/share/crystal/src/gc/boehm.cr:71:5 in '->'
  from /usr/share/crystal/src/fiber.cr:255:3 in '???'
  from ???

Unhandled exception in spawn: AMQ::Protocol::Frame::Body-end was 99, expected 206 (AMQ::Protocol::Error::InvalidFrameEnd)
  from /usr/share/crystal/src/gc/boehm.cr:71:5 in '->'
  from /usr/share/crystal/src/fiber.cr:255:3 in '???'
  from ???

Unhandled exception in spawn: AMQ::Protocol::Frame::Body-end was 115, expected 206 (AMQ::Protocol::Error::InvalidFrameEnd)
  from /usr/share/crystal/src/gc/boehm.cr:71:5 in '->'
  from /usr/share/crystal/src/fiber.cr:255:3 in '???'
  from ???

Unhandled exception in spawn: AMQ::Protocol::Frame::Body-end was 105, expected 206 (AMQ::Protocol::Error::InvalidFrameEnd)
  from /usr/share/crystal/src/gc/boehm.cr:71:5 in '->'
  from /usr/share/crystal/src/fiber.cr:255:3 in '???'
  from ???

Unhandled exception in spawn: Could not write the full body (AMQ::Protocol::Error::FrameEncode)
  from lib/amq-protocol/src/amq/protocol/properties.cr:136:23 in '???'
  from src/amqproxy/server.cr:112:13 in '???'
  from src/amqproxy/pool.cr:17:7 in '->'
  from /usr/share/crystal/src/fiber.cr:255:3 in '???'
  from ???

[Question] Possible to limit the number of connections to RabbitMQ?

I took a look at the code (note: I have zero experience of Crystal), but could not find anything related to configuring this.

I'm asking this since RabbitMQ metrics are indicated that the number of connections grows endlessly until amqproxy gets OOM killed by k8s. The metrics show thousands of connections on RabbitMQ's side but I know I don't have that many clients connecting to RabbitMQ (and each client is limited in the number of connections it can opened simultaneously).

Having such a configurable parameter would help control the proxy's pool size according to known usage pattern. For instance, if I know that I'll have at most 200 clients each with 1 active connection, I'll limit the proxy to 250 or so.

Thank you in advance.

Concurrent connections are not limited, breaking RabbitMQ server

My use-case consists of openresty responding with empty_gif and then pushing the necessary data into a queue in RabbitMQ. Using the AMQP protocol directly, RabbitMQ was able to cope with the number of messages, ingesting at a rate of about 300-400 messages per second. When I installed and configured AMQProxy, that number just skyrocketed, reaching over 600msgs/s.

At one point RabbitMQ just stopped ingesting, and the management interface in CloudAMQP became unavailable and displayed an error message (see below).

I do see the maxConnections setting in the example.ini file, although I don't see mentions of said upper limit in the codebase, nor documentation on how to set the maximum number of connections to hold open. When I checked the management interface, there were over 6000 connections open, with many of them with a status of blocking or blocked. The instance is a Big Bunny.

The load test is being run with Apache AB, ab -k -c200 -n100000 -r 'http://<some address>/1234567890.gif'. Openresty (nginx) was running with a single worker allowing a maximum of 400 worker connections (which I believed would limit concurrency, and therefore connections). The publisher is using basic_publish, and consumers for this particular queue are in a different server. Successfully published messages are also successfully processed.

We have other pieces of software using CloudAMQP, and AMQProxy will be very helpful in case we are able to get this prototype right.

Server log sample:

(... alarm setting and clearing repeated many many times above ...)

**********************************************************
*** Publishers will be blocked until this alarm clears ***
**********************************************************

=INFO EVENT==== Fri, 18 Jan 2019 19:27:50 GMT ===
vm_memory_high_watermark clear. Memory used:839187024 allowed:842953236

=WARNING EVENT==== Fri, 18 Jan 2019 19:27:50 GMT ===
memory resource limit alarm cleared on node 'rabbit@<server name>'

=INFO EVENT==== Fri, 18 Jan 2019 19:27:51 GMT ===
vm_memory_high_watermark set. Memory used:955837240 allowed:842953236

=WARNING EVENT==== Fri, 18 Jan 2019 19:27:51 GMT ===
memory resource limit alarm set on node 'rabbit@<server name>'.

**********************************************************
*** Publishers will be blocked until this alarm clears ***
**********************************************************

(... there were many of the message below ...)

=ERROR EVENT==== Fri, 18 Jan 2019 19:28:13 GMT ===
closing AMQP connection <0.16260.3> (x.x.x.x:46390 -> y.y.y.y:5672):
{handshake_timeout,frame_header}

=WARNING EVENT==== Fri, 18 Jan 2019 19:29:07 GMT ===
file descriptor limit alarm set.

********************************************************************
*** New connections will not be accepted until this alarm clears ***
********************************************************************

=WARNING EVENT==== Fri, 18 Jan 2019 19:29:18 GMT ===
file descriptor limit alarm cleared

=INFO EVENT==== Fri, 18 Jan 2019 19:30:06 GMT ===
started TCP Listener on 0.0.0.0:5672

=INFO EVENT==== Fri, 18 Jan 2019 19:30:07 GMT ===
started SSL Listener on 0.0.0.0:5671

=INFO EVENT==== Fri, 18 Jan 2019 19:30:07 GMT ===
rabbit_stomp: default user 'guest' enabled

=INFO EVENT==== Fri, 18 Jan 2019 19:30:07 GMT ===
started STOMP TCP Listener on 0.0.0.0:61613

=INFO EVENT==== Fri, 18 Jan 2019 19:30:07 GMT ===
started STOMP SSL Listener on 0.0.0.0:61614

=INFO EVENT==== Fri, 18 Jan 2019 19:30:07 GMT ===
rabbit_web_stomp: listening for HTTP connections on 0.0.0.0:15674

=INFO EVENT==== Fri, 18 Jan 2019 19:30:07 GMT ===
Management plugin started. Port: 15672

=INFO EVENT==== Fri, 18 Jan 2019 19:30:07 GMT ===
Statistics database started.

=INFO EVENT==== Fri, 18 Jan 2019 19:30:07 GMT ===
Server startup complete; 15 plugins started.
 * rabbitmq_shovel_management
 * rabbitmq_federation_management
 * rabbitmq_management
 * rabbitmq_web_dispatch
 * webmachine
 * mochiweb
 * rabbitmq_management_agent
 * rabbitmq_shovel
 * rabbitmq_federation
 * rabbitmq_web_stomp
 * rabbitmq_stomp
 * rabbitmq_consistent_hash_exchange
 * amqp_client
 * cowboy
 * sockjs

One can see the spikes in connections:

and how they seem unrelated to the number of messages in the queues, over that time period:

UNEXPECTED_FRAME - expected content header for class 60, got non content header frame instead

hi, I am use
client: amqplib 0.7.1 , https://www.npmjs.com/package/amqplib
rabbitmq-server: 3.7
It's will got the fllowing error on my client

UNEXPECTED_FRAME - expected content header for class 60, got non content header frame instead

any idea, thanks.

This is my docker-compose yaml file of mq deploy.

version: "3.7"
services:
  mq-n1:
    image: rabbitmq:3.7-management
    hostname: rabbit-c1
    network_mode: host
    restart: always
    environment:
      - RABBITMQ_ERLANG_COOKIE=rabbit-cookie
      - RABBITMQ_DEFAULT_USER=private
      - RABBITMQ_DEFAULT_PASS=private
  mq-proxy:
    image: cloudamqp/amqproxy
    network_mode: host
    restart: always
    command: amqp://127.0.0.1:5672 --port=55672

Unknown error (read_loop)

Anybody know this error?

missing connection reset when pipe is broken

there was already an update by @Kaggggggga #8 to close the upstream when the pipe is broken on reading. but that closing should also be placed in the write method. i worked out a pull request #16

Connection closed by AMQP Proxy Shutdown

Hello, I am evaluating this on my staging environment and after a few hours running some of our PHP workers started throwing exceptions when consuming from the Proxy.

I would like to know if there is any restriction over connections (like connections can only be done via sockets), and what could be the cause of the following exception:

Our pods didn't have any restart so I think AMQP Proxy shutdown by itself somehow.

Docker build failed

$ git clone https://github.com/cloudamqp/amqproxy /tmp/amqproxy
Cloning into '/tmp/amqproxy'...
$ cd /tmp/amqproxy
$ docker build -t amqproxy:latest .
Step 1/8 : FROM crystallang/crystal
latest: Pulling from crystallang/crystal
d7c3167c320d: Pulling fs layer
131f805ec7fd: Pulling fs layer
322ed380e680: Pulling fs layer
6ac240b13098: Pulling fs layer
22da88a5d530: Pulling fs layer
6ac240b13098: Waiting
22da88a5d530: Waiting
322ed380e680: Verifying Checksum
322ed380e680: Download complete
131f805ec7fd: Verifying Checksum
131f805ec7fd: Download complete
d7c3167c320d: Verifying Checksum
d7c3167c320d: Download complete
6ac240b13098: Verifying Checksum
6ac240b13098: Download complete
d7c3167c320d: Pull complete
22da88a5d530: Verifying Checksum
22da88a5d530: Download complete
131f805ec7fd: Pull complete
322ed380e680: Pull complete
6ac240b13098: Pull complete
22da88a5d530: Pull complete
Digest: sha256:afd58df611c04807961539f884bbe8dfd25ea9fd54132cbea2c01f5475689ebd
Status: Downloaded newer image for crystallang/crystal:latest
---> ee810d01edd1
Step 2/8 : WORKDIR /app
---> Running in b5e32584d9af
Removing intermediate container b5e32584d9af
---> 99bff998ec9a
Step 3/8 : COPY . .
---> ba5593b5f51b
Step 4/8 : RUN shards build --release --production
---> Running in 1f9eb895802c
Resolving dependencies
Fetching https://github.com/cloudamqp/amq-protocol.cr.git
Installing amq-protocol (0.3.12)
Building: amqproxy
Error target amqproxy failed to compile:
Showing last frame. Use --error-trace for full trace.
In lib/amq-protocol/src/amq/protocol/frames.cr:416:21
416 | bytes += io.write_bytes(@reply_code, format)
^
Error: no overload matches 'Int64#+' with type Nil
Overloads are:

• Int64#+(other : Int8)
• Int64#+(other : Int16)
• Int64#+(other : Int32)
• Int64#+(other : Int64)
• Int64#+(other : Int128)
• Int64#+(other : UInt8)
• Int64#+(other : UInt16)
• Int64#+(other : UInt32)
• Int64#+(other : UInt64)
• Int64#+(other : UInt128)
• Int64#+(other : Float32)
• Int64#+(other : Float64)
• Number#+()
  The command '/bin/sh -c shards build --release --production' returned a non-zero code: 1

Cant compile with crystal 0.27.0

Crystal 0.27.0 [c9d1eef8f] (2018-11-01)
LLVM: 4.0.0
Default target: x86_64-unknown-linux-gnu

Please find the output of below
crystal build --verbose --error-trace --release -o bin/amqproxy src/amqproxy.cr

server.listen(listen_address, listen_port)
       �[32;1m^~~~~~�[0m

in src/amqproxy/server.cr:31: instantiating 'TCPServer.class#open(String, Int32)'

      TCPServer.open(address, port) do |socket|
                �[32;1m^~~~�[0m

in src/amqproxy/server.cr:31: instantiating 'TCPServer.class#open(String, Int32)'

      TCPServer.open(address, port) do |socket|
                �[32;1m^~~~�[0m

in src/amqproxy/server.cr:41: expanding macro

            spawn handle_connection(client, client.remote_address)
            �[32;1m^�[0m

in macro 'spawn' /usr/share/crystal/src/concurrent.cr:97, line 11:

   1.   
   2.     ->(
   3.       
   4.         __arg0 : typeof(client),
   5.       
   6.         __arg1 : typeof(client.remote_address),
   7.       
   8.       
   9.       ) {
  10.       spawn(name: nil) do
�[32;1m>�[0m�[1m 11.         handle_connection(�[0m
  12.           
  13.             __arg0,
  14.           
  15.             __arg1,
  16.           
  17.           
  18.         )
  19.       end
  20.     
  21.       }.call(client, client.remote_address)
  22.     
  23.   

instantiating 'handle_connection(TCPSocket+, Socket::IPAddress)'
in src/amqproxy/server.cr:84: instantiating 'AMQProxy::Client.class#new(TCPSocket+)'

      c = Client.new(socket)
                 �[32;1m^~~�[0m

in src/amqproxy/client.cr:11: instantiating 'negotiate_client(IO+)'

      @vhost, @user, @password = negotiate_client(@socket)
                                 �[32;1m^~~~~~~~~~~~~~~~�[0m

in src/amqproxy/client.cr:53: instantiating 'AMQProxy::AMQP::Frame.class#decode(IO+)'

      start_ok = AMQP::Frame.decode(socket).as(AMQP::Connection::StartOk)
                             �[32;1m^~~~~~�[0m

in src/amqproxy/amqp/frames.cr:38: undefined method 'at' for Slice(UInt8)

        frame_end = payload.at(size)
                            �[32;1m^~�[0m

================================================================================

Slice(UInt8) trace:

  src/amqproxy/amqp/frames.cr:35

            payload = Bytes.new(size + 1)
            �[32;1m^~~~~~~�[0m

  src/amqproxy/amqp/frames.cr:35

            payload = Bytes.new(size + 1)
                            �[32;1m^~~�[0m

  /usr/share/crystal/src/slice.cr:83

      def self.new(size : Int, *, read_only = false)


  /usr/share/crystal/src/slice.cr:83

      def self.new(size : Int, *, read_only = false)
                                  �[32;1m^�[0m

  /usr/share/crystal/src/slice.cr:84

        {% unless T <= Int::Primitive || T <= Float::Primitive %}
        �[32;1m^�[0m

  /usr/share/crystal/src/slice.cr:89

        new(pointer, size, read_only: read_only)
        �[32;1m^~~�[0m

  /usr/share/crystal/src/slice.cr:67

      def initialize(@pointer : Pointer(T), size : Int, *, @read_only = false)


  /usr/share/crystal/src/slice.cr:67

      def initialize(@pointer : Pointer(T), size : Int, *, @read_only = false)
      �[32;1m^�[0m

  /usr/share/crystal/src/slice.cr:67

      def initialize(@pointer : Pointer(T), size : Int, *, @read_only = false)

Memory leak in 0.4.4

In version 0.4.4 there is a still a memory leak similar to #25 It is leaking at ~40kbps.

This is a memory usage graph between 0.4.0 & 0.4.4

Error reading from upstream end of file reached

We get a connection error in the amqproxy after our php AMQP client does a basic_publish. The messages still goes through but the proxy connection to the upstream AMQP server opens and then closes every time instead of staying open.

Here is some sample code showing how we're publishing from our PHP:

try {
   $conn = self::getConnectionForVhost($vhost);
   $ch = $conn->channel();

    $eventMsg = array(
      'properties' => $properties
     );
     $amqpMessage = new AMQPMessage(
        $json_encode($eventMsg),
            array(
              'content_type' => 'application/json',
               'delivery_mode' => 2
             )
         );
      $ch->basic_publish($amqpMessage, $exchange, $routingKey);
      $ch->close();
      $conn->close();
 } catch (Exception $e) {
      error_log('ERROR PUBLISHING EVENT ' . $e->getMessage());
      return false;
 }

amqproxy log

Connected to upstream 18.233.168.244:5671
Error reading from upstream: #<IO::EOFError:End of file reached>
Problem closing upstream: #<Errno:Error reading socket: Connection reset by peer>

RabbitMQ Server log (using cloud AMQP dedicated node version 3.7.10 Erlang 20.1)

[INFO] 2019-08-23 22:25:41.969 [info] <0.18763.5995> connection <0.18763.5995> (54.175.220.154:40096 -> 10.50.144.213:5671): user 'preprod' authenticated and granted access to vhost 'preprod'
[WARNING] 2019-08-23 22:26:33.565 [warning] <0.18763.5995> closing AMQP connection <0.18763.5995> (54.175.220.154:40096 -> 10.50.144.213:5671, vhost: 'preprod', user: 'preprod'): client unexpectedly closed TCP connection

EDIT: added rabbit + erlang version

BernardPHP connections hang when polling an empty queue

What I've observed is when our BernardPHP / PhpAmqplib consumer is targeting the Amqproxy, it will hang when no messages remain in the queue. Essentially, it's not able to poll for messages and will hang at a variable point in time when the queue depth is zero.

I put together a set of php files that reproduces the issue.
bernard_test.tar.gz

Question

If I want to use it in a Containerized environment like Kubernetes or DC/OS , what is the best practice :

1. having 1 centralized proxy for all the micro services ?
2. having an embedded proxy inside the service image?

Thanks