isabella232 / harp

HTTP-to-AMQP Relay Project

HARP

HTTP-based AMQP message Relay Project

Introduction

HARP is an experimental project to facilitate the bi-directional relay of AMQP message traffic between two or more decoupled AMQP brokers using HTTP. The use of HTTP as a relay protocol is particularly convenient in situations where firewall traversal is required, since most corporate firewalls are already configured to properly forward HTTP traffic. Thus, HARP is particularly well suited to relay AMQP messages in remote (on-premise) client-to-cloud scenarios, allowing remote client-side and cloud-hosted messaging clients to communicate over AMQP transparently & securely while preserving “standard” HTTP cloud service connection semantics.

HARP versus Shovel

Compared to RabbitMQ’s Shovel relay mechanism ( http://www.rabbitmq.com/shovel.html ), HARP has several differences and potential advantages:

• HARP uses web-standard HTTP as the relay protocol, rather than AMQP.
• HARP relay forwarding is dynamic. Clients on either end of the relay connection can programmatically request forwarding of specific queues and/or exchanges at runtime. This means HARP can be used effectively as a relay for dynamic AMQP routing patterns such as RPC-over-AMQP.
• HARP is capable of selectively forwarding specific routing key values from a Direct exchange. This allows you to set up a Direct exchange where some consumer queues are local and others are remote.
• HARP is specifically designed to support cloud-based architectures. Each relay channel has a client-side and a (servlet-based) server-side, and a single server instance can support multiple clients. This facilitates the establishment of a large number of AMQP client message bus deployments that exchange messages with a single, centralized (potentially cloud-hosted) message bus.

Building HARP

HARP is a multi-module Maven project, so you will need to have Maven installed, as well as a java-7-compliant JDK (JDK 6 also works, but you will need to change the jdk.version property in the main POM file).

All of the project dependencies are hosted in Maven central except for the Google Protobuf compiler, which you can download here. Currently the project is pinned to Protobuf version 2.4.1, although newer versions can be used with a few simple POM file tweaks. The build expects the protoc compiler to be installed in /usr/local/bin, although, again, this can be changed easily in the harp-amqp-relay-lib POM.

Once Protobuf is installed properly, a simply running “mvn install” at the top level should build everything (fingers crossed!).

It’s built. What now?

Once you get the build done, all of the “good stuff” will be in the target directories of the three Maven modules. These modules and what they produce are described below:

• harp-amqp-relay-lib
  • This is the main relay logic library module. It builds a jar file, harp-amqp-relay-lib-.jar, which contains the logic and APIs necessary to build relay clients or server-side implementations.
• harp-amqp-relay-web
  • This builds a server-side implementation of the HARP relay, packaged as a standard web servlet. The target WAR archive, harp-amqp-relay-web.war, can be drop-in deployed in standard servlet containers, including recent releases of Tomcat and Jetty.
  • The servlet accepts several initialization parameters that control its connection to the server-side AMQP broker. These parameters are listed and described in src/main/webapp/WEB-INF/web.xml. By default the serlet will connect to a broker running on localhost with the standard AMQP port and no username/password.
• harp-amqp-relay-apps
  • This project build a set of useful sample and utility command line applications, including a sample relay client implementation. These command line apps can found in target/appassembler/bin, and come in both *nix and Windows (BAT) versions:
    • simple-relay-client
      • A basic example of a client-side relay application. While this can be used as a standalone relay client, the real intent is to illustrate how to use the client-side relay APIs incorporated into harp-amqp-relay-lib, which can easily be embedded into any client java application.
    • amqp-forward
      • A simple client to generate queue or exchange forwarding requests from the command line. Again, this is meant as an illustration, since in most cases the API for making a forwarding request would be embedded into message clients & called programmatically.
    • amqp-generator
      • A test client for generating simple, textual AMQP messages and sending them to a targeted local queue or exchange. Useful, along with amqp-monitor, for testing relay functionality.
    • amqp-monitor
      • A test client that pulls and displays the contents of AMQP messages sent to the specified local queue. Useful, along with amqp-generator, for testing relay functionality. Note that the monitor automatically generates a relay forwarding request for the monitored queue.

Setting up an example

To see how HARP works, you can set up a simple example installation. For this we need two hosts running an AMQP bus, which we will call “SERVER” and “CLIENT”.

1. Install, if necessary, Tomcat or Jetty on the SERVER host and deploy harp-amqp-relay-web.
2. On the the CLIENT host, run simple-relay-client, specifying the root URL of the Tomcat/Jetty instance running on SERVER (e.g. “simple-relay-client http://SERVER.mydomain.com:8080/”).
   1. At this point, the relay between the two brokers on SERVER and CLIENT has been established.
3. On SERVER, run amqp-monitor, specifying a test queue name (e.g. “amqp-monitor testA”).
4. On CLIENT run amqp-generator, specifying the same test queue name (e.g. “amqp-generator testA”) and type a few lines of text. These lines should appear on SERVER in the terminal window running amqp-monitor. This shows the full end-to-end relay chain has been established. Messages are being sent by amqp-generator to the local queue (on CLIENT) and then are transparently forwarded to the queue with the same name on SERVER, where they are picked up by amqp-monitor.
5. You can also try running the generator/monitor with a different queue name with the hosts reversed – in other words run the monitor on CLIENT and the generator on SERVER. This demonstrates the bi-directional nature of the relay.
   1. You can also test the auto-message-generation features of amqp-generator. This can be used to generate a large number of messages quite rapidly, which is useful for testing relay performance.

Implementation Notes

• HARP uses the RabbitMQ client API to connect to the AMQP broker. This should work with non-RabbitMQ broker implementations, including QPid.
• HARP cycles the HTTP connection between client and server regularly (they cycle rate can be controlled by parameters specified by the client when connecting). Note that decreasing the cycle rate/increasing the connection time may improve efficiency somewhat, particularly for HTTPS connections, but if the connection time is too long intervening infrastructure, such as web proxies, may cause timeout errors. HARP uses an HTTP/1.1 connection to reduce the overhead from connection cycling.