openmatchmaking / pathfinder

Async WebSocket-over-RabbitMQ reverse proxy

pathfinder

An asynchronous WebSocket-over-RabbitMQ reverse proxy, based on Tokio and futures-rs crates.

Features

• Configuring a behaviour of the reverse proxy via CLI options and YAML files
• Communicating with Auth/Auth microservice for validating a JSON Web Token, getting a list of permissions before getting an access to other microservices
• Transferring requests to the certain microservices via RabbitMQ queues and returning responses in JSON format

Usage

USAGE:
    pathfinder [FLAGS] [OPTIONS]

FLAGS:
    -s, --secured     Enable the SSL/TLS mode for connections with RabbitMQ
    -h, --help        Prints help information
    -V, --version     Prints version information

OPTIONS:
    -c, --config <config>                                  Path to a custom settings file [default: ]
    -i, --ip <ip>                                          The used IP for a server [default: 127.0.0.1]
    -p, --port <port>                                      The listened port [default: 9000]
    -l, --log-level <log_level>                            Verbosity level filter of the logger [default: info]
        --rabbitmq-host <rabbitmq_host>                    The used host by RabbitMQ broker [default: 127.0.0.1]
        --rabbitmq-port <rabbitmq_port>                    The listened port by RabbitMQ broker [default: 5672]
        --rabbitmq-virtual-host <rabbitmq_virtual_host>    The virtual host of a RabbitMQ node [default: vhost]
        --rabbitmq-user <rabbitmq_username>                A RabbitMQ application username [default: user]
        --rabbitmq-password <rabbitmq_password>            A RabbitMQ application password [default: password]
        --ssl-cert <ssl_certificate>                       Path to a SSL certificate [default: ]
        --ssl-key <ssl_public_key>                         Path to a SSL public key [default: ]

Configuration file

For using a custom configuration for reverse proxy, you will need to specify -c (or --config) option with a path to a file. For example:

pathfinder --config=myconfig.yaml -p 8001

At the current stage of this project, reverse proxy is support only endpoints list, which is using for mapping URLs into certain RabbiMQ exchanges and queues. Each of those endpoints contains four fields:

• url - URL that specified by a client in each request. Required.
• routing_key - Means the name of topic (or queue) where will be storing the message. This topic (or queue) is listening by certain microservice. Required.
• request_exchange - Defines the name of exchange point for RabbitMQ, through which the reverse proxy should publish a message. Optional. Default: "open-matchmaking.direct"
• response_exchange - Defines the name of exchange point for RabbitMQ, through which the reverse proxy should consume a message. Optional. Default: "open-matchmaking.responses.direct"
• token_required - Defines does the endpoint need any extra checks for credentials before getting an access to it. Optional. Default: true.

Example

endpoints:
  - search:
      url: "/api/matchmaking/search"
      routing_key: "microservice.search"
  - leaderboard:
      url: "/api/matchmaking/leaderboard"
      routing_key: "microservice.leaderboard"
      request_exchange: "amqp.direct"
      response_exchange:  "open-matchmaking.default.direct"

Documentation

Information about why this reverse proxy was implemented you can find here.

Benchmarks

For performance benchmark was used the MacBook Pro 13" (mid 2012) with 2,5Ghz Intel Core i5 (2x cores, 4 threads) processor and 16Gb memory. The tests were running in the following conditions:

1. All required microservices and external resources were running via Docker containers.
2. The test cluster was using 12 of 16Gb of available memory and all available cores / threads from my notebook.
3. During the tests were simulated 1000 concurrent users via Gatling tool, with splitting the traffic onto 5 stages so that it gradually increased.

The test were made in two passes:

1. Relies on processing requests without validating tokens and passing the data as is to the Echo microservice.
2. Uses an information that necessary to register and to generate a token for getting an access to microservice:

• Communicating with Auth/Auth for registering a new user and generating JSON Web Token
• Token from the previous step must be used with data for getting an access to the Echo microservice. During this simulation step the token will be verified by the Auth/Auth microservice before passing a requests further

The repository with benchmarks can be found here.

Metric name \ Test name	Without token	With Json Web Token (JWT)
Startup RAM usage, Mb	2.21	2.23
Max RAM usage, Mb	69.54	66.48
Avg RAM usage, Mb	42.02	48.54
Max CPU usage, %	114.04	120.398
Avg CPU usage, %	69.54	45.66
Total requests	6075	2241
Successfully processed	6075	2241
Error responses	0	0
Min response time, ms	3	15
Max response time, ms	13243	38945
Mean response time, ms	2450	8948
Std dev response time, ms	3933	13833
50th percentile, ms	17	25
75th percentile, ms	4824	20372
95th percentile, ms	11358	37672
99th percentile, ms	12968	38587
Requests / sec	162.723	53.357

Note #1: Keep in mind that the response time and RPS (requests per second) are much lower on the second pass because necessary to communicate with Auth/Auth microservice a couple of times before doing an actual work.
Note #2: Potentially, reverse proxy could process more requests per second which is mostly depends on performance of the used microservice, rather than reverse proxy itself.

License

The pathfinder is published under BSD license. For more details read the LICENSE file.

pathfinder's Issues

Custom message routing for the proxy engine

In actual moment of time a developer should define all desired endpoints in configuration file that have YAML format and will be passed as a parameter in CLI before the start. It's fine for a couple of endpoints, however in the case of multiple microservices the efforts for maintaining this table significally increases, when necessary to specfy all of those.

To minimize a maintenance efforts, we can optimize this process, so that it will use much more cleaner and easier way to handle those cases:

• Give an opportunity to specify "dynamic paths", so that the developer will define only a couple of generic resources on the base of some template
• For some corner cases let to developers to use the "static routing" (that currently using for all of it)

As for example I will use the following configuration, that must be supported and valid:

endpoints:
  - search:
      url: "/api/matchmaking/search"
      routing_key: "matchmaking.users.{action}"
      request_exchange: "open-matchmaking.matchmaking.search.direct"
  - users:
      url: "/api/users/{action}"
      routing_key: "auth.users.{action}"
      request_exchange: "open-matchmaking.auth.{action}.direct"

In this example we have 2 endpoints:

1. The search endpoints that uses static routing and gives to proxy an information about which AMQP queue must be used for sending a request.
2. The users endpoints that uses dynamic routing for balancing all incoming requests to AMQP queues / request exchanges of Auth/Auth microservice:
   • An action in curly braces is a parameter that stores a suffix and going to be used later in the templates for generating names of AMQP queues, request and response exchanges. For example the /auth/users/retrieve/ URL will be transformed by proxy engine into auth.users.retrieve, because the action has the retrieve value and was used in /api/users/{action} template, where that value was used.
   • The same rules applies to routing_key, request_exchange and response_exchange, if were specified any substitutions.

Any patterns or substitutions must not be defined only in the end of strings. They can be specified in the middle of the template, or in some cases may be even starts with it.
Besides of it the developer must replace "microservice" onto "routing_key", that must be used in configuration files instead, because shows that this field is required for passing messages.

For the cases, when the passed URL in the incoming request format is valid, but AMQP queues, request and response exchanges may not exist, we must return an error with a message like "Endpoint wasn't found" instead of waiting infinitely. It can be done with creating a "error queue" per each client that linked to the special exchange (or may be even "open-matchmaking.responses.direct") for tracking a messages that can't be delivered to the certain microservice.

Replace "redis-async" crate onto "redis-rs"

In actual moment "redis-rs" is actively updating for letting an access to developers to use async implementation out of the box, that gives us to write asynchronous applications and services with "tokio" and "future-rs" crates.

Reusing channels for publishing and consuming messages

After performance tests I've found an interesting bug that can be found in the case a big amount of users that constantly doing API calls and use a separate channel for each action. And when the limit is reached, reverse proxy is starting to deny to process any requests. In addition to it, it is writing messages in log, like this:

[2019-01-09][13:37:29][pathfinder::proxy][ERROR] Error in RabbitMQ client. Reason: The maximum number of channels for this connection has been reached

As a solution for it, we can reuse created channels for each connected user to reverse proxy, so that:

• 1 channel uses for publishing
• 1 channel uses for consuming

For the beginning, I think, it will more than enough use 1 connection and 2 separate channels per each user, so that approximately the single connection can be used for ~32k users simultaneously. And if we face with that issue once again, then, probably, we will need to move each type of channels (publishing, consuming, etc.) per each connection.

Optimizing a processing client requests

An actual implementation of the pathfinder project is using a separate connections (RabbitMQ and Redis) per each active user it leads to inefficient use of resources and increasing a response time (especially for the first requests). What can be done for solving this issue:

• Use a single RabbitMQ connection per each pathfinder instance with a separate channel per each active user, which is much cheaper (in terms of used resources) and faster
• Replace manually getting tokens from Redis instance and validating them in run-time onto the separate request to Auth/Auth microservice, so that it will make all required checks and will return a list of permissions, assigned to the certain user.

This solution let to use the server resources more efficiently and will minimize the time, required for processing requests per each message which is coming from a user via WebSockets.

Update an existing codebase after Tokio reform

Recently the tokio-rs crate was reformed so that existing APIs was changed. Therefore, for avoiding a broken pathfinder app, necessary to apply a new changes to Tokio:

• Replace in Cargo.toml file the tokio-core crate onto the tokio crate
• Remove deprecated parts of code with setting up the event loop via tokio_core::reactor::Core
  and replace it onto tokio::executor::current_thread task executor instead
• Make ensure that &handle as a part of old Core functionality won't broke the existing code (for example, when we're using middlewares)

Name clashes in the Rust ecosystem

There are already at least two other Rust projects using the Pathfinder name:

https://crates.io/crates/pathfinder
https://github.com/pcwalton/pathfinder

So you might want to consider picking another name.