jollopre / harmoniser Goto Github PK

The Open Telemetry initiative has defined a set of semantic conventions to address messaging operations.

We would like to provide all the minimum and necessary data for publish/consume through harmoniser thats permits an easy collection of telemetry data through external agents such as Honeycomb, NR or the like.

• Manage channel creation efficiently
• Ensure consumer creation is thread-safe
• Start consuming once on_delivery method is defined
• Offer possibility for receiving on_cancellation from rabbitMQ
• Handle errors as transparent as possible
• Trap OS signals smooth

The existing Harmoniser::Publisher creates connection/channel/exchange right when harmoniser_publisher class method gets called. Since we want to avoid all sort of problems related to pre-fork servers like unicorn, passenger, we believe is better to defer the resources creation until the first publication wants to be made.

• Update Dockerfile to its latest ruby
• Update RabbitMQ docker to its latest
• Update bunny gem dependency to its latest
• Update Gemfile.lock to ensure reproducible development builds

We'd like to introduce the concept of Topology into harmoniser, so that we can explicitly configure:

• exchanges
• queues
• bindings

This should simplify the building of Publisher/Consumer since it will be assumed that exchanges/queues/bindings are already defined.

We'd like to have a dedicated channel for each publisher/subscriber class. Some of the reason described below:

• Isolation: Issues in one channel such as high message processing time will not affect the rest of subscribing cases.
• Fine-Grained control: Channel specific settings such as prefetch, acknowledgments can be configured differently for each case
• Resource utilisation: For subscribing cases, we can tune the concurrency per queue thanks to the ConsumerWorkPool increasing or decreasing the throughput on demand

We'd like to introduce CI github action containing capturing the following:

• Code building
• Definition of service container for RabbitMQ
• Linting through standardrb
• Capability to run specs

This action should be triggered anytime push to branch happen or PR is opened. Consider including also:

• Ruby version badge
• Workflow status badge

The existing Harmoniser::Subscriber creates connection/channel/queue right when harmoniser_subscriber class method gets called. Since we want to avoid all sort of problems related to pre-fork servers like unicorn, passenger, we believe is better to defer the resources creation until the consumption starting is marked explicitly

The existing signal handler only takes place when harmoniser is the main process, however, during the process of reading the require file, a Harmoniser.connection could happen. If there are errors trying to connect, the connection is attempted forever unless a SignalException is received, see here.

This code could become problematic when the exit is instructed if:

• harmoniser is not the main process, i.e. bundle exec harmoniser was not launched and
• the process where harmoniser run does not define Signal.trap beforehand.

In order to prevent this scenario, we'd like to make sure that ONLY exit is instructed when harmoniser is the main process, otherwise and under the event of capturing SignalException, we are going to raise it again.

We would like to introduce a functionality that allows publishing messages into RabbitMQ exchanges. This functionality should:

• permit easy set up with sensible defaults
• flexibility for setting up advance configuration for exchanges
• manage channel creation efficiently
• handle errors as transparent as possible
• handle OS signals smooth
• ensure that publishing runs on a thread-safe environment

Our first approach to perform data sync with publisher/subscribers passes by using a message broker, concretely rabbitmq. We would like to have things in place such as:

• development/test environment with rabbitmq
• Configuration object accepting properties to establish connection through Bunny
• Adequate handing/memoization of Bunny session taking into account network failures and abrupt rabbitmq server disconnection

We'd like to offer default options for:

• timeouts
• recovery_attempt_started
• recovery_completed

Also, make sure that connection is close regardless whether or not Harmoniser is used as a Ruby separated process or not.

We'd like to update the README to explain:

• what's this gem
• how to use it
• explain the different concepts

Offer a CLI for running harmoniser in the background

Instead of passing a Bunny instance, we prefer to pass a hash of options, that map with the options available to instantiate Bunny::Session class. This will allow us to better control the following:

• Create a unique Bunny::Session shared across Harmoniser module
• Ensure the creation is thread-safe
• Prevent leaking Bunny::Session across the library through the delegation of specific methods like (create_channel or start)

The existing connection to RabbitMQ performed here does not have retry in place for when the server is not available.
Although an established connection is properly handling re-connection, it is true that under failure for connection, the process can be break.

We would like to introduce a retry strategy for connection establishment so that harmoniser can have:

• Improved resilience: For instance, intermitent network failures while trying to connect
• Enhanced availability: Permitting the process to continue as well as reducing its chances of terminated harmoniser process with non zero exit code

In order to better debug an application when errors occur at:

• channel level or
• through processing messages from a queue

We'd like to introduce default handler for the above mentioned scenarios.

NOTE: Ideas to structure the content can be taken from Bunny guides