gnagel / promiscuous

Replicate data across your applications

Introduction

Promiscuous is a publisher-subscriber framework for easily replicating data across your Ruby applications.

Motivation

If you hit the Amazon.com gateway page, the application calls more than 100 services to collect data and construct the page for you.

— Werner Vogels, CTO, Amazon.com, 2006

When it comes to scaling a team, having just one codebase adversely impacts productivity and performance. A more sustainable approach is to adopt a service-oriented architecture (SOA), with a system composed of several loosely coupled applications, each existing in isolation with its own database. In this manner, each service can be tested separately, deployed separately and even owned separately by developers. Unfortunately, it has not always been easy to achieve this with Ruby.

Promiscuous facilitates designing Ruby based SOA services. It does this by watching models in publisher applications and sending corresponding model operations on a common message bus powered by RabbitMQ. Each subscriber has its own queue to receive messages asynchronously.

Role in our Infrastructure

At Crowdtap, we use Promiscuous as the central tier of our system. It replicates a subset of our core models backed by MongoDB to internal services, such as our e-commerce store on PostgreSQL and our analytics engine on ElasticSearch.

Difference from traditional replication

By using Redis to synchronize and order operations, Promiscuous guarantees that a subscriber never sees out of order updates, even when using shards. This guarantee considerably reduces the complexity of an application, while improving its robustness.

Rails Quick Tutorial

1. Preparation

We need to few things for the promiscuous tutorial:

• The AMQP broker RabbitMQ up and running.
• The key-value storage system Redis (at least 2.6) up and running.
• Two Rails applications with the promiscuous gem installed.
• Both applications must be running on separate databases.
• Both applications must have a User model (ActiveRecord or Mongoid) with two attributes name and email.

2. Publishing

By including the Promiscuous publisher mixin, we can publish the model attributes:

# app/models/user.rb on the publisher app
class User
  include Promiscuous::Publisher
  publish :name, :email
end

Note: With ActiveRecord on the publisher side, promiscuous only supports PostgreSQL at the moment. You also need to change max_prepared_transactions = 10 in the config of PostgreSQL.

3. Subscribing

Similarly to the publisher app, we can subscribe to the attributes:

# app/models/user.rb on the subscriber app
class User
  include Promiscuous::Subscriber
  subscribe :name, :email

  after_create { Rails.logger.info "Hi #{name}!" }
end

4. Replication

The subscriber must listen for new data to arrive. Promiscuous has a worker that we can launch with the following command:

bundle exec promiscuous subscribe

You should start the subscriber first, otherwise the appropriate queues will not be created. From now on, you should see the queue in the RabbitMQ web admin page. Create a new user in the publisher's Rails console with:

User.create(:name => 'Yoda')`

You should see the message "Hi Yoda!" appearing in the log file of the subscriber.

Promiscuous in Depth

Features and Recipes

• Attributes
• Ephemerals & Observers
• Polymorphism
• Embedded Documents
• Foreign Keys
• Namespace Mapping
• Promiscuous DSL

The Replication Mechanism

• Pipeline Description

Configuration

• Configuration Options
• Subscriber CLI Options

Testing

• Exporting Publishers Definitions
• Unit Testing
• Integration Testing
• Gemify your Apps

Going to Production

• RabbitMQ & Redis
• Initial Sync
• Error Handling
• Instrumentation
• Managing Deploys
• Setup with Unicorn, Resque, etc.

Miscellaneous

• Roadmap
• Coding Restrictions
• Limitations
• FAQs
• License