bbc / alephant-broker

Brokers requests for alephant components

Alephant::Broker

Brokers requests for rendered templates, retrieved from S3 or a HTML endpoint.

Installation

Add this line to your application's Gemfile:

gem 'alephant-broker'

And then execute:

bundle install

Or install it yourself as:

gem install alephant-broker

Usage

The Broker is capable of retrieving rendered templates from either S3 or a HTML endpoint (e.g. alephant-publisher-request). This must be decided when creating an instance of the Broker, as a load strategy is given as a parameter (see below for examples).

Barebones

S3 Load Strategy

require 'alephant/broker'

config = {
  :s3_bucket_id      => 'test_bucket',
  :s3_object_path    => 'foo',
  :lookup_table_name => 'test_lookup'
}

request = {
  'PATH_INFO'      => '/component/foo',
  'QUERY_STRING'   => 'variant=bar',
  'REQUEST_METHOD' => 'GET'
}

Alephant::Broker::Application.new(
  Alephant::Broker::LoadStrategy::S3.new,
  config
).call(request).tap do |response|
  puts "status:  #{response.status}"
  puts "content: #{response.content}"
end

HTML Load Strategy

require 'alephant/broker'

class UrlGenerator < Alephant::Broker::LoadStrategy::HTTP::URL
  def generate(id, options)
    "http://example-api.com/data?id=#{id}"
  end
end

request = {
  'PATH_INFO'      => '/component/foo',
  'QUERY_STRING'   => 'variant=bar',
  'REQUEST_METHOD' => 'GET'
}

Alephant::Broker::Application.new(
  Alephant::Broker::LoadStrategy::HTTP.new(UrlGenerator.new),
  {}
).call(request).tap do |response|
  puts "status:  #{response.status}"
  puts "content: #{response.content}"
end

Note

The HTML load strategy relies upon being given a URLGenerator, which is used to generate the URL of the HTML endpoint (see below for example). The class must:

• be implemented within your own application.
• extend Alephant::Broker::LoadStrategy::HTTP::URL.
• include a #generate method which takes id (string) and options (hash) as parameters.

require 'alephant/broker'
require 'rack'

class UrlGenerator < Alephant::Broker::LoadStrategy::HTTP::URL
  def generate(id, options)
    "http://api.my-app.com/component/#{id}?#{to_query_string(options)}"
  end

  private

  def to_query_string(hash)
    Rack::Utils.build_query hash
  end
end

Revalidate Strategy

require 'alephant/broker'

config = {
  :s3_bucket_id                => 'test_bucket',
  :s3_object_path              => 'foo',
  :lookup_table_name           => 'test_lookup',
  :sqs_queue_name              => 'test_queue',
  :elasticache_config_endpoint => 'test_elisticache'
  :revalidate_cache_ttl        => 30
}

Alephant::Broker::Application.new(
  Alephant::Broker::LoadStrategy::Revalidate::Strategy.new,
  config
).call(request).tap do |response|
  puts "status:  #{response.status}"
  puts "content: #{response.content}"
end

The "Revalidate" strategy uses the following process flow:

And uses the following AWS resources:

• S3 - for storage of rendered components
• DynamoDB - for recording S3 component locations
• Elasticache - for caching S3 content and recording "inflight" messages
• SQS - for communication between the broker and renderer

Rack App

Create config.ru using example below, and then run:

rackup config.ru

require 'alephant/broker'
require 'alephant/broker/load_strategy/http'

class UrlGenerator < Alephant::Broker::LoadStrategy::HTTP::URL
  def generate(id, options)
    "http://example-api.com/data?id=#{id}"
  end
end

run Alephant::Broker::Application.new(
  Alephant::Broker::LoadStrategy::HTTP.new(UrlGenerator.new),
  {}
)

Cache version number

The broker looks for a configuration value elasticache_cache_version and if it exists it uses it to construct the cache key. This allows the cache to be busted if the data in the cache changes, or for any other reason that it needs to be invalidated.

This version is added as a header to the response in the following format:

X-Cache-Version: {CACHE_VERSION}

Contributing

1. Fork it!
2. Create your feature branch: git checkout -b my-new-feature
3. Commit your changes: git commit -am 'Add some feature'
4. Push to the branch: git push origin my-new-feature
5. Create a new Pull Request.

Feel free to create a new issue if you find a bug.

alephant-broker's Issues

Allow header prefix to be set via config

Currently the header is hardcoded into the gem, it would be better if it was grabbed from the config.

Inconsistent batch request components

Problem

Scenario:

• A batch request for 5 components.
• 4 of the components are retrieved from the cache, however 1 component's cache has expired/not exist.
• Broker requests content for the expired component, and updates the cache before returning it.

In this situation, one of the components may have more recent data which could contradict the other 4 components (e.g. stock price).

Solution

• If 1+ of many components in a batch request is required to fetch new content, then all components must follow suit.
• This can be given as an option with the JSON body of the batch POST request (defaults to false).

Implementation

• Before the point where the components are loaded in parallel, the broker must evaluate if all requested components exist in the cache.
• This boolean value is then handed to the LoadStrategy, which uses it to determine if it is necessary to check the cache.
• Make sure that we aren't checking the cache twice.

Remove Multi

Multi is no longer used and has become stale after the changes seen in #14 - therefore meaning that it has become an overhead to support.

It can be removed within a PR (reverted in future if necessary).

Params passed to URL Generator

The HTTP load strategy (L#58-59) currently hands the id and options of the component to the URL generator.

Both of these parameters should default to nil, as users may not need them.

README Example

I'm not sure if the README example (https://github.com/BBC-News/alephant-broker#s3-load-strategy) is correct?

config = {
  :bucket_id         => 'test_bucket',
  :path              => 'foo',
  :lookup_table_name => 'test_lookup'
}

We can see in https://github.com/BBC-News/alephant-broker/blob/master/lib/alephant/broker/load_strategy/s3.rb#L59-L60 that Broker.config tries to get bucket and path using the keys s3_bucket_id and s3_object_path which don't correlate to the example config used in the README.

Cache Dynamo Read/Write

Caching Dynamo read/write may help with hitting limits with Dynamo. Had issues in Elections.

This should be a very small cache time.

Now part of: https://jira.dev.bbc.co.uk/browse/CONNPOL-2821

Loading components in parallel (Request::Batch)

Problem

This functionality was removed within #14 as we were seeing errors, most likely due to race conditions. It may also be due to using Peach which is now quite an old gem.

Solution

Refactor Request::Batch so that the following method loads components in parallel.

def components_for(env)
  env.data['components'].map do |c|
    @component_factory.create(
      c['component'],
      batch_id,
      c['options']
    )
  end
end

Caching non 2xx responses

Currently in the http load strategy we cache the content regardless of the error code returned:

https://github.com/BBC-News/alephant-broker/blob/master/lib/alephant/broker/load_strategy/http.rb#L33-L35

We should look at either not caching these responses or caching them for a short or proportionally shorter cache time.

Allow cache class to take a TTL to set on a per object basis

Currently the TTL is specified when we create the Dalli::Client, it would be nice if we could set this on a per object basis.