Xorlev/summingbird-hybrid-example

This is a modification of the summingbird-example provided by Twitter,
using a Kafka source for events and implementing the hybrid batch-realtime
approach.

PREREQUISITES
-------------

You need to have https://www.docker.com/ and http://www.fig.sh/ installed
so that you can launch memcache, zookeeper and kafka. Modify fig.yml and change the 
KAFKA_ADVERTISED_HOST_NAME to your IP address and run:
  fig up -d

Make sure all 3 initialized corectly
  fig ps

Example output from fig:
                Name                              Command               State                      Ports                    
---------------------------------------------------------------------------------------------------------------------------
summingbirdhybridexample_kafka_1       /bin/sh -c start-kafka.sh        Up      0.0.0.0:49155->9092/tcp                     
summingbirdhybridexample_memcached_1   memcached                        Up      0.0.0.0:49153->11211/tcp                    
summingbirdhybridexample_zookeeper_1   /opt/zookeeper-3.4.5/bin/z ...   Up      0.0.0.0:49154->2181/tcp, 2888/tcp, 3888/tcp 

This example also relies on a patched version of Tormenta which can be found https://github.com/upio/tormenta.git 
Check this package out and publish it to your local maven repository:
  sbt publishM2

RUNNING
---------------------------------

To demo the hybrid mode:

  sbt 'run-main summingbird.proto.RunHybrid'

You'll see a bunch of start-up messages, and then it should settle into regularly
spitting out some "sanity check" messages once a minute, both a detailed view of
what's been computed on a single key, and aggregate data about how many messages
have been produced and seen at various stages.  (Some small lag is expected, but
should remain fairly steady over time.)

Once every 5 minutes, the demo rolls over a new log file and kicks off a batch
computation to update the "offline" store.


Example output after running for a while:

14/03/27 13:31:30 INFO example.HybridRunner$: Offline: Some((BatchID.4653174,296))
14/03/27 13:31:30 INFO example.HybridRunner$: Online: (BatchID.4653174,Some(10))
14/03/27 13:31:30 INFO example.HybridRunner$: Hybrid: Some(306)
14/03/27 13:31:30 INFO example.RunHybrid$: Events Produced: 2875
14/03/27 13:31:30 INFO example.RunHybrid$: Events Ingested: 2875
14/03/27 13:31:30 INFO example.RunHybrid$: Events Counted (offline): 2784
14/03/27 13:31:30 INFO example.RunHybrid$: Events Counted (online): 81
14/03/27 13:31:30 INFO example.RunHybrid$: Events Counted (hybrid): 2865