cdapio / tigon

High Throughput Real-time Stream Processing Framework

Tigon

Introduction

Tigon is an open-source, real-time, low-latency, high-throughput stream processing framework.

Tigon is a collaborative effort between Cask Data, Inc. and AT&T that combines technologies from these companies to create a disruptive new framework to handle a diverse set of real-time streaming requirements.

Cask Data has built technology that provides scalable, reliable, and persistent high-throughput event processing with high-level Java APIs using Hadoop and HBase.

AT&T has built a streaming engine that provides massively scalable, flexible, and in-memory low-latency stream processing with a SQL-like query Language.

Together, they have combined to create Tigon.

There are many applications that can take advantage of its features:

• Ability to handle extremely large data flows;
• Exactly-once event processing using an app-level Java API with consistency, reliability, and persistence;
• Streaming database using a SQL-like language to filter, group and join data streams in-memory;
• Runs collections of queries using pipelined query plans;
• Able to transparently handle complex record routing in large parallelized implementations;
• Runs and scales as a native Apache Hadoop YARN Application;
• Reads, writes, and tightly integrates with HDFS and HBase;
• Supports a significant amount of parallelization;
• Fault-tolerance and horizontal scalability without burdening the developer;
• Enterprise security features with debugging, logging, and monitoring tools; and
• Simpler programming model, tools and UI; and
• Open-source software and development process.

For more information, see our collection of Guides and other documentation.

Is It Building?

Builds

Bamboo Build
GitHub Version

Getting Started

Prerequisites

Tigon is supported on *NIX systems such as Linux and Macintosh OS X. It is not supported on Microsoft Windows.

To install and use Tigon and its included examples, there are a few prerequisites:

1. JDK 6 or JDK 7 (required to run Tigon; note that $JAVA_HOME should be set)
2. GCC
3. G++
4. Apache Maven 3.0+ (required to build the example applications)

Note: To run the TigonSQL Stream Engine outside of Tigon, libz, Perl 5.x, and Python 3.x are required.

Download

Pre-compiled sources and related files can be downloaded in a zip file: tigon-developer-release-0.2.1.zip.

Install

Once the download has completed, unzip the file in a suitable location.

Run Instructions

To run Tigon in standalone mode:

$ run_standalone.sh <path-to-flow-jar> <flow-class-name> <run-time-args>

To run Tigon in distributed mode:

$ run_distributed.sh <zookeeper-quorum> <hdfs-namespace>

Building from Source

You can also build Tigon directly from the latest source code:

git clone https://github.com/caskdata/tigon.git
cd tigon
mvn clean package -DskipTests -Pdist

After the build completes, you will have a distribution of Tigon under the tigon-distribution/target/ directory.

Take the tigon-sdk-<version>.zip file and unzip it into a suitable location.

Getting Started Guide

Visit our web site for a Getting Started Guide that will guide you through installing Tigon and running an example.

Where to Go Next

Now that you've had a look at the Tigon SDK, take a look at:

• Examples, located in the /tigon-examples directory of Tigon
• Online Examples (demonstrating basic features of Tigon) are located on-line
• Reference Applications:
  • AdNetworkFlow: Demonstrates using Tigon to write a realtime bidding (RTB) advertisement framework
  • ClickStreamFlow: Demonstrates using TigonSQL to write a webpage click stream data processing application.
• Developer Guides, located in the source distribution in /tigon-docs/developer-guide/source or online

CDAP

Tigon is the realtime stream processing framework used in the Cask Data Application Platform (CDAP). CDAP provides a high-level Dataset abstraction for User Data Stores and allows developers to interact with them in their flowlets. In addition to the Dataset abstraction, CDAP integrates Batch Processing with MapReduce and Spark support. By combining these with security, metrics, and an intuitive GUI, CDAP greatly simplifies Big Data application development. If you like the features and architecture of Tigon, we recommend that you take a look at CDAP.

How to Contribute

Interested in helping to improve Tigon? We welcome all contributions, whether in filing detailed bug reports, submitting pull requests for code changes and improvements, or by asking questions and assisting others on the mailing list.

Bug Reports & Feature Requests

Bugs and suggestions should be made by filing an Issue.

Pull Requests

We have a simple pull-based development model with a consensus-building phase, similar to Apache's voting process. If you’d like to help make Tigon better by adding new features, enhancing existing features, or fixing bugs, here's how to do it:

1. If you are planning a large change or contribution, discuss your plans on the cask-tigon-dev mailing list first. This will help us understand your needs and best guide your solution in a way that fits the project.
2. Fork Tigon into your own GitHub repository.
3. Create a topic branch with an appropriate name.
4. Work on the code to your heart's content.
5. Once you’re satisfied, create a pull request from your GitHub repo (it’s helpful if you fill in all of the description fields).
6. After we review and accept your request, we’ll commit your code to the cask/tigon repository.

Thanks for helping to improve Tigon!

Mailing List

Tigon User Group: [email protected]

Tigon Development Discussion: [email protected]

IRC Channel

Tigon IRC Channel #tigon on irc.freenode.net

License and Trademarks

Copyright © 2014 Cask Data, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Cask is a trademark of Cask Data, Inc. All rights reserved.

Apache, Apache HBase, and HBase are trademarks of The Apache Software Foundation. Used with permission. No endorsement by The Apache Software Foundation is implied by the use of these marks.

tigon's Issues

Support RTS generated sysTime

Add support to allow users to access sysTime (from RTS) through the java code.

COUNT() should return INT

As discussed earlier, since java doesn't support unsigned data types. Please change count() to return INT instead of UINT

Pings from GSEXIT

Each instance of GSEXIT should register and log metrics with the discovery server, similar to the behavior of FTAs.

Register @ http:///v1/announce-fta-instance
Log metrics @ http:///v1/log-metrics

Syntax error in generated files

UPDATED

There are missing semi colons in the auto generated file (hfta_0.cc) when the SQL query contains an INNER JOIN

Update: It is not just INNER JOIN, other queries may fail as well.

Schemas:

StreamSchema nameSchema = new StreamSchema.Builder()
.addField("uid", GDATFieldType.INT, GDATSlidingWindowAttribute.INCREASING)
.addField("name", GDATFieldType.STRING)
.build();
addJSONInput("nameInput", nameSchema);

StreamSchema ageSchema = new StreamSchema.Builder()
.addField("uid", GDATFieldType.INT, GDATSlidingWindowAttribute.INCREASING)
.addField("age", GDATFieldType.INT)
.build();
addJSONInput("ageInput", ageSchema);

Sample 1:

SELECT nI.uid, nI.name, aI.age INNER_JOIN FROM nameInput nI, ageInput aI WHERE nI.uid = aI.uid AND aI.age > 40

Error Log:

$ make
g++ -O3 -g -I ../../include -I ../../include/hfta -o hfta_0.o -c hfta_0.cc
hfta_0.cc:272:1: warning: control reaches end of non-void function [-Wreturn-type]
}
^
hfta_0.cc:383:15: warning: shift count >= width of type [-Wshift-count-overflow]
return( (0) >> 32);
^ ~~
hfta_0.cc:499:11: error: expected ';' after return statement
return 0
^
;
hfta_0.cc:649:11: error: expected ';' after return statement
return 0
^
;
2 warnings and 2 errors generated.
make: *** [hfta_0.o] Error 1

Sample 2

SELECT uid, COUNT(*) FROM nameInput nI GROUP BY uid

Error log:

$ make
gcc -O3 -g -I . -I ../../include -I ../../include/lfta -o lfta.o -c localhost_lfta.c
localhost_lfta.c:297:17: warning: equality comparison with extraneous parentheses [-Wparentheses-equality]
if((gb_attr_0 == t->aggr_table[probe].gb_var0)){
~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
localhost_lfta.c:297:17: note: remove extraneous parentheses around the comparison to silence this warning
if((gb_attr_0 == t->aggr_table[probe].gb_var0)){
~ ^ ~
localhost_lfta.c:297:17: note: use '=' to turn this equality comparison into an assignment
if((gb_attr_0 == t->aggr_table[probe].gb_var0)){
^~
=
localhost_lfta.c:757:17: warning: equality comparison with extraneous parentheses [-Wparentheses-equality]
if((gb_attr_0 == t->aggr_table[probe].gb_var0)){
~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
localhost_lfta.c:757:17: note: remove extraneous parentheses around the comparison to silence this warning
if((gb_attr_0 == t->aggr_table[probe].gb_var0)){
~ ^ ~
localhost_lfta.c:757:17: note: use '=' to turn this equality comparison into an assignment
if((gb_attr_0 == t->aggr_table[probe].gb_var0)){
^~
=
2 warnings generated.
g++ -O3 -g -o rts lfta.o -L../../lib -lgscplftaaux -lgscprts -lgscphost -lm -lgscpaux -lgscplftaaux -lclearinghouse -lresolv -lpthread -lgscpinterface -lgscpaux -lcurl
g++ -O3 -g -I ../../include -I ../../include/hfta -o hfta_0.o -c hfta_0.cc
hfta_0.cc:423:11: error: expected ';' after return statement
return 0
^
;
1 error generated.
make: *** [hfta_0.o] Error 1

Remove libcurl dependency

Remove dependency on libcurl (external library) by defining our own C functions to perform HTTP GET & POST in gshub.cpp

Update tigon-sql examples README

Currently it is empty https://github.com/caskdata/tigon/blob/develop/tigon-examples/tigon-sql/README

Fix SQL library test suite failures

• The test suite has to be made more robust.
• The generated test results and outputs should be placed under tigon-sql/target/test-suite-reports//

Remove signed variable code

For e.g. The external_fcns.def contains function definitions that use uint as the parameter or return type.

We should clean such references from the tigon repository.

Handle FTA instantiation failure

With the current setup we cannot handle FTA instantiation failure. For e.g. Lets say the HFTA could not be initialized, then it never announces itself and hence the java process can never identify that we are in an error state.

Note : The above issue occurred on my MAC because of too many GSTool processes in memory, which were occupying all of the available shared memory. Hence a new FTA could not be instantiated

Proposed solution:

Option I

Always register the FTA. Since an external process is responsible for making all the calls to the discover server, it can register for the FTA.

Option II (less preferred)

Add an additional call to share the number of pings to expect. The discovery server will expect the number of expected pings to be shared at a new REST end-point. The java program can then check if the correct number of processes have registered to detect a failure state. (Assumption : At runtime, we can evaluate the number of FTAs and GSEXITs expected to run.)

The drawback with this approach is that it will rely on a timeout to check the number of processes registered. On a slow machine a small timeout may fail, while on fast machines a big timeout would mean that it takes longer to detect failure.

Object as DataType for Out Emit Funtion

Hi, currently only the primitive types are supported to emit the java object from one flowlet to another. Therefore, an abstract or generic datatype cannot be send or pipeline. is it possible to add such support as I would like to send custom made objects. Secondly, currently Tigon only supports the old version of Guava library (ver 13.0.1), however the newest version of Guava is 18.0. Cheers