doitintl / banias

Opinionated serverless event analytics pipeline

Banias

Blog Post

Banias (Arabic: بانياس الحولة‎; Hebrew: בניאס‬) is the Arabic and modern Hebrew name of an ancient site developed around a river once associated with the Greek god Pan.

So, like the flow of the Banias, events are flowing into our system. So we decided to build a reference architecture and actual implementation of event analytics pipeline. You can take the code as it is and use it or use it a design reference.

Banias Architecture:

• API receiving the events data from the producers (e.g. web apps, mobile app or backend servers)
• The events are sent to Google Pub/Sub topic
• Apache Beam/Google Cloud Dataflow streams the events into BigQuery with or without mutations or agregations

Installation

API Deployment

Apache Beam/Google Cloud Dataflow Deployment

banias's Issues

Resource requests and limits

Need to add Resource requests and limits to the banias/frontend/deploy/frontend-deployment.yaml

Add support for AWS Kinesis Streams / Firehose

https://docs.aws.amazon.com/firehose/latest/dev/writing-with-sdk.html#putrecordbatch

Consider exporting monitoring metrics to opencensus.io

Consider exporting monitoring metrics to opencensus.io instead of just Prometheus

Ingress

banias/frontend/deploy/frontend-service.yaml has to be changed to Ingress to support global ip and federation

From DavidB

went over it, looks ok but I miss comments so don't really understand what many parts of code do without looking at the import packages first. For example, g.Add that uses oklog which i don't know what it is and so on. So from the main func code i could hardly understand what it does, especially with things like 'logger.Info("exit", zap.Error(g.Run()))'. I think g.Run() should be more explicit because it's your main loop and my eye automatically skips log lines. Also pls highlight somewhere what the endpoints are (/track and /metrics), unless you have a dedicated client in which case it's not that important.
collector looks fine, needs more comments though. Things like 'if counter != 7' need to be more clear
Also I'd do some wrapper around tag, for example instead of tag.NewKey("banias/keys/code") it would be mytag.Key("code") and instead of "banias/measures/request_latency" it would be mytag.Measure("request_latency") because 'banias' is constant and keys/measures/etc are categories

Invalid argument in backend/Makefile

There are a few occurrences of --GCSSchemasBucketName=(SCHEMAS_BUCKET) in the Makefile, which cause errors. I presume it is meant to read --GCSSchemasBucketName=$(SCHEMAS_BUCKET).

Is the current Makefile working for you with this error?