rupnikj / qtopology

Distributed stream processing layer

QTopology

QTopology is a distributed stream processing layer, written in node.js.

NPM package: https://www.npmjs.com/package/qtopology

Documentation https://qminer.github.io/qtopology

Travis CI: https://travis-ci.org/qminer/qtopology

Installation

npm install qtopology

Intro

QTopology is a distributed stream processing layer, written in node.js.

It uses the following terminology, originating in Storm project:

• Topology - Organization of nodes into a graph that determines paths where messages must travel.
• Bolt - Node in topology that receives input data from other nodes and emits new data into the topology.
• Spout - Node in topology that reads data from external sources and emits the data into the topology.
• Stream - When data flows through the topology, it is optionaly tagged with stream ID. This can be used for routing.

When running in distributed mode, qtopology also uses the following:

• Coordination storage - must be resilient, receives worker registrations and sends them the initialization data. Also sends shutdown signals. Implementation is custom. QTopology provides REST-based service out-of-the-box, but the design is similar for other options like MySQL storage etc.
• Worker - Runs on single server. Registers with coordination storage, receives initialization data and instantiates local topologies in separate subprocesses.
  • Leader - one of the active workers is announced leader and it performs leadership tasks such as assigning of topologies to workers, detection of dead or inactive workers.

Quick start

See documentation