qminer / qtopology

Distributed stream processing layer

QMiner

QMiner is an analytics platform for large-scale real-time streams containing structured and unstructured data. It is designed for scaling to millions of data points on high-end commodity hardware, providing efficient storage, retrieval and analytics mechanisms with real-time response.

Project homepage

Examples

• Twitter sentiment extraction
• Recommendation system

Prerequisites

• node.js v14.x, v12.x, v10.x and npm 5.3 or higher To test that your node.js version is correct, run node --version and npm --version. Not compatible with nodejs v0.10 or older.

Windows

• Visual C++ Redistributable Packages for Visual Studio 2015 download vcredist_x64.exe for node.js x64 or vcredist_x86.exe for node.js x86.

Install

To Install the qminer package run:

npm install qminer

Test. To test if the package was successfully installed run:

node -e "require('qminer'); console.log('OK')"

Compile from Source

If you wish to compile the package from source, please address the instructions.

Documentation

The package has a full documentation available online:

• JavaScript API
  • qm module
  • la module
  • analytics module
• C++ Documentation

Acknowledgments

QMiner is developed by Department of Artificial Intelligence at Jozef Stefan Institute, Quintelligence, Qlector and other contributors.

The authors would like to acknowledge funding from the European Union Seventh Framework Programme, under Grant Agreements 288342 (XLike), 611346 (XLime), 611875 (Symphony), 317534 (Sophocles), 318452 (Mobis), 600074 (NRG4Cast), 619437 (Sunseed), 632840 (FI-Impact) and 612329 (ProaSense).

This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 636160-2 (Optimum).

qtopology's Issues

RSS spout

A spout that would read data from RSS or ATOMcfeed and emit parsed posts.

Dir-monitor spout

Spout that periodically monitors given directory and reports new files

Create npm package

GET spout

REST spout

Test spout

Spout that emits predefined tuples

REST spout enhancements

• optionally attach request data (path, user ip, etc.)
• limit pending queue

Implement common coordination storage

Shutdown step of whole local topology

Write demo with qminer

GET bolt

Implement flow control

• Spouts and bolts should be monitored with respect to emitted tuples
• Queuing mechanism should be implemented before each bolt to allow some flexibility regarding processed tuples
  • Using high_water and low_water signals (from settings)
• Bolts need to ack their data and thus send control feedback to previous nodes in the topology

Topology weight indication

Topology definition can include its weight - an approximation of the resource intensity of running this topology.
Leader should take this indication into account when doing load distribution.

Default weight is 1.

Bolts & spouts - in-process

Create initial version of in-process version of bolts and spouts

File-appender bolt

Bolt should append each incoming data as plain text into given file.
Creates file if it doesn't exist, appends if exists.

Robustness - call timeouts

Define common coordination storage

Implement required changes in workers

System telemetry

Advanced flow control

Each bolt should have a queue in front of it and handle multi-spout inbound requests

JSON file spout

Write docs

Implement needed changes in coordinator

Add system (out-of-the-box) bolts

• Filter bolt
• Delay bolt
• Telemetry bolt

Create Github pages

Initial tests of distributed mode

Breaking change: context as parameter to init()

Breaking change

context should be sent as parameter to init(), not to constructor

child-process restarter - error limit

Add limit to child-process restarting attempts, similar to what worker code uses.

Topology error handling

If topology repeatedly crashes, worker should try up to 5 restarts per 10 minutes and then disable it.

Local topologies should execute in separate processes

Bolt - allow_parallel

Bolts by design execute in sync manner - the topology takes care of waiting.
Allow (opt-in) async execution (non-blocking) when bolt settings require it.

Added std node - constant fields attacher

Bolts & spouts - child process

Create initial version of child-process version of bolts and spouts

Set up Travis CI

Unify API - inproc and subproc

GET bolt

result should be text, not Buffer

GREP bolt

Initialization data for initialization code

Heartbeat

Make heartbeat part of the engine

Stream-assigner BOLT

This bolt will assign different stream ids to data, depending on the content. Match is determined using the same pattern matching mechanism as in filter bolt.

process-stdout spout

Initialization step of whole local topology

Worker affinity

JsonSchema - warnings

Schema validator should report errors when optional fields names are miss-spelled

Dashboard - close button for ribbon doesn't work

Dashboard - worker details - leadership flag

All worker s are flagged as leaders when "More >>" is clicked.

Unit tests

POST bolt

REPL spout

Stream mechanism

Spouts and bolts can tag tuples with stream names, while bolts can subscribe to specific stream names.