datasystemsgrouput / adaptive-watermarks

An approach to apply concept drifts and ADWIN on the streams event time to reason about the progress of watermarks

Adaptive Watermarks: A Concept Drift-based Approach for Predicting Event-Time Progress in Data Streams

This repository contains the source code for the adaptive watermark generator.

This depends on the ADWIN library. As well as Apache Flink.

Idea

Event-time based stream processing is concerned with analyzing data with respect to its generation time. In most of the cases, data gets delayed during its journey from the source(s) to the stream processing engine. This is known as out-of-order data arrival. Among the different approaches for out-of-order stream processing, low watermarks are proposed to inject special records within data streams, i.e., watermarks. A watermark is a timestamp that indicates that no data with a timestamp older than the watermark should be observed later on. Any element as such is considered a late arrival. Watermark generation is usually periodic and heuristic-based. The limitation of such watermark generation strategy is its rigidness regarding the frequency of data arrival as well as the delay data may encounter. Here, we propose an adaptive watermark generation strategy. Adaptively, it can decide when to generate a new watermark and with what timestamp without a priori adjustment. We treat the change in data arrival frequency and change in delays as concept drifts in stream data mining. We use Adaptive Window (ADWIN) as our concept drift sensor for the change of the distribution of arrival rate and delay. We have implemented our approach on top of Apache Flink. We compare our approach to the baseline periodic watermark generation.

How to run?

Import the maven project into your IDE.

You can unpack the data (csv files) in the data folder to folder on your computer.

You can check the example run in ~\src\main\java\ee\ut\cs\dsg\adaptivewatermark\flink\StreamingJob

You can pass a command line arguments as

--input "path to input file" --output "path to output file" --adaptive true --allowedLateness 100 --oooThreshold 1 --sensitivity 1 --sensitivityChangeRate 1.0 --windowWidth 1000 --period 10 parameters:

• Input: Path to the input file that should have timestamps as long value,
• Output: Path to the output file that shall contain the window start and end and the number of elements assigned to the window,
• Adaptive: if set to true will run the adaptive watermark generator. Otherwise, if will run the baseline periodic watermark generator,
• Allowed lateness (m): This parameter is relevant to the periodic watermark generator. It is the number of milliseconds by which the watermark is delayed behind the max timestamp seen,
• Out-of-order threshold (l): This parameter is used for the adaptive watermark generator. It controls the percentage of out of order (late) records with respect to the total number of records in a chunk. By default it is set to 1.0 meaning no restriction on out-of-order records,
• Sensitivity (delta): This parameter is used to control how sensitive ADWIN is to the change in data arrival rate. Default value is 1 which means most sensitive,
• Sensitivity change rate: the rate by which the sensitivity parameter is decreased/increased in case of late arrivals go above the threshold (l),
• Window width: In milliseconds, controls the width of the tumbling time window in the pipeline. Default is 1000 milliseconds,
• Period (s): In milliseconds controls the periodic watermark assigner interval between watermark generations.