htm-community / flink-htm Goto Github PK

Publish the first release of the library to Central.

All of life is out of balance and seems far away due to vision impairment occurring when viewing the authors profile! Please help...

Let's develop a connector to read Socrata streams.

Consider using the soda-java library, but be aware that its dependencies are fairly old (jackson 1, jersey 1). We would probably need to shade the dependencies and/or update them.

Alternatively, we could write something from scratch (consumer-side only, to begin with). The River source is written with Spray, which is easy to use. There are quite a few soda model classes (for geo data, timestamps, etc) and query classes that would need to be re-implemented.

It is possible to send a reset to HTM to indicate that a temporal sequence has ended. Investigate how to facilitate a reset in the flink-htm API.

A simple option would be to expose the network object to direct manipulation in the select function. But I would rather a more filter-like approach like HTMStream::resetOn(f: ResetFunction).

Is it better to reset before or after an event?

Update the library to use Flink 1.3 and likewise refresh the HTM dependency to 0.6.13.

Template:

DataStream<T> result = HTM.learn(stream, new Network())
				.select(new InferenceSelectFunction<R, T>() {
                                           @Override
                                            public T select(Tuple2<R, NetworkInference> inference) throws Exception {
					              	return inference.f1.getAnomalyScore();
                                             }
				});

The overrided select is never called. I verified that following the call hierarchy. This lead to no one output produced. Anyway, I verified the anomaly score, internally, is correctly computed.

Tried to resolve it, no great results reached. Please, fix it.

Flink's reliability is based on checkpointing of operator state. In flink-htm the critical state is that of the HTM network.

The operator code is well-prepared to support checkpointing. A state holder is allocated from Flink (see L63 of KeyedHTMInferenceOperator) into which the network instance is stored. Flink expects to use the serializer passed to it, and will automatically serialize the network at the appropriate time. The issue is that the network is not actually serializable (as of htm.java 0.6.5), and the job will crash if checkpointing is enabled.

Stretch goal: Flink supports both synchronous and asynchronous checkpointing. In the latter case, the operator is free to process elements while the state is being checkpointed. Depending on how the network state is ultimately checkpointed, consider it.

Flink provides a unit test framework for operators - or at least an integration test using an embedded Flink instance.

• Use a mock Network.
• Enhance the existing test (HTMIntegrationTest) to validate the output.
• Test keyed streams.
• Test the 'encoder input function' code (GenerateEncoderInputFunction), which takes a user type (i.e. case class, POJO, tuple as described by the subclasses of CompositeType) and produces a map of input data for the multi-encoder. Test the various types of user types, and test the value conversion logic (e.g. int -> double for ScalarEncoder).

It is possible for events to arrive out-of-order into the HTM operator. We should reorder the events using an internal queue, using the watermark to make progress.

See the flink-cep library for an example of this: CEPPatternOperator.

The Flink API supports stateful mapper functions and so should the HTM DSL.

An important scenario is to store predictions over time for comparison purposes with later events, to calculate an error rate (for example). See the HotGym example.

The workaround at this time is to use a trivial select function followed by mapWithState.

River View rivers come in two varieties - scalar and geospatial. It is unclear (to me) what extra work is needed to support geospatial rivers - maybe nothing. Please test the RiverSource against one (here's a possible list).

The RiverSource should support a graceful stop by implementing StoppableFunction. Streaming jobs are stoppable by the user as per FLINK-2111.

RiverSource[T] should provide a continuous stream of data, never terminating as it does now.

Ideally we would still provide a batch mode of operation too.

1. unbounded - the paging iterator should poll the stream on the interval specified in the meta.
2. bounded - the paging iterator should read until a given point in event time, then terminate. All streams should end at the same (event) time - "now" by default.

The learn function that flink-htm adds to DataStream is the primary API that inserts HTM into a dataflow graph. I consider the name of the function to be an open issue, since learn has another meaning in an HTM context.

The function name must be a verb, similar to existing functions like map, filter, and groupBy.

I suggest that the function name be grok.

Old:

val anomalyScores = env
      .addSource(nycTraffic)
      .keyBy("streamId")
      .learn(network)
      .select(...)

New:

val anomalyScores = env
      .addSource(nycTraffic)
      .keyBy("streamId")
      .grok(network)
      .select(...)

flink-htm provides a convenience DSL for encoders (see Input Data wiki page).

Many but not all encoders are supported. Simply create additional companion objects as needed in Encoders.scala.