htm-community / flink-htm Goto Github PK
View Code? Open in Web Editor NEWDistributed, streaming anomaly detection and prediction with HTM in Apache Flink
License: GNU Affero General Public License v3.0
Distributed, streaming anomaly detection and prediction with HTM in Apache Flink
License: GNU Affero General Public License v3.0
Publish the first release of the library to Central.
All of life is out of balance and seems far away due to vision impairment occurring when viewing the authors profile! Please help...
Let's develop a connector to read Socrata streams.
Consider using the soda-java library, but be aware that its dependencies are fairly old (jackson 1, jersey 1). We would probably need to shade the dependencies and/or update them.
Alternatively, we could write something from scratch (consumer-side only, to begin with). The River source is written with Spray, which is easy to use. There are quite a few soda model classes (for geo data, timestamps, etc) and query classes that would need to be re-implemented.
It is possible to send a reset
to HTM to indicate that a temporal sequence has ended. Investigate how to facilitate a reset in the flink-htm API.
A simple option would be to expose the network object to direct manipulation in the select
function. But I would rather a more filter-like approach like HTMStream::resetOn(f: ResetFunction)
.
Is it better to reset before or after an event?
Update the library to use Flink 1.3 and likewise refresh the HTM dependency to 0.6.13.
Template:
DataStream<T> result = HTM.learn(stream, new Network())
.select(new InferenceSelectFunction<R, T>() {
@Override
public T select(Tuple2<R, NetworkInference> inference) throws Exception {
return inference.f1.getAnomalyScore();
}
});
The overrided select is never called. I verified that following the call hierarchy. This lead to no one output produced. Anyway, I verified the anomaly score, internally, is correctly computed.
Tried to resolve it, no great results reached. Please, fix it.
Flink's reliability is based on checkpointing of operator state. In flink-htm the critical state is that of the HTM network.
The operator code is well-prepared to support checkpointing. A state holder is allocated from Flink (see L63 of KeyedHTMInferenceOperator) into which the network instance is stored. Flink expects to use the serializer passed to it, and will automatically serialize the network at the appropriate time. The issue is that the network is not actually serializable (as of htm.java 0.6.5), and the job will crash if checkpointing is enabled.
Stretch goal: Flink supports both synchronous and asynchronous checkpointing. In the latter case, the operator is free to process elements while the state is being checkpointed. Depending on how the network state is ultimately checkpointed, consider it.
Flink provides a unit test framework for operators - or at least an integration test using an embedded Flink instance.
It is possible for events to arrive out-of-order into the HTM operator. We should reorder the events using an internal queue, using the watermark to make progress.
See the flink-cep library for an example of this: CEPPatternOperator.
The Flink API supports stateful mapper functions and so should the HTM DSL.
An important scenario is to store predictions over time for comparison purposes with later events, to calculate an error rate (for example). See the HotGym example.
The workaround at this time is to use a trivial select function followed by mapWithState
.
River View rivers come in two varieties - scalar and geospatial. It is unclear (to me) what extra work is needed to support geospatial rivers - maybe nothing. Please test the RiverSource against one (here's a possible list).
The RiverSource
should support a graceful stop by implementing StoppableFunction
. Streaming jobs are stoppable by the user as per FLINK-2111.
RiverSource[T] should provide a continuous stream of data, never terminating as it does now.
Ideally we would still provide a batch mode of operation too.
paging
iterator should poll the stream on the interval specified in the meta.paging
iterator should read until a given point in event time, then terminate. All streams should end at the same (event) time - "now" by default.The learn
function that flink-htm adds to DataStream
is the primary API that inserts HTM into a dataflow graph. I consider the name of the function to be an open issue, since learn
has another meaning in an HTM context.
The function name must be a verb, similar to existing functions like map
, filter
, and groupBy
.
I suggest that the function name be grok
.
Old:
val anomalyScores = env
.addSource(nycTraffic)
.keyBy("streamId")
.learn(network)
.select(...)
New:
val anomalyScores = env
.addSource(nycTraffic)
.keyBy("streamId")
.grok(network)
.select(...)
flink-htm provides a convenience DSL for encoders (see Input Data wiki page).
Many but not all encoders are supported. Simply create additional companion objects as needed in Encoders.scala.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.