Comments (4)
This is an interesting pattern. This issue has come up on github before but I wasn't able to come up with a solution. I think I'm going to put this in some form into the wiki.
Thanks for your writeup.
from flink-spector.
This was not an issue with the framework, but rather my understanding (or lack thereof) of the timing of events. I'll explain here with a contrived example so others don't make the same mistake I did.
Basically, lets say I have two streams - a "Number" stream, and a "Multiplier" stream. My CoProcessFunction accepts both these streams as input. It keeps track (via ValueState) both the last Number it has seen as well as the last Multiplier it has seen. Upon receiving a Number N, if it has already seen a multiplier LM (last multiplier), then it produces a new Number with value N * LM. Similarly, upon seeing a multiplier M, if it has already seen a number LN (last number), then it produces a new Number with value LN * M.
For example, given a Number Stream [1,] and a Multiplier Stream [10,20], the output stream should be Numbers [10, 20]....or so I thought!
But what actually happens is that for the Test that "passed", the sequence of events was -
- Saw Number 1, stored it in LastNumberSeen. LastMultiplierSeen is NULL, so no output.
- Saw Multiplier 10, stored in LastMultiplierSeen. LastNumberSeen is 1 - so output 1*10 = 10
- Saw Multiplier 20, stored in LastMultiplierSeen. LastNumberSeen is 1 - so output 1*20 = 20
and the output was [10, 20]
However, the test that failed, the sequence of events was
- Saw Multiplier 10, stored in LastMultiplierSeen. LastNumberSeen is NULL - so no output
- Saw Multiplier 20, stored in LastMultiplierSeen. LastNumberSeen is NULL - so no output
- Saw Number 1, stored it in LastNumberSeen. LastMultiplierSeen is 20, so output 1*20 = 20
and the output was [20]
My mistake was that I had an invalid assumption on the ordering of events between the 2 streams.
We can control the ordering on a single stream (by using EventTimeInputBuilder). But not sure how we can control the ordering on 2 separate streams. In reality we cannot, but for testing it would be good to have this determinism.
If you have any thoughts on that please share.
Thanks, and sorry for the false alarm.
from flink-spector.
"But not sure how we can control the ordering on 2 separate streams. "
I was able to guarantee ordering by creating one ordered stream that contains both Number and Multiplier types (using a stream of type DataStream) - using the EventTimeBuilder. I then filtered this stream twice - to get 2 streams one containing only Number and the other only Multiplier. Finally, I map both those streams to produce a DataStream and DataStream.
Here's a code snippet for illustration.
EventTimeInputBuilder<Object> myCombinedBuilder = EventTimeInputBuilder.startWith(new Object())
.emit(new Number(1), after(1, TimeUnit.SECONDS))
.emit(new Multiplier(10), after(1, TimeUnit.SECONDS))
.emit(new Multiplier(20), after(1, TimeUnit.SECONDS));
DataStream<Object> combinedStream = this.createTestStream(myCombinedBuilder);
DataStream<Number> numberOnlyStream = combinedStream.filter(new FilterFunction<Object>() {
@Override
public boolean filter(Object value) throws Exception {
return value instanceof Number;
}
}).map(new MapFunction<Object, Number>() {
@Override
public Number map(Object value) throws Exception {
return (Number) value;
}
});
DataStream<Multiplier> multiplierOnlyStream = combinedStream.filter(new FilterFunction<Object>() {
@Override
public boolean filter(Object value) throws Exception {
return value instanceof Multiplier;
}
}).map(new MapFunction<Object, Multiplier>() {
@Override
public Multiplier map(Object value) throws Exception {
return (Multiplier)value;
}
});
from flink-spector.
One oddity you may notice is that I start the EventTimeInputBuilder with a placeholder object that is a super class of both Number and Multiplier types -- in my case I just used Object(). This is to get the builder to produce a stream which can accept both types. It gets filtered out, so a bit wasteful.
EventTimeInputBuilder.startWith(new Object())
from flink-spector.
Related Issues (20)
- Low performance when using EventTimeInputBuilder HOT 1
- Tests don't work while making mvn test on CircleCI HOT 2
- Convert Message Headers to Long
- Global Windows don't emit data on time HOT 2
- Publish versions to OSS Repository Hosting HOT 2
- "Task was cancelled" exception when testing CoProcessFunction HOT 6
- Publish 0.8.2 to Maven Central HOT 5
- NPE in test HOT 6
- Consider splitting out JUnit 4 functionality into separate module and documenting HOT 2
- assertStream always passes HOT 6
- There is no DataStreamTestBase.emit variant that accepts an arbitrary timestamp HOT 1
- Run pipeline without flushing all the windows HOT 2
- TestBaseUtils.startCluster is deprecated as of Flink 1.7.0 HOT 5
- Release 0.8.5? HOT 3
- Flinkspector 0.9.1 references package/classes not published by Flink HOT 4
- Is there any plan to support flink 1.8? HOT 6
- setParallelism changes test outcome HOT 6
- unresolved dependencies scala 2.12
- unmaintained? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from flink-spector.