Sessionization Example never times out about spark-the-definitive-guide

[aremash]

I also tried changing the GroupStateTimeout from EventTimeTimeout to ProcessingTimeTimeout on line 315 plus oldState.setTimeoutTimestamp(newState.timestamp.getTime(), "5 seconds") to oldState.setTimeoutDuration("5 seconds") on line 295, but still don't observe any time-outs.

[bllchmbrs]

There might be a couple of issues here.

1. That's because the file sink doesn't support complete mode. In append mode, it's going to have to wait for the entire watermark to run out, which can take some time. More info here. You'd start seeing results after 10+ minutes if you're doing append because it has to wait for the watermark to complete.

2. Also if there's no data left, or it gets processed too quickly, Spark won't "age out" old state if it processes an empty batch. This is also something that's slated for future changes.