Possible New Updates about amazon-kinesis-client

[knorwood]

1. This is an interesting idea, thanks for suggesting it. For merges, the benefit seems straightforward (apps may want to avoid repopulating a cache on another instance). In the case of a split, the split may have been due to capacity restrictions of the Worker instance, in which case, keeping the child shards on the same instance as their parent will perpetuate whatever capacity issue the split was intended to relieve. Do you have any suggestions or preferences for how the KCL would behave in this scenario?
2. This configuration option, cleanupLeasesUponShardCompletion makes the KCL cleanup leases for shards which no longer need processing because all their records have been processed. By default, it is set to true, so after a reshard event, the KCL worker apps should eventually converge to only processing (and only holding leases for) shards which are open.

[ajreid21]

Thanks for the response.

For 1, similar use cases for merge and split come up where the Worker instance is keeping state as it reads records and needs any future records with the same partition key to be processed by that same Worker. For the scenario where you need to shard for instance capacity restrictions, the KCL could make this behavior be a configurable flag (keep the current behavior as the default, but be able to configure it to force child shards to be processed by the parent).

For 2, I will test this again, but I believe last time that it shutdown the Worker threads for "closed" shards but still kept them as shard assignments (and therefore, the instance did not try and re-balance and grab other open shards from other instances).

[pfifer]

It looks like there is the same suggestion in #78 for this.

Please reopen if you're still seeing an issue with 2.