Comments (6)
Hi @wushujames,
Thanks for the suggestion. We are considering adding this feature. Would you be able to share how you are planning to use this metric in your application?
Btw, in case you weren't aware, KCL currently sends this metric to Cloudwatch (which may be useful for monitoring purposes).
Thanks,
-Adnan.
from amazon-kinesis-client.
I am also interested in this feature, particularly when exposed to the MultiLangDaemon
protocol. My use-case is easy enough to explain:
We batch thousands of smaller subrecords into each Kinesis record and we track latency of these subrecords individually, but to track the record itself we're presently using the ExplicitHashKey
to the same thing that HashKey
is normally set to, but this allows us to use HashKey
to store the time of record insertion and compute latency of Kinesis itself while not changing any Kinesis behavior.
This little bit of clever work-around gives us a per-record metadata field and the ability to easily use our own monitoring and alerting systems with extremely low latency. Cloudwatch is more difficult and less effective in this case due to reporting latency and having to query another system for simple data that is actually provided by the underlying API.
There's a way to address most of this in our record encoding format, to add this data in, but by adding it within the Kinesis record we would be required to do another layer of encoding and encapsulation for 1 simple field.
from amazon-kinesis-client.
I only just recently discovered that it is in CloudWatch. Thanks for that.
We have an application that needs to be mostly caught up to "now" before it is allowed to answer API requests. 1 minute back from "now" might be acceptable, 5 minutes might be acceptable, but hours back is unacceptable. This metric would let the application and operations decide when it is "mostly caught up" and when the application is allowed to start servicing requests.
Since this metric is per GetRecords() call, and the GetRecords() calls is per shard, we would have to have some heuristics that examines the MillisBehindLatest value for each shard, and only takes action when all shards have MillisBehindLatest lower than our desired threshold.
from amazon-kinesis-client.
I was looking to get MillisBehindLatest in IRecordProcessor as well, because we want to report it to CW.
The KCL already reports it to CW, but AFAICT it's only reported on the (combined) Operation+ShardId dimension, which means we don't have a single application-wide metric/dimension that we could use for both CW dashboards and auto scaling, e.g. to scale/up down based on MillisBehindLatest.
(Or if I want to put MillisBehindLatest in a CW dashboard, and I've got ~50 shards, I've got to select all 50 lines one-by-one. And then reselect 100-shards if I reshard from to 50 to 100.)
Is there an easy way for KCL to publish MillisBehindLatest both with and without shard id? Or am I overlooking something in CW?
from amazon-kinesis-client.
You have that nice ProcessRecordsInput now, it wouldn't even be a breaking change. :-)
from amazon-kinesis-client.
The KCL exposes MillisBehindLatest value to the record processor in 1.5.0 release. Thanks everyone for feedback.
from amazon-kinesis-client.
Related Issues (20)
- Clarification about DynamoDB initial configuration for KCL with consumer application HOT 3
- Guava dependencies with vulnerabilities in 1.x HOT 1
- Shutdown has been called on the cache, can't accept new requests.
- Feature Request: Stream name in aws kcl consumer when MultiStreamTracker is used HOT 4
- Problem in ts-tsbgservice-kinesis-incoming-emea-prod HOT 2
- Unexpected exception was thrown - IllegalStateException
- RetrievalConfig.initialPositionInStreamExtended deprecated HOT 1
- KCL continues to hold leases after unexpected shutdown due to Error
- Please update your schema-registry-serde library in order to solve CVE issue HOT 1
- Need more test cases, samples, documentation for StreamConfigs in case of Multi Streams with KCL2.x.
- Support for both polling and fanout retrieval mode for multi-stream consumer configuration HOT 1
- amazon-kinesis-client-pom using old awssdk.version HOT 2
- i.n.c.ChannelInitializer Failed to initialize a channel. Closing: [id: 0x9a4e56d6] java.lang.VerifyError: Bad return type HOT 2
- Lease table is not updated when new shards are added causing stale workers HOT 1
- Support artifacts with all third-party dependencies relocated HOT 2
- Consolidate metrics to a common name space, like /aws/kinesis-client HOT 2
- Uneven distribution of shards over the consumer application workers HOT 1
- Retrieving shard consumer's current lease's hash range key HOT 2
- KCL2 -Multi stream consumer - Configured streams can be in same account and cross account HOT 4
- graceful shutdown of MultiLangDaemon worker that is assigned for completed shards is always timeout HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from amazon-kinesis-client.