awslabs / amazon-kinesis-data-generator

A UI that simplifies testing with Amazon Kinesis Streams and Firehose. Create and save record templates, and easily send data to Amazon Kinesis.

amazon-kinesis-data-generator

Easily send data to your Kinesis Stream or Kinesis Firehose

---

About

The Amazon Kinesis Data Generator is a UI that simplifies how you send test data to Amazon Kinesis Streams or Amazon Kinesis Firehose. Using the Amazon Kinesis Data Generator, you can create templates for your data, create random values to use for your data, and save the templates for future use.

Getting Started

The easiest and recommended approach is to use the Github-hosted version of this project. The second method is to clone/download this project and host it in your own S3 bucket.

Using the Github-hosted UI

Because the project is a collection of static HTML and JavaScript, you can use the application without deploying the code into your own AWS account. However, you will need to create an Amazon Cognito User Pool in your account, and create a user in that pool. A simple CloudFormation template and Lambda function is provided to simplify the creation of the Cognito User Pool and user.

To get started, go to the hosted Kinesis Data Generator. Instructions for creating the Cognito user in your account are located in the application's documentation.

amazon-kinesis-data-generator's Issues

Lambda does not support node.js 8 runtime

Updating to nodejs12.x manually fixes the problem

Stream/delivery stream drop-down only contain partial list

and does not allow typing in a stream name - making it impossible to select the stream I want.

Need to improve the generator generation speed

Can I change the configuration to make the speed of the generator reach 100k record/s or more? How to achieve it?

Issue when deploying cloudformation template - sha512 and bucket dependency

Hello,
I open this issue to track fixes I had to do for deploying cloudformation template.

I had 2 errors.

First one was:

name 'RunTimeError' is not defined

I fixed it changing the sha512 in the template from this:

bbe80f52ec7a246c065069f2bb0112a1a968472e6b48946ab88d73e5284787cd56acbd1d7adaa07a8120f3e1bd8d6644b96d18d7f0d7a2e60013d77b00d07eaa

to this:

779f78833de27c4523d27b0f792c7d3be7070fbe4bca76d480af2cb030049601e0081d44712c420e972c4bb9546c4167368671135ea0e62fe7d5d026eea584f6

Once fixed the first error, the second one appered: "cannot get bucket Arn"

I fixed this error by adding a DependsOn to StagingLambdaRole,
from this:

StagingLambdaRole:
    Type: "AWS::IAM::Role"
    Properties:

to this:

  StagingLambdaRole:
    Type: "AWS::IAM::Role"
    DependsOn: StagingS3Bucket
    Properties:

Issue when deploying cloudformation template

I'm trying to deploy the CloudFormation template in my AWS account and get the following error when proceeding after uploading the template:

The following resource types are not supported for resource import: Custom::BootstrapStagingLambdaFunc,Custom::DataGenCognitoSetupLambdaFunc

How do we monitor logs from KDG to Firehose?

I try the example KDG setup with my Firehose, but I don't see the content in S3, and not sure where I can get some logging/debugging information that I can look into KDG and how it is emitting data stream to my Firehose instance.

Node JS version needs update to version 18.

Have a configurable "Send total number of records"

It would be nice to have a configurable "Send total number of records" in addition to the throughput.

So, the application sends N records / second, with a total of T records and then stops.

I will take a look at the code, and if that's relatively easy to do, I shall submit a PR.

How do we send multiple records to simulate batch put?

Trying to setup a transformer to process batch records being sent in 15 minute intervals from kinesis.

The process works great utilizing the kinesis-data-generator for a single record but seems to fail when I try and format multiple records.

I've tried formatting the json in the following ways with no success:
[{jsonPayload},{jsonPayload}]

{jsonPayload}, {jsonPayload}

{jsonPayload}{jsonPayload}

Feeling a little lost if anyone knows what I'm doing wrong.

Something wrong with password RegEx

When using the hosted Kinesis Data Generator, no matter what password I use, I still get a "Template validation error: Parameter Password failed to satisfy constraint: must be at least 6 alpha-numeric characters, and contain at least one number" error.

Stuck at "Signing In"

If I try a user that doesn't exist or a wrong password I get an error so the authentication should work. It's just showing Signing In with the spinning animation and doesn't progress.

Unable to create stack on any other region other than us-west-2

I have been trying to generate the stack provided on the help page on other region other than us-west-2 and it keeps failing. When I generate stack on us-west-1 or us-east-1 I'd get this error. Also have posted the similar question in AWS dev forum and stackoverflow but no help so far.

lambda node4.3 runtime deprecated

i couldn't find the cloudformation template in the repo, but i copied it locally and updated the lambda to the latest node runtime and everything worked, so i think that's all that's needed.

Why does KDG shows sending data but kinesis stream does not receive it

I tried sending 1500 records/sec but after a while even though KDG shows data being sent, the stream and cloudwatch logs show no data whatsoever.
Is there a limit to the data being sent? In that case, why doesn't KDG stop automatically?

Using between, or adding a random number of X time to a date

I have multiple timestamps in the streamed record I'm trying to reproduce. Each one has a start and end date. I'm trying to randomise the end date by adding a random number of milliseconds to the start date.

I get the start date from {{date.utc}} and I can see there is a function for date.between() - however it's not clear how you would use this. I'm stuck on how to get the current date into the function:

{{date.between(<current_date>,<num_of_milliseconds>)}} doesn't work and neither have any of my other guesses. Any help on how to format this would be appreciated.

eneccccrkncgguevgbclrechldljkvhbjgbfvdihftrk

JSON get's converted to Base64

When sending the record template

{
	"sensorId": 40,
	"currentTemperature": 76,
	"status": "OK"
}

the payload delivered to Kinesis is Base64 encoded.

{
    "kinesis": {
        "kinesisSchemaVersion": "1.0",
        "partitionKey": "2264290559",
        "sequenceNumber": "49583841783469136626947675794180856103183100390328500226",
        "data": "ewkic2Vuc29ySWQiOiA0MCwJImN1cnJlbnRUZW1wZXJhdHVyZSI6IDc2LAkic3RhdHVzIjogIk9LIn0K",
        "approximateArrivalTimestamp": 1524572040.471
    },
    "eventSource": "aws:kinesis",
    "eventVersion": "1.0",
    "eventID": "shardId-000000000000:49583841783469136626947675794180856103183100390328500226",
    "eventName": "aws:kinesis:record",
    "invokeIdentityArn": "arn:aws:iam::294557341534:role/lambda_firehoseTranformer",
    "awsRegion": "eu-central-1",
    "eventSourceARN": "arn:aws:kinesis:eu-central-1:294557341534:stream/test"
}

Is this expected behavior? If yes, please update the documentation. Using the sample test events and documentation for AWS Kinesis/Firehose and the AWS Kinesis-Lambda-Templates this does not work without a conversion back to JSON.

random.float does not work

random.float function does not work for me.

CloudFormation template Failure

I am trying to deploy the KdG via cloudformation but stack fails. The arn boundary parameter says optional but stack fails with " handler return error, permission block must be set for cognito provider " have been on this for sometime now, i created a condition block policy snd passed as parameter for the policy boundery but same error.