sendgridlabs / go-kinesis

GO library for AWS Kinesis

go-kinesis

GO-lang library for AWS Kinesis API.

Documentation

• Core API
• Batch Producer API

Example

Example you can find in folder examples.

Command line interface

You can find a tool for interacting with kinesis from the command line in folder kinesis-cli.

Testing

Local Kinesis Server

The tests require a local Kinesis server such as Kinesalite to be running and reachable at http://127.0.0.1:4567.

To make the tests complete faster, you might want to have Kinesalite perform stream creation and deletion faster than the default of 500ms, like so:

kinesalite --createStreamMs 5 --deleteStreamMs 5 &

The & runs Kinesalite in the background, which is probably what you want.

go test

Some of the tests are marked as safe to be run in parallel, so to speed up test execution you might want to run go test with the -parallel n flag.

go-kinesis's Issues

Example rate-limit issues

Kinesis rate limits GetRecords requests to 5 per second per shard. The example's "getRecords" method will quickly hit that, and by not handling the rate limit error will fail as you proceed to attempt iterating over the records. It's recommended to sleep after every request for 500-1000ms, and to check for errors after GetRecords

http://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetRecords.html
If a call returns 10 MB of data, subsequent calls made within the next 5 seconds throw ProvisionedThroughputExceededException. If there is insufficient provisioned throughput on the shard, subsequent calls made within the next 1 second throw ProvisionedThroughputExceededException. Note that GetRecords won't return any data when it throws an exception. For this reason, we recommend that you wait one second between calls to GetRecords; however, it's possible that the application will get exceptions for longer than 1 second

Python library calling sleep:
https://github.com/awslabs/kinesis-poster-worker/blob/master/worker.py

Code should be run through go fmt.

I was preparing a patch to fix a bug when encoding records for kinesis, and noticed that you're using spaces for indentation instead of tabs, which indicates that Go FMT isn't being used on the code. This complicates patch writing when many editors automatically go-fmt code on saving, causing substantial diffs as most lines are re-written to use tabs, or are otherwise restructured to follow go-fmt styles.

See http://blog.golang.org/go-fmt-your-code

BatchProducer doesn't check size of payload

Kinesis rejects PutRecords requests over 5 megabytes, but the batch producer doesn't do anything to make sure it doesn't exceed 5 megabytes in its requests.

When this failure is hit, we've seen our batch producer fail entirely as AWS simply kills the HTTP connection. None of the records get through, and every subsequent PutRecords request from the batch producer fails because the batches are too large, and it can't come back by sending smaller payloads.

Wrong repo

Sorry, wrong repo.

Sample code keeps asking for AWS_SECURITY_TOKEN

I'm trying to run the sample code provided in the examples folder but it just keeps asking for the AWS_SECURITY_TOKEN env variable.

According to the Shared config file section in the AWS Go SDK "Credentials are the values the SDK should use for authenticating requests with AWS Services. They arfrom a configuration file will need to include both aws_access_key_id and aws_secret_access_key must be provided together in the same file to be considered valid. The values will be ignored if not a complete group. aws_session_token is an optional field that can be provided if both of the other two fields are also provided."
As you can see, the token is supposed to be optional if you provide both the secret and access keys but this code https://github.com/sendgridlabs/go-kinesis/blob/master/auth.go#L78
tells me that, in go-kinesis, it's mandatory.

Why isn't AWS_SECURITY_TOKEN optional?
Am I missing something here?

Thanks in advance for any help

Provide automatic fallbacks when creating auth object for AWS

Looking at auth.go, the three methods of retrieving the access keys to make API calls to AWS are available:

• the consumer provides the access keys
• get the access keys from environment variables
• get the access keys from the local metadata server

The problem with this is that, as a consumer, I need to add logic to handle falling back to one of the above methods if a previous one fails (provided by consumer -> checked env -> check metadata server). Comparing this code to goamz's GetAuth method, goamz provides a nice wrapper method to encapsulate this fallback logic so that the code will seamlessly work in all three scenarios.

It would be nice to have a method like this for go-kinesis.

goamz

Any plans on eventually moving it to goamz? Using both goamz and go-kinesis in one project is a bit confusing at the moment. 🚶

No error when call PutRecords, but no data when get records from kinesis

Hi, Experter

I encounter an issue that records lost when called PutRecords api to send data to kinesis.
It returns no error when call PutRecords in producer side.
But no records get in customer side.
It happens in low probability, and hard to reproduce.
Did you encounter this, or it is fixed in latest version?
tag:0.3.0
Thank you in advance!

Other question is:
Where can I find AWS_SECURITY_TOKEN.
There is no such a key in tag:0.3.0. And I can't find the key on aws console.

Expand CLI to add a PUT command

It would be cool if you there was a CLI for doing the following:

cat something.txt | kinesis-cli <stream-name>

something along those lines. I could imagine creating very simple producers this way. Thoughts?

Updated batchproducer implementation

We wanted to use the batchproducer logic but had issues with the customized kinesis client here and wanted to use the standard AWS SDK for go which has since been released.

We updated the batchproducer to use the standard SDK and released it here (retaining original license) in case anyone else runs into this; also made a few other small improvements:
https://github.com/qntfy/frinesis

Just sharing for awareness. Thanks!

Host check in Sign method doesn't work for local Kinesis

This host check in the Sign method doesn't work when using a local Kinesis server such as Kinesalite.

Unexpected EOF when using PutRecords

Post https://kinesis.ap-southeast-1.amazonaws.com: EOF

Why?

Mock request to kinesis

Any hint on how to mock requests to kinesis in tests?

Currently my test freezes for 30 seconds and then fails. :(

Question from Example App

Really enjoying the library thus far. Great stuff 👍

Was curious about the usage of timeout here?
https://github.com/sendgridlabs/go-kinesis/blob/master/examples/example.go#L58

When I adjust the number from 30 -> 1 to try to emulate a timeout the execution just pauses. Is that the anticipated behavior? Any insight would be much appreciated.

not enough arguments in call to kinesis.NewAuth

1. and 4. line of code is what I had before and it worked. Now I wanted to implement Godep but it looks like that I am a little bit too late.

First four lines below are what I had before and it worked. Now I am trying to implement it like you implemented it in example.go ... but keep getting errors...

• not enough arguments in call to kinesis.NewAuth
• undefined: kinesis.Region (even if i pass kinesis.Region{} as second parameter to kinesis.New function
• cannot use &auth (type *_kinesis.AuthCredentials) as type kinesis.Auth in argument to kinesis.New:
  *_kinesis.AuthCredentials does not implement kinesis.Auth (missing GetAccessKey method)

I would really appreciate some help here... since code don't work for me the way it is in example.go.

Best,
Tomaz

// KINESIS AUTHENTIACTION
// set env variables AWS_ACCESS_KEY and AWS_SECRET_KEY AWS_REGION_NAME
//auth := kinesis.NewAuth()
//ksis := kinesis.New(&auth, kinesis.Region{})

var (
    err  error
    auth kinesis.Auth
)

auth, err = kinesis.NewAuthFromEnv()
if err != nil {
    fmt.Printf("unable to retrieve authentication credentials from the environment: %v", err)
    os.Exit(1)
}

ksis := kinesis.New(auth, "")

Would a shard utility cli be a good example to add?

I find myself wanting a simple command line interface to manage my kinesis streams.

I imagine it would look something like:

$ ./shardutil --help
  Usage: ./shardutil [command] [args]  
  Commands:
    ls <stream name>
      Describes the stream

    split <stream name> <shard id> [<starting hash key>]
      Splits the stream's shard at the hash key (or half if not given). Prompts for confirmation.

    merge <stream name> <shard id> <adjacent shard id>
      Merges the stream's shards. Prompts for confirmation.

I'd be happy to write this if it would be useful to include as an example (or just as a useful tool). But I'm happy to just keep it for myself if you'd rather not have something like that as part of the repo, too. Let me know!

Support for Batch send (PutRecords)

Kinesis now supports batch send of messages (up to 500). It would be nice to have that supported too. It doesn't seem to be very difficult, just a different structure of sending messages.

Is there any examples for the batchProducer

A example would make it easier to understand how it should be used.

Any help would be gratefully received.

Thanks.

batchproducer tries to import "github.com/timehop/go-kinesis"

When using batchproducer, I got the following compilation error:

batchproducer.go:10:2: cannot find package "github.com/timehop/go-kinesis" in any of:
    /usr/local/Cellar/go/1.4.2/libexec/src/github.com/timehop/go-kinesis (from $GOROOT)
    Development/go/src/github.com/timehop/go-kinesis (from $GOPATH)

Looks like there is a bad import in batchproducer.go

NewAuth usage

I can't figure out how to new NewAuth() function together with NewClient(). Could you provide an example please? This, of course, doesn't work"

kinesis_client = kinesis.New(kinesis.NewAuth(), kinesis.Region{os.Getenv("KINESIS_REGION")})

It gives me panic: runtime error: invalid memory address or nil pointer dereference, though compiles okay

Add sendgrid kinesis to AWS developer tools

This package should be listed here.

http://aws.amazon.com/kinesis/streams/developer-resources/

which can be done by posting here:
https://forums.aws.amazon.com/forum.jspa?forumID=169&start=0

NewAuthFromEnv() does not support AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

I found that NewAuthFromEnv does not support looking for the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY for API key values. This will cause problems for some users as the AWS cli docs does recommend setting those environment variables.

Firehose integration

I see @fjordan may have a working solution:
rewardStyle@6c4ffb2

Is this something that can be merged in?

IAM Role Support

Hello,

what do you think about IAM Role Support? I don't think it would be hard thing to do: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html. For us it would be extremely usefull since we do everything with IAM Roles instead of shipping credentials.