smarty-archives / go-aws-auth

[DEPRECATED] Signs requests to Amazon Web Services (AWS) using IAM roles or signed signature versions 2, 3, and 4. Supports S3 and STS.

go-aws-auth

Go-AWS-Auth is a comprehensive, lightweight library for signing requests to Amazon Web Services.

It's easy to use: simply build your HTTP request and call awsauth.Sign(req) before sending your request over the wire.

Supported signing mechanisms

• Signed Signature Version 2
• Signed Signature Version 3
• Signed Signature Version 4
• Custom S3 Authentication Scheme
• Security Token Service
• S3 Query String Authentication
• IAM Role

For more info about AWS authentication, see the comprehensive docs at AWS.

Install

Go get it:

$ go get github.com/smartystreets/go-aws-auth

Then import it:

import "github.com/smartystreets/go-aws-auth"

Using your AWS Credentials

The library looks for credentials in this order:

1. Hard-code: You can manually pass in an instance of awsauth.Credentials to any call to a signing function as a second argument:

   awsauth.Sign(req, awsauth.Credentials{
   	AccessKeyID: "Access Key ID", 
   	SecretAccessKey: "Secret Access Key",
   	SecurityToken: "Security Token",	// STS (optional)
   })

2. Environment variables: Set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables with your credentials. The library will automatically detect and use them. Optionally, you may also set the AWS_SECURITY_TOKEN environment variable if you are using temporary credentials from STS.

3. IAM Role: If running on EC2 and the credentials are neither hard-coded nor in the environment, go-aws-auth will detect the first IAM role assigned to the current EC2 instance and use those credentials.

(Be especially careful hard-coding credentials into your application if the code is committed to source control.)

Signing requests

Just make the request, have it signed, and perform the request as you normally would.

url := "https://iam.amazonaws.com/?Action=ListRoles&Version=2010-05-08"
client := new(http.Client)

req, err := http.NewRequest("GET", url, nil)

awsauth.Sign(req)  // Automatically chooses the best signing mechanism for the service

resp, err := client.Do(req)

You can use Sign to have the library choose the best signing algorithm depending on the service, or you can specify it manually if you know what you need:

• Sign2
• Sign3
• Sign4
• SignS3 (deprecated for Sign4)
• SignS3Url (for pre-signed S3 URLs; GETs only)

Contributing

Please feel free to contribute! Bug fixes are more than welcome any time, as long as tests assert correct behavior. If you'd like to change an existing implementation or see a new feature, open an issue first so we can discuss it. Thanks to all contributors!

go-aws-auth's Issues

URL needs double-encoding for non-S3 endpoints

Hey,

I'm using this library to sign requests to an AWS ES Service endpoint. Basic requests work fine, but those with a * have a signing error:

The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

The Canonical String for this request should have been
'GET
/.kibana-4/_mapping/%252A/field/_source
_=1447104608354
...

The canonical request generated by go-aws-auth only single-encoded the symbol:
/.kibana-4/_mapping/%2A/field/_source

Adding another encodePathFrag() call around the existing one in normuri() fixed the request.

Docs don't talk much about this but a spec on node's aws-sdk checks that URIs are double-encoded for non-S3 endpoints only. https://github.com/aws/aws-sdk-js/blob/333b9d1507a70c8e5b6459a1b513f9fcddd16457/test/signers/v4.spec.coffee#L124

unable to use EC2 service

I'm using the lib for ec2 service.
For url :
https://ec2.eu-central-1.amazonaws.com/?ImageId=ami-9bf712f4&MaxCount=1&MinCount=0&Action=RunInstances

when I use the Sign() func, I get response 401 :

<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>AuthFailure</Code><Message>AWS was not able to validate the provided access credentials</Message></Error></Errors><RequestID>4bfd2d54-242d-4d4b-bfe5-768339e1559a</RequestID></Response>.

when I use the Sign4() func I get http response 500:

<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InternalError</Code><Message>An internal error has occurred</Message></Error></Errors><RequestID>823256fe-0681-4578-862e-fa3458b84686</RequestID></Response>

Any idea?

Broken S3 sign

Just trying out signing a request for S3 but I get

The request signature we calculated does not match the signature you provided. back from amazon.

I like the light approach of only signing http.Request if it would work.

Credentials need to be thread-safe

Issue #13 addressed the thread safety of signing requests, but highlighted that now the credentials are potential hazards for race conditions.

Unnecessary mutex in awsauth.go?

I am not sure why the "signMutex" is necessary for concurrent access to the signing version map. In go maps are safe for concurrent access for any number of readers, it's only when something else may be writing to the map that you would need to protect it:

https://groups.google.com/forum/#!msg/golang-nuts/HpLWnGTp-n8/hyUYmnWJqiQJ

Unless I am missing something, the global map is never written to.

MissingAuthenticationTokenException

When I print the response, I get a 403 Forbidden with the following error MissingAuthenticationTokenException. I have nevertheless added the SecurityToken generated with

$ aws sts get_session-token

What am I doing wrong ? The error should be at least : WrongToken

Thank you

Edit

After calling :

awsauth.Sign(req)

req.Header is an empty map. Shouldn't it be populated with signed attributes ?

Edit 2

You may have inverted indexes, at least for my issue where host = "https://xxxxxxxxxx.execute-api.us-east-1.amazonaws.com"

Should be :

• service at index 1
• region at index 2

https://github.com/smartystreets/go-aws-auth/blob/0c1422d1fdb9fef5a1ad5b2e13ac663467eb767e/common.go#L49-L50

master (2043e6d0bb7e4c18464a7bba562acbe482e3cabd) does not pass tests!

go version go1.8 darwin/amd64
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/tmornini/go"
GORACE=""
GOROOT="/usr/local/Cellar/go/1.8/libexec"
GOTOOLDIR="/usr/local/Cellar/go/1.8/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/z7/rq6vhb0s7299yt54gljf0f600000gn/T/go-build673217083=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
GOROOT/bin/go version: go version go1.8 darwin/amd64
GOROOT/bin/go tool compile -V: compile version go1.8 X:framepointer
uname -v: Darwin Kernel Version 15.6.0: Mon Jan  9 23:07:29 PST 2017; root:xnu-3248.60.11.2.1~1/RELEASE_X86_64
ProductName:	Mac OS X
ProductVersion:	10.11.6
BuildVersion:	15G1217
lldb --version: lldb-360.1.70

$ go test
........
8 total assertions

..
10 total assertions

..
12 total assertions

...
15 total assertions

..
17 total assertions

..
19 total assertions

...
22 total assertions

.
23 total assertions

.
24 total assertions

.
25 total assertions

.
26 total assertions

..............
40 total assertions

.
41 total assertions

.
42 total assertions

..
44 total assertions

...
47 total assertions

...
50 total assertions

.
51 total assertions

.....
56 total assertions

...
59 total assertions

..
61 total assertions

......
67 total assertions

..
69 total assertions

.....
74 total assertions

..
76 total assertions

x......
Failures:

  * /Users/tmornini/go/src/github.com/affinion-group/go-aws-auth/sign4_test.go 
  Line 21:
  Expected: '(*http.Request){Method:"POST", URL:(*url.URL){Scheme:"http", Opaque:"", User:(*url.Userinfo)(nil), Host:"iam.amazonaws.com", Path:"/", RawPath:"", ForceQuery:false, RawQuery:"", Fragment:""}, Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{"Content-Type":[]string{"application/x-www-form-urlencoded; charset=utf-8"}, "X-Amz-Date":[]string{"20080814T170848Z"}}, Body:ioutil.nopCloser{Reader:(*strings.Reader){s:"Action=ListUsers&Version=2010-05-08", i:0, prevRune:-1}}, GetBody:(func() (io.ReadCloser, error))(0x0000000001234750), ContentLength:35, TransferEncoding:[]string(nil), Close:false, Host:"iam.amazonaws.com", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:"", RequestURI:"", TLS:(*tls.ConnectionState)(nil), Cancel:(<-chan struct {})(0x0000000000000000), Response:(*http.Response)(nil), ctx:context.Context(nil)}'
  Actual:   '(*http.Request){Method:"POST", URL:(*url.URL){Scheme:"http", Opaque:"", User:(*url.Userinfo)(nil), Host:"iam.amazonaws.com", Path:"/", RawPath:"", ForceQuery:false, RawQuery:"", Fragment:""}, Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{"Content-Type":[]string{"application/x-www-form-urlencoded; charset=utf-8"}, "X-Amz-Date":[]string{"20080814T170848Z"}}, Body:ioutil.nopCloser{Reader:(*strings.Reader){s:"Action=ListUsers&Version=2010-05-08", i:0, prevRune:-1}}, GetBody:(func() (io.ReadCloser, error))(0x0000000001234750), ContentLength:35, TransferEncoding:[]string(nil), Close:false, Host:"iam.amazonaws.com", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:"", RequestURI:"", TLS:(*tls.ConnectionState)(nil), Cancel:(<-chan struct {})(0x0000000000000000), Response:(*http.Response)(nil), ctx:context.Context(nil)}'
  (Should resemble)!


83 total assertions

x
Failures:

  * /Users/tmornini/go/src/github.com/affinion-group/go-aws-auth/sign4_test.go 
  Line 50:
  Expected: '(*http.Request){Method:"POST", URL:(*url.URL){Scheme:"http", Opaque:"", User:(*url.Userinfo)(nil), Host:"iam.amazonaws.com", Path:"/", RawPath:"", ForceQuery:false, RawQuery:"", Fragment:""}, Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{"Content-Type":[]string{"application/x-www-form-urlencoded; charset=utf-8"}, "X-Amz-Date":[]string{"20110909T233600Z"}}, Body:ioutil.nopCloser{Reader:(*strings.Reader){s:"Action=ListUsers&Version=2010-05-08", i:0, prevRune:-1}}, GetBody:(func() (io.ReadCloser, error))(0x0000000001234750), ContentLength:35, TransferEncoding:[]string(nil), Close:false, Host:"iam.amazonaws.com", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:"", RequestURI:"", TLS:(*tls.ConnectionState)(nil), Cancel:(<-chan struct {})(0x0000000000000000), Response:(*http.Response)(nil), ctx:context.Context(nil)}'
  Actual:   '(*http.Request){Method:"POST", URL:(*url.URL){Scheme:"http", Opaque:"", User:(*url.Userinfo)(nil), Host:"iam.amazonaws.com", Path:"/", RawPath:"", ForceQuery:false, RawQuery:"", Fragment:""}, Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{"Content-Type":[]string{"application/x-www-form-urlencoded; charset=utf-8"}, "X-Amz-Date":[]string{"20110909T233600Z"}}, Body:ioutil.nopCloser{Reader:(*strings.Reader){s:"Action=ListUsers&Version=2010-05-08", i:0, prevRune:-1}}, GetBody:(func() (io.ReadCloser, error))(0x0000000001234750), ContentLength:35, TransferEncoding:[]string(nil), Close:false, Host:"iam.amazonaws.com", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:"", RequestURI:"", TLS:(*tls.ConnectionState)(nil), Cancel:(<-chan struct {})(0x0000000000000000), Response:(*http.Response)(nil), ctx:context.Context(nil)}'
  (Should resemble)!


84 total assertions

--- FAIL: TestVersion4RequestPreparer (0.00s)
..
86 total assertions

...
89 total assertions

.
90 total assertions

.
91 total assertions

......
97 total assertions

.
98 total assertions

..
100 total assertions

FAIL
exit status 1
FAIL	github.com/affinion-group/go-aws-auth	4.417s

Does not support API Gateway with custom host names

With API Gateway, you need to sign your URL if you have AWS_IAM authentication enabled and this would be a great service to use if it supported it. Unfortunately, if you are using custom domain with API Gateway, you can't derive the service and region from the URL like you can with other API requests. There should be another method similar to Sign4 that allows the user to manually provide the region and service (execute-api for API Gateway).

Implement Route 53 authentication

Route 53 is not like the others; it apparently uses a custom authentication scheme (Signed Signature Version 3?).

See comments at the end of issue #2 for more info. Currently looking into this.

serviceAndRegion logic is incorrect

Right now I am trying to sign requests to AWS Elasticsearch service, and those domains are in the form:

https://search-[domain]-[id].[region].es.amazonaws.com

My request are failing because they're defaulting to us-east-1 and service is also incorrect.

Looks like this is fixed by #27

Rename the `awsauth.Keys` variable?

Is Keys really a good name for it? It stores the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, but only ever 1 pair at a time. It's just a type named Credentials, a struct with those two values. What would be a better name? Credential?

Sign2 succeeds, but Sign4 403 forbidden

The following test for an SQS resource, succeeds with Sign2, but fails when Sign4 is substituted, otherwise code is identical. Error code returned is 403 forbidden.

package main

import (
    "fmt"
    "awsauth"
    "log"
    "net/http"
    "net/http/httputil"
)


const kAccessKeyID = "ACCESSKEY"
const kSecretAccessKey = "SECRETKEY"
const kAWSAccountID = "###########"
const kQueueName = "testqueue"

func main() {

    awsauth.Keys = &awsauth.Credentials{ kAccessKeyID, kSecretAccessKey }

    url := "https://sqs.us-east-1.amazonaws.com/" + kAWSAccountID +
        "/" + kQueueName +
        "?Action=SendMessage&Version=2012-11-05&MessageBody=YesHelloWorldmessage"

    client := &http.Client{}

    req, err := http.NewRequest("GET", url, nil)
    out, _ := httputil.DumpRequestOut((*http.Request)(req), true)
    fmt.Printf("%s\n", string(out))

    //awsauth.Sign4(req)   // This fails
    awsauth.Sign2(req)   // This succeeds
    out2, _ := httputil.DumpRequestOut((*http.Request)(req), true)
    fmt.Printf("%s\n", string(out2))

    resp, err := client.Do(req)
    if err != nil {
        log.Fatal(err)
    }

    fmt.Println(resp.StatusCode)
    fmt.Println(resp.Status)
}

Support temporary credentials from EC2 metadata

This way IAM roles can be utilized within EC2 instances.

I may be wrong (let me know), but I think the process is simple:

GET http://169.254.169.254/latest/meta-data/iam/security-credentials/<role-name>

where <role-name> is the name of an IAM role (I presume it must be assigned to that instance?).

Sample output:

{
  "Code" : "Success",
  "LastUpdated" : "2012-04-26T16:39:16Z",
  "Type" : "AWS-HMAC",
  "AccessKeyId" : "AKIAIOSFODNN7EXAMPLE",
  "SecretAccessKey" : "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
  "Token" : "token",
  "Expiration" : "2012-04-27T22:39:16Z"
}

There's also this API call using Security Token Service... I've actually not used either of these features before, though... someone is welcome to dive in and add this functionality if they desire it sooner than later!

If this is as simple as providing an IAM role name (or even simpler), then perhaps the library could do this implicitly: if awsauth.Keys has not been set, and it can't find environment variables with the credentials, then it could automatically suppose it's on an EC2 instance and get security credentials. It would just have to keep track of the expiration date...

Signature invalid when using for "Product Advertising API"

My code is the following:

func (o *AWSHandler) InteractWithMethod(method, u string, values url.Values, header http.Header) (*http.Response, error) {
    var req *http.Request
    var err error

    values.Add("AWSAccessKeyId", o.config["access_key"])
    values.Add("Timestamp", time.Now().Format("2006-01-02T15:04:05Z"))
    values.Add("SignatureVersion", "3")
    if method == "GET" {
        req, err = http.NewRequest(method, u+"?"+values.Encode(), nil)
    } else {
        req, err = http.NewRequest(method, u, bytes.NewBufferString(values.Encode()))
    }
    req.Header = header
    if err != nil {
        return nil, err
    }
    awsauth.SignS3Url(req, time.Now().Add(time.Duration(10)*time.Second), awsauth.Credentials{
        AccessKeyID:     o.config["access_key"],
        SecretAccessKey: o.config["access_secret"],
    })
    resp, err := o.client.Do(req)
}

And here is the response I get:

<?xml version="1.0"?>
<ItemSearchErrorResponse xmlns="http://ecs.amazonaws.com/doc/2005-10-05/"><Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.</Message></Error><RequestId>227b35f0-bd51-4fe0-9896-aa54c25f5c68</RequestId></ItemSearchErrorResponse>

I'm pretty sure my keys are correct...

Cannot find package?

I've followed the instructions:

go get github.com/smartystreets/go-aws-auth
import "github.com/smartystreets/awsauth" into main.go

When I compile:

cannot find package "github.com/smartystreets/awsauth" in any of:
/usr/local/go/src/pkg/github.com/smartystreets/awsauth (from $GOROOT)
/Users/sc28/projects/src/github.com/smartystreets/awsauth (from $GOPATH)

I changed to the pkg directory and deleted the .a file, went to the source directory and rebuilt/installed - running go test worked fine and the .a is refreshed in the pkg directory - same error.

Not sure why path isn't found - the package .a file is found in the $GOPATH/pkg/darwin_and64/github.com/smartystreets/ folder

The GOPATH is looking for awsauth in the smartystreets folder, where as the go get command seems to install it to the go-aws-auth folder in smartystreets

Support S3 query string URLs

There's an alternative form of S3 authentication which makes it easy to serve files (by a certain expiration date) with only a URL, no fancy headers required.

• Introduction
• Implementation details

Warning: go-aws-auth is no longer maintained and will soon be removed!

Announcement:

This repository (go-aws-auth) will soon be archived and will subsequently be removed on September 1, 2018. Between now and then, any who wish may fork the repo and continue development on said forks.

Background

We have come to the realization that our usage of this package is limited to only 2 aws operations:

• S3: GetObject
• S3: PutObject

So, we have opted to create a separate, single-purpose library to accomplish this specific purpose and replace go-aws-auth. We realize that there are those who have much grander ambitions for this library and we're sorry we haven't engaged with your issues and pull requests--we just no longer have a need for the kind of general-purpose AWS signing library that this package originally set out to be. Also, we created this repo when there were no official AWS SDKs for go (v1 and v2). There's also the recently released go-cloud SDK to think about now, too.

That's all for now. Thanks for reading!

Suggestion: Make http request signing safe from multiple goroutines

Since working with the http.Client and creating new requests is usually safe from multiple goroutines, I would like to suggest either guarding the things in awsauth that manipulates http.Requests with a mutex or getting rid of the shared states as well.

I'm not entirely sure which parts of it might need changes but the race detector complained on using Sign4 for my requests on multiple routines.

SigV4 query-string URI-signing for AWS IoT Data service

I had need for a different query-string-based SigV4 signing for AWS IoTData service. The mechanism is quite different from the S3 or ES signing mechanism (already implemented in this library), so I wrote a new function to provide a SigV4 query string signing mechanism intended rather narrowly for the IoTData service for MQTT-over-WebSocket URIs, such as
wss://data.iot.us-east-1.amazonaws.com/mqtt
or account-specific endpoint like
wss://ABCD1234EFGH56.iot.us-east-1.amazonaws.com/mqtt
to produce output like
wss://ABCD1234EFGH56.iot.us-east-1.amazonaws.com/mqtt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=...
Implemented in a separate file for convenience for now,
brianfaull@105b5fb
It uses a different signature than the S3-signing mechanism, handling URIs merely as string rather than embedded within http.Request for practical reasons.
I hope this is helpful. Feel free to incorporate it if you'd like. Alternatively, if you have requests for better integration and a proper pull-request, let's talk more.

request.Host may include port, causes incorrect signature

Right now setting the Host header to the value of request.Host may also include the port (if specified) and causes an incorrect canonicalURL string to get hashed.

Looks like this is fixed by PR #27

tag a release

Any chance you could tag a release? coreos/rkt much prefer to only depend on tagged projects (see rkt/rkt#2428). As they put it:

Please consider assigning version numbers and tagging releases. Tags/releases
are useful for downstream package maintainers (in Debian and other distributions) to export source tarballs, automatically track new releases and to declare dependencies between packages. Read more in the Debian Upstream Guide.

Versioning provides additional benefits to encourage vendoring of a particular (e.g. latest stable) release contrary to random unreleased snapshots.

Versioning provides safety margin for situations like "ops, we've made a mistake but reverted problematic commit in master". Presumably tagged version/release is better tested/reviewed than random snapshot of "master" branch.

can't override service/region magic

The magic code for parsing service/region from the service domain does not work in our environment since we refer to our services via an indirect CNAME which does not follow the AWS format.

If we could explicitly provide service and region this would not be an issue. Will attach PR with workaround.