go-amz / amz Goto Github PK

How come we have two goamz now? Isn't https://github.com/mitchellh/goamz used more?

@kat-co One issue v4 signing introduces is that it uses a lot of memory use when uploading files to S3 (eg. with PutReader). For example, I concurrently upload 10 x 50 MB videos and consume 500MB of memory.

Would it be possible to do this without using ioutil.ReadAll(req.Body)?

https://github.com/go-amz/amz/blob/v4-unstable/aws/sign.go#L272

I started looking into v4 signing a week or two ago, but haven't done an implementation so far. Might we need to go so far as chucked uploads? http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html

Maybe there is a way I can use v2 signing for the moment?

While adding v4 signature support for S3, I had to modify our signing algorithm to add additional headers required for S3. The existing v4 signing tests compare against a pre-computed hash which was calculated without these headers. This invalidates these tests for now, but we should revisit these tests to see if we can't get more testing for v4.

I want to add CloudFront cache invalidation, but to where?

https://github.com/goamz/goamz already has some CloudFront stuff

https://github.com/mitchellh/goamz I've been using this fork up until now (due to #4).

also https://github.com/crowdmob/goamz

https://gopkg.in/amz.v1 reports the error:

GitHub repository at https://github.com/go-amz/amz has no branch or tag "v1", "v1.N" or "v1.N.M"

See niemeyer/gopkg#20.

A temporary workaround may be to create a master branch.

Hello,

 We are trying to retrieve the first 64 bytes of an object and we see that there is no method to achieve this in your package. The S3 documentation knows it as Range Header, is there a way to achieve that request by adding that header or will it be necessary to create a new method?

 GetContent(path string, offset uint64, size uint64) (data []byte, err error)
 GetContentReader(path string, offset uint64, size uint64) (rc io.ReadCloser, err error)

 Greetings

I'm testing the readiness of deployments to cn-north-1. The stream data was fixed yesterday (great - #21), but when bootstrapping now, I get the error:

2015-01-28 02:00:39 ERROR juju.cmd supercommand.go:323 cannot determine if environment is already bootstrapped.: The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.

It's worth noting that accessing cn-north-1 via the awscli works without issue (as far as I tested): https://pastebin.canonical.com/124392/

More details: https://pastebin.canonical.com/124394/

Expected Results

When server running on UTC+8 timezone and current time is Thu, 18 Aug 2016 08:12:34 UTC+8, the generated X-Amz-Date should be 20160818T001234Z.

Actual Results

Currently, the result is 20160818T081234Z. A 403 response will be received when following the generated URL.

http://golang.org/pkg/net/http/#Client.Do
Callers should close resp.Body when done reading from it. If resp.Body is not closed, the Client's underlying RoundTripper (typically Transport) may not be able to re-use a persistent TCP connection to the server for a subsequent "keep-alive" request.

We encountered stale sockets for the process and at the end it fails with too many open files. It happens with s3-eu-west-1.amazonaws.com backend.

go version 1.3

SessionToken is important to us because we use Temporary Security Credentials to access AWS resources for our applications.

In order to support Temporary Security Credentials, you must supply the AccessKey, SecretKey, and the SessionToken which is missing from this version.

It would be very handy to have a bucket.GetWriter() similar to bucket.GetReader() for writing to an S3 object. It would allow people to leverage more io.* and ioutils.* functionality, as well as libraries which gather write throughput statistics.

A surface level comparison of:

• https://godoc.org/gopkg.in/amz.v1/s3
• https://godoc.org/github.com/mitchellh/goamz/s3
• https://godoc.org/github.com/crowdmob/goamz/s3
• https://godoc.org/github.com/goamz/goamz/s3

mitchellh/goamz/s3

mitchellh/goamz/s3 doesn't have func RetryAttempts(retry bool) but there are several added methods:

func (b *Bucket) Copy(oldPath, newPath string, perm ACL) error
func (b *Bucket) GetBucketContents() (*map[string]Key, error)
func (b *Bucket) GetKey(path string) (*Key, error)
func (b *Bucket) GetResponse(path string) (*http.Response, error)
func (b *Bucket) GetTorrent(path string) ([]byte, error)
func (b *Bucket) GetTorrentReader(path string) (io.ReadCloser, error)
func (b *Bucket) Head(path string) (*http.Response, error)
func (b *Bucket) MultiDel(paths []string) error
func (b *Bucket) PutHeader(path string, data []byte, customHeaders map[string][]string, perm ACL) error
func (b *Bucket) PutReaderHeader(path string, r io.Reader, length int64, customHeaders map[string][]string, perm ACL) error
func (s3 *S3) ListBuckets() (result *ListBucketsResp, err error)

crowdmob/goamz/s3

crowdmob/goamz/s3 has added some of the same methods as mitchellh/goamz/s3

func (b *Bucket) GetResponse(path string) (resp *http.Response, err error)

and others, but with a different method signature:

func (b *Bucket) Head(path string, headers map[string][]string) (*http.Response, error)

it has also changed some of the amz.v1 methods to take an Options struct:

func (b *Bucket) InitMulti(key string, contType string, perm ACL, options Options) (*Multi, error)
func (b *Bucket) Put(path string, data []byte, contType string, perm ACL, options Options) error
func (b *Bucket) PutReader(path string, r io.Reader, length int64, contType string, perm ACL, options Options) error

and then added a whole slough of new methods, some that sound like they accomplish the same things as mitchellh/goamz/s3, but in a different way:

func (b *Bucket) DelMulti(objects Delete) error
func (b *Bucket) DeleteLifecycleConfiguration() error
func (b *Bucket) Exists(path string) (exists bool, err error)
func (b *Bucket) GetLifecycleConfiguration() (*LifecycleConfiguration, error)
func (b *Bucket) GetResponseWithHeaders(path string, headers map[string][]string) (resp *http.Response, err error)
func (b *Bucket) Location() (string, error)
func (b *Bucket) PostFormArgs(path string, expires time.Time, redirect string) (action string, fields map[string]string)
func (b *Bucket) PostFormArgsEx(path string, expires time.Time, redirect string, conds []string) (action string, fields map[string]string)
func (b *Bucket) PutBucketSubresource(subresource string, r io.Reader, length int64) error
func (b *Bucket) PutBucketWebsite(configuration WebsiteConfiguration) error
func (b *Bucket) PutCopy(path string, perm ACL, options CopyOptions, source string) (*CopyObjectResult, error)
func (b *Bucket) PutLifecycleConfiguration(c *LifecycleConfiguration) error
func (b *Bucket) SignedURLWithArgs(path string, expires time.Time, params url.Values, headers http.Header) string
func (b *Bucket) UploadSignedURL(path, method, content_type string, expires time.Time) string
func (b *Bucket) Versions(prefix, delim, keyMarker string, versionIdMarker string, max int) (result *VersionsResp, err error)
# ... and a whole lot more (those are just under bucket)

goamz/goamz/s3 looks pretty similar to crowdmob/goamz/s3.

This isn't to say all these methods should be added. But it does mean I can't take even a trivial project that was using mitchellh/goamz/s3 and recompile it with amz.v1/s3. Handy methods like GetBucketContents and MultiDel don't exist.

As another point of comparison, there is Stripe's generated API:
http://godoc.org/github.com/stripe/aws-go/gen/s3

Note: I can't currently refer to http://godoc.org/gopkg.in/amz.v2-dev/s3 for comparison.

When uploading a file to S3 using PutReaderWithHeaders.

Query-string authentication requires the Signature, Expires and AWSAccessKeyId parameters

I'm curious why SignV2 chooses to use query-string authentication?

req.URL.RawQuery = queryVals.Encode()

vs. setting a header:

req.Header.Set("Authorization", "AWS "+ auth.AccessKey +":"+ string(signature))

The endpoints used in goamz for cn-north-1 [1] use the domain:

amazonaws.com.cn

whereas that published in the stream [2] uses:

amazonaws-cn.com.cn

This means that when bootstrapping (from a non-cn client machine), it fails to find a matching endpoint [3]

That said, as Haw pointed out, the amazonaws.com.cn one from goamz resolves, while the stream one doesn't resolve (from outside .cn) [4], so if it is just the stream being incorrect, this bug is invalid :)

We'd like to try to deploy a staging service to cn-north-1 as soon as we're allowed, but would be blocked on this.

[1] https://github.com/go-amz/amz/blob/v1/aws/aws.go#L145
[2] http://cloud-images.ubuntu.com/releases/streams/v1/index.json
[3] http://paste.ubuntu.com/9890087/
[4] http://paste.ubuntu.com/9890097/

Hi,

I was going through the apis available for S3 operations. I see that i can set the custom header info using PutHeader() API while creating the object. But I don't see a way to retrieve the the header that was set with the object.
My requirement is to set the custom details while putting it to the bucket and get the list of custom-details that i had set earlier.
Is this possible ?

Thanks and Regards,
Madhukar