rdwilliamson / aws Goto Github PK

Hello,

I've been doing some extensive debugging and believe I've found an issue with the treehasher: it doesn't pass tests for parts greater than 3MB. If I construct a data array in advance of, say, 3.5 MB of the byte 'a', and write it all at once to the TreeHasher, the checksum it calculates is wrong.

If instead I write the byte 'a' to the treehasher iteratively (3.5 MB worth), it outputs the correct checksum. For cases (1 << 20), (2 << 20), (3 << 20), both ways output the same result, and are correct. Do you know what might be the problem? I've hand-computed the checksums and also checked them against other repos (for example, https://github.com/jclouds/jclouds-labs-aws/blob/master/glacier/src/test/java/org/jclouds/glacier/util/TreeHashTest.java#L68 for the canonical treehash of the 3.5 MB bytestring; https://chromium.googlesource.com/external/boto/+/master/tests/unit/glacier/test_utils.py#108 for 4MB), so I don't think the checksum in the test case itself is the problem.

I've uploaded the failing test cases. 3.5 and 4 fail. Note that everything smaller, passes. https://github.com/recursionpharma/aws/blob/treehasher/glacier/treehash_test.go#L93

Please let me know.

Thanks!

The function ListMultipartParts adds the limit and marker parameters to the header, instead of the URL string. So, the request doesn't recognize them properly.

I fixed it and my tests ensure that pagination now works. I added the code to the pull request in #11. I suspect that ListMultipartUploads and any other GET requests suffer from the same problem, but I don't use them so I haven't fixed them (perhaps you can take a look this weekend).

By the way, I have a pretty thorough multipart-uploading test suite I would like to add (takes 77s to run). It requires AWS credentials so I'll add them in a separate pull request, but I'm not sure what the best way to structure it would be (simply don't run the tests if the env variables aren't set?)

Hi,

I'm successfully uploading/restoring/describing job, but the ListJobs operation seems not to work, I got this strange error:

ResourceNotFoundException: Client: Vault not found for ARN: arn:aws:glacier:eu-west-1:687173749091:vaults/jobs

I don't understand why there is /jobs at the end of the ARN instead of my vault name, the ARN for a describe job query is: arn:aws:glacier:eu-west-1:687173749091:vaults/thenameofmyvault ?

Any thoughts ?

Thanks!

Hi Robert, nice library -- any license info?

Thx!

If you call Connection.UploadArchive with a ReadSeeker that's empty (e.g., for a zero byte file, or the buffer is already fully consumed before passing it to the function), the function panic()s in line 38: th.TreeHash() with runtime error: index out of range.
It would be good to trap this error by checking if request.ContentLength > 0, and if it isn't, return with an error (return nil, error.New("Emtpy buffer passed")).

Hi,

Thanks for this great library. I'm trying to upload some massive (TB++) files to Glacier, and I noticed that completing the multipart upload requires a TreeHash. I don't want to re-read all the parts and calculate a TreeHash from that, or compute another TreeHash every part, since UploadMultipart already does this.

Is the recommended method to upload all the parts, call ListMultipartParts, go through the resulting []MultipartPart, self-compute the resulting TreeHash, and then call CompleteMultipart?

If so, shall I write a function for TreeHash that can compute the full TreeHash for a multipart upload given the treehash of all of its parts, as described here: http://docs.aws.amazon.com/amazonglacier/latest/dev/checksum-calculations.html

Thanks in advance.