ttlabs / evaporatejs Goto Github PK

I'm starting to use EvaporateJS, great library!

I'd like to contribute with a Java signing example:

import java.io.IOException;

import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import javax.servlet.ServletException;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import org.apache.commons.codec.binary.Base64;
import org.apache.commons.lang3.StringUtils;

@WebServlet("/signAuth")
public class SigningExample extends HttpServlet
{
    private static final long   serialVersionUID    = 1L;

    private static final String HMAC_SHA1_ALGORITHM = "HmacSHA1";

    @Override
    protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException
    {
        String data = request.getParameter("to_sign");

        if(StringUtils.isEmpty(data))
        {
            response.sendError(HttpServletResponse.SC_BAD_REQUEST, "Invalid data: 'to_sign' parameter not informed");
        }
        else
        {
            response.getWriter().write(calculateRFC2104HMAC(data, "<<YOUR_AWS_SECRET_KEY>>"));
        }
    }

    /**
     * http://docs.aws.amazon.com/AmazonSimpleDB/latest/DeveloperGuide/HMACAuth.html#AuthJavaSampleHMACSignature
     * 
     * Computes RFC 2104-compliant HMAC signature.
     * 
     * @param data
     *           The data to be signed.
     * @param key
     *           The signing key.
     * @return The Base64-encoded RFC 2104-compliant HMAC signature.
     * @throws ServletException
     * @throws java.security.SignatureException
     *            when signature generation fails
     */
    public static String calculateRFC2104HMAC(String data, String key) throws ServletException
    {
        String result;
        try
        {
            // get an hmac_sha1 key from the raw key bytes
            SecretKeySpec signingKey = new SecretKeySpec(key.getBytes(), HMAC_SHA1_ALGORITHM);

            // get an hmac_sha1 Mac instance and initialize with the signing key
            Mac mac = Mac.getInstance(HMAC_SHA1_ALGORITHM);
            mac.init(signingKey);

            // compute the hmac on input data bytes
            byte[] rawHmac = mac.doFinal(data.getBytes());

            // base64-encode the hmac
            result = new String(Base64.encodeBase64(rawHmac));
        }
        catch(Exception e)
        {
            throw new ServletException("Failed to generate HMAC: " + e.getMessage());
        }
        return result;
    }
}

I recently run into some RequestTimeTooSkewed errors because of daylight saving time: Example:

My computer: new Date() output is  Thu Nov 06 2014 13:49:36 GMT-0200 (Horário brasileiro de verão) // works fine
Client computer: new Date() output is Thu Nov 06 2014 13:47:15 GMT-0300 (Hora oficial do Brasil) // RequestTimeTooSkewed error

Because of that new Date().toUTCString() gives different values.

I solved this by getting the time from my server, which has NTP installed, so I changed the setupRequest function:

           if(!con.timeUrl)
           {
              requester.dateString = new Date().toUTCString();
           }
           else
           {
              requester.dateString = $.ajax({
                  type: "GET",
                  url: con.timeUrl,
                  cache: false,
                  async: false
              }).responseText;
           }

Server returns the date formatted in the RFC 1123 pattern, same output from JavaScript new Date().toUTCString().

I'd like to suggest this idea to be merged in the official code.

Thanks

Hello everyone !

I'm currently trying to setup a direct upload from angular-file-upload to Amazon S3 bucket.
The upload process works fine, ex: If I upload an image of 3MB, the image is correctly transferred on my Amazon S3 bucket (located in Francfort btw, using the new AWS V4 signature)

The issue is the following one: The uploaded files cannot be opened and when I try to "cat" to see the binary content of the file, there are 3 lines added:

------WebKitFormBoundaryfOtaE6TQBF0hzBnw
Content-Disposition: form-data; name="file"; filename="01_-_Rock_Or_Bust.mp3"
Content-Type: audio/mp3

For each uploaded file, there is this "webkitformboundary" stuff added inside my file's binary content which is the reason of my issue I guess.

Does anyone know how to solve it?

I have a Node.js server that is basically generating pre-signed url and my app using this plugin to do the upload to S3.

I've been looking to solve this with my team for 24 hours now but no clue at moment.
Any ideas?

Best regards,

The url signing endpoint that I am using requires Authentication, therefore I need to configure evaporate.add calls with the signHeaders option which causes the error in the subject.

signHeaders: {
Authorization:'Bearer ' + Authentication.token
},

I was able to alter the evaporate library to set the headers after the call to xhr.open which solved the issue. Perhaps I can submit a pull request

( in upload.on200)
var eTag = xhr.getResponseHeader('ETag')
This returns null when running in Android 4.x versions until version 4.4 on StockBroswer even though I see in Fiddler that eTag is returned.
This seems to be a bug in Android implementation - https://bugs.webkit.org/show_bug.cgi?id=41210
(Cross Origin XMLHttpRequest can not expose headers indicated in Access-Control-Expose-Headers HTTP Response Header)
Works on Stock browser in versions 4.4+

The progress and complete callbacks appear in the .add documentation, but it looks like there are more than that - cancelled, info, warn, and error:

https://github.com/TTLabs/EvaporateJS/blob/master/evaporate.js#L129-L134

Are these ok to use? If so, can we get these added to the docs?

As I read it:

• cancelled() gets called when a successful cancel is called for an upload id.
• info(msg) gets called with a debug/info message, usually logged as well.
• warn(msg) gets called on a potentially recoverable error, and will be retried (e.g. part upload).
• error(msg) gets called on an irrecoverable error.

Are those about right?

Is there any ways to calculate the remaining time to upload in the progress callback?

Once this is in place, the library will become much more useful.

Map out browsers that EvaporateJS actually work well on so it is more clear where it works and where it does not.

I have started working on an angular module to make using evaporate even easier.

https://github.com/sourcec0de/ng-evaporate

This is just my proof of concept but now that this is finished I will implement the directive so you can just include the HTML and drop it into your angular project for file upload awesomeness.

https://github.com/aws/aws-sdk-js has existed for a few months at least, and includes official implementations of things like uploadPart, completeMultipartUpload, etc. It's probably a good idea to use those officially maintained and tested methods.

This is the issue to track the existing TODO to add support for the Content-MD5 request header feature supported by AWS.

In my app I send the POST request, successfully https://s3.amazonaws.com/linguakzweb/1d5jq34iclihx1ij7gni.webm?uploads . Then PUT request invoked, the response says that I have wrong signature while the POST request was successful. PUT https://s3.amazonaws.com/linguakzweb/dxqyi6jk216p...Z9_qKTHllKVFO3u03mB_nCNTku6RpDOqKyGKp8qx85eY5w--.
Log says - ### undefined.

Response:

SignatureDoesNotMatchThe request signature we calculated does not match
the signature you provided. Check your key and signing method.AKIAIWG6FO4WCPJ3Z5MQ
PUT

text/plain; charset=UTF-8

x-amz-acl:public-read
x-amz-date:Tue, 18 Aug 2015 14:07:31 GMT
/linguakzweb/164mc2sl97jvgah3jji.webm?partNumber=1&uploadId=fjoMSLJgmYhRnPjuZF4ql5jTOGAgZgHQLekx
.f1gxn05uoUo_VOjEuthU814VMGBPOI.Nn0rvbdKGy4fNfmdLQ--d1JrURO9ezOUHCDrhSH0fP424dI
=50 55 54 0a 0a 74 65 78 74 2f 70 6c 61 69 6e 3b 20 63 68 61 72
73 65 74 3d 55 54 46 2d 38 0a 0a 78 2d 61 6d 7a 2d 61 63 6c 3a 70 75 62 6c 69 63 2d 72 65 61 64 0a 78
2d 61 6d 7a 2d 64 61 74 65 3a 54 75 65 2c 20 31 38 20 41 75 67 20 32 30 31 35 20 31 34 3a 30 37 3a 33
31 20 47 4d 54 0a 2f 6c 69 6e 67 75 61 6b 7a 77 65 62 2f 31 36 34 6d 63 32 73 6c 39 37 6a 76 67 61 68
33 6a 6a 69 2e 77 65 62 6d 3f 70 61 72 74 4e 75 6d 62 65 72 3d 31 26 75 70 6c 6f 61 64 49 64 3d 66 6a
6f 4d 53 4c 4a 67 6d 59 68 52 6e 50 6a 75 5a 46 34 71 6c 35 6a 54 4f 47 41 67 5a 67 48 51 4c 65 6b 78
2e 66 31 67 78 6e 30 35 75 6f 55 6f 5f 56 4f 6a 45 75 74 68 55 38 31 34 56 4d 47 42 50 4f 49 2e 4e 6e
30 72 76 62 64 4b 47 79 34 66 4e 66 6d 64 4c 51 2d 2d5DE43823DC3E540A
9tqPAQ/xY0trw5J1IKX4E5iFpWKMJmimXEmDI3HaJHOyFgvgrJ1VQlgA/MqkVU4W

Please help me.

Clarify the usage of PUT vs POST for adding a new part to S3, provide a sample CORS policy that works in development and provide implementations that might make more sense for production (e.g. embedding an appropriate policy in the returned signature).

When trying to run the example, I get "Error onfailedauth for step initiate". What do I need to provide in order to track this error down?

Here's what I changed in evaporate_example.html:

     var _e_ = new Evaporate({
        signerUrl: 'http://localhost:8080/sign_auth',
        aws_key: 'AKIAJVBW466OXU6WHYLQ',
        bucket: 'infogen2-storage',
     });

... and I added an error handler function so I could see what was going wrong. Also I set my secret key in signing_example.py.

thanks for any help you can provide.

Just make it easier for people want to use this lib for distribution

Excuse my ignorance if this is already possible, but would it be considered better practise to determine the file storage location when signing the upload to prevent the user from setting any storage location they like?

I was just wondering if there is any way that I can change the configuration for each file. I have an upload service that has a unique signerUrl and AWS public key per upload. Is this possible without any major overhauls?

Thanks for the great software!

I am uploading file using Server-Side Encryption with Customer-Provided Encryption Keys (SSE-C), but I got SignatureDoesNotMatch error.

actual request:

x-amz-acl:private
x-amz-date:Wed, 08 Oct 2014 15:44:52 GMT
x-amz-server-side-encryption-customer-algorithm:AES256
x-amz-server-side-encryption-customer-key:***
x-amz-server-side-encryption-customer-key-MD5:***

AWS calculated:

x-amz-acl:private
x-amz-date:Wed, 08 Oct 2014 15:44:52 GMT
x-amz-server-side-encryption-customer-algorithm:AES256
x-amz-server-side-encryption-customer-key:***
x-amz-server-side-encryption-customer-key-md5:***

the different is MD5 be changed to md5.

I suspect the issue is in
function makeStringToSign(request){

        var x_amz_headers = '', to_sign, header_key_array = [];

        for (var key in request.x_amz_headers) {
            if (request.x_amz_headers.hasOwnProperty(key)) {
                header_key_array.push(key);
            }
        }
        header_key_array.sort();

        header_key_array.forEach(function(header_key,i){
            x_amz_headers += (header_key + ':'+ request.x_amz_headers[header_key] + '\n');
        });


        to_sign = request.method+'\n'+
            '\n'+
            (request.contentType || '')+'\n'+
            '\n'+
            x_amz_headers +
            request.path;
        return encodeURIComponent(to_sign);
    }
}

any comments?

I am evaluating your Evaporate for uploading documents to s3. I have got the basic example working but I have a couple of questions. First, is your auth_sign method same as pre-signed s3 urls, if not how can I add its support. Second, is there a way to add encryption to the upload function or should I use server side encryption?

Thanks.

Uncaught ReferenceError: log is not defined evaporatejs.js:1
(anonymous function) evaporatejs.js:1

version 0.0.2
In the head I have

Would love to be able to install this from npm. Would you mind adding a package.json and publishing to npm?

The comments mention calculating MD5 and sending them to server and sending ETags of each part to server on part complete to provide support for resumability across different sessions. Can you please elaborate a bit more on how do you plan to use MD5 and ETags to achieve this? I am planning to implement it.

Thanks for your work here, this is just what I am looking for.
Does EvaporateJS support parallel multi-part uploads?

In order to run the demo a app.yaml file is needed, this file seems to be not checked into git.

I am wondering if there is any way to enable dropzone for the upload component, so that instead of clicking to upload, people can drag the file to the upload zone to start upload the component. It would be helpful to have the dropzone setting as jquery file upload. Thanks.

I have a costumer that has a database update process where there is a set of SQL scripts, named in sequence. He needs to upload all files, even if some are empty to don't break the sequence.

Evaporate by default fails when the file is empty because no part is uploaded, causing the complete upload request to be malformed:

<Error><Code>MalformedXML</Code><Message>The XML you provided was not well-formed or did not validate against our published schema</Message><RequestId>E00B21BDF42C8090</RequestId><HostId>Pv/R73AHwE/9KN8mNv+hihZGB3AkmVJXM9GlMvG1SG68FZ1a62DsitoVUxbK81rt</HostId></Error>

I made some minor changes to have at least one part upload and I created a isEmpty flag on the part to respect the current empty blob etag md5 check.

Can I create a pull request for you to evaluate the possibility of merging it on master?

Thanks

Incase something break, the website could notify the user and take other required action.

It should be possible to do a speed test by profiling how long it takes to upload each part. This used in conjunction with the total file size can inform the user of time remaining for each upload.

I cannot seem to retrieve the ETag from the multipart upload response headers. I can successfully upload each chunk of the file but when it comes time to complete the multipart upload I am seeing this error:

<Error><Code>InvalidPart</Code><Message>One or more of the specified parts could not be found.  The part may not have been uploaded, or the specified entity tag may not match the part's entity tag.</Message><UploadId>cawD5yowF97n6lErE88PcRtNfz8qeZIcoSLuWk77yPVq0.SgpbcRT_ShZjGnFFB61Vyq.caQGCPx28bSR_CaEA--</UploadId><ETag>null</ETag><RequestId>A8DB1B2F4F3F3C34</RequestId><HostId>/hQ8RMrnOYwtxWrOFD1vvNixxUAVmmQOqjXSPMsjDQ7qcBuQWBggINpa4IPGRagX</HostId><PartNumber>1</PartNumber></Error>

This is the error I see when I am uploading each chunk

I am running the application using the example python script provided and have only added my bucket and AWS key.

Around Daylight Savings Time shift we started receiving issues with uploads failing silently. The underlying issue was AWS was responding to uploads with: RequestTimeTooSkewed - The difference between the request time and the current time is too large.

The date is using new Date().toUTCString(); which can be incorrect if the client's time is incorrect. I believe this should come from the server and not the client.

If I submit a pull request to fall back on the server time (received when signing the request) would this be accepted?

Also, I could be missing another way to handle this. Is there one?

Hi,
Any ETA on the pause and resume functions?

Thanks,
Julien

In order to minimize cross-talk, EvaporateJS should batch-generate signatures. This avoids hitting the application server whenever a new part can be uploaded so there is less time wasted between parts.

Modifications can go into makeParts().

This was brought up in another issue.

Basically, in most cases the signing request should be checked server-side to make sure the user should have access to the place they upload to. It would be nice if this kind of check were included in the example. Blindly signing will result in a malicious user being able to overwrite every file in the bucket.

It seems the client can completely control what key an object gets when uploaded to S3. If you wanted to enforce a particular key from the server, what are the best practices?

When the signing URL is called, I could technically extract the object key from the to_sign data, but I was wondering if there was a more elegant approach.

I saw that there was a bower.json file in the repository, but couldn’t install via Bower using the short name (EvaporateJS). Even searching for just “evaporate” (bower search evaporate) yielded no results.

I can work around this by specifying the GitHub username and repository name (bower install TTLabs/EvaporateJS) but would be good if this package could be published to Bower so I can install using the short name.

We have an application that works great with Evaporate during browser testing. Unfortunately when running within a Capybara-Webkit session it does not.

The string that is signed for the initiate request contains no content-type:

POST\n\n\n\nx-amz-date:Tue, 06 Jan 2015 11:33:18 GMT\n/new-distrify-test-uploads/1%2Fblack.mp4?uploads

Whereas the one in the error returned by Amazon does:

POST\n\napplication/x-www-form-urlencoded\n\nx-amz-date:Tue, 06 Jan 2015 11:33:18 GMT\n/new-distrify-test-uploads/1%2Fblack.mp4?uploads

I am not sure why this is the case. The content-type does not seem to be getting explicitly set on the initiate request. Perhaps it always should be, and the signature generated with this?

Store the file in local storage before uploading, and store relevant metadata (parts uploaded, etc) with the file, so if the browser had to relaunch, the upload session can continue.

This is probably not practical for video files or other kinds of very large files.

Let's say a file is 60% done, and the upload irrecoverably fails. Then the user tries to upload again a few hours later (not necessarily from the same browser or even machine). We could 'resume' the upload starting at 61%, if we have the etags for the first 60%.

1. as etags of completed parts are received, POST to an application server (probably in batches, to guarantee order)
2. when a file is passed to evaporate, check with application server whether some etags already exist for the file (file would need some 'unique' id, which would probably have to be name+lengthInBytes). Maybe could be hash of file, but not if it requires entire file being loaded into memory :)
3. if existing etags are available then use those, rather than uploading the corresponding part

Need to check how long the uncompiled parts are kept by AWS.

As temporary security credentials (IAM roles) will have different access key id and secure token on different EC2 machine, current implementation will get 403 (Forbidden) when the first request get security-token and a following request to sign the request params hits different ec2 instance.
These seems need to be setup in one request to server to get all params (including anything need to be signed) and signed signature.

I am trying to upload a file with the name "Scree-n Shot 2014-01[]' Ѯ Ч Л `-ê â î û-22 at 5.27.45 PM (1) (2) (1).png" and Firefox always fails the signature verification from S3. Chrome and Safari work great, so I am nervous to say this is a definite bug. Have you seen this issue yourself before?

I'd like to pull in EvaporateJS into a Rails project using https://rails-assets.org/. However to do this the repository needs to have semver-compliant git tags.

Would it be possible to create a tag for the 0.0.2 release in this Github repo?

I've got an error (latest Chrome, Firefox, Opera):

evaporate.js:167 processQueue length: 1
evaporate.js:194 starting FileUpload 0
evaporate.js:153 onFileUploadStatusChange
evaporate.js:167 processQueue length: 1
evaporate.js:578 setupRequest() Object {method: "POST", path: "/BUCKETNAME/test_585987047?uploads", step: "initiate", x_amz_headers: Object, not_signed_headers: Object…}
evaporate.js:659 authorizedSend() initiate
evaporate.js:680 Uncaught InvalidStateError: Failed to execute 'setRequestHeader' on 'XMLHttpRequest': The object's state must be OPENED.authorizedSend @ evaporate.js:680initiateUpload @ evaporate.js:264me.start @ evaporate.js:197processQueue @ evaporate.js:182

Looking at our logs, we see that occassionally parts get stuck in status == 2, and never do anything else. This causes failed uploads. I plan to implement a fix to this, which will essentially be:

on a periodic basis, check the byteloaded of each part
if the bytesloaded havent progressed then kill the part, and set back to 'pending'

If anyone has any thoughts / suggestions / comments then please chip in now, so I can include them when I code it up.

I've got a signer endpoint that creates an hmac using SHA + the access key to our aws account,

here's the add call

tmpOption = {
          name: 'test_' + Math.floor(1000000000*Math.random()),
          file: files[i],
          notSignedHeadersAtInitiate: {
            'Cache-Control': 'max-age=3600'
          },
          // xAmzHeadersAtInitiate : {
          //    'x-amz-acl': 'public-read'
          // },
          complete: function(){
             console.log('complete................yay!');
          },
          progress: function(progress){
             console.log('making progress: ' + progress);
          }
        };
        uploadOptions.push(tmpOption)
        var result = evaporator.add(tmpOption);
}

amazon returns the error:

The request signature we calculated does not match the signature you provided. Check your key and signing method.

What gives?

It is possible to replace parts of the code with the official js browser sdk for AWS. Specifically uploading parts and multipart upload initialization.

http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/frames.html

When signature is generated and the content type is posted to the method, any + in the content type are converted to spaces. Obviously this is because of the use of escape(). The problem is that means the signature is incorrect when it is generated. This could be handled in the server side code but wondering if there was a better fix?

My example is application/epub+zip which is for a ".epub" file.

Is there support for using the new AWS4 signature in EvaporateJS? This is required when connecting to buckets which are located in Frankfurt (eu-central-1).

I think the signing requires SHA256 instead of SHA1, but changing my signing-handler to use SHA256 only results in a fail-message:
"failed to get authorization (readyState=4) for initiate. xhr.status: 200. xhr.response: <...>"

Is it only the singing-handler which needs to be fixed? I found some thing here which might be interesting:
http://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html

Any ideas?

Do you have a sample of what signer_example.py returns?

EvaporateJS should support cancellation or expose a way to cancel things easily. Each upload can be represented as an individual object which exposes observable progress, and allows cancellation, pausing, and resumption.

Update: Cancellation seems already supported.