Resumable uploads about drive

[kcwu]

I experimented resumable upload. Here are my findings

1. need to local patch google-api-go-client
   https://groups.google.com/forum/#!topic/golang-nuts/QlUQWq061XA
   otherwise you will get "resumable uploads must set the Name parameter. " error.
2. good news is, no longer auth error when push huge file (issue #53, #116)
   I succeeded uploaded a 22GB file in 214 minutes.
3. The upload chunk size is hard coded 256kb. You may want to local patch the chunk size. Otherwise it is very slow.
   For my case, 22GB/214minutes = 1.76MB/s
   Compared to my non-resumable upload 4.5MB/s
4. Bad news: uploaded file name always become "Untitled"
   I found google-api-go-client has this line
   req.Body = nil
   so the file name is not sent to server.
   I don't know how to fix it yet.

[odeke-em]

Thank you for investigating these. Actually I mostly calcuate upload chunk sizes, but have a default. For some services like AWS's S3, for multi part uploads, the chunks should be something like 5MBs size minimum. I'll see how we what's up as we step through the code when implementing it. If need be, we shall patch the Go api client. I actually already have a clone at https://github.com/odeke-em/google-api-go-client in case we need to fix it for our uses.

[kcwu]

FYI, my experiment code kcwu@a4d9a33

[odeke-em]

Nice. However, you might need to change a whole bunch of things related to resumable uploads. You need to actually start a resumable upload session and then save the resumable upload URI token and pass that on every request. To get more information on this, checkout https://developers.google.com/drive/web/manage-uploads#resumable

[odeke-em]

Oh btw we shall need a mechanism for saving that info either in memory or to disk for every resumable upload, in scenarios like on Interrupt (Ctrl + C) as well as on upload fails and then read from there on the next retrial. Also we need to get in a retry mechanism, I have a simple lib started for exponential back-off so that we can detect when an upload might have failed while the data is still in memory.

[kcwu]

If we only want to "resume" in the same upload process, google-api-go-client already did handle these session/token/retry things.

BTW, current implementation of google-api-go-client requires file size known before upload start. If we want to support -piped upload, we have to hack google-api-go-client as well.

[odeke-em]

But resumable uploads are necessary if for example someone hits Ctrl-C as well as if you get a network error so we still to save this information.
Yeah we shall see about that, or for a simple start use resumable uploads only for stat-able files.

[odeke-em]

@kcwu, just a heads-up there should no longer be OAuth timeout errors.

[odeke-em]

@kcwu Happy New Year! Just as a random note: I was about to go to bed and tried pushing up a few files but saw the need to change the chunksize, so I implemented #840.