rclone / rclone Goto Github PK

"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files

Home Page: https://rclone.org

License: MIT License

Makefile 0.16% Shell 0.46% Go 98.06% Python 0.41% HTML 0.48% Batchfile 0.01% Dockerfile 0.04% JavaScript 0.29% C 0.05% PHP 0.03%

golang go cloud-storage s3 openstack-swift google-drive azure-blob backblaze-b2 sftp ftp

rclone's People

Contributors

Stargazers

Watchers

Forkers

perezkarjee rayleyva tjasond jmcarbo briceburg tinygrasshopper madralphw brin100 light9gmailcom alexcouper shalupov zeshanb shimondoodkin darren12345 x1011 klauspost simon joeygo23 vichetuc bobobo1618 nicot dapob1 danieldc hiddenloop bigmarvin meirelles starsoftanalysis ei-grad zhitko k4y5 dibu28 yami offspring bsdwatch bjornharrtell kpabba vlinhd11 corokada rivy escribano wernight vanphong3783 stengaard mafm greenyard pombredanne bogdan-pr abhaypsingh robmurtha kazagistar tuefekci binarytemple-clients tmancill srsmithdata ratuka wouterv heinsm nullprojects josejamilena devopsbox delian zakeerbasha mnjstwins mwitkow databus23 fafaman orgads yeonsh listeng stefhen rawdigits cloudxtreme rlapray mfhbam boreys shuhaowu jakop345 unixorn multikatt mo2tion alidzapp thebrave arcimboldo orinocoz yut148 cscazorla chriscorwin stefangweichinger olax youmuyou nwgat tjcomserv yogendersolanki91 curehsu colivi freedge veridian69 secfork-org linhua55 gcb23fish

rclone's Issues

Logging / stats should go out on stderr

Logging / stats should go out on stderr whereas listings etc should go out on stdout.

They can then be separated with a bit of shell redirection.

Implement a move command

Part of keeping things in sync is moving files from one place to another via Drive. Could you implement a command that moves all files from one place to another. Basically, it copies them and then after verifies successful copy on receiving end, deletes on source end?

"Deletes any files that exist in source that don't exist in destination" - optional

This is from the sync command documentation.

Am I correct in my interpretation here:

Create a bunch of files
Copy a bunch of files to Drive
Make a new file
Run sync <local> <remote>
New file is deleted

If so, I would very much like for this to be optional. If not, the documentation needs to be a bit clearer. When I sync, I'd prefer it if files that don't exist locally are deleted on the remote rather than the reverse.

drive: multiple files of same name being created

Great tool! Thanks for all your work on this and making it available.

Using rclone sync to a drive location, I'm seeing many copies of the same file/same name being created. Saw in the release notes that this was a fix in 1.06, but I've only ever used versions 1.07 and 1.09 (just downloaded 1.10 and will try that as well).

Possible causes include:

interrupted transfers? (it's a huge directory and hasn't finished)
maybe the deletions only occur after a sync is successful/complete?

Any thoughts?

Drive doesn't delete files the account doesn't own

1.10 is not deleting files that are not owned by the account owner. This gets in the way of keeping shared folders in sync.

Through the web-based UI you can still delete non-owned files, and you can also do it through Google Apps Scripts, so I'm pretty sure there is an API call you can make that will allow it. Try removeFile - that worked in an Apps Script I tried (and some other API calls to send the file to trash did not work).

Thank you

OK, more info:

When you purge a non-empty directory, even if that directory contains non-owned files, it works. But the problem is the only way to do it now is to remove the whole directory, and if it's a share directory, then you have to create a new directory and reshare it which isn't a good workflow.
When you run: rclone purge remote:SharedFolder/nonOwnedFile.txt it gives the errors:
-Error 403, insufficient permissions. This is incorrect, as the owner of the folder does have permission to delete a shared file.
-Failed to purge: can't rmdir in limited fs
When you run rclone purge remote:SharedFolder/OwnedFile.txt it actually does delete the file but it gives the error "Failed to purge: can't rmdir in limited fs."

Memory leak when copying from swift to google cloud storage bucket

While copying from swift to google storage bucket. Things were pretty smooth for a few hours and then GO application started issuing these errors. I've pasted the error log below this message.

rclone --checkers=1 --transfers=1 copy -q downloadfrom:container uploadto:bucket-name/folder

What if I use modify window option and set a long time difference? Not sure if this is what modify window option is used for? --modify-window=600000000000ns (600sec). Then the command for rclone will look like this:

rclone --checkers=1 --transfers=1 --modify-window=600000000000ns copy -q downloadfrom:container uploadto:bucket-name/folder

Thank you,
Zeshan

github.com/ncw/swift.(_Connection).ObjectsWalk(0xc208286240, 0x7fff38d0a6fa, 0x6, 0xc2083e9560, 0xc20805ef08, 0x0
, 0x0)
/home/ncw/Code/Go/src/github.com/ncw/swift/swift.go:840 +0xe7
github.com/ncw/rclone/swift.(_FsSwift).list(0xc208286240, 0xc208034000, 0xc20805efc0)
/home/ncw/Code/Go/src/github.com/ncw/rclone/swift/swift.go:223 +0x155
github.com/ncw/rclone/swift.func·003()
/home/ncw/Code/Go/src/github.com/ncw/rclone/swift/swift.go:246 +0xc4
created by github.com/ncw/rclone/swift.(_FsSwift).List
/home/ncw/Code/Go/src/github.com/ncw/rclone/swift/swift.go:247 +0x1a1
goroutine 1806459 [select]:
net/http.(_persistConn).writeLoop(0xc208048370)
/home/ncw/Code/go/src/net/http/transport.go:945 +0x41d
created by net/http.(_Transport).dialConn
/home/ncw/Code/go/src/net/http/transport.go:661 +0xcbc
goroutine 295 [select]:
net/http.(_persistConn).readLoop(0xc2082eea50)
/home/ncw/Code/go/src/net/http/transport.go:928 +0x9ce
created by net/http.(_Transport).dialConn
/home/ncw/Code/go/src/net/http/transport.go:660 +0xc9f
goroutine 296 [select]:
net/http.(_persistConn).writeLoop(0xc2082eea50)
/home/ncw/Code/go/src/net/http/transport.go:945 +0x41d
created by net/http.(_Transport).dialConn
/home/ncw/Code/go/src/net/http/transport.go:661 +0xcbc
goroutine 625446 [IO wait]:
net.(_pollDesc).Wait(0xc20800b5d0, 0x72, 0x0, 0x0)
/home/ncw/Code/go/src/net/fd_poll_runtime.go:84 +0x47
net.(_pollDesc).WaitRead(0xc20800b5d0, 0x0, 0x0)
/home/ncw/Code/go/src/net/fd_poll_runtime.go:89 +0x43
net.(_netFD).Read(0xc20800b570, 0xc20a6f0000, 0x8000, 0x8000, 0x0, 0x7fc5fc3d2c58, 0xc20a816b50)
/home/ncw/Code/go/src/net/fd_unix.go:242 +0x40f
net.(_conn).Read(0xc26cc44c50, 0xc20a6f0000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/home/ncw/Code/go/src/net/net.go:121 +0xdc
crypto/tls.(_block).readFromUntil(0xc2082f7380, 0x7fc5fc3d43a8, 0xc26cc44c50, 0x5, 0x0, 0x0)
/home/ncw/Code/go/src/crypto/tls/conn.go:454 +0xe6
crypto/tls.(_Conn).readRecord(0xc2080b3b80, 0x17, 0x0, 0x0)
/home/ncw/Code/go/src/crypto/tls/conn.go:539 +0x2da
crypto/tls.(_Conn).Read(0xc2080b3b80, 0xc26cbd0000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/home/ncw/Code/go/src/crypto/tls/conn.go:904 +0x166
net/http.noteEOFReader.Read(0x7fc5fc3d8f30, 0xc2080b3b80, 0xc208048688, 0xc26cbd0000, 0x1000, 0x1000, 0xc26d31a0c
0, 0x0, 0x0)
/home/ncw/Code/go/src/net/http/transport.go:1270 +0x6e
net/http.(_noteEOFReader).Read(0xc2173d7400, 0xc26cbd0000, 0x1000, 0x1000, 0xc208010000, 0x0, 0x0)
:125 +0xd4
bufio.(_Reader).fill(0xc247417f20)
/home/ncw/Code/go/src/bufio/bufio.go:97 +0x1ce
bufio.(_Reader).Peek(0xc247417f20, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/ncw/Code/go/src/bufio/bufio.go:132 +0xf0
net/http.(_persistConn).readLoop(0xc208048630)
/home/ncw/Code/go/src/net/http/transport.go:842 +0xa4
created by net/http.(_Transport).dialConn
/home/ncw/Code/go/src/net/http/transport.go:660 +0xc9f
goroutine 625447 [select]:
net/http.(_persistConn).writeLoop(0xc208048630)
/home/ncw/Code/go/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
/home/ncw/Code/go/src/net/http/transport.go:661 +0xcbc

google cloud storage - manipulation beyond root path fails

Manipulation commands that modify files/directories underneath a bucket fail silently using google cloud storage. Root path commands (such as bucket creation) do work. E.g.

rclone mkdir google:my-bucket/test-dir will NOT create test-dir under my-bucket.
rclone mkdir google:my-other-bucket will create a bucket named my_other_bucket

The inability to manipulate files under buckets also breaks sync commands -- they appear to do nothing.

details

rclone version 1.07 , precompiled amd64 binary from downloads.rclone.org

I've setup a google cloud storage remote named google via rclone config :

[google]
type=google cloud storage
client_id=__HIDDEN__
client_secret=__HIDDEN__
project_number=__HIDDEN__
object_acl=
bucket_acl=
token={"AccessToken":"__HIDDEN__","RefreshToken":"__HIDDEN__","Expiry":"2015-01-19T02:43:39.634659884-05:00","Extra":null}

I left the ACL fields empty, although previously had set them to projectPrivate


## list contents of iceburg-vault
#########################

[nesta@hartford ~]$ rclone --verbose=true ls google:iceburg-vault
2015/01/19 02:03:09 Storage bucket iceburg-vault: Modify window is 1ns
        0 manually-created-folder/

Transferred:            0 Bytes (   0.00 kByte/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:  5.395056198s

2015/01/19 02:03:14 *** Go routines at exit 9


## create a directory under iceburg-vault
#################################

[nesta@hartford ~]$ rclone --verbose=true mkdir google:iceburg-vault/test
2015/01/19 02:03:52 Storage bucket iceburg-vault path test/: Modify window is 1ns

Transferred:            0 Bytes (   0.00 kByte/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:  5.513743952s

2015/01/19 02:03:52 *** Go routines at exit 9


## list contents of iceburg-vault
#########################
# NOTE: it doesn't list "test", only the previously manually-created-folder...

[nesta@hartford ~]$ rclone --verbose=true ls google:iceburg-vault
2015/01/19 02:04:12 Storage bucket iceburg-vault: Modify window is 1ns
        0 manually-created-folder/

Transferred:            0 Bytes (   0.00 kByte/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:  5.394211563s

2015/01/19 02:04:17 *** Go routines at exit 9

crash when copy too large file to google drive

throw memory out error. file larger then 1GB .

config doesn't exit properly

From Kim:

The only bug I found, not critical, is that there is no graceful quit in rclone config. When finished configuring the google drive profile, it prompts for the name (of the remote) instead of e/n/d/q> so I can quit.

Swift keys with slashes do not work with Ceph in swift emulation mode

I had to regenerate my swift keys. If the key has a / or \ then rclone refuses to work and gives the following error:

./rclone lsd betaimages:/
2015/04/08 15:53:13 Failed to create file system for "betaimages:/": Operation forbidden

using the same config and using a swift key without \ and / works great

Request: Implement rsync-like exclude/include

I know you are focused on working out kinks with Drive and S3, but when you're ready for new features:

It would be terrific to have the following abilities work exactly as they do in rsync. Here is a prioritized list:

--max-size and --min-size. Why? - with backing up to cloud storage, it is important not to waste time accidentally backing up a 2GB movie file or something when you're backing up someone's Documents folder...
--exclude-from. I hope you will stick as closely as you can to Rsync's exact behavior here. The AWS CLI team is going off and reinventing the wheel with their exclude/include function and it's really frustrating/unecessary, when the rsync community has spent years coming up with an elegant and totally functional solution. The best part about rsync's is you can list the excludes in a file, so the actual command remains elegant and easy to read.

Thanks!

feature request: an option to delete duplicates during sync

With the new "duplicate file found" message, can we have a command line option to delete these duplicates as they're found?

Apologies if I've just missed this in the documentation—I didn't see anything there.

Make a travis build to check compilation under supported go versions

support for amazon cloud drive?

https://developer.amazon.com/public/apis/experience/cloud-drive/

Need synchronization in list output

Multiple go routines outputting to stdout causes corrupted output.

Should synchronize by outputting the data into a channel and printing in one goroutine.

Drive: Rclone doesn't sync Google Docs

Attached is a screen capture of my G Drive showing all the folders and some of the files.

Below is what:

rclone lsd GD: | sort -k 5

shows:

      -1 2014-06-28 09:18:09        -1 CNASALERC
      -1 2013-04-28 17:12:33        -1 Drive Sync Images
      -1 2013-03-20 08:38:03        -1 HandWrittenNotes
      -1 2014-07-03 08:23:16        -1 Health
      -1 2015-04-09 14:01:41        -1 Kathalina
      -1 2015-03-03 00:35:17        -1 LESTp
      -1 2014-08-20 17:11:08        -1 LEV
      -1 2013-03-27 07:53:36        -1 My Tracks
      -1 2014-07-13 07:36:47        -1 PhD_UNSW
      -1 2015-02-21 02:00:25        -1 SMSs
      -1 2014-06-05 08:02:51        -1 Writing
      -1 2015-04-22 11:31:58        -1 video

ie ALL the folders. Below is what:

rclone copy GD:/ ./

gives:

drwxrws--- 3 phil phil 4096 Apr 24 14:05 Drive Sync Images
drwxrws--- 2 phil phil 4096 Apr 24 14:10 HandWrittenNotes
drwxrws--- 2 phil phil 4096 Apr 24 14:02 Kathalina
drwxrws--- 2 phil phil 4096 Apr 24 14:05 LEV
drwxrws--- 2 phil phil 4096 Apr 24 14:06 My Tracks
drwxrws--- 2 phil phil 4096 Apr 24 14:05 PhD_UNSW
drwxrws--- 2 phil phil 4096 Apr 24 14:05 SMSs
drwxrws--- 6 phil phil 4096 Apr 24 14:06 Writing
-rw-rw-r-- 1 phil phil 151613 Feb 15 2014 AguadoSec02No10_gnote2_3_24pt.pdf
-rw-rw-r-- 1 phil phil 44509 Feb 15 2014 JulioSalvadorSagreras_1_p07.pdf
-rw-rw-r-- 1 phil phil 123555 Feb 12 2014 Julio_Salvador_Sagreras_Las_Primeras_Lecciones_de_guitarra.xml
-rw-rw-r-- 1 phil phil 139 Feb 13 2014 Julio_Salvador_Sagreras_Las_Primeras_Lecciones_de_guitarra_p10_w7.mid
-rw-rw-r-- 1 phil phil 652 Feb 12 2014 Julio_Salvador_Sagreras_Las_Primeras_Lecciones_de_guitarra_p2.mid
-rw-rw-r-- 1 phil phil 2319437 Feb 20 2012 Report20209_1.png
-rw-rw-r-- 1 phil phil 1688514 Feb 20 2012 Report20209_2.png
-rw-rw-r-- 1 phil phil 933 Mar 7 14:18 SajeCrt.txt
-rw-rw-r-- 1 phil phil 36164 Apr 24 17:45 Screenshot-042415-17:43:36.png
-rw-rw-r-- 1 phil phil 602 Feb 16 2014 ssh-dss.txt
-rw-rw-r-- 1 phil phil 240 Feb 3 15:19 t.csv

ie missing folders and files.

So between the lsd and the copy commands, something goes wrong . . ideas?

Thanks,

Phil.

Add region parameter to swift backend

From Roberto via email

How could it be possible to select the zone to storage
the info. By default it selects France (SBG-1) but we would like to
storage to America (BHS-1) region.

Region parameter is needed.

Dropbox support

Support dropbox as a sync source and destination

Compilation error with go 1.1.2 and gccgo (gcc 4.8, 4.9)

config.go:150:16: error: reference to undefined identifier 'strings.IndexByte'

Is it possible to maintain compatibility with older versions?

dry run not dry enough

I may be overlooking something but I ran rclone --dry-run sync drive:Foo local:. and it actually syncs and removes local files from disk that are not on drive. If I remove the dry run option, the only visible change is some additional logging

$ rclone sync drive:Foo local:.
2014/06/26 08:44:30 Local file system at .: Building file list
2014/06/26 08:44:30 Local file system at .: Waiting for checks to finish
2014/06/26 08:44:31 Local file system at .: Waiting for transfers to finish
2014/06/26 08:44:31 Waiting for deletions to finish
...

Use new location of Google API package

From Brad Fitzpatrick:

If you previously used the auto-generated Google APIs for Go, like:

import "code.google.com/p/google-api-go-client/storage/v1"

Now you should do:

import "google.golang.org/api/storage/v1"

And we've moved to git+github+gerrit like Go itself. New home is:

https://github.com/google/google-api-go-client

Ability to use rclone separately

Hello!

I have very challenging backup task and you will help me with rclone because it's looks like what I want.

I have about 5Tb of data in ~200 different binary files. This files has 1MB blocks and represents VPS filesystems which changed every minute.

I need ability to check what blocks was changed since previous full backups and send to backup server only changed pieces.

I'm tried to use rdiff signature and rdiff delta but without any success because it's really slow.

Could I use rclone in my task?

Support for syncing to GCS

Does this or will this tool support syncing to and from Google Cloud Storage? I see it works with Drive but wasn't sure if it also supported GCS?

My other issue, which no other tool has really addressed is that I am syncing millions of files and none of them were designed for that. I haven't looked at the code but can you comment on how will this scales?

This looks like a sweet tool, I am excited to check it out.

Need tenant support for swift backend

Some openstack deployments need the tenant parameter and rclone doesn't support it.

drive: authError when uploading big files

Factored from #28

Eg from @benfry

(50GB file)
2015/02/27 17:17:22 
Transferred: 51805750272 Bytes (12582.34 kByte/s)
Errors: 0
Checks: 0
Transferred: 0
Elapsed time: 1h7m0.837699975s
Transferring: Blade2_C_Drive006.v2i

2015/02/27 17:17:54 Blade2_C_Drive006.v2i: Failed to copy: Upload failed: googleapi: Error 401: Invalid Credentials, authError

Transferred: 52428762515 Bytes (12631.21 kByte/s)
Errors: 1
Checks: 0
Transferred: 1
Elapsed time: 1h7m33.449969925s

This can be reproduced very easily using the --bwlimit flag to make transfer take an arbitrarily long file.

The auth errors look very much like these problems

I have tried

refreshing the auth every 5 minutes during long transfers
- doesn't work - as it doesn't refresh the auth that is in progress
using the Resumable Media API in the google drive API
- doesn't work as it isn't finished according to the author

That leaves using the chunked API

https://developers.google.com/drive/web/manage-uploads#resumable

NB might need to do this for google cloud storage too which probably has the same underlying problem

drive: crash on failed to update remote mtime

From Kim via email

2015/03/01 13:58:23 Google drive root 'xxx': Error recived: Put https://www.googleapis.com/drive/v2/files/xxx?alt=json&setModifiedDate=true: net/http: TLS handshake timeout
2015/03/01 13:58:23 file_name.txt: Failed to update remote mtime: Put https://www.googleapis.com/drive/v2/files/xxx?alt=json&setModifiedDate=true: net/http: TLS handshake timeout
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x1 addr=0x0 pc=0x493ccd]

goroutine 1386 [running]:
github.com/ncw/rclone/drive.(*FsObjectDrive).setMetaData(0x12a0c6c0, 0x0)
    /home/ncw/Code/Go/src/github.com/ncw/rclone/drive/drive.go:836 +0x12d
github.com/ncw/rclone/drive.(*FsObjectDrive).SetModTime(0x12a0c6c0, 0xcb91ae50, 0xe, 0x2315b468, 0x990980)
    /home/ncw/Code/Go/src/github.com/ncw/rclone/drive/drive.go:914 +0x322
github.com/ncw/rclone/fs.Equal(0x32dd00c8, 0x130881b0, 0x137a80, 0x12a0c6c0, 0x1344ff7c)
    /home/ncw/Code/Go/src/github.com/ncw/rclone/fs/operations.go:94 +0x531
github.com/ncw/rclone/fs.checkOne(0x32dd00c8, 0x130881b0, 0x137a80, 0x12a0c6c0, 0x1288a2c0)
    /home/ncw/Code/Go/src/github.com/ncw/rclone/fs/operations.go:199 +0xfb
github.com/ncw/rclone/fs.PairChecker(0x1288a180, 0x1288a2c0, 0x1315c420)
    /home/ncw/Code/Go/src/github.com/ncw/rclone/fs/operations.go:214 +0xc3
created by github.com/ncw/rclone/fs.Sync
    /home/ncw/Code/Go/src/github.com/ncw/rclone/fs/operations.go:288 +0x2ce

s3: ls doesn't read all the files

I did a check between directories in Google Drive and S3 that I'm fairly sure are in sync. However it came up that 7K files were missing. I then did a ls on the S3 bucket and it didn't list all of the files. I know that S3's ls function only returns 1000 objects at a time. Could that be why the ls and the check failed?

OSX-Current not implemented on darwin/amd64

OSX AMD 64 Yesomite When initiated with the command shown below:
"Kates-iMac:rclone-v1.05 Paul$ ~/rclone-v1.05/rclone copy ~/Pictures/Archive GDrive:TestSync"
it initially returns
"2014/12/11 11:09:52 Couldn't find home directory: user: Current not implemented on darwin/amd64
However this then proceeds to work correctly and copy/sync to Google Drive - not sure what "Current" is - does it mean the full path is required which is why it is working?

runtime error: invalid memory address or nil pointer dereference (Google Cloud Storage)

This error is always happening to me.
I'm using google cloud storage.

uname:

# uname -a
Linux 1k6 3.5.0-23-generic #35~precise1-Ubuntu SMP Fri Jan 25 17:13:26 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

error output:

runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x497bf4]

goroutine 55 [running]:
runtime.panic(0x82fd00, 0xada3d3)
    /home/ncw/Code/go/src/pkg/runtime/panic.c:279 +0xf5
github.com/ncw/rclone/googlecloudstorage.(*FsObjectStorage).setMetaData(0xc2083088a0, 0x0)
    /home/ncw/Code/Go/src/github.com/ncw/rclone/googlecloudstorage/googlecloudstorage.go:423 +0x534
github.com/ncw/rclone/googlecloudstorage.(*FsObjectStorage).Update(0xc2083088a0, 0x7fd70b0c5060, 0xc20821b220, 0xec9b728b6, 0x0, 0xae0c80, 0x475, 0x0, 0x0)
    /home/ncw/Code/Go/src/github.com/ncw/rclone/googlecloudstorage/googlecloudstorage.go:559 +0x523
github.com/ncw/rclone/googlecloudstorage.(*FsStorage).Put(0xc208004660, 0x7fd70b0c5060, 0xc20821b220, 0xc20824f91e, 0x27, 0xec9b728b6, 0x0, 0xae0c80, 0x475, 0x0, ...)
    /home/ncw/Code/Go/src/github.com/ncw/rclone/googlecloudstorage/googlecloudstorage.go:356 +0xea
github.com/ncw/rclone/fs.Copy(0x7fd70b0a4bf8, 0xc208004660, 0x0, 0x0, 0x7fd70b0bc008, 0xc20824fbd0)
    /home/ncw/Code/Go/src/github.com/ncw/rclone/fs/operations.go:132 +0xe27
github.com/ncw/rclone/fs.Copier(0xc20818c7e0, 0x7fd70b0a4bf8, 0xc208004660, 0xc208725940)
    /home/ncw/Code/Go/src/github.com/ncw/rclone/fs/operations.go:218 +0x19e
created by github.com/ncw/rclone/fs.Sync
    /home/ncw/Code/Go/src/github.com/ncw/rclone/fs/operations.go:287 +0x41a

goroutine 16 [semacquire, 4 minutes]:
sync.runtime_Semacquire(0xc208189a54)
    /home/ncw/Code/go/src/pkg/runtime/sema.goc:199 +0x30
sync.(*WaitGroup).Wait(0xc2087258e0)
    /home/ncw/Code/go/src/pkg/sync/waitgroup.go:129 +0x14b
github.com/ncw/rclone/fs.Sync(0x7fd70b0a4bf8, 0xc208004660, 0x7fd70b0a4a58, 0xc208025260, 0x7fd70aef9f00, 0x0, 0x0)
    /home/ncw/Code/Go/src/github.com/ncw/rclone/fs/operations.go:306 +0x4d9
main.func·001(0x7fd70b0a4bf8, 0xc208004660, 0x7fd70b0a4a58, 0xc208025260)
    /home/ncw/Code/Go/src/github.com/ncw/rclone/rclone.go:67 +0x56
main.main()
    /home/ncw/Code/Go/src/github.com/ncw/rclone/rclone.go:357 +0x2d2

goroutine 19 [finalizer wait, 4 minutes]:
runtime.park(0x415990, 0xadf298, 0xadd509)
    /home/ncw/Code/go/src/pkg/runtime/proc.c:1369 +0x89
runtime.parkunlock(0xadf298, 0xadd509)
    /home/ncw/Code/go/src/pkg/runtime/proc.c:1385 +0x3b
runfinq()
    /home/ncw/Code/go/src/pkg/runtime/mgc0.c:2644 +0xcf
runtime.goexit()
    /home/ncw/Code/go/src/pkg/runtime/proc.c:1445

goroutine 17 [syscall, 4 minutes]:
runtime.goexit()
    /home/ncw/Code/go/src/pkg/runtime/proc.c:1445

goroutine 20 [chan receive]:
main.func·012()
    /home/ncw/Code/Go/src/github.com/ncw/rclone/rclone.go:325 +0x57
created by main.StartStats
    /home/ncw/Code/Go/src/github.com/ncw/rclone/rclone.go:328 +0x22

goroutine 27 [chan receive, 3 minutes]:
github.com/ncw/rclone/fs.PairChecker(0xc20818c620, 0xc20818c7e0, 0xc2087258e0)
    /home/ncw/Code/Go/src/github.com/ncw/rclone/fs/operations.go:201 +0x7a
created by github.com/ncw/rclone/fs.Sync
    /home/ncw/Code/Go/src/github.com/ncw/rclone/fs/operations.go:281 +0x381

goroutine 28 [chan receive, 3 minutes]:
github.com/ncw/rclone/fs.PairChecker(0xc20818c620, 0xc20818c7e0, 0xc2087258e0)
    /home/ncw/Code/Go/src/github.com/ncw/rclone/fs/operations.go:201 +0x7a
created by github.com/ncw/rclone/fs.Sync
    /home/ncw/Code/Go/src/github.com/ncw/rclone/fs/operations.go:281 +0x381

goroutine 29 [chan receive, 3 minutes]:
github.com/ncw/rclone/fs.PairChecker(0xc20818c620, 0xc20818c7e0, 0xc2087258e0)
    /home/ncw/Code/Go/src/github.com/ncw/rclone/fs/operations.go:201 +0x7a
created by github.com/ncw/rclone/fs.Sync
    /home/ncw/Code/Go/src/github.com/ncw/rclone/fs/operations.go:281 +0x381

goroutine 30 [chan receive, 3 minutes]:
github.com/ncw/rclone/fs.PairChecker(0xc20818c620, 0xc20818c7e0, 0xc2087258e0)
    /home/ncw/Code/Go/src/github.com/ncw/rclone/fs/operations.go:201 +0x7a
created by github.com/ncw/rclone/fs.Sync
    /home/ncw/Code/Go/src/github.com/ncw/rclone/fs/operations.go:281 +0x381

goroutine 31 [chan receive, 3 minutes]:
github.com/ncw/rclone/fs.PairChecker(0xc20818c620, 0xc20818c7e0, 0xc2087258e0)
    /home/ncw/Code/Go/src/github.com/ncw/rclone/fs/operations.go:201 +0x7a
created by github.com/ncw/rclone/fs.Sync
    /home/ncw/Code/Go/src/github.com/ncw/rclone/fs/operations.go:281 +0x381

goroutine 49 [chan receive, 3 minutes]:
github.com/ncw/rclone/fs.PairChecker(0xc20818c620, 0xc20818c7e0, 0xc2087258e0)
    /home/ncw/Code/Go/src/github.com/ncw/rclone/fs/operations.go:201 +0x7a
created by github.com/ncw/rclone/fs.Sync
    /home/ncw/Code/Go/src/github.com/ncw/rclone/fs/operations.go:281 +0x381

goroutine 50 [chan receive, 3 minutes]:
github.com/ncw/rclone/fs.PairChecker(0xc20818c620, 0xc20818c7e0, 0xc2087258e0)
    /home/ncw/Code/Go/src/github.com/ncw/rclone/fs/operations.go:201 +0x7a
created by github.com/ncw/rclone/fs.Sync
    /home/ncw/Code/Go/src/github.com/ncw/rclone/fs/operations.go:281 +0x381

goroutine 51 [chan receive, 3 minutes]:
github.com/ncw/rclone/fs.PairChecker(0xc20818c620, 0xc20818c7e0, 0xc2087258e0)
    /home/ncw/Code/Go/src/github.com/ncw/rclone/fs/operations.go:201 +0x7a
created by github.com/ncw/rclone/fs.Sync
    /home/ncw/Code/Go/src/github.com/ncw/rclone/fs/operations.go:281 +0x381

goroutine 52 [select]:
net/http.(*persistConn).roundTrip(0xc2084be2c0, 0xc20815c4f0, 0x0, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/net/http/transport.go:1015 +0x6db
net/http.(*Transport).RoundTrip(0xc208050100, 0xc2081949c0, 0xd, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/net/http/transport.go:208 +0x49a
code.google.com/p/goauth2/oauth.(*Transport).RoundTrip(0xc20808e6e0, 0xc2081949c0, 0x17b, 0x0, 0x0)
    /home/ncw/Code/Go/src/code.google.com/p/goauth2/oauth/oauth.go:278 +0x318
net/http.send(0xc2081948f0, 0x7fd70b0a4bd0, 0xc20808e6e0, 0xc2083a6480, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/net/http/client.go:195 +0x43d
net/http.(*Client).send(0xc208025740, 0xc2081948f0, 0xba, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/net/http/client.go:118 +0x15b
net/http.(*Client).doFollowingRedirects(0xc208025740, 0xc2081948f0, 0x96de98, 0x0, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/net/http/client.go:343 +0x97f
net/http.(*Client).Do(0xc208025740, 0xc2081948f0, 0xa, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/net/http/client.go:153 +0x193
code.google.com/p/google-api-go-client/storage/v1.(*ObjectsInsertCall).Do(0xc208541c00, 0x0, 0x0, 0x0)
    /home/ncw/Code/Go/src/code.google.com/p/google-api-go-client/storage/v1/storage-gen.go:4042 +0x15a8
github.com/ncw/rclone/googlecloudstorage.(*FsObjectStorage).Update(0xc20846a840, 0x7fd70b0c5060, 0xc2081ef440, 0xeca7bb0a0, 0x12351e00, 0xae0c80, 0x8650, 0x0, 0x0)
    /home/ncw/Code/Go/src/github.com/ncw/rclone/googlecloudstorage/googlecloudstorage.go:557 +0x4ee
github.com/ncw/rclone/googlecloudstorage.(*FsStorage).Put(0xc208004660, 0x7fd70b0c5060, 0xc2081ef440, 0xc20852d41e, 0x27, 0xeca7bb0a0, 0x12351e00, 0xae0c80, 0x8650, 0x0, ...)
    /home/ncw/Code/Go/src/github.com/ncw/rclone/googlecloudstorage/googlecloudstorage.go:356 +0xea
github.com/ncw/rclone/fs.Copy(0x7fd70b0a4bf8, 0xc208004660, 0x0, 0x0, 0x7fd70b0bc008, 0xc20852c3c0)
    /home/ncw/Code/Go/src/github.com/ncw/rclone/fs/operations.go:132 +0xe27
github.com/ncw/rclone/fs.Copier(0xc20818c7e0, 0x7fd70b0a4bf8, 0xc208004660, 0xc208725940)
    /home/ncw/Code/Go/src/github.com/ncw/rclone/fs/operations.go:218 +0x19e
created by github.com/ncw/rclone/fs.Sync
    /home/ncw/Code/Go/src/github.com/ncw/rclone/fs/operations.go:287 +0x41a

goroutine 53 [select]:
net/http.(*persistConn).roundTrip(0xc2084bf550, 0xc20819c9e0, 0x0, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/net/http/transport.go:1015 +0x6db
net/http.(*Transport).RoundTrip(0xc208050100, 0xc2082772b0, 0xd, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/net/http/transport.go:208 +0x49a
code.google.com/p/goauth2/oauth.(*Transport).RoundTrip(0xc20808e6e0, 0xc2082772b0, 0x17b, 0x0, 0x0)
    /home/ncw/Code/Go/src/code.google.com/p/goauth2/oauth/oauth.go:278 +0x318
net/http.send(0xc2082771e0, 0x7fd70b0a4bd0, 0xc20808e6e0, 0xc208010a80, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/net/http/client.go:195 +0x43d
net/http.(*Client).send(0xc208025740, 0xc2082771e0, 0xba, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/net/http/client.go:118 +0x15b
net/http.(*Client).doFollowingRedirects(0xc208025740, 0xc2082771e0, 0x96de98, 0x0, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/net/http/client.go:343 +0x97f
net/http.(*Client).Do(0xc208025740, 0xc2082771e0, 0xa, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/net/http/client.go:153 +0x193
code.google.com/p/google-api-go-client/storage/v1.(*ObjectsInsertCall).Do(0xc2083ca140, 0x0, 0x0, 0x0)
    /home/ncw/Code/Go/src/code.google.com/p/google-api-go-client/storage/v1/storage-gen.go:4042 +0x15a8
github.com/ncw/rclone/googlecloudstorage.(*FsObjectStorage).Update(0xc2086f6ae0, 0x7fd70b0c5060, 0xc2081ca040, 0xec9e7eb16, 0x0, 0xae0c80, 0x580a, 0x0, 0x0)
    /home/ncw/Code/Go/src/github.com/ncw/rclone/googlecloudstorage/googlecloudstorage.go:557 +0x4ee
github.com/ncw/rclone/googlecloudstorage.(*FsStorage).Put(0xc208004660, 0x7fd70b0c5060, 0xc2081ca040, 0xc2084bd7de, 0x27, 0xec9e7eb16, 0x0, 0xae0c80, 0x580a, 0x0, ...)
    /home/ncw/Code/Go/src/github.com/ncw/rclone/googlecloudstorage/googlecloudstorage.go:356 +0xea
github.com/ncw/rclone/fs.Copy(0x7fd70b0a4bf8, 0xc208004660, 0x0, 0x0, 0x7fd70b0bc008, 0xc2084bd9a0)
    /home/ncw/Code/Go/src/github.com/ncw/rclone/fs/operations.go:132 +0xe27
github.com/ncw/rclone/fs.Copier(0xc20818c7e0, 0x7fd70b0a4bf8, 0xc208004660, 0xc208725940)
    /home/ncw/Code/Go/src/github.com/ncw/rclone/fs/operations.go:218 +0x19e
created by github.com/ncw/rclone/fs.Sync
    /home/ncw/Code/Go/src/github.com/ncw/rclone/fs/operations.go:287 +0x41a

goroutine 54 [select]:
net/http.(*persistConn).roundTrip(0xc208328840, 0xc20819d670, 0x0, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/net/http/transport.go:1015 +0x6db
net/http.(*Transport).RoundTrip(0xc208050100, 0xc208277520, 0xd, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/net/http/transport.go:208 +0x49a
code.google.com/p/goauth2/oauth.(*Transport).RoundTrip(0xc20808e6e0, 0xc208277520, 0x17b, 0x0, 0x0)
    /home/ncw/Code/Go/src/code.google.com/p/goauth2/oauth/oauth.go:278 +0x318
net/http.send(0xc208277450, 0x7fd70b0a4bd0, 0xc20808e6e0, 0xc208010c00, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/net/http/client.go:195 +0x43d
net/http.(*Client).send(0xc208025740, 0xc208277450, 0xba, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/net/http/client.go:118 +0x15b
net/http.(*Client).doFollowingRedirects(0xc208025740, 0xc208277450, 0x96de98, 0x0, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/net/http/client.go:343 +0x97f
net/http.(*Client).Do(0xc208025740, 0xc208277450, 0xa, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/net/http/client.go:153 +0x193
code.google.com/p/google-api-go-client/storage/v1.(*ObjectsInsertCall).Do(0xc2083ca900, 0x0, 0x0, 0x0)
    /home/ncw/Code/Go/src/code.google.com/p/google-api-go-client/storage/v1/storage-gen.go:4042 +0x15a8
github.com/ncw/rclone/googlecloudstorage.(*FsObjectStorage).Update(0xc2086f7aa0, 0x7fd70b0c5060, 0xc2081cb0e0, 0xecaca0280, 0x33caf7db, 0xae0c80, 0x3c2f, 0x0, 0x0)
    /home/ncw/Code/Go/src/github.com/ncw/rclone/googlecloudstorage/googlecloudstorage.go:557 +0x4ee
github.com/ncw/rclone/googlecloudstorage.(*FsStorage).Put(0xc208004660, 0x7fd70b0c5060, 0xc2081cb0e0, 0xc2085ca10e, 0x27, 0xecaca0280, 0x33caf7db, 0xae0c80, 0x3c2f, 0x0, ...)
    /home/ncw/Code/Go/src/github.com/ncw/rclone/googlecloudstorage/googlecloudstorage.go:356 +0xea
github.com/ncw/rclone/fs.Copy(0x7fd70b0a4bf8, 0xc208004660, 0x0, 0x0, 0x7fd70b0bc008, 0xc2085ca280)
    /home/ncw/Code/Go/src/github.com/ncw/rclone/fs/operations.go:132 +0xe27
github.com/ncw/rclone/fs.Copier(0xc20818c7e0, 0x7fd70b0a4bf8, 0xc208004660, 0xc208725940)
    /home/ncw/Code/Go/src/github.com/ncw/rclone/fs/operations.go:218 +0x19e
created by github.com/ncw/rclone/fs.Sync
    /home/ncw/Code/Go/src/github.com/ncw/rclone/fs/operations.go:287 +0x41a

goroutine 56 [chan send]:
github.com/ncw/rclone/fs.func·003()
    /home/ncw/Code/Go/src/github.com/ncw/rclone/fs/operations.go:299 +0x284
created by github.com/ncw/rclone/fs.Sync
    /home/ncw/Code/Go/src/github.com/ncw/rclone/fs/operations.go:303 +0x46c

goroutine 34 [chan send]:
github.com/ncw/rclone/local.func·001(0xc20824fd10, 0x45, 0x7fd70b0a4a08, 0xc20824fdb0, 0x0, 0x0, 0x0, 0x0)
    /home/ncw/Code/Go/src/github.com/ncw/rclone/local/local.go:117 +0x452
path/filepath.walk(0xc20824fd10, 0x45, 0x7fd70b0a4a08, 0xc20824fdb0, 0x7fd7082ecf68, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/path/filepath/path.go:343 +0x8c
path/filepath.walk(0xc208812c90, 0x2b, 0x7fd70b0a4a08, 0xc20896a8c0, 0x7fd7082ecf68, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/path/filepath/path.go:368 +0x4f5
path/filepath.walk(0xc2086d58f0, 0x29, 0x7fd70b0a4a08, 0xc2084bd630, 0x7fd7082ecf68, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/path/filepath/path.go:368 +0x4f5
path/filepath.walk(0xc208364870, 0x27, 0x7fd70b0a4a08, 0xc20824e2d0, 0x7fd7082ecf68, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/path/filepath/path.go:368 +0x4f5
path/filepath.walk(0xc208812540, 0x25, 0x7fd70b0a4a08, 0xc2084bc140, 0x7fd7082ecf68, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/path/filepath/path.go:368 +0x4f5
path/filepath.walk(0xc2086d4480, 0x23, 0x7fd70b0a4a08, 0xc2081b2320, 0x7fd7082ecf68, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/path/filepath/path.go:368 +0x4f5
path/filepath.walk(0x7fff4595184d, 0x1d, 0x7fd70b0a4a08, 0xc2081b20f0, 0x7fd7082ecf68, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/path/filepath/path.go:368 +0x4f5
path/filepath.Walk(0x7fff4595184d, 0x1d, 0x7fd7082ecf68, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/path/filepath/path.go:390 +0xe5
github.com/ncw/rclone/local.func·002()
    /home/ncw/Code/Go/src/github.com/ncw/rclone/local/local.go:122 +0x70
created by github.com/ncw/rclone/local.(*FsLocal).List
    /home/ncw/Code/Go/src/github.com/ncw/rclone/local/local.go:128 +0xaf

goroutine 101 [select]:
net/http.(*persistConn).writeLoop(0xc208328840)
    /home/ncw/Code/go/src/pkg/net/http/transport.go:885 +0x38f
created by net/http.(*Transport).dialConn
    /home/ncw/Code/go/src/pkg/net/http/transport.go:601 +0x957

goroutine 67 [IO wait]:
net.runtime_pollWait(0x7fd70b0a5a20, 0x72, 0x0)
    /home/ncw/Code/go/src/pkg/runtime/netpoll.goc:146 +0x66
net.(*pollDesc).Wait(0xc208513870, 0x72, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/net/fd_poll_runtime.go:84 +0x46
net.(*pollDesc).WaitRead(0xc208513870, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/net/fd_poll_runtime.go:89 +0x42
net.(*netFD).Read(0xc208513810, 0xc2081c5800, 0x800, 0x800, 0x0, 0x7fd70b098440, 0xb)
    /home/ncw/Code/go/src/pkg/net/fd_unix.go:232 +0x34c
net.(*conn).Read(0xc2082a0090, 0xc2081c5800, 0x800, 0x800, 0x0, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/net/net.go:122 +0xe7
crypto/tls.(*block).readFromUntil(0xc2085e7650, 0x7fd70b0a5de0, 0xc2082a0090, 0x5, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/crypto/tls/conn.go:451 +0xd9
crypto/tls.(*Conn).readRecord(0xc20810c2c0, 0x17, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/crypto/tls/conn.go:536 +0x1ff
crypto/tls.(*Conn).Read(0xc20810c2c0, 0xc2084a3000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/crypto/tls/conn.go:901 +0x16a
net/http.noteEOFReader.Read(0x7fd70b0aa790, 0xc20810c2c0, 0xc2084bf5a8, 0xc2084a3000, 0x1000, 0x1000, 0xaf1680, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/net/http/transport.go:1203 +0x72
net/http.(*noteEOFReader).Read(0xc20825df00, 0xc2084a3000, 0x1000, 0x1000, 0xc20804eb00, 0x0, 0x0)
    <autogenerated>:124 +0xca
bufio.(*Reader).fill(0xc2086f6720)
    /home/ncw/Code/go/src/pkg/bufio/bufio.go:97 +0x1b3
bufio.(*Reader).Peek(0xc2086f6720, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/bufio/bufio.go:132 +0x101
net/http.(*persistConn).readLoop(0xc2084bf550)
    /home/ncw/Code/go/src/pkg/net/http/transport.go:782 +0x95
created by net/http.(*Transport).dialConn
    /home/ncw/Code/go/src/pkg/net/http/transport.go:600 +0x93f

goroutine 115 [IO wait]:
net.runtime_pollWait(0x7fd70b0a5b80, 0x72, 0x0)
    /home/ncw/Code/go/src/pkg/runtime/netpoll.goc:146 +0x66
net.(*pollDesc).Wait(0xc2080cb020, 0x72, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/net/fd_poll_runtime.go:84 +0x46
net.(*pollDesc).WaitRead(0xc2080cb020, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/net/fd_poll_runtime.go:89 +0x42
net.(*netFD).Read(0xc2080cafc0, 0xc20839b000, 0x1000, 0x1000, 0x0, 0x7fd70b098440, 0xb)
    /home/ncw/Code/go/src/pkg/net/fd_unix.go:232 +0x34c
net.(*conn).Read(0xc2082a0408, 0xc20839b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/net/net.go:122 +0xe7
crypto/tls.(*block).readFromUntil(0xc208292420, 0x7fd70b0a5de0, 0xc2082a0408, 0x5, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/crypto/tls/conn.go:451 +0xd9
crypto/tls.(*Conn).readRecord(0xc20850f080, 0x17, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/crypto/tls/conn.go:536 +0x1ff
crypto/tls.(*Conn).Read(0xc20850f080, 0xc2082dc000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/crypto/tls/conn.go:901 +0x16a
net/http.noteEOFReader.Read(0x7fd70b0aa790, 0xc20850f080, 0xc2084be318, 0xc2082dc000, 0x1000, 0x1000, 0xaf1680, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/net/http/transport.go:1203 +0x72
net/http.(*noteEOFReader).Read(0xc20823a160, 0xc2082dc000, 0x1000, 0x1000, 0xc20804eb08, 0x0, 0x0)
    <autogenerated>:124 +0xca
bufio.(*Reader).fill(0xc208308180)
    /home/ncw/Code/go/src/pkg/bufio/bufio.go:97 +0x1b3
bufio.(*Reader).Peek(0xc208308180, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/bufio/bufio.go:132 +0x101
net/http.(*persistConn).readLoop(0xc2084be2c0)
    /home/ncw/Code/go/src/pkg/net/http/transport.go:782 +0x95
created by net/http.(*Transport).dialConn
    /home/ncw/Code/go/src/pkg/net/http/transport.go:600 +0x93f

goroutine 100 [IO wait]:
net.runtime_pollWait(0x7fd70b0a5c30, 0x72, 0x0)
    /home/ncw/Code/go/src/pkg/runtime/netpoll.goc:146 +0x66
net.(*pollDesc).Wait(0xc2084ce6f0, 0x72, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/net/fd_poll_runtime.go:84 +0x46
net.(*pollDesc).WaitRead(0xc2084ce6f0, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/net/fd_poll_runtime.go:89 +0x42
net.(*netFD).Read(0xc2084ce690, 0xc20820c000, 0x1000, 0x1000, 0x0, 0x7fd70b098440, 0xb)
    /home/ncw/Code/go/src/pkg/net/fd_unix.go:232 +0x34c
net.(*conn).Read(0xc2085cc058, 0xc20820c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/net/net.go:122 +0xe7
crypto/tls.(*block).readFromUntil(0xc208292210, 0x7fd70b0a5de0, 0xc2085cc058, 0x5, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/crypto/tls/conn.go:451 +0xd9
crypto/tls.(*Conn).readRecord(0xc2083c8000, 0x17, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/crypto/tls/conn.go:536 +0x1ff
crypto/tls.(*Conn).Read(0xc2083c8000, 0xc20810e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/crypto/tls/conn.go:901 +0x16a
net/http.noteEOFReader.Read(0x7fd70b0aa790, 0xc2083c8000, 0xc208328898, 0xc20810e000, 0x1000, 0x1000, 0xaf1680, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/net/http/transport.go:1203 +0x72
net/http.(*noteEOFReader).Read(0xc208220660, 0xc20810e000, 0x1000, 0x1000, 0xc20804eb18, 0x0, 0x0)
    <autogenerated>:124 +0xca
bufio.(*Reader).fill(0xc2086eea80)
    /home/ncw/Code/go/src/pkg/bufio/bufio.go:97 +0x1b3
bufio.(*Reader).Peek(0xc2086eea80, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/bufio/bufio.go:132 +0x101
net/http.(*persistConn).readLoop(0xc208328840)
    /home/ncw/Code/go/src/pkg/net/http/transport.go:782 +0x95
created by net/http.(*Transport).dialConn
    /home/ncw/Code/go/src/pkg/net/http/transport.go:600 +0x93f

goroutine 119 [IO wait]:
net.runtime_pollWait(0x7fd70b0a5ad0, 0x72, 0x0)
    /home/ncw/Code/go/src/pkg/runtime/netpoll.goc:146 +0x66
net.(*pollDesc).Wait(0xc2089561b0, 0x72, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/net/fd_poll_runtime.go:84 +0x46
net.(*pollDesc).WaitRead(0xc2089561b0, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/net/fd_poll_runtime.go:89 +0x42
net.(*netFD).Read(0xc208956150, 0xc208320000, 0x1000, 0x1000, 0x0, 0x7fd70b098440, 0xb)
    /home/ncw/Code/go/src/pkg/net/fd_unix.go:232 +0x34c
net.(*conn).Read(0xc20803c030, 0xc208320000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/net/net.go:122 +0xe7
crypto/tls.(*block).readFromUntil(0xc208364b70, 0x7fd70b0a5de0, 0xc20803c030, 0x5, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/crypto/tls/conn.go:451 +0xd9
crypto/tls.(*Conn).readRecord(0xc2080c9340, 0x17, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/crypto/tls/conn.go:536 +0x1ff
crypto/tls.(*Conn).Read(0xc2080c9340, 0xc20836c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/crypto/tls/conn.go:901 +0x16a
net/http.noteEOFReader.Read(0x7fd70b0aa790, 0xc2080c9340, 0xc20877e268, 0xc20836c000, 0x1000, 0x1000, 0xaf1680, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/net/http/transport.go:1203 +0x72
net/http.(*noteEOFReader).Read(0xc2081b6120, 0xc20836c000, 0x1000, 0x1000, 0xc20804eb10, 0x0, 0x0)
    <autogenerated>:124 +0xca
bufio.(*Reader).fill(0xc208309e60)
    /home/ncw/Code/go/src/pkg/bufio/bufio.go:97 +0x1b3
bufio.(*Reader).Peek(0xc208309e60, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/ncw/Code/go/src/pkg/bufio/bufio.go:132 +0x101
net/http.(*persistConn).readLoop(0xc20877e210)
    /home/ncw/Code/go/src/pkg/net/http/transport.go:782 +0x95
created by net/http.(*Transport).dialConn
    /home/ncw/Code/go/src/pkg/net/http/transport.go:600 +0x93f

goroutine 120 [select]:
net/http.(*persistConn).writeLoop(0xc20877e210)
    /home/ncw/Code/go/src/pkg/net/http/transport.go:885 +0x38f
created by net/http.(*Transport).dialConn
    /home/ncw/Code/go/src/pkg/net/http/transport.go:601 +0x957

goroutine 68 [select]:
net/http.(*persistConn).writeLoop(0xc2084bf550)
    /home/ncw/Code/go/src/pkg/net/http/transport.go:885 +0x38f
created by net/http.(*Transport).dialConn
    /home/ncw/Code/go/src/pkg/net/http/transport.go:601 +0x957

goroutine 116 [select]:
net/http.(*persistConn).writeLoop(0xc2084be2c0)
    /home/ncw/Code/go/src/pkg/net/http/transport.go:885 +0x38f
created by net/http.(*Transport).dialConn
    /home/ncw/Code/go/src/pkg/net/http/transport.go:601 +0x957

drive: Fix 403 rateLimitExceeded errors

This is a great product, thank you! Wish I could contribute actual code, but for now can only contribute testing results. It's great to be able to create simple scripts that interact with these different cloud storage services. I'm surprised this seems to be one of the few cross platform command line tools doing this.

I'm using it mostly with Google Drive for now but plan to use with S3 as well. With Google Drive, I'm experiencing the error I think you are aware of when it operates over several GB worth of data and a large directory tree - Drive seems to not like all of the requests or something and it throws an error.

Interestingly, when I map Google Drive using Expandrive, and then use rsync, I don't get this error. This may be because Expandrive slows down the requests due to it's own internal workings? There are other problems with mapping a drive and then using the command line so I'd much rather not have to involve the drive mapping and just use the command line.

If there is some way I can help please let me know.

copy traverses the entire destination file system

copy traverses the entire destination file system which works but is potentially inefficient if we are copying a single file to a large filesystem tree for instance.

Perhaps add a command line flag --no-traverse to override?

Could also read source files into a file list and use the directory traversing optimisations for that.

Sync command documentation and versioning

Sync command documentation is incorrect
The documentation states that the sync command "Sync the source to the destination" and that it "Deletes any files that exist in source that don't exist in destination".
It is actually the other way around: the source is the master, and the destination is changed to match it (deleting files, if necessary).
Tested on versions 1.10 (Linux) and 1.12 (both Linux and Windows), and the behaviour is the same on all of them

No information about versioning
The documentation does not mention that both the copy and the sync commands creates a new version of the files that have been updated on the source (at least on Google Drive) instead of deleting the old one and adding the updated version, like other similar tools.
This was one of the main reasons that made me use this tool, and I only found that out through testing. It might be a good idea to increase the awareness about this feature.

A file and a directory with shared path confuses rclone

It is quite possible to make objects like this (in s3 or swift)

test_file
test_file/1

However when rsync tries to copy them it is confused about the status of "test_file" whether it should be a file or a directory, and gives errors like this

2014/11/27 13:17:54 test_file/1: Failed to copy: mkdir /tmp/test/test_file: not a directory

Not sure how to fix this as it isn't possible to represent this in a filesystem. syncing a bucket/container like this to s3/swift does work fine though.

This can be caused by the s3fs program which creates a file object "dir/" for every directory - presumably to store permissions, eg

        0 2014-11-27 13:17:54.000000000 test/
       10 2014-11-27 13:12:28.000000000 test/test_file

Could just ignore zero length files with trailing / ?

Windows \ separator problems

From Kim via email:

Rclone upload works only to the first main subdirectory under Google drive.

This command works, rclone copy "C:\NetworkData\New folder" "remote:New folder"

But commands below with a \ separator creates a directory name with \ in it (eg. Newfolder\New folder) instead of copying to the 2nd subdirectory New folder.

rclone copy "C:\NetworkData\New folder" "remote:Newfolder\New folder"
rclone copy "C:\NetworkData\New folder" "remote:New folder\New folder"
rclone copy "C:\NetworkData\New folder" "remote:Newfolder\Newfolder"

HDFS support

Hi,

Any thoughts or plans around HDFS support? It could be very nice to have a way to, for example, bring a given data set out of Swift into Hadoop to run jobs on.

Make the difference betwen drive and google cloud storage clearer

Hi, I just installed rclone via AUR and followed the tutorial on rclone.org, but I seem to keep getting this error: 2015/03/25 22:29:59 Failed to copy: Can't make bucket without project number

Would anyone know what I'm doing wrong?

Here's my rc file:

[remote]
type = google cloud storage
client_id = 
client_secret = 
project_number = 
object_acl =
bucket_acl =
token = {"AccessToken":"XXXXXXXXXXXXXXXXXXX","RefreshToken":"XXXXXXXXXXXX","Expiry":"2015-03-25T23:20:45.958816454-04:00","Extra":null}

What's next in testing rclone for Drive?

Hi Nick,
It looks like rclone is doing better with Drive. I would like to move it into a production environment where I'd script uploads and downloads with Drive. Before doing so, I wanted to know if you had any reservations based on how it's working for you? Are there any tests you'd like help running?

1 question is whether, if it encounters errors, it will automatically re-run?

Thanks again for all of this great work.

Allow copying a single file?

I don't know what's involved, but it would be great to be able to copy or move a single file either from local to cloud, cloud to local, or between 2 cloud providers.

Yours is the only stable tool I know of that connects to each of these different cloud providers so it would be invaluable to be able to copy/move individual files between them from the command line!

Have Support for Symlinks in Rclone

copy or sync command does NOT work for symlink(soft links) folders when syncing to Google Drive. It would be nice if rclone can follow symlinks.

Ran Sync and then Check and it didn't Check

Hi - I ran a sync command and it completed with 0 errors, then I ran a check, and it showed that a PDF file had a different MD5 sum. I re-ran the sync and then check several times with the same results.

I then ran sync with -v and saw that it said, for this one file, that it was the same size and modification.

The docs say rclone checks by modtime and then MD5.

Might this be a bug, or am I misinterpreting the docs?

Thanks

Drive: "Shared with me" not visible

I got rclone working without too much trouble (Fedora 21 x86_64) - compared to gsync - it is very nice! - thanks!

. . but it looks like using "lsd" does not show the "Shared with me" folder - is that fixable?

More thanks in advance!

Regards,
Phil.

Swift: Cannot copy/sync file larger than 5GB

Swift maximum file size is 5GB and if you want to upload bigger files to this storage, it has to be segmented (more info: http://docs.openstack.org/developer/swift/overview_large_objects.html).

Unfortunately it looks like rclone does not support segmenting and if I try to upload 6GB file, it fails with:
large_file.img: Failed to copy: Too Large Object

Otherwise, great tool!!!

copy to googledrive

2014/07/02 11:18:12 blog/wp-content/plugins/google-sitemap-generator/sitemap-wpmu.php: Failed to remove failed copy: googleapi: got HTTP response code 404 and error reading body: invalid character 'N' looking for beginning of value
2014/07/02 11:18:12 blog/wp-content/plugins/google-sitemap-generator/sitemap.xsl: Failed to remove failed copy: googleapi: got HTTP response code 404 and error reading body: invalid character 'N' looking for beginning of value
2014/07/02 11:18:12 blog/wp-content/plugins/index.php: Failed to copy: Upload failed: googleapi: Error 400: Invalid value for: Invalid format: "2011-06-30T15:22:09Z" is malformed at "Z"
2014/07/02 11:18:12 blog/wp-content/plugins/index.php: Failed to remove failed copy: googleapi: got HTTP response code 404 and error reading body: invalid character 'N' looking for beginning of value
2014/07/02 11:18:13 blog/wp-content/plugins/headmeta/headmeta.php: Failed to copy: Upload failed: googleapi: Error 400: Invalid value for: Invalid format: "2011-04-29T00:16:24Z" is malformed at "Z"
2014/07/02 11:18:13 blog/wp-content/plugins/headmeta/readme.txt: Failed to copy: Upload failed: googleapi: Error 400: Invalid value for: Invalid format: "2011-04-29T00:16:24Z" is malformed at "Z"
2014/07/02 11:18:13 blog/wp-content/plugins/headmeta/headmeta.php: Failed to remove failed copy: googleapi: got HTTP response code 404 and error reading body: invalid character 'N' looking for beginning of value
2014/07/02 11:18:13 blog/wp-content/plugins/headmeta/readme.txt: Failed to remove failed copy: googleapi: got HTTP response code 404 and error reading body: invalid character 'N' looking for beginning of v

local: subdirectories not working under windows

From Kim via email:

For example, I have a file underC:\Temp\Folder1\sample1.jpg. when I used

rclone copy /Temp remote:backup

on GoogleDrive, it does not create a folder called \Folder1, it creates a file with a
name called Folder1\sample1.jpg.

I tried creating a folder called Folder1 on Google drive before syncing
but it still creates a file with a name called Folder1\sample1.jpg
ignoring the directory Folder1.

OneDrive support

Would be nice to support Microsoft OneDrive. There seems to be some API documentation available at http://msdn.microsoft.com/en-us/library/windows/apps/hh826521.aspx

drive: fix scan of whole file system - unworkable for large numbers of files

Hey.

Found rclone and it's not working at all even with single file:

$ rclone -v copy  rclone-v1.07-linux-amd64.zip  drive:RCLONE/
2015/02/03 20:21:23 Google drive root 'RCLONE': Modify window is 1ms
2015/02/03 20:21:23 Google drive root 'RCLONE': Building file list
2015/02/03 20:22:23 
Transferred:            0 Bytes (   0.00 kByte/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:  1m6.178419486s

2015/02/03 20:23:23 
Transferred:            0 Bytes (   0.00 kByte/s)
Errors:                 0 
Checks:                 0
Transferred:            0
Elapsed time:  2m6.17841706s

2015/02/03 20:24:23 
Transferred:            0 Bytes (   0.00 kByte/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:  3m6.17849835s

2015/02/03 20:25:23 
Transferred:            0 Bytes (   0.00 kByte/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:  4m6.17840109s

Any thoughts ?

rclone config bug

With the latest version, there is a small bug when you make a new remote:

Make a new remote of type "Drive"
After you paste in the token, and click "y" that it's OK, it then asks you for the name again - it leads you back in a loop. You have to type CTRL-C to escape it.

rclone backup: new command with incremental strategy

Hi Nick,

I'm currently using s3ql to mount remote S3/GCS data as a local filesystem (through fuse), and then use a shell script to implement rsync based backups to this filesystem.

I'd like to make use of rclone based backups in ansible-pcd -- for its simplicity and for user friendliness (e.g. browsing on the remote end will display the backed up files themselves -- versus s3ql which displays indecipherable filesystem metadata).

For the majority of "backups", it's important to have incremental functionality that allows you to a restore a file from "yesterday" w/o having "todays" changes override and make that impossible. E.g. snapshots/rotations/&c.

Most incremental strategies also make subsequent backups efficient by limiting what gets backed up to changed components only; speeding up runtime and transfer time, and saving space @ the backup destination.

Protecting against file corruption making its way into downstream [snapshots/rotations/&c] is also a plus.

I think rclone handles the syncing component well -- although don't see anything apparent re: incremental snapshotting. Do you plan to implement or could you share your thoughts on this feature?

How to use GitHub

Please use the 👍 reaction to show that you are affected by the same issue.
Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
Subscribe to receive notifications on status change and new comments.

Is it possible to omit the need of specifying path?

What I meant is, currently if I want to sync a folder, I would do

rclone sync source:path dest:path

But the most convenient way would be I simply go to that folder and do

rclone sync

Because remote path is always hard to remember for storage service like GCS. Is it possible? I imagine having some kind of mapping metadata saved in config file would do it.

Not copying all folders and files from Drive - attempt #2

Attached is a screen capture of my G Drive showing all the folders and some of the files. Below is what: