juicedata / juicesync Goto Github PK
View Code? Open in Web Editor NEWA tool to move your data between any clouds or regions.
License: Apache License 2.0
juicesync's People
Contributors
Stargazers
Watchers
Forkers
suzaku curiositychina hugoren gitking lihangel yujunz nmgliangwei thinking-one-hour-everyday bonecto zhanglei swimfish09 drag0nglass shunpeizhang accpet wprobot koksen ocakgun chenzishi zjnxyz acautman sawdog showsmall 1008610010 tdreaming arvintian angristan vivicai sky365365 heartshare davies dubeno chenyongwang morrishe bitboxclub h2atecnologia zyclove qqshow dhoke4tdb chenjie4255 philipluk suryatmodulus joseangelramos louie007 s21109 ycstech gaojianlipplabs mxlabs youngerliucn xsren luis-sousa-pinto iq-scm 5l1v3r1 tangguoqiang172528725 rohankumardubeyjuicesync's Issues
能支持传输时过滤某些文件吗？
我们有些需求是只同步目录中的部分文件，有些文件时不需要传输的，但是目前没找到有支持这个功能的参数，我们都是先把文件移到其他地方，删掉不需要传输的文件之后再同步，这样做有些繁琐，请问后续有支持文件传输过滤这个功能的开发吗？
More flexible options for authentication
Sometime we need to assume dedicated IAM roles in addition to access key and secret key to grant access to the bucket.
sftp 能否能支持非22端口？
sftp 能否增加支持非22端口？ 我看代码实现内没有实现这个功能
problem with --perms in sftp
debug logs as follow:
2021/03/23 15:17:30.195700 juicefs[86870] : Iterating objects from [email protected]:/opt/data/ start ""
*panic: interface conversion: object.sftpStore is not object.FileSystem: missing method Chmod
goroutine 53 [running]:
github.com/juicedata/juicefs/pkg/sync.worker(0xc000498120, 0x26fa460, 0xc0000a9480, 0x26fade0, 0xc000092300, 0xc000309ea0)
/home/runner/go/pkg/mod/github.com/juicedata/[email protected]/pkg/sync/sync.go:382 +0x110f
github.com/juicedata/juicefs/pkg/sync.Sync.func1(0xc00048a120, 0xc000498120, 0x26fa460, 0xc0000a9480, 0x26fade0, 0xc000092300, 0xc000309ea0)
/home/runner/go/pkg/mod/github.com/juicedata/[email protected]/pkg/sync/sync.go:627 +0xa3
created by github.com/juicedata/juicefs/pkg/sync.Sync
/home/runner/go/pkg/mod/github.com/juicedata/[email protected]/pkg/sync/sync.go:625 +0x171
Comparision to rclone
How do you compare juicesync to rclone sync ( https://rclone.org/commands/rclone_sync/ )? What are the advantages? What are the expected benefits? What caused the creation of a new project, instead of developing the existing ones?
[Question] Choose region
Hello,
I'm user of Scaleway provider, he provide standard S3 compatible with amazon S3, but when i try to connect to, i have a problem with the region. Do you have an option to change the region during the connection ?
valentin@nuc01:~/juicefs$ ./juicefs format --storage s3 --bucket https://xxx.s3.fr-par.scw.cloud/ --access-key XXXX --secret-key XXXX localhost test
2021/01/22 22:37:14.515694 juicefs[443170] <INFO>: Meta address: redis://localhost
2021/01/22 22:37:14.517149 juicefs[443170] <WARNING>: AOF is not enabled, you may lose data if Redis is not shutdown properly.
2021/01/22 22:37:14.517503 juicefs[443170] <INFO>: Data uses s3://xxxx/test/
2021/01/22 22:37:26.994285 juicefs[443170] <FATAL>: Storage s3://xxxxx/test/ is not configured correctly: Failed to create s3://xxxxt/test/: AuthorizationHeaderMalformed: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'fr-par'
status code: 400, request id: txf8d357f0026e4f, host id: txf8d357f0026ef, previous error: AuthorizationHeaderMalformed: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'fr-par'
status code: 400, request id: tx110744ef80c84f, host id: tx110744ef80c8f
please create bucket s3://xxxx/test/ manually, then format again
Thanks
Error running make under macOS
$ make
go build -ldflags="-L/usr/local/opt/openssl/lib -L/usr/local/opt/readline/lib -L/usr/local/opt/zlib/lib -L/usr/local/opt/sqlite3/lib" -o juicesync
# github.com/juicedata/juicesync
flag provided but not defined: -L/usr/local/opt/openssl/lib
usage: link [options] main.o
...
make: *** [juicesync] Error 2
Failed to build with `GO111MODULE=on`
✗ GO111MODULE=on go get github.com/juicedata/juicesync
...
build github.com/juicedata/juicesync: cannot load huawei/obs: cannot find module providing package huawei/obs
It looks good when it is off
GO111MODULE=off go get github.com/juicedata/juicesync
SerializationError: failed to decode S3 XML error response
./juicesync qiniu://KEY:[email protected]/ qingstor://KEY:[email protected]/Output log:
2018/11/04 11:17:01.184505 <WARNING>: Failed to replicate testdir/testfile from qiniu://test-bucket to qingstor://other-test-bucket: SerializationError: failed to decode S3 XML error response
status code: 404, request id: , host id:
caused by: expected element type <Error> but have <html>
读取数据失败时的异常处理
使用场景是oss 两个bucket之间做数据同步，使用过程中发现了src 和dst 文件大小一致，md5sum不一致的情况；
检查了oss 访问日志，发现出现异常的文件有过一次返回500 的get请求, 并且后续没有重试，在出错的时间点，观察到juicesync有debug日志:
Copied xxxx part 16，没有失败的日志，并且正常退出；
咨询了oss 的研发人员，get 请求服务端返回500时sdk 会有异常的返回值，请帮忙确认下在get object 返回异常时的处理是否正确：
https://github.com/juicedata/juicesync/blob/master/sync/sync.go#L252
if dst object is file, temporary tiles will not be deleted
if dst object is file, the file in /tmp will not be deleted.
Missing full synopsis in help message
$ juicesync -h
Usage of juicesync:
-end string
the last keys to sync
-p int
number of concurrent threads (default 50)
-q change log level to ERROR
-start string
the start of keys to sync
-v turn on debug log
vs
$ juicesync
juicesync [options] SRC DST
panic: runtime error: index out of range
juicesync s3://*:*@https://s3-us-west-1.amazonaws.com/bucket/prefix/ s3://*:*@https://s3.cn-north-1.amazonaws.com.cn/bucket/prefix/
panic: runtime error: index out of range
goroutine 1 [running]:
github.com/juicedata/juicesync/object.newS3(0xc000023fd0, 0xe, 0x7ffd71fb45de, 0x14, 0x7ffd71fb45f3, 0x28, 0xc0001e9dc8, 0x448247)
/home/ubuntu/go/src/github.com/juicedata/juicesync/object/s3.go:211 +0x4a6
github.com/juicedata/juicesync/object.CreateStorage(0x7ffd71fb45d9, 0x2, 0xc000023fd0, 0xe, 0x7ffd71fb45de, 0x14, 0x7ffd71fb45f3, 0x28, 0x0, 0xc0001e9e80)
/home/ubuntu/go/src/github.com/juicedata/juicesync/object/object_storage.go:101 +0xbb
main.createStorage(0x7ffd71fb45d9, 0x93, 0xc000023fc0, 0xe)
/home/ubuntu/go/src/github.com/juicedata/juicesync/main.go:71 +0x197
main.main()
/home/ubuntu/go/src/github.com/juicedata/juicesync/main.go:97 +0x110
Getting panic when using Google Cloud Storage
version 0.9.3
Getting following error when trying to use GCS storage
/juicefs format --storage gs --bucket gs://bucket-example-test localhost test2
2021/01/20 00:46:45.475761 juicefs[954814] <INFO>: Meta address: redis://localhost
2021/01/20 00:46:45.476692 juicefs[954814] <WARNING>: AOF is not enabled, you may lose data if Redis is not shutdown properly.
panic: runtime error: index out of range [1] with length 1
goroutine 1 [running]:
github.com/juicedata/juicesync/object.newGS(0x7ffd6cdcf170, 0x10, 0x0, 0x0, 0x0, 0x0, 0x2, 0x0, 0x16, 0xc00011cd80)
/go/pkg/mod/github.com/juicedata/[email protected]/object/gs.go:145 +0x3e6
github.com/juicedata/juicesync/object.CreateStorage(0x7ffd6cdcf164, 0x2, 0x7ffd6cdcf170, 0x10, 0x0, 0x0, 0x0, 0x0, 0x16, 0xc00011cd80, ...)
/go/pkg/mod/github.com/juicedata/[email protected]/object/object_storage.go:123 +0x1ad
github.com/juicedata/juicefs/pkg/object.CreateStorage(0x7ffd6cdcf164, 0x2, 0x7ffd6cdcf170, 0x10, 0x0, 0x0, 0x0, 0x0, 0xc000559920, 0x45f8c7, ...)
/go/src/github.com/juicedata/juicefs/pkg/object/interface.go:57 +0x26d
main.createStorage(0xc000559b68, 0x15894c1, 0x8, 0xc00012ba50, 0xe)
/go/src/github.com/juicedata/juicefs/cmd/format.go:54 +0xa5
main.format(0xc00009b600, 0x8, 0xe)
/go/src/github.com/juicedata/juicefs/cmd/format.go:165 +0x8fe
github.com/urfave/cli/v2.(*Command).Run(0xc000169440, 0xc00009b380, 0x0, 0x0)
/go/pkg/mod/github.com/urfave/cli/[email protected]/command.go:163 +0x4ed
github.com/urfave/cli/v2.(*App).RunContext(0xc000243a00, 0x178e700, 0xc0000400b8, 0xc00003c080, 0x8, 0x8, 0x0, 0x0)
/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:313 +0x81f
github.com/urfave/cli/v2.(*App).Run(...)
/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:224
main.main()
/go/src/github.com/juicedata/juicefs/cmd/main.go:74 +0x93c
"out of memory" when sync from s3 to bos
The used percent of memory will rise continuously.
You can start a process of juicesync, which do synchronizing objects from s3(or s3 compatible, like ceph and minio) to bos. And confirm it by print MEM output.
while true; do ps aux|grep "juice\|MEM"|grep -v grep; sleep 1; done
And after a while, the process will be killed by system.
panic: runtime error: slice bounds out of range
When try to sync ALL objects from one bucket to another.
$ juicesync s3://*:*@bucket.s3-us-west-1.amazonaws.com/ s3://*:*@bucket.s3.cn-north-1.amazonaws.com.cn/
panic: runtime error: slice bounds out of range
goroutine 1 [running]:
github.com/juicedata/juicesync/object.newS3(0xc00016a550, 0x4f, 0x7ffda78e852b, 0x14, 0x7ffda78e8540, 0x28, 0xc00027fdc8, 0x448247)
/home/admin/go/src/github.com/juicedata/juicesync/object/s3.go:211 +0x49f
github.com/juicedata/juicesync/object.CreateStorage(0x7ffda78e8526, 0x2, 0xc00016a550, 0x4f, 0x7ffda78e852b, 0x14, 0x7ffda78e8540, 0x28, 0xe645a0, 0xc0001e97c0)
/home/admin/go/src/github.com/juicedata/juicesync/object/object_storage.go:101 +0xbb
main.createStorage(0x7ffda78e8526, 0x8b, 0xe645a0, 0xc0001e97c0)
/home/admin/go/src/github.com/juicedata/juicesync/main.go:71 +0x197
main.main()
/home/admin/go/src/github.com/juicedata/juicesync/main.go:98 +0x13f
format panic with qiniu kodo
What happened:
juicefs format --storage=qiniu --bucket=http://some_qiniu_cdn_domain localhost test
2021/01/20 14:41:10.771521 juicefs[1728] : Meta address: redis://localhost
panic: runtime error: slice bounds out of range [:-1]
goroutine 1 [running]:
github.com/juicedata/juicesync/object.newQiniu(0x7ffd3dca278f, 0x1e, 0xc00004200b, 0x28, 0xc00004204b, 0x28, 0x2, 0x157d0a6, 0x7ffd3dca2780, 0x5)
/home/xxx/go/pkg/mod/github.com/juicedata/[email protected]/object/qiniu.go:164 +0x106e
github.com/juicedata/juicesync/object.CreateStorage(0x7ffd3dca2780, 0x5, 0x7ffd3dca278f, 0x1e, 0xc00004200b, 0x28, 0xc00004204b, 0x28, 0x24, 0xc000046540, ...)
/home/xxx/go/pkg/mod/github.com/juicedata/[email protected]/object/object_storage.go:123 +0x1ad
github.com/juicedata/juicefs/pkg/object.CreateStorage(0x7ffd3dca2780, 0x5, 0x7ffd3dca278f, 0x1e, 0xc00004200b, 0x28, 0xc00004204b, 0x28, 0xc0005df920, 0x45fb87, ...)
/home/xxx/Downloads/juicefs/pkg/object/interface.go:57 +0x26d
main.createStorage(0xc0005dfb68, 0x15806a1, 0x8, 0xc000548a40, 0x1c)
/home/xxx/Downloads/juicefs/cmd/format.go:54 +0xa5
main.format(0xc00054cd40, 0x8, 0xe)
/home/xxx/Downloads/juicefs/cmd/format.go:165 +0x8fe
github.com/urfave/cli/v2.(*Command).Run(0xc000141320, 0xc00054ca80, 0x0, 0x0)
/home/xxx/go/pkg/mod/github.com/urfave/cli/[email protected]/command.go:163 +0x4ed
github.com/urfave/cli/v2.(*App).RunContext(0xc0004b4ea0, 0x1784d80, 0xc00012a010, 0xc000130120, 0x6, 0x6, 0x0, 0x0)
/home/xxx/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:313 +0x81f
github.com/urfave/cli/v2.(*App).Run(...)
/home/xxx/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:224
main.main()
/home/xxx/Downloads/juicefs/cmd/main.go:74 +0x93c
What you expected to happen:
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
Environment:
- JuiceFS version (use
./juicefs --version): juicefs version 0.9.3-5 (2021-01-19 18baa89) - Cloud provider or hardware configuration running JuiceFS:
- OS (e.g:
cat /etc/os-release): Debian GNU/Linux 10 (buster) - Kernel (e.g.
uname -a): SMP Debian 4.19.160-2 (2020-11-28) x86_64 GNU/Linux - Object storage (cloud provider and region): qiniu
- Redis info (version, cloud provider managed or self maintained): 5.0.9
- Network connectivity (JuiceFS to Redis, JuiceFS to object storage): ok
- Others:
REDIS_PASSWORD, ACCESS_KEY and SECRET_KEY environment variables are used
Add flag to gate copy links
Similar to rsync
-L, --copy-links transform symlink into referent file/dir
$ 符号结尾问题
object/s3.go
func (s *s3client) List(prefix, marker string, limit int64) ([]*Object, error) {
param := s3.ListObjectsInput{
Bucket: &s.bucket,
Prefix: &prefix,
Marker: &marker,
MaxKeys: &limit,
}
resp, err := s.s3.ListObjects(¶m)
if err != nil {
return nil, err
}
n := len(resp.Contents)
objs := make([]*Object, 0)
for i := 0; i < n; i++ {
o := resp.Contents[i]
if strings.HasSuffix(*o.Key, "$") {
logger.Infof("s3 path: %s was empty dir, skip it!", *o.Key)
continue
}
mtime := int(o.LastModified.Unix())
objs = append(objs, &Object{*o.Key, *o.Size, mtime, mtime})
}
/*
objs := make([]*Object, n)
for i := 0; i < n; i++ {
o := resp.Contents[i]
mtime := int(o.LastModified.Unix())
objs[i] = &Object{*o.Key, *o.Size, mtime, mtime}
}
*/
return objs, nil
}
目标端linux的ssh端口不是默认的22，无法使用
从本机向一台linux同步文件，该linux系统的ssh端口不是22，执行失败，将该端口修改回22后可以执行。
gorelease is broken
gorelease fail to publish a new release
Run actions/[email protected]
with:
go-version: 1.13.x
Error: Unable to process command '##[set-env name=GOROOT;]/opt/hostedtoolcache/go/1.13.15/x64' successfully.
Error: The `set-env` command is disabled. Please upgrade to using Environment Files or opt into unsecure command execution by setting the `ACTIONS_ALLOW_UNSECURE_COMMANDS` environment variable to `true`. For more information see: https://github.blog/changelog/2020-10-01-github-actions-deprecating-set-env-and-add-path-commands/
Error: Unable to process command '##[add-path]/opt/hostedtoolcache/go/1.13.15/x64/bin' successfully.
Error: The `add-path` command is disabled. Please upgrade to using Environment Files or opt into unsecure command execution by setting the `ACTIONS_ALLOW_UNSECURE_COMMANDS` environment variable to `true`. For more information see: https://github.blog/changelog/2020-10-01-github-actions-deprecating-set-env-and-add-path-commands/
更好地处理部分传输失败的情况
可选的设计：
- 添加一下参数控制，允许这种传输失败时，抛出异常
- 将所有失败的传输项加入一个 retry list，允许用户手工重试
Another way to install juicefs and juicesync
See details LinuxSuRen/hd-home#125
Need to support native symbolic link semantics
In the current version, if there is a symbolic link in the copy directory, it will copy the actual file of the symbolic link to the target directory instead of directly copying the symbolic link. But in some cases, it is necessary to copy the symbolic link as it is. It is better to add a --no-follow-link parameter option to copy the file directly instead of copying the symbolic link.
Support copying symlink
When syncing from file:// to object storage, it would be nice to support symlink by copying content like rclone copy $DIST_BASE/ $RCLONE_DST --copy-links
Currently it throws an error
+ mkdir -p release/dist/channels
+ ln -fnsv ../branches/release-juicesync release/dist/channels/stable
'release/dist/channels/stable' -> '../branches/release-juicesync'
+ juicesync release/dist oss://***:***@jfs-release.oss-cn-hongkong.aliyuncs.com/dist
2020/01/02 12:57:38.829321 <INFO>: Syncing between file:///home/runner/work/jfs/jfs/release/dist/ and oss://jfs-release/dist (starting from "")
2020/01/02 12:57:45.926890 <ERROR>: Failed to copy channels/stable: read /home/runner/work/jfs/jfs/release/dist/channels/stable: is a directory
Support HDFS
HDFS is widely used storage for big data.
3P 数据跨云传多久？
Q1. 3P 数据跨云传多久？
Q2. 拷贝的时候会拷贝metadata 么？
Q3. 支持集群拷贝么？
Q4. 有流量统计或者进度统计么？
panic: runtime error: index out of range [1] with length 1
juicesync file://dist s3://jfs-release/dist
panic: runtime error: index out of range [1] with length 1
goroutine 1 [running]:
github.com/juicedata/juicesync/object.newS3(0xc0000c1700, 0x13, 0x0, 0x0, 0x0, 0x0, 0x2, 0x2)
/home/runner/work/juicesync/juicesync/object/s3.go:214 +0x86b
github.com/juicedata/juicesync/object.CreateStorage(0x7ffeefbfeafe, 0x2, 0xc0000c1700, 0x13, 0x0, 0x0, 0x0, 0x0, 0x1ce0260, 0xc0003079e0)
/home/runner/work/juicesync/juicesync/object/object_storage.go:101 +0xac
main.createStorage(0x7ffeefbfeafe, 0x15, 0x1ce0260, 0xc0003079e0)
/home/runner/work/juicesync/juicesync/main.go:100 +0x19d
main.main()
/home/runner/work/juicesync/juicesync/main.go:133 +0x1ab
qiniu cloud kodo, please
use juicesync copy data from jfs without mount
it will be more convenient to use juicesync to copy data to or from jfs without mount jfs to local filesystem
使用sftp传输，即使指定了SSH_PRIVATE_KEY_PATH，还是提示输入密码
即使指定了环境变量SSH_PRIVATE_KEY_PATH，还是提示输入密码，输入空密码，使用的是证书认证。
如果src 和tgt 都是本地文件系统，同步的速度会超过rsync吗？
具体使用场景是将大量小文件从一个磁盘 sync到另外一个磁盘。
不知道使用juicesync 相对于rsync 会有提速吗？
使用rsync时明显没有将磁盘的读取、写入带宽用满。
另外可以解释下在sync时，terminal上显示的信息具体代表什么含义吗？
Web GUI
能否有个gui界面？
图形化配置更方便
out of memory
sync qiniu large file request canceled
large file like video
2018/11/12 09:30:30.952158 <WARNING>: Failed to replicate dir/bigfile from qiniu://test-blob to qingstor://test-blob: net/http: request canceled (Client.Timeout exceeded while reading body)
version: current master branch
文件夹权限和 "/" 文件的问题
感谢 JuiceFS 团队制作如此优秀的工具，近期在从 七牛云 以 SFTP 传输数据的时候发现了两个问题：
第一个：目标服务器的文件夹权限发生了更改：
755 是正确的权限（我看到其他的都是这个权限），而传输过去的文件夹是 700。由于在 Unix 系统里采用了「只读」权限的策略，导致只读权限账号无法读取 700 权限的文件夹内容：
请帮忙查证一下看是否有必要在这里进行一些修补或者完善，需要指明的是，这个问题并不是在最新的 0.1.0 中出现，而且是很偶然的。
第二个：特殊文件处理 比如 "/" 这样的文件：
因为使用了镜像存储，所以七牛云会把诸如 example.com/assets/ 和 example.com/assets// 在存储系统里认为是两个文件，所以这可能是如下报错产生的原因：
2020/01/30 05:07:55.263889 <ERROR>: Failed to copy /: sftp: "Failure" (SSH_FX_FAILURE)
请帮忙查证一下这个问题，需要指明的是，这个问题仅出现在最新的，从 releases 里下载的 0.1.0 版本中存在，之前应该是没有这样的问题。
感谢制作这个优秀的工具并持续维护。
以上，
谢谢！
希望支持天翼云 对象存储【OOS】
希望和天翼元对象存储进行对接。
天翼云对象存储的开发者文档可以通过这里获取。
https://www.ctyun.cn/help2/10000101/10001738
有2个文档
对象存储网络 和 其他区域他们所对应的endpoint
可以在这里查到
https://www.ctyun.cn/help2/10000101/10474062
please add Windows support
Sometimes I have to run JuiceSync under Windows. Please add Windows support.
空文件不传输问题
// sync.go
52 if obj.Size == 0 || strings.HasSuffix(key, "/")
我们会有一些空文件同样需要传输的，已经修改成
// sync.go
52 if strings.HasSuffix(key, "/")
能否支持一下传输失败时，抛出异常的功能
我们在传输oss文件到s3的时候，传输已经失败了，但是没有抛出异常，Succeed里还认为成功了，不清楚这是基于什么原因设计的，如果这个算正常功能的话，能否添加一下参数控制，允许这种传输失败时，抛出异常
下面是我们的报错日志：
2020/03/13 03:41:12.538091 <DEBUG>: Creating oss storage at endpoint https://xxxx.oss-cn-shanghai.aliyuncs.com
2020/03/13 03:41:12.538152 <DEBUG>: Creating s3 storage at endpoint https://xxxx.s3.cn-north-1.amazonaws.com.cn
2020/03/13 03:41:12.538249 <INFO>: Syncing from oss://xxxxxx/xxxx/xxx/xx/data_date=2020-03-12/ to s3://xxxxxx/xxxx/xxx/xx/data_date=2020-03-12/
2020/03/13 03:41:12.538255 <DEBUG>: maxResults: 10240, defaultPartSize: 5242880, maxBlock: 10485760
2020/03/13 03:41:12.538264 <DEBUG>: Listing objects from oss://xxxxxx/xxxx/xxx/xx/data_date=2020-03-12/ marker ""
2020/03/13 03:41:12.586283 <DEBUG>: Found 1 object from oss://xxxxxx/xxxx/xxx/xx/data_date=2020-03-12/ in 47.999306ms
2020/03/13 03:41:12.586797 <DEBUG>: Listing objects from s3://xxxxxx/xxxx/xxx/xx/data_date=2020-03-12/ marker ""
2020/03/13 03:41:12.587225 <DEBUG>: Continue listing objects from oss://xxxxxx/xxxx/xxx/xx/data_date=2020-03-12/ marker "data-00000-of-00001.tsv.gz"
2020/03/13 03:41:12.591886 <DEBUG>: Found 0 object from oss://xxxxxx/xxxx/xxx/xx/data_date=2020-03-12/ in 5.356395ms
2020/03/13 03:41:12.959493 <DEBUG>: Found 0 object from s3://xxxxxx/xxxx/xxx/xx/data_date=2020-03-12/ in 373.069188ms
2020/03/13 03:41:13.007916 <DEBUG>: Copying object data-00000-of-00001.tsv.gz as 55 parts (size: 5242880): EVf5BS8pLJIKHHDPgu8I.FWYHXZ4KW1E44vZfC3legCZ7m_k5RqsvNGlMcqEfjrQHeqk_X5DEQ7iT7FNHUk58zyKaqwmKuwdeISkWjWalxVTQ0EyLz2NBDcnF2Fl0CU7
2020/03/13 03:46:53.907529 <DEBUG>: Copied data-00000-of-00001.tsv.gz part 33
2020/03/13 03:47:54.303755 <DEBUG>: Copied data-00000-of-00001.tsv.gz part 52
2020/03/13 03:48:36.707127 <DEBUG>: Copied data-00000-of-00001.tsv.gz part 26
2020/03/13 03:49:09.955661 <DEBUG>: Copied data-00000-of-00001.tsv.gz part 28
2020/03/13 03:50:47.422721 <DEBUG>: Copied data-00000-of-00001.tsv.gz part 20
2020/03/13 03:50:57.765377 <DEBUG>: Copied data-00000-of-00001.tsv.gz part 53
2020/03/13 03:52:20.123703 <DEBUG>: Copied data-00000-of-00001.tsv.gz part 38
2020/03/13 03:52:39.790954 <DEBUG>: Copied data-00000-of-00001.tsv.gz part 46
2020/03/13 03:53:06.143590 <DEBUG>: Copied data-00000-of-00001.tsv.gz part 10
2020/03/13 03:53:17.350579 <DEBUG>: Copied data-00000-of-00001.tsv.gz part 54
2020/03/13 03:53:29.880241 <DEBUG>: Copied data-00000-of-00001.tsv.gz part 36
2020/03/13 03:53:33.591953 <DEBUG>: Copied data-00000-of-00001.tsv.gz part 1
2020/03/13 03:53:44.180034 <DEBUG>: Copied data-00000-of-00001.tsv.gz part 50
2020/03/13 03:54:04.815573 <DEBUG>: Copied data-00000-of-00001.tsv.gz part 22
2020/03/13 03:54:13.727680 <DEBUG>: Copied data-00000-of-00001.tsv.gz part 25
2020/03/13 03:54:29.634812 <DEBUG>: Copied data-00000-of-00001.tsv.gz part 45
2020/03/13 03:54:54.144061 <DEBUG>: Copied data-00000-of-00001.tsv.gz part 8
2020/03/13 03:56:39.667419 <DEBUG>: Copied data-00000-of-00001.tsv.gz part 41
2020/03/13 03:58:15.571480 <DEBUG>: Copied data-00000-of-00001.tsv.gz part 30
2020/03/13 03:58:16.822260 <DEBUG>: Copied data-00000-of-00001.tsv.gz part 16
2020/03/13 03:59:29.646285 <DEBUG>: Copied data-00000-of-00001.tsv.gz part 14
2020/03/13 04:00:23.323447 <DEBUG>: Copied data-00000-of-00001.tsv.gz part 24
2020/03/13 04:01:12.741026 <DEBUG>: Copied data-00000-of-00001.tsv.gz part 40
2020/03/13 04:01:16.116275 <DEBUG>: Copied data-00000-of-00001.tsv.gz part 19
2020/03/13 04:01:22.773482 <DEBUG>: Copied data-00000-of-00001.tsv.gz part 48
2020/03/13 04:01:23.387022 <DEBUG>: Copied data-00000-of-00001.tsv.gz part 29
2020/03/13 04:02:17.003057 <WARNING>: Failed to copy data-00000-of-00001.tsv.gz part 43: RequestTimeout: Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.
status code: 400, request id: D7422D8F25A99995, host id: HMmsZm3jrdbkrEAnCQUbEeGA3wyfTatUVAgkJ480Lo5y44TEFLZdjvFMTnQg5RhxT1pdLhCAQss=
2020/03/13 04:02:17.141562 <ERROR>: Failed to copy data-00000-of-00001.tsv.gz: multipart: part 43: RequestTimeout: Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.
status code: 400, request id: D7422D8F25A99995, host id: HMmsZm3jrdbkrEAnCQUbEeGA3wyfTatUVAgkJ480Lo5y44TEFLZdjvFMTnQg5RhxT1pdLhCAQss=
2020/03/13 04:02:17.141589 <INFO>: Found: 1, copied: 0, deleted: 0, failed: 1
关于文件同步流程的优化
对象存储的文件同步能否不经过本地中转，实现对象存储之间的直接同步操作？
Support a file list as SRC
Hi @davies, thanks for sharing this great tool juicesync !
I'm wondering if you can add a new feature like supporting a plain text file as SRC ?
eg.
s3://bucket1/file1
oss://buckert2/Dir1/
etc
请增加对天翼云（融合版）的支持
感谢作者的这个产品，实测速度非常快，尤其注意到最近增加了对天翼云对象存储的支持。不过天翼云对象存储有两个版本（经典版、融合版），其中融合版基于ceph实现，测试过程中发现使用oos://前缀无法访问融合版（我测试使用的endpoint为http://cqoss.xstore.ctyun.cn）。
Kerberized HDFS Support
Hi,juicesync is an awsome data sync tool.But I found that when I start to sync data from 2 hdfss,it shows "no available namenodes: write tcp x.x.x.x:59738->x.x.x.x:8020: write: broken pipe".Does it support kerberized HDFS?Thank you.
aws bucket name 包含点就无法同步
object/s3.go
212 hostParts := strings.SplitN(uri.Host, ".", 2)
这样写的话 bucket name 中包含点 就 GG 了
麻烦 Fix
如下是我自己修改的
hostParts := strings.SplitN(uri.Host, ".s3.", 2)
bucket := hostParts[0]
endpoint = hostParts[1]
endpoint = "s3." + endpoint
能支持 单个 File 传输吗？
我们有部分功能是需要可以单个文件传输，现在就有点麻烦，需要挪动到目录中再传输
从文件系统同步时符号链接会follow link，有循环时导致无限循环
Add mime-type to objects based on extension
现在上传文件，默认content-type字段是 application/octet-stream，当对象存储用于网站支持时，浏览器不能识别对应类型，导致网站不能正常访问。
希望可以添加此功能
update 文件 -u 比对逻辑无精确到毫秒问题
mtime := int(o.LastModified.Unix())
objs = append(objs, &Object{*o.Key, *o.Size, mtime, mtime})
是否可以支持 毫秒
Warnings on `go get github.com/juicedata/juicesync`
✗ go get github.com/juicedata/juicesync
# github.com/yunify/qingstor-sdk-go/request/signer
../../pkg/mod/github.com/yunify/[email protected]+incompatible/request/signer/qingstor.go:90:3: undefined: convert.StringToUnixTimestamp
../../pkg/mod/github.com/yunify/[email protected]+incompatible/request/signer/qingstor.go:116:3: undefined: convert.StringToUnixTimestamp
../../pkg/mod/github.com/yunify/[email protected]+incompatible/request/signer/qingstor.go:146:3: undefined: convert.StringToUnixTimestamp
../../pkg/mod/github.com/yunify/[email protected]+incompatible/request/signer/qingstor.go:172:3: undefined: convert.StringToUnixTimestamp
../../pkg/mod/github.com/yunify/[email protected]+incompatible/request/signer/qingstor.go:236:3: undefined: convert.StringToUnixTimestamp
# github.com/yunify/qingstor-sdk-go/request/unpacker
../../pkg/mod/github.com/yunify/[email protected]+incompatible/request/unpacker/base.go:74:5: undefined: convert.StringToUnixTimestamp
../../pkg/mod/github.com/yunify/[email protected]+incompatible/request/unpacker/base.go:87:3: undefined: convert.StringToUnixTimestamp
../../pkg/mod/github.com/yunify/[email protected]+incompatible/request/unpacker/base.go:155:6: undefined: convert.StringToUnixTimestamp
../../pkg/mod/github.com/yunify/[email protected]+incompatible/request/unpacker/base.go:178:5: undefined: convert.StringToUnixTimestamp
# github.com/yunify/qingstor-sdk-go/request/builder
../../pkg/mod/github.com/yunify/[email protected]+incompatible/request/builder/qingstor.go:80:3: undefined: convert.StringToUnixTimestamp
../../pkg/mod/github.com/yunify/[email protected]+incompatible/request/builder/qingstor.go:86:3: undefined: convert.StringToUnixTimestamp
../../pkg/mod/github.com/yunify/[email protected]+incompatible/request/builder/qingstor.go:93:4: undefined: convert.StringToUnixTimestamp
✗ go version
go version go1.13.3 darwin/amd64
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.