qiniu / qshell Goto Github PK
View Code? Open in Web Editor NEWShell Tools for Qiniu Cloud
License: MIT License
Shell Tools for Qiniu Cloud
License: MIT License
现在有m3u8delete和m3u8replace,却没有个创建m3u8列表的功能。
由于七牛的API有变动,编译时会出现cannot find package,原github.com/qiniu/api已发生改变:
qshell应该是基于老版本的api实现,可否考虑更新一下?
$ qshell batchrename liyang oldFile.txt
<DANGER> Input gjdjdj to confirm operation: gjdjdj
All Renamed!
oldFile.txt
/123/123.png /234/234.png
All Renamed但是文件没有被rename
每次upload完都要重新设置一下mimetype,感觉好麻烦
2016/07/11 21:34:32 [INFO][qshell] qupload.go:232: Use the last cached local sync dir file list ...
2016/07/11 21:34:32 [ERROR][qshell] qupload.go:340: Invalid cache line 14 14412684965838689
----------Upload Result----------
Total: 0
Success: 0
Failure: 0
Skipped: 0
在windows系统下, 通过 qdownload 命令执行后,下载到本地的图片文件都只有1K。
而且图片全部打不开
上传或者其他原因运行失败后应该返回非0的 error code, use os.Exit(x)
方便后续脚本判断运行状态
需要根据qdownload
配置文件的设置来设置List的域名。
如果我把qshell放置在/root/下,并且配置了account....
当我在/root/a/目录下执行程序就报错..
Open account file failed, open .qshell/account.json: no such file or directory ,please use
account
to set AccessKey and SecretKey first
~/qshell rput backup nginx_20160213.tar.gz ~/backup/nginx_20160213.tar.gz http://up.qiniug.com
"file_list":"test.min.js",
我设置了file_list但是仍然上传了所有的文件
Go里面的path/filepath/WalkFunc
在遇到无法访问目录时,会导致列取终止,除非WalkFunc返回SkipDir。
本地路径包含中文文件夹的情况下,上传照片失败。
另外每个依赖ak,sk的命令运行时输出一行提示信息,提示ak,sk加密存储的位置。
文件被修改 但是文件大小没有改变的情况下 无法通过qdownload 同步到本地
qdownload配置文件如何指定多个prefix
同样的配置,在 mac 下可以,在windows下就会报 401 错误,下载下来的只有文件夹,没有文件。windows 只是 dest_dir 不一样, 我配置的是 "D:\svnbak",使用 1.8.0 版本。
{
"dest_dir" : "/Users/xxx/Downloads/svn",
"bucket" : "svnbak",
"domain" : "http://svnbak.xxx.com",
"access_key" : "xxx",
"secret_key" : "xxx",
"is_private" : true
}
我大概要上载1TB文件 几十万个 。。还不知道怎么能准确无误上传
2016/08/30 17:49:17 [ERROR][cli] cdn.go:26: Get cdn supplier of ip error, unexpected end of JSON input
qshell在遍历大的数据集合的时候,常常因为服务端超时导致list失败,这个时候需要记录下当前的marker,以便于下次重新list的时候,从marker的地方继续开始。原来设计的逻辑对小数据集有效,但是大的数据集,不能不支持从断掉的地方开始重试并写入到同一个文件中。
{
"src_dir" : "`隐去`",
"access_key" : "`隐去`",
"secret_key" : "`隐去`",
"bucket" : "`隐去`",
"file_list" : "/Users/mac/Documents/Qiniu_Tool/qshell-v1.8.1/ao.md",
"zone" : "nb",
"ignore_dir" : false,
"key_prefix" : "",
"up_host" : "http://upload.qiniu.com",
"overwrite" : true,
"check_exists" : true,
"check_hash" : false,
"check_size" : true,
"skip_file_prefixes" : ".git,bin",
"skip_path_prefixes" : "hello/,temp/",
"skip_fixed_strings" : ".svn",
"skip_suffixes" : ".DS_Store,.exe",
"rescan_local" : false,
"log_file" : "upload.log",
"log_level" : "info"
}
预测可能问题为:
╮(╯_╰)╭
MacOS Sierra
问题如题,我想这应该也是为了安全起见,但能否加一个字段,允许将本地不存在的文件删除。
Uploading /Users/dysun/fan/data/happy.png => tewe : 2016/happy.png ...
Progress: 100%
Put file error, invalid character '<' looking for beginning of value
Last time: 0.04 s, Average Speed: 391.8 KB/s
ubuntu@pt-bj-bgp3:/data/qiniu$ ./qshell qupload 10 sync.conf
Listing local sync dir, this can take a long time, please wait patiently...
2015/12/10 20:10:59 [ERROR][qshell] qupload.go:240: Invalid cache line `33 14449019115095594'
-------Upload Result-------
Total: 0
Success: 0
Failure: 0
Skipped: 0
cd $BACKUP_DIR mysqldump $OPTIONS >> $DUMPFILE echo $OPTIONS if [[ $? == 0 ]] then tar czvf $ARCHIVE $DUMPFILE >> $LOGFILE 2>&1 echo "[$ARCHIVE] Backup Successful!" >> $LOGFILE rm -f $DUMPFILE /root/qs/qshell account $AK $SK /root/qs/qshell fput $QN_ZONE $ARCHIVE $BACKUP_DIR/$ARCHIVE else echo "Database Backup fail!" >> $LOGFILE #mail -s "database : $DATABASE Daily Backup Fail!" $MAIL fi
代码如上
执行结果:
Open account file failed, open .qshell/account.json: not a directory Open account file failed, open .qshell/account.json: not a directory, please use
accountto set AccessKey and SecretKey first
该功能的特点要求,所有的文件列表必须是相对于 Src_Dir 的相对路径。
设置 qupload
check_hash
为 false
后,每次上传,同一文件都会重新上传一遍。对于这种文件,浏览器缓存还有效吗?
[WARN][qiniu/api.v6/resumable/io] resumable_api.go:233: resumable.Put 4181 failed: {"error":"","reqid":"2iQAAOQj0eq9IZIU","details":["UP/400"],"code":400}
版本:qshell v1.8.0 win x64
我有上万个小文件想通过qupload进行同步
但同步时似乎对于每个文件都会尝试上传,与服务器交互,导致速度极慢
之前使用的qrsbox有对本地文件的版本数据进行缓存,仅尝试上传有更新的文件
是我配置方法不当还是qshell没有这个功能?
基本做法是:
提示用户要做危险操作了,请输入验证码。验证码用6位随机字符串来避免y/n的误输入。
当前输出信息太多了,能否用进度条替代:https://github.com/cheggaaa/pb
看了一下,只需要三行代码。
qdownload当前是否只支持将整个空间符合规则的一次下下来,能否指定文件起始点,与结束点,因为当前的机制下很可能硬盘被塞满了,但空间可能没完整下载下来
更新至七牛最新支持的API。
通常都要先根据 prefix listbucket, 然后在根据 list 来删除,比较麻烦。可以一步直接根据 prefix 删除。
现在的下载链接中有空格,是这样的:
http://qdisk.qiniudn.com/qshell%20v1.4.5.zip
建议别用空格,按照github release惯例(比如 https://github.com/wifidog/wifidog-gateway/releases ),变成这样:
http://qdisk.qiniudn.com/qshell-1.4.5.zip
{
"dest_dir" : "/path/to/",
"bucket" : "tower",
"access_key" : "xxxx",
"secret_key" : "xxxx",
"domain" : "http://obd7m26p3.bkt.clouddn.com",
"is_private": true,
"prefix" : "717buy_user/12.txt"
}
qshell qdownload
does download the file but show like this:
2016/08/17 12:13:27 [INFO][qshell] qdownload.go:170: Total: 1
2016/08/17 12:13:27 [INFO][qshell] qdownload.go:171: Local: 0
2016/08/17 12:13:27 [INFO][qshell] qdownload.go:172: Success: 0
2016/08/17 12:13:27 [INFO][qshell] qdownload.go:173: Failure: 1
2016/08/17 12:13:27 [INFO][qshell] qdownload.go:174: Duration: 432.707361ms
I print out the resp and find resp.Header.get('Content-Length')
failed. There is no 'Content-Length' in resp.Header
.
So I print out the req to get the URL like this:
http://obd7m26p3.bkt.clouddn.com/717buy_user/12.txt\?e=1471493749&token=xxxxx
Use httpie
to send this request and get the header which does contains 'Content-Length'.
That's so weird.
I try to help find which file could cause this error but I can't. Some other files in the same bucket can download successfully.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.