qiniu / api.v7 Goto Github PK
View Code? Open in Web Editor NEWQiniu SDK for Golang (v7.x)
License: MIT License
Qiniu SDK for Golang (v7.x)
License: MIT License
> go get -v qiniupkg.com/api.v7
Fetching https://qiniupkg.com/api.v7?go-get=1
https fetch failed.
import "qiniupkg.com/api.v7": https fetch: Get https://qiniupkg.com/api.v7?go-get=1: x509: certificate is valid for *.qbox.me, qbox.me, not qiniupkg.com
package qiniupkg.com/api.v7: unrecognized import path "qiniupkg.com/api.v7"
环境是 Go 1.10.
在我们的 log 里发现大量的 [INFO] github.com/qiniu/api.v7/storage/resume_upload.go:251: resumable.Put retrying ... 2 reason: token expired 12345 seconds ago 这样的错误,经过调查,认为应该是 SDK 的 bug。
表现为在一个 long running 的程序里,持续需要上传不同的文件时,后续上传所使用的 upToken 始终是第一次设置的 upToken(虽然每次都有重新设置),导致过了默认的 expire 时间(1h)后,所有上传都会失败。
用示例的 https://github.com/qiniu/api.v7/blob/master/examples/resume_upload_advanced.go 改写了一个可以复现的程序(其中部分环境变量需要自己设置):
(注意程序中的 Expires 被故意设置为了 1s。)
package main
import (
"crypto/md5"
"encoding/hex"
"encoding/json"
"fmt"
"io/ioutil"
"os"
"path/filepath"
"sync"
"time"
"github.com/qiniu/api.v7/auth/qbox"
"github.com/qiniu/api.v7/storage"
"golang.org/x/net/context"
)
func md5Hex(str string) string {
h := md5.New()
h.Write([]byte(str))
return hex.EncodeToString(h.Sum(nil))
}
type ProgressRecord struct {
Progresses []storage.BlkputRet `json:"progresses"`
}
func UploadToQiNiu(localFile string, remoteKey string) (error, string) {
var accessKey = os.Getenv("qiniuAccessKey")
var secretKey = os.Getenv("qiniuSecretKey")
bucket := os.Getenv("bucket")
putPolicy := storage.PutPolicy{
Scope: bucket,
Expires: 1, //*** 为了暴露问题,故意设置为很短
}
mac := qbox.NewMac(accessKey, secretKey)
upToken := putPolicy.UploadToken(mac)
// println(upToken)
cfg := storage.Config{}
// 空间对应的机房
cfg.Zone = &storage.ZoneHuadong
// 是否使用https域名
cfg.UseHTTPS = false
// 上传是否使用CDN上传加速
cfg.UseCdnDomains = false
// 必须仔细选择一个能标志上传唯一性的 recordKey 用来记录上传进度
// 我们这里采用 md5(bucket+key+local_path+local_file_last_modified)+".progress" 作为记录上传进度的文件名
fileInfo, statErr := os.Stat(localFile)
if statErr != nil {
return statErr, ""
// fmt.Println(statErr)
// return
}
fileSize := fileInfo.Size()
fileLmd := fileInfo.ModTime().UnixNano()
recordKey := md5Hex(fmt.Sprintf("%s:%s:%s:%s", bucket, remoteKey, localFile, fileLmd)) + ".progress"
// 指定的进度文件保存目录,实际情况下,请确保该目录存在,而且只用于记录进度文件
recordDir := "/tmp"
mErr := os.MkdirAll(recordDir, 0755)
if mErr != nil {
fmt.Println("mkdir for record dir error,", mErr)
return mErr, ""
}
recordPath := filepath.Join(recordDir, recordKey)
progressRecord := ProgressRecord{}
// 尝试从旧的进度文件中读取进度
recordFp, openErr := os.Open(recordPath)
if openErr == nil {
progressBytes, readErr := ioutil.ReadAll(recordFp)
if readErr == nil {
mErr := json.Unmarshal(progressBytes, &progressRecord)
if mErr == nil {
// 检查context 是否过期,避免701错误
for _, item := range progressRecord.Progresses {
if storage.IsContextExpired(item) {
fmt.Println(item.ExpiredAt)
progressRecord.Progresses = make([]storage.BlkputRet, storage.BlockCount(fileSize))
break
}
}
}
}
recordFp.Close()
}
if len(progressRecord.Progresses) == 0 {
progressRecord.Progresses = make([]storage.BlkputRet, storage.BlockCount(fileSize))
}
resumeUploader := storage.NewResumeUploader(&cfg)
ret := storage.PutRet{}
progressLock := sync.RWMutex{}
putExtra := storage.RputExtra{
Progresses: progressRecord.Progresses,
Notify: func(blkIdx int, blkSize int, ret *storage.BlkputRet) {
progressLock.Lock()
progressLock.Unlock()
//将进度序列化,然后写入文件
progressRecord.Progresses[blkIdx] = *ret
progressBytes, _ := json.Marshal(progressRecord)
fmt.Println("write progress file", blkIdx, recordPath)
wErr := ioutil.WriteFile(recordPath, progressBytes, 0644)
if wErr != nil {
fmt.Println("write progress file error,", wErr)
}
},
}
err := resumeUploader.PutFile(context.Background(), &ret, upToken, remoteKey, localFile, &putExtra)
if err != nil {
return err, ""
// fmt.Println(err)
// return
}
//上传成功之后,一定记得删除这个进度文件
os.Remove(recordPath)
fmt.Println(ret.Key, ret.Hash)
return nil, fmt.Sprintf("http://domain/%s\n", remoteKey)
}
func main() {
var localFile string
var remotePath string
if len(os.Args) > 1 {
localFile = os.Args[1]
}
if len(os.Args) > 2 {
remotePath = os.Args[2]
}
if len(localFile) == 0 || len(remotePath) == 0 {
println("Invalid input")
os.Exit(1)
}
time.Sleep(2)
// [INFO] github.com/qiniu/api.v7/storage/resume_upload.go:251: resumable.Put retrying ... 2 reason: token expired 2 seconds ago
err, url := UploadToQiNiu(localFile, remotePath)
if err != nil {
fmt.Printf("Upload %s failed: %s\n", localFile, err)
}
time.Sleep(10)
// [INFO] github.com/qiniu/api.v7/storage/resume_upload.go:251: resumable.Put retrying ... 2 reason: token expired 32 seconds ago
err, url = UploadToQiNiu(localFile, remotePath)
if err != nil {
fmt.Printf("Upload %s failed: %s\n", localFile, err)
}
time.Sleep(10)
err, url = UploadToQiNiu(localFile, remotePath)
if err != nil {
fmt.Printf("Upload %s failed: %s\n", localFile, err)
}
fmt.Println(url)
}
输出是:
❯❯❯ http_proxy=http://localhost:8888 go run upload.go /tmp/some-file.data some-file.data
============ first upload
2018/05/08 22:36:03.010548 [INFO] github.com/qiniu/api.v7/storage/resume_upload.go:251: resumable.Put retrying ... 2 reason: token expired 5 seconds ago
2018/05/08 22:36:04.296303 [INFO] github.com/qiniu/api.v7/storage/resume_upload.go:251: resumable.Put retrying ... 1 reason: token expired 7 seconds ago
2018/05/08 22:36:10.197309 [INFO] github.com/qiniu/api.v7/storage/resume_upload.go:251: resumable.Put retrying ... 1 reason: token expired 13 seconds ago
2018/05/08 22:36:10.424459 [INFO] github.com/qiniu/api.v7/storage/resume_upload.go:251: resumable.Put retrying ... 2 reason: token expired 13 seconds ago
2018/05/08 22:36:14.926204 [INFO] github.com/qiniu/api.v7/storage/resume_upload.go:251: resumable.Put retrying ... 0 reason: token expired 17 seconds ago
2018/05/08 22:36:18.863737 [WARN] github.com/qiniu/api.v7/storage/resume_upload.go:254: resumable.Put 1 failed: token expired 21 seconds ago
2018/05/08 22:36:19.186937 [WARN] github.com/qiniu/api.v7/storage/resume_upload.go:254: resumable.Put 2 failed: token expired 22 seconds ago
2018/05/08 22:36:21.925414 [INFO] github.com/qiniu/api.v7/storage/resume_upload.go:251: resumable.Put retrying ... 0 reason: token expired 24 seconds ago
2018/05/08 22:36:24.542282 [WARN] github.com/qiniu/api.v7/storage/resume_upload.go:254: resumable.Put 0 failed: token expired 27 seconds ago
Upload /tmp/some-file.data failed: resumable put failed
============ second upload
2018/05/08 22:36:32.037926 [INFO] github.com/qiniu/api.v7/storage/resume_upload.go:251: resumable.Put retrying ... 2 reason: token expired 35 seconds ago
2018/05/08 22:36:33.509196 [INFO] github.com/qiniu/api.v7/storage/resume_upload.go:251: resumable.Put retrying ... 0 reason: token expired 36 seconds ago
2018/05/08 22:36:36.557253 [INFO] github.com/qiniu/api.v7/storage/resume_upload.go:251: resumable.Put retrying ... 1 reason: token expired 39 seconds ago
2018/05/08 22:36:40.112608 [INFO] github.com/qiniu/api.v7/storage/resume_upload.go:251: resumable.Put retrying ... 2 reason: token expired 43 seconds ago
2018/05/08 22:36:43.892976 [INFO] github.com/qiniu/api.v7/storage/resume_upload.go:251: resumable.Put retrying ... 0 reason: token expired 46 seconds ago
2018/05/08 22:36:45.613949 [INFO] github.com/qiniu/api.v7/storage/resume_upload.go:251: resumable.Put retrying ... 1 reason: token expired 48 seconds ago
2018/05/08 22:36:49.932773 [WARN] github.com/qiniu/api.v7/storage/resume_upload.go:254: resumable.Put 2 failed: token expired 52 seconds ago
2018/05/08 22:36:52.516190 [WARN] github.com/qiniu/api.v7/storage/resume_upload.go:254: resumable.Put 0 failed: token expired 55 seconds ago
2018/05/08 22:36:53.386208 [WARN] github.com/qiniu/api.v7/storage/resume_upload.go:254: resumable.Put 1 failed: token expired 56 seconds ago
Upload /tmp/some-file.data failed: resumable put failed
============ third upload
2018/05/08 22:36:59.208120 [INFO] github.com/qiniu/api.v7/storage/resume_upload.go:251: resumable.Put retrying ... 2 reason: token expired 62 seconds ago
2018/05/08 22:37:03.457293 [INFO] github.com/qiniu/api.v7/storage/resume_upload.go:251: resumable.Put retrying ... 2 reason: token expired 66 seconds ago
2018/05/08 22:37:06.115405 [INFO] github.com/qiniu/api.v7/storage/resume_upload.go:251: resumable.Put retrying ... 0 reason: token expired 69 seconds ago
2018/05/08 22:37:09.576919 [INFO] github.com/qiniu/api.v7/storage/resume_upload.go:251: resumable.Put retrying ... 0 reason: token expired 72 seconds ago
2018/05/08 22:37:13.082099 [WARN] github.com/qiniu/api.v7/storage/resume_upload.go:254: resumable.Put 2 failed: token expired 76 seconds ago
2018/05/08 22:37:14.687961 [WARN] github.com/qiniu/api.v7/storage/resume_upload.go:254: resumable.Put 0 failed: token expired 77 seconds ago
2018/05/08 22:37:14.899853 [INFO] github.com/qiniu/api.v7/storage/resume_upload.go:251: resumable.Put retrying ... 1 reason: token expired 77 seconds ago
2018/05/08 22:37:20.281203 [INFO] github.com/qiniu/api.v7/storage/resume_upload.go:251: resumable.Put retrying ... 1 reason: token expired 83 seconds ago
2018/05/08 22:37:22.990254 [WARN] github.com/qiniu/api.v7/storage/resume_upload.go:254: resumable.Put 1 failed: token expired 85 seconds ago
Upload /tmp/some-file.data failed: resumable put failed
注意 second upload 和 third upload 里,token expired xx seconds ago,这个 xx
已经不是第一次的 5 seconds 了,而是 35 和 62。说明服务器收到的还是第一次的 token。
命令中设置了 http_proxy=http://localhost:8888 ,也可以从相应代理工具中看到 Request Header 里的 Authorization 确实没有变过。
Fix 方式是:
https://github.com/qiniu/api.v7/blob/develop/storage/resume_base.go#L73 里,newUptokenClient 里,既然是 new client 了,清空一次复用 client 的 transport 吧,函数第一行加一句:client.Transport = http.DefaultTransport。
目前的实现方式,会导致每次调用 UploadToQiNiu,都会给复用的 client 嵌套一层 Transport,最后 req.Header.Set("Authorization", t.token) 会被重复执行很多遍,而且总是第一次设置的 uptoken 会覆盖后来所有的设置。
kodo.SetMac("", "")
zone := 1 // 您空间(Bucket)所在的区域: 华东 0,华北 1
c := kodo.New(zone, nil) // 用默认配置创建 Client
GOARCH="amd64"
GOBIN="/wwwroot/gopath//pkg"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/wwwroot/gopath/"
GORACE=""
GOROOT="/usr/local/Cellar/go/1.7.4/libexec"
GOTOOLDIR="/usr/local/Cellar/go/1.7.4/libexec/pkg/tool/darwin_amd64"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/fm/29zg12712ql3y0l0j3t9_mkw0000gn/T/go-build332331909=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"
go get github.com/qiniu/api.v7
# github.com/qiniu/api.v7/auth/qbox
qiniu/api.v7/auth/qbox/qbox_auth.go:84: req.URL.Port undefined (type *url.URL has no field or method Port)
qiniu/api.v7/auth/qbox/qbox_auth.go:86: req.URL.Port undefined (type *url.URL has no field or method Port)
运行 v7 sdk 示例
//生成一个上传token
token := c.MakeUptoken(policy)
token总是为空。调试发现
//初始化AK,SK
conf.ACCESS_KEY = "l6jgr7CYCGyXy57yml25TntkQHbo7Vu1t72ZkGOI"
conf.SECRET_KEY = "aQ5EplBfKG1nZ6MQQ4XNi79UJtazM0rtc9T4Apgf"
后
//创建一个Client
c := kodo.New(0, nil)
时ACCESS_KEY和SECRET_KEY都为空。
https://github.com/qiniu/api.v7/blob/develop/kodo/main.go#L71 初始化一个没有数据的mac
,造成后面生成的token
没有AccessKey
➜ src go get -u github.com/qiniu/api.v7
package qiniupkg.com/x/reqid.v7: unrecognized import path "qiniupkg.com/x/reqid.v7" (https fetch: Get https://qiniupkg.com/x/reqid.v7?go-get=1: x509: certificate has expired or is not yet valid)
package qiniupkg.com/x/bytes.v7: unrecognized import path "qiniupkg.com/x/bytes.v7" (https fetch: Get https://qiniupkg.com/x/bytes.v7?go-get=1: x509: certificate has expired or is not yet valid)
package qiniupkg.com/x/log.v7: unrecognized import path "qiniupkg.com/x/log.v7" (https fetch: Get https://qiniupkg.com/x/log.v7?go-get=1: x509: certificate has expired or is not yet valid)
go get 默认是master分支啊,能否尽快修复这个问题?
尚未支持华南机房。
cannot find package "qiniupkg.com/api.v7/kodo" in any of:
/usr/local/Cellar/go/1.7.3/libexec/src/qiniupkg.com/api.v7/kodo (from $GOROOT)
/Users/ch999/Documents/gopath/src/qiniupkg.com/api.v7/kodo (from $GOPATH)
~$ go get qiniupkg.com/api.v7/api
package qiniupkg.com/api.v7/api: unrecognized import path "qiniupkg.com/api.v7/api" (parse https://qiniupkg.com/api.v7/api?go-get=1: no go-import meta tags)
各种包工具发现有版本,就会拉最后一个。但是那都是前年的了。
要么没版本,要么就不时的发布一下。
您好,同样的代码,在半个月前是正常能用的。
在今天(2018-01-06 23:15) 几乎无法保存数据。报错为:
Post http://up.qiniup.com: EOF
今天测试发了约 200 个请求。成功了 27 个。
希望有空能帮忙查看一下。
代码里是并发16个请求的,不知是否有影响。
谢谢。
kodo能否支持分块并行上传,
我们从国外服务器传国内的qiniu好像会比较慢 我想使用并行上传
$ go get qiniupkg.com/api.v7/kodocli
package qiniupkg.com/api.v7/kodocli: cannot find package "qiniupkg.com/api.v7/kodocli" in any of:
$ go get qiniupkg.com/api.v7/kodo
package qiniupkg.com/api.v7/kodo: cannot find package "qiniupkg.com/api.v7/kodo" in any of:
对象下载的操作应该也由 bucket 来承担,不应该直接调用 http,会造成使用理解上的混乱。很多细节性的操作比如range,断点续传类的功能还需要去查 http 的api。可以参考 s3 的 get 相关 api ,设计的还是比较人性化。
创建连接部分的配置需要有些文档说明,至今不明白IoHost,rshost 还有 rsfhost 是干什么的,
beego配合七牛生成token,偶尔出现,fatal error: concurrent map writes
调用代码如下
returnBody := {"key":$(key),"etag":$(etag),"fsize":$(fsize),"mimeType":$(mimeType),"imageInfo_width":$(imageInfo.width),"imageInfo_height":$(imageInfo.height)}
putPolicy := kodo.PutPolicy{
Expires: uint32(expires),
Scope: bucketName,
ReturnBody: returnBody,
MimeLimit: "image/*",
}
return qiniu_client.MakeUptoken(&putPolicy)
错误如下
fatal error: concurrent map writes
goroutine 218 [running]:
runtime.throw(0x885129, 0x15)
/usr/local/go/src/runtime/panic.go:566 +0x95 fp=0xc420276d68 sp=0xc420276d48
runtime.mapassign1(0x7b62c0, 0xc42013f860, 0xc420276ee8, 0xc420276f28)
/usr/local/go/src/runtime/hashmap.go:458 +0x8ef fp=0xc420276e50 sp=0xc420276d68
qiniupkg.com/api.v7/api.(*Client).GetBucketInfo(0xc42013fe00, 0xc42013e900, 0x28, 0xc42000dbe0, 0x6, 0xc4201c5900, 0x3, 0x4, 0xc4203e4740, 0x14, ...)
/Users/alicksnake/go/src/qiniupkg.com/api.v7/api/api.go:57 +0x3a2 fp=0xc420276f68 sp=0xc420276e50
qiniupkg.com/api.v7/kodo.(*Client).GetBucketInfo(0xc4200ff6b0, 0xc42000dbe0, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc4202770f8, ...)
/Users/alicksnake/go/src/qiniupkg.com/api.v7/kodo/token.go:115 +0x98 fp=0xc420277010 sp=0xc420276f68
qiniupkg.com/api.v7/kodo.(*Client).MakeUptokenWithSafe(0xc4200ff6b0, 0xc42041f320, 0x0, 0x3a0000c420472d84, 0x0, 0x0)
/Users/alicksnake/go/src/qiniupkg.com/api.v7/kodo/token.go:92 +0x190 fp=0xc4202770c0 sp=0xc420277010
qiniupkg.com/api.v7/kodo.(*Client).MakeUptoken(0xc4200ff6b0, 0xc42041f320, 0xc4201540f8, 0xc4202e2ea0)
/Users/alicksnake/go/src/qiniupkg.com/api.v7/kodo/token.go:81 +0x3c fp=0xc420277138 sp=0xc4202770c0
childschool/models/utils.Image_Uptoken_NO_CallBack(0xc42000dbe0, 0x6, 0x278d00, 0xc42000dbe0, 0x6)
/Users/alicksnake/go/src/childschool/models/utils/util_qiniu.go:54 +0xfd fp=0xc420277290 sp=0xc420277138
childschool/controllers/suiyin/app/api/v1.(*QiniuController).GetUpToken(0xc4200cca50)
/Users/alicksnake/go/src/childschool/controllers/suiyin/app/api/v1/qiniu.go:64 +0x19b fp=0xc420277380 sp=0xc420277290
runtime.call32(0xc420428fc0, 0xc4203de8d8, 0xc4203de8e0, 0x800000008)
/usr/local/go/src/runtime/asm_amd64.s:479 +0x4c fp=0xc4202773b0 sp=0xc420277380
reflect.Value.call(0x85a820, 0xc4200cca50, 0x6a13, 0x87204d, 0x4, 0x0, 0x0, 0x0, 0xc4203de8d0, 0x13, ...)
/usr/local/go/src/reflect/value.go:434 +0x5c8 fp=0xc420277700 sp=0xc4202773b0
reflect.Value.Call(0x85a820, 0xc4200cca50, 0x6a13, 0x0, 0x0, 0x0, 0xc4200cca50, 0x6a13, 0xc42003b848)
/usr/local/go/src/reflect/value.go:302 +0xa4 fp=0xc420277768 sp=0xc420277700
github.com/astaxie/beego.(*ControllerRegister).ServeHTTP(0xc4200937c0, 0xca52a0, 0xc420071e10, 0xc4201b63c0)
/Users/alicksnake/go/src/github.com/astaxie/beego/router.go:785 +0x1eb1 fp=0xc420277cd0 sp=0xc420277768
net/http.serverHandler.ServeHTTP(0xc42008a300, 0xca52a0, 0xc420071e10, 0xc4201b63c0)
/usr/local/go/src/net/http/server.go:2202 +0x7d fp=0xc420277d18 sp=0xc420277cd0
net/http.(*conn).serve(0xc42030e480, 0xca5ca0, 0xc42044abc0)
/usr/local/go/src/net/http/server.go:1579 +0x4b7 fp=0xc420277f78 sp=0xc420277d18
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc420277f80 sp=0xc420277f78
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2293 +0x44d
goroutine 1 [chan receive, 8 minutes]:
github.com/astaxie/beego.(*App).Run(0xc42000d8f0)
/Users/alicksnake/go/src/github.com/astaxie/beego/app.go:190 +0x6fd
github.com/astaxie/beego.Run(0x0, 0x0, 0x0)
/Users/alicksnake/go/src/github.com/astaxie/beego/beego.go:67 +0x51
main.main()
/Users/alicksnake/go/src/childschool/main.go:241 +0x4a0
goroutine 17 [syscall, 8 minutes, locked to thread]:
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2086 +0x1
goroutine 5 [syscall, 8 minutes]:
os/signal.signal_recv(0x0)
/usr/local/go/src/runtime/sigqueue.go:116 +0x157
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.1
/usr/local/go/src/os/signal/signal_unix.go:28 +0x41
goroutine 6 [chan receive, 8 minutes]:
database/sql.(*DB).connectionOpener(0xc4200ff550)
/usr/local/go/src/database/sql/sql.go:730 +0x4a
created by database/sql.Open
/usr/local/go/src/database/sql/sql.go:493 +0x1e9
goroutine 247 [select]:
net/http.(*persistConn).writeLoop(0xc42006c100)
/usr/local/go/src/net/http/transport.go:1641 +0x3bd
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1058 +0x50e
goroutine 36 [select]:
childschool/models/suiyin.(*Timmer).InitTimer.func1()
/Users/alicksnake/go/src/childschool/models/suiyin/timer.go:53 +0x1f4
created by childschool/models/suiyin.(*Timmer).InitTimer
/Users/alicksnake/go/src/childschool/models/suiyin/timer.go:72 +0x65
goroutine 37 [IO wait]:
net.runtime_pollWait(0x1108da8, 0x72, 0x0)
/usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc420460d80, 0x72, 0xc420468c70, 0xc42000c078)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc420460d80, 0xca13a0, 0xc42000c078)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).accept(0xc420460d20, 0x0, 0xc9f3a0, 0xc420288c80)
/usr/local/go/src/net/fd_unix.go:419 +0x238
net.(*TCPListener).accept(0xc420324028, 0x29e8d60800, 0x0, 0x0)
/usr/local/go/src/net/tcpsock_posix.go:132 +0x2e
net.(*TCPListener).AcceptTCP(0xc420324028, 0xc420468d98, 0xc420468da0, 0xc420468d90)
/usr/local/go/src/net/tcpsock.go:209 +0x49
net/http.tcpKeepAliveListener.Accept(0xc420324028, 0x933bd8, 0xc42030e480, 0xca5d60, 0xc420462270)
/usr/local/go/src/net/http/server.go:2608 +0x2f
net/http.(*Server).Serve(0xc42008a300, 0xca5620, 0xc420324028, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2273 +0x1ce
net/http.(*Server).ListenAndServe(0xc42008a300, 0xc420456620, 0x0)
/usr/local/go/src/net/http/server.go:2219 +0xb4
github.com/astaxie/beego.(*App).Run.func4(0xc42000d8f0, 0xc4204565e0, 0x5, 0xc420460cb0)
/Users/alicksnake/go/src/github.com/astaxie/beego/app.go:182 +0x4aa
created by github.com/astaxie/beego.(*App).Run
/Users/alicksnake/go/src/github.com/astaxie/beego/app.go:188 +0x748
goroutine 193 [IO wait]:
net.runtime_pollWait(0x1108e68, 0x72, 0xf6)
/usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc42006fd40, 0x72, 0xc420189790, 0xc42000c078)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc42006fd40, 0xca13a0, 0xc42000c078)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).Read(0xc42006fce0, 0xc42023a000, 0x1000, 0x1000, 0x0, 0xca13a0, 0xc42000c078)
/usr/local/go/src/net/fd_unix.go:243 +0x1a1
net.(*conn).Read(0xc42002c620, 0xc42023a000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:173 +0x70
net/http.(*connReader).Read(0xc4203dd6c0, 0xc42023a000, 0x1000, 0x1000, 0x1acd69, 0xcc29a0, 0x0)
/usr/local/go/src/net/http/server.go:586 +0x144
bufio.(*Reader).fill(0xc420295560)
/usr/local/go/src/bufio/bufio.go:97 +0x10c
bufio.(*Reader).ReadSlice(0xc420295560, 0xa, 0x0, 0x1e, 0x0, 0x33, 0x0)
/usr/local/go/src/bufio/bufio.go:330 +0xb5
bufio.(*Reader).ReadLine(0xc420295560, 0xc4201b61e0, 0xf0, 0xf0, 0x8471c0, 0x29f1c3, 0xccfbe8)
/usr/local/go/src/bufio/bufio.go:359 +0x37
net/textproto.(*Reader).readLineSlice(0xc4203a4540, 0xc420189a88, 0xc420189a88, 0x17df8, 0xf0, 0x8471c0)
/usr/local/go/src/net/textproto/reader.go:55 +0x5e
net/textproto.(*Reader).ReadLine(0xc4203a4540, 0xc4201b61e0, 0xc, 0x0, 0x31f4c)
/usr/local/go/src/net/textproto/reader.go:36 +0x2f
net/http.readRequest(0xc420295560, 0xc42023b000, 0xc4201b61e0, 0x0, 0x0)
/usr/local/go/src/net/http/request.go:793 +0xa5
net/http.(*conn).readRequest(0xc420367680, 0xca5ca0, 0xc4201c53c0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:765 +0x10d
net/http.(*conn).serve(0xc420367680, 0xca5ca0, 0xc4201c53c0)
/usr/local/go/src/net/http/server.go:1532 +0x3d3
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2293 +0x44d
goroutine 246 [IO wait]:
net.runtime_pollWait(0x1108028, 0x72, 0x108)
/usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc420180a00, 0x72, 0xc42003e9d0, 0xc42000c078)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc420180a00, 0xca13a0, 0xc42000c078)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).Read(0xc4201809a0, 0xc420379000, 0x1000, 0x1000, 0x0, 0xca13a0, 0xc42000c078)
/usr/local/go/src/net/fd_unix.go:243 +0x1a1
net.(*conn).Read(0xc4203de930, 0xc420379000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:173 +0x70
net/http.(*persistConn).Read(0xc42006c100, 0xc420379000, 0x1000, 0x1000, 0x1d5af0, 0xc42003eb58, 0xba1d)
/usr/local/go/src/net/http/transport.go:1256 +0x154
bufio.(*Reader).fill(0xc420255e60)
/usr/local/go/src/bufio/bufio.go:97 +0x10c
bufio.(*Reader).Peek(0xc420255e60, 0x1, 0xc42003ebbd, 0x1, 0x0, 0xc420255ec0, 0x0)
/usr/local/go/src/bufio/bufio.go:129 +0x62
net/http.(*persistConn).readLoop(0xc42006c100)
/usr/local/go/src/net/http/transport.go:1413 +0x1a1
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1057 +0x4e9
goroutine 199 [runnable]:
bytes.(*Buffer).WriteString(0xc420470a50, 0xc4202890a0, 0x17, 0x0, 0x0, 0x0)
/usr/local/go/src/bytes/buffer.go:141
encoding/json.(*encodeState).string(0xc420470a50, 0xc4202890a0, 0x17, 0xc420289001, 0x17)
/usr/local/go/src/encoding/json/encode.go:918 +0x4df
encoding/json.stringEncoder(0xc420470a50, 0x788c20, 0xc42044ad90, 0x198, 0x780100)
/usr/local/go/src/encoding/json/encode.go:564 +0x228
encoding/json.(*arrayEncoder).encode(0xc4203de0c8, 0xc420470a50, 0x781680, 0xc4203e6340, 0x197, 0x100)
/usr/local/go/src/encoding/json/encode.go:723 +0xee
encoding/json.(*arrayEncoder).(encoding/json.encode)-fm(0xc420470a50, 0x781680, 0xc4203e6340, 0x197, 0x750100)
/usr/local/go/src/encoding/json/encode.go:730 +0x64
encoding/json.(*sliceEncoder).encode(0xc4203de0d0, 0xc420470a50, 0x781680, 0xc4203e6340, 0x197, 0x100)
/usr/local/go/src/encoding/json/encode.go:697 +0xc1
encoding/json.(*sliceEncoder).(encoding/json.encode)-fm(0xc420470a50, 0x781680, 0xc4203e6340, 0x197, 0x100)
/usr/local/go/src/encoding/json/encode.go:709 +0x64
encoding/json.(*structEncoder).encode(0xc42041c480, 0xc420470a50, 0x848560, 0xc4203e6240, 0x199, 0x100)
/usr/local/go/src/encoding/json/encode.go:601 +0x253
encoding/json.(*structEncoder).(encoding/json.encode)-fm(0xc420470a50, 0x848560, 0xc4203e6240, 0x199, 0xc4203e0100)
/usr/local/go/src/encoding/json/encode.go:615 +0x64
encoding/json.(*ptrEncoder).encode(0xc4203de0e0, 0xc420470a50, 0x7761c0, 0xc4203e6240, 0x16, 0x770100)
/usr/local/go/src/encoding/json/encode.go:742 +0xe3
encoding/json.(*ptrEncoder).(encoding/json.encode)-fm(0xc420470a50, 0x7761c0, 0xc4203e6240, 0x16, 0xc4203e0100)
/usr/local/go/src/encoding/json/encode.go:747 +0x64
encoding/json.(*encodeState).reflectValue(0xc420470a50, 0x7761c0, 0xc4203e6240, 0x16, 0x100)
/usr/local/go/src/encoding/json/encode.go:307 +0x82
encoding/json.(*encodeState).marshal(0xc420470a50, 0x7761c0, 0xc4203e6240, 0x60100, 0x0, 0x0)
/usr/local/go/src/encoding/json/encode.go:280 +0xb8
encoding/json.Marshal(0x7761c0, 0xc4203e6240, 0xcd0540, 0xc42044ad80, 0x3, 0x4, 0xc420289000)
/usr/local/go/src/encoding/json/encode.go:145 +0x8f
qiniupkg.com/api.v7/kodo.(*Client).MakeUptokenWithSafe(0xc4200ff6b0, 0xc4203e6120, 0x0, 0x3a0000c4203bb7c4, 0x0, 0x0)
/Users/alicksnake/go/src/qiniupkg.com/api.v7/kodo/token.go:103 +0xe5
qiniupkg.com/api.v7/kodo.(*Client).MakeUptoken(0xc4200ff6b0, 0xc4203e6120, 0xc4201540f8, 0xc420482ea0)
/Users/alicksnake/go/src/qiniupkg.com/api.v7/kodo/token.go:81 +0x3c
childschool/models/utils.Image_Uptoken_NO_CallBack(0xc42000dbe0, 0x6, 0x278d00, 0xc42000dbe0, 0x6)
/Users/alicksnake/go/src/childschool/models/utils/util_qiniu.go:54 +0xfd
childschool/controllers/suiyin/app/api/v1.(*QiniuController).GetUpToken(0xc4201b6000)
/Users/alicksnake/go/src/childschool/controllers/suiyin/app/api/v1/qiniu.go:64 +0x19b
reflect.Value.call(0x85a820, 0xc4201b6000, 0x6a13, 0x87204d, 0x4, 0x0, 0x0, 0x0, 0xc420324038, 0x13, ...)
/usr/local/go/src/reflect/value.go:434 +0x5c8
reflect.Value.Call(0x85a820, 0xc4201b6000, 0x6a13, 0x0, 0x0, 0x0, 0xc4201b6000, 0x6a13, 0xc420185828)
/usr/local/go/src/reflect/value.go:302 +0xa4
github.com/astaxie/beego.(*ControllerRegister).ServeHTTP(0xc4200937c0, 0xca52a0, 0xc420341a00, 0xc420218960)
/Users/alicksnake/go/src/github.com/astaxie/beego/router.go:785 +0x1eb1
net/http.serverHandler.ServeHTTP(0xc42008a300, 0xca52a0, 0xc420341a00, 0xc420218960)
/usr/local/go/src/net/http/server.go:2202 +0x7d
net/http.(*conn).serve(0xc42008a800, 0xca5ca0, 0xc4203356c0)
/usr/local/go/src/net/http/server.go:1579 +0x4b7
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2293 +0x44d
goroutine 234 [IO wait]:
net.runtime_pollWait(0x14018d0, 0x72, 0x10a)
/usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc4204611e0, 0x72, 0xc4204649d0, 0xc42000c078)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc4204611e0, 0xca13a0, 0xc42000c078)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).Read(0xc420461180, 0xc420306000, 0x1000, 0x1000, 0x0, 0xca13a0, 0xc42000c078)
/usr/local/go/src/net/fd_unix.go:243 +0x1a1
net.(*conn).Read(0xc42002c710, 0xc420306000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:173 +0x70
net/http.(*persistConn).Read(0xc420216100, 0xc420306000, 0x1000, 0x1000, 0x1d5af0, 0xc420464b58, 0xba1d)
/usr/local/go/src/net/http/transport.go:1256 +0x154
bufio.(*Reader).fill(0xc420268a80)
/usr/local/go/src/bufio/bufio.go:97 +0x10c
bufio.(*Reader).Peek(0xc420268a80, 0x1, 0xc420464bbd, 0x1, 0x0, 0xc420268ae0, 0x0)
/usr/local/go/src/bufio/bufio.go:129 +0x62
net/http.(*persistConn).readLoop(0xc420216100)
/usr/local/go/src/net/http/transport.go:1413 +0x1a1
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1057 +0x4e9
goroutine 235 [select]:
net/http.(*persistConn).writeLoop(0xc420216100)
/usr/local/go/src/net/http/transport.go:1641 +0x3bd
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1058 +0x50e
goroutine 204 [IO wait]:
net.runtime_pollWait(0x1108328, 0x72, 0x104)
/usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc4202b8bc0, 0x72, 0xc42027b790, 0xc42000c078)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc4202b8bc0, 0xca13a0, 0xc42000c078)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).Read(0xc4202b8b60, 0xc420246000, 0x1000, 0x1000, 0x0, 0xca13a0, 0xc42000c078)
/usr/local/go/src/net/fd_unix.go:243 +0x1a1
net.(*conn).Read(0xc4201de148, 0xc420246000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:173 +0x70
net/http.(*connReader).Read(0xc420273f20, 0xc420246000, 0x1000, 0x1000, 0x1acd69, 0xcc29a0, 0x0)
/usr/local/go/src/net/http/server.go:586 +0x144
bufio.(*Reader).fill(0xc420294120)
/usr/local/go/src/bufio/bufio.go:97 +0x10c
bufio.(*Reader).ReadSlice(0xc420294120, 0xa, 0x0, 0x1e, 0x6, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:330 +0xb5
bufio.(*Reader).ReadLine(0xc420294120, 0xc4201b62d0, 0xf0, 0xf0, 0x8471c0, 0xc42041a580, 0x830880)
/usr/local/go/src/bufio/bufio.go:359 +0x37
net/textproto.(*Reader).readLineSlice(0xc42044ec00, 0xc42027ba88, 0xc42027ba88, 0x17df8, 0xf0, 0x8471c0)
/usr/local/go/src/net/textproto/reader.go:55 +0x5e
net/textproto.(*Reader).ReadLine(0xc42044ec00, 0xc4201b62d0, 0xc, 0x0, 0x31f4c)
/usr/local/go/src/net/textproto/reader.go:36 +0x2f
net/http.readRequest(0xc420294120, 0xc42019f000, 0xc4201b62d0, 0x0, 0x0)
/usr/local/go/src/net/http/request.go:793 +0xa5
net/http.(*conn).readRequest(0xc42008ae00, 0xca5ca0, 0xc4202c6100, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:765 +0x10d
net/http.(*conn).serve(0xc42008ae00, 0xca5ca0, 0xc4202c6100)
/usr/local/go/src/net/http/server.go:1532 +0x3d3
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2293 +0x44d
Add PutExtra like this:
func PutFile(bucket, key, filePath string, uid string) error {
token := PutToken(bucket, key, uint32(120))
putExtra := storage.PutExtra{
Params: map[string]string{
"x:uid": uid,
},
}
formUploader := storage.NewFormUploader(cfg)
ret := storage.PutRet{}
return formUploader.PutFile(context.Background(), &ret, token, key, filePath, &putExtra)
}
I just try:
func GetFileInfo(bucket, key string) (storage.FileInfo, error) {
bucketManager := storage.NewBucketManager(mac, cfg)
fileInfo, err := bucketManager.Stat(bucket, key)
return fileInfo, err
}
but the return info of fileInfo is:
storage.FileInfo{Hash:"Fm5gOFQjePErFAvE7g4CjSJUQiEZ", Fsize:545715, PutTime:15302384628621008, MimeType:"image/png", Type:0}
How can I get the x:uid info?
上传文件时如果使用 zones[1] 即 http://up-z1.qiniu.com
会得到错误信息 incorrect zone, please use up-z0.qiniu.com
package qiniupkg.com/x/url.v7: unrecognized import path "qiniupkg.com/x/url.v7" (https fetch: Get https://qiniupkg.com/x/url.v7?go-get=1: dial tcp: lookup qiniupkg.com on 10.143.22.116:53: no such host)
解决一下呗
我在本地也dns 查找不到
PutFile上传图片时,间歇性出错:
安装官方文档实现分片上传,总是报错:
too many data to read, block capacity is 2048 bytes, 2048 byte(s) used
官方能不能给一个完整的分片上传的实例?
V7有没有类似于qshell domains bucket的功能?
qshell使用了V6来实现, 以后会使用v7来实现吗?
不然 storage.MakePublicURL("http://a.b.c/", "hi.jpg")
会返回 http://a.b.c//hi.jpg
,同时影响 storage.MakePrivateURL
签名的计算,导致鉴权错误
编译通不过么?
uploader
的 Rput
函数文档里写着 key
可以设定访问路径。
以下为我的代码:
ACCESSKEY
和 SECRETKEY
皆已设置。
但在我使用,如果上传的文件 key
加入了前缀 rikka/
,则无法上传成功:
上图里有很多我自己项目的 Log,所以可能有些难看清。但是注意第一个红框,key
是 rikka/<uuid>
,然后二三两个框里就输出了上传重试/失败的消息。
如果 key
之前不加前缀 rikka/
则可上传成功:
可以看到上传成功了,并且尝试访问最后的文件路径也能得到上传的文件。
所以我想问如果想要上传的时候设置路径,到底应该怎么使用 key
这个参数呢?难道需要 escape 么?
谢谢。
请问下 我用go sdk包生成上传uptoken 在客户端用js-sdk凭借uptoken上传资源到七牛服务器 ,在哪里配置我的 两个KEY呢?还有bucketName?文档貌似没写清楚?
用linux命令:
echo "/v2/tune/refresh" |openssl dgst -binary -hmac "用户SK" -sha1 |base64 | tr + - | tr / _
和用包的方法
mac := qbox.NewMac("AK", "SK")
sign := mac.Sign([]byte(data))
两个的生成出来的token不同,现在只能用第一种方法生成出来的token,第二种生成出来使用显示bad token.
i use the command "go get -u qiniupkg.com/api.v7", but throw a error msg package qiniupkg.com/api.v7: unrecognized import path "qiniupkg.com/api.v7".
What should i do?
I propose exposing bput
、mkblk
and mkfile
interface to coder to bring them more flexibility.
Though the possibility of being called is small, it will be convenient for a coder to have the ability of calling them separately according to the situation. Since I am writing a storage driver based on qiniu for docker/distribution, so I do think it not a good idea to implement them again by myself due to the lack of interface above.
If needed, I can submit the pull requests.
unrecognized import path "qiniupkg.com/x/ctype.v7"
maintenance or migrated?
在rput函数中,task指向的函数中使用的的变量,tasks队列中存在覆盖问题。特别是blkSize1,当上传的文件最后一个block小于4m的时候,tasks队列里如果还存在数据,会导致所有的tasks里的函数调用传递的blksize被覆盖。
package main
import (
"qiniupkg.com/api.v7/kodo"
"golang.org/x/net/context"
"fmt"
)
func main() {
kodo.SetMac("accesskey", "secretkey")
zone := 0
c := kodo.New(zone, nil)
bucket := c.Bucket("oilbeater")
ctx := context.Background()
err := bucket.RputFile(ctx, nil, "hello.go", "main.go", nil)
fmt.Println(err.Error())
}
输出
scope not specified
这是预期的结果还是我使用的姿势有问题
有关于持久化处理(pfop)的api么?
睇了一下文档,好像没有发现持久化处理(pfop)的api。还是我没有看清楚文档。请指教。
要直接写按照文档写http连接调用么?怎么go的api没有这个现成调用的。
这个主要是url.QueryEscape把空格变成了 + 号,这个不对,应该是 %20
具体位置在这里
rpc.v7/rpc_clieng.go:157 : PrettyPrintRequest(req)
应该去掉。。。
../qiniupkg.com/api.v7/kodo/bucket.go:303: syntax error: unexpected URIDeleteAfterDays, expecting (
go get的时候还要多下14m的文件,挺欺负小水管的
类似于formUploader.PutWithoutKey
在 bucketManager.Fetch 中自定义返回的数据,比如 imageInfo 和 exif 信息
有什么办法让他及时更新显示最新的图片吗?(浏览器无痕模式下)
I use this code to uploader file.but I can't get a usable token.
zone := 0
client := kodo.New(0, nil)
policy := &kodo.PutPolicy{Scope: bucket, Expires: 3600}
token := client.MakeUptoken(policy) //this is nil
fmt.Println("token:", token)
uploader := kodocli.NewUploader(zone, nil)
var ret PutRet
filePath := "D:\\111.doc"
res := uploader.PutFileWithoutKey(nil, &ret, token, filePath, nil)
fmt.Print(ret)
if res != nil {
fmt.Println("io.Put failed:", res)
} else {
fmt.Println("io.Put success.")
}
who can help me? Thanks.
err := Bucket("mine").Put(context.Background(), nil, "", bytes.NewBufferString(req.File), req.Size, nil)
if err != nil {
fmt.Println(err)
}
// Got:
/*
Post http://up.qiniu.com: http: ContentLength=32963 with Body length 43791
*/
能否像node的sdk一样自动添加一个地区选择的功能
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.