cloudmindsrobot / dagger Goto Github PK
View Code? Open in Web Editor NEWDagger 是一个基于 Loki 的日志查询和管理系统,它是由达闼科技( CloudMinds )云团队的`大禹基础设施平台`派生出来的一个项目。Dagger 运行在 Loki 前端,具备日志查询、搜索,保存和下载等特性,适用于云原生场景下的容器日志管理场景。
License: Apache License 2.0
Dagger 是一个基于 Loki 的日志查询和管理系统,它是由达闼科技( CloudMinds )云团队的`大禹基础设施平台`派生出来的一个项目。Dagger 运行在 Loki 前端,具备日志查询、搜索,保存和下载等特性,适用于云原生场景下的容器日志管理场景。
License: Apache License 2.0
详细日志:
2021-01-13T04:12:09Z [INFO] "GET /api/v1/loki/snapshot?page=1&page_size=10 HTTP/1.0 200 952.044µs "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36" "
2021-01-13T04:12:12Z [INFO] "GET /api/v1/loki/history?page=1&page_size=10 HTTP/1.0 200 863.445µs "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36" "
2021-01-13T04:12:14Z [INFO] "GET /api/v1/loki/labels HTTP/1.0 200 41.342206ms "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36" "
2021-01-13T04:12:14Z [INFO] "GET /api/v1/loki/labels?start=1610510833000000000&end=1610511133000000000 HTTP/1.0 200 42.362902ms "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36" "
2021-01-13T04:12:24Z [INFO] "GET /swagger/index.html HTTP/1.0 200 163.233µs "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36" "
2021-01-13T04:12:24Z [INFO] "GET /swagger/doc.json HTTP/1.0 200 1.197638ms "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36" "
2021-01-13T04:13:05.449Z [ERROR] controllers/query.go:555 send message to viewer error, write tcp 172.27.0.2:8000->172.27.0.3:60410: write: broken pipe
dagger/backend/controllers.LokiTail
/go/src/dagger/backend/controllers/query.go:555
github.com/gin-gonic/gin.(*Context).Next
/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:161
github.com/gin-gonic/gin.RecoveryWithWriter.func1
/go/pkg/mod/github.com/gin-gonic/[email protected]/recovery.go:83
github.com/gin-gonic/gin.(*Context).Next
/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:161
github.com/gin-gonic/gin.LoggerWithConfig.func1
/go/pkg/mod/github.com/gin-gonic/[email protected]/logger.go:241
github.com/gin-gonic/gin.(*Context).Next
/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:161
github.com/gin-contrib/sessions.Sessions.func1
/go/pkg/mod/github.com/gin-contrib/[email protected]/sessions.go:52
github.com/gin-gonic/gin.(*Context).Next
/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:161
github.com/gin-gonic/gin.(*Engine).handleHTTPRequest
/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:409
github.com/gin-gonic/gin.(*Engine).ServeHTTP
/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:367
net/http.serverHandler.ServeHTTP
/usr/local/go/src/net/http/server.go:2802
net/http.(*conn).serve
/usr/local/go/src/net/http/server.go:1890
2021-01-13T04:13:05Z [INFO] "GET /ws/tail?namespace_service=a-prd-whzb/omni-adapter&start=1610511022000000000&level=&pod=&filters= HTTP/1.1 200 2m41.627422872s "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36" "
panic: send on closed channel
goroutine 805 [running]:
dagger/backend/utils.WebSocketServerHandler(0xc0001f3ce0, 0xc0003c0c00, 0xc0003c0c60)
/go/src/dagger/backend/utils/lokiwebsocket.go:97 +0x16c
created by dagger/backend/controllers.LokiTail
/go/src/dagger/backend/controllers/query.go:547 +0xc5b
①dagger-mysql 容器无法启动
解决办法: 在docker服务器上,useradd -u 999 mysql, chonw -R mysql /usr/local/src/dagger/dagger/production/config
②dagger-backend 容器无法与loki服务交互
解决办法:修改 docker-compose.yaml dagger-backend 容器内挂载的 ../backend/dagger.ini 修改为./config/dagger.ini
支持马总大佬!
正常登陆后,想在登出,没有作用,还是一样能够查询看到相应的界面
Is this a BUG REPORT or FEATURE REQUEST?:
Uncomment only one, leave it on its own line:
/kind bug
/kind feature
What happened:
在查询时额外添加了k8s_pod_name标签(而且还无法取消),但我的loki中并没有这个标签,导致所有查询都为空。
What you expected to happen:
删除这个自主添加的标签,或者可以取消掉。
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
请问增加这个的意义是什么?
dagger/backend/controllers/query.go
Line 548 in 7a63f5e
grafana通过配置多个loki数据源来查询日志,但是由于不同的loki数据源中日志label并不一样,这也导致在grafana上查询日志还是具备一定的门槛。
dagger的目的是尽量简化日志查询过程中的条件选择,引入多个loki数据源可能会打破这样的初衷。
我这里另外一个解决方案,看各位有啥好的想法:
方案一、一套Loki集群收集多个集群日志,然后跟dagger对接
优势:
劣势:
方案二、dagger支持多个loki数据源
优势:
劣势:
有其他的想法,欢迎在下方留言讨论
Is this a BUG REPORT or FEATURE REQUEST?:
Uncomment only one, leave it on its own line:
/kind bug
/kind feature
What happened:
What you expected to happen:
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
请问这个项目还在维护吗? 看好久没更新了,包括 #17 看是之前说要有更新修复,貌似没有下文了,感觉挺好的一个项目,希望能继续下去,加油!
loki有按标签进行日志量(存储大小)统计接口,你们这边有实现吗?
另外基于日志使用量告警有没有什么方案?
Is this a BUG REPORT or FEATURE REQUEST?:
Uncomment only one, leave it on its own line:
kind bug
/kind feature
What happened: 登录后报红色背景的通知: Error 获取标签数据失败 和 Error: 获取数据失败 两个错误,同时也有蓝绿背景的 Success: 登录成功
What you expected to happen: 展现出 logs
How to reproduce it (as minimally and precisely as possible):
正常安装,pods 正常,pvc 也正常 bound,登录后出现异常
Anything else we need to know?:
Is this a BUG REPORT or FEATURE REQUEST?:
yes
Uncomment only one, leave it on its own line:
/kind bug
2020-11-04 18:33:34 | 2020-11-04T10:33:34.323Z [INFO] controllers/query.go:102 query expr: {app="xxxxappxxxx",namespace="dev1"} \|~ "200"
2020-11-04 18:33:34 | 2020-11-04T10:33:34.323Z [INFO] utils/lokiapi.go:16 loki api query url: http://loki-headless.monitoring:3100/loki/api/v1/query_range?query=%7Bapp%3D%22xxxxappxxxx%22%2Cnamespace%3D%22dev1%22%7D+%7C~+%22200%22&start=1604485500000000000&end=1604485800000000000&limit=500&direction=forward
2020-11-04 18:33:34 | 2020-11-04T10:33:34Z [INFO] "GET /api/v1/loki/query_range?namespace=dev1&app=xxxxxapppxxx&start=1604485500000000000&end=1604485800000000000&level=&limit=500&size=20&middleStart=&middleEnd=&pod=&all=true&dsc=false&filters[]=200 HTTP/1.0 500 141.573176ms "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4310.0 Safari/537.36" "
2020-11-04 18:33:34 |
2020-11-04 18:33:34 |
2020-11-04 18:33:34 | 2020/11/04 10:33:34 [Recovery] 2020/11/04 - 10:33:34 panic recovered:
2020-11-04 18:33:34 | runtime error: index out of range [-289]
2020-11-04 18:33:34 | /usr/local/go/src/runtime/panic.go:75 (0x438932)
2020-11-04 18:33:34 | /go/src/dagger/backend/gin/controllers/query.go:178 (0xa5b8c2)
2020-11-04 18:33:34 | /go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:161 (0xa3910a)
2020-11-04 18:33:34 | /go/src/dagger/backend/gin/middlewares/middlewares.go:72 (0xa64a71)
2020-11-04 18:33:34 | /go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:161 (0xa3910a)
2020-11-04 18:33:34 | /go/pkg/mod/github.com/gin-gonic/[email protected]/recovery.go:83 (0xa4cc03)
2020-11-04 18:33:34 | /go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:161 (0xa3910a)
2020-11-04 18:33:34 | /go/pkg/mod/github.com/gin-gonic/[email protected]/logger.go:241 (0xa4bd30)
2020-11-04 18:33:34 | /go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:161 (0xa3910a)
2020-11-04 18:33:34 | /go/pkg/mod/github.com/gin-contrib/[email protected]/sessions.go:52 (0xa4f2fd)
2020-11-04 18:33:34 | /go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:161 (0xa3910a)
2020-11-04 18:33:34 | /go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:409 (0xa4304c)
2020-11-04 18:33:34 | /go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:367 (0xa4274d)
2020-11-04 18:33:34 | /usr/local/go/src/net/http/server.go:2802 (0x81b003)
2020-11-04 18:33:34 | /usr/local/go/src/net/http/server.go:1890 (0x8168a4)
2020-11-04 18:33:34 | /usr/local/go/src/runtime/asm_amd64.s:1357 (0x467610)
2020-11-04 18:33:34
选择级别和对应指定 Pod 无效,返回 200 状态码;
backend 无日志,response 返回 200
点击下载日志仍然失败
2020-11-04 18:36:59 | 2020-11-04T10:36:59.484Z [INFO] controllers/query.go:303 download expr: {app="xxxxapppxaxx",namespace="dev1"}
2020-11-04 18:36:59 | 2020-11-04T10:36:59.484Z [INFO] utils/lokiapi.go:16 loki api query url: http://loki-headless.monitoring:3100/loki/api/v1/query_range?query=%7Bapp%3D%22xxxxapppxaxx%22%2Cnamespace%3D%22dev1%22%7D&start=1604485500000000000&end=1604485800000000000&limit=5000&direction=backward
2020-11-04 18:36:59 | 2020-11-04T10:36:59.639Z [INFO] utils/lokiapi.go:16 loki api query url: http://loki-headless.monitoring:3100/loki/api/v1/query_range?query=%7Bapp%3D%22xxxxapppxaxx%22%2Cnamespace%3D%22dev1%22%7D&start=1604485500000000000&end=1604485092838469523&limit=5000&direction=backward
2020-11-04 18:37:00 |
2020-11-04 18:37:00 | 2020-11-04T10:37:00Z [INFO] "GET /api/v1/loki/export?namespace=dev1&app=xxxxapppxaxx&start=1604485500000000000&end=1604485800000000000&level=&pod=&dsc=true HTTP/1.0 500 748.953057ms "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4310.0 Safari/537.36" "
2020-11-04 18:37:00 |
2020-11-04 18:37:00 | 2020/11/04 10:37:00 [Recovery] 2020/11/04 - 10:37:00 panic recovered:
2020-11-04 18:37:00 | interface conversion: interface {} is nil, not string
2020-11-04 18:37:00 | /usr/local/go/src/runtime/iface.go:255 (0x4162f5)
2020-11-04 18:37:00 | /go/src/dagger/backend/gin/controllers/query.go:316 (0xa5db92)
2020-11-04 18:37:00 | /go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:161 (0xa3910a)
2020-11-04 18:37:00 | /go/src/dagger/backend/gin/middlewares/middlewares.go:72 (0xa64a71)
2020-11-04 18:37:00 | /go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:161 (0xa3910a)
2020-11-04 18:37:00 | /go/pkg/mod/github.com/gin-gonic/[email protected]/recovery.go:83 (0xa4cc03)
2020-11-04 18:37:00 | /go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:161 (0xa3910a)
2020-11-04 18:37:00 | /go/pkg/mod/github.com/gin-gonic/[email protected]/logger.go:241 (0xa4bd30)
2020-11-04 18:37:00 | /go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:161 (0xa3910a)
2020-11-04 18:37:00 | /go/pkg/mod/github.com/gin-contrib/[email protected]/sessions.go:52 (0xa4f2fd)
2020-11-04 18:37:00 | /go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:161 (0xa3910a)
2020-11-04 18:37:00 | /go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:409 (0xa4304c)
2020-11-04 18:37:00 | /go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:367 (0xa4274d)
2020-11-04 18:37:00 | /usr/local/go/src/net/http/server.go:2802 (0x81b003)
2020-11-04 18:37:00 | /usr/local/go/src/net/http/server.go:1890 (0x8168a4)
2020-11-04 18:37:00 | /usr/local/go/src/runtime/asm_amd64.s:1357 (0x467610)
2020-11-04 18:37:00
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.