Comments (6)
日志如下:
t=2022-06-09T14:16:26+0800 lvl=eror msg="Request Completed" logger=context userId=0 orgId=1 uname= method=GET path=/api/datasources/proxy/1/api/v1/query_range status=502 remote_addr=10.168.2.252 time_ms=30002 size=0 referer="http://10.168.2.252/grafana/d/all-node/all-node?orgId=1&refresh=5s"
t=2022-06-09T14:16:26+0800 lvl=eror msg="Data proxy error" logger=data-proxy-log userId=0 orgId=1 uname= path=/api/datasources/proxy/1/api/v1/query_range remote_addr=10.168.2.252 referer="http://10.168.2.252/grafana/d/all-node/all-node?orgId=1&refresh=5s" error="http: proxy error: dial tcp 10.43.83.231:9090: i/o timeout"
t=2022-06-09T14:16:26+0800 lvl=eror msg="Request Completed" logger=context userId=0 orgId=1 uname= method=GET path=/api/datasources/proxy/1/api/v1/query_range status=502 remote_addr=10.168.2.252 time_ms=30001 size=0 referer="http://10.168.2.252/grafana/d/all-node/all-node?orgId=1&refresh=5s"
t=2022-06-09T14:16:31+0800 lvl=eror msg="Alert Rule Result Error" logger=alerting.evalContext ruleId=1 name="System load alert" error="alert execution exceeded the timeout" changing state to=alerting
查到一个贴子:https://grafana.docs.cern.ch/4._Troubleshooting/2-proxy-error-context-canceled/
还是不大明白,这个贴子说的是让改哪个js代码?
from cube-studio.
grafana 可以看到界面包含了几个地方的功能需要:
1、每个机器的采集器是否正常 包含node-export/dcgm-exporter(gpu)
2、prometheus是否正常,可以在prometheus里面看数据是否正常
3、查看grafana是否正常,主要是看板的配置
4、最后是网关带来了grafana的url访问
从你的截图里面看,应该grafana链接不上prometheus,所以怀疑是你的prometheus未正常启动
from cube-studio.
prometheus正常启动着呢
日志里面有一些警告,不知道有没有影响
level=warn ts=2022-06-14T04:58:58.12469297Z caller=klog.go:86 component=k8s_client_runtime func=Warningf msg="/app/discovery/kubernetes/kubernetes.go:262: watch of *v1.Endpoints ended with: too old resource version: 3201509 (3202538)"
level=info ts=2022-06-14T05:00:03.954078526Z caller=compact.go:496 component=tsdb msg="write block" mint=1655172000000 maxt=1655179200000 ulid=01G5G9ADRC0DSEC1K1KEM5TZ81 duration=3.301850879s
level=info ts=2022-06-14T05:00:04.202635464Z caller=head.go:536 component=tsdb msg="head GC completed" duration=108.122048ms
level=info ts=2022-06-14T05:00:09.252410301Z caller=head.go:583 component=tsdb msg="WAL checkpoint complete" first=173 last=175 duration=5.049629249s
from cube-studio.
kubeflow-prometheus-adapter运行正常
日志报这个错
E0614 06:59:25.408045 1 provider.go:229] unable to update list of all metrics: unable to fetch metrics for query "{name="DCGM_FI_DEV_GPU_UTIL",exported_pod!="",exported_namespace="service"}": Get "http://prometheus-k8s.monitoring.svc:9090/api/v1/series?match%5B%5D=%7B__name__%3D%22DCGM_FI_DEV_GPU_UTIL%22%2Cexported_pod%21%3D%22%22%2Cexported_namespace%3D%22service%22%7D&start=1655189875.407": dial tcp 10.43.83.231:9090: i/o timeout
from cube-studio.
kubeflow-prometheus-adapter这个模块的shell进去报错:
from cube-studio.
commit 0b6b2aa
修复了这个问题
from cube-studio.
Related Issues (20)
- 【模型训练】 创建的任务模板无法删除 HOT 2
- external_ip 多ip 逻辑bug HOT 1
- 数据集删除遗留问题 HOT 1
- 单机部署之后访问http:xx.xx.xx.xx 返回rancher web页面咋回事呀,rancher web按照部署手册也是设置的xx.xx.xx.xx HOT 1
- 启动以后【标注平台】功能入口是disabled的状态 HOT 1
- 项目空间-资源配置模块为disabled状态 HOT 1
- 私有仓库镜像节点无法运行 failed to look-up entrypoint/cmd for image \"172.40.20.82:8443/aiclube/model_download\" HOT 1
- 请问是否支持推理服务部署在非k8s集群机器上
- 怎么更新已经单机部署好的cube-studio平台 HOT 1
- 使用sealos部署的k8s单机,在点击k8s Dashboard、Grafana以及进入到notebook时都会出现404错误 HOT 1
- 如何将Geforce RTX3090GPU 加入平台的K8S集群中? HOT 1
- 登录界面没有相应代码 HOT 4
- 单机部署 数据库可以用单独机器的数据库吗 HOT 1
- docker-compose部署单机出现版本问题
- 创建notebook后,点进去404
- 如何接入label studio
- 部分镜像拉取失败
- 可以新增多模态信息抽取模型?
- 请问链接公众号和企业微信部分的代码在哪里啊?
- 关于start.sh HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cube-studio.