Comments (6)
@cheyang done, I deploy kube-scheduler not in container, so the IP addr in the scheduler-policy-config.json file should not be '127.0.0.1', should be clusterIP.
from gpushare-scheduler-extender.
I think "aliyun.com/gpu-mem":"10G"
should be changed to "aliyun.com/gpu-mem":"10"
. Please take a look at https://github.com/AliyunContainerService/gpushare-scheduler-extender/blob/master/docs/userguide.md
from gpushare-scheduler-extender.
Btw, can you show me the result of kubectl describe po gpushare
?
from gpushare-scheduler-extender.
@cheyang I use the "aliyun.com/gpu-mem":"2", and result is "Error: failed to start container "test": Error response from daemon: OCI runtime create failed: container_linux.go:344: starting container process caused "process_linux.go:424: container init caused "process_linux.go:407: running prestart hook 0 caused \"error running hook: exit status 1, stdout: , stderr: exec command: [/usr/bin/nvidia-container-cli --load-kmods configure --ldconfig=@/sbin/ldconfig --device=no-gpu-has-2MiB-to-run --utility --pid=14123 /home/docker/overlay2/b20cfd4438c42624b7b077786963b307545df9e2e205687291a3f5436908b306/merged]\\nnvidia-container-cli: device error: unknown device id: no-gpu-has-2MiB-to-run\\n\""": unknown"
from gpushare-scheduler-extender.
@cheyang my kubernetes version is v1.10.6
from gpushare-scheduler-extender.
Please run kubectl-inspect-gpushare
first to check the gpu memory unit you are using. Looks like it's MiB.
You can set "aliyun.com/gpu-mem":"1024" to try.
from gpushare-scheduler-extender.
Related Issues (20)
- plugin does not evenly distribute the pods. 这个插件无法均匀分配Pod。 HOT 2
- GPU cores scheduling / GPU核心调度
- 你好,kubectl logs这个命令在gpu容器上无效,在普通容器上却可以
- Back-off restarting failed container: gpushare-device-plugin-ds-xxxxx HOT 1
- 这个GPU共享插件支持使用dcgm-exporter做监控吗 HOT 4
- 如果想要指定使用两张显卡多卡训练 该怎么做 HOT 1
- 如果一个机器上有两张卡,第一张卡的内存使满了,之后的任务会调度到另一张卡上吗
- Support for Horizontal Pod Autoscaling (HPA) with GPU Pods? 是否支持使用GPU Pods的水平Pod自动扩展(HPA)? HOT 1
- 该项目还在维护吗
- 关于显存申请基本单位改为MiB但不起作用的问题
- 调度层有bug吧,请求8G,实际设备最大7G,居然最终能创建成功pod
- 这个项目目前在使用过程中存在的问题
- ALIYUN_COM_GPU_MEM_IDX in the annotation is different than ALIYUN_COM_GPU_MEM_IDX inside the pod
- 方案只能在阿里云上的机器里使用吗 HOT 1
- kubelet版本問題
- [AKS] kube-scheduler static POD not running for Aliyun GPU Scheduler Extender HOT 1
- 版本兼容性问题
- k8s上安装好插件,无法识别到集群GPU资源 HOT 1
- 副本问题
- 节点上有多个GPU时,无法正常分配GPU HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gpushare-scheduler-extender.