DRA: no handler registered for plugin type: DRAPlugin at socket /var/lib/kubelet/plugins_registry/ about kubernetes

[cyclinder]

Nope, It doesn't relate to this issue, I believe this line of the log has misled you. I used go run main.go to start the dra server; the logs are from when I stopped the dra server.

Can you reproduce this with a dev cluster if you run it this way from the v1.29.1 k/k source tree. ?

Yes, I will be trying to reproduce this later and give feedback to you.

[bart0sh]

@cyclinder I have another guess. You probably didn't enable DynamicResourceAllocation feature gate. This explains why the handler is not registered. Can you show your kubelet command line?

[cyclinder]

/sig node

[bart0sh]

/assign

[bart0sh]

@cyclinder

How can we reproduce it (as minimally and precisely as possible)?
write a dra kubelet-plugin with kubeletplugin.Start()

Here is such a plugin: https://github.com/kubernetes/kubernetes/blob/master/test/e2e/dra/test-driver/app/kubeletplugin.go#L143 and it's used heavily in the DRA e2e and e2e_node tests and registers just fine.

Can you provide more details so I can investigate the issue further?

[cyclinder]

Hi @bart0sh, thanks for the response!

Can you provide more details so I can investigate the issue further?

Yes, the k8s version info is shown below:

root@controller-node-1:~# kubectl  version
Client Version: v1.29.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.1
root@controller-node-1:~# containerd --version
containerd github.com/containerd/containerd v1.7.13 7c3aca7a610df76212171d200ca3811ff6096eb8
root@controller-node-1:~# kubelet --version
Kubernetes v1.29.1

and the codebase of my plugin is here: https://github.com/spidernet-io/spiderpool/blob/0d8872cdb891dde80ef1cc86fb195ee31e126ce4/pkg/dra/dra-plugin/plugin.go#L37

and the version of the dra library I used is v0.29.2: https://github.com/spidernet-io/spiderpool/blob/0d8872cdb891dde80ef1cc86fb195ee31e126ce4/go.mod#L61

and the sock file of my plugin is here:

root@controller-node-1:~# ls /var/lib/kubelet/plugins_registry/netresources.spidernet.io.sock
/var/lib/kubelet/plugins_registry/netresources.spidernet.io.sock

and the error logs of the plugin is show below:

I0522 03:16:50.689661       1 nonblockinggrpcserver.go:105] "GRPC server started" logger="dra"
I0522 03:16:50.689801       1 nonblockinggrpcserver.go:105] "GRPC server started" logger="registrar"
E0522 03:18:20.947948       1 nonblockinggrpcserver.go:125] "handling request failed" err="failed registration process: RegisterPlugin error -- no handler registered for plugin type: DRAPlugin at socket /var/lib/kubelet/plugins_registry/netresources.spidernet.io.sock" logger="registrar" requestID=4 request="&RegistrationStatus{PluginRegistered:false,Error:RegisterPlugin error -- no handler registered for plugin type: DRAPlugin at socket /var/lib/kubelet/plugins_registry/netresources.spidernet.io.sock,}"
E0522 03:18:21.948430       1 nonblockinggrpcserver.go:125] "handling request failed" err="failed registration process: RegisterPlugin error -- no handler registered for plugin type: DRAPlugin at socket /var/lib/kubelet/plugins_registry/netresources.spidernet.io.sock" logger="registrar" requestID=6 request="&RegistrationStatus{PluginRegistered:false,Error:RegisterPlugin error -- no handler registered for plugin type: DRAPlugin at socket /var/lib/kubelet/plugins_registry/netresources.spidernet.io.sock,}"
E0522 03:18:23.948515       1 nonblockinggrpcserver.go:125] "handling request failed" err="failed registration process: RegisterPlugin error -- no handler registered for plugin type: DRAPlugin at socket /var/lib/kubelet/plugins_registry/netresources.spidernet.io.sock" logger="registrar" requestID=8 request="&RegistrationStatus{PluginRegistered:false,Error:RegisterPlugin error -- no handler registered for plugin type: DRAPlugin at socket /var/lib/kubelet/plugins_registry/netresources.spidernet.io.sock,}"

Please let me know if there is any other information you would like me to provide :)

[bart0sh]

@cyclinder I'm not able to reproduce this with the below code and with the hacked version of spiderpool code.

package main

import (
	"context"
	"log"
	"os"

	"k8s.io/dynamic-resource-allocation/kubeletplugin"
	drapbv1alpha3 "k8s.io/kubelet/pkg/apis/dra/v1alpha3"
)

const (
	DRADriverName             = "netresources.spidernet.io"
	DRAPluginRegistrationPath = "/var/lib/kubelet/plugins_registry/" + DRADriverName + ".sock"
	DRADriverPluginPath       = "/var/lib/kubelet/plugins/" + DRADriverName
	DRADriverPluginSocketPath = DRADriverPluginPath + "/plugin.sock"
)

type driver struct {
	logger *log.Logger
}

func (d *driver) NodePrepareResources(ctx context.Context, req *drapbv1alpha3.NodePrepareResourcesRequest) (*drapbv1alpha3.NodePrepareResourcesResponse, error) {
	d.logger.Print("NodePrepareResource is called")
	return &drapbv1alpha3.NodePrepareResourcesResponse{Claims: map[string]*drapbv1alpha3.NodePrepareResourceResponse{}}, nil
}

func (d *driver) NodeUnprepareResources(ctx context.Context, req *drapbv1alpha3.NodeUnprepareResourcesRequest) (*drapbv1alpha3.NodeUnprepareResourcesResponse, error) {
	d.logger.Print("NodeUnprepareResources is called")
	return &drapbv1alpha3.NodeUnprepareResourcesResponse{Claims: map[string]*drapbv1alpha3.NodeUnprepareResourceResponse{}}, nil
}

func StartDRAPlugin(logger *log.Logger) (kubeletplugin.DRAPlugin, error) {
	err := os.MkdirAll(DRADriverPluginPath, 0750)
	if err != nil {
		return nil, err
	}

	driver := &driver{logger}

	dp, err := kubeletplugin.Start(driver,
		kubeletplugin.DriverName(DRADriverName),
		kubeletplugin.RegistrarSocketPath(DRAPluginRegistrationPath),
		kubeletplugin.PluginSocketPath(DRADriverPluginSocketPath),
		kubeletplugin.KubeletPluginSocketPath(DRADriverPluginSocketPath),
	)
	if err != nil {
		return nil, err
	}

	return dp, nil
}

func main() {
	logger := log.Default()
	logger.Println("Starting DRA plugin")
	_, err := StartDRAPlugin(logger)
	if err != nil {
		logger.Fatalf("failed to start DRA plugin: %w", err)
	}
	logger.Println("DRA plugin started")
	select {}
}

Here is what I see in the kubelet log:

I0527 18:24:49.049099 1647750 plugin.go:186] "Validate DRA plugin" name="netresources.spidernet.io" endpoint="/var/lib/kubelet/plugins/netresources.spidernet.io/plugin.sock" versions="1.0.0"
I0527 18:24:49.049149 1647750 plugin.go:106] "Register new DRA plugin" name="netresources.spidernet.io" endpoint="/var/lib/kubelet/plugins/netresources.spidernet.io/plugin.sock"

Can you try to run either of those in your environment just to see if it's still reproducible?

[cyclinder]

Thanks @bart0sh for looking at this! I tried to run your branch in my environment, but it still failed with the same error:

root@worker-node-1:/home/cyclinder/cyclinder/spiderpool/cmd/spiderpool-agent# go run main.go
2024/05/28 10:56:07 >>> Starting DRA plugin
I0528 10:56:07.635516  136753 nonblockinggrpcserver.go:105] "GRPC server started" logger="dra"
I0528 10:56:07.635616  136753 nonblockinggrpcserver.go:105] "GRPC server started" logger="registrar"
E0528 10:56:08.299190  136753 nonblockinggrpcserver.go:125] "handling request failed" err="failed registration process: RegisterPlugin error -- no handler registered for plugin type: DRAPlugin at socket /var/lib/kubelet/plugins_registry/dra.spidernet.io.sock" logger="registrar" requestID=2 request="&RegistrationStatus{PluginRegistered:false,Error:RegisterPlugin error -- no handler registered for plugin type: DRAPlugin at socket /var/lib/kubelet/plugins_registry/dra.spidernet.io.sock,}"
E0528 10:56:09.299408  136753 nonblockinggrpcserver.go:125] "handling request failed" err="failed registration process: RegisterPlugin error -- no handler registered for plugin type: DRAPlugin at socket /var/lib/kubelet/plugins_registry/dra.spidernet.io.sock" logger="registrar" requestID=4 request="&RegistrationStatus{PluginRegistered:false,Error:RegisterPlugin error -- no handler registered for plugin type: DRAPlugin at socket /var/lib/kubelet/plugins_registry/dra.spidernet.io.sock,}"
E0528 10:56:11.300439  136753 nonblockinggrpcserver.go:125] "handling request failed" err="failed registration process: RegisterPlugin error -- no handler registered for plugin type: DRAPlugin at socket /var/lib/kubelet/plugins_registry/dra.spidernet.io.sock" logger="registrar" requestID=6 request="&RegistrationStatus{PluginRegistered:false,Error:RegisterPlugin error -- no handler registered for plugin type: DRAPlugin at socket /var/lib/kubelet/plugins_registry/dra.spidernet.io.sock,}"
E0528 10:56:14.302561  136753 nonblockinggrpcserver.go:125] "handling request failed" err="failed registration process: RegisterPlugin error -- no handler registered for plugin type: DRAPlugin at socket /var/lib/kubelet/plugins_registry/dra.spidernet.io.sock" logger="registrar" requestID=8 request="&RegistrationStatus{PluginRegistered:false,Error:RegisterPlugin error -- no handler registered for plugin type: DRAPlugin at socket /var/lib/kubelet/plugins_registry/dra.spidernet.io.sock,}"

show the kubelet logs:

May 28 10:56:08 worker-node-1 kubelet[2430]: E0528 10:56:08.299430    2430 goroutinemap.go:150] Operation for "/var/lib/kubelet/plugins_registry/dra.spidernet.io.sock" failed. No retries permitted until 2024-05-28 10:56:08.799408329 +0800 CST m=+3495.843079650 (durationBeforeRetry 500ms). Error: RegisterPlugin error -- failed to send error at socket /var/lib/kubelet/plugins_registry/dra.spidernet.io.sock, err: RegisterPlugin error -- no handler registered for plugin type: DRAPlugin at socket /var/lib/kubelet/plugins_registry/dra.spidernet.io.sock: rpc error: code = Unknown desc = failed registration process: RegisterPlugin error -- no handler registered for plugin type: DRAPlugin at socket /var/lib/kubelet/plugins_registry/dra.spidernet.io.sock
May 28 10:56:09 worker-node-1 kubelet[2430]: E0528 10:56:09.299734    2430 goroutinemap.go:150] Operation for "/var/lib/kubelet/plugins_registry/dra.spidernet.io.sock" failed. No retries permitted until 2024-05-28 10:56:10.299701237 +0800 CST m=+3497.343372589 (durationBeforeRetry 1s). Error: RegisterPlugin error -- failed to send error at socket /var/lib/kubelet/plugins_registry/dra.spidernet.io.sock, err: RegisterPlugin error -- no handler registered for plugin type: DRAPlugin at socket /var/lib/kubelet/plugins_registry/dra.spidernet.io.sock: rpc error: code = Unknown desc = failed registration process: RegisterPlugin error -- no handler registered for plugin type: DRAPlugin at socket /var/lib/kubelet/plugins_registry/dra.spidernet.io.sock
May 28 10:56:11 worker-node-1 kubelet[2430]: E0528 10:56:11.300773    2430 goroutinemap.go:150] Operation for "/var/lib/kubelet/plugins_registry/dra.spidernet.io.sock" failed. No retries permitted until 2024-05-28 10:56:13.300741111 +0800 CST m=+3500.344412443 (durationBeforeRetry 2s). Error: RegisterPlugin error -- failed to send error at socket /var/lib/kubelet/plugins_registry/dra.spidernet.io.sock, err: RegisterPlugin error -- no handler registered for plugin type: DRAPlugin at socket /var/lib/kubelet/plugins_registry/dra.spidernet.io.sock: rpc error: code = Unknown desc = failed registration process: RegisterPlugin error -- no handler registered for plugin type: DRAPlugin at socket /var/lib/kubelet/plugins_registry/dra.spidernet.io.sock
May 28 10:56:14 worker-node-1 kubelet[2430]: E0528 10:56:14.302841    2430 goroutinemap.go:150] Operation for "/var/lib/kubelet/plugins_registry/dra.spidernet.io.sock" failed. No retries permitted until 2024-05-28 10:56:18.3028145 +0800 CST m=+3505.346485853 (durationBeforeRetry 4s). Error: RegisterPlugin error -- failed to send error at socket /var/lib/kubelet/plugins_registry/dra.spidernet.io.sock, err: RegisterPlugin error -- no handler registered for plugin type: DRAPlugin at socket /var/lib/kubelet/plugins_registry/dra.spidernet.io.sock: rpc error: code = Unknown desc = failed registration process: RegisterPlugin error -- no handler registered for plugin type: DRAPlugin at socket /var/lib/kubelet/plugins_registry/dra.spidernet.io.sock
cMay 28 10:56:18 worker-node-1 kubelet[2430]: W0528 10:56:18.304259    2430 logging.go:59] [core] [Channel #2723 SubChannel #2724] grpc: addrConn.createTransport failed to connect to {Addr: "/var/lib/kubelet/plugins_registry/dra.spidernet.io.sock", ServerName: "%2Fvar%2Flib%2Fkubelet%2Fplugins_registry%2Fdra.spidernet.io.sock", }. Err: connection error: desc = "transport: Error while dialing: dial unix /var/lib/kubelet/plugins_registry/dra.spidernet.io.sock: connect: connection refused"
May 28 10:56:19 worker-node-1 kubelet[2430]: W0528 10:56:19.304832    2430 logging.go:59] [core] [Channel #2723 SubChannel #2724] grpc: addrConn.createTransport failed to connect to {Addr: "/var/lib/kubelet/plugins_registry/dra.spidernet.io.sock", ServerName: "%2Fvar%2Flib%2Fkubelet%2Fplugins_registry%2Fdra.spidernet.io.sock", }. Err: connection error: desc = "transport: Error while dialing: dial unix /var/lib/kubelet/plugins_registry/dra.spidernet.io.sock: connect: connection refused"

Can you tell me which k8s version you use in your environment?

[bart0sh]

@cyclinder Can you reproduce this with a dev cluster if you run it this way from the v1.29.1 k/k source tree. ?

FEATURE_GATES='DynamicResourceAllocation=true,APIServerTracing=false' RUNTIME_CONFIG=resource.k8s.io/v1alpha2 DNS_ADDON="coredns" CGROUP_DRIVER=systemd ALLOW_PRIVILEGED=1 CONTAINER_RUNTIME_ENDPOINT=unix:///var/run/containerd/containerd.sock LOG_LEVEL=6 API_SECURE_PORT=6444 PATH=$(pwd)/third_party/etcd:$PATH ./hack/local-up-cluster.sh -O

[bart0sh]

@cyclinder I tried to use exactly the same versions as you've mentioned above. Here is my go.mod for the code I've shown above:

$ cat ~/go/src/misc/kubereg/go.mod
module misc/kubereg

go 1.22.1

require (
        k8s.io/dynamic-resource-allocation v0.29.2
        k8s.io/kubelet v0.29.2
)

require (
        github.com/go-logr/logr v1.3.0 // indirect
        github.com/gogo/protobuf v1.3.2 // indirect
        github.com/golang/protobuf v1.5.4 // indirect
        golang.org/x/net v0.23.0 // indirect
        golang.org/x/sys v0.18.0 // indirect
        golang.org/x/text v0.14.0 // indirect
        google.golang.org/genproto/googleapis/rpc v0.0.0-20230822172742-b8732ec3820d // indirect
        google.golang.org/grpc v1.58.3 // indirect
        google.golang.org/protobuf v1.33.0 // indirect
        k8s.io/klog/v2 v2.110.1 // indirect
)

k/k version is 1.29.1

[bart0sh]

@cyclinder
This is the core of the issue I think:

"transport: Error while dialing: dial unix /var/lib/kubelet/plugins_registry/dra.spidernet.io.sock: connect: connection refused"

It looks like Kubelet can't connect to the plugin socket for some reason. It would make sense to run Kubelet with a higher verbosity level to get more info.

[haircommander]

/triage accepted

[cyclinder]

woo, thanks for the tips, I enabled the DynamicResourceAllocation feature gate via kubelet flag before, but It didn't work, and after I added the feature-gates into /var/lib/kubelet/config.yaml, It works now.

root@worker-node-1:~# cat /etc/kubernetes/kubelet.env
KUBE_LOG_LEVEL="--v=2"
KUBELET_ADDRESS="--node-ip=10.20.1.230"
KUBELET_HOSTNAME="--hostname-override=worker-node-1"


KUBELET_ARGS="--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf \
--config=/etc/kubernetes/kubelet-config.yaml \
--kubeconfig=/etc/kubernetes/kubelet.conf \
--container-runtime-endpoint=unix:///var/run/containerd/containerd.sock \
--runtime-cgroups=/system.slice/containerd.service \
--feature-gates=DynamicResourceAllocation=true,AllAlpha=true \
 "
KUBELET_CLOUDPROVIDER=""

PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
root@worker-node-1:~# cat /var/lib/kubelet/config.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/ssl/ca.crt
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 0s
    cacheUnauthorizedTTL: 0s
cgroupDriver: systemd
clusterDNS:
- 169.254.25.10
clusterDomain: cluster.local
containerRuntimeEndpoint: ""
cpuManagerReconcilePeriod: 0s
evictionPressureTransitionPeriod: 0s
fileCheckFrequency: 0s
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 0s
imageMaximumGCAge: 0s
imageMinimumGCAge: 0s
kind: KubeletConfiguration
logging:
  flushFrequency: 0
  options:
    json:
      infoBufferSize: "0"
  verbosity: 0
memorySwap: {}
nodeStatusReportFrequency: 0s
nodeStatusUpdateFrequency: 0s
resolvConf: /run/systemd/resolve/resolv.conf
rotateCertificates: true
featureGates:
  DynamicResourceAllocation: true
  AllAlpha: true
runtimeRequestTimeout: 0s
shutdownGracePeriod: 0s
shutdownGracePeriodCriticalPods: 0s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
volumeStatsAggPeriod: 0s
May 30 09:55:22 worker-node-1 kubelet[1202942]: I0530 09:55:22.907570 1202942 plugin.go:186] "Validate DRA plugin" name="netresources.spidernet.io" endpoint="/var/lib/kubelet/plugins/netresources.spidernet.io/plugin.sock" versions="1.0.0"
May 30 09:55:22 worker-node-1 kubelet[1202942]: I0530 09:55:22.907661 1202942 plugin.go:106] "Register new DRA plugin" name="netresources.spidernet.io" endpoint="/var/lib/kubelet/plugins/netresources.spidernet.io/plugin.sock"

Thanks @bart0sh for the help! Let me close the issue.

/close

[k8s-ci-robot]

@cyclinder: Closing this issue.

In response to this:

woo, thanks for the tips, I enabled the DynamicResourceAllocation feature gate via kubelet flag before, but It didn't work, and after I added the feature-gates into /var/lib/kubelet/config.yaml, It works now.

root@worker-node-1:~# cat /etc/kubernetes/kubelet.env
KUBE_LOG_LEVEL="--v=2"
KUBELET_ADDRESS="--node-ip=10.20.1.230"
KUBELET_HOSTNAME="--hostname-override=worker-node-1"


KUBELET_ARGS="--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf \
--config=/etc/kubernetes/kubelet-config.yaml \
--kubeconfig=/etc/kubernetes/kubelet.conf \
--container-runtime-endpoint=unix:///var/run/containerd/containerd.sock \
--runtime-cgroups=/system.slice/containerd.service \
--feature-gates=DynamicResourceAllocation=true,AllAlpha=true \
"
KUBELET_CLOUDPROVIDER=""

PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
root@worker-node-1:~# cat /var/lib/kubelet/config.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
 anonymous:
   enabled: false
 webhook:
   cacheTTL: 0s
   enabled: true
 x509:
   clientCAFile: /etc/kubernetes/ssl/ca.crt
authorization:
 mode: Webhook
 webhook:
   cacheAuthorizedTTL: 0s
   cacheUnauthorizedTTL: 0s
cgroupDriver: systemd
clusterDNS:
- 169.254.25.10
clusterDomain: cluster.local
containerRuntimeEndpoint: ""
cpuManagerReconcilePeriod: 0s
evictionPressureTransitionPeriod: 0s
fileCheckFrequency: 0s
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 0s
imageMaximumGCAge: 0s
imageMinimumGCAge: 0s
kind: KubeletConfiguration
logging:
 flushFrequency: 0
 options:
   json:
     infoBufferSize: "0"
 verbosity: 0
memorySwap: {}
nodeStatusReportFrequency: 0s
nodeStatusUpdateFrequency: 0s
resolvConf: /run/systemd/resolve/resolv.conf
rotateCertificates: true
featureGates:
 DynamicResourceAllocation: true
 AllAlpha: true
runtimeRequestTimeout: 0s
shutdownGracePeriod: 0s
shutdownGracePeriodCriticalPods: 0s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
volumeStatsAggPeriod: 0s
May 30 09:55:22 worker-node-1 kubelet[1202942]: I0530 09:55:22.907570 1202942 plugin.go:186] "Validate DRA plugin" name="netresources.spidernet.io" endpoint="/var/lib/kubelet/plugins/netresources.spidernet.io/plugin.sock" versions="1.0.0"
May 30 09:55:22 worker-node-1 kubelet[1202942]: I0530 09:55:22.907661 1202942 plugin.go:106] "Register new DRA plugin" name="netresources.spidernet.io" endpoint="/var/lib/kubelet/plugins/netresources.spidernet.io/plugin.sock"

Thanks @bart0sh for the help! Let me close the issue.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.