KIND kubelet fails to start about kind

[tyler-dunkel]

Ok it created successfully with those changes. But I wonder why it was picking up the wrong arch?

[stmcginnis]

/remove-kind bug
/kind support

Can you check if DOCKER_DEFAULT_PLATFORM is set? I've seen some mention of this error happening when trying to run amd64 on arm64 hosts.

[tyler-dunkel]

I dont see it set. Should I set it export DOCKER_DEFAULT_PLATFORM=linux/amd64 ?

[BenTheElder]

I dont see it set. Should I set it export DOCKER_DEFAULT_PLATFORM=linux/amd64 ?

unset is preferable, so we can just use the docker default (which matches the host), but if it is set it should match the host platform, which in your case is not amd64

see #2718

[tyler-dunkel]

yea it is unset. It should be using the docker default. NOTE: I tested other docker containers (like hello-world) and they worked fine.

[BenTheElder]

Based on the logs it looks like #2718, this appears to be an amd64 image on an arm64 host. Maybe the image was already pulled as amd64 previously?

[tyler-dunkel]

Ive cleared all the images and containers out of docker and tried to run on a "clean" docker but still hit same issue.

[tyler-dunkel]

one thing to note compaired to #2718 is my install passes the prepare nodes step and fails on start control plane

[tyler-dunkel]

Adding my cluster yaml for additional details

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    kubeadmConfigPatches:
      - |
        kind: InitConfiguration
        nodeRegistration:
          kubeletExtraArgs:
            node-labels: "ingress-ready=true"
    extraPortMappings:
      - containerPort: 80
        hostPort: 80
        protocol: TCP
      - containerPort: 443
        hostPort: 443
        protocol: TCP

[BenTheElder]

From images.log we can see amd64 images preloaded into this node image, indicating it's an amd64 image.

This still looks like a variation on #2718

Can you try explicitly using the arm64 image?

$ crane manifest kindest/node:v1.30.0@sha256:047357ac0cfea04663786a612ba1eaba9702bef25227a794b52890dd8bcd692e
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 743,
         "digest": "sha256:2af5d1b382926abcd6336312d652cd045b7cc47475844a608669c71b1fefcfbc",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 743,
         "digest": "sha256:5e4ce6f9033bdb9ce81a7fd699c8e67cfcacfab57076058e3e6f33c32036b42b",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      }
   ]
}

You can use:
--image=kindest/node:v1.30.0@sha256:5e4ce6f9033bdb9ce81a7fd699c8e67cfcacfab57076058e3e6f33c32036b42b

[BenTheElder]

Better yet also with DOCKER_DEFAULT_PLATFORM=linux/arm64

It still looks like something is causing the wrong architecture to be run, and kind doesn't do anything this internally, that would be docker.

[BenTheElder]

That env may be set or there may be some other related docker setting set.

Docker sort of supports cross-platform image running but the way it does it is not sufficient to run something like Kubernetes.

If you figure out what setting this is overriding please let us know, I'm not aware of another option like DOCKER_DEFAULT_PLATFORM unless you supply --platform to docker run, which kind is not doing, so I'm not sure what it would be besides the env, but it seems plausible that docker would have some other way to set it.