K3s fails to import tarball produced by `ctr image export` about k3s

[mdrahman-suse]

Validated on master branch with commit d9b8ba8

Environment and config

Ubuntu 22.04, Single server

• Config.yaml

write-kubeconfig-mode: 644
cluster-init: true
node-name: server1

Testing

• Install k3s
• Ensure cluster is up and running
• Deploy pod nginx (Can be from any other image)
• Check nginx is running and image is showed when listed
• Export the image
• Copy the image in to a new server and store it in path /var/lib/rancher/k3s/agent/images
• Install k3s on the new server
• Ensure the image is shown when listed and no error related to image import is observed in the logs

Replication

$ k3s -v
k3s version v1.30.1+k3s1 (80978b5b)
go version go1.22.2

• Deploy nginx, ensure its on the list

$ kubectl run nginx --image=nginx
pod/nginx created

$ sudo ctr images ls | grep nginx
docker.io/library/nginx:latest
...

• Export that image and copy over to the new server

$ sudo ctr image export nginx-image.tar docker.io/library/nginx:latest

• Install k3s version on the new server
• Copy the image tarball on the new server in path

$ sudo mkdir -p /var/lib/rancher/k3s/agent/images && sudo cp nginx-image.tar /var/lib/rancher/k3s/agent/images

• Observed the error on the logs

$ sudo journalctl -u k3s | grep nginx-image
Jun 12 22:46:49 ip-172-31-47-24 k3s[23292]: time="2024-06-12T22:46:49Z" level=info msg="Importing images from /var/lib/rancher/k3s/agent/images/nginx-image.tar"
Jun 12 22:46:51 ip-172-31-47-24 k3s[23292]: time="2024-06-12T22:46:51Z" level=error msg="Error encountered while importing /var/lib/rancher/k3s/agent/images/nginx-image.tar: failed to import images from /var/lib/rancher/k3s/agent/images/nginx-image.tar: content digest sha256:c36751d7f6324d496c55c7fae626cda3efcc62b11a3b6cfb28d5ae53d8d6fc7a: not found"

• No image is observed when searched on the list

$ sudo ctr images ls | grep nginx
$

Validation

$ k3s -v
k3s version v1.30.1+k3s-d9b8ba8d (d9b8ba8d)
go version go1.22.2

• Deploy nginx, ensure its on the list

$ kubectl run nginx --image=nginx
pod/nginx created

$ sudo ctr images ls | grep nginx
docker.io/library/nginx:latest
...

• Export that image and copy over to the new server

$ sudo ctr image export nginx-image.tar docker.io/library/nginx:latest

• Install k3s commit on the new server
• Copy the image tarball on the new server in path

$ sudo mkdir -p /var/lib/rancher/k3s/agent/images && sudo cp nginx-image.tar /var/lib/rancher/k3s/agent/images

• Observed no related error on the logs

$ sudo journalctl -u k3s | grep nginx-image
Jun 12 22:37:58 k3s[20247]: time="2024-06-12T22:37:58Z" level=info msg="Importing images from /var/lib/rancher/k3s/agent/images/nginx-image.tar"
Jun 12 22:38:00 k3s[20247]: time="2024-06-12T22:38:00Z" level=info msg="Imported images from /var/lib/rancher/k3s/agent/images/nginx-image.tar in 2.262639703s"

• Image is observed when searched on the list

$ sudo ctr images ls | grep nginx
docker.io/library/nginx:latest
...