SELinux prevents local-path provisioner PV dirs from being cleaned up about k3s

[galal-hussein]

Okay I think I have figured out the problem:

the problem is related MCS labels, and multi container access on the same file:

 ps auxZ | grep local-path
system_u:system_r:container_t:s0:c608,c1006 root 10449 0.0  0.9 1265432 36588 ?  Ssl  18:05   0:00 local-path-provisioner start --config /etc/config/config.json

As you can see local-path-provisioner started with MCS label s0:c608,c1006 however the helper pod that tries to delete the files has started with MCS label s0:c212,c497 which doesnt have access to that file category

type=AVC msg=audit(1716402374.003:883): avc:  denied  { read } for  pid=14858 comm="ls" name="pvc-31023300-a21e-45ea-b177-70d8f26389b6_kube-system_test-pvc" dev="xvda4" ino=184552397 scontext=system_u:system_r:container_t:s0:c212,c497 tcontext=system_u:object_r:container_file_t:s0:c310,c608 tclass=dir permissive=0

for more info https://www.redhat.com/en/blog/how-selinux-separates-containers-using-multi-level-security

The fix is simply giving the helper pod more range of security context MCS categories, I have added a PR to fix that in local-path-provisioner to make it a permanent solution

[brandond]

cc @galal-hussein

[galal-hussein]

The log doesnt make a lot of sense in my opinion, the denial is a read access from source (container_t) to (container_file_t) which should be available if container-selinux is available, can you check the following:

rpm -qa | grep selinux

also

semodule -l

I need to see what version of container-selinux version is used and if the k3s-selinux is actually loaded not only installed

[zc-devs]

can you check

Sure:
rpm.txt
semodule.txt

[VestigeJ]

Reproduced with selinux in audit

Validated with COMMIT=d9b8ba8d7109ca098c379d170eb412879c5ee94e

type=AVC msg=audit(1717714359.907:540): avc:  denied  { remove_name } for  pid=7527 comm="rm" name="pvc-34f0f671-74e7-4637-bc8d-621c631cda7d_kube-system_test-pvc" dev="xvda3" ino=92284451 scontext=system_u:system_r:container_t:s0:c62,c578 tcontext=system_u:object_r:var_lib_t:s0 tclass=dir permissive=1

//showing expanded file categories
$ sudo ps auxZ | grep local-path

system_u:system_r:container_t:s0:c483,c762 root 3740 0.0  0.8 1265184 33320 ?  Ssl  23:34   0:00 local-path-provisioner start --config /etc/config/config.json

$ kg pv,pvc,pod -A

NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                  STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
persistentvolume/pvc-956284c7-c03a-4955-a634-a7bb4b0960d3   10Mi       RWO            Delete           Bound    kube-system/test-pvc   local-path     <unset>                          4m28s

NAMESPACE     NAME                             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
kube-system   persistentvolumeclaim/test-pvc   Bound    pvc-956284c7-c03a-4955-a634-a7bb4b0960d3   10Mi       RWO            local-path     <unset>                 4m31s

NAMESPACE     NAME                                          READY   STATUS      RESTARTS   AGE
kube-system   pod/coredns-576bfc4dc7-67gmf                  1/1     Running     0          5m51s
kube-system   pod/helm-install-traefik-95djw                0/1     Completed   1          5m52s
kube-system   pod/helm-install-traefik-crd-pfdj8            0/1     Completed   0          5m52s
kube-system   pod/local-path-provisioner-86f46b7bf7-xsmpb   1/1     Running     0          5m51s
kube-system   pod/metrics-server-557ff575fb-x7bqh           1/1     Running     0          5m51s
kube-system   pod/svclb-traefik-9a2525f6-hg9nv              2/2     Running     0          5m36s
kube-system   pod/test-pod                                  1/1     Running     0          4m31s
kube-system   pod/traefik-5fb479b77-pdp75                   1/1     Running     0          5m37s

$ k delete -f pvc.yaml -f podpvc.yaml

persistentvolumeclaim "test-pvc" deleted
pod "test-pod" deleted

$ sudo ausearch -m AVC,USER_AVC,SELINUX_ERR,USER_SELINUX_ERR -ts recent | grep -i denied

type=AVC msg=audit(1717717275.427:887): avc:  denied  { remove_name } for  pid=7378 comm="rm" name="pvc-956284c7-c03a-4955-a634-a7bb4b0960d3_kube-system_test-pvc" dev="xvda3" ino=26007582 scontext=system_u:system_r:container_t:s0-s0:c0.c1023 tcontext=system_u:object_r:var_lib_t:s0 tclass=dir permissive=1