Set container logs to an "admin" GID other than root to make life easier for users and logging/o11y implementers about kubernetes

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[kannon92]

#124228

Related issue.

[yong-jie-gong]

Still have customers adopting Kubernetes and having the discussion about why logging daemonsets need user 0.

By now, is there no adopted way to do what we'd do in nix days, and add the logging agents to a admin group that can access Kubelet logs on disk - /var/log/pods

Has this been documented and solved in the last few years? Would make a lot of folks lives easier!

Even a documented security profile that explains to k8s admins, devs and compliance departments, why theres all these tools trying to "run as root". Would definitely help ISVs as well!

Specifically talking about node level logging architectures: https://kubernetes.io/docs/concepts/cluster-administration/logging/

currently, raw logs(.log) under /var/logs/pods is generated by container runtime instead of kubelet. there is nothing kubelet can do. archive logs (.gz) is genrated by kubelet but is world-wide visible considerd as security issue. which logs do you refer to,?

# ls -rlt /var/log/pods/*/**/*
-rw-r-----. 1 root root /var/log/pods/kube-system_apiserver-demo1.test.com_712b37831be464ccc2fb1553ef89aa91/apiserver/66.log
-rw-r--r--. 1 root root /var/log/pods/kube-system_coredns-cfngt_8a5bacf2-063b-4ffc-b86a-237b94a07246/coredns/18.log.20240329-044703.gz

[matthewmodestino]

I am referring to /var/log/pods. I am not personally concerned about the world readable archives.

Most logging vendor agents only read the live raw log, and yes, I understand the container engine generates them, but in service of the kubelet. Kubelet already exposes container engine settings like rotation size (which are also super low defaults in production environments k8s runs in).

The only issue we have is they are owned as root, thus we have to mess with privileged security context and running as user 0.

Some setting between kubelet and container engine should set it to some acceptable admin group like, root:adm - just using adm as an example of an admin group - then us logging vendors and k8s admins won't need to go thru security friction every single time a new user adopts k8s and runs a logging daemonset.

Its way better than the hacky ways im seeing it done now. Logging vendors shouldnt have to step on these perms out of band, or be on the hook to explain why k8s and their chosen container engines are the way they are and to solve for this.

Otherwise this should be at very least covered in the logging architecure documentation. And perhaps with guidance on a locked down security profile that everyone can jump off from that allows read only to hostpath.

Discussions have happened at the container engine level around this in the past - for example -

containerd/cri#613

containerd/containerd#8920

Would like to see some k8s leadership on this.