ulimit changed in pods between kops 1.28 -> master about kops

[zetaab]

seems that /lib/systemd/system/containerd.service file contains in old instance LimitNOFILE=infinity but it does not exist anymore in kOps master.

[zetaab]

imo it is quite radical just remove that setting in #16151 without providing way to configure that and migrate projects to new behaviour. In slack conversation it was told that kops supports containerd nri already https://kops.sigs.k8s.io/cluster_spec/#nri-configuration and its possible to somehow configure ulimits using that. However, we should have somekind of window to migrate from old to new one, instead of just removing the old solution.

1. we should mark this ulimit as deprecated and provide instructions how to migrate new way of doing
2. delete this old ulimit behaviour in next release
3. with boolean flag we could still allow the old behaviour longer.

This change is quite big. We have currently issues running databases in production because ulimit is limited to 1024. I have now done hotfix (own compiled version) to revert this ulimit change, but still we should have migration window for projects. Most of them are developers who are not interested of changing their deployment manifests just because something changes in platform itself. Also it might be problem for like third party helm charts, if those does not support specifying annotations for pods.