Add sysctl support about enhancements

[ehashman]

Hi @arunmk,

There is a PR for the e2es, however those aren't required to merge by code freeze, they will follow the test freeze deadline. Code changes are complete.

kubernetes/kubernetes#99734

[ehashman]

@arunmk currently @wgahnagl is checking if we need to promote e2e tests to conformance as a result of the GA. There are no other code changes.

I'll see about the documentation changes for GA. @pacoxu do you want to take that on or should I find someone else?

[pacoxu]

@tengqm I opened kubernetes/website#26981 for tracking.

[sttts]

There are a number of people using sysctls now. I have not heard any issues with them.

I suggest to promote the current API (transformed to native fields in the PSP and on pods) to beta for 1.11.

@jeremyeder @vishh @derekwaynecarr @php-coder

@kubernetes/sig-node-api-reviews

[sttts]

@kacole2 there is nothing planned to my knowledge in 1.12 about this feature. /cc @derekwaynecarr @ingvagabund @sjenning

[sttts]

Nothing planned here as far as I know.

[sttts]

Nothing planned afaik.

[pacoxu]

@ehashman
I will update sysctls docs next week if no one is working on it.

[arunmk]

Hi @ehashman, @wgahnagl,

Could you mention if there is going to be a PR for the e2e tests? Code freeze is on 3/9 and it should make it by then. If it's not going to come in this KEP can be marked done.

Thanks!

[tengqm]

@ehashman @pacoxu Please open a placeholder PR in k/website for tracking. Thanks.

[annajung]

Hi @ehashman, @pacoxu

Can you update the kep.yaml to reflect a status of implemented:

enhancements/keps/sig-node/34-sysctl-fields/kep.yaml

Line 19 in 29a891b

status: implementable

Once that merges, we can close out this issue.

[sttts]

@kubernetes/docs here are the sysctl docs: kubernetes/website#1126

[sttts]

/cc @kubernetes/feature-reviewers

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[jeremyeder]

Thanks, @sttts!

[derekwaynecarr]

@sttts it needs a feature gate.

from node side, it would be @sjenning who could help push this in sig-node. will sync w/ @dchen1107 next week. we discussed this briefly in last weeks sig-node.

[sttts]

@derekwaynecarr in the kubelet not much would change code-wise. But of course we need a "go" from the node team that they think using sysctls is safe enough for beta. Note, that graduation to beta does not say anything about extending the list of safe sysctls.

It's already feature gated. As beta we would switch the default to true. Doesn't look like we had a feature gate sjenning/kubernetes@f4f7220

[justaugustus]

@sttts
Any plans for this in 1.11?

If so, can you please ensure the feature is up-to-date with the appropriate:

• Description
• Milestone
• Assignee(s)
• Labels:
  • stage/{alpha,beta,stable}
  • sig/*
  • kind/feature

cc @idvoretskyi

[php-coder]

@sttts Do we need to wait until pod annotations become fields or it doesn't block us from graduating it to beta?

[liggitt]

@sttts Do we need to wait until pod annotations become fields or it doesn't block us from graduating it to beta?

yes, they need to become fields

[justaugustus]

@php-coder @liggitt so just to clarify, no work planned for 1.11?
Also, would you mind updating the description to fit the new feature description template?

[sttts]

@justaugustus promotion to beta is discussed in sig-node /cc @derekwaynecarr

[sjenning]

/remove-lifecycle stale

[derekwaynecarr]

@justaugustus - per sig-node planning, goal is to promote to beta.

I have updated assignees with those doing development and review.

[justaugustus]

@derekwaynecarr thanks for the update!

[ingvagabund]

Working on the KEP for the graduation here: kubernetes/community#2093

[twilfong]

Are there also plans to include more sysctls in the safe set as part of this? My company would definitely make use of the ability to set net.ipv4.tcp_keepalive_time, tcp_keepalive_intvl, and tcp_keepalive_probes on a per-pod basis.

Example use: Java applications that depend on TCP keepalive, but which rely on the standard Socket class, can turn keepalive on with that class, but can't set those three parameters.

[php-coder]

There are also 2 open PRs for adding more safe sysctls: kubernetes/kubernetes#54896 and kubernetes/kubernetes#55011

[sttts]

@twilfong compare my comment kubernetes/kubernetes#54896 (comment). We are open to adding more sysctls to the safe set, but we need a kernel source analysis why it is safe. Note that also unsafe sysctls can be used, but they must be whitelisted in the kubelet.

[twilfong]

Thanks @php-coder and @sttts.

@sttts: I've read your comment and read through https://github.com/kubernetes/community/pull/700/files#diff-0e864ea85fc8d72b3bd0b0f39c34d143R342 and understand the basic requirements for whitelisting.

I have verified that the three net.ipv4.tcp_keepalive_* parameters are namespaced in net ns, but have not done an analysis to find if the memory resources caused by the sysctl are accounted for by the associated cgroup.

My guess is that this should meet the bar of not causing harm to the node or other containers on the same node where the pod with changed kernel parameter is run, since the keepalive parameters only control the timing of keepalive probes and when the socket is closed. (e.g. there should be no difference in memory allocation for any given socket, regardless of how these parameters are set.)

What is the recommended way to move forward with this? Should my team do a more deep analysis and then submit a pull request for a commit touching pkg/kubelet/sysctl/whitelist.go and pkg/kubelet/sysctl/whitelist_test.go? Or is there a different (better) recommended way to go about this?

[sttts]

@twilfong I would suggest to add a convincing discussion to the proposal in the community repo for documentation and a counter part PR in k/k against the whitelist. @sjenning @derekwaynecarr @vishh are the ones who can review this.

[ingvagabund]

Promotion of annotations to API fields PR: kubernetes/kubernetes#63717

[mdlinville]

@sttts please fill out the appropriate line item of the
1.11 feature tracking spreadsheet
and open a placeholder docs PR against the
release-1.11 branch
by 5/25/2018 (tomorrow as I write this) if new docs or docs changes are
needed and a relevant PR has not yet been opened.

[sttts]

@ingvagabund ^^

[ingvagabund]

Docs: kubernetes/website#8804

[fejta-bot]

Feature issues opened in kubernetes/features should never be marked as frozen.
Feature Owners can ensure that features stay fresh by consistently updating their states across release cycles.

/remove-lifecycle frozen

[kacole2]

@sttts This feature was worked on in the previous milestone, so we'd like to check in and see if there are any plans for this to graduate stages in Kubernetes 1.12 since there is nothing in the original post.

If there are any updates, please explicitly ping @justaugustus, @kacole2, @robertsandoval, @rajendar38 to note that it is ready to be included in the Features Tracking Spreadsheet for Kubernetes 1.12.

---

Please note that the Features Freeze is July 31st, after which any incomplete Feature issues will require an Exception request to be accepted into the milestone.

In addition, please be aware of the following relevant deadlines:

• Docs deadline (open placeholder PRs): 8/21
• Test case freeze: 8/28

Please make sure all PRs for features have relevant release notes included as well.

Happy shipping!

[justaugustus]

Thanks for the update, @sttts!
Can you modify this issue description to match the issue template?

[sttts]

@justaugustus @derekwaynecarr we need an owner of this feature. Is it sig-node?

[justaugustus]

@sttts -- Based on the comment history, looks like this belongs to SIG Node & @derekwaynecarr.
Happy to chase people down if that isn't sufficient though.

[kacole2]

Hi
This enhancement has been tracked before, so we'd like to check in and see if there are any plans for this to graduate stages in Kubernetes 1.13. This release is targeted to be more ‘stable’ and will have an aggressive timeline. Please only include this enhancement if there is a high level of confidence it will meet the following deadlines:

• Docs (open placeholder PRs): 11/8
• Code Slush: 11/9
• Code Freeze Begins: 11/15
• Docs Complete and Reviewed: 11/27

Please take a moment to update the milestones on your original post for future tracking and ping @kacole2 if it needs to be included in the 1.13 Enhancements Tracking Sheet

Thanks!

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[krmayankk]

/remove-lifecycle rotten

[kacole2]

Hello @sttts @krmayankk , I'm the Enhancement Lead for 1.15. Is this feature going to be graduating alpha/beta/stable stages in 1.15? Please let me know so it can be tracked properly and added to the spreadsheet. As usual, a formal KEP will need to be merged for this to be included in 1.15. The KEP that @ingvagabund created at kubernetes/community#2093 needs to be migrated.

Once coding begins, please list all relevant k/k PRs in this issue so they can be tracked properly.

[kacole2]

Hi @sttts @krmayankk , I'm the 1.16 Enhancement Lead. Is this feature going to be graduating alpha/beta/stable stages in 1.16? Please let me know so it can be added to the 1.16 Tracking Spreadsheet. If not's graduating, I will remove it from the milestone and change the tracked label.

Once coding begins or if it already has, please list all relevant k/k PRs in this issue so they can be tracked properly.

As a reminder, every enhancement requires a KEP in an implementable state with Graduation Criteria explaining each alpha/beta/stable stages requirements.

Milestone dates are Enhancement Freeze 7/30 and Code Freeze 8/29.

Thank you.

[annajung]

Hello @sttts @sjenning @derekwaynecarr @ingvagabund, 1.17 Enhancement Shadow here! 🙂

I wanted to reach out to see if this enhancement will be graduating to alpha/beta/stable in 1.17?


Please let me know so that this enhancement can be added to 1.17 tracking sheet.

Thank you!

🔔Friendly Reminder

• The current release schedule is

  • Monday, September 23 - Release Cycle Begins
  • Tuesday, October 15, EOD PST - Enhancements Freeze
  • Thursday, November 14, EOD PST - Code Freeze
  • Tuesday, November 19 - Docs must be completed and reviewed
  • Monday, December 9 - Kubernetes 1.17.0 Released

• A Kubernetes Enhancement Proposal (KEP) must meet the following criteria before Enhancement Freeze to be accepted into the release

  • PR is merged in
  • In an implementable state
  • Include test plans and graduation criteria

• All relevant k/k PRs should be listed in this issue

[sttts]

I am not aware of a graduation.

[annajung]

@sttts Thank you for letting me know, I will remove this from v1.17 release 👍

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[palnabarun]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[palnabarun]

/remove-lifecycle stale

[msedzins]

Hey there @sttts -- 1.19 Enhancements shadow here. I wanted to check in and see if you think this Enhancement will be graduating in 1.19?

In order to have this part of the release:

1. The KEP PR must be merged in an implementable state
2. The KEP must have test plans
3. The KEP must have graduation criteria.

The current release schedule is:

• Monday, April 13: Week 1 - Release cycle begins
• Tuesday, May 19: Week 6 - Enhancements Freeze
• Thursday, June 25: Week 11 - Code Freeze
• Thursday, July 9: Week 14 - Docs must be completed and reviewed
• Tuesday, August 4: Week 17 - Kubernetes v1.19.0 released
• Thursday, August 20: Week 19 - Release Retrospective

If you do, I'll add it to the 1.19 tracking sheet (http://bit.ly/k8s-1-19-enhancements). Once coding begins please list all relevant k/k PRs in this issue so they can be tracked properly. 👍

Thanks!

[msedzins]

Hi there @sttts , @derekwaynecarr ,

Kind reminder about my question above.

Regards,
Mirek

[msedzins]

Hi there @sttts , @derekwaynecarr ,

Kind reminder about my question above.

Regards,
Mirek

[msedzins]

Hi there @sttts , @derekwaynecarr ,

Kind reminder about my question above.

Regards,
Mirek

[msedzins]

Hey @sttts @derekwaynecarr , Enhancement shadow for the v1.19 release cycle here. Just following up on my earlier update to inform you of the
upcoming Enhancement Freeze scheduled on Tuesday, May 19.

Regards,
Mirek

[palnabarun]

@sttts @derekwaynecarr -- Unfortunately the deadline for the 1.19 Enhancement freeze has passed. For now this is being removed from the milestone and 1.19 tracking sheet. If there is a need to get this in, please file an enhancement exception.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[palnabarun]

/remove-lifecycle stale

[kikisdeliveryservice]

Hi @sttts @derekwaynecarr

Enhancements Lead here. Any plans to graduate this in 1.20?

Thanks!
Kirsten

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[pacoxu]

kubernetes/kubernetes#72593 (comment)
As SysctlForbidden is a probelm， we need a proposal to make the experience better.

[ehashman]

/milestone v1.21

[arunmk]

@ehashman I am an enhancements shadow for 1.21 and am reviewing this KEP. In the kep.yaml file. I have a few comments:

1. I do not see fields as per the current template. We will need to migrate to the new version
2. Is the Graduation Criteria mentioned applicable to the current stable graduation?
3. This issue is marked as lifecycle/rotten. Will you remove it?
4. Should there be a test plan if there are any issues under consideration now?
5. There also needs to be a Product Readiness Review if applicable as per the template.

What are your thoughts on this?

[pacoxu]

We may update it.
/remove-lifecycle rotten

Graduation Criteria:

• API changes allowing to configure the pod-scoped sysctl via spec.securityContext field.（cancelled）
• Promote --experimental-allowed-unsafe-sysctls kubelet flag to kubelet config api option
• feature gate enabled by default
• e2e tests promote WIP
• documentation

Some discussion in
https://docs.google.com/document/d/1FbThdQQVNPISNjK4IEqfliuRCA6pLbUiayb3OASOXHA/edit?usp=sharing

[arunmk]

Hi @pacoxu @ehashman

Enhancements Freeze is 2 days away, Feb 9th EOD PST

Enhancements team is aware that KEP update is currently in progress (as per comment). Please make sure to work on PRR questionnaires and requirements and get it merged before the freeze. For PRR related questions or to boost the PR for PRR review, please reach out in slack #prod-readiness

The KEP looks good.

Any enhancements that do not complete the following requirements by the freeze will require an exception.

[DONE] The KEP must be merged in an implementable state: state is currently provisional
[DONE] The KEP must have test plans
[DONE] The KEP must have graduation criteria
[DONE] The KEP must have a production readiness review: needs file under https://github.com/kubernetes/enhancements/tree/master/keps/prod-readiness/sig-node

EDIT: updated status in place. Thanks for the update @ehashman .

[ehashman]

I will try to get the KEP doc updated today.

[ehashman]

@arunmk this is now good to go for 1.21

[arunmk]

Thanks @ehashman . I am looking at it now and will update the status in-place.

[arunmk]

Hi @ehashman, @pacoxu,

Since your Enhancement is scheduled to be in 1.21, please keep in mind the important upcoming dates:

• Tuesday, March 9th: Week 9 - Code Freeze
• Tuesday, March 16th: Week 10 - Docs Placeholder PR deadline
  • If this enhancement requires new docs or modification to existing docs, please follow the steps in the Open a placeholder PR doc to open a PR against k/website repo.

As a reminder, please link all of your k/k PR(s) and k/website PR(s) to this issue so we can track them.

Thanks!

[pacoxu]

I will work on the flag promotion today:

• - Promote --experimental-allowed-unsafe-sysctls kubelet flag to kubelet config api option Done in 1.11(remove experimental) & 1.16(move to kubeadm/kubelet config)
• WIP by @wgahnagl kubernetes/kubernetes#99158 or kubernetes/kubernetes#99263 focus on lock feature gate on. (Will be done in 1.21 need to rebase and test fix)
• WIP: Promote e2e to conformance testing kubernetes/kubernetes#99734
• Check documentations: https://kubernetes.io/docs/tasks/administer-cluster/sysctl-cluster/

Read the history implementation and do a summary for this feature:

1. 1.4 add security.alpha.kubernetes.io/unsafe-sysctls annotation support, move to client-go annotation_key_constants.go in 1.7 , move to pkg/api/ annotation_key_constants.go in 1.8, move to pkg/apis/core/ in 1.9-1.10.
2. 1.11 kubernetes/kubernetes#63717 Promote sysctl annotations to fields. Add feature gate sysctls, meanwhile, the kubelet "experimental-allowed-unsafe-sysctls" promote to "allowed-unsafe-sysctls".
3. 1.14 kubernetes/kubernetes#72752 Moves feature gate checking of Sysctls out of validation into strategy utility methods, and avoids dropping data on update.
4. 1.16 kubernetes/kubernetes#72974 by @sjenning： kubelet: add allowed sysctl to KubeletConfiguration (add support in kubeadm as well)
5. 1.21 GA and lock to true: kubernetes/kubernetes#99158
6. 1.23 Remove the feature gate.

[arunmk]

Hi @pacoxu,

Enhancements team is currently tracking the following PR

• kubernetes/kubernetes#99158

With the PR merged, can we mark this enhancement complete for code freeze or do you have other PR(s) that are being worked on as part of the release?

Thanks

[arunmk]

(Adding this as a note sent to all)

Hi @ehashman @wgahnagl,

A friendly reminder that Code freeze is 3 days away, March 9th EOD PST

Any enhancements that are NOT code complete by the freeze will be removed from the milestone and will require an exception to be added back.

Please also keep in mind that if this enhancement requires new docs or modification to existing docs, you'll need to follow the steps in the Open a placeholder PR doc to open a PR against k/website repo by March 16th EOD PST

Thanks!

[pacoxu]

Fine