• Release Signoff Checklist
• Summary
• Motivation
  • Goals
  • Non-Goals
• Proposal
  • First release N (when the deprecation happens)
  • Release N+1
  • Release N+3 (two releases after Beta)
  • Release N+4
  • User Stories (Optional)
    • Story 1
    • Story 2
  • Notes/Constraints/Caveats (Optional)
  • Risks and Mitigations
    • Risk 1
    • Risk 2
    • Risk 3
• Design Details
  • Test Plan
  • Graduation Criteria
    • Alpha
    • Beta
    • GA
    • Post GA
  • Upgrade / Downgrade Strategy
    • Steps per release:
  • Version Skew Strategy
• Production Readiness Review Questionnaire
• Implementation History
• Drawbacks
• Alternatives
• Infrastructure Needed (Optional)

Items marked with (R) are required prior to targeting to a milestone / release.

• (R) Enhancement issue in release milestone, which links to KEP dir in kubernetes/enhancements (not the initial KEP PR)
• (R) KEP approvers have approved the KEP status as implementable
• (R) Design details are appropriately documented
• (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
  • e2e Tests for all Beta API Operations (endpoints)
  • (R) Ensure GA e2e tests for meet requirements for Conformance Tests
  • (R) Minimum Two Week Window for GA e2e tests to prove flake free
• (R) Graduation criteria is in place
  • (R) all GA Endpoints must be hit by Conformance Tests
• (R) Production readiness review completed
• (R) Production readiness review approved
• "Implementation History" section is up-to-date for milestone
• User-facing documentation has been created in kubernetes/website, for publication to kubernetes.io
• Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes

During kubeadm init, a ConfigMap is written under kube-system called kubelet-config-x.y where x is the Kubernetes MAJOR version and y the MINOR version. It includes a KubeletConfiguration object Kind. The same ConfigMap is read during kubeadm join and downloaded to a file so that the joining kubelet can use it via --config. The access to this ConfigMap for joining nodes, is made possible via RBAC rules created during kubeadm init using a similar versioned naming format - Role kubeadm:kubelet-config-x.y and RoleBinding kubeadm:kubelet-config-x.y, both under kube-system.

Simplify the naming of the default kubelet configuration ConfigMap and related RBAC rules that kubeadm manages to the format kubelet-config, which does not include a version in the name. Given the stored configuration object is versioned (has GroupVersionKind), it would be up to the running kubeadm instance that GETs it to determine if this GroupVersionKind is supported.

Currently the overall design is faulty as kubeadm init, despite supporting a version skew with the kubelet of N-1, writes a single ConfigMap kubelet-config-x.y where x.y is the version of the control plane and not the version of the kubelet on the host where kubeadm init is called.

Later, during join operations the version x.y is determined by looking up the control plane version of the cluster, via the kubernetesVersion field of the ClusterConfiguration object stored in the kube-system/kubeadm-config ConfigMap.

During upgrade a new kubelet-config-x.y+1 ConfigMap and RBAC rules are written where y+1 is the target MINOR upgrade version. The old kubelet-config-x.y are not cleaned up.

Encoding the version in the API object names is redundant and creates unnecessary complications for both kubeadm maintainers and users that wish to modify the object in the cluster.

• Simplify the name of the kubelet related ConfigMap and RBAC rules that kubeadm manages to exclude encoded version information in the API object names.
• Remove confusion around the version that is encoded in the name - control plane vs kubelet version?
• Simplify user access to the ConfigMap, allowing them to always rely on kube-system/kubelet-config as the default location.
• Simplify the kubeadm logic to not manage the x.y version during PUT/GET operations.
• Gradually apply the change to allow users to adapt using a feature gate.

• Apply the change within a single release of kubeadm, breaking the users.

The proposal is to declare the default ConfigMap and RBAC object format of kubelet-config-x.y deprecated but support it for a GA deprecation period of 1 year (3 kubeadm releases). In parallel, introduce objects named kubelet-config (without x.y) that will become the source of truth.

The switch will be controlled by a new kubeadm specific feature gate (not a core one, stored in kubeadm's ClusterConfiguration) called UnversionedKubeletConfigMap. Using a feature gate for similar changes is common in changes in core Kubernetes components and is familiar mechanism to users.

Note that depending on feedback from users the N+x release schedule might be adjusted - e.g. extending or shortening the Alpha or Beta periods.

• The feature gate is added as Alpha and disabled by default.
• Users can continue using the old naming format or opt-in into the new naming format by enabling the feature gate.

• The feature gate becomes Beta and is enabled by default.
• Users can opt-out of the new feature by disabling the feature gate.

• The feature gate becomes GA and is locked to enabled.
• Users can no longer opt-out and must adapt their infrastructure that reads/writes from/to the ConfigMap.
• Users should remove usage of the feature gate from their infrastructure. Alternatively they can do it in the next release.

• The feature gate is removed.

As a kubeadm user that wishes to modify the KubeletConfiguration that kubeadm stores in the cluster before joining new nodes, I wish to have easy access to a static location such as kube-system/kubelet-config without looking up the control plane version first.

As a kubeadm maintainer, I wish to simplify the logic in kubeadm that manages the kubelet configuration read/write operations.

NONE

Users do not have sufficient visibility of the change

Communicate the change on the following channels to ensure as much visibility as possible:

• kubeadm release notes
• kubernetes-dev mailing list
• #kubeadm on k8s slack
• Blog posts, Twitter, Reddit, other

Users that do not use kubeadm upgrade would not be able to join new nodes

In the Alpha stage of the change, the command kubeadm upgrade apply will be used to create a ConfigMap and the related RBAC rules that follow the new naming format. Nodes that are joined to the cluster later would be able to use this new ConfigMap. During Beta the presence of these new API objects will become a requirement, unless the users opt-out by disabling the feature gate stored in the ClusterConfiguration.

Cluster API is an example of a project that uses kubeadm for cluster creation, but does not use the kubeadm upgrade command for upgrades. These upgrades are called "immutable node upgrades", where new version nodes join the cluster and replace the old ones. Similar projects and users would have to implement their own means to support the Alpha->Beta transition of this feature by populating the required ConfigMap and RBAC rules before joining new nodes during an immutable upgrade.

The kubelet deprecates the API version stored in the ConfigMap

Ideally, when the kubelet introduces a new API version (e.g. v1beta2) and deprecates the API currently used by kubeadm (e.g. v1beta1), kubeadm has to have a way to upconvert the configuration in the ConfigMap, so that users don't have to do it manually. Currently the kubelet does not expose a way to perform this upconversion and the process is delegated as manual for the user to perform.

To continue supporting its version skew against the kubelet (N, N-1), kubeadm will:

• Delay the switch from the old to new kubelet API by one or more releases for new clusters
• For existing clusters during "join" and "upgrade", kubeadm will continue using the old API for as long as possible.

This would give more time for the users to update the contents of the ConfigMap and also to properly support the kubeadm / kubelet skew.

The new feature gate will be tested in a specific e2e test jobs managed by the existent tooling that kubeadm has established.

Unit tests

Unit tests will be included in the kubeadm source code to ensure that the feature gate correctly toggles the used names of API objects.

Upgrade e2e test job

This upgrade job will ensure that kubeadm upgrade apply correctly creates API objects with the new name format and that kubeadm upgrade node correctly consumes them.

This test job would not be required for the Beta->GA upgrade stage, since at that point the feature gate will be enabled by default and the existing kubeadm tests will provide the same coverage.

Join e2e test job

This test job will be similar to the existing regular kubeadm e2e test jobs, except that it would have the feature gate enabled. Its purpose would be to ensure that kubeadm join to an existing cluster when the feature gate is enabled would GET the correct ConfigMap.

This test job would be viable only during the Alpha period, as during Beta the feature will be enabled by default. Not having the feature enabled during Beta would be tested by existing e2e test jobs.

Other kubeadm e2e tests

Tests under kubernetes/kubernetes/test/e2e_kubeadm will be updated to correctly consume the correct ConfigMap and RBAC depending on the feature gate state.

These tests would be relevant for the Alpha period and once the feature gate is enabled by default (Beta) the checks for the legacy naming can be removed from them.

Future e2e tests

In the future, when the kubelet introduces a new API version, we should add kubeadm e2e tests that ensure that kubeadm continues to operate using its preferred kubelet API version stored in the kubelet-config ConfigMap.

• The feature is implemented behind a feature flag.
• Announce the feature to users.
• E2e tests are enabled.

• The feature is enabled by default, but users can opt out.
• Gather more feedback from users if needed.
• E2e tests are updated.
• Documentation at the k8s.io website is updated.

• The feature is locked to enabled.
• Documentation at the k8s.io website is updated.
• E2e tests are updated.
• Gather more feedback from users if needed.

• The feature gate is removed.

• First release (N): during kubeadm upgrade to a kubeadm version N (feature gate as Alpha and disabled by default) kubeadm must write the new ConfigMap and RBAC rules to allow the feature gate to work if the user wishes to switch it to enabled.
• Second release (N+1): during kubeadm upgrade to a kubeadm version N+1 (feature gate is Beta and enabled by default) the nodes can start using the new ConfigMap and RBAC rules by default, unless the user has explicitly disabled the feature gate before upgrade.
• Third release (N+2): kubeadm upgrade and join will continue to honor the value of the feature gate.
• Fourth release (N+3): kubeadm upgrade and join will now only use the new format. Cleanup of the old ConfigMaps and RBAC rules is out of scope but kubeadm can perform it.
• Fifth release (N+4): kubeadm no longer uses the feature gate and the feature is enabled by default.

Non-applicable, since the feature gate will be specific to the kubeadm binary and its operation and components that kubeadm can be skewed against such as the kubelet, kube-proxy and control plane will not be affected.

Note that kubeadm does not support conversion of the kubelet KubeletConfiguration Kind and this is left as a manual operation to the user, unless the kubelet exposes conversion on an API endpoint or via the CLI, eventually.

While the some of the questions in the PRR could be qualified as relevant to this proposal, KEPs for kubeadm have been established as "out-of-tree" and the PRR does not apply to them.

• 2021-08-30: Initial KEP draft for 1.23 (Alpha).
• 2022-01-10: Update KEP for 1.24 (Beta).
• 2022-05-18: Update KEP for 1.25 (GA).

Deprecation of GA features is always disruptive and hard for the users to adapt to. The usage of the old naming format is documented in a number of places in the kubeadm documentation and a percentage of the users have already integrated it in scripts and tooling.

This KEP proposes a standard deprecation procedure for a GA feature in a Kubernetes component. Users should already be familiar with similar procedures in other components, yet the likelihood of this change causing disruptions is relatively high.

Not using a feature gate has been considered, but the alternative of performing a hard switch to the new format after a GA deprecation period is less desired. A feature gate gives control to the users that are slower to adapt to opt-out during the Beta period and is a standard practice in Kubernetes.

NONE