How we can start supporting reading versioned configuration for all our components after a code move for ComponentConfig to staging

• Summary
  • The current state of the world
    • Current kubelet
    • Current kube-proxy
    • Current kube-scheduler
    • Current kube-controller-manager
    • Current kube-apiserver
    • Current cloud-controller-manager
  • Goals
  • Non-goals
  • Related proposals and further references
• Proposal
  • Migration strategy per component or k8s.io repo
    • k8s.io/apimachinery changes
    • k8s.io/apiserver changes
    • kubelet changes
    • kube-proxy changes
    • kube-scheduler changes
    • k8s.io/controller-manager changes
    • kube-controller-manager changes
    • cloud-controller-manager changes
    • kube-apiserver changes
  • Vendoring
    • Vgo
    • Dep
  • Timeframe and Implementation Order
  • OWNERS files for new packages and repos

Currently all ComponentConfiguration API types are in the core Kubernetes repo. This makes them practically inaccessible for any third-party tool. With more and more generated code being removed from the core Kubernetes repo, vendoring gets even more complicated. Last but not least, efforts to move out kubeadm from the core repo are blocked by this.

This KEP is about creating new staging repos, k8s.io/{component}, which will host the external types of the core components’ ComponentConfig in a top-level config/ package. Internal types will eventually be stored in k8s.io/{component}/pkg/apis/config (but a non-goal for this KEP). Shared types will go to k8s.io/{apimachinery,apiserver,controller-manager}/pkg/apis/config.

• Package: k8s.io/kubernetes/pkg/kubelet/apis/kubeletconfig
• GroupVersionKind: kubelet.config.k8s.io/v1beta.KubeletConfiguration
• Supports reading config from file with flag precedence, well-tested.

• Package: k8s.io/kubernetes/pkg/proxy/apis/kubeproxyconfig
• GroupVersionKind: kubeproxy.config.k8s.io/v1alpha1.KubeProxyConfiguration
• Supports reading config from file, without flag precedence, not tested.
• This API group has its own copy of ClientConnectionConfiguration instead of a shared type.

• Package: k8s.io/kubernetes/pkg/apis/componentconfig
• GroupVersionKind: componentconfig/v1alpha1.KubeSchedulerConfiguration
• Supports reading config from file, without flag precedence, not tested
• This API group has its own copies of ClientConnectionConfiguration & LeaderElectionConfiguration instead of shared types.

• Package: k8s.io/kubernetes/pkg/apis/componentconfig
• GroupVersionKind: componentconfig/v1alpha1.KubeControllerManagerConfiguration
• No support for config from file
• This API group has its own copies of ClientConnectionConfiguration & LeaderElectionConfiguration instead of shared types.

• Doesn’t expose component configuration anywhere
• No support for config from file
• The most similar thing to componentconfig for the API server is the ServerRunOptions struct in k8s.io/kubernetes/cmd/kube-apiserver/app/options/options.go

• Package: k8s.io/kubernetes/pkg/apis/componentconfig
• GroupVersionKind: componentconfig/v1alpha1.CloudControllerManagerConfiguration
• No support for config from file
• This API group has its own copies of ClientConnectionConfiguration & LeaderElectionConfiguration instead of shared types.

• Find a home for the ComponentConfig API types, hosted as a staging repo in the "core" repo that is kubernetes/kubernetes
• Make ComponentConfig API types consumable from projects outside of kube and from different parts of kube itself
  • Resolve dependencies from the external ComponentConfig API types so that everything can depend on them
  • The only dependency of the ComponentConfig API types should be k8s.io/apimachinery
• Split internal types from versioned types
• Remove the monolithic componentconfig/v1alpha1 API group
• Enable the staging bot so that a [https://github.com/kubernetes/](https://github.com/kubernetes/){component} (imported as k8s.io/{component}) repos are published regularly.
• The future API server componentconfig code should be compatible with the proposed structure

• Graduate the API versions
  • For v1.12, we’re working incrementally and will keep the API versions of the existing ComponentConfigs.
• Do major refactoring of the ComponentConfigs. This PR is about code moves, not about re-defining the structure. We will do the latter in follow-ups.
• Change the components to support reading a config file, do flag precedence correctly or add e2e testing
  • Further, the "load-versioned-config-from-flag" feature in this proposal should not be confused with the Dynamic Kubelet Configuration feature. There is nothing in this proposal advocating for that a component should support a similar feature. This is all about the making the one-off “read bytes from a source and unmarshal into internal config state” possible for both the internal components and external consumers of these APIs
  • This is work to be done after this proposal is implemented (for every component but the kubelet which has this implemented already), and might or might not require further, more component-specific proposals/KEPs
• Create a net-new ComponentConfiguration struct for the API server
• Publish the internal types to the new k8s.io/{component} repo
• Support ComponentConfiguration for the cloud-controller-manager, as it’s a stop-gap for the cloud providers to move out of tree. This effort is in progress.
  • When the currently in-tree cloud providers have move out of tree, e.g. to k8s.io/cloud-provider-gcp, they should create their own external and internal types and make the command support loading configuration files.
  • The new repo can reuse the generic types from the to-be-created, k8s.io/controller-manager repo eventually.
  • Meanwhile, the cloud-controller-manager will reference the parts it needs from the main repo, and live privately in cmd/cloud-controller-manager
• Expose defaulting functions for the external ComponentConfig types in k8s.io/{component}/config packages.
  • Defaulting functions will still live local to the component in e.g. k8s.io/kubernetes/pkg/{component}/apis/config/{version} and be registered in the default scheme, but won't be publicly exposed in the k8s.io/{component} repo.
  • The only defaulting functions that are published to non-core repos are for the shared config types, in other words in k8s.io/{apimachinery,apiserver,controller-manager}/pkg/apis/config/{version}, but they are not registered in the scheme by default (with the normal SetDefault_Foo method and the addDefaultingFunc(scheme *runtime.Scheme) { return RegisterDefaults(scheme) } function). Instead, there will be RecommendedDefaultFoo methods exposed, which the consumer of the shared types may or may not manually run in SetDefaults_Bar functions (where Bar wraps Foo as a field).

• Original Google Docs version of this KEP
• Kubernetes Component Configuration by @mikedanese
• Versioned Component Configuration Files by @mtaufen
• Creating a ComponentConfig struct for the API server by @luxas & @sttts
• Related tracking issues in kubernetes/kubernetes:
  • Move KubeControllerManagerConfiguration to pkg/controller/apis/
  • kube-proxy config should move out of ComponentConfig apigroup
  • Move ClientConnectionConfiguration struct to its own api group
  • Move CloudControllerManagerConfiguration to cmd/cloud-controller-manager/apis

• for component in [kubelet kubeproxy kubecontrollermanager kubeapiserver kubescheduler]
  • API group name: {component}.config.k8s.io
  • Kind name: {Component}Configuration
  • Code location:
    • External types: k8s.io/{component}/config/{version}/types.go
      • Like k8s.io/api
    • Internal types: k8s.io/kubernetes/pkg/{component}/apis/config
      • Alternatives, if applicable
        • k8s.io/{component}/pkg/apis/config (preferred, in the future)
        • k8s.io/kubernetes/cmd/{component}/app/apis/config
      • If dependencies allow it, we can move them to k8s.io/{component}/pkg/apis/config/types.go. Not having the external types there is intentional because the pkg/ package tree is considered as "on-your-own-risks / no code compatibility guarantees", while config/ is considered as a code API.
    • Internal scheme package: k8s.io/kubernetes/pkg/{component}/apis/config/scheme/scheme.go
      • The scheme package should expose Scheme *runtime.Scheme, Codecs *serializer.CodecFactory, and AddToScheme(*runtime.Scheme), and have an init() method that runs AddToScheme(Scheme)
  • For the move to a staging repo to be possible, the external API package must not depend on the core repo.
    • Hence, all non-staging repo dependencies need to be removed/resolved before the package move.
  • Conversions from the external type to the internal type will be kept in {internal_api_path}/{external_version}, like for k8s.io/api
    • Defaulting code will be kept in this package, besides the conversion functions.
    • The defaulting code here is specific for the usage of the component, and internal by design. If there are defaulting functions we feel would be generally useful, they might be exposed in k8s.io/{component}/config/{version}/defaults.go as RecommendedDefaultFoo functions that can be used by various consumers optionally.
  • Add at least some kind of minimum validation coverage for the types (e.g. ranges for integer values, URL scheme/hostname/port parsing, DNS name/label/domain validation) in the {internal_api_path}/validation package, targeting the internal API version. The consequence of these validations targeting the internal type is that they can't be exposed in the k8s.io/{component} repo, but publishing more functionality like that is out of scope for this KEP and left as a future task.
• Create a "shared types"-package with structs generic to all or many componentconfig API groups, in the k8s.io/apimachinery, k8s.io/apiserver and k8s.io/controller-manager repos, depending on the struct.
  • Location: k8s.io/{apimachinery,apiserver,controller-manager}/pkg/apis/config/{,v1alpha1}
  • These aren’t "real" API groups, but they have both internal and external versions
  • Conversions and internal types are published to the staging repo.
  • Defaulting functions are of the RecommendedDefaultFoo format and opt-ins for consumers. No defaulting functions are registered in the scheme.
• Remove the monolithic componentconfig/v1alpha1 API group (pkg/apis/componentconfig)
• Enable the staging bot to create the Github repos
• Add API roundtrip (fuzzing), defaulting, conversion, JSON tag consistency and validation tests.

• Not a "real" API group, instead shared packages only with both external and internal types.
• External Package with defaulting (where absolutely necessary) & conversions: k8s.io/apimachinery/pkg/apis/config/v1alpha1/types.go
• Internal Package: k8s.io/apimachinery/pkg/apis/config/types.go
• Structs to be hosted initially:
  • ClientConnectionConfiguration
• Assignee: @hanxiaoshuai

• Not a "real" API group, instead shared packages only with both external and internal types.
• External Package with defaulting (where absolutely necessary) & conversions: k8s.io/apiserver/pkg/apis/config/v1alpha1/types.go
• Internal Package: k8s.io/apiserver/pkg/apis/config/types.go
• Structs to be hosted initially:
  • LeaderElectionConfiguration
  • DebuggingConfiguration
  • later to be created: SecureServingConfiguration, AuthenticationConfiguration, AuthorizationConfiguration, etc.
• Assignee: @hanxiaoshuai

• GroupVersionKind: kubelet.config.k8s.io/v1beta.KubeletConfiguration
• External Package: k8s.io/kubelet/config/v1beta1/types.go
• Internal Package: k8s.io/kubernetes/pkg/kubelet/apis/config/types.go
• Internal Scheme: k8s.io/kubernetes/pkg/kubelet/apis/config/scheme/scheme.go
• Conversions & defaulting (where absolutely necessary) Package: k8s.io/kubernetes/pkg/kubelet/apis/config/v1beta1
• Future Internal Package: k8s.io/kubelet/pkg/apis/config/types.go
• Assignee: @mtaufen

• GroupVersionKind: kubeproxy.config.k8s.io/v1alpha1.KubeProxyConfiguration
• External Package: k8s.io/kube-proxy/config/v1alpha1/types.go
• Internal Package: k8s.io/kubernetes/pkg/proxy/apis/config/types.go
• Internal Scheme: k8s.io/kubernetes/pkg/proxy/apis/config/scheme/scheme.go
• Conversions & defaulting (where absolutely necessary) Package: k8s.io/kubernetes/pkg/proxy/apis/config/v1alpha1
• Future Internal Package: k8s.io/kube-proxy/pkg/apis/config/types.go
• Start referencing ClientConnectionConfiguration from the generic ComponentConfig packages
• Assignee: @m1093782566

• GroupVersionKind: kubescheduler.config.k8s.io/v1alpha1.KubeSchedulerConfiguration
• External Package: k8s.io/kube-scheduler/config/v1alpha1/types.go
• Internal Package: k8s.io/kubernetes/pkg/scheduler/apis/config/types.go
• Internal Scheme: k8s.io/kubernetes/pkg/scheduler/apis/config/scheme/scheme.go
• Conversions & defaulting (where absolutely necessary) Package: k8s.io/kubernetes/pkg/scheduler/apis/config/v1alpha1
• Future Internal Package: k8s.io/kube-scheduler/pkg/apis/config/types.go
• Start referencing ClientConnectionConfiguration & LeaderElectionConfiguration from the generic ComponentConfig packages
• Assignee: @dixudx

• Not a "real" API group, instead shared packages only with both external and internal types.
• External Package with defaulting (where absolutely necessary) & conversions: k8s.io/controller-manager/pkg/apis/config/v1alpha1/types.go
• Internal Package: k8s.io/controller-manager/pkg/apis/config/types.go
• Will host structs:
  • GenericComponentConfiguration (which will be renamed to GenericControllerManagerConfiguration)
• Assignee: @stewart-yu

• GroupVersionKind: kubecontrollermanager.config.k8s.io/v1alpha1.KubeControllerManagerConfiguration
• External Package: k8s.io/kube-controller-manager/config/v1alpha1/types.go
• Internal Package: k8s.io/kubernetes/pkg/controller/apis/config/types.go
• Internal Scheme: k8s.io/kubernetes/pkg/controller/apis/config/scheme/scheme.go
• Conversions & defaulting (where absolutely necessary) Package: k8s.io/kubernetes/pkg/controller/apis/config/v1alpha1
• Future Internal Package: k8s.io/kube-controller-manager/pkg/apis/config/types.go
• Start referencing ClientConnectionConfiguration & LeaderElectionConfiguration from the generic ComponentConfig packages
• Assignee: @stewart-yu

• Not a "real" API group, instead only internal types in cmd/.
• Internal Package: cmd/cloud-controller-manager/app/apis/config/types.go
• We do not plan to publish any external types for this in a staging repo.
• The internal cloud-controller-manager ComponentConfiguration types will reference both k8s.io/controller-manager/pkg/apis/config and k8s.io/kubernetes/pkg/controller/apis/config/
• Assignee: @stewart-yu

• Doesn’t have a ComponentConfig struct at the moment, so there is nothing to move around.
• Eventually, we want to create this ComponentConfig struct, but exactly how to do that is out of scope for this specific proposal.
• See Creating a ComponentConfig struct for the API server for a proposal on how to refactor the API server code to be able to expose the final ComponentConfig structure.

Vgo – as the future standard vendoring mechanism in Golang – supports vgo modules using a k8s.io/{component}/config/go.mod file. Tags of the shape config/vX.Y on k8s.io/{component} will define a version of the component config of that component. Such a tagged module can be imported into a 3rd-party program without inheriting dependencies outside of the k8s.io/{component}/config package.

The k8s.io/{component}/config/go.mod file will look like this:

module "k8s.io/{component}/config"
require (
	"k8s.io/apimachinery" v1.12.0
)

The exact vgo semver versioning scheme we will use is out of scope of this document. We will be able to version the config package independently from the main package k8s.io/{component} if we want to, e.g. to implement correct semver semantics.

Other 3rd-party code can import the config module as usual. Vgo does not add the dependencies from code outside of k8s.io/{component}/config (actually, vgo creates a separate vgo.sum for the config package with the transitive dependencies).

Compare http://github.com/sttts/kubeadm for a test project using latest vgo.

Dep supports the import of sub-packages without inheriting dependencies from outside of the sub-package.

Objective: Done for v1.12

Implementation order:

• Start with copying over the necessary structs to the k8s.io/apiserver and k8s.io/apimachinery shared config packages, with external and internal API versions. The defaulting pattern for these types are of the RecommendedDefaultFoo form, in other words defaulting is not part of the scheme.
• Remove as many unnecessary references to pkg/apis/componentconfig from the rest of the core repo as possible
• Make the types in pkg/apis/componentconfig reuse the newly-created types in k8s.io/apiserver and k8s.io/apimachinery .
• Start with the scheduler as the first component to be moved out.
  • One PR for moving KubeSchedulerConfiguration to staging/src/k8s.io/kube-scheduler/config/v1alpha1/types.go, and the internal type to pkg/scheduler/apis/config/types.go.
    • Set up the conversion for the external type by creating the package pkg/scheduler/apis/config/v1alpha1, without types.go, like how k8s.io/api is set up.
    • This should be a pure code move.
• Set up staging publishing bot (async, non-critical)
• The kubelet, kube-proxy and kube-controller-manager types follow, each one independently.

• Approvers:
  • @kubernetes/api-approvers
  • @sttts
  • @luxas
  • @mtaufen
• Reviewers:
  • @kubernetes/api-reviewers
  • @sttts
  • @luxas
  • @mtaufen
  • @dixudx
  • @stewart-yu