Feature Gates

This page contains an overview of the various feature gates an administrator can specify on different Kubernetes components.

See feature stages for an explanation of the stages for a feature.

Overview

Feature gates are a set of key=value pairs that describe Kubernetes features. You can turn these features on or off using the --feature-gates command line flag on each Kubernetes component.

Each Kubernetes component lets you enable or disable a set of feature gates that are relevant to that component. Use -h flag to see a full set of feature gates for all components. To set feature gates for a component, such as kubelet, use the --feature-gates flag assigned to a list of feature pairs:

--feature-gates=...,GracefulNodeShutdown=true

The following tables are a summary of the feature gates that you can set on different Kubernetes components.

• The "Since" column contains the Kubernetes release when a feature is introduced or its release stage is changed.
• The "Until" column, if not empty, contains the last Kubernetes release in which you can still use a feature gate.
• If a feature is in the Alpha or Beta state, you can find the feature listed in the Alpha/Beta feature gate table.
• If a feature is stable you can find all stages for that feature listed in the Graduated/Deprecated feature gate table.
• The Graduated/Deprecated feature gate table also lists deprecated and withdrawn features.

Feature gates for Alpha or Beta features

Feature gates for graduated or deprecated features

Using a feature

Feature stages

A feature can be in Alpha, Beta or GA stage. An Alpha feature means:

• Disabled by default.
• Might be buggy. Enabling the feature may expose bugs.
• Support for feature may be dropped at any time without notice.
• The API may change in incompatible ways in a later software release without notice.
• Recommended for use only in short-lived testing clusters, due to increased risk of bugs and lack of long-term support.

A Beta feature means:

• Enabled by default.
• The feature is well tested. Enabling the feature is considered safe.
• Support for the overall feature will not be dropped, though details may change.
• The schema and/or semantics of objects may change in incompatible ways in a subsequent beta or stable release. When this happens, we will provide instructions for migrating to the next version. This may require deleting, editing, and re-creating API objects. The editing process may require some thought. This may require downtime for applications that rely on the feature.
• Recommended for only non-business-critical uses because of potential for incompatible changes in subsequent releases. If you have multiple clusters that can be upgraded independently, you may be able to relax this restriction.

A General Availability (GA) feature is also referred to as a stable feature. It means:

• The feature is always enabled; you cannot disable it.
• The corresponding feature gate is no longer needed.
• Stable versions of features will appear in released software for many subsequent versions.

List of feature gates

Each feature gate is designed for enabling/disabling a specific feature:

• AdmissionWebhookMatchConditions: Enable match conditions on mutating & validating admission webhooks.

• APIListChunking: Enable the API clients to retrieve (LIST or GET) resources from API server in chunks.

• APIPriorityAndFairness: Enable managing request concurrency with prioritization and fairness at each server. (Renamed from RequestManagement)

• APIResponseCompression: Compress the API responses for LIST or GET requests.

• APISelfSubjectReview: Activate the SelfSubjectReview API which allows users to see the requesting subject's authentication information. See API access to authentication information for a client for more details.

• APIServerIdentity: Assign each API server an ID in a cluster, using a Lease.

• APIServerTracing: Add support for distributed tracing in the API server. See Traces for Kubernetes System Components for more details.

• AdvancedAuditing: Enable advanced auditing

• AggregatedDiscoveryEndpoint: Enable a single HTTP endpoint /discovery/<version> which supports native HTTP caching with ETags containing all APIResources known to the API server.

• AnyVolumeDataSource: Enable use of any custom resource as the DataSource of a PVC.

• AppArmor: Enable use of AppArmor mandatory access control for Pods running on Linux nodes. See AppArmor Tutorial for more details.

• CPUManager: Enable container level CPU affinity support, see CPU Management Policies.

• CPUManagerPolicyAlphaOptions: This allows fine-tuning of CPUManager policies, experimental, Alpha-quality options This feature gate guards a group of CPUManager options whose quality level is alpha. This feature gate will never graduate to beta or stable.

• CPUManagerPolicyBetaOptions: This allows fine-tuning of CPUManager policies, experimental, Beta-quality options This feature gate guards a group of CPUManager options whose quality level is beta. This feature gate will never graduate to stable.

• CPUManagerPolicyOptions: Allow fine-tuning of CPUManager policies.

• CSIMigrationAzureFile: Enables shims and translation logic to route volume operations from the Azure-File in-tree plugin to AzureFile CSI plugin. Supports falling back to in-tree AzureFile plugin for mount operations to nodes that have the feature disabled or that do not have AzureFile CSI plugin installed and configured. Does not support falling back for provision operations, for those the CSI plugin must be installed and configured. Requires CSIMigration feature flag enabled.

• CSIMigrationGCE: Enables shims and translation logic to route volume operations from the GCE-PD in-tree plugin to PD CSI plugin. Supports falling back to in-tree GCE plugin for mount operations to nodes that have the feature disabled or that do not have PD CSI plugin installed and configured. Does not support falling back for provision operations, for those the CSI plugin must be installed and configured. Requires CSIMigration feature flag enabled.

• CSIMigrationRBD: Enables shims and translation logic to route volume operations from the RBD in-tree plugin to Ceph RBD CSI plugin. Requires CSIMigration and csiMigrationRBD feature flags enabled and Ceph CSI plugin installed and configured in the cluster. This flag has been deprecated in favor of the InTreePluginRBDUnregister feature flag which prevents the registration of in-tree RBD plugin.

• CSIMigrationvSphere: Enables shims and translation logic to route volume operations from the vSphere in-tree plugin to vSphere CSI plugin. Supports falling back to in-tree vSphere plugin for mount operations to nodes that have the feature disabled or that do not have vSphere CSI plugin installed and configured. Does not support falling back for provision operations, for those the CSI plugin must be installed and configured. Requires CSIMigration feature flag enabled.

• CSIMigrationPortworx: Enables shims and translation logic to route volume operations from the Portworx in-tree plugin to Portworx CSI plugin. Requires Portworx CSI driver to be installed and configured in the cluster.

• CSINodeExpandSecret: Enable passing secret authentication data to a CSI driver for use during a NodeExpandVolume CSI operation.

• CSIStorageCapacity: Enables CSI drivers to publish storage capacity information and the Kubernetes scheduler to use that information when scheduling pods. See Storage Capacity. Check the csi volume type documentation for more details.

• CSIVolumeHealth: Enable support for CSI volume health monitoring on node.

• CloudControllerManagerWebhook: Enable webhooks in cloud controller manager.

• CloudDualStackNodeIPs: Enables dual-stack kubelet --node-ip with external cloud providers. See Configure IPv4/IPv6 dual-stack for more details.

• ClusterTrustBundle: Enable ClusterTrustBundle objects and kubelet integration.

• ComponentSLIs: Enable the /metrics/slis endpoint on Kubernetes components like kubelet, kube-scheduler, kube-proxy, kube-controller-manager, cloud-controller-manager allowing you to scrape health check metrics.

• ConsistentHTTPGetHandlers: Normalize HTTP get URL and Header passing for lifecycle handlers with probers.

• ContainerCheckpoint: Enables the kubelet checkpoint API. See Kubelet Checkpoint API for more details.

• ContextualLogging: When you enable this feature gate, Kubernetes components that support contextual logging add extra detail to log output.

• CronJobTimeZone: Allow the use of the timeZone optional field in CronJobs

• CrossNamespaceVolumeDataSource: Enable the usage of cross namespace volume data source to allow you to specify a source namespace in the dataSourceRef field of a PersistentVolumeClaim.

• CustomCPUCFSQuotaPeriod: Enable nodes to change cpuCFSQuotaPeriod in kubelet config.

• CustomResourceValidationExpressions: Enable expression language validation in CRD which will validate customer resource based on validation rules written in the x-kubernetes-validations extension.

• DelegateFSGroupToCSIDriver: If supported by the CSI driver, delegates the role of applying fsGroup from a Pod's securityContext to the driver by passing fsGroup through the NodeStageVolume and NodePublishVolume CSI calls.

• DevicePlugins: Enable the device-plugins based resource provisioning on nodes.

• DisableAcceleratorUsageMetrics: Disable accelerator metrics collected by the kubelet.

• DisableCloudProviders: Disables any functionality in kube-apiserver, kube-controller-manager and kubelet related to the --cloud-provider component flag.

• DisableKubeletCloudCredentialProviders: Disable the in-tree functionality in kubelet to authenticate to a cloud provider container registry for image pull credentials.

• DownwardAPIHugePages: Enables usage of hugepages in downward API.

• DryRun: Enable server-side dry run requests so that validation, merging, and mutation can be tested without committing.

• DynamicResourceAllocation: Enables support for resources with custom parameters and a lifecycle that is independent of a Pod.

• ElasticIndexedJob: Enables Indexed Jobs to be scaled up or down by mutating both spec.completions and spec.parallelism together such that spec.completions == spec.parallelism. See docs on elastic Indexed Jobs for more details.

• EndpointSliceTerminatingCondition: Enables EndpointSlice terminating and serving condition fields.

• EfficientWatchResumption: Allows for storage-originated bookmark (progress notify) events to be delivered to the users. This is only applied to watch operations.

• EventedPLEG: Enable support for the kubelet to receive container life cycle events from the container runtime via an extension to CRI. (PLEG is an abbreviation for “Pod lifecycle event generator”). For this feature to be useful, you also need to enable support for container lifecycle events in each container runtime running in your cluster. If the container runtime does not announce support for container lifecycle events then the kubelet automatically switches to the legacy generic PLEG mechanism, even if you have this feature gate enabled.

• ExecProbeTimeout: Ensure kubelet respects exec probe timeouts. This feature gate exists in case any of your existing workloads depend on a now-corrected fault where Kubernetes ignored exec probe timeouts. See readiness probes.

• ExpandedDNSConfig: Enable kubelet and kube-apiserver to allow more DNS search paths and longer list of DNS search paths. This feature requires container runtime support(Containerd: v1.5.6 or higher, CRI-O: v1.22 or higher). See Expanded DNS Configuration.

• ExperimentalHostUserNamespaceDefaulting: Enabling the defaulting user namespace to host. This is for containers that are using other host namespaces, host mounts, or containers that are privileged or using specific non-namespaced capabilities (e.g. MKNODE, SYS_MODULE etc.). This should only be enabled if user namespace remapping is enabled in the Docker daemon.

• GracefulNodeShutdown: Enables support for graceful shutdown in kubelet. During a system shutdown, kubelet will attempt to detect the shutdown event and gracefully terminate pods running on the node. See Graceful Node Shutdown for more details.

• GracefulNodeShutdownBasedOnPodPriority: Enables the kubelet to check Pod priorities when shutting down a node gracefully.

• GRPCContainerProbe: Enables the gRPC probe method for {Liveness,Readiness,Startup}Probe. See Configure Liveness, Readiness and Startup Probes.

• HonorPVReclaimPolicy: Honor persistent volume reclaim policy when it is Delete irrespective of PV-PVC deletion ordering. For more details, check the PersistentVolume deletion protection finalizer documentation.

• HPAContainerMetrics: Enable the HorizontalPodAutoscaler to scale based on metrics from individual containers in target pods.

• HPAScaleToZero: Enables setting minReplicas to 0 for HorizontalPodAutoscaler resources when using custom or external metrics.

• IPTablesOwnershipCleanup: This causes kubelet to no longer create legacy iptables rules.

• InPlacePodVerticalScaling: Enables in-place Pod vertical scaling.

• InTreePluginAWSUnregister: Stops registering the aws-ebs in-tree plugin in kubelet and volume controllers.

• InTreePluginAzureDiskUnregister: Stops registering the azuredisk in-tree plugin in kubelet and volume controllers.

• InTreePluginAzureFileUnregister: Stops registering the azurefile in-tree plugin in kubelet and volume controllers.

• InTreePluginGCEUnregister: Stops registering the gce-pd in-tree plugin in kubelet and volume controllers.

• InTreePluginOpenStackUnregister: Stops registering the OpenStack cinder in-tree plugin in kubelet and volume controllers.

• InTreePluginPortworxUnregister: Stops registering the Portworx in-tree plugin in kubelet and volume controllers.

• InTreePluginRBDUnregister: Stops registering the RBD in-tree plugin in kubelet and volume controllers.

• InTreePluginvSphereUnregister: Stops registering the vSphere in-tree plugin in kubelet and volume controllers.

• JobMutableNodeSchedulingDirectives: Allows updating node scheduling directives in the pod template of Job.

• JobPodFailurePolicy: Allow users to specify handling of pod failures based on container exit codes and pod conditions.

• JobReadyPods: Enables tracking the number of Pods that have a Ready condition. The count of Ready pods is recorded in the status of a Job status.

• JobTrackingWithFinalizers: Enables tracking Job completions without relying on Pods remaining in the cluster indefinitely. The Job controller uses Pod finalizers and a field in the Job status to keep track of the finished Pods to count towards completion.

• KMSv2: Enables KMS v2 API for encryption at rest. See Using a KMS Provider for data encryption for more details.

• KubeletCredentialProviders: Enable kubelet exec credential providers for image pull credentials.

• KubeletInUserNamespace: Enables support for running kubelet in a user namespace. See Running Kubernetes Node Components as a Non-root User.

• KubeletPodResources: Enable the kubelet's pod resources gRPC endpoint. See Support Device Monitoring for more details.

• KubeletPodResourcesGet: Enable the Get gRPC endpoint on kubelet's for Pod resources. This API augments the resource allocation reporting.

• KubeletPodResourcesGetAllocatable: Enable the kubelet's pod resources GetAllocatableResources functionality. This API augments the resource allocation reporting

• KubeletPodResourcesDynamiceResources: Extend the kubelet's pod resources gRPC endpoint to to include resources allocated in ResourceClaims via DynamicResourceAllocation API. See resource allocation reporting for more details. with informations about the allocatable resources, enabling clients to properly track the free compute resources on a node.

• KubeletTracing: Add support for distributed tracing in the kubelet. When enabled, kubelet CRI interface and authenticated http servers are instrumented to generate OpenTelemetry trace spans. See Traces for Kubernetes System Components for more details.

• LegacyServiceAccountTokenNoAutoGeneration: Stop auto-generation of Secret-based service account tokens.

• LegacyServiceAccountTokenTracking: Track usage of Secret-based service account tokens.

• LocalStorageCapacityIsolationFSQuotaMonitoring: When LocalStorageCapacityIsolation is enabled for local ephemeral storage and the backing filesystem for emptyDir volumes supports project quotas and they are enabled, use project quotas to monitor emptyDir volume storage consumption rather than filesystem walk for better performance and accuracy.

• LogarithmicScaleDown: Enable semi-random selection of pods to evict on controller scaledown based on logarithmic bucketing of pod timestamps.

• LoggingAlphaOptions: Allow fine-tuing of experimental, alpha-quality logging options.

• LoggingBetaOptions: Allow fine-tuing of experimental, beta-quality logging options.

• MatchLabelKeysInPodTopologySpread: Enable the matchLabelKeys field for Pod topology spread constraints.

• MaxUnavailableStatefulSet: Enables setting the maxUnavailable field for the rolling update strategy of a StatefulSet. The field specifies the maximum number of Pods that can be unavailable during the update.

• MemoryManager: Allows setting memory affinity for a container based on NUMA topology.

• MemoryQoS: Enable memory protection and usage throttle on pod / container using cgroup v2 memory controller.

• MinDomainsInPodTopologySpread: Enable minDomains in Pod topology spread constraints.

• MinimizeIPTablesRestore: Enables new performance improvement logics in the kube-proxy iptables mode.

• MixedProtocolLBService: Enable using different protocols in the same LoadBalancer type Service instance.

• MultiCIDRRangeAllocator: Enables the MultiCIDR range allocator.

• MultiCIDRServiceAllocator: Track IP address allocations for Service cluster IPs using IPAddress objects.

• NetworkPolicyStatus: Enable the status subresource for NetworkPolicy objects.

• NewVolumeManagerReconstruction: Enables improved discovery of mounted volumes during kubelet startup. Since this code has been significantly refactored, we allow to opt-out in case kubelet gets stuck at the startup or is not unmounting volumes from terminated Pods. Note that this refactoring was behind SELinuxMountReadWriteOncePod alpha feature gate in Kubernetes 1.25.

  Before Kubernetes v1.25, the kubelet used different default behavior for discovering mounted volumes during the kubelet startup. If you disable this feature gate (it's enabled by default), you select the legacy discovery behavior.

  In Kubernetes v1.25 and v1.26, this behavior toggle was part of the SELinuxMountReadWriteOncePod feature gate.

• NodeInclusionPolicyInPodTopologySpread: Enable using nodeAffinityPolicy and nodeTaintsPolicy in Pod topology spread constraints when calculating pod topology spread skew.

• NodeLogQuery: Enables querying logs of node services using the /logs endpoint.

• NodeOutOfServiceVolumeDetach: When a Node is marked out-of-service using the node.kubernetes.io/out-of-service taint, Pods on the node will be forcefully deleted if they can not tolerate this taint, and the volume detach operations for Pods terminating on the node will happen immediately. The deleted Pods can recover quickly on different nodes.

• NodeSwap: Enable the kubelet to allocate swap memory for Kubernetes workloads on a node. Must be used with KubeletConfiguration.failSwapOn set to false. For more details, please see swap memory

• OpenAPIEnums: Enables populating "enum" fields of OpenAPI schemas in the spec returned from the API server.

• OpenAPIV3: Enables the API server to publish OpenAPI v3.

• PDBUnhealthyPodEvictionPolicy: Enables the unhealthyPodEvictionPolicy field of a PodDisruptionBudget. This specifies when unhealthy pods should be considered for eviction. Please see Unhealthy Pod Eviction Policy for more details.

• PodDeletionCost: Enable the Pod Deletion Cost feature which allows users to influence ReplicaSet downscaling order.

• PodAndContainerStatsFromCRI: Configure the kubelet to gather container and pod stats from the CRI container runtime rather than gathering them from cAdvisor. As of 1.26, this also includes gathering metrics from CRI and emitting them over /metrics/cadvisor (rather than having cAdvisor emit them directly).

• PodDisruptionConditions: Enables support for appending a dedicated pod condition indicating that the pod is being deleted due to a disruption.

• PodHasNetworkCondition: Enable the kubelet to mark the PodHasNetwork condition on pods.

• PodSchedulingReadiness: Enable setting schedulingGates field to control a Pod's scheduling readiness.

• PodSecurity: Enables the PodSecurity admission plugin.

• ProbeTerminationGracePeriod: Enable setting probe-level terminationGracePeriodSeconds on pods. See the enhancement proposal for more details.

• ProcMountType: Enables control over the type proc mounts for containers by setting the procMount field of a SecurityContext.

• ProxyTerminatingEndpoints: Enable the kube-proxy to handle terminating endpoints when ExternalTrafficPolicy=Local.

• QOSReserved: Allows resource reservations at the QoS level preventing pods at lower QoS levels from bursting into resources requested at higher QoS levels (memory only for now).

• ReadWriteOncePod: Enables the usage of ReadWriteOncePod PersistentVolume access mode.

• RecoverVolumeExpansionFailure: Enables users to edit their PVCs to smaller sizes so as they can recover from previously issued volume expansion failures. See Recovering from Failure when Expanding Volumes for more details.

• RemainingItemCount: Allow the API servers to show a count of remaining items in the response to a chunking list request.

• RemoveSelfLink: Sets the .metadata.selfLink field to blank (empty string) for all objects and collections. This field has been deprecated since the Kubernetes v1.16 release. When this feature is enabled, the .metadata.selfLink field remains part of the Kubernetes API, but is always unset.

• RetroactiveDefaultStorageClass: Allow assigning StorageClass to unbound PVCs retroactively.

• RotateKubeletServerCertificate: Enable the rotation of the server TLS certificate on the kubelet. See kubelet configuration for more details.

• SELinuxMountReadWriteOncePod: Speeds up container startup by allowing kubelet to mount volumes for a Pod directly with the correct SELinux label instead of changing each file on the volumes recursively. The initial implementation focused on ReadWriteOncePod volumes.

• SeccompDefault: Enables the use of RuntimeDefault as the default seccomp profile for all workloads. The seccomp profile is specified in the securityContext of a Pod and/or a Container.

• SecurityContextDeny: This gate signals that the SecurityContextDeny admission controller is deprecated.

• ServerSideApply: Enables the Sever Side Apply (SSA) feature on the API Server.

• ServerSideFieldValidation: Enables server-side field validation. This means the validation of resource schema is performed at the API server side rather than the client side (for example, the kubectl create or kubectl apply command line).

• ServiceInternalTrafficPolicy: Enables the internalTrafficPolicy field on Services

• ServiceIPStaticSubrange: Enables a strategy for Services ClusterIP allocations, whereby the ClusterIP range is subdivided. Dynamic allocated ClusterIP addresses will be allocated preferently from the upper range allowing users to assign static ClusterIPs from the lower range with a low risk of collision. See Avoiding collisions for more details.

• SizeMemoryBackedVolumes: Enable kubelets to determine the size limit for memory-backed volumes (mainly emptyDir volumes).

• StableLoadBalancerNodeSet: Enables less load balancer re-configurations by the service controller (KCCM) as an effect of changing node state.

• StatefulSetStartOrdinal: Allow configuration of the start ordinal in a StatefulSet. See Start ordinal for more details.

• StorageVersionAPI: Enable the storage version API.

• StorageVersionHash: Allow API servers to expose the storage version hash in the discovery.

• TopologyAwareHints: Enables topology aware routing based on topology hints in EndpointSlices. See Topology Aware Hints for more details.

• TopologyManager: Enable a mechanism to coordinate fine-grained hardware resource assignments for different components in Kubernetes. See Control Topology Management Policies on a node.

• TopologyManagerPolicyAlphaOptions: Allow fine-tuning of topology manager policies, experimental, Alpha-quality options. This feature gate guards a group of topology manager options whose quality level is alpha. This feature gate will never graduate to beta or stable.

• TopologyManagerPolicyBetaOptions: Allow fine-tuning of topology manager policies, experimental, Beta-quality options. This feature gate guards a group of topology manager options whose quality level is beta. This feature gate will never graduate to stable.

• TopologyManagerPolicyOptions: Allow fine-tuning of topology manager policies,

• UserNamespacesStatelessPodsSupport: Enable user namespace support for stateless Pods.

• ValidatingAdmissionPolicy: Enable ValidatingAdmissionPolicy support for CEL validations be used in Admission Control.

• VolumeCapacityPriority: Enable support for prioritizing nodes in different topologies based on available PV capacity.

• WatchBookmark: Enable support for watch bookmark events.

• WatchList : Enable support for streaming initial state of objects in watch requests.

• WinDSR: Allows kube-proxy to create DSR loadbalancers for Windows.

• WinOverlay: Allows kube-proxy to run in overlay mode for Windows.

• WindowsHostNetwork: Enables support for joining Windows containers to a hosts' network namespace.

• WindowsHostProcessContainers: Enables support for Windows HostProcess containers.

What's next

• The deprecation policy for Kubernetes explains the project's approach to removing features and components.
• Since Kubernetes 1.24, new beta APIs are not enabled by default. When enabling a beta feature, you will also need to enable any associated API resources. For example, to enable a particular resource like storage.k8s.io/v1beta1/csistoragecapacities, set --runtime-config=storage.k8s.io/v1beta1/csistoragecapacities. See API Versioning for more details on the command line flags.