KEP-5836: Scheduler Preemption for In-Place Pod Resize

Release Signoff Checklist
Summary
Motivation
- Goals
- Non-Goals
Proposal
- User Stories (Optional)
- Risks and Mitigations
Design Details
Production Readiness Review Questionnaire
Implementation History
Drawbacks
Alternatives

Release Signoff Checklist

Items marked with (R) are required prior to targeting to a milestone / release.

(R) Enhancement issue in release milestone, which links to KEP dir in kubernetes/enhancements (not the initial KEP PR)
(R) KEP approvers have approved the KEP status as implementable
(R) Design details are appropriately documented
(R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
- e2e Tests for all Beta API Operations (endpoints)
- (R) Ensure GA e2e tests meet requirements for Conformance Tests
- (R) Minimum Two Week Window for GA e2e tests to prove flake free
(R) Graduation criteria is in place
- (R) all GA Endpoints must be hit by Conformance Tests within one minor version of promotion to GA
(R) Production readiness review completed
(R) Production readiness review approved
“Implementation History” section is up-to-date for milestone
User-facing documentation has been created in kubernetes/website, for publication to kubernetes.io
Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes

Summary

In-Place Pod Resize reached GA in 1.35, allowing pod resizing without restarts. However, unlike traditional pod recreation, in-place resizes that exceed node capacity do not currently trigger the preemption of lower-priority pods. Instead, they remain in a Deferred resize status and require manual intervention. We propose that the Scheduler actively creates space for resize requests by preempting lower-priority pods, following the same preemption logic used when scheduling new pods.

Motivation

The introduction of In-Place Pod Resize gives platform teams new ways to optimize cloud resources without restarting pods, marking a significant shift in how Kubernetes operates. This is especially powerful when In-Place Pod Resize is integrated with higher-level autoscaling controllers such as VPA’s new InPlaceOrRecreate mode. However, users currently lack a way to control scale-up behavior when a node lacks the capacity to fulfill the request. Consequently, workloads may face disruptions such as being moved to a larger node, or suffering an OOM-kill because memory could not be scaled up in-place.

Scheduler preemption eliminates the gap by introducing a configurable ability to free up capacity on a fully-utilized node to allow the scale up to succeed in-place.

In production Kubernetes environments, cluster administrators strive to maximize resource utilization and efficiency. A common strategy is to bin-pack unused capacity on not-yet-full nodes with lower-priority workloads, such as batch jobs, background data processing, or best-effort tasks.

Without scheduler preemption for in-place resizing, this created a major operational dilemma. If lower-priority workloads consumed the remaining headroom on a node, higher-priority applications running on that same node would become blocked (Deferred) when they needed to scale up to handle sudden traffic surges or memory spikes. Operators were forced to choose between running low-utilization clusters with idle buffer capacity or risking that critical workloads could not resize when needed.

With scheduler preemption for in-place Pod resize, you can confidently bin-pack unused space across your clusters with lower-priority workloads without worrying about them degrading higher-priority Pods or blocking their scale-up requests. If a high-priority workload requires an in-place resize that exceeds available node capacity, the scheduler automatically preempts the lower-priority Pods to clear headroom. You achieve high cluster utilization and cost efficiency while preserving the responsiveness and reliability of critical services.

Goals

Introduce the ability to instruct the Scheduler to preempt lower-priority pods to make room for a Deferred resize.
Keep the preemption behavior for Deferred resizes aligned with the preemption behavior for newly created pods.

Non-Goals

Perform the entire scheduling cycle (including binding, DRA, etc) on Deferred resizes; the Scheduler should trigger preemption and nothing more.
Coordinate with higher-level autoscalers such as VPA or CA on the preemption decision. This is considered out of scope for this KEP, but may be considered as a future enhancement.
Changing the source of truth for resize allocations. The Kubelet remains responsible for evaluating incoming resize requests, and the scheduler must account for cases where the Kubelet’s runtime decisions differ from the scheduler’s calculated predictions.

Proposal

In-Place Pod Resize requests that result in a Deferred resize status trigger scheduler preemption, keeping the behavior aligned with scheduler preemption for newly created pods.

The Scheduler watches for Deferred resizes. To make room for the resize, it tries to preempt (evict) lower-priority pods to make the resize possible. The Kubelet today watches for pod removals and reacts to the eviction by reattempting the previously Deferred resize.

User Stories (Optional)

Story 1: Pods in the Deferred resize status no longer require manual intervention.

Users are enabled to instruct the system to automatically evict lower-priority workloads rather than the resize request sitting there indefinitely.

Story 2: Reduction of disruption for critical workloads.

The Scheduler would preferentially evict lower-priority pods before an external controller, such as VPA’s InPlaceOrRecreate mode, would be forced to fall back to the more disruptive action of evicting and recreating the critical pod itself.

Story 3: Driving cluster autoscaling

If a pod managed by a workload controller is preempted, the corresponding controller will likely create new pods in response to the preemption, which will appear in the scheduling queue. By moving the resource deficit from an indefinite Deferred status into the active scheduling queue, preempted pods will naturally trigger cluster autoscalers to provision new capacity.

Risks and Mitigations

Performance impact

If there are a significant number of deferred pods, periodic processing of those pods can affect scheduling throughput. To mitigate this, deferred pods are only reconsidered if something on the node has changed that could make preemption succeed.

Interaction with workload-aware preemption

With workload-aware preemption, the resize may end up triggering preemption of a potentially large workload. In these cases, the preempted workload would always be lower priority than the pod that is being resized, so we consider this working as intended.

Race between a Deferred resize and a new higher-priority pod

From the scheduler’s view, once the spec is updated, the resources are already reserved. It doesn’t matter from the scheduler perspective whether the resize has been actuated by the Kubelet yet.

This means that if a new, higher-priority pod comes in and the only way to fit it is by taking the space the Deferred pod is trying to grow into, the standard preemption logic applies. This might mean the resizing pod itself gets evicted if it’s the best victim candidate. While we would rather not kill pods unnecessarily, this behavior is consistent with the rest of the scheduler’s logic.

Design Details

How Deferred Resizes Integrate into the Scheduling Queue

The scheduling queue is shared between new pods and Deferred pods, using existing queuing logic to sort them by their current scheduling priority.

See alternative considerations for separate scheduling queues here and prioritized resize logic here.

Detecting Deferred Resizes in UpdatePod

The UpdatePod event handler is responsible for maintaining the state of the scheduling queue based on changes to the pod objects, and includes logic to handle pods with a Deferred resize:

If the deferred condition is added to a pod (or PodResizeSchedulerFeasible is removed while the pod remains Deferred): The Scheduler adds the pod to the scheduling queue if it has PodResizePending=True (reason: Deferred) and does not have the PodResizeSchedulerFeasible condition set (see Synchronizing State via the PodResizeSchedulerFeasible Condition).
If the deferred condition is removed: The Scheduler removes the pod from whatever queue it’s currently in. This can occur if the user reverts the resize request or if the Kubelet is able to satisfy the resize request due to other pods being removed or downsized.

Queueing Hints in NodeResourcesFit

If the Deferred pod’s spec (desired) resources change (which can happen if the desired resources are updated before the previously requested resize can complete), the Scheduler updates its view of the pod by leveraging NodeResourcesFit queueing hints (QHints). The NodeResourcesFit QHint for target pod updates detects this change and automatically moves the pod back to the active queue if resource changes are detected.

Scheduler Restart and State Recovery

Because the scheduler’s internal activeQ and Unschedulable pods pool are maintained in-memory, a scheduler restart clears the state of all pods currently undergoing preemption for a resize. To prevent these pods from remaining in a Deferred state indefinitely, the scheduler must proactively re-identify them upon startup.

Cold-Start Re-Queueing: As part of initialization, the scheduler adds deferred pods that do not already have the PodResizeSchedulerFeasible condition set to the scheduling queue in the AddPod event handler (see Synchronizing State via the PodResizeSchedulerFeasible Condition).
Re-evaluation: Once in the queue, these pods undergo the standard evaluation flow (Node Fit -> Preemption).

Conflicting Sources of Truth for the Pod

Adding a snapshot of a deferred pod directly to the scheduling queue means the pod exists in two places simultaneously: the scheduling cache and the scheduling queue. Any subsequent pod updates (such as resize requests, priority updates, or label changes) must be propagated to both the cache and the queue in parallel.

The primary risk with this approach is ensuring that no existing or future scheduler logic implicitly assumes a pod has a single unique representation or reference. We must audit the scheduler codebase to ensure that:

Pod updates correctly target both locations. The pod should be updated in the cache first, then in the queue, to make any divergences between the two more predictable.
No downstream scheduling or preemption logic relies on strict pointer equality of pod objects across the cache and queue boundaries.
Resource double-counting is avoided on the target node by excluding the pod’s additional resources during resource fit calculations as described in the section below.

Even with these considerations in mind, asynchronous event delivery and concurrent pod updates mean that the pod snapshot popped from the scheduling queue can occasionally carry resource requests that temporarily diverge from the cache’s snapshot. In Alpha, NodeResourcesFit amortized this discrepancy at runtime by assuming the pod’s resources as the maximum between the queued and cached snapshots. For Beta, we eliminate this divergence at the source for deferred pods by refreshing the queued pod snapshot directly from the node’s cache snapshot at the start of the scheduling cycle, ensuring all plugins evaluate the exact same immutable pod instance.

If the refreshed pod in the cache is no longer Deferred (for example, because the Kubelet already actuated the resize or the resize was reverted), its resources already fit on the node in the cache snapshot. The pod therefore succeeds in the scheduling cycle, skips setting the PodResizeSchedulerFeasible condition during Bind, and is removed from the scheduling queue (see Handling Node Evaluation Results).

See alternative considerations for how to handle this without dual representation (such as using a dynamic PodGetter interface) here.

Processing Deferred Resizes in the Scheduling Cycle

When processing a pod with a Deferred resize, the scheduler runs it through a modified scheduling cycle. Historically, the scheduler operated under the fundamental assumption that every pod in the scheduling pipeline was an unbound pod seeking initial placement across the cluster. With in-place pod resize, this assumption no longer holds: the scheduler must now also process already-bound, running pods to evaluate capacity expansion and preemption limited to only their assigned node.

Bypassing Irrelevant Constraints via `RelevancePlugin`

To bypass plugins that are irrelevant to deferred pods without hardcoding plugin lists or scattering ad-hoc bypass checks across individual plugins, we introduce the RelevancePlugin extension point interface in the scheduling framework:

// RelevancePlugin allows a scheduling plugin to declare whether it is
// relevant for evaluating a given pod during its scheduling cycle.
type RelevancePlugin interface {
    Plugin
    // IsRelevantForPodResize returns a Status indicating whether this plugin is relevant
    // for evaluating the given resizing pod. If it returns fwk.Skip, the framework skips all
    // extension points for this plugin during the scheduling cycle and preemption.
    IsRelevantForPodResize(ctx context.Context, cycleState *CycleState, pod *v1.Pod) *fwk.Status
}

At the beginning of a scheduling cycle for a resizing pod, IsRelevantForPodResize is evaluated so the scheduling framework runtime knows which plugins should participate in evaluating the pod.

If a plugin does not implement RelevancePlugin, the framework considers it as only handling standard unscheduled pods (no node assigned). For a resizing pod, un-scoped plugins automatically default to being skipped (fwk.Skip) across all extension points (PreFilter, Filter, Score, Reserve, Permit, PreBind, Bind, PostFilter), including during the preemption algorithm’s dry-run simulations. Wake-up events and Queueing Hints for plugins are also filtered based on the applicability check, preventing placement plugins from waking up deferred pods on unrelated node updates.

Only the plugins responsible for evaluating deferred resize pods (NodeName, NodeResourcesFit, DeferredPodScheduling, and DefaultPreemption) implement RelevancePlugin to return success for resizing pods.

This eliminates the need to add resize checks to plugins irrelevant to deferred pods. Furthermore, custom third-party plugins do not need to be updated to ignore deferred pods, as they will only be invoked for resizing pods if they explicitly implement RelevancePlugin.

`NodeName` Plugin: Node Restriction via the PreFilter Phase

The NodeName plugin is adjusted to implement the PreFilter phase. Because the deferred resize pod is already bound to and running on a host, the scheduler limits the candidate search space at the start of the scheduling cycle. If a pod specifies Spec.NodeName, its PreFilter returns a PreFilterResult containing only that node. This restricts the candidate node evaluation search space to just the pod’s currently assigned node.

`NodeResourcesFit` Plugin: Calculating Resource Fit in the Filter Phase

The NodeResourcesFit plugin will run its PreFilter and Filter phases for deferred resize pods. It runs a modified resource-fit check to specially handle Deferred pods.

To prevent the double-counting of resources when a deferred-resize pod is evaluated for scheduling or preemption fit on its assigned node, the NodeResourcesFit plugin is adjusted to leverage the scheduler cache’s existing resource accounting logic. If the pod is already assigned to the node being evaluated, the scheduler’s cache has already factored the pod’s resource footprint into the node’s aggregated requested resources (nodeInfo.Requested), so the plugin uses nodeInfo.Requested directly, without adding the pod’s resource requests again.

Handling Node Evaluation Results

In most cases, this restricted search results in a resource fit error, naturally triggering the scheduler’s preemption pathway in PostFilter to identify and evict victims on the assigned node. As in standard scheduling, once PostFilter identifies and initiates eviction of the victim pods, the resizing pod returns to unschedulablePods while the victims terminate. Because NodeResourcesFit registers the AssignedPodDelete QueueingHint for the pod, once the preempted victims finish terminating and are deleted from the API server, the pod is moved from unschedulablePods back to activeQ for a subsequent scheduling cycle.

When the resize fits in Filter (either immediately, or on the subsequent scheduling cycle after preempted victims have finished terminating and been removed from the node), the pod progresses through the Permit and Bind phases to complete the scheduling cycle. Because the pod is already bound to a node, DefaultBinder is skipped. Instead, the DeferredPodScheduling plugin implements the Bind extension point to set the PodResizeSchedulerFeasible condition (with status: True and reason: SchedulerPreemptionCompleted) on the pod’s status, and the pod is removed from the scheduling queue. If the refreshed pod in the cache was already no longer Deferred, DeferredPodScheduling.Bind is a no-op and does not patch the condition.

Because the scheduler cache accounts for the pod using max(desired, allocated, actual), the required capacity remains reserved in nodeInfo.Requested. However, even after the pod passes Filter, sets PodResizeSchedulerFeasible=True in Bind, and exits the scheduling queue, race conditions (such as a competing pod scaling up on the node, the deferred pod itself scaling up further while in-flight, a pod binding directly to the node, or node allocatable capacity decreasing) can still prevent the Kubelet from actuating the resize. When the Kubelet re-evaluates the pod and determines that the resize still cannot be admitted, it removes the PodResizeSchedulerFeasible condition while keeping PodResizePending=True (reason: Deferred), which triggers the scheduler’s updatePod event handler to re-enqueue the pod into activeQ.

The details of this condition and how it resolves reconsideration race conditions are described in Synchronizing State via the PodResizeSchedulerFeasible Condition.

`DefaultPreemption` Plugin: Preemption Mechanism Adjustments in the PostFilter Phase

Only the DefaultPreemption plugin runs during the PostFilter phase for deferred resize pods; other PostFilter plugins are bypassed via IsRelevantForPodResize(ctx, cycleState, pod).

The modifications to the NodeName, NodeResourcesFit, and DeferredPodScheduling plugins ensure three important components of the desired behavior:

Victim Selection Constraint: The search for preemption victims is isolated exclusively to the deferred pod’s active node. This is already done by the NodeName plugin in the PreFilter phase. However, because of workload-aware group scheduling, preempting a victim on the target node might still trigger the cascaded preemption of related pods on other nodes.
Policy Enforcement: If preemption is disabled on the node, the DeferredPodScheduling plugin rejects the node with UnschedulableAndUnresolvable during the Filter phase, causing DefaultPreemption to skip victim selection.
Resource Accounting Adjustment: During the preemption evaluation, the fit plugin is adjusted to specially handle the deferred pod’s resources, as described in NodeResourcesFit Plugin: Calculating Resource Fit in the Filter Phase.

Similar to the Filter phase, the preemption and reprieve logic runs only the resource-fit checks, skipping irrelevant constraints like affinity and topology spread constraints.

Preventing Unnecessary Victim Preemption

When evaluating resource availability, the scheduler and Kubelet use different accounting models today:

The Kubelet determines resource fit in the event of a resize by evaluating max(desired, allocated, actual) only for the pod undergoing admission or resize, while evaluating max(allocated, actual) for all other pods on the node (ignoring their unfulfilled desired requests).
The scheduler assumes the resources of all pods on the node to be max(desired, allocated, actual). This ensures the node does not end up overcommitted in the event that Kubelet actuates a resize while a new pod is being scheduled.

However, this divergence in accounting can result in unnecessary preemption during scheduler evaluation of Deferred pods when multiple pods on the same host have pending resizes.

Consider a scenario with 4 pods on a node with allocatable capacity $2X$, all in a Deferred resize state:

Pod 1 (high priority): desired = X, allocated = X/2, actual = X/2 (needs $+X/2$ additional capacity)
Pods 2, 3, 4 (low priority): desired = 2X, allocated = X/2, actual = X/2

From the node’s physical allocation standpoint, the current allocated usage is X/2 + X/2 + X/2 + X/2 = 2X. Evicting just one low-priority pod frees X/2 of allocated capacity, which is physically sufficient for Pod 1 to expand from X/2 to X (total 2X <= 2X).

If the scheduler evaluated all pods using max(desired, allocated, actual) during preemption reprieve:

Preemption logic first removes all candidate pods, so it sees the node is using X (Pod 1’s max).
The scheduler first tries to reprieve Pod 2, which is assumed to have max(2X, X/2, X/2) resources. 2X + X = 3X, which is more than node allocatable 2X. The scheduler determines this pod cannot be reprieved.
The scheduler goes through the same process for Pods 3 and 4, and determines that they cannot be reprieved.

All 3 pods (Pod 2, 3, and 4) would be preempted, even though it was only necessary to preempt one of them.

To prevent unnecessary victim preemption when evaluating a Deferred resize pod, we adjust how candidate victim pods are added back to copy during the reprieve phase:

First, preemption logic removes all candidate victim pods
During reprieve, when adding candidate victim pods back to the NodeInfo copy, the scheduler adds each candidate victim back with its resource requests set to max(allocated, actual) (ignoring unfulfilled desired requests).

With this adjustment, only the resizing preemptor pod uses max(desired, allocated, actual) and candidate victim pods use max(allocated, actual) during reprieve (similar to Kubelet’s calculations). The above scenario now results in:

Preemption logic first removes all candidate pods, so it sees the node is using X (Pod 1’s max).
The scheduler first tries to reprieve Pod 2, which is added back to the NodeInfo copy with max(X/2, X/2) resources. X/2 + X = 1.5X, which is less than node allocatable 2X, so Pod 2 is reprieved.
The scheduler tries to reprieve Pod 3, which is added back with max(X/2, X/2) resources. X/2 + 1.5X = 2X, which still fits within the node allocatable 2X, so Pod 3 is reprieved.
The scheduler tries to reprieve Pod 4, which is added back with max(X/2, X/2) resources. X/2 + 2X = 2.5X, which is more than node allocatable 2X, so Pod 4 is not reprieved.

This results in only 1 pod (Pod 4) being preempted.

If reprieve determines that no victims need to be preempted (all candidate victims are reprieved because the resizing pod fits alongside their max(allocated, actual) resources), the scheduler treats the deferred resize as fitting on the node without preemption, proceeds through Bind to set the PodResizeSchedulerFeasible condition, and removes the pod from the scheduling queue. If the Kubelet actuates a reprieved lower-priority pod’s resize in the meantime and consumes the available headroom before the higher-priority preemptor’s resize is actuated, the Kubelet removes PodResizeSchedulerFeasible when re-evaluating the higher-priority pod, causing the scheduler to re-insert it into the scheduling queue (see Synchronizing State via the PodResizeSchedulerFeasible Condition).

Preventing Additional Preemption Across Grace Periods and Restarts

When an in-place pod resize triggers preemption, lower-priority victim pods on the assigned node are scheduled for deletion. Because pods typically have a termination grace period and termination cleanup hooks, victim pods remain present on the node in a terminating state for a non-trivial duration while the resizing pod waits in unschedulablePods. During this window, two scenarios can cause the scheduler to re-evaluate the resizing pod before the victim has finished terminating, which could otherwise trigger redundant victim evictions to satisfy the same resize deficit:

Long Grace Periods and Early Queue Wake-Ups: While a victim pod is terminating in unschedulablePods, another event on the node (such as one of multiple preempted victims finishing termination earlier than another, an unrelated pod being deleted or downsized on the node, or the periodic unschedulablePods flush) can move the resizing pod back into activeQ. Because the remaining terminating victim’s resources are still accounted for on the node until full deletion, the scheduler encounters a fit error again and re-invokes preemption.
Scheduler Restarts: The scheduler crashes or restarts while a victim pod is terminating. Upon startup, AddPod re-enqueues the Deferred pod (since it has not yet passed Filter and entered Bind to receive PodResizeSchedulerFeasible), and the scheduler re-evaluates the pod before the victim has finished terminating.

In standard initial pod placement, the scheduler prevents duplicate preemption via a two-stage mechanism in DefaultPreemption. When an unscheduled pod preempts victims on a node, the scheduler records the pod’s status.NominatedNodeName. On subsequent scheduling attempts, the default preemption plugin inspects NominatedNodeName. If it finds any lower-priority pod on the nominated node that is marked as terminating due to scheduler preemption (recorded by the DisruptionTarget condition), it suppresses further preemption. Because NominatedNodeName and the victim’s DisruptionTarget condition are persisted in etcd, a newly restarted scheduler reads them from cache and immediately suppresses duplicate preemptions.

For in-place resize pods, the pod is already bound and NominatedNodeName is not set. To prevent double preemption for resizing pods, we modify the default preemption plugin to instead evaluate the pod’s spec.NodeName when NominatedNodeName is empty. If there exists an already-terminating lower-priority pod on the node that the pod is bound to, the default preemption plugin safely suppresses duplicate preemption cycles across long victim termination grace periods and scheduler restarts.

Summary of Scheduling Cycle Flow

Taking all the above into account, the logic for processing a Deferred resize is as follows:

Identify Deferred Status: Confirm the pod has a Deferred resize (without PodResizeSchedulerFeasible set) and is already bound to a node. Enqueue it in the Scheduling queue.
Queued Pod Refresh: Refresh the queued pod snapshot directly from the node cache snapshot to eliminate divergence.
Evaluate Fit: Perform node evaluation restricted to the current node.
- The node evaluation logic should run only the logic for the resource-fit and preemption policy check, skipping filters that are relevant only to initial scheduling, such as affinity/anti-affinity rules and topology spread constraints.
- The resource-fit logic is adjusted to specially handle the deferred pod’s resources, as described in NodeResourcesFit Plugin: Calculating Resource Fit in the Filter Phase.
- If preemption is disabled on the node, DeferredPodScheduling.Filter returns UnschedulableAndUnresolvable, skipping preemption.
- If the resize fits on the node, the pod proceeds directly to the Bind phase.
Trigger Preemption & Calculate Victims: If a FitError occurs, initiate the Scheduler preemption logic in PostFilter. Identify suitable preemption victims, with the search restricted to the current node. The resource-fit logic is adjusted to specially handle the deferred pod’s resources, as described in NodeResourcesFit Plugin: Calculating Resource Fit in the Filter Phase, and irrelevant constraints such as affinity and topology spread are skipped. The pod is placed in unschedulablePods while any preempted victims terminate (or while waiting for node capacity if no victims were found).
Victim Termination & Second Scheduling Cycle: Once preempted victims finish terminating and are deleted from the API server, NodeResourcesFit’s AssignedPodDelete QueueingHint moves the deferred pod from unschedulablePods back to activeQ. On the subsequent scheduling cycle, if the freed capacity is sufficient, Filter succeeds and the pod proceeds to the Bind phase where DeferredPodScheduling.Bind sets the PodResizeSchedulerFeasible condition (with reason: SchedulerPreemptionCompleted) and removes the pod from the scheduling queue.
Kubelet Retry & Reconsideration: Once a preempted victim pod finishes terminating and is removed, the Kubelet retries the Deferred resize. If the resize fits, the Kubelet actuates it and removes both PodResizePending and PodResizeSchedulerFeasible. If the resize still cannot fit, the Kubelet removes PodResizeSchedulerFeasible while keeping PodResizePending=True (reason: Deferred), causing the scheduler’s updatePod handler to re-enqueue the pod into activeQ for reevaluation (see Synchronizing State via the PodResizeSchedulerFeasible Condition for more details).

Scheduler Resource Reservation

After a preemption occurs, we want to ensure that the space is not taken up by another pod that has not yet been scheduled. However, due to the fact that the Scheduler assumes that the resources being used by a pod are equal to max(desired, allocated, actual), and that the new requested resizes are reflected in the pod’s spec, the scheduler naturally already reserves the space for the resizing pod. This means that the Scheduler does not need to take any special action to ensure the resources are reserved.

Kubelet-Scheduler Preemption Interaction

The Kubelet monitors pod removals, including evictions, and automatically retries Deferred resizes as soon as it detects that a victim pod has finished terminating and been cleared from the node.

Reconsideration Race Conditions

Once preempted victims finish terminating and the scheduler determines in a subsequent scheduling cycle that a Deferred resize pod fits on its node (or determines via Filter that sufficient capacity is already available without evicting victims) and drops the pod from the scheduling queue in Bind, several race conditions can still prevent the Kubelet from admitting the resize:

Competing Pod Scale-Up: It is possible that the Kubelet chooses to resize a different pod than the one that triggered the eviction. For example, Pod A has a deferred resize and triggers the eviction of Pod B. After Pod B is deleted and Pod A passes Filter (or passes PostFilter reprieve when a lower-priority Pod C also has a pending resize), a competing resize of Pod C on the same node is prioritized or admitted first by the Kubelet, consuming the freed capacity before Pod A’s resize is actuated.
In-Flight Scale-Up of the Deferred Pod Itself: While Pod A is in-flight in the scheduler’s binding cycle, a user or controller updates Pod A’s spec with an even larger resource request that no longer fits within the freed capacity.
Direct Node Binding or Static Pod Admission: Another pod is bound directly to the node (bypassing the scheduler) or a static pod is admitted by the Kubelet, consuming the freed capacity before Pod A’s resize is actuated.
Node Allocatable Decrease: The node’s status.allocatable resources decrease before the Kubelet actuates Pod A’s resize.

Synchronizing State via the `PodResizeSchedulerFeasible` Condition

To handle all of these race conditions uniformly and maintain declarative state across scheduler restarts, we introduce a new pod status condition, PodResizeSchedulerFeasible, to coordinate handoff between the scheduler and the Kubelet:

type: PodResizeSchedulerFeasible
status: Always True when the condition is present on the pod. Following the same lifecycle convention as PodResizePending and PodResizeInProgress in KEP-1287, the condition is only present on pod.status.conditions while active (status: True) and is removed entirely by the Kubelet once it re-evaluates or completes the resize, rather than transitioning to False.
reason: Set to SchedulerPreemptionCompleted. Because this condition is only added by the scheduler in Bind when the Deferred resize passes node fit evaluation in Filter (either immediately or after preempted victims have finished terminating and been removed) or when PostFilter reprieve determines no victims need to be evicted, it only represents an affirmative state. Preemption or fit failures are never recorded on this condition (they continue to be emitted as FailedScheduling warning events).
message: Always left empty (""), as the condition is a state signal indicating that the scheduler has completed its evaluation/preemption for the resize and never carries an error message.
lastTransitionTime: Populated with the timestamp when the scheduler added the condition.

The lifecycle of this condition across the scheduler and Kubelet is as follows:

Victim Preemption and Termination (PostFilter $\rightarrow$ unschedulablePods): When a Deferred resize pod fails Filter and DefaultPreemption identifies and evicts victims in PostFilter, the pod is placed back into unschedulablePods (without PodResizeSchedulerFeasible set) while the victims terminate. Once the preempted victims finish terminating and are deleted from the API server, NodeResourcesFit’s AssignedPodDelete QueueingHint moves the pod from unschedulablePods back to activeQ for a subsequent scheduling cycle.
Scheduler Sets PodResizeSchedulerFeasible in Bind: When the pod passes Filter on the subsequent scheduling cycle after victims have terminated (or passes Filter / PostFilter reprieve immediately without needing victim eviction), the pod enters the binding cycle. DeferredPodScheduling.Bind patches the pod’s status to add PodResizeSchedulerFeasible=True with reason: SchedulerPreemptionCompleted, and the pod is removed from the scheduling queue.
Scheduler Queueing Guard (AddPod and UpdatePod): The scheduler’s AddPod and UpdatePod event handlers only enqueue an assigned pod into the scheduling queue if it has PodResizePending=True (reason: Deferred) and does not have PodResizeSchedulerFeasible=True. This ensures that if the scheduler restarts after the pod has exited the queue in Bind and is waiting for Kubelet actuation, AddPod sees PodResizeSchedulerFeasible=True and does not re-enqueue the pod.
Kubelet Re-evaluates the Resize and Removes the Condition: When the Kubelet evaluates pending resizes on the node:
- If the resize fits: The Kubelet allocates the new resources, removes both PodResizePending and PodResizeSchedulerFeasible from pod.status.conditions, and actuates the resize.
- If the resize still does not fit (due to any of the race conditions above): The Kubelet keeps PodResizePending=True (reason: Deferred) and removes PodResizeSchedulerFeasible from pod.status.conditions, patching the updated status to the API server.
- If the resize request was reverted: The Kubelet removes both PodResizePending and PodResizeSchedulerFeasible.
Scheduler Re-enqueues the Pod: When the Kubelet removes PodResizeSchedulerFeasible while PodResizePending=True (reason: Deferred) remains on the pod, the scheduler’s updatePod event handler observes that the pod is Deferred without PodResizeSchedulerFeasible and re-enqueues the pod into activeQ to run a new evaluation and preemption cycle.

Walkthrough: Competing Pod Scale-Up Race Cases

Returning to the example where Pod A triggers the eviction of Victim Pod B, and a competing resize of Pod C is requested on the same node, there are two timing cases depending on when Pod C’s resize request arrives relative to Pod A’s second scheduling cycle:

Case 1: Pod C’s resize arrives while Victim Pod B is terminating (before Pod A’s second scheduling cycle runs)
1. The Kubelet marks Pod A as Deferred, and the scheduler enqueues Pod A.
2. In Cycle 1, Filter fails for Pod A; PostFilter (DefaultPreemption) evicts Victim Pod B and returns Pod A to unschedulablePods while Pod B terminates.
3. While Pod B is terminating, a resize request for Pod C arrives on the same node. Because the scheduler cache computes nodeInfo.Requested as max(desired, allocated, actual) across all pods on the node, Pod C’s new desired resources are immediately reflected in nodeInfo.Requested.
4. Once Victim Pod B finishes terminating and is deleted, NodeResourcesFit’s AssignedPodDelete QueueingHint moves Pod A from unschedulablePods back to activeQ for Cycle 2, and the Kubelet also retries Deferred pods on the node.
5. In Cycle 2, if scheduler does not see pod C’s upsize yet, the Filter phase determines that it fits and sends Pod A to Bind, which adds the PodResizeSchedulerFeasible condition. This transitions into Case 2 below if Kubelet ends up admitting pod C first.
Case 2: Pod A passes Filter in Cycle 2 and exits the queue in Bind, and then the Kubelet cannot admit Pod A
1. Following Victim Pod B’s deletion, AssignedPodDelete wakes Pod A for Cycle 2. Pod A passes Filter.
2. DeferredPodScheduling.Bind sets PodResizeSchedulerFeasible=True (reason: SchedulerPreemptionCompleted) on Pod A, and Pod A is removed from the scheduling queue.
3. Before the Kubelet actuates Pod A’s resize, Pod C scales up and is admitted by the Kubelet first (or Pod A receives an even larger resize request while in-flight in Bind, or a new pod binds directly to the node), consuming the freed capacity. At this point, Pod A is out of the scheduling queue and Victim Pod B has already been deleted (so no future AssignedPodDelete event will ever wake Pod A).
4. When the Kubelet evaluates Pod A (including when the Kubelet’s pod sync loop observes PodResizeSchedulerFeasible=True on Deferred Pod A), the Kubelet sees that Pod A still cannot fit, keeps PodResizePending=True (reason: Deferred), and removes PodResizeSchedulerFeasible from pod.status.conditions.
5. The scheduler’s updatePod handler detects that Pod A is Deferred without PodResizeSchedulerFeasible and re-enqueues Pod A into activeQ, initiating a new preemption cycle to clear space for Pod A.

graph TD
    A[Pod A Enters activeQ] --> B{Filter: Does Resize Fit?}
    B -- No --> C[PostFilter: Calculate & Evict Victim Pod B]
    C --> D[Park Pod A in unschedulablePods While Victim B Terminates]
    D --> E[Victim Pod B Deleted: AssignedPodDelete QHint Moves Pod A to activeQ]
    E --> B
    B -- "Yes)" --> F["Bind: Set PodResizeSchedulerFeasible=True & Pop from Queue"]
    F --> G{Kubelet Evaluates Pod A}
    G -- "Pod A Fits" --> H["Clear PodResizePending & PodResizeSchedulerFeasible (Resize Actuated)"]
    G -- "Pod A Still Does Not Fit (e.g., Pod C Took Freed Space)" --> I["Kubelet Clears PodResizeSchedulerFeasible (Keeps Deferred)"]
    I --> J[updatePod Detects Deferred Without PodResizeSchedulerFeasible]
    J --> A

See alternative considerations for the deferred resize lifecycle here.

Preemption Policies

The existing preemption policies for scheduling new pods now also apply to Deferred resizes.

This means that the default preemptionPolicy, PreemptLowerPriority, allows the resize of pods of that PriorityClass to preempt lower-priority pods. If preemptionPolicy is set to Never, the resize of pods in that PriorityClass is non-preempting.

We considered creating a separate, independent preemption policy for Deferred resizes, but introducing this divergence adds unnecessary complexity for both users and maintainers:

Users would now be required to manage two different preemption behaviors for one pod.
The behavior may end up hard to predict, where a pod can preempt upon starting, but not to grow.
We would have to define the matrix of interactions between two independent preemption fields, leading to ambiguous states (e.g. what happens if preemptionPolicy is set to Never but resizePreemptionPolicy is set to PreemptLowerPriority)?
Adds code complexity, including updates to the Pod API, validation logic, and new branches in the Scheduler.

If a pod is marked with preemptionPolicy: PreemptLowerPriority, the user has already communicated that this workload is more important than lower-priority tasks. Whether that importance is manifested during initial placement or vertical scale-up, the intent of the priority remains the same.

See alternative considerations for preemption policies here.

Pod Priority, Graceful Termination, and Pod Disruption Budget

Because we are reusing the existing Scheduling preemption logic, the behavior is aligned with existing preemption behavior for new pods. This means:

Pods are preempted according to their Pod Priority.
The graceful termination period of victim pods is honored.
PodDisruptionBudget is supported, but not guaranteed.

Node-level Preemption Policy for In-Place Pod Resize

To avoid unnecessary disruption on nodes that have their own autoscaling solutions (such as resizable nodes), cluster operators and controllers need a way to disable scheduler-triggered preemption specifically when it is caused by an in-place pod resize request on those nodes. This protects critical workloads by preventing the scheduler from preempting low-priority pods to make room for a higher-priority resize, allowing the system to prefer node upsizing over preemption-induced disruption.

We propose a new field on the Node object to configure this policy, wrapped within a high-level PodPreemptionPolicy struct in NodeSpec:

// NodeSpec contains the desired configuration of a node.
type NodeSpec struct {
    // PodPreemptionPolicy controls the node-level preemption behaviors for pods on this node.
    // +optional
    PodPreemptionPolicy *NodePodPreemptionPolicy `json:"podPreemptionPolicy,omitempty"`
}

// NodePodPreemptionPolicy defines the node-level policies governing preemption for pods on this node.
type NodePodPreemptionPolicy struct {
    // DisableResizePreemption lists the owners (e.g., autoscalers, operators, administrators) 
    // that have requested to disable scheduler and Kubelet preemption for in-place pod resize on this node.
    // If this list is non-empty, resize-induced preemption is disabled on this node.
    // +listType=set
    // +optional
    // +k8s:optional
    // +k8s:eachVal=+k8s:format=k8s-label-key
    // +k8s:maxItems=20
    // +k8s:listType=set
    DisableResizePreemption []string `json:"disableResizePreemption,omitempty"`
}

An example YAML configuration for the Node:

apiVersion: v1
kind: Node
metadata:
  name: resizable-node
spec:
  podPreemptionPolicy:
    disableResizePreemption:
      - autoscaling.k8s.io/cluster-autoscaler
      - autoscaling.k8s.io/vpa-updater

The strings under disableResizePreemption have the same validation and constraints as standard Kubernetes label keys (e.g., a DNS subdomain prefix and a name separated by a /), and the list can have maximum of 20 entries.

See alternative considerations for the node-level preemption policy API here.

Policy Enforcement via `DeferredPodScheduling` Plugin

The node-level preemption policy is enforced directly in the scheduler via the DeferredPodScheduling plugin:

Configuration: A cluster operator or controller adds its identifier to the spec.podPreemptionPolicy.disableResizePreemption set on the Node object.
Scheduler Inspection in Filter Phase: When the scheduler evaluates a pod with a Deferred resize, DeferredPodScheduling.Filter checks node.Spec.PodPreemptionPolicy.DisableResizePreemption on the pod’s assigned node. If the list is non-empty, Filter returns status UnschedulableAndUnresolvable with reason ErrReasonNodeDisablesResizePreemption ("node had resize preemption disabled").
Skipping Preemption: Because the node status is UnschedulableAndUnresolvable (rather than Unschedulable), the DefaultPreemption PostFilter plugin skips preemption victim selection on the node. The pod is placed in the unschedulablePods pool without evicting lower-priority workloads.
Queueing Hints & Dynamic Wakeup: DeferredPodScheduling registers QueueingHints for Node Add and Update events. If the node’s spec.podPreemptionPolicy.disableResizePreemption field is cleared or modified to allow preemption, the QHint moves deferred pods assigned to that node back to the activeQ for immediate re-evaluation.

Multiple Owner Support

In environments with multiple controllers (e.g., multiple autoscalers) managing the same node, conflicts can arise. To support this, the spec.podPreemptionPolicy.disableResizePreemption field is designed as a set-type list (+listType=set, max 20 entries). The scheduler honors the policy if the list contains any entries (representing at least one active owner requesting preemption disablement).

Policy Scope: Exclusivity to Pod Resize

This policy is strictly limited to pod resizes and does not apply to new pod scheduling. In a mixed cluster (containing both resizable and non-resizable nodes), applying a preemption-disabling policy to new pod scheduling on a resizable node would cause the scheduler to simply select victims on a non-resizable node instead, defeating the purpose of the policy. For in-place pod resize, however, preemption is already strictly confined to the target node where the pod is running.

Kubelet Preemption Bypass for Resize Requests

The Kubelet’s critical admission handler runs during kubelet admission. Prior to this KEP, it would trigger preemption for system-critical pods that require more resources than currently available on the node. When the InPlacePodVerticalScalingSchedulerPreemption feature gate is enabled, the Kubelet bypasses admission preemption entirely for resize operations.

All preemption decisions for in-place pod resize are centralized exclusively in the scheduler. If an in-place resize cannot be accommodated immediately on the node, the Kubelet marks the resize request as Pending with reason Deferred, and relies on the scheduler to asynchronously handle preemption if priority rules permit.

This design provides key architectural benefits:

Single Preemption Authority: Consolidates all preemption logic (priority evaluation, graceful termination periods, PDB constraints) in the scheduler, avoiding duplicated or inconsistent preemption logic in the Kubelet.
Consistent Operational Model: Workloads on nodes with preemption disabled (non-empty spec.podPreemptionPolicy.disableResizePreemption) will not experience preemption from either the scheduler or Kubelet, allowing autoscaling solutions to scale up node capacity cleanly.

Failures and Reconsideration of Deferred pods

To ensure efficiency and avoid unnecessary reconciliation loops, the Scheduler treats Deferred resizes that fail preemption (when preemption is enabled) as a standard FitError (Unschedulable). By classifying the resize failure this way, the existing Scheduler retry and backoff logic is triggered automatically, requiring no new dedicated retry infrastructure.

Queue Placement: When a preemption attempt fails to find enough victims on the node to satisfy the resize, the pod is moved to the unschedulablePods pool within the scheduling queue.
Triggering Re-entry: Because the pod is in the unschedulablePods pool, it is moved back to the activeQ (or backoffQ) automatically when cluster events occur that could potentially change the outcome of the preemption logic. These events include:
- Pod Deletion: A pod on the same node is deleted or removed (freeing up capacity).
- Pod Downsize: A pod on the same node has its resource requests decreased.
- Node Update: The node’s Allocatable resources increase (e.g., via a Kubelet config update).
Backoff Mechanism: Standard exponential backoff applies. This ensures that “stuck” resize requests do not degrade the scheduling throughput for the rest of the cluster.

By utilizing the existing movePodsToActiveOrBackoffQueue logic and treating the resize deficit as a native scheduling constraint, we ensure that Deferred resizes do not consume excessive CPU cycles unless the cluster state has changed in a way that makes success plausible.

To prevent ’thundering herd’ issues where unrelated node activity causes resizes to be reevaluated unnecessarily, a pod is only moved back to the activeQ when the pod deletion, pod downsize, or node update events occur on the same node as the pod being resized. This is implemented by making the QueueingHint (Qhint) functions of the NodeResourcesFit plugin node-aware; if the pod is already assigned to a node (indicating a resize request), the Qhints evaluate to QueueSkip for any event occurring on a different node.

Failure Handler Adjustments for Deferred Pods

When a deferred resize scheduling cycle fails (i.e., there is a fit error and preemption fails or is disabled), the scheduler executes its failure handler (handleSchedulingFailure). Because the deferred pod is already scheduled and running on a node, several steps in the standard failure handler must be modified or skipped:

Bypassing the NodeName Re-queue Abort: For a standard pod, if the pod already has Spec.NodeName populated in the informer cache, the failure handler assumes it has been successfully bound and aborts re-queuing it. For Deferred pods, this abort check is skipped so that the pod is successfully returned to the Unschedulable queue to await future retry attempts.
Skipping PodScheduled Condition Updates: The standard failure handler sets the pod’s PodScheduled status condition to False (with reason Unschedulable). For a Deferred pod, updating this condition is skipped. For beta, we will implement emitting events to surface the results of the scheduling cycle for observability.
No Nominated Node Name (NNN) Assignment: Standard scheduling failures can record a NominatedNodeName on the pod status to reserve space on a target candidate node. For a Deferred pod, no nominated node is set or updated. The pod is already bound to its host, and its resize can only be evaluated and satisfied on that specific host; nominating another node is invalid.

Interaction with Workload-Aware Scheduling (WAS) and Gang Scheduling

When operating in clusters with Workload-Aware Scheduling (WAS / gang scheduling):

Resizing Preemptor Pods: Deferred resize pods belonging to a PodGroup are evaluated and preempted individually (as standard single pods). We have no established use cases yet for coordinated gang-resizing of pods; if concrete use cases emerge in the future, we can reevaluate adding group-level resize coordination.
Candidate Victim Pods: Because deferred resize preemption reuses DefaultPreemption (SelectVictimsOnNode), victim selection automatically inherits KEP-5710: Workload-Aware Preemption behavior. If a candidate victim on the node belongs to a PodGroup with DisruptionMode: All, evicting it disrupts the entire victim PodGroup across nodes; the scheduler’s victim ordering (MoreImportantVictim) and reprieval logic account for this by prioritizing standalone pods for eviction over disrupting PodGroup workloads.

(See WAS Integration Alternatives for a summary of coordinated gang-resize approaches considered and deferred.)

Metrics and Events

Metrics

kube-scheduler exports a significant number of metrics. We introduce dedicated ALPHA metrics (mirroring STABLE and BETA metrics whose label sets cannot be modified), extend existing ALPHA metrics, and clarify skipped metrics to provide observability into scheduler behavior for in-place pod resizing.

New ALPHA metrics for deferred resize pods

Because scheduler_pending_pods, scheduler_preemption_attempts_total, and scheduler_preemption_victims are STABLE, and scheduler_scheduling_algorithm_duration_seconds is BETA, their label sets cannot be modified. These existing STABLE and BETA metrics will exclude deferred resize pods (recording only standard initial-placement pods). Following the pattern established by workload-aware scheduling and preemption (scheduler_workload_preemption_attempts_total, scheduler_workload_preemption_victims, and scheduler_podgroup_scheduling_algorithm_duration_seconds), we introduce dedicated ALPHA metrics for deferred resize evaluation cycles:

scheduler_pending_resize_pods (Gauge with queue label, mirroring scheduler_pending_pods)
scheduler_resize_scheduling_algorithm_duration_seconds (Histogram, mirroring scheduler_scheduling_algorithm_duration_seconds)
scheduler_resize_preemption_attempts_total (Counter, mirroring scheduler_preemption_attempts_total)
scheduler_resize_preemption_victims (Histogram, mirroring scheduler_preemption_victims)

Extending existing ALPHA metrics with a new `operation` label

We update the following existing ALPHA metrics to include a new operation label:

scheduler_queued_entities
scheduler_preemption_evaluation_duration_seconds
scheduler_preemption_execution_duration_seconds
scheduler_preemption_pdb_violations_total
scheduler_preemption_goroutines_execution_total

To distinguish between in-place resize workflows and initial placement workflows, the operation label has possible values of initial_placement or pod_resize.

Extending metrics with a new `AssignedPodResize` event

The following metrics already expose an event label:

scheduler_event_handling_duration_seconds
scheduler_queue_incoming_pods_total
scheduler_queue_incoming_entities_total

When an assigned pod transitions to a deferred resize and enters the scheduling queue, the scheduler currently records event="UnscheduledPodAdd". We introduce a new cluster event value AssignedPodResize so incoming queue metrics and event handling durations accurately reflect resize queueing rather than new pod arrivals.

Metrics skipped for deferred resizing pods

Because resizing pods are already bound to a node and actuated by Kubelet rather than bound by the scheduler (or are tracked via the dedicated resize metrics above), the following initial-placement metrics are skipped entirely for deferred resize evaluation cycles:

scheduler_pending_pods
scheduler_scheduling_algorithm_duration_seconds
scheduler_preemption_attempts_total
scheduler_preemption_victims
scheduler_schedule_attempts_total
scheduler_scheduling_attempt_duration_seconds
scheduler_pod_scheduling_sli_duration_seconds
scheduler_pod_scheduling_attempts
scheduler_pod_scheduled_after_flush_total
scheduler_unschedulable_pods
scheduler_permit_wait_duration_seconds

Events

Preemption events for in-place resize follow existing scheduler preemption conventions:

Preemptor Pod: Emits a FailedScheduling warning event when preemption cannot proceed, such as when node preemption policy disables preemption (0/1 nodes are available: 1 node had resize preemption disabled.) or when available victims cannot satisfy the resize deficit (0/1 nodes are available: 1 Insufficient cpu.).
Victim Pods: Emits a Preempted normal event indicating the victim was evicted to accommodate the resizing pod (Preempted by pod <namespace>/<name> on node <node-name>), and sets the DisruptionTarget pod condition with reason PreemptionByScheduler.

Pod Status & Conditions

Once the scheduler completes preemption for a Deferred resize and determines in Filter that the resize fits on the node, it sets the PodResizeSchedulerFeasible condition (with reason: SchedulerPreemptionCompleted) in Bind to coordinate handoff with the Kubelet (see Synchronizing State via the PodResizeSchedulerFeasible Condition).

However, we decided not to expose a pod status condition surfacing preemption failure or blocker states when a Deferred resize cannot fit or preempt on its node. There is currently no concrete use case requiring a failure condition, and omitting it avoids generating unnecessary API status churn on every failed preemption evaluation cycle. This is analogous to how scheduler preemption behaves today for initial pod placement, where the scheduler does not write a preemption-specific failure condition on pending pods.

There are two primary actors that may be interested in the results of the preemption cycle for resizing pods: external controllers and users; both are discussed below.

External controllers

The current recommended pattern for external controller integration with this feature is to utilize the existing PodResizePending condition and the node-level podPreemptionPolicy field. For example, if a controller can scale down another pod to free up capacity or increase capacity of the node itself, these alternative actions should be attempted first as they are less disruptive and generally preferred over eviction. The controller can use podPreemptionPolicy to disable resize preemption, perform its own autoscaling actions to free up capacity based on the PodResizePending condition, and dynamically reenable resize preemption only when it needs help from the scheduler as a last resort to free up capacity.

The podPreemptionPolicy field covers the currently known integration needs for external controllers. That said, one can imagine a controller wanting the scheduler to proactively preempt for a resize, and react accordingly when preemption fails. However, we do not currently have a concrete use-case requiring a preemption failure condition, so we leave it out of scope for the initial beta and may consider introducing such a mechanism in a future release if a clear requirement arises that is not addressed by our existing mechanisms.

End-user observability

For user observability, users can inspect preemption blockers via standard kubectl describe pod event streams, where FailedScheduling warning events detail why preemption was blocked or could not find sufficient capacity. Users can also observe the PodResizePending and PodResizeSchedulerFeasible conditions to infer the current pod resize status.

This is analogous to user-observability today for initial pod placement, where users can inspect FailedScheduling events to see why a pod couldn’t be scheduled and the PodScheduled condition to infer if the pod has been scheduled. Initial pod placement does not expose any additional conditions to surface why preemption failed, and we do not believe that the same is necessary for resizing pods.

Should we receive feedback that users want more fine-grained condition-based observability into preemption failures for resizing pods, we can consider extending the pod status conditions in a future enhancement, but we consider it out of scope for the initial beta.

Test Plan

[x] I/we understand the owners of the involved components may require updates to existing tests to make this code solid enough prior to committing the changes necessary to implement this enhancement.

Prerequisite testing updates

Unit tests

k8s.io/kubernetes/pkg/apis/scheduling/v1alpha1: 2026-02-23 - 62.7%
k8s.io/kubernetes/pkg/apis/scheduling/validation: 2026-02-23 - 97.8%
k8s.io/kubernetes/pkg/scheduler: 2026-02-23 - 81.7%

Integration tests

Integration tests in test/integration/scheduler/preemption/deferred_resize_preemption_test.go cover the following scenarios:

Preemption Evaluation (TestDeferredResizePodPreemption):
- Ensure that a single lower-priority pod can be preempted on the assigned node by a higher-priority pod with a Deferred resize.
- Ensure that multiple lower-priority pods can be preempted to satisfy the resource deficit of a Deferred resize.
- Ensure that pods with preemptionPolicy: PreemptNever do not trigger preemption when deferred.
- Ensure that when a deferred resize fits without preemption, the pod completes the scheduling cycle through Bind, sets the PodResizeSchedulerFeasible condition (reason: SchedulerPreemptionCompleted), and exits the scheduling queue without creating a Binding object or triggering victim preemption.
- Ensure that when a node disables resize preemption via spec.podPreemptionPolicy.disableResizePreemption, preemption is bypassed and the deferred pod remains parked without victim eviction.
- Ensure that double preemption is prevented during long graceful termination periods (a terminating victim pod suppresses further preemption for the resizing pod).
- Ensure that restarting the scheduler while a victim is terminating preserves preemption suppression until the victim is completely deleted.
Queueing Hints (TestDeferredResizeQueueingHints):
- Ensure that relevant events on the same node (pod deletion, pod downsize, node allocatable capacity increase, target pod spec downsize) trigger QueueingHints to move deferred pods from unschedulablePods back to the activeQ.
- Ensure that irrelevant events (pod deletion or downsize on other nodes, non-resource label changes) are filtered out and do not wake the queue.
Node Preemption Policy Events (TestDeferredResizeNodePreemptionPolicy):
- Ensure that updating a node’s preemption policy to enable preemption (clearing disableResizePreemption) wakes up deferred pods on that node.
- Ensure that disabling preemption policy or updating policies on unrelated nodes does not wake the queue.
Scheduling Queue Handlers (TestDeferredResizeQueueingHandlers):
- Ensure that existing deferred pods in the cluster (that do not already have PodResizeSchedulerFeasible=True) are enqueued upon scheduler startup and informer synchronization.
- Ensure that pods transitioning to a Deferred resize are automatically enqueued in the scheduling queue.
- Ensure that when PodResizeSchedulerFeasible is removed while the pod remains Deferred (for example, when Kubelet re-evaluates the resize and determines it still cannot fit), the pod is automatically re-enqueued into activeQ.
- Ensure that pods are removed from the queue when their Deferred resize condition is cleared or resolved.
- Ensure that deleting a deferred pod removes it from both the scheduler cache and scheduling queue.
Metrics Verification:
- Ensure scheduler_pending_resize_pods and scheduler_resize_preemption_attempts_total are recorded during deferred resize processing.
- Verify that scheduler_schedule_attempts_total is not incremented for deferred resize cycles.

e2e tests

We will create a single e2e test to ensure that a lower-priority pod can be preempted by a higher-priority Deferred resize, and that the Kubelet successfully actuates the Deferred resize after the preemption completes.

Graduation Criteria

Alpha

Feature and API implemented behind a feature flag
Initial unit, integration, and e2e tests completed and enabled

Beta

Gather feedback from alpha; specifically:
- validation of primary user stories and motivation
- validation of podPreemptionPolicy API usage
- no significant issues or gaps reported by users (to be evaluated at Beta promotion time)
Metrics and events are defined and implemented
Additional integration tests are defined, implemented and linked in KEP
Interaction with workload aware scheduling is clarified
Address scenarios where a preemption victim’s grace period exceeds the time window allocated for executing the resize operation (such as VPA’s fallback time limit).
Evaluate using nominated node name to prevent double preemption
Evaluate whether we need to solve the additional preemption risk
If KEP-5517 Alpha2 is implemented, ensure that the Deferred resize scheduling cycle includes resources from pod.status.nodeAllocatableResourceClaimStatuses.

GA

Allowing time for feedback, with at least 2 releases have passed with the feature default-on
All GA-blocking issues and gaps identified as feedback during beta are resolved

Upgrade / Downgrade Strategy

Standard procedures for features introducing new API fields should be used:

On upgrade, kube-apiservers should be upgraded first before kube-scheduler can read, and operators/controllers can configure, the new node-level PodPreemptionPolicy field.
On downgrade, kube-schedulers and Kubelets should be downgraded first (to stop using the new feature and fields) before kube-apiservers are downgraded; note that downgrade of kube-apiserver(s) and/or disabling the new API fields will not clear their contents for objects already stored in the storage (etcd).

Version Skew Strategy

N-3 kubelet already marks resizes without enough capacity as Deferred and retries when room is made. An older kubelet that is unaware of the PodResizeSchedulerFeasible condition will still clear PodResizePending when the resize fits on the node (which removes the pod from the scheduler’s consideration); if the resize still cannot fit on an older kubelet due to a race condition, the kubelet continues to retry pending resizes periodically until capacity becomes available.

Because the node-level preemption policy is enforced directly in the scheduler by inspecting the Node spec via the DeferredPodScheduling plugin, version skew between Kubelet and Scheduler does not affect policy enforcement.

Production Readiness Review Questionnaire

Feature Enablement and Rollback

How can this feature be enabled / disabled in a live cluster?

Feature gate (also fill in values in kep.yaml)
- Feature gate name: InPlacePodVerticalScalingSchedulerPreemption
- Components depending on the feature gate: kube-apiserver, kube-scheduler, kubelet

Does enabling the feature change any default behavior?

Yes:

Deferred resizes will trigger preemption of lower-priority pods. InPlacePodVerticalScaling is GA and enabled by default, and PriorityClasses are widely used.
The Kubelet critical admission handler will no longer run on resizing pods. Previously, if there was not enough room for the upsize of a system-critical pod, the Kubelet critical admission handler would evict other pods to free up resources. With this KEP, it instead allows the pod to be marked as Deferred and relies on scheduler preemption to free up space. See Kubelet Preemption Bypass for Resize Requests for the relevant section.

Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)?

Yes, the feature can be rolled back by disabling the feature gate.

What happens if we reenable the feature if it was previously rolled back?

The feature will start working again; Deferred resizes will trigger preemption of lower-priority pods.

Are there any tests for feature enablement/disablement?

The scheduler logic will be covered by regular feature tests, which is sufficient because preemption code changes occur purely in-memory.

For the newly introduced API, we added dedicated tests for feature gate enablement/disablement in the API server:

Rollout, Upgrade and Rollback Planning

How can a rollout or rollback fail? Can it impact already running workloads?

This feature effectively consists of two parts:

Preemption triggered by Deferred resize updates: Implemented inside kube-scheduler (with status condition handoff via PodResizeSchedulerFeasible).
- Failure Modes: Bugs in scheduler resource accounting, queue handling, or QueueingHints could cause either unnecessary/duplicate preemptions (evicting more victims than needed) or excessive queue churn degrading scheduling throughput.
The per-node API (node.spec.podPreemptionPolicy) to enable/disable resize preemption: Component rollout order matters here. If kube-scheduler enables the feature before kube-apiserver, the API server will drop node.spec.podPreemptionPolicy on write. Controllers managing resizable nodes will fail to disable resize preemption, causing the scheduler to evict lower-priority pods on nodes intended to be protected from preemption.

Impact on running workloads: Enabling the feature directly impacts running workloads: running lower-priority pods can now be preempted and evicted as a result of in-place scale-up requests on higher-priority pods—an action that previously never triggered pod eviction.

What specific metrics should inform a rollback?

Abnormal spike in scheduler_resize_preemption_attempts_total without a corresponding increase in successful Kubelet resizes.
Unexpected increase in scheduler_resize_scheduling_algorithm_duration_seconds causing queue starvation.

Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?

Integration tests verify that disabling the feature gate stops queueing deferred pods and ignores resize events without affecting initial pod placement.

For manual testing we followed the following steps:

Pre-Upgrade Baseline (Feature Gate Disabled):

Start the cluster with InPlacePodVerticalScalingSchedulerPreemption=false.
Deploy low-priority workloads on a worker node until allocatable capacity is nearly saturated.
Deploy a high-priority pod on the same node and issue a resource resize patch that exceeds remaining node capacity.
Verify that the high-priority pod transitions to PodResizePending with reason Deferred. Confirm that the scheduler does not trigger preemption and low-priority pods remain uninterrupted.

Upgrade Step (Feature Gate Enabled):

Enable the feature gate InPlacePodVerticalScalingSchedulerPreemption=true on kube-apiserver first
Set node.spec.podPreemptionPolicy.disableResizePreemption = [“test-owner”] on the test node
Enable the feature gate InPlacePodVerticalScalingSchedulerPreemption=true on kube-scheduler and kubelet.
Trigger a deferred resize on that node; verify kube-scheduler skips preemption and logs 0/1 nodes available: 1 node had resize preemption disabled.
Dynamically clear the node.spec.podPreemptionPolicy.disableResizePreemption on the test node
Verify that the scheduler evaluates the node, identifies lower-priority victims, and initiates eviction (victims receive Preempted events and DisruptionTarget condition).
Verify that once victims terminate, Kubelet actuates the resize and clears Deferred.

Rollback / Downgrade Step (Feature Gate Disabled):

Disable the feature gate InPlacePodVerticalScalingSchedulerPreemption=false on kube-scheduler and restart the component.
Trigger a new resize on a saturated node resulting in Deferred status.
Verify that kube-scheduler ignores the deferred resize and does not evict lower-priority pods.
Verify that existing workloads (including previously resized pods) continue running normally.

Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?

No.

Monitoring Requirements

How can an operator determine if the feature is in use by workloads?

Inspect the gauge metric scheduler_pending_resize_pods > 0 or counter scheduler_resize_preemption_attempts_total > 0.

How can someone using this feature know that it is working for their instance?

Events
- Event Reason: Preempted on the victim pod and ResizeStarted or ResizeCompleted on the preempting pod.
API .status
- Condition name: The preempting pod has PodResizeSchedulerFeasible set when scheduler evaluation completes, and has both PodResizePending and PodResizeSchedulerFeasible cleared once Kubelet actuates the resize.
- Other field: The preempting pod’s status.Resources matches its spec.Resources.
Other (treat as last resort)
- Details:

What are the reasonable SLOs (Service Level Objectives) for the enhancement?

Pod resize preemption evaluation duration (scheduler_preemption_evaluation_duration_seconds{operation="pod_resize"}) completes within 1s for 99% of cases.

What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?

Metrics
- Metric name: scheduler_resize_preemption_attempts_total
- Components exposing the metric: kube-scheduler

Are there any missing metrics that would be useful to have to improve observability of this feature?

None.

Dependencies

Does this feature depend on any specific services running in the cluster?

None.

Scalability

Will enabling / using this feature result in any new API calls?

For watching and queueing deferred resizes, we do not introduce any new list or watch API calls. The Scheduler is already watching for pod and node updates, with states cached in memory via Informers, and the Kubelet already updates pod status when a resize is Deferred as part of standard in-place pod resize behavior.

When a Deferred resize triggers preemption, kube-scheduler issues standard preemption API calls to evict each selected victim pod:

PATCH on the victim pod’s /status subresource to set the DisruptionTarget (PreemptionByScheduler) condition.
DELETE on the victim pod to initiate eviction.
CREATE / PATCH on Events to record Preempted events on victim pods (and FailedScheduling events on the resizing pod when preemption fails).

Once preempted victims finish terminating and the scheduler determines in Filter that a Deferred resize fits on the node:

PATCH on the resizing pod’s /status subresource by kube-scheduler during Bind to set the PodResizeSchedulerFeasible (SchedulerPreemptionCompleted) condition (1 call per handled Deferred resize).
PATCH on the resizing pod’s /status subresource by kubelet to remove the PodResizeSchedulerFeasible condition upon re-evaluation. When the resize is admitted by the Kubelet, removing PodResizeSchedulerFeasible is combined with the Kubelet’s existing /status update that clears PodResizePending (resulting in no extra API call); an additional /status PATCH from the Kubelet only occurs when a race condition prevents the Kubelet from admitting the resize and the pod remains Deferred.

Will enabling / using this feature result in introducing new API types?

No.

Will enabling / using this feature result in any new calls to the cloud provider?

No.

Will enabling / using this feature result in increasing size or count of the existing API objects?

The new optional podPreemptionPolicy field on NodeSpec will slightly increase the size of Node objects when configured. On PodStatus, the PodResizeSchedulerFeasible condition temporarily adds one condition entry to pod.status.conditions while a Deferred pod that has completed scheduler evaluation awaits Kubelet actuation, and is removed once the Kubelet re-evaluates the resize.

Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs?

There may be some performance impact for scheduling of the pods, as the scheduler now has to additionally process Deferred pods.

Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, …) in any components?

No.

Can enabling / using this feature result in resource exhaustion of some node resources (PIDs, sockets, inodes, etc.)?

No.

Troubleshooting

How does this feature react if the API server and/or etcd is unavailable?

If the API server and/or etcd are unavailable, kube-scheduler will not be informed of any new resize requests, and thus will not be able to process new resize requests, resulting in pods remaining in PodResizePending: Reason=Deferred. Existing resized pods will continue to run, but no new preemptions would be triggered.

What are other known failure modes?

Preemption Victim Stuck in Terminating
- Description: The scheduler selects a lower-priority victim and issues a delete call. However, the victim has a hanging finalizer, an unresponsive CSI volume detachment, or a stuck graceful termination process. Because the scheduler’s double-preemption prevention logic detects an already-terminating victim on spec.NodeName, it correctly suppresses further preemption, but the deferred pod(s) remains stuck waiting for capacity that never frees up.
- Detection: The resizing pod remains in PodResizePending: Reason=Deferred. A lower-priority pod on the same node has metadata.deletionTimestamp != nil and a DisruptionTarget condition with reason PreemptionByScheduler for longer than its expected termination grace period.
- Mitigations: Operator inspects the stuck victim pod, resolves the finalizer or volume unmount blocker, or force-deletes the victim (kubectl delete pod <victim> --force --grace-period=0).
- Diagnostics: Inspect victim pod conditions and events: kubectl describe pod <victim> to check for unmount or finalizer errors.
- Testing: Integration tests verify that a terminating victim suppresses duplicate preemption until fully removed from the cache.

What steps should be taken if SLOs are not being met to determine the problem?

If scheduling latency or preemption SLOs are degraded (e.g., scheduler_scheduling_algorithm_duration_seconds or scheduler_preemption_evaluation_duration_seconds exceed targets), operators should follow this diagnostic procedure:

Isolate the source of latency:
- Compare scheduler_resize_scheduling_algorithm_duration_seconds against scheduler_scheduling_algorithm_duration_seconds.
- Compare scheduler_pending_resize_pods against scheduler_pending_pods.
- If initial placement latency is elevated while resize counts are negligible, the degradation is unrelated to resize preemption (investigate general cluster load, API server latency, or placement plugins).
- If scheduler_preemption_evaluation_duration_seconds{operation="pod_resize"} is elevated, proceed to investigate preemption evaluation bottlenecks.
Check for Queue Flooding and QHint Churn:
- Inspect scheduler_queue_incoming_pods_total{event="AssignedPodResize"}.
- A high rate of incoming resize events indicates that deferred pods are repeatedly waking from unschedulablePods back into activeQ.
- Check if a misconfigured QueueingHint in NodeResourcesFit or DeferredPodScheduling is waking deferred pods on unrelated node events (such as pod additions or deletions on other nodes) rather than strictly filtering to events on the pod’s assigned host.
Inspect Preemption Thrashing and PDB Deadlocks:
- Compare scheduler_resize_preemption_attempts_total against scheduler_resize_preemption_victims.
- A high ratio of preemption attempts to actual victims evicted indicates that preemption is repeatedly evaluating nodes but failing to clear capacity.
- Inspect scheduler_preemption_pdb_violations_total{operation="pod_resize"}: if elevated, preemption cycles are spending time repeatedly dry-running candidate victim sets only to be blocked by strict PodDisruptionBudgets.
Mitigation Actions:
- To stop preemption immediately on problematic nodes without cluster downtime, patch the affected nodes to disable resize preemption.
- If scheduling throughput cluster-wide is severely impacted, disable the feature gate on kube-scheduler.

Implementation History

2026-02-23: KEP Created for alpha release 2026-08-17: KEP updated to align with actual alpha implementation 2026-09-10: KEP beta update

Drawbacks

Alternatives

Separate scheduling queue

We considered keeping Deferred resizes in a separate scheduling queue from new pods. However, the interaction and prioritization between these two queues was difficult to reason about. It is simpler and more intuitive to consider Deferred resizes as warranting largely the same behavior as new pods with regards to preemption, so keeping them together in a single queue made more sense.

Implementing prioritized resizes logic

The Kubelet has a priority order that it attempts to retry Deferred resizes, as defined here.

We considered integrating a similar prioritization logic for the Deferred resizes in the scheduling queue. However, this adds a lot of complexity for questionable benefit. Scheduler preemption today considers only Pod Priority, and unless there is a strong use case to do something more complex, the preemption behavior for Deferred resizes should stay aligned with that of new pods.

Preemption Policies

We considered creating a separate, independent preemption policy for Deferred resizes, but introducing this divergence adds unnecessary complexity for both users and maintainers:

Users would now be required to manage two different preemption behaviors for one pod.
The behavior may end up hard to predict, where a pod can preempt upon starting, but not to grow.
We would have to define the matrix of interactions between two independent preemption fields, leading to ambiguous states (e.g. what happens if preemptionPolicy is set to Never but resizePreemptionPolicy is set to PreemptLowerPriority)?
Adds code complexity, including updates to the Pod API, validation logic, and new branches in the Scheduler.

Conflicting Sources of Truth for the Pod

To address the complexity of the pod existing in both the cache and the scheduling queue simultaneously, we considered avoiding duplicate state by using a dynamic reference interface.

Instead of storing a direct snapshot of the pod in the scheduling queue, we considered changing the pod representation inside the queue to a PodGetter interface:

type PodGetter interface {
    GetPod() *v1.Pod
}

Under this alternative:

For standard pods, GetPod() would simply return the stored pod snapshot.
For deferred resize pods, GetPod() would dynamically query the pod directly from the scheduler’s shared cache, ensuring a single source of truth.

While this abstraction avoids the need to synchronize pod updates across multiple places, we rejected it because:

Bypassing critical scheduling code paths: Storing a dynamic reference might skip essential queue event handlers. For example, PriorityQueue.Update would not be called correctly upon pod updates. This could prevent a pod’s preemption from being retried when a relevant event (such as a status or request change) is updated.
Increased codebase management overhead: Circumventing the standard queue update flow would make the scheduling queue’s state harder to reason about and manage.
Complex refactoring footprint: It introduces a significant refactoring footprint across core scheduler structures and scheduling plugins that expect immutable pod snapshots throughout their scheduling cycle.

Instead, we chose to explicitly keep the pod object in both places (the cache and the queue) while ensuring we audit and eliminate any downstream logic that assumes the pod exists in only one place.

Kubelet-Scheduler Preemption Interaction

We considered introducing a mechanism to communicate to the Kubelet exactly which resize event initiated a preemption cascade. This would eliminate opportunistic race conditions between the Kubelet and the scheduler. However, this adds a lot of complexity for arguably worse behavior.

For example, if a higher-priority resize is requested while room is being made by evicting a lower-priority pod, we want to honor the priority. We want the higher-priority pod to get resized, and the other deferred pod to be put back into the scheduling queue. By avoiding explicit signaling, we let Kubernetes naturally honor pod priority; the Kubelet can allocate capacity to the highest-priority workload, and the displaced pod safely returns to the scheduling queue.

Tracking Preemption Nominations to Avoid Double Preemption

If a Deferred resize triggers preemption, there is a risk of double preemption (where the scheduler terminates a different set of victims for the same resize request) during long graceful termination periods or across scheduler restarts.

We considered several alternative options for addressing this risk before settling on having the preemption plugin evaluate spec.nodeName for pods that are already bound. The alternative options were:

Option 1: Accept Double Preemption (Rejected)

Description: Treat double preemption as an acceptable edge-case behavior, relying on API server delete idempotency.
Why Rejected: Evicting additional workloads while a previous victim is terminating leads to unnecessary workload disruption in production clusters.

Option 2: Internal Nomination Tracking (Rejected)

Description: Maintain an internal scheduler map tracking active preemption nominations for deferred resizes without persisting state to the API server.
Why Rejected: In-memory state is wiped on scheduler crashes and restarts, leaving the restart scenario unmitigated.

Option 3: Reuse `status.nominatedNodeName` (Rejected)

Description: Reuse pod.Status.NominatedNodeName on the resizing pod to record the assigned node during preemption.
Why Rejected: Adds two unnecessary API status writes per resize (setting and clearing the field) and breaks downstream assumptions for external controllers (such as cluster autoscaler or monitoring dashboards) that treat nominatedNodeName as indicating an unscheduled pod waiting to bind.

Handling Successfully Fitted Deferred Resizes

When the scheduler triggers preemption for a deferred resize pod (or determines that it fits on its assigned node), the scheduler must handle its lifecycle while awaiting Kubelet actuation. We considered several alternatives before settling on the PodResizeSchedulerFeasible condition:

1: Parking in Unschedulable Queue via `Permit` Rejection (Alpha Decision - Rejected for Beta)

Description: The DeferredPodScheduling.Permit plugin returns UnschedulableAndUnresolvable when a deferred resize fits, parking the pod in unschedulablePods. To wake the pod if another workload consumed the space before Kubelet actuation, NodeResourcesFit registered cluster-wide UpdatePodScaleUp QueueingHints (isSchedulableAfterAssignedPodScaleUp).
Why Rejected:
1. Queue Pollution: Successfully fitted pods sat in unschedulablePods in memory indefinitely until Kubelet actuation, consuming queue resources.
2. QHint Overhead: UpdatePodScaleUp QHints ran on every pod scale-up event across the entire cluster, iterating over all unschedulable pods to check matching node names.

2: Re-queueing on Node Scale-Up Events in `updatePod` (Rejected)

Description: Drop the deferred pod from the scheduling queue via a no-op Bind, and modify the scheduler’s updatePod event handler to watch for assigned pod scale-up events on a node, using a spec.nodeName informer index and SchedulingQueue.AddOrActivate to re-enqueue any Deferred pods on that node.
Why Rejected:
1. Edge-Triggered & Incomplete Coverage: Watching only for assigned pod scale-ups misses other scenarios that can prevent Kubelet admission after preemption (such as pods bound directly to the node, static pod admission, or node allocatable decreases) and requires special handling for in-flight scale-ups of the deferred pod itself.
2. No Declarative State Across Scheduler Restarts: Without a persisted condition on the pod, a restarted scheduler cannot distinguish between a newly Deferred pod that needs preemption and a Deferred pod whose preemption has already been triggered and is waiting for victim termination and Kubelet actuation.

3: Re-queueing on `PodResizePending.LastProbeTime` Updates (Rejected)

Description: Drop the deferred pod from the scheduling queue via a no-op Bind, update the Kubelet to persist LastProbeTime changes on the PodResizePending (reason: Deferred) condition whenever it retries a deferred resize, and have the scheduler’s updatePod handler re-enqueue the pod whenever LastProbeTime advances.
Why Rejected: Like the updatePod scale-up watcher, LastProbeTime alone does not record whether the scheduler has already handled the pod since the last probe, so a newly restarted scheduler still cannot tell whether a Deferred pod has already had preemption completed and is waiting for the Kubelet. Additionally, persisting LastProbeTime on every periodic Kubelet retry generates recurring API status updates for all Deferred pods even when nothing has changed on the node.

WAS Integration Alternatives

Coordinated Group-Level Resize Preemption (Deferred)

Description: When an assigned pod belonging to a PodGroup transitions to Deferred, enqueue the entire PodGroup into scheduleOnePodGroup. Enforce a readiness guard in PreFilter comparing the number of Deferred members against spec.schedulingPolicy.gang.minCount (with a 5s settling window to account for sequential multi-pod PATCH skew), and execute atomic all-or-nothing multi-node preemption across all resizing members of the gang.
Why Deferred:
1. Lack of Established Use Cases: Real-world distributed training workloads typically run in a pod-per-node topology or scale individual pods independently; there is no concrete production demand yet for coordinated in-place resizing across an entire gang.
2. Coordination Complexity & Edge Cases: Obeying minCount for an already running PodGroup introduces subtle edge cases (e.g., active pods dropping below minCount, mixed Deferred and unscheduled pods sharing the same parent group, and timer management for sequential patches) that are not justified without proven user demand.

Node-Level Preemption Policy API Options

We identified several API options for implementing the node-level preemption policy before selecting the proposed first-class Node Field.

1. Node Annotations (Rejected)

Description: Setting the policy via a standard or prefixed annotation, such as scheduler.policy/disable-pod-resize-preemption: "true".
Why Rejected: Core Kubernetes components (such as the Scheduler and Kubelet) should generally not depend on unstructured annotations to drive critical runtime decisions. Annotations are meant for metadata and not for policy enforcement.

2. Pod-Level Preemption Disabled Condition (Rejected)

Description: The Kubelet would watch the Node’s preemption policy and propagate it to the scheduler by setting a PodResizePreemptionDisabled condition on deferred pods to instruct the scheduler to skip preemption.
Why Rejected: Setting a pod condition from the Kubelet introduces extra API write load and latency on the API server for every deferred pod when node policy changes. The scheduler already caches Node objects in NodeInfo and can evaluate the policy directly in the DeferredPodScheduling plugin with zero additional API writes, while using node-level QueueingHints to automatically requeue pods when the node policy is updated.

3. Node Labels (Rejected)

Description: Setting the policy via a node label, e.g., kubectl label node <node-name> scheduler.policy/disable-pod-resize-preemption="true".
Why Rejected: Labels are intended for identifying and grouping objects (e.g., for node selectors or affinity). They are not meant for carrying fine-grained configuration or policy intent. They lack structured representation and do not support multiple owners cleanly.

4. Separate API Object (Rejected)

Description: Defining a new API object (e.g., a NodePreemptionPolicy CRD) to manage preemption policies outside the core Node object, linked via a node selector.
Why Rejected: This approach is too heavyweight for a simple boolean toggle. It introduces the overhead of managing a new API type, implementing new controllers, and increasing memory usage in the control plane to cache these objects. The operational complexity outweighs the benefits.

5. Node Condition (Rejected)

Description: Representing the policy status using a Node Condition, e.g., type: PodResizePreemptionDisabled, status: "True".
Why Rejected: Node Conditions are designed to report the current observable state of a node (e.g., Ready, DiskPressure) as determined by the Kubelet or other controllers. They are not intended to be used by operators to configure desired behavior. Furthermore, conditions do not easily support the “multiple owner” scenario (e.g., multiple autoscalers) where multiple entities need to independently express their intent without overwriting each other.

KEP-5836: Scheduler Preemption for In-Place Pod Resize

KEP-5836: Scheduler Preemption for In-Place Pod Resize

Release Signoff Checklist

Summary

Motivation

Goals

Non-Goals

Proposal

User Stories (Optional)

Story 1: Pods in the Deferred resize status no longer require manual intervention.

Story 2: Reduction of disruption for critical workloads.

Story 3: Driving cluster autoscaling

Risks and Mitigations

Performance impact

Interaction with workload-aware preemption

Race between a Deferred resize and a new higher-priority pod

Design Details

How Deferred Resizes Integrate into the Scheduling Queue

Detecting Deferred Resizes in UpdatePod

Queueing Hints in NodeResourcesFit

Scheduler Restart and State Recovery

Conflicting Sources of Truth for the Pod

Processing Deferred Resizes in the Scheduling Cycle

Bypassing Irrelevant Constraints via RelevancePlugin

NodeName Plugin: Node Restriction via the PreFilter Phase

NodeResourcesFit Plugin: Calculating Resource Fit in the Filter Phase

Handling Node Evaluation Results

DefaultPreemption Plugin: Preemption Mechanism Adjustments in the PostFilter Phase

Preventing Unnecessary Victim Preemption

Preventing Additional Preemption Across Grace Periods and Restarts

Summary of Scheduling Cycle Flow

Scheduler Resource Reservation

Kubelet-Scheduler Preemption Interaction

Reconsideration Race Conditions

Synchronizing State via the PodResizeSchedulerFeasible Condition

Walkthrough: Competing Pod Scale-Up Race Cases

Preemption Policies

Pod Priority, Graceful Termination, and Pod Disruption Budget

Node-level Preemption Policy for In-Place Pod Resize

Policy Enforcement via DeferredPodScheduling Plugin

Multiple Owner Support

Policy Scope: Exclusivity to Pod Resize

Kubelet Preemption Bypass for Resize Requests

Failures and Reconsideration of Deferred pods

Failure Handler Adjustments for Deferred Pods

Interaction with Workload-Aware Scheduling (WAS) and Gang Scheduling

Metrics and Events

Metrics

New ALPHA metrics for deferred resize pods

Extending existing ALPHA metrics with a new operation label

Extending metrics with a new AssignedPodResize event

Metrics skipped for deferred resizing pods

Events

Pod Status & Conditions

External controllers

End-user observability

Test Plan

Prerequisite testing updates

Unit tests

Integration tests

e2e tests

Graduation Criteria

Alpha

Beta

GA

Upgrade / Downgrade Strategy

Version Skew Strategy

Production Readiness Review Questionnaire

Feature Enablement and Rollback

How can this feature be enabled / disabled in a live cluster?

Does enabling the feature change any default behavior?

Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)?

What happens if we reenable the feature if it was previously rolled back?

Are there any tests for feature enablement/disablement?

Rollout, Upgrade and Rollback Planning

How can a rollout or rollback fail? Can it impact already running workloads?

What specific metrics should inform a rollback?

Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?

Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?

Monitoring Requirements

Bypassing Irrelevant Constraints via `RelevancePlugin`

`NodeName` Plugin: Node Restriction via the PreFilter Phase

`NodeResourcesFit` Plugin: Calculating Resource Fit in the Filter Phase

`DefaultPreemption` Plugin: Preemption Mechanism Adjustments in the PostFilter Phase

Synchronizing State via the `PodResizeSchedulerFeasible` Condition

Policy Enforcement via `DeferredPodScheduling` Plugin

Extending existing ALPHA metrics with a new `operation` label

Extending metrics with a new `AssignedPodResize` event

Option 3: Reuse `status.nominatedNodeName` (Rejected)

1: Parking in Unschedulable Queue via `Permit` Rejection (Alpha Decision - Rejected for Beta)

2: Re-queueing on Node Scale-Up Events in `updatePod` (Rejected)

3: Re-queueing on `PodResizePending.LastProbeTime` Updates (Rejected)