Appearance

Termination, disruption, and drain

Connect graceful shutdown to readiness, voluntary eviction budgets and node maintenance.

What you will understand

  • Trace the Pod termination grace period and application signal handling
  • Calculate whether a PDB permits a voluntary eviction
  • Drain a node while recognizing failures that a PDB cannot prevent

Builds on: DaemonSets, Jobs, and CronJobs

Mental model: shutdown is a coordinated transition with a deadline

Deleting a Pod starts a termination workflow rather than instantly erasing every process and connection. The kubelet observes the grace period and arranges shutdown of containers. If configured, a preStop hook runs as part of that grace-period budget before the normal stop signal. The application should handle its stop signal, stop admitting new work, complete or hand off existing requests, and exit within the available time. If it does not exit in time, the runtime can force termination.

Endpoint changes occur alongside shutdown, not as one atomic global switch. A terminating endpoint is marked accordingly and ordinarily not ready for new normal traffic; implementations may use additional serving information for graceful handling. Load balancers, proxies and clients learn these changes asynchronously, and existing connections may remain active. A fixed sleep in preStop can allow propagation time, but it is not a substitute for a correct application shutdown protocol or a guarantee about every network device.

Traffic removal and process shutdown share a deadline

Trace deletion, endpoint readiness, preStop, stop signal and final process exit.

Lifecycle viewStep 1 of 4
Deletion beginsThe API records deletion and a grace period. Endpoint state and node shutdown react asynchronously to that transition.. API Pod, Handles active requests, warning. EndpointSlice, Terminating / not ready, active. kubelet, Grace-period deadline, active. preStop hook, Runs within grace budget, idle. Application process, Handle stop signal and exit, idle. Clients / proxy, Observe endpoint updates, idleAPI Pod: Handles active requests
API PodHandles active requestsWarning
EndpointSlice: Terminating / not ready
EndpointSliceTerminating / not readyIn focus
kubelet: Grace-period deadline
kubeletGrace-period deadlineIn focus
preStop hook: Runs within grace budget
preStop hookRuns within grace budgetContext
Application process: Handle stop signal and exit
Application processHandle stop signal and exitContext
Clients / proxy: Observe endpoint updates
Clients / proxyObserve endpoint updatesContext
API PodHandles active requestsWarning
EndpointSliceTerminating / not readyIn focus
kubeletGrace-period deadlineIn focus
preStop hookRuns within grace budgetContext
Application processHandle stop signal and exitContext
Clients / proxyObserve endpoint updatesContext
Animation controls
Speed
0.5×
1×
2×
Explore a scenario
Normal behavior
Hook consumes the grace budget
Deletion begins

The API records deletion and a grace period. Endpoint state and node shutdown react asynchronously to that transition.

Read the complete visual sequence
  1. Deletion begins

    The API records deletion and a grace period. Endpoint state and node shutdown react asynchronously to that transition.

  2. Stop new traffic and run hook

    Endpoint updates discourage new ordinary traffic, while preStop uses part of the same grace-period budget.

  3. Signal the process

    After the hook, the runtime sends the normal stop signal. The application should finish or hand off in-flight work and stop accepting new work.

  4. Exit before the deadline

    A graceful process exit completes shutdown. Existing connections and upstream propagation must be considered in the application design.

Hook consumes the grace budget

A long hook leaves insufficient time for application shutdown.

  1. Slow preStop starts

    The grace countdown is already running while the hook waits.

  2. Application receives little time

    The hook consumes most of the budget before the process can finish active work.

  3. Deadline forces termination

    If shutdown does not complete within the available grace handling, the runtime can kill remaining processes. Increase budgets only with a measured shutdown protocol.

About this model

One ordinary container and a normal deletion; precise propagation time depends on components.

Walkthrough: allow maintenance without removing too many healthy replicas

A PodDisruptionBudget, or PDB, expresses the availability that should remain during covered voluntary disruptions. It is normally enforced through the Eviction API used by maintenance tooling. This PDB requires two healthy matching Pods to remain available.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-availability
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: api

With three healthy selected replicas, one voluntary eviction can usually proceed while respecting this requirement. Once only two healthy replicas remain, another covered eviction must wait until health recovers. The budget does not create replicas or enough capacity for replacements. A Deployment must maintain the intended count, and the scheduler needs somewhere eligible to put the replacements. A single-replica application with minAvailable: 1 can block ordinary maintenance indefinitely unless another healthy instance can be introduced.

A PDB does not prevent involuntary node crashes, kernel failures, storage failures or every direct Pod deletion. It is not a cap on unavailability caused by a Deployment’s own rolling-update strategy. Rollout controls such as maxUnavailable and maxSurge govern that controller’s behavior separately. Empty selectors in policy/v1 can select every Pod in the namespace, so keep the intended label boundary explicit.

Eviction asks permission from the current health budget

Follow cordon, PDB health evaluation, eviction and replacement during maintenance.

Policy viewStep 1 of 4
Cordon the workerThe node becomes unschedulable for ordinary new Pods. Existing workloads do not move merely because it is cordoned.. Worker A, Node to maintain, warning. Drain, Cordon then eviction requests, idle. PDB, minAvailable=2; healthy=3, idle. API Pod A, On Worker A, idle. Replica controller, Maintains desired count, idle. Worker B, Has replacement capacity, idleWorker A: Node to maintain
Worker ANode to maintainWarning
Drain: Cordon then eviction requests
DrainCordon then eviction requestsContext
PDB: minAvailable=2; healthy=3
PDBminAvailable=2; healthy=3Context
API Pod A: On Worker A
API Pod AOn Worker AContext
Replica controller: Maintains desired count
Replica controllerMaintains desired countContext
Worker B: Has replacement capacity
Worker BHas replacement capacityContext
Worker ANode to maintainWarning
DrainCordon then eviction requestsContext
PDBminAvailable=2; healthy=3Context
API Pod AOn Worker AContext
Replica controllerMaintains desired countContext
Worker BHas replacement capacityContext
Animation controls
Speed
0.5×
1×
2×
Explore a scenario
Normal behavior
A replica was already unready
Cordon the worker

The node becomes unschedulable for ordinary new Pods. Existing workloads do not move merely because it is cordoned.

Read the complete visual sequence
  1. Cordon the worker

    The node becomes unschedulable for ordinary new Pods. Existing workloads do not move merely because it is cordoned.

  2. Ask for an eviction

    Drain requests voluntary eviction. With three healthy replicas and minimum two, the PDB can permit one disruption.

  3. Replace the removed replica

    The controller sees fewer current replicas than desired and creates a replacement, which still needs suitable scheduling and storage.

  4. Wait for health before more removal

    The replacement must become healthy before the next covered disruption can use a renewed budget.

A replica was already unready

Only two selected replicas are healthy before drain begins.

  1. Health budget exhausted

    minAvailable=2 and healthy=2 leave no room for another covered healthy eviction.

  2. Eviction waits

    The PDB rejects the voluntary removal that would reduce healthy availability below the budget.

  3. Repair health or plan capacity

    Recover the unhealthy replica or add a viable replacement. A PDB cannot prevent node crashes or create capacity by itself.

About this model

Three selected replicas with minAvailable=2; ordinary Eviction API path and a controller-managed workload.

Practice: read the budget before touching the node

kubectl get pdb api-availability
kubectl describe pdb api-availability
kubectl get pods -l app=api -o wide
# Only use a disposable worker obtained from kubectl get nodes.
kubectl cordon LAB_WORKER
kubectl drain LAB_WORKER --ignore-daemonsets
# After maintenance and health checks:
kubectl uncordon LAB_WORKER

Cordon marks a node unschedulable for ordinary new placement. It does not move existing Pods. Drain attempts to remove appropriate workloads, normally using evictions that respect PDBs, while handling node-local and unmanaged workloads according to its flags. DaemonSet Pods are normally handled separately; --ignore-daemonsets permits drain to proceed without deleting those agents. Local emptyDir data and Pods without suitable controllers require conscious choices; do not add force or deletion flags merely to make an error disappear.

Watch the replacement Pods, events and PDB disruptionsAllowed while a disposable node is drained in the workload lab. If drain waits, first ask whether enough matching Pods are healthy, whether a replacement is Pending, and whether storage topology or affinity prevents relocation. A budget reporting zero permitted disruptions may be doing exactly what was requested. Increasing a timeout does not create missing capacity.

Pitfalls: a force action changes the safety boundary

Direct deletion, forceful termination and bypassing eviction checks can trade a blocked maintenance operation for an availability or data-integrity problem. This is particularly serious for stateful members on unreachable nodes: API deletion is not evidence that the old process has stopped writing. Establish node fencing or application-specific ownership before allowing a replacement writer. Test shutdown with long-lived requests so you can observe both application completion and traffic removal rather than relying only on a Pod phase.

Check your understanding

Three replicas have minAvailable=2, but one is already unready. Can drain evict another healthy replica immediately?

Normally no. Only two are currently healthy, so evicting another healthy member would violate the requested availability. Investigate and restore health or deliberately change the availability and maintenance plan. The PDB cannot compensate for the pre-existing unhealthy replica, and bypassing it removes the protection it was designed to provide.

Next, define who can perform these operations. Availability policy is meaningful only when identities and permissions also constrain who can change it or bypass its workflow.

Primary documentation

Go to the source

Reviewed 2026-10-27 · Kubernetes 1.37 · Related local exercise

Search the book

Search loads when you open it.