CPU throttling, OOM, and eviction
Identify the resource boundary before choosing a remedy.
What you will understand
- Distinguish CPU quota throttling from memory OOM termination.
- Explain node-pressure eviction and its evidence.
- Relate requests, limits and QoS without promising immunity.
Builds on: Topology and resource governance, Informers, caches, and workqueues
Mental model: different limits fail differently
A CPU limit can cause throttling when the container exhausts its allowed execution quota. The process is delayed rather than automatically killed for crossing a CPU limit. Latency and request queues can grow even while average CPU graphs look modest, particularly when sampling hides short bursts. A CPU request instead informs scheduling and relative allocation under contention; it is not a promise that every request receives a dedicated physical core.
Memory has a different failure boundary. A container that cannot satisfy allocations within its enforced memory limit can be terminated by the kernel OOM mechanism. Kubernetes can report OOMKilled in container termination state, and restart policy determines what happens next. Memory requests guide placement and eviction considerations; they do not reserve every byte against every failure. A process can also fail allocations itself, so an application exception is not automatically evidence of a kernel OOM kill.
CPU delays work; memory can terminate it
The enforced memory boundary can trigger a kernel OOM kill recorded in container termination state.
CPU and memory requests inform scheduling; they are distinct from enforced runtime limits.
Read the complete visual sequence
- Requests inform placement
CPU and memory requests inform scheduling; they are distinct from enforced runtime limits.
- CPU quota is exhausted
CPU work waits for execution budget, potentially increasing latency without terminating the process.
- Memory allocation cannot continue
The enforced memory boundary can trigger a kernel OOM kill recorded in container termination state.
Misdiagnose throttling as OOM
A slow process needs evidence before a memory-limit change.
- Latency rises
The CPU-limited container remains Running while requests queue.
- No memory termination
There is no OOMKilled evidence; changing memory does not address CPU quota exhaustion.
- Inspect the right signal
Correlate quota throttling with latency and concurrency before changing CPU configuration.
About this model
Linux cgroup-style CPU quota and memory behavior is summarized; kernel and runtime accounting affect exact observations.
Walkthrough: pressure can remove a whole Pod
Node-pressure eviction is the kubelet reclaiming resources when thresholds for memory, disk space, inodes or other supported signals are met. It differs from a container reaching its own memory limit. An evicted Pod is normally marked Failed with an eviction reason and message; a workload controller may create a replacement. The replacement may land on another node, but only if scheduling and capacity allow it. Eviction is not equivalent to a container restart inside the same Pod identity.
QoS classes help characterize resource configuration, but Guaranteed is not immunity from every eviction or host failure. Eviction ranking considers whether usage exceeds requests, Pod priority and relative excess for the pressured resource. Disk and inode pressure have their own accounting complications. Preserve the termination message, events, node conditions and container lastState before a cleanup or new rollout hides the evidence. A single exit code without context is usually insufficient to distinguish all these paths.
Node pressure differs from a container OOM
A controller can create a new Pod, but it still needs a feasible node and does not retain the evicted UID.
The kubelet detects a configured resource pressure signal and attempts reclaim.
- Worker node to Kubeletthreshold signal
Read the complete visual sequence
- Observe node pressure
The kubelet detects a configured resource pressure signal and attempts reclaim.
- Choose eviction when needed
If reclaim is insufficient, the kubelet ranks eligible Pods using relevant usage, requests and priority.
- Replacement needs capacity
A controller can create a new Pod, but it still needs a feasible node and does not retain the evicted UID.
Replacement also cannot fit
Eviction does not create additional cluster capacity.
- Pressure removes a Pod
A node-pressure eviction produces Pod failure evidence distinct from a container restart.
- Controller requests replacement
Desired replicas remain, so a new Pod is created.
- Capacity remains exhausted
The new Pod can remain Pending or encounter pressure again until demand or capacity changes.
About this model
Eviction thresholds and ranking depend on the pressured resource and kubelet configuration; QoS is not universal protection.
This complete memory-limited Pod is an optional small demonstration similar to the m17 download. It intentionally allocates memory through a memory-backed volume so the allocation is charged to the container. Its local outcome depends on runtime and kernel accounting; the downloadable script checks for OOMKilled rather than claiming every environment behaves identically.
apiVersion: v1
kind: Pod
metadata:
name: m17-memory
namespace: kv-lab
labels:
lab: m17
spec:
restartPolicy: Never
containers:
- name: fill
image: busybox:1.37.0
command: [dd, if=/dev/zero, of=/memory/blob, bs=1M, count=96]
resources:
requests:
memory: 16Mi
cpu: 10m
limits:
memory: 32Mi
cpu: 100m
volumeMounts:
- name: memory
mountPath: /memory
volumes:
- name: memory
emptyDir:
medium: Memory
kubectl --context kind-kv-lab -n kv-lab get pod m17-oom -o yaml
kubectl --context kind-kv-lab -n kv-lab describe pod m17-oom
kubectl --context kind-kv-lab get nodes
kubectl --context kind-kv-lab -n kv-lab get events --sort-by=.metadata.creationTimestamp
Practice: explain the observed termination
Run the m17 local exercise rather than generating node-wide pressure on your laptop. The scripted Pod has a bounded memory limit and the script waits for OOMKilled. Inspect the exact container termination reason and compare it with the Pod phase. If the expected reason does not appear, inspect logs and runtime accounting; do not automatically increase allocation until the host becomes unstable. This is a container-limit exercise, not a proof of production eviction ordering.
For a throttling investigation, collect CPU quota/throttling metrics from the runtime or monitoring system and correlate them with application latency, concurrency and queue length. kubectl top requires a metrics provider and gives a sampled utilization view, not the complete throttling history. For eviction, inspect node pressure conditions and kubelet events before changing container memory limits. Choose the remedy at the boundary the evidence actually identifies.
Pitfalls: bigger limits can move the bottleneck
Increasing memory limits without increasing requests can let Pods consume more than the node can safely support. Increasing CPU limits may improve one container’s burst behavior while increasing contention for neighbors. A memory-backed emptyDir consumes memory and can contribute to memory pressure; a disk-backed emptyDir has different pressure behavior. Include startup peaks, caches, heap behavior and realistic load when setting requests. Avoid using restart count alone as the health signal: repeated replacement Pods can reset the visible history under new UIDs.
Check your understanding
A request becomes slow at its CPU limit but the container stays Running. Is that inconsistent?
No. CPU quota exhaustion commonly throttles execution rather than killing the container. Check throttling and queueing evidence. OOMKilled and node-pressure eviction are different mechanisms with different termination and event evidence.
Next, examine how an autoscaler interprets CPU usage relative to requests. Resource configuration changes both placement and the feedback signal used to scale.
Go to the source
Reviewed 2026-10-27 · Kubernetes 1.37 · Related local exercise