Probes and graceful recovery
Give the kubelet separate startup, traffic-readiness, and restart signals.
What you will understand
- Distinguish startup, readiness, and liveness consequences
- Choose probe paths and budgets that match failure semantics
- Explain termination grace without assuming instant traffic draining
Builds on: Rolling updates and rollback
Mental model: three questions with different consequences
A startup probe asks whether a container’s application has completed startup. While a configured startup probe has not succeeded, the kubelet does not run that container’s readiness or liveness probes. This gives slow initialization a deliberate budget without making ongoing liveness checks permanently slow. When consecutive startup failures reach failureThreshold, the kubelet terminates the container and applies its restart policy.
A readiness probe asks whether this container should currently contribute to a ready Pod. Failed readiness can make the Pod unready, affecting ordinary Service endpoint eligibility after observers propagate the change. It does not restart the container. The process can stay alive, warm caches, retry dependencies, or recover itself. Readiness is about traffic eligibility rather than punishment for an error.
A liveness probe asks whether restarting this container is a useful repair. After enough consecutive failures, the kubelet terminates the container and applies restart behavior. Use it for conditions such as a local deadlock that a restart can plausibly fix. Making liveness fail because a shared database is temporarily unavailable can cause every healthy application process to restart together, adding load to a failing dependency.
Three probes, three decisions
Show startup gating, readiness eligibility, and liveness-driven local recovery.
The startup probe runs while initialization proceeds. Readiness and liveness wait until it succeeds. Reaching its consecutive failure threshold makes the kubelet terminate and restart this container under the assumed Always policy.
- Kubelet probes to Application initializingstartup checks
- Startup not yet successful blocked toward Readiness/liveness gatedwait until success
Read the complete visual sequence
- Startup has its own budget
The startup probe runs while initialization proceeds. Readiness and liveness wait until it succeeds. Reaching its consecutive failure threshold makes the kubelet terminate and restart this container under the assumed Always policy.
- Readiness controls eligibility
After startup succeeds, readiness can mark the Pod unready. The process stays running and endpoint readiness is updated asynchronously.
- Liveness requests a restart
If liveness repeatedly fails beyond tolerance, the kubelet terminates and restarts this container under its restart policy. This is local recovery, not a new Pod UID.
Failure: shared dependency used as liveness
Failure: shared dependency used as liveness
- The database becomes unavailable
Every app process is alive, but a liveness check includes the same failed database dependency.
- Healthy processes restart together
The kubelets restart apps even though restarting cannot repair the database. Reconnection traffic can worsen recovery.
- Match signal to repair
Use liveness for locally recoverable stalls. Model dependency degradation with carefully designed readiness and application retry behavior.
About this model
One ordinary container uses Always restart policy; Service routing behavior is the normal non-publishNotReadyAddresses case.
Walkthrough: configure real paths
This complete training Pod serves nginx’s existing root path. It demonstrates mechanics rather than an ideal application health API. A real application should expose deliberately designed startup, readiness, and liveness semantics, with cheap checks and controlled dependency behavior.
apiVersion: v1
kind: Pod
metadata:
name: probes-demo
spec:
terminationGracePeriodSeconds: 30
containers:
- name: web
image: nginx:1.27.5
ports:
- name: http
containerPort: 80
startupProbe:
httpGet:
path: /
port: http
periodSeconds: 2
failureThreshold: 30 # Roughly a 60-second startup window, not an SLA.
readinessProbe:
httpGet:
path: /
port: http
periodSeconds: 5
failureThreshold: 2
livenessProbe:
httpGet:
path: /
port: http
periodSeconds: 10
failureThreshold: 3
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "nginx -s quit"]
The named http port resolves to the container’s declared port 80. HTTP probe success depends on the returned status, not whether a page body contains expected business data. Defaults such as timeout also matter. A threshold times period is a useful planning approximation, but probe scheduling, execution time, initial delay, and status propagation mean it is not an exact user-visible failure duration.
kubectl apply -f probes-demo.yaml
kubectl get pod probes-demo -w
kubectl describe pod probes-demo
kubectl get pod probes-demo -o jsonpath='{.status.containerStatuses[0].restartCount}'
A healthy lab should see startup succeed, readiness become true, and no unexpected restart growth. For a new disposable Pod, change only the readiness path to /does-not-exist. Expect readiness failures without liveness-triggered restarts because liveness still checks /. Change only the liveness path in another new Pod to observe a different consequence. Keep these failure exercises isolated from real workloads.
Practice: trace a deliberate shutdown
Pod deletion sets a deletion timestamp and starts graceful termination. The kubelet runs a configured preStop hook within the termination grace period and asks the runtime to signal the container’s main process. The default signal behavior is commonly TERM, though image or explicit stop-signal configuration can affect it. After the grace period, remaining processes can be forcibly killed. Hook execution consumes the same grace budget; it is not extra unlimited time.
Spend the grace period deliberately
Trace deletion, endpoint eligibility changes, hooks, and bounded exit.
The Pod receives a deletion timestamp. Endpoint observers update terminating/readiness conditions while the kubelet begins local shutdown.
- Pod terminating to Endpoint updatesobserve deletion
- Pod terminating to Kubelet shutdownobserve deletion
Read the complete visual sequence
- Deletion begins two asynchronous paths
The Pod receives a deletion timestamp. Endpoint observers update terminating/readiness conditions while the kubelet begins local shutdown.
- Run the hook inside the budget
A preStop hook asks nginx to quit gracefully. The hook consumes the existing grace period; it does not add unlimited draining time.
- Exit before the deadline
The app finishes bounded work and exits. New traffic eligibility and existing connections depend on propagation and the actual networking implementation.
Failure: hook consumes the whole grace period
Failure: hook consumes the whole grace period
- A slow hook blocks progress
preStop sleeps for most of the grace period while the app still has work to drain.
- Little shutdown time remains
The app receives its termination handling opportunity late. Requests may still be active when the grace budget expires.
- Remaining processes can be killed
Design a bounded hook and a signal-aware application, then test real request draining against the actual load-balancer path.
About this model
Traffic propagation and kubelet termination happen concurrently; default graceful deletion is shown.
Endpoint changes happen asynchronously with termination. A terminating endpoint is normally not ready for ordinary new Service traffic, while its serving/terminating conditions can help implementations handle draining. Existing connections, load-balancer refresh, and special fallback behavior complicate any claim of immediate zero traffic. The application should stop accepting unwanted new work, finish bounded in-flight work, and exit before the grace deadline. A fixed sleep is only a timing guess unless validated against the actual request path.
Pitfalls
A kubelet HTTP probe targets the Pod endpoint; it does not validate public DNS, ingress routing, TLS at an external gateway, or every application route. Supplement it with end-to-end request tests. Keep probe responses cheap so a high replica count does not create a new health-check bottleneck.
Check your understanding
Readiness fails for one minute while liveness succeeds. Should the restart count necessarily increase?
No. Readiness affects the Pod’s readiness and ordinary endpoint eligibility; it does not request a restart. The process may recover in place. If restart count rises, inspect exits, liveness/startup failures, OOM evidence, or other actual restart causes instead of attributing the restart to readiness alone.
Use the module lab to compare probe consequences and a bounded shutdown. Next, inspect the Service and EndpointSlice objects that consume readiness information.
Primary documentation
Go to the source
Reviewed 2026-10-27 · Kubernetes 1.37 · Related local exercise