Debug the failing workload
Turn a visible Pod status into evidence about scheduling, image retrieval, process failure or runtime inspection.
What you will understand
- Distinguish Pod phase from container waiting and termination reasons
- Choose evidence for Pending, ImagePullBackOff and CrashLoopBackOff
- Use ephemeral containers within explicit permission and runtime limits
Builds on: Protecting Secrets and tenant boundaries
Mental model: the status column is a clue, not a root cause
Start a workload investigation by locating its current lifecycle boundary. Has the API accepted the object? Has a controller created a Pod? Has the scheduler assigned a node? Has the runtime retrieved images and prepared volumes? Has the process started and failed? Each boundary has different evidence. Reading application logs cannot explain a scheduler that has never placed the Pod, and adding memory cannot repair an image name that does not exist.
Pending is a Pod phase, not a synonym for insufficient CPU. It can include time waiting for scheduling and time preparing containers after assignment. A CrashLoopBackOff display is a container waiting reason indicating repeated restarts with restart backoff; it is not a separate Pod phase. A process that repeatedly exits successfully can also enter this loop under restartPolicy: Always. ImagePullBackOff likewise describes delayed retry after image retrieval failure. Inspect the structured container states, Pod conditions and events so that a convenient CLI status string does not become the entire diagnosis.
Find the earliest boundary that has not succeeded
Distinguish an unscheduled Pod, image retrieval backoff and a repeatedly crashing process.
An unassigned Pod with FailedScheduling needs scheduler evidence, not application logs from a process that never ran.
- Pod object blocked toward Schedulernot yet assigned
- Scheduler to EvidenceFailedScheduling
Read the complete visual sequence
- Pending before assignment
An unassigned Pod with FailedScheduling needs scheduler evidence, not application logs from a process that never ran.
- Assigned but image unavailable
After node assignment, a pull error can produce ImagePullBackOff. Read the exact registry or network failure in events.
- Process exits repeatedly
A retrieved image can start and then exit. Restart backoff produces CrashLoopBackOff as a container waiting reason.
- Choose evidence by boundary
Previous logs explain a terminated process when available, while events explain placement and setup. Change only a demonstrated cause.
Liveness causes the restart loop
The application is slow to start and a probe repeatedly terminates it.
- Process starts
The image and configuration are usable enough for the process to begin startup.
- Probe kills before readiness
An overly aggressive liveness probe repeatedly triggers restarts during legitimate startup.
- Correct probe semantics
Use a suitable startup probe and evidence-based timings; increasing memory would not repair this demonstrated probe failure.
About this model
One application container; displayed waiting reasons are not separate Pod phases.
Walkthrough: capture the evidence before changing the workload
kubectl get deployment,replicaset,pod -n learning -o wide
kubectl describe pod FAILING_POD -n learning
kubectl get events -n learning --sort-by=.metadata.creationTimestamp
kubectl logs FAILING_POD -n learning -c app --tail=100
kubectl logs FAILING_POD -n learning -c app --previous --tail=100
kubectl get pod FAILING_POD -n learning -o yaml
Replace the resource and container names with those shown by the first commands. --previous retrieves logs from the previous terminated container instance when available, which is useful when the current instance has not yet emitted the failing startup message. An error saying no previous instance exists is itself a limit of that evidence source. Capture logs promptly because log rotation, Pod replacement and node loss can remove local history. Avoid copying tokens, connection strings or sensitive user payloads into an incident note.
For an unscheduled Pending Pod, read FailedScheduling events and compare requests, labels, affinity, taints and volume constraints. If a node is already assigned, inspect image, volume and sandbox events instead. For ImagePullBackOff, distinguish image-not-found, authentication, authorization, rate limiting, network and certificate failures from the actual event message. The kubelet pulls images using its configured credentials and referenced pull Secrets; a successful docker pull on your laptop does not prove the worker has the same network path or identity.
For CrashLoopBackOff, read the last termination reason, exit code, restart count and previous logs. A failing liveness probe can repeatedly kill a process that would otherwise remain alive; a startup probe can protect a deliberately slow startup. An OOM kill has a different evidence pattern from an application configuration exception. Fix the smallest demonstrated cause and preserve the manifest revision so you can explain what changed.
Bring tools to the Pod network namespace
Follow authorization and admission before adding an ephemeral container for scoped investigation.
The application has no shell, but the operator needs to test the listener from its network namespace.
- Operator to API subresourceadd debug container
Read the complete visual sequence
- Ask a specific question
The application has no shell, but the operator needs to test the listener from its network namespace.
- Check permission and policy
The subresource request must pass authorization and admission. Debugging does not automatically grant host access.
- Start tools beside the app
The ephemeral container joins the Pod network and may target process visibility when supported by the runtime.
- Observe without replacing the app
Test the relevant localhost port or namespace evidence. The debug container does not replace the application or become its deployment revision.
Debug permission is denied
The operator is not allowed to update the ephemeral-container subresource.
- Application remains unchanged
The application Pod is still running or failing in its existing state.
- API rejects debug addition
Authorization denies the subresource update before any new container can run.
- Use authorized evidence
Read permitted logs and events or use an approved separate diagnostic Pod; request narrowly scoped access only if the task needs this capability.
About this model
Runtime supports the requested target visibility; privileges and process namespace access are not universal.
Practice: inspect a minimal image without changing its production contents
Ephemeral containers add a debug container to an existing Pod through a dedicated subresource. They are useful when the application image lacks a shell or diagnostic tools. They need permission and appropriate admission settings. They share the Pod’s network namespace, while process visibility targeting depends on the runtime and configuration. Seeing localhost does not automatically mean seeing every target process or its full filesystem.
# Use a throwaway Pod and an approved debug image in the local lab.
kubectl debug -it FAILING_POD -n learning \
--image=busybox:1.36 --target=app
# From the debug shell, inspect connectivity without printing credentials.
# An exit returns to your local shell; it does not delete the application Pod.
A debug container is added to the Pod’s specification and is not a general replacement for the application container or a routine deployment mechanism. It cannot make a not-yet-running image supply logs, and the chosen image may still lack the tool you need. A copied debug Pod or a disposable separate client can be safer when changing the live Pod would introduce unnecessary risk. Use only the privileges required for the question being tested.
Pitfalls: repeated restarts can hide the first useful error
Deleting all failed Pods before capturing events and previous logs erases the clearest evidence. A controller may instantly create fresh Pods with the same defect, making the system appear active without progress. Compare the failing Pod’s image, configuration references and owning ReplicaSet to the intended revision. Avoid an unrelated bundle of fixes because a green Pod afterward would not establish which hypothesis was correct.
Check your understanding
A Pod shows Pending, has a nodeName, and reports FailedMount. Should you first lower its CPU request?
No. Assignment has already occurred, so the immediate failing boundary is volume setup on the selected node. Read the mount event, claim and driver evidence. Lowering CPU may create another revision but does not address a demonstrated mount failure.
Use the diagnostics lab to record a symptom, a hypothesis and the single observation that can disprove it. Next, apply the same boundary-by-boundary method to networking and storage.
Primary documentation
Go to the source
Reviewed 2026-10-27 · Kubernetes 1.37 · Related local exercise