Appearance

Your first debugging workflow

Turn a broken request into evidence at the API, workload, listener, Service, and DNS boundaries.

What you will understand

  • Choose logs and events according to the failure stage
  • Trace a request without restarting away useful evidence
  • Verify both a targeted repair and the original caller path

Builds on: DNS and service exposure

Mental model: locate the first failed boundary

“A request failed” is a symptom, not a diagnosis. Start by identifying the caller, exact destination and port, response or error, time window, namespace, and recent change. Then separate the API object path from the application packet path. A successful kubectl get proves access to the API, not that the web server works. A successful Pod port-forward tests a selected backend through a debugging tunnel, not necessarily the Service’s ordinary forwarding path.

Use the narrowest useful evidence first. Confirm the context and namespace so you do not investigate a different workload with the same name. Inspect the owning Deployment, its ReplicaSets, and current Pods. Check whether Pods exist, have node assignments, started containers, and became ready. Each transition has a responsible component and different failure evidence. Avoid deleting the failing Pod before reading its status and logs; replacement can discard the very observation needed to explain the incident.

Choose evidence for the startup stage

Separate unscheduled objects, image preparation failures, and executed-process failures.

Flow viewStep 1 of 3
Check assignment and conditionsA Pending Pod without a node needs scheduling evidence. A bound Pending Pod can instead be preparing images or mounts.. get -o wide + describe, active. Pod assignment + conditions, warning. Component event reasons, readyget -o wide + describe
get -o wide + describeIn focus
Pod assignment + conditions
Pod assignment + conditionsWarning
Component event reasons
Component event reasonsReady
get -o wide + describeIn focus
Pod assignment + conditionsWarning
Component event reasonsReady
Animation controls
Speed
0.5×
1×
2×
Explore a scenario
Normal behavior
Failure: delete before collecting evidence
Check assignment and conditions

A Pending Pod without a node needs scheduling evidence. A bound Pending Pod can instead be preparing images or mounts.

Read the complete visual sequence
  1. Check assignment and conditions

    A Pending Pod without a node needs scheduling evidence. A bound Pending Pod can instead be preparing images or mounts.

  2. Distinguish before-execution failure

    ImagePullBackOff means image preparation is retrying. No application process from that failed start exists to write useful application logs.

  3. Inspect a process that did execute

    For repeated exits, read termination reason, exit code, current logs, and previous-container logs if retained. These explain a different boundary than scheduling.

Failure: delete before collecting evidence

Failure: delete before collecting evidence

  1. A failing Pod contains useful state

    The current Pod has termination details and prior container output. The replacement may not preserve these observations.

  2. Deletion discards a useful target

    A controller creates Pod B, but commands aimed at A can no longer retrieve its ordinary live-object state or logs.

  3. Collect first, then change deliberately

    Capture necessary status and redacted logs, identify the likely cause, and make a targeted repair. Use centralized observability for durable incident history.

About this model

One current Pod is inspected; events and prior logs exist only while retained by the environment.

Walkthrough: inspect startup before transport

Use service-demo from the preceding lessons in its dedicated lab namespace. Replace ACTUAL_POD_NAME with a current Pod name from the first listing. Keep your verified context and namespace explicit when multiple clusters are configured.

kubectl get deployment service-demo
kubectl get pods -l app=service-demo -o wide
kubectl describe pod ACTUAL_POD_NAME
kubectl logs ACTUAL_POD_NAME -c web --tail=100 --timestamps
# Useful only if that container has a previous terminated instance:
kubectl logs ACTUAL_POD_NAME -c web --previous --tail=100
kubectl get events --sort-by=.metadata.creationTimestamp

Events describe observations by components, such as failed scheduling, image pulling, mount preparation, or probe failures. They are not an indefinite audit log and may be aggregated or expire. Container logs are the application’s stdout/stderr made available through the runtime and node log path. They are not every file in the container, and they are not durable centralized storage by themselves. A failed image pull has no running application process to emit its startup logs.

An unassigned Pending Pod points toward scheduling evidence. A bound Pod waiting for an image points toward reference, credentials, and registry access. A restarting container points toward termination reason, exit code, logs, probe results, or memory pressure. CrashLoopBackOff describes delayed retries after repeated failures; it does not identify the underlying application defect. Read the immediate cause before choosing a repair.

Follow a broken request one hop at a time

Once a backend runs, inspect the Service selector, ports, and EndpointSlices. Compare endpoint addresses with actual Pod IPs, readiness conditions with probe evidence, and target ports with the actual listener. Test from the same caller network when possible. A probe from your laptop may use a different DNS resolver, network route, firewall policy, or credential path from the failing Pod.

kubectl get service service-demo -o yaml
kubectl get pods -l app=service-demo --show-labels
kubectl get endpointslices -l kubernetes.io/service-name=service-demo -o yaml
# Use the existing debug client from the prior lesson.
kubectl exec discovery-client -- nslookup service-demo
kubectl exec discovery-client -- wget -qO- http://service-demo:8080/

If name lookup fails, inspect the caller’s resolver configuration and cluster DNS path. If lookup succeeds, retain that evidence and test address/port reachability. If a direct Pod-IP request works from the same client but the Service request fails, focus on frontend port mapping, endpoint records, and Service forwarding. If both fail, the listener, network reachability, policy, or application becomes more likely. No single test proves every possible path healthy.

Narrow a broken Service request

Compare DNS, frontend routing, endpoint selection, and backend listener evidence.

Flow viewStep 1 of 3
Reproduce from the actual caller networkThe client asks for debug-mismatch:8080. Record whether failure is DNS, connection, or an HTTP response before changing anything.. Debug client Pod, active. Cluster DNS answer, ready. debug-mismatch:8080, warningDebug client Pod
Debug client PodIn focus
Cluster DNS answer
Cluster DNS answerReady
debug-mismatch:8080
debug-mismatch:8080Warning
Debug client PodIn focus
Cluster DNS answerReady
debug-mismatch:8080Warning
Animation controls
Speed
0.5×
1×
2×
Explore a scenario
Normal behavior
Failure: a tunnel result is overgeneralized
Reproduce from the actual caller network

The client asks for debug-mismatch:8080. Record whether failure is DNS, connection, or an HTTP response before changing anything.

Read the complete visual sequence
  1. Reproduce from the actual caller network

    The client asks for debug-mismatch:8080. Record whether failure is DNS, connection, or an HTTP response before changing anything.

  2. Inspect endpoint and listener boundaries

    The Service has no matching ready endpoint because its selector is wrong. A direct backend or port-forward can still show the application itself responds.

  3. Repair and verify the original path

    Correct only the selector, observe ready endpoint records, then repeat the same client request. API acceptance alone is not the repair proof.

Failure: a tunnel result is overgeneralized

Failure: a tunnel result is overgeneralized

  1. A Pod port-forward succeeds

    The operator forwards directly to one Pod and receives a response. This proves that selected backend works through the tunnel.

  2. The original Service path still fails

    The client Pod uses DNS and Service forwarding, which the tunnel bypassed. Its selector mismatch remains unfixed.

  3. Validate each distinct path

    Keep the tunnel success as backend evidence. Repair the Service boundary and repeat the original client request before declaring recovery.

About this model

A normal ClusterIP Service and one same-namespace debug client are used; no external gateway is involved.

Practice: repair one deliberate mismatch

This valid Service object intentionally selects nothing in the previous workload. Save it as debug-mismatch.yaml and apply it only in the disposable lab namespace:

apiVersion: v1
kind: Service
metadata:
  name: debug-mismatch
spec:
  selector:
    app: deliberately-unmatched # The intended Pods use app: service-demo.
  ports:
    - port: 8080
      targetPort: http

Predict that the Service can receive a ClusterIP and DNS record while lacking usable endpoints. Verify each prediction, change only the selector to app: service-demo, apply, and repeat the original client request to http://debug-mismatch:8080/. The repair should change the backend evidence and the request result. Merely seeing “configured” after apply is not the success condition.

Pitfalls

A minimal image may have no shell, nslookup, or curl. Missing debug tools do not establish an application failure. Use an approved client Pod or later an ephemeral-debug-container workflow rather than modifying the production image during an incident. Redact credentials and user data from logs before sharing them. Also avoid assuming every HTTP 500 comes from the Pod you intended; proxies and upstream dependencies can produce their own responses.

Check your understanding

The Pod answers over port-forward, but the Service request fails from a client Pod. Is the Service proven healthy?

No. Port-forward reaches a selected Pod through a different path. Inspect Service selection, endpoint readiness, client-facing and target ports, DNS, and node forwarding from the failing caller. Preserve the successful backend test as a useful narrowing result, then verify the repaired original path and several representative requests.

Complete the module lab by recording symptom, evidence, responsible layer, targeted change, and repeated verification. You are ready to study resources and placement with this same boundary-by-boundary method.

Primary documentation

Go to the source

Reviewed 2026-10-27 · Kubernetes 1.37 · Related local exercise

Search the book

Search loads when you open it.