Appearance

Debug network and storage failures

Use comparative probes to separate DNS, Service selection, forwarding, policy, binding and mounting.

What you will understand

  • Locate a failed request using DNS, Service and direct endpoint probes
  • Distinguish no-ready-endpoint failures from a broken data plane
  • Classify storage failures as binding, attachment, mount or application access

Builds on: Debug the failing workload

Mental model: compare paths while preserving the source

A failed request has several possible boundaries: name resolution, Service selection, ready endpoint publication, data-plane forwarding, policy and the application listener. Test these in an order that turns each success into a constraint on the remaining hypotheses. Keep the same source Pod for comparisons because two sources may have different network policies, DNS settings, nodes or routes. A successful curl from your laptop is not equivalent to a successful in-cluster request.

Begin with the symptom in precise terms: source Pod and node, destination name or address, protocol, port and observed timeout or response. A DNS error is different from a TCP connection refusal, and both differ from an HTTP 500 response. These observations guide the next test, but implementation-generated responses and timeouts are not absolute proof of one root cause. Record enough context to reproduce the result before editing the workload.

Remove one indirection from the failing request

Compare DNS, ClusterIP and direct endpoint traffic from one diagnostic client.

Flow viewStep 1 of 4
Test resolutionResolve the Service name from the same client and inspect resolver configuration if this fails.. Client Pod, Keep source constant, active. DNS, Resolve api Service name, active. Service VIP, ClusterIP:80, idle. EndpointSlice, Ready Pod address, idle. Backend Pod, Listens on 8080, idle. Data plane / policy, Implementation-specific forwarding, idleClient Pod: Keep source constant
Client PodKeep source constantIn focus
DNS: Resolve api Service name
DNSResolve api Service nameIn focus
Service VIP: ClusterIP:80
Service VIPClusterIP:80Context
EndpointSlice: Ready Pod address
EndpointSliceReady Pod addressContext
Backend Pod: Listens on 8080
Backend PodListens on 8080Context
Data plane / policy: Implementation-specific forwarding
Data plane / policyImplementation-specific forwardingContext
Client PodKeep source constantIn focus
DNSResolve api Service nameIn focus
Service VIPClusterIP:80Context
EndpointSliceReady Pod addressContext
Backend PodListens on 8080Context
Data plane / policyImplementation-specific forwardingContext
Animation controls
Speed
0.5×
1×
2×
Explore a scenario
Normal behavior
DNS egress is blocked
Test resolution

Resolve the Service name from the same client and inspect resolver configuration if this fails.

Inspect this state in YAML / kubectl
kubectl exec client -- nslookup api
Read the complete visual sequence
  1. Test resolution

    Resolve the Service name from the same client and inspect resolver configuration if this fails.

  2. Check usable selection

    Inspect selector labels and readiness conditions. A Service with no usable backend can still resolve successfully.

  3. Probe the Service path

    Send to ClusterIP and Service port. This tests virtual-address forwarding plus the remaining backend path.

  4. Compare direct endpoint traffic

    Bypass DNS and Service selection using the ready Pod address and target port from the same client. A difference narrows the failing boundary.

DNS egress is blocked

The client cannot resolve names but direct address traffic is permitted.

  1. Name lookup times out

    An egress-isolated client has no suitable DNS allow rule.

  2. Direct address request works

    The application path can work when discovery is bypassed; this does not prove DNS is healthy.

  3. Repair DNS access deliberately

    Verify DNS workload health and actual labels, then allow the required UDP/TCP DNS path rather than opening every egress destination.

About this model

One healthy-looking backend; actual provider may transform packet addresses at different points.

Walkthrough: take discovery out of the path one step at a time

kubectl get service api -o yaml
kubectl get pods -l app=api -o wide
kubectl get endpointslices -l kubernetes.io/service-name=api -o yaml
kubectl exec client -- nslookup api.learning.svc.cluster.local
kubectl exec client -- cat /etc/resolv.conf
kubectl exec client -- wget -T 3 -qO- http://api:80
# Substitute a ready endpoint address and its actual target port.
kubectl exec client -- wget -T 3 -qO- http://POD_IP:8080

If DNS fails, inspect the client’s resolver configuration, DNS policy and namespace-qualified name. Then inspect the cluster DNS Service, its ready endpoints and DNS workload health. Egress isolation can block UDP or TCP DNS, while a custom cluster domain can make a copied fully qualified name wrong. Do not restart all DNS Pods simply because a short name in the wrong namespace failed.

If DNS works, inspect whether the Service selector matches actual Pod labels and whether slices contain ready endpoints. Missing readiness can reflect a correct probe protecting clients from an unhealthy process or a misconfigured probe that never succeeds. If direct endpoint traffic works while the Service virtual address fails from the same source, investigate the Service data plane and port mapping. If both fail, compare listener state, ingress and egress policies, node reachability and provider-specific routing. A localhost test inside the destination does not prove the process listens on the Pod interface.

Follow claim, attach, mount and file access separately

Classify storage errors using PVC status, Pod events and process permissions.

Storage viewStep 1 of 4
Check binding firstA Pending claim needs class, provisioner and consumer-topology evidence. A Bound claim has passed that checkpoint.. PVC, Pending or Bound, active. Provisioner, Class and topology, active. Attachment, Backend → selected worker, idle. Node mount, Publish filesystem into Pod, idle. App process, UID/GID accesses /data, idlePVC: Pending or Bound
PVCPending or BoundIn focus
Provisioner: Class and topology
ProvisionerClass and topologyIn focus
Attachment: Backend → selected worker
AttachmentBackend → selected workerContext
Node mount: Publish filesystem into Pod
Node mountPublish filesystem into PodContext
App process: UID/GID accesses /data
App processUID/GID accesses /dataContext
PVCPending or BoundIn focus
ProvisionerClass and topologyIn focus
AttachmentBackend → selected workerContext
Node mountPublish filesystem into PodContext
App processUID/GID accesses /dataContext
Animation controls
Speed
0.5×
1×
2×
Explore a scenario
Normal behavior
Filesystem permission denied after mount
Check binding first

A Pending claim needs class, provisioner and consumer-topology evidence. A Bound claim has passed that checkpoint.

Read the complete visual sequence
  1. Check binding first

    A Pending claim needs class, provisioner and consumer-topology evidence. A Bound claim has passed that checkpoint.

  2. Check attachment ownership

    An attachment error can reflect topology or another active node attachment. Establish writer ownership before forcing detach.

  3. Check node publication

    FailedMount points to a later checkpoint such as backend connectivity, mount options, credentials or filesystem setup.

  4. Check application permission

    A running container with permission denied may have a healthy mount but incompatible UID, group, mode or read-only settings.

Filesystem permission denied after mount

The claim is Bound and mounted but the process cannot write its intended directory.

  1. Infrastructure checkpoints pass

    The backend attaches and the filesystem appears in the Pod.

  2. Application write fails

    The process UID/group lacks write permission, or the mounted path is intentionally read-only.

  3. Fix the precise access requirement

    Inspect ownership, mount flags and the security context. Recreating the PVC risks data without addressing the demonstrated access mismatch.

About this model

A CSI driver with attachment is illustrated; some drivers skip that checkpoint.

Practice: storage has its own checkpoint ladder

kubectl get pod writer -o wide
kubectl describe pod writer
kubectl get pvc
kubectl describe pvc app-data
kubectl get pv
kubectl get storageclass
kubectl get volumeattachments

The last command may require broader permissions and is relevant only to drivers using attachment. First decide whether the claim is Pending or Bound. For Pending, inspect class selection, provisioning events and delayed binding. WaitForFirstConsumer can be expected before a consuming workload is placed. For Bound claims with attachment errors, examine node topology, previous attachment ownership and supported access mode. Do not force-detach a disk until the old writer is known to be stopped or fenced.

For FailedMount, read the exact message and node-side driver evidence: invalid mount options, inaccessible backend, missing credentials or filesystem setup are different causes. If the container starts but gets permission denied when accessing /data, mounting may already be healthy. Inspect process UID and groups, mount read-only settings, filesystem ownership and application path. A security context change should address a demonstrated permission requirement; privileged mode is too broad as a default diagnostic fix.

Use the diagnostics lab to record the first successful checkpoint and the first failing checkpoint. Make one targeted correction, repeat the same probe, and also repeat a nearby negative case so that a broad accidental permission change is not mistaken for a precise repair. Keep persistent claims intact until the exercise’s retention policy is understood.

Pitfalls: repair the failed boundary rather than its neighbors

Changing a Service selector cannot repair a disk mount failure. Recreating a PVC cannot repair an HTTP route’s hostname mismatch. Such changes can introduce new faults and destroy the comparison that made the investigation useful. Conversely, multiple simultaneous failures are possible: fixing DNS may reveal a policy denial that was already present. Treat the newly exposed symptom as the next boundary rather than claiming the previous diagnosis was necessarily wrong.

Check your understanding

The Service name resolves and direct Pod-IP traffic works from the same client, but the ClusterIP request fails. Where should you focus next?

Focus on the Service’s port mapping, usable endpoint state and the node’s Service forwarding implementation. DNS and the tested direct endpoint path have passed useful checks. Continue with evidence about that virtual-address path rather than changing the backend image or DNS configuration without a new observation.

Next, preserve this evidence across Pod replacements using metrics, centralized logs and distributed traces, each with a defined question and retention boundary.

Primary documentation

Go to the source

Reviewed 2026-10-27 · Kubernetes 1.37 · Related local exercise

Search the book

Search loads when you open it.