Capstone: recover a stateless app
Find the broken selector through evidence and restore the request path.
What you will understand
- Trace DNS, Service selection, EndpointSlices and application reachability.
- Repair the smallest proven cause without restarting healthy replicas.
- Verify recovery with both control-plane state and a real client request.
Builds on: Protecting Secrets and tenant boundaries, Capacity and scale trade-offs, Service meshes and platform boundaries
Mental model: a healthy Pod is one link in the request path
This capstone deploys two nginx replicas, a Service and a BusyBox client. The application Pods are healthy, but the Service intentionally selects the wrong label. DNS can resolve the Service name and the Service can have a stable virtual IP while no matching backends are available. The failure is therefore neither proof that DNS is broken nor proof that nginx needs restarting. The task is to identify the first broken relationship using evidence.
The path to examine is client name resolution, Service configuration, selector-to-Pod matching, EndpointSlice contents, target port and application response. Different networking implementations realize Service forwarding differently, so avoid assuming every node always uses a particular iptables chain. EndpointSlices and the declared Service/Pod relationship are the portable control-plane evidence. A request from the actual client confirms whether the assembled path works in this environment.
Ready Pods can sit outside a Service selector
The client can address the Service but cannot reach selected application backends through this relationship.
Two nginx Pods become Ready with app=m20-web labels.
Read the complete visual sequence
- Application is healthy
Two nginx Pods become Ready with app=m20-web labels.
- Service selects another label
The Service selector is app=broken-selector, so those Ready Pods do not become its usable backends.
- Client path fails
The client can address the Service but cannot reach selected application backends through this relationship.
Restart the healthy Pods
An unrelated restart does not correct the selection contract.
- New Pods appear
A restart replaces Pods but their Deployment template still uses app=m20-web.
- Wrong selector remains
The Service still requires app=broken-selector.
- Failure repeats
No usable matching backends appear; repair the proven selector mismatch instead.
About this model
Exact client error and dataplane forwarding differ by networking implementation; the label mismatch is implementation-independent.
Walkthrough: narrow the investigation before changing state
Read stateless.yaml from the m20 download. The Deployment template labels Pods app=m20-web, while the Service selector initially uses app=broken-selector. The EndpointSlice controller therefore has no matching ready application backends for that Service. A request can fail with a timeout or another network error depending on implementation; the specific client error is less useful than the selector and backend evidence. Readiness and endpoint publication still matter after the selector is corrected.
The script first waits for the Deployment rollout and the client Pod, then prints Pod, Service and EndpointSlice observations. It confirms the request does not succeed, patches only the Service selector, and retries the client request for a bounded period. This separation captures before-and-after behavior and allows normal controller and networking propagation. The loop’s budget is a local test bound, not a measured recovery objective or guarantee for other clusters.
Repair one relationship, then verify a real request
A successful HTTP response proves this client-to-Service-to-application path now works.
The repair changes only the Service selector to match the Deployment Pod labels.
- Selector patch to Service m20-webminimal proven fix
Read the complete visual sequence
- Correct the selector
The repair changes only the Service selector to match the Deployment Pod labels.
- Wait for propagation
The EndpointSlice and networking paths observe matching Ready Pods; the script retries within a local bound.
- Verify from the client
A successful HTTP response proves this client-to-Service-to-application path now works.
Endpoints exist, wrong port
A different failure needs new evidence after selection is correct.
- Selection is correct
The Service now discovers the intended Ready Pods.
- Port contract is wrong
A hypothetical incorrect targetPort sends traffic to a port where the application is not listening.
- Inspect the next boundary
Endpoint presence alone is insufficient; compare targetPort and the actual listener before another repair.
About this model
The actual capstone uses matching port 80; the alternate branch is a separate diagnostic hypothesis.
This is the intentionally broken Service from the exercise. Save and inspect the complete provided files rather than applying only this object without its Pods and client.
apiVersion: v1
kind: Service
metadata:
name: m20-web
namespace: kv-lab
labels:
lab: m20
spec:
selector:
app: broken-selector # Deployment Pods actually use app=m20-web.
ports:
- port: 80
targetPort: 80
# Download all m20 files first, then run only this investigation.
bash capstone-stateless.sh
# Read-only follow-up evidence after the script's repair.
kubectl --context kind-kv-lab -n kv-lab get pods -l app=m20-web --show-labels
kubectl --context kind-kv-lab -n kv-lab get service m20-web -o yaml
kubectl --context kind-kv-lab -n kv-lab get endpointslices -l kubernetes.io/service-name=m20-web -o yaml
kubectl --context kind-kv-lab -n kv-lab exec m20-client -- wget -qO- -T 3 http://m20-web
Practice: produce a short recovery record
Run the m20 local exercise using capstone-stateless.sh if you want this investigation independently; run.sh runs all three capstones. Record the initial Pod readiness, the incorrect selector, absence of usable selected backends, the failed request, the corrected selector and the successful request. These facts support a precise conclusion: this failure came from a Service-to-Pod selection mismatch and was repaired by aligning that relationship.
For a second investigation, discuss a different failure without mixing it into the primary result. A wrong targetPort can leave endpoints present while traffic fails. A readiness failure can remove otherwise matching Pods from usable backends. A NetworkPolicy can block an otherwise correct Service path if the networking implementation enforces it. Use a new hypothesis and evidence set for each change; applying several repairs at once makes the cause ambiguous.
Pitfalls: avoid fixing the symptom by changing unrelated layers
Restarting healthy Pods may accidentally change timing but leaves the wrong selector wrong. Deleting the Service can change its assigned identity and disrupt clients unnecessarily. Replacing an image does not repair a label mismatch. A direct Pod-IP request can help isolate the application, but success on that path does not prove Service DNS or forwarding works. Preserve enough evidence to explain why the selected repair follows from the observed failure.
The lab uses a disposable cluster and does not validate external ingress, cloud load balancers, production DNS, TLS termination or multi-zone failover. Do not generalize a successful BusyBox request into a platform availability claim. If you extend the exercise, add one boundary at a time and define what its health observation proves.
Check your understanding
DNS resolves and both nginx Pods are Ready, but the Service has no usable endpoints. What relationship should you inspect first?
Compare the Service selector with actual Pod labels and readiness evidence. DNS resolution identifies the Service, while Pod readiness describes the Pods; neither proves the Service selected those Pods. The intentional mismatch in this exercise is visible directly in those fields.
Next, recover data rather than connectivity. The stateful capstone requires a backup outside the cluster and a deliberate delete-and-restore sequence before Pod replacement can prove persistence.
Go to the source
Reviewed 2026-10-27 · Kubernetes 1.37 · Related local exercise