Topology and resource governance
Spread replicas across failure domains and make namespace budgets visible before admission.
What you will understand
- Calculate topology skew across eligible domains
- Explain ResourceQuota admission versus scheduling capacity
- Use LimitRange defaults without hiding the admitted requests
Builds on: Scheduling constraints
Mental model: distribute replicas, then budget their promises
A Deployment’s replica count says how many copies should exist; it does not by itself say which machines or zones should host them. Topology spread constraints compare the number of matching Pods across eligible topology domains. The topologyKey names a node label such as kubernetes.io/hostname or topology.kubernetes.io/zone. The labelSelector names the Pods in the incoming Pod’s namespace whose distribution should be counted. These are different label sets on different object kinds.
The skew measures an imbalance relative to the eligible domain counts under the constraint’s rules. With two eligible zones containing two and one matching Pods, placing the next Pod in the second zone yields two and two. maxSkew: 1 with DoNotSchedule makes excessive imbalance a hard scheduling constraint. ScheduleAnyway keeps distribution as a scoring preference, subject to other scheduling constraints. Domain counting follows topology labels and the nodeAffinityPolicy and nodeTaintsPolicy inclusion rules; by default affinity is honored and taints are ignored for that count. Resource fit is a separate check: a domain can count even when none of its nodes has room for the new Pod. Do not infer placement availability from the number of zone names in a diagram alone.
Count domains before choosing the next placement
Compare two zone counts as a fourth replica is scheduled and explain a hard-skew failure.
The selector counts api Pods, not every Pod in the namespace. The current eligible zone counts are east=2 and west=1.
- Zone east to Schedulercount 2
- Zone west to Schedulercount 1
Read the complete visual sequence
- Count matching replicas
The selector counts api Pods, not every Pod in the namespace. The current eligible zone counts are east=2 and west=1.
- Test the crowded zone
Putting the new replica in east creates counts 3 and 1. Skew 2 exceeds the configured maximum 1.
- Test the less crowded zone
Putting it in west creates counts 2 and 2, so this topology constraint allows the placement.
- Bind after other filters
The scheduler can choose a fitting west node after all other resource and placement rules pass. Existing Pods do not move merely because this rule exists.
West has no fitting capacity
West remains an eligible domain for counting but its nodes cannot fit this Pod.
- East would violate skew
Placing in the already crowded east zone violates the hard spread constraint.
- West fails resources
The preferred balancing domain exists but has insufficient resources for this Pod.
- Choose capacity or a softer rule
Add fitting capacity in west or deliberately choose ScheduleAnyway when temporary imbalance is acceptable.
About this model
Two eligible zones with consistent labels; examples omit advanced minDomains and inclusion-policy settings.
Walkthrough: a spread rule belongs to the Pod template
This fragment belongs under a Deployment’s spec.template.spec. The Deployment selector and template labels must also include app: api. Multiple constraints are combined, so a hostname constraint and zone constraint must both be satisfiable when both are hard rules.
# Fragment: spec.template.spec of a Deployment labeled app=api
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: api
containers:
- name: api
image: nginx:1.27-alpine
resources:
requests: {cpu: 100m, memory: 64Mi}
limits: {cpu: 500m, memory: 128Mi}
Use actual zone labels only on nodes representing those domains. Labeling three local containers as three “zones” is useful for scheduler experiments, but it creates no protection against losing the laptop. Spread also does not continuously rebalance existing Pods after a node is added. The scheduler places newly unscheduled Pods; controllers or separate rebalancing tooling may cause new placement opportunities later.
A ResourceQuota is a namespace admission budget. It can cap aggregate CPU requests, memory limits, storage requests and object counts. Admission rejects a new object that would exceed the relevant budget. This is different from Pending: a rejected object may never be persisted, while a Pending Pod has already entered the API and is waiting for placement or startup. Quota does not reserve physical node capacity for a namespace. Several namespaces can be below their quotas while competing for the same exhausted nodes.
A LimitRange can set default requests and limits and require minimums, maximums or request-to-limit ratios. Its defaults apply when objects are admitted, not as a live rewrite of every existing Pod. Defaults make behavior safer only when they are understood. A surprising default CPU request can exhaust quota or scheduler capacity even if the submitted manifest omitted the field.
Admission budget precedes placement
Follow LimitRange mutation into ResourceQuota accounting and then scheduler placement.
The submitted Pod has no CPU request, but admission has not yet completed.
- Submitted Pod to API admissioncreate request
Read the complete visual sequence
- Submit incomplete resources
The submitted Pod has no CPU request, but admission has not yet completed.
- Apply defaults
LimitRange supplies a 100m CPU request, changing the amount quota and scheduling must account for.
- Reject an over-budget create
950m plus 100m exceeds the 1 CPU namespace quota. Admission rejects the Pod rather than leaving that Pod Pending.
- Free budget then place
After accounted requests fall sufficiently, a new create can pass quota. The scheduler still independently checks node capacity.
Surge exceeds a fitting final state
A rollout needs temporary additional Pods that exceed quota.
- Final state would fit
Eight replicas at 100m request total 800m, below the 1 CPU budget.
- Overlap exceeds quota
Three surge replicas would raise requests to 1100m. The ReplicaSet receives creation failures for the excess.
- Inspect the controller events
Describe the ReplicaSet and adjust surge, budget or resource settings with the intended availability trade-off.
About this model
One namespace and one CPU request budget; real quotas can track many resources.
Practice: calculate a rollout before admitting it
apiVersion: v1
kind: ResourceQuota
metadata:
name: team-budget
spec:
hard:
requests.cpu: "1"
requests.memory: 1Gi
pods: "10"
---
apiVersion: v1
kind: LimitRange
metadata:
name: container-defaults
spec:
limits:
- type: Container
defaultRequest: {cpu: 100m, memory: 64Mi}
default: {cpu: 500m, memory: 256Mi}
Apply these only inside the training namespace. Inspect kubectl describe resourcequota team-budget and kubectl get limitrange container-defaults -o yaml. Then create a small Pod with no resource fields and read the admitted Pod back. Expect defaulted values, subject to any other admission rules installed on the cluster. Describe the owning ReplicaSet if a Deployment cannot create its next Pod; quota rejection events often appear on the controller that attempted creation.
Before scaling, multiply replica count by per-Pod requests and include rolling-update surge. Eight 100m Pods plus three overlapping surge Pods request 1.1 CPUs, exceeding this quota even though the final eight-Pod state would fit. Compute the budget for termination overlap and other workloads too. These quota and spread exercises are extensions to the module’s local practice; its baseline demonstrates selector-based placement without installing these governance objects.
Pitfalls: governance is not physical isolation
Quotas control API consumption, not per-tenant network access or dedicated hardware. Namespace boundaries require additional RBAC, admission and network controls when tenants are untrusted. Likewise, raising quota cannot make a too-small node fit a large Pod, and loosening topology spread cannot bypass a missing volume zone. Diagnose admission, scheduling and runtime separately, then change the policy responsible for the actual constraint.
Check your understanding
A namespace has unused quota, but a new Pod requests more memory than any eligible node provides. Does admission guarantee it will run?
No. Quota admission can succeed while every node fails the scheduler’s fit check. The Pod should exist in Pending state with scheduling events. Quota answers whether this namespace may declare more demand; scheduling answers whether an actual eligible node can currently satisfy that demand.
Next, trace a packet between two placed Pods. Correct placement provides capacity and failure-domain choices; connectivity needs its own implementation and policies.
Primary documentation
Go to the source
Reviewed 2026-10-27 · Kubernetes 1.37 · Related local exercise