Scheduling constraints
Combine node selection, Pod affinity, and taints without turning preferences into guarantees.
What you will understand
- Explain filtering and scoring before binding
- Choose between node affinity and inter-Pod affinity
- Predict why a toleration permits but does not attract a Pod
Builds on: Requests, limits, and QoS
Mental model: first eligibility, then preference
The scheduler chooses a node for an unscheduled Pod by filtering unsuitable nodes and scoring the feasible remainder. Resource requests are one filter. Node labels, required affinity, taints, storage topology, and other constraints also contribute. A preferred rule changes scoring but does not make an otherwise invalid node eligible. When every node fails at least one hard rule, the correct result is a Pending Pod with scheduling evidence, not an arbitrary placement that ignores the manifest.
A nodeSelector requires matching labels on a node. Node affinity expresses richer conditions: required terms filter nodes, while preferred terms influence scoring. In requiredDuringSchedulingIgnoredDuringExecution, “ignored during execution” means changing a matching node label later does not by itself evict an already running Pod. Multiple expressions inside one selector term are combined with AND; alternative node selector terms are combined with OR. This small logical distinction can double the intended set of eligible nodes accidentally.
Build the feasible node set
Show node labels, required affinity and preferred inter-Pod locality contributing different decisions.
The required node selector admits Worker A and rejects Worker B, regardless of current CPU usage.
- Client Pod to Worker Alabel matches
- Client Pod blocked toward Worker Blabel mismatch
Inspect this state in YAML / kubectl
nodeSelector:
workload: toolsRead the complete visual sequence
- Read node labels
The required node selector admits Worker A and rejects Worker B, regardless of current CPU usage.
- Resolve Pod affinity
The cache Pod is selected by its Pod label. Its node zone identifies the locality domain for the preference.
- Score feasible nodes
Preferred Pod affinity only scores nodes that survived the hard filters; it cannot rescue Worker B.
- Bind without continual relocation
Binding assigns the client to Worker A. IgnoredDuringExecution affinity does not continually reschedule it after label changes.
Required anti-affinity cannot fit
Two replicas require different hostnames but only one worker is eligible.
- First replica occupies Worker A
An existing replica occupies the only eligible hostname domain; the next replica must avoid that hostname.
- Second replica waits
Required anti-affinity rejects the occupied domain; the node-label mismatch rejects the other worker.
- Choose a conscious trade-off
Add an eligible domain or make separation a preference if co-location is acceptable. More retries cannot satisfy an impossible constraint.
About this model
Two labeled worker nodes and a cache Pod; real scoring combines multiple plugins.
Walkthrough: dedicate capacity with two independent controls
Taints repel Pods that lack matching tolerations. They do not identify the node a Pod should seek. A toleration lets a Pod pass a taint check; node affinity can require the intended pool. The following Pod combines both, assuming a lab worker is deliberately labeled workload=tools and tainted dedicated=tools:NoSchedule.
apiVersion: v1
kind: Pod
metadata:
name: tools-client
labels:
app: tools-client
spec:
nodeSelector:
workload: tools # Requires this label on the chosen node.
tolerations:
- key: dedicated
operator: Equal
value: tools
effect: NoSchedule # Allows this taint; does not select the node.
containers:
- name: client
image: busybox:1.36
command: ["sleep", "3600"]
resources:
requests:
cpu: 25m
memory: 16Mi
NoSchedule prevents new placement without a toleration but does not evict already assigned Pods. PreferNoSchedule is a preference to avoid the node. NoExecute can also cause existing Pods without a matching toleration to be evicted; a matching toleration with tolerationSeconds can give a bounded stay. These effects are operationally different. Never experiment with cluster-wide worker taints on a shared cluster simply to see which Pods disappear.
Inter-Pod affinity uses labels on other Pods and a topology key on nodes. A cache client might prefer to share a zone with cache Pods. Pod anti-affinity can avoid placing replicas in the same hostname domain. Required anti-affinity improves separation but can make replicas unschedulable when there are fewer eligible domains than replicas. Preferred anti-affinity can reduce that availability trade-off, at the cost of accepting some co-location. Topology labels must be consistently present; a missing label is not a new physical failure domain.
Permission to enter is not an attraction
Follow a dedicated pool taint and distinguish NoSchedule from NoExecute.
The dedicated NoSchedule taint rejects a new ordinary Pod without the matching toleration.
- Ordinary Pod blocked toward Tools nodeuntolerated
Read the complete visual sequence
- Repel ordinary placement
The dedicated NoSchedule taint rejects a new ordinary Pod without the matching toleration.
- Permit tools placement
A matching toleration lets the tools Pod pass this check. It says nothing about which feasible node wins.
- Select the intended pool
Adding workload=tools as a required selector restricts the tools Pod to the labeled pool.
- Leave existing Pods alone
NoSchedule does not remove the already running Pod. It restricts future scheduler placements on this node.
Change to NoExecute
A new NoExecute taint also affects existing Pods.
- Taint effect changes
NoExecute is evaluated for already assigned Pods as well as new candidates.
- Untolerated Pod evicted
The running Pod without a matching NoExecute toleration becomes subject to taint-based eviction.
- Bound the stay deliberately
A matching NoExecute toleration may include tolerationSeconds. Expiration ends the temporary permission to remain.
About this model
A single training taint; other scheduler filters are assumed satisfied.
Practice: prove every filter before adding another
# Substitute a disposable lab worker name obtained from the first command.
kubectl get nodes --show-labels
kubectl label node LAB_WORKER workload=tools
kubectl taint node LAB_WORKER dedicated=tools:NoSchedule
kubectl apply -f tools-client.yaml
kubectl get pod tools-client -o wide
kubectl describe pod tools-client
# Undo only the training changes after deleting the Pod.
kubectl delete pod tools-client
kubectl taint node LAB_WORKER dedicated=tools:NoSchedule-
kubectl label node LAB_WORKER workload-
Run this exercise only in the module’s disposable cluster. If there is no eligible worker, stay with the visual scenario instead of changing a control-plane node’s scheduling policy. Expect the Pod to bind only if the labeled node also passes resource and other scheduler checks. Remove its toleration in a separately named copy and predict the scheduling event before applying it. The tainted node should be rejected despite its matching label.
Inspect FailedScheduling messages as a list of reasons across nodes. A message mentioning both insufficient memory and untolerated taints does not necessarily mean the same node failed both checks. Inspect each node’s labels, taints, allocatable resources and existing requests to build the actual feasible set. Avoid setting spec.nodeName as a shortcut: it bypasses normal scheduler selection and hides the mechanism you are learning.
Pitfalls: hard constraints compose into an empty set
Copying required anti-affinity from a three-node environment into a one-node local cluster often blocks the second replica. Similarly, a toleration broad enough to tolerate every taint can allow access to pools that were intended to remain dedicated. Use narrow keys, values and effects, and apply an admission policy when pool access must be a security or governance rule. Placement configuration alone should not be treated as an authorization boundary against users who can freely change it.
Check your understanding
A Pod tolerates the GPU node’s taint but has no selector. Must the scheduler put it on a GPU node?
No. The toleration only removes one reason to reject GPU nodes. Other untainted nodes may also be feasible and may receive a higher score. Use a resource request for the extended GPU resource and appropriate placement rules when the workload needs that hardware. A matching toleration alone neither reserves nor requests a device.
Use the module lab to remove one constraint at a time. Next, spread multiple replicas across topology domains while reserving enough namespace budget for their rollout.
Primary documentation
Go to the source
Reviewed 2026-10-27 · Kubernetes 1.37 · Related local exercise