Informers, caches, and workqueues
Build convergence from imperfect observations and repeatable work.
What you will understand
- Trace a shared informer from list and watch to cache handlers.
- Explain key deduplication and rate-limited reconciliation.
- Design retries that recover from partial success.
Builds on: Watches and server-side apply
Mental model: events schedule a question, not a command
An informer combines list/watch observation with a local object cache and event handlers. A shared informer lets multiple consumers share that observation machinery. Handlers usually enqueue a resource key such as kv-lab/m16, rather than perform long network calls immediately. A worker later asks: given the desired object and observed dependents now, what must change? This separates API event delivery from expensive reconciliation and keeps the system responsive during bursts.
The cache is eventually consistent with the API server. A worker may successfully create a child and then briefly read a cached parent or child that predates that write. Reconciliation must tolerate that window. It should also tolerate an object disappearing between enqueue and processing. A missing parent often means no creation work is needed; it is not automatically a retryable outage. Do not derive correctness from receiving exactly one event in exactly the sequence you expected.
Observation feeds a cache and a queue
A worker reads cached state and compares desired state with dependents; it may need another pass after cache lag.
The informer lists and watches resources, storing observed objects in its local cache.
- API list/watch to Informer cacheupdate objects
Read the complete visual sequence
- Build a baseline
The informer lists and watches resources, storing observed objects in its local cache.
- Keep handlers short
A handler enqueues the object key instead of performing a slow external request in the watch path.
- Reconcile the latest observation
A worker reads cached state and compares desired state with dependents; it may need another pass after cache lag.
Read after write is stale
A successful API write may not yet be reflected in the local cache.
- Write completed
The worker created a dependent through the API, but its informer has not observed that change yet.
- Old observation returns
The cache still reports no child. A blind second create can meet AlreadyExists.
- Recover by comparison
Treat the conflict as new information, requeue appropriately, and reuse the stable child identity.
About this model
A single informer is shown; production controllers often depend on several caches and must synchronize them.
Walkthrough: one key may represent many changes
A workqueue can deduplicate pending keys and coordinate processing of a key so workers do not repeatedly execute the same pending item concurrently. If another change arrives while a key is being processed, the queue can arrange another pass after the current one finishes. This reduces unnecessary work but means an event stream is not a transaction log. Reconcile the current object’s desired replica count, rather than attempting to replay every historical increase and decrease.
Rate-limited queues let failures re-enter with delay. On success, a typical client-go worker forgets accumulated failure history; it always marks the item done after processing. On transient errors it adds the key with rate limiting. On permanent invalid input it records a useful condition and waits for a relevant change rather than retrying rapidly forever. Per-item backoff and overall throughput limits reduce pressure, but neither grants capacity to an overloaded downstream API.
Dedupe, delay, and retry safely
Success resets failure history; transient failure requeues with backoff instead of a tight loop.
Multiple pending notifications for one key can produce one current-state reconciliation.
- Object changes to Rate-limited queueadd key
Read the complete visual sequence
- Collapse redundant work
Multiple pending notifications for one key can produce one current-state reconciliation.
- Attempt bounded work
The worker reads current intent and issues repeatable operations with stable identities.
- Forget or rate-limit
Success resets failure history; transient failure requeues with backoff instead of a tight loop.
Timeout after partial success
The next pass must discover work already committed by the dependency.
- Remote create committed
The dependency created the resource but its response was lost.
- Retry is delayed
Backoff protects the dependency while the key remains eligible for another attempt.
- Discover before replacing
A stable identity lets reconciliation find the existing resource and complete status without duplication.
About this model
The scene illustrates standard workqueue behavior without prescribing one library version or retry duration.
The following configuration is used by the local mechanism exercise. The ConfigMap illustrates stable desired input, while the Lease is inspected separately in the leadership lesson. Reapplying a declaration is not a production informer implementation; it demonstrates why repeating an intention should preserve object identity.
apiVersion: v1
kind: ConfigMap
metadata:
name: m15-repeat
namespace: kv-lab
labels:
lab: m15
data:
desired: stable
kubectl --context kind-kv-lab -n kv-lab apply -f repeat.yaml
kubectl --context kind-kv-lab -n kv-lab get configmap m15-repeat -o jsonpath='{.metadata.uid}{"\n"}'
# Apply again; the object UID should remain the same.
kubectl --context kind-kv-lab -n kv-lab apply -f repeat.yaml
kubectl --context kind-kv-lab -n kv-lab get configmap m15-repeat -o jsonpath='{.metadata.uid}{"\n"}'
kubectl --context kind-kv-lab -n kv-lab get configmap m15-repeat --watch --request-timeout=5s
Practice: reason about the crash boundary
Run the m15 local exercise, then connect it to the polling operator in m16. Draw a reconcile attempt that creates a ConfigMap, creates a Deployment, and updates status. Place a process crash after each write. The next attempt must discover and reuse an already-created object, finish missing work, and avoid inventing a new child name on every attempt. This is idempotence across partial progress, not merely making a function return the same string twice.
Inspect the example operator’s deliberately limited polling loop. Polling is useful for exposing the state comparison, but it does not provide shared informer efficiency, queue fairness or rate-limited retry machinery. A production conversion would use maintained client libraries, wait for required caches to synchronize, handle deleted objects, and measure queue depth, queue wait time, reconcile duration and error categories. Choose observations that distinguish a slow worker from an unhealthy API dependency.
Pitfalls: retries are a load generator
Do not retry every exception with no delay. A rejected schema, missing authorization, exhausted quota and temporary timeout require different remedies. If an external request timed out after the external system committed it, retrying a blind create can duplicate resources. Use a stable external identity, an idempotency key where available, and discovery before replacement. Cap concurrency so a short API outage does not produce an uncontrolled recovery surge. Logging every retry at maximum detail can itself consume storage and hide the first useful failure.
Check your understanding
Ten updates enqueue the same object before a worker runs. Must reconciliation execute ten times?
No. A deduplicating key queue may collapse pending work. The worker should read the latest desired state and converge toward it. If every individual transition matters to a business workflow, that workflow needs a durable event or transaction design beyond a controller workqueue.
Next, trace a specific controller that consumes unscheduled Pods. Its queue feeds a scheduling framework with feasibility checks, scoring and a separate binding outcome.
Go to the source
Reviewed 2026-10-27 · Kubernetes 1.37 · Related local exercise