Watches and server-side apply
Follow a resumable watch cursor and negotiate field ownership explicitly.
What you will understand
- Resume watches without treating resourceVersion as a clock.
- Explain stale caches and relisting after expired history.
- Resolve server-side apply conflicts through an ownership decision.
Builds on: Request lifecycle and admission
Mental model: observation and ownership solve different problems
A watch delivers changes to a selected API collection. A client typically lists a baseline, then watches from its resourceVersion, preserving the string unchanged. Current kube-apiserver resources, including CRDs, support ordering versions within the same API group and resource type using arbitrary-size decimal comparison; conformant implementations guarantee this from Kubernetes 1.35. Extension APIs require checking their contract and numeric format. This permits constrained ordering, not subtraction to count changes, cross-resource ordering, or elapsed-time calculations.
A disconnected client has not necessarily lost its ability to recover. It can reconnect from its last accepted version while history remains available. If the requested history is too old, the server can return an expired-history response such as HTTP 410 Gone. The client must rebuild its baseline and continue watching. A watch is therefore a resumable observation protocol, not a permanent audit log or a guarantee that every intermediate state reaches every controller.
List, resume, and rebuild a watch
A worker reads the cache and compares current observations with desired state. Cache visibility can lag an API write.
A list supplies collection state and a resourceVersion cursor. Preserve the string unchanged and follow the documented same-resource ordering contract.
- API server to List baselinelist response
- List baseline to Watch streamresume cursor
Read the complete visual sequence
- Establish the baseline
A list supplies collection state and a resourceVersion cursor. Preserve the string unchanged and follow the documented same-resource ordering contract.
- Observe changes
The client watches from the cursor and updates local state as events arrive; this is not a durable audit log.
- Reconcile cached state
A worker reads the cache and compares current observations with desired state. Cache visibility can lag an API write.
History expired
A disconnected client cannot assume old watch history is retained forever.
- Reconnect from old cursor
The client resumes with its last accepted token after a long interruption.
- 410 Gone
The server cannot provide that retained history; the client discards its incomplete baseline.
- Relist and resume
A new list rebuilds current state and supplies a fresh cursor before the watch resumes.
About this model
A traditional list-then-watch client is shown; optional streaming-list optimizations are omitted. Documented same-resource version ordering does not make the cursor a clock or event count.
Walkthrough: watch current state, negotiate field changes
An informer uses watch information to maintain a local cache, but cache reads can lag a successful write. A reconcile loop must handle an object that is already changed on the API server while its cached copy still looks old. Kubernetes controllers work toward current desired state rather than executing an irreversible business transaction once for every watch notification. For durable audit requirements, configure an audit pipeline with its own retention and access model instead of retaining a controller’s event stream.
Server-side apply solves a separate coordination problem: multiple actors declare intentions for fields. The API server records field managers in managedFields. When an apply changes a field owned by another manager to a different value, it normally reports a conflict. Managers may share ownership when applying the same value; an ordinary update or patch can also change field ownership according to its operation. Ownership is field-specific and schema-dependent, including list merge behavior; it is not an exclusive lock on the entire object.
Two managers and one declared field
A second manager can declare a separate field, or share ownership by applying the same value.
kv-gitops applies data.release and becomes an owner of that declared field.
- kv-gitops to Server-side applyapply intention
- Server-side apply to ConfigMapaccepted fields
Read the complete visual sequence
- First declaration
kv-gitops applies data.release and becomes an owner of that declared field.
- Record the contract
The stored value and managedFields describe the result; unrelated fields can belong to other managers.
- Agree or separate fields
A second manager can declare a separate field, or share ownership by applying the same value.
Conflicting declaration
A second manager changes an owned field to a different value.
- Another intention
kv-human applies data.release with a value different from the git-managed value.
- Conflict is returned
The API rejects that apply field change; the conflict names the competing field manager.
- Make an ownership decision
The original value remains until actors agree, relinquish the field, or an authorized takeover is explicit.
About this model
One scalar field illustrates ownership; map and list ownership depend on the API schema.
Save this as ownership.yaml in a disposable directory. Manager kv-gitops owns the declared data field after applying it. A second apply using another manager and a different value should conflict.
apiVersion: v1
kind: ConfigMap
metadata:
name: m14-ownership
namespace: kv-lab
labels:
lab: m14
# Only this key is part of this manager's intention.
data:
release: git-declared
kubectl --context kind-kv-lab -n kv-lab apply --server-side --field-manager=kv-gitops -f ownership.yaml
kubectl --context kind-kv-lab -n kv-lab get configmap m14-ownership -o yaml --show-managed-fields
# Keep a second terminal watching while you edit and reapply the file.
kubectl --context kind-kv-lab -n kv-lab get configmap m14-ownership --watch
# After changing data.release locally, expect a field conflict here.
kubectl --context kind-kv-lab -n kv-lab apply --server-side --field-manager=kv-human -f ownership.yaml
Practice: capture a conflict instead of suppressing it
Run the m14 local exercise. It creates a ConfigMap with one field manager and attempts a conflicting declaration with another. Observe the exact field path in the conflict and inspect managedFields before choosing a resolution. For this experiment the intended result is the conflict itself; do not add force simply to make every command green. Watch delivery may group closely timed changes differently from your terminal expectations, so inspect final state as well as the event stream.
There are three useful resolutions to discuss. The second actor can stop declaring the field, both actors can agree on the same value, or an authorized owner can deliberately take ownership using force conflicts. That last option changes the ownership contract. Record who will manage the field on the next reconciliation, otherwise two automated systems can repeatedly undo each other’s changes.
Pitfalls: persistence is not cache synchronization
A successful apply is evidence about the API write, not proof that every cache has observed it. Check the resource’s documented status conditions and, where reported, whether observedGeneration covers the desired generation; metadata.generation alone records desired-state changes, not convergence. A watch bookmark advances progress information without representing a normal object change, and clients must interpret event types correctly. Use maintained client libraries for list/watch recovery. Do not invent a cursor or use resourceVersion: "0" as a universal request for the newest possible read; resourceVersion semantics depend on the operation and associated options.
Check your understanding
A watch cursor changes from one numeric-looking string to another. How many updates occurred?
The strings do not count updates. Documented same-resource ordering does not make resourceVersion a per-object event counter or a clock. Reconcile observed state; use an audit or application log when individual changes matter.
Next, connect field ownership to deletion ownership. managedFields controls declarations of fields; ownerReferences and finalizers control different lifecycle relationships.
Go to the source
Reviewed 2026-10-27 · Kubernetes 1.37 · Related local exercise