Appearance

Backup, restore, and disaster recovery

Recover API state without confusing it with application data.

What you will understand

  • Identify assets missing from an etcd snapshot.
  • Explain offline restore, new membership and revision handling.
  • Verify recovery from an isolated target before declaring success.

Builds on: Termination, disruption, and drain, PKI, kubeadm, and safe upgrades

Mental model: back up the state you intend to recover

An etcd snapshot captures Kubernetes API data stored in etcd, including workload specifications, resource relationships and Secrets. It does not capture the bytes inside PersistentVolumes, external databases, object stores, container images or every file under a control-plane host. PKI, service-account signing keys, encryption-at-rest configuration and keys, and infrastructure configuration require their own protected recovery plan. A PVC object in the snapshot is a claim describing storage, not a copy of the application’s records.

Backup usefulness therefore depends on a recovery objective. Define the acceptable loss of recent changes and the acceptable time to restore useful service, then choose a schedule and verification process. A file that exists is evidence of capture, not proof that the environment can decrypt it, restore it with compatible tooling, rebuild member identities or recover application data consistently. Protect snapshots as sensitive credentials-bearing material, even if the learning environment contains only disposable Secrets.

API state and data-plane bytes need separate backups

Trust, signing and encryption material plus compatible infrastructure and application backups complete the plan.

Storage viewStep 1 of 3
Capture API dataAn etcd snapshot contains stored API objects and may contain sensitive Secret data.. Kubernetes API objects, Specs, identities and Secrets, idle. etcd snapshot, API database backup, active. PersistentVolume bytes, Application records, idle. PKI + encryption keys, Separate protected assets, idle. Recovery plan, Assemble compatible assets, idleKubernetes API objects: Specs, identities and Secrets
Kubernetes API objectsSpecs, identities and SecretsContext
etcd snapshot: API database backup
etcd snapshotAPI database backupIn focus
PersistentVolume bytes: Application records
PersistentVolume bytesApplication recordsContext
PKI + encryption keys: Separate protected assets
PKI + encryption keysSeparate protected assetsContext
Recovery plan: Assemble compatible assets
Recovery planAssemble compatible assetsContext
Kubernetes API objectsSpecs, identities and SecretsContext
etcd snapshotAPI database backupIn focus
PersistentVolume bytesApplication recordsContext
PKI + encryption keysSeparate protected assetsContext
Recovery planAssemble compatible assetsContext
Animation controls
Speed
0.5×
1×
2×
Explore a scenario
Normal behavior
Snapshot-only recovery
Capture API data

An etcd snapshot contains stored API objects and may contain sensitive Secret data.

Read the complete visual sequence
  1. Capture API data

    An etcd snapshot contains stored API objects and may contain sensitive Secret data.

  2. Identify missing bytes

    A PVC object is not its volume contents; application files and external stores need their own backup path.

  3. Assemble recovery assets

    Trust, signing and encryption material plus compatible infrastructure and application backups complete the plan.

Snapshot-only recovery

Recovering API objects can leave the application without its data or keys.

  1. API restore succeeds

    The Deployment, PVC and Secret objects can be listed in the target.

  2. Data path is absent

    The target has no recovered database volume or external-store backup.

  3. Service recovery is unproven

    API availability alone does not establish application data correctness or decryptability.

About this model

The exact asset list depends on cluster encryption, providers and application dependencies.

Walkthrough: restore is an isolated reconstruction

The m18 exercise captures a snapshot from the named kind control plane and copies it outside the cluster. Its optional offline rehearsal creates a separate kv-recovery cluster and refuses an existing target of that name. It stops the target API server and etcd static Pods, waits for their containers to exit, restores to a new data directory with the target’s new single-member identity, retains the previous directory, and restarts the components. Read the downloaded recovery guide and script before running it.

Restoring older state can move the apparent revision history backward for Kubernetes watchers. The documented etcd recovery approach includes revision bump and compaction marking so clients with newer observations are forced to rebuild their state rather than continue from incompatible history. This is an etcd recovery concern; Kubernetes resourceVersion is not a wall clock or a client-side restore control. Choose the bump using a reasoned bound and the tool version’s instructions, not an arbitrary number copied without understanding.

Restore isolated membership and invalidate stale watch history

The API returns backed-up objects; stale controller watches must rebuild rather than assume old history remains valid.

Lifecycle viewStep 1 of 3
Isolate and stop target writersThe rehearsal uses a separate cluster and stops its API server and etcd before replacing target state.. Verified snapshot, Older committed API state, idle. Stopped target etcd, No live writer, active. New data directory, New member identity + revision plan, idle. Recovered API, Restart against restored etcd, idle. Controller caches, Relist after invalidated history, idleVerified snapshot: Older committed API state
Verified snapshotOlder committed API stateContext
Stopped target etcd: No live writer
Stopped target etcdNo live writerIn focus
New data directory: New member identity + revision plan
New data directoryNew member identity + revision planContext
Recovered API: Restart against restored etcd
Recovered APIRestart against restored etcdContext
Controller caches: Relist after invalidated history
Controller cachesRelist after invalidated historyContext
Verified snapshotOlder committed API stateContext
Stopped target etcdNo live writerIn focus
New data directoryNew member identity + revision planContext
Recovered APIRestart against restored etcdContext
Controller cachesRelist after invalidated historyContext
Animation controls
Speed
0.5×
1×
2×
Explore a scenario
Normal behavior
Live writer or wrong target
Isolate and stop target writers

The rehearsal uses a separate cluster and stops its API server and etcd before replacing target state.

Read the complete visual sequence
  1. Isolate and stop target writers

    The rehearsal uses a separate cluster and stops its API server and etcd before replacing target state.

  2. Restore with explicit identity

    Compatible tools restore into a new directory with the planned membership, revision bump and compaction marking.

  3. Verify and rebuild clients

    The API returns backed-up objects; stale controller watches must rebuild rather than assume old history remains valid.

Live writer or wrong target

Restoring into an active source directory risks corruption and disruption.

  1. Source and target confused

    A command points at the original control-plane data directory instead of the isolated target.

  2. Concurrent writes persist

    A running etcd process invalidates the offline replacement assumptions.

  3. Stop before mutation

    Use the guarded rehearsal and explicit identities; do not improvise a live directory replacement.

About this model

The local script performs a single-member API-data rehearsal, not production membership recovery or worker migration.

This marker is a small API object you can inspect before and after the rehearsal. It is not an application backup artifact.

apiVersion: v1
kind: ConfigMap
metadata:
  name: m18
  namespace: kv-lab
  labels:
    lab: m18
data:
  marker: captured-before-disaster
# In the downloaded m18 directory: captures to a private host directory.
bash run.sh
backup_dir=$(python3 backup-path.py m18)
# Inspect RECOVERY.md and restore-offline.sh before this optional step.
bash restore-offline.sh
# Use the separate file produced by the recovery script, never your active context.
KUBECONFIG="$backup_dir/recovery.kubeconfig" kubectl -n kv-lab get configmap m18 -o yaml
# Discard only the isolated rehearsal target after inspection.
KUBECONFIG="$backup_dir/recovery.kubeconfig" kind delete cluster --name kv-recovery

Practice: verify the boundary of the rehearsal

Use the m18 local exercise only when Docker has capacity for the additional control plane. Expect a snapshot file outside the source cluster, then the API marker accessible through the recovery kubeconfig after the optional restore. The snapshot may also contain source Node objects. Their appearance in the restored API does not prove that a working worker fleet, networking or storage was migrated. This exercise demonstrates API-data recovery into an isolated target, not a supported kind cluster migration.

Write a recovery checklist that includes snapshot integrity, compatible etcd tools, encryption and trust assets, member topology, API readiness, controller startup and application verification. Keep source and target identities explicit in every command. For a stateful application, use its native consistent backup or a validated quiescing/snapshot protocol and test transaction-level correctness after restore. The m20 stateful capstone deliberately uses a quiescent single file so that boundary can be observed without claiming database consistency.

Pitfalls: restored intent can immediately act

When controllers return, they reconcile the restored desired state. An old custom resource may cause an operator to recreate infrastructure, change replicas or remove objects according to old policy. Isolate the recovery target from production dependencies until ownership and intent are understood. Retain evidence of the previous environment and avoid letting the old and restored control planes compete over the same external resources.

Do not restore over a live etcd data directory while its process is writing. Do not treat copying a volume directory as an application-consistent backup while the database is mutating it. A checksum can detect some file corruption but cannot prove transaction consistency or missing-key recovery. Restoration must be tested at the API and application levels appropriate to the recovery objective.

Check your understanding

The restored API contains the PVC and StatefulSet. Are database records restored?

Not from that evidence. etcd restored their API objects, while database bytes belong to the storage and application backup path. Verify the recovered volume contents and application consistency using the database’s supported recovery procedure.

Next, examine GitOps as another controller that may act after recovery. A source repository and a snapshot can describe different intended states, so startup ordering and ownership need an explicit plan.

Go to the source

Reviewed 2026-10-27 · Kubernetes 1.37 · Related local exercise

Search the book

Search loads when you open it.