Capstone: recover stateful data
Back up a quiescent file, delete it, restore it, and prove persistence.
What you will understand
- Separate Pod replacement from restoration of deleted records.
- Verify an external backup and restored contents.
- State the consistency boundary and limits of the file-based exercise.
Builds on: Termination, disruption, and drain, Backup, restore, and disaster recovery, Capstone: recover a stateless app
Mental model: persistence also preserves mistakes
A PersistentVolumeClaim lets a replacement Pod reconnect to storage beyond the old Pod’s lifetime. This preserves useful files, but it also preserves an accidental deletion. Deleting and recreating the consumer Pod does not recreate a file removed from its mounted filesystem. Recovery needs a retained copy from before the loss and a procedure that restores it. This distinction is the central failure boundary of the stateful capstone.
The exercise writes one small CSV record to a PVC, captures a backup in the private host directory selected by backup-path.py, deliberately deletes the file, restores it with kubectl cp, and checks exact contents. It then replaces the Pod and checks the restored record again. The backup exists outside the cluster so destruction of the disposable cluster would not automatically remove that host file. Protect and clean up the host copy intentionally as part of the exercise lifecycle.
Persistence preserves deletion as well as data
The script removes the file from the mounted PVC and verifies absence; a Pod restart cannot undo this deletion.
The single writer finishes the CSV record before a backup is captured.
- m20-data Pod to PVC m20write complete record
Read the complete visual sequence
- Write then quiesce
The single writer finishes the CSV record before a backup is captured.
- Keep an external copy
The script captures the bounded file in a private host directory and verifies its exact bytes before deleting the live file.
- Delete the actual record
The script removes the file from the mounted PVC and verifies absence; a Pod restart cannot undo this deletion.
No retained copy
Restarting a consumer cannot recover bytes no longer present in its volume.
- File is removed
The current persistent filesystem no longer contains orders.csv.
- Pod is recreated
The new Pod mounts the same claim and sees the same missing file.
- Recovery source is absent
Without a retained backup or application recovery history, persistence alone provides no earlier record.
About this model
The CSV is quiescent during copy; this does not model concurrent database transaction consistency.
Walkthrough: consistency is an explicit assumption
The single writer completes its write before backup, and no concurrent writer changes the file during capture. That quiescent-file assumption makes a byte copy meaningful for this example. A database with concurrent transactions, multiple files, write-ahead logs and caches needs its supported backup or quiescing procedure. A storage snapshot may be crash-consistent without being application-consistent; the required guarantees depend on the application and snapshot implementation. Do not describe this CSV copy as a database backup.
Read capstone-stateful.sh. It writes order-42,confirmed, verifies the captured backup bytes against that fixture before deleting the in-cluster file, confirms absence, then copies the backup into the mounted volume with kubectl cp. It compares exact restored content before and after Pod replacement, reporting byte counts and SHA-256 digests. The PVC remains while the consumer Pod is deleted and recreated from data-consumer.yaml. The data survives because the new Pod mounts the same claim, not because Kubernetes knows how to regenerate the order record.
Restore bytes, then prove a new Pod sees them
A new Pod UID mounts the same PVC and reads the restored content, demonstrating persistence after recovery.
kubectl cp restores /data/orders.csv in the existing volume; byte counts and SHA-256 digests must match the private host backup.
- Host CSV backup to Restore file copyread retained copy
- Restore file copy to Existing PVCwrite recovered bytes
Read the complete visual sequence
- Restore the record
kubectl cp restores /data/orders.csv in the existing volume; byte counts and SHA-256 digests must match the private host backup.
- Verify before replacement
The script checks exact expected content so a successful write command alone is insufficient.
- Verify after replacement
A new Pod UID mounts the same PVC and reads the restored content, demonstrating persistence after recovery.
Backup captured during database writes
A byte copy can be nonempty yet fail application consistency requirements.
- Concurrent writers remain active
A hypothetical database changes related files while capture is in progress.
- Restore returns bytes
The files copy successfully, but their relationships may reflect different transaction points.
- Integrity is unproven
Use application-native backup or validated quiescing and verify transaction-level invariants.
About this model
The actual lab has one completed CSV write and tests file restoration, not database recovery or node-loss resilience.
The PVC below requests a small dynamically provisioned volume using the environment’s default StorageClass. ReadWriteOnce describes a supported access mode; it is not an application-level transaction lock or a guarantee of cross-node portability for local storage.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: m20
namespace: kv-lab
labels:
lab: m20
spec:
accessModes: [ReadWriteOnce]
resources:
requests:
storage: 64Mi
# Run in the downloaded m20 directory.
bash capstone-stateful.sh
# Verify the backup is retained on the host and the restored value is mounted.
backup_dir=$(python3 backup-path.py m20)
cat "$backup_dir/orders.csv"
kubectl --context kind-kv-lab -n kv-lab exec m20-data -- cat /data/orders.csv
kubectl --context kind-kv-lab -n kv-lab get pvc m20 -o wide
kubectl --context kind-kv-lab -n kv-lab get pod m20-data -o jsonpath='{.metadata.uid}{"\n"}'
Practice: make each recovery claim testable
Run the m20 local exercise using capstone-stateful.sh. Record five observations: the original record, the host backup, file absence after deliberate deletion, exact restored content, and the same content after a new Pod UID mounts the PVC. The script performs these assertions so a green result has a clear meaning. It does not measure restore throughput, loss under concurrent writes, database consistency or recovery from a failed storage node.
The script restores this bounded fixture with kubectl cp, which requires tar inside the container; the pinned BusyBox image provides it. It compares byte counts and SHA-256 digests before and after Pod replacement. Its private host output defaults to an operating-system temporary directory outside the website assets. Move real backups to durable protected storage with a defined retention policy.
Read the StorageClass, bound PV and reclaim policy before cleanup. A local provisioner can tie volume data to a particular node or container filesystem. Preserving a PVC while restarting its Pod tests a different failure than losing the node that stores its bytes. Deleting the PVC may trigger backend deletion according to reclaim policy and implementation. Keep the backup outside that deletion scope before removing storage resources.
Pitfalls: data integrity needs application knowledge
A nonempty backup can still contain incomplete records. Checksums prove correspondence between bytes being compared, not that those bytes form a valid application state. For a real database, verify catalog integrity, transaction recovery, record counts or business invariants using its supported tools. Define whether restore overwrites existing records, merges them, or requires an empty target. Rehearse credentials and access to encrypted backups before an incident removes the original key service.
An etcd restore can reconstruct the PVC object without reconstructing the volume’s CSV file. Conversely, the host CSV backup cannot recreate RBAC, Service configuration or control-plane trust. Recovery is the assembly of several asset classes under a tested order. Keep the operator or GitOps controller from immediately reapplying conflicting old intent while that assembly is incomplete.
Check your understanding
The CSV is deleted from the PVC, and the Pod is replaced. Why is the CSV still missing until restoration?
The replacement mounts the same persistent filesystem, including its deletion. Persistence preserves current bytes and metadata; it does not keep an automatic historical version of every file. The external backup supplies the earlier record that the restore procedure writes back.
Next, recover declarative ownership itself. The final capstone coordinates GitOps intent, an operator’s generated resources, conflict handling and garbage collection without making two controllers own the same field.
Go to the source
Reviewed 2026-10-27 · Kubernetes 1.37 · Related local exercise