CKA etcd snapshot save and restore to a new data dir

Save with the server's certs, restore to a new dir, move the pod.

Everything the cluster knows lives in etcd. Take the snapshot with the server's own certs, restore to a new directory, and move the pod, not the files.

§I. Frame

Poulton calls etcd the cluster store: the only stateful part of the control plane. Lose it and the API server has nothing to serve. A CKA task usually reads: save a snapshot to a given path, then restore from a provided snapshot. On kubeadm, etcd is a static pod, so the 10-09 audit lesson's skill (edit a manifest, wait for the kubelet) applies again.

The task for today: save /opt/backup/etcd.db, restore it into /var/lib/etcd-restore, and bring the cluster back on the restored data.

§II. Three techniques

Borrow the Pod's Flags (named technique). Do not guess endpoints or cert paths. Read them from the manifest:

grep -E 'listen-client-urls|cert-file|key-file|trusted-ca-file|data-dir' \
  /etc/kubernetes/manifests/etcd.yaml

On a default kubeadm node that shows https://127.0.0.1:2379, certs under /etc/kubernetes/pki/etcd/, and --data-dir=/var/lib/etcd.

Save Talks to the Server, Restore Talks to Disk (named technique). etcdctl snapshot save is a client call over TLS, so it needs the endpoint and three cert flags. Restore does not talk to a server; it writes a fresh data directory. The etcd docs move offline restore to etcdutl; etcdctl snapshot restore still works on many exam images but prints a deprecation warning.

New Dir, Then Move the Pod (named technique). Restore into a new directory. Then change the etcd static pod to use it. Never restore over the live /var/lib/etcd.

§III. Save

ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /opt/backup/etcd.db

etcdutl snapshot status /opt/backup/etcd.db -w table

Illustrative status output (not captured): a table with hash, revision, total keys, and total size. A non-empty key count is the quick sanity check.

§IV. Restore

etcdutl snapshot restore /opt/backup/etcd.db --data-dir /var/lib/etcd-restore

Then edit /etc/kubernetes/manifests/etcd.yaml. The kubernetes.io page shows two working edits; the smaller one changes only the hostPath volume:

  volumes:
  - hostPath:
      path: /var/lib/etcd-restore    # was /var/lib/etcd
      type: DirectoryOrCreate
    name: etcd-data

The container still mounts at /var/lib/etcd and --data-dir stays the same; only the host directory behind it changes. If you instead change --data-dir, change the volumeMounts path to match, or etcd starts on an empty directory inside the container.

The kubelet sees the manifest change and restarts etcd. The API server reconnects once etcd answers. Check with crictl ps -a | grep etcd, then kubectl get pods -A. Expect objects created after the snapshot to be gone.

§V. Drills

1. snapshot save fails with a TLS or deadline error. What is usually missing? (One of --cacert, --cert, --key, or the https:// endpoint.)
2. Does etcdutl snapshot restore need the cert flags? (No: it writes to disk and talks to no server.)
3. You changed --data-dir to /var/lib/etcd-restore but left the hostPath and mount alone. What happens? (etcd sees an empty directory inside the container; the cluster comes up empty.)
4. After restore, a Deployment you created ten minutes ago is gone. Bug? (No: the snapshot predates it.)

Close

Borrow the Pod's Flags, Save Talks to the Server, New Dir Then Move the Pod. Read the manifest, save with the server certs, restore to a new directory, and point the static pod at it.

Related

• Tome: Poulton ch2, cluster store (grounded-in)
• Web: Operating etcd clusters for Kubernetes · etcd disaster recovery · CKA curriculum
• Prior: 10-09 CKS audit Policy (same static-pod edit loop)