Project 2: Build a kubeadm Cluster, Back Up etcd, Upgrade and Recover From Failures

Certified Kubernetes Administrator 45 min min read Updated: Nov 17, 2026 Advanced
Project 2: Build a kubeadm Cluster, Back Up etcd, Upgrade and Recover From Failures
Advanced Topic 2 of 3

Goal

Operate a cluster through its whole life: build, back up, upgrade, break and repair. These are the same skills the CKA tests, and the same skills you need on call.

PhaseSkill from the course
1. BuildInstall lesson (containerd, kubeadm, CNI, join)
2. Workload and baselineDeployments, Services, PVC, RBAC
3. Backupetcd snapshot, off-node copy, verification
4. UpgradeOne minor version with drain and uncordon
5. Failure drillsTroubleshooting lessons
6. Restoreetcd restore, rebuild checks
7. DocumentRunbook

Phase 1: Build the Lab

VMRoleSize
cp1Control plane2 CPU, 4 GB
w1Worker2 CPU, 2 GB
w2Worker2 CPU, 2 GB
  1. Create three Linux VMs (Vagrant, Multipass, VirtualBox, or cloud) with static IPs and the same OS
  2. On all nodes: swap off, kernel modules, sysctl, containerd with SystemdCgroup = true, kubeadm/kubelet/kubectl at the older minor you will upgrade from (for example 1.36)
  3. On cp1: kubeadm init with --pod-network-cidr, copy the kubeconfig, install a CNI
  4. Join w1 and w2 and check kubectl get nodes
bash kubectl get nodes -o wide kubectl get pods -A # Save the facts you will need later kubectl version -o yaml > /root/cluster-before.yaml sudo kubeadm certs check-expiration > /root/certs-before.txt sudo cp -a /etc/kubernetes /root/kubernetes-backup-$(date +%F)

Phase 2: A Baseline Workload

bash kubectl create namespace demo kubectl -n demo create deployment web --image=nginx:1.27 --replicas=4 kubectl -n demo expose deployment web --port=80 kubectl -n demo create configmap marker --from-literal=created=before-backup kubectl -n demo create secret generic note --from-literal=text=important cat <<EOF | kubectl apply -f - apiVersion: policy/v1 kind: PodDisruptionBudget metadata: {name: web, namespace: demo} spec: minAvailable: 3 selector: {matchLabels: {app: web}} EOF kubectl create role viewer -n demo --verb=get,list --resource=pods kubectl -n demo get all,cm,secret,pdb,role

Phase 3: Back Up etcd

bash export ETCDCTL_API=3 ETCDCTL_ENDPOINTS=https://127.0.0.1:2379 export ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key sudo mkdir -p /backup sudo -E etcdctl snapshot save /backup/etcd-before-upgrade.db sudo etcdutl snapshot status /backup/etcd-before-upgrade.db -w table sudo cp -a /etc/kubernetes/pki /backup/pki scp -r /backup user@another-machine:/srv/k8s-backups/ # keep a copy OFF the node # After the backup, create one more object to test the restore later kubectl -n demo create configmap marker2 --from-literal=created=after-backup

Phase 4: Upgrade One Minor Version

  1. Read the release notes of the target minor and check deprecated API usage
  2. Change the package repository on cp1 to the new minor, upgrade kubeadm, run kubeadm upgrade plan
  3. kubeadm upgrade apply v1.37.x on cp1
  4. Drain cp1, upgrade kubelet and kubectl, restart kubelet, uncordon
  5. For w1 and then w2: upgrade kubeadm, kubeadm upgrade node, drain, upgrade kubelet, restart, uncordon
  6. Verify: versions, pods, the demo namespace, PDB behaviour during drain
bash # Watch that the app stays available during the node upgrades kubectl -n demo run watcher --image=busybox:1.36 --restart=Never -- sh -c 'while true; do wget -q -T2 -O- http://web >/dev/null && echo ok || echo FAIL; sleep 1; done' kubectl -n demo logs -f watcher | grep -c FAIL # should stay 0 or very low kubectl get nodes ; sudo kubeadm certs check-expiratio

Phase 5: Failure Drills

Do each drill, find the cause using only the method from the troubleshooting lessons, fix it, then write one line in your runbook.

DrillHow to break it (lab only)What you should do
A. API server will not startEdit /etc/kubernetes/manifests/kube-apiserver.yaml and change the --etcd-certfile path to a wrong name (back up the file first outside the folder)crictl ps -a, crictl logs, fix the path, wait for kubelet to restart the pod
B. Scheduler brokenChange the scheduler image name in its manifest to a wrong tagNew pods stay Pending with no events. crictl/kubelet logs show ImagePull errors. Fix the image
C. Node NotReadysystemctl stop kubelet on w1See NotReady, wait for eviction timers, restart the kubelet. Then break it differently: set cgroupDriver: cgroupfs in the kubelet config
D. Container runtime downsystemctl stop containerd on w2Kubelet logs show CRI errors. Restart containerd, check pods
E. Certificate expiryMove the clock forward in a throwaway VM, or inspect kubeadm certs check-expiration and run kubeadm certs renew allRestart control plane static pods and refresh kubeconfig
F. DNS brokenkubectl -n kube-system scale deploy coredns --replicas=0Name lookups fail from pods. Scale back and verify with nslookup
G. Bad webhookCreate a ValidatingWebhookConfiguration pointing to a missing Service with failurePolicy: FailCreates are rejected. Find and delete the webhook config
H. Namespace deletedkubectl delete ns demoRestore from etcd snapshot (next phase) or from Git/YAML

Phase 6: Restore etcd From the Snapshot

bash # 1) Stop the API server by moving its manifest away sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml /root/ # 2) Restore to a new data directory sudo etcdutl snapshot restore /backup/etcd-before-upgrade.db --data-dir /var/lib/etcd-restored # 3) Point the etcd static pod hostPath to the restored directory sudo cp /etc/kubernetes/manifests/etcd.yaml /root/etcd.yaml.bak sudo sed -i 's#path: /var/lib/etcd$#path: /var/lib/etcd-restored#' /etc/kubernetes/manifests/etcd.yaml sudo grep -n 'etcd-restored' /etc/kubernetes/manifests/etcd.yaml # 4) Wait for etcd, bring the API server back sleep 30 ; sudo crictl ps | grep etcd sudo mv /root/kube-apiserver.yaml /etc/kubernetes/manifests/ kubectl get ns ; kubectl -n demo get cm # marker exists, marker2 (created after the snapshot) is gone
A snapshot taken before an upgrade restores the old state, not the old binaries: If you restore a pre-upgrade snapshot after upgrading control plane components, the API objects roll back but the running components stay at the new version. For a real rollback you must also reinstall the previous component versions. Use this phase to understand restore mechanics, and treat it as part of a documented disaster plan.

Phase 7: Write the Runbook

SectionContents
InventoryNode names, IPs, versions, CIDRs, endpoints, who owns what
BackupCommand, schedule, storage location, retention, last restore test date
RestoreExact numbered steps and paths for this cluster
UpgradeVersion path, order, checks, rollback points
Failure guidesOne page per drill: symptom, checks, fix
Contacts and escalationWho to call and when

Self-Assessment

SkillCan you do it in under 10 minutes without notes?
Find and read a failing static pod's logs with crictl
Take and verify an etcd snapshot
Drain, upgrade and uncordon a node
Create a user with a certificate and limited RBAC
Create a PV/PVC and mount it in a pod
Write a NetworkPolicy from a description
Fix a Pending pod caused by a taint or an affinity rule

Practice Tasks

  1. Complete phases 1 to 4 and show the node versions before and after
  2. Complete at least five of the failure drills and write the fix for each
  3. Restore etcd and prove which objects came back
  4. Time yourself doing each CKA-style task and set a goal to halve the time
  5. Publish your runbook in a Git repository

Conclusion

Cluster administration is a lifecycle: build, protect, upgrade, break and repair. By practising each stage in a lab and documenting it in a runbook, you gain the exact skills the CKA exam and production on-call work demand.

Quick Summary: build kubeadm; baseline workload; etcd snapshot+pki copy; upgrade cp then workers; drills: apiserver manifest, scheduler image, kubelet, containerd, certs, DNS, webhook; restore; runbook.

Get Newsletter

Subscibe to our newsletter and we will notify you about the newest updates on Edugators