Project 2: Build a kubeadm Cluster, Back Up etcd, Upgrade and Recover From Failures
Goal
Operate a cluster through its whole life: build, back up, upgrade, break and repair. These are the same skills the CKA tests, and the same skills you need on call.
| Phase | Skill from the course |
| 1. Build | Install lesson (containerd, kubeadm, CNI, join) |
| 2. Workload and baseline | Deployments, Services, PVC, RBAC |
| 3. Backup | etcd snapshot, off-node copy, verification |
| 4. Upgrade | One minor version with drain and uncordon |
| 5. Failure drills | Troubleshooting lessons |
| 6. Restore | etcd restore, rebuild checks |
| 7. Document | Runbook |
Phase 1: Build the Lab
| VM | Role | Size |
| cp1 | Control plane | 2 CPU, 4 GB |
| w1 | Worker | 2 CPU, 2 GB |
| w2 | Worker | 2 CPU, 2 GB |
- Create three Linux VMs (Vagrant, Multipass, VirtualBox, or cloud) with static IPs and the same OS
- On all nodes: swap off, kernel modules, sysctl, containerd with
SystemdCgroup = true, kubeadm/kubelet/kubectl at the older minor you will upgrade from (for example 1.36)
- On cp1:
kubeadm init with --pod-network-cidr, copy the kubeconfig, install a CNI
- Join w1 and w2 and check
kubectl get nodes
bash
kubectl get nodes -o wide
kubectl get pods -A
# Save the facts you will need later
kubectl version -o yaml > /root/cluster-before.yaml
sudo kubeadm certs check-expiration > /root/certs-before.txt
sudo cp -a /etc/kubernetes /root/kubernetes-backup-$(date +%F)
Phase 2: A Baseline Workload
bash
kubectl create namespace demo
kubectl -n demo create deployment web --image=nginx:1.27 --replicas=4
kubectl -n demo expose deployment web --port=80
kubectl -n demo create configmap marker --from-literal=created=before-backup
kubectl -n demo create secret generic note --from-literal=text=important
cat <<EOF | kubectl apply -f -
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: {name: web, namespace: demo}
spec:
minAvailable: 3
selector: {matchLabels: {app: web}}
EOF
kubectl create role viewer -n demo --verb=get,list --resource=pods
kubectl -n demo get all,cm,secret,pdb,role
Phase 3: Back Up etcd
bash
export ETCDCTL_API=3 ETCDCTL_ENDPOINTS=https://127.0.0.1:2379
export ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key
sudo mkdir -p /backup
sudo -E etcdctl snapshot save /backup/etcd-before-upgrade.db
sudo etcdutl snapshot status /backup/etcd-before-upgrade.db -w table
sudo cp -a /etc/kubernetes/pki /backup/pki
scp -r /backup user@another-machine:/srv/k8s-backups/ # keep a copy OFF the node
# After the backup, create one more object to test the restore later
kubectl -n demo create configmap marker2 --from-literal=created=after-backup
Phase 4: Upgrade One Minor Version
- Read the release notes of the target minor and check deprecated API usage
- Change the package repository on cp1 to the new minor, upgrade
kubeadm, run kubeadm upgrade plan
kubeadm upgrade apply v1.37.x on cp1
- Drain cp1, upgrade
kubelet and kubectl, restart kubelet, uncordon
- For w1 and then w2: upgrade
kubeadm, kubeadm upgrade node, drain, upgrade kubelet, restart, uncordon
- Verify: versions, pods, the
demo namespace, PDB behaviour during drain
bash
# Watch that the app stays available during the node upgrades
kubectl -n demo run watcher --image=busybox:1.36 --restart=Never -- sh -c 'while true; do wget -q -T2 -O- http://web >/dev/null && echo ok || echo FAIL; sleep 1; done'
kubectl -n demo logs -f watcher | grep -c FAIL # should stay 0 or very low
kubectl get nodes ; sudo kubeadm certs check-expiratio
Phase 5: Failure Drills
Do each drill, find the cause using only the method from the troubleshooting lessons, fix it, then write one line in your runbook.
| Drill | How to break it (lab only) | What you should do |
| A. API server will not start | Edit /etc/kubernetes/manifests/kube-apiserver.yaml and change the --etcd-certfile path to a wrong name (back up the file first outside the folder) | crictl ps -a, crictl logs, fix the path, wait for kubelet to restart the pod |
| B. Scheduler broken | Change the scheduler image name in its manifest to a wrong tag | New pods stay Pending with no events. crictl/kubelet logs show ImagePull errors. Fix the image |
| C. Node NotReady | systemctl stop kubelet on w1 | See NotReady, wait for eviction timers, restart the kubelet. Then break it differently: set cgroupDriver: cgroupfs in the kubelet config |
| D. Container runtime down | systemctl stop containerd on w2 | Kubelet logs show CRI errors. Restart containerd, check pods |
| E. Certificate expiry | Move the clock forward in a throwaway VM, or inspect kubeadm certs check-expiration and run kubeadm certs renew all | Restart control plane static pods and refresh kubeconfig |
| F. DNS broken | kubectl -n kube-system scale deploy coredns --replicas=0 | Name lookups fail from pods. Scale back and verify with nslookup |
| G. Bad webhook | Create a ValidatingWebhookConfiguration pointing to a missing Service with failurePolicy: Fail | Creates are rejected. Find and delete the webhook config |
| H. Namespace deleted | kubectl delete ns demo | Restore from etcd snapshot (next phase) or from Git/YAML |
Phase 6: Restore etcd From the Snapshot
bash
# 1) Stop the API server by moving its manifest away
sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml /root/
# 2) Restore to a new data directory
sudo etcdutl snapshot restore /backup/etcd-before-upgrade.db --data-dir /var/lib/etcd-restored
# 3) Point the etcd static pod hostPath to the restored directory
sudo cp /etc/kubernetes/manifests/etcd.yaml /root/etcd.yaml.bak
sudo sed -i 's#path: /var/lib/etcd$#path: /var/lib/etcd-restored#' /etc/kubernetes/manifests/etcd.yaml
sudo grep -n 'etcd-restored' /etc/kubernetes/manifests/etcd.yaml
# 4) Wait for etcd, bring the API server back
sleep 30 ; sudo crictl ps | grep etcd
sudo mv /root/kube-apiserver.yaml /etc/kubernetes/manifests/
kubectl get ns ; kubectl -n demo get cm # marker exists, marker2 (created after the snapshot) is gone
A snapshot taken before an upgrade restores the old state, not the old binaries: If you restore a pre-upgrade snapshot after upgrading control plane components, the API objects roll back but the running components stay at the new version. For a real rollback you must also reinstall the previous component versions. Use this phase to understand restore mechanics, and treat it as part of a documented disaster plan.
Phase 7: Write the Runbook
| Section | Contents |
| Inventory | Node names, IPs, versions, CIDRs, endpoints, who owns what |
| Backup | Command, schedule, storage location, retention, last restore test date |
| Restore | Exact numbered steps and paths for this cluster |
| Upgrade | Version path, order, checks, rollback points |
| Failure guides | One page per drill: symptom, checks, fix |
| Contacts and escalation | Who to call and when |
Self-Assessment
| Skill | Can you do it in under 10 minutes without notes? |
Find and read a failing static pod's logs with crictl | |
| Take and verify an etcd snapshot | |
| Drain, upgrade and uncordon a node | |
| Create a user with a certificate and limited RBAC | |
| Create a PV/PVC and mount it in a pod | |
| Write a NetworkPolicy from a description | |
| Fix a Pending pod caused by a taint or an affinity rule | |
Practice Tasks
- Complete phases 1 to 4 and show the node versions before and after
- Complete at least five of the failure drills and write the fix for each
- Restore etcd and prove which objects came back
- Time yourself doing each CKA-style task and set a goal to halve the time
- Publish your runbook in a Git repository
Conclusion
Cluster administration is a lifecycle: build, protect, upgrade, break and repair. By practising each stage in a lab and documenting it in a runbook, you gain the exact skills the CKA exam and production on-call work demand.
Quick Summary: build kubeadm; baseline workload; etcd snapshot+pki copy; upgrade cp then workers; drills: apiserver manifest, scheduler image, kubelet, containerd, certs, DNS, webhook; restore; runbook.