16. Cluster Operations and Maintenance

Standing up a cluster is the beginning; keeping it healthy, current, recoverable, and cost-efficient over years is the real work — "Day 2 operations." This chapter covers the operational disciplines: upgrading clusters safely, backing up and recovering from disaster, controlling cost, and troubleshooting the failures you'll inevitably face. These are the skills that separate running a demo cluster from operating production.

---

16.1 Cluster Upgrades

Kubernetes releases frequently, and clusters must be upgraded to stay supported and secure. This subchapter covers doing it safely.

---

kubeadm upgrade workflow

Theory

Kubernetes has a fast release cadence (roughly three minor versions per year, each supported ~1 year), so upgrading is a recurring, mandatory operation. For self-managed clusters, kubeadm provides a structured upgrade workflow. The cardinal rule: upgrade the control plane before the nodes, and go one minor version at a time (you can't skip minors, e.g., 1.29 → 1.31 directly).

The kubeadm upgrade sequence:

1. Upgrade the first control-plane node: upgrade the kubeadm binary, run kubeadm upgrade plan (shows available versions/what will change) and kubeadm upgrade apply <version> (upgrades the control-plane components/etcd), then upgrade that node's kubelet/kubectl and restart the kubelet.
2. Upgrade other control-plane nodes: kubeadm upgrade node on each, then their kubelets.
3. Upgrade worker nodes one at a time: drain the node (evict Pods safely), kubeadm upgrade node, upgrade its kubelet, restart, then uncordon it.

Throughout, the version skew policy (next) bounds allowed differences. Doing this carefully — control plane first, one minor at a time, nodes drained during their turn — is what makes upgrades safe and non-disruptive.

Example

# On the first control-plane node:
apt-get install -y kubeadm=1.30.x         # upgrade kubeadm binary
kubeadm upgrade plan                       # preview available upgrades
kubeadm upgrade apply v1.30.2              # upgrade control plane + etcd
apt-get install -y kubelet=1.30.x kubectl=1.30.x && systemctl restart kubelet

# On each worker (one at a time):
kubectl drain node-2 --ignore-daemonsets --delete-emptydir-data
kubeadm upgrade node && apt-get install -y kubelet=1.30.x && systemctl restart kubelet
kubectl uncordon node-2

Exercises

1. (Beginner) Do you upgrade the control plane or the worker nodes first?
2. (Beginner) Can you skip minor versions (e.g., go from 1.29 straight to 1.31)?
3. (Intermediate) What does kubeadm upgrade plan do versus kubeadm upgrade apply?
4. (Interview) Why must worker nodes be drained before upgrading, and what role does one-node-at-a-time play? (Hint: evict/reschedule workloads; maintain availability.)

Answers

1. The control plane first, then the worker nodes.
2. No — you must upgrade one minor version at a time (e.g., 1.29 → 1.30 → 1.31); skipping minors isn't supported.
3. kubeadm upgrade plan inspects the cluster and shows available target versions and what the upgrade would change (a preview/validation step); kubeadm upgrade apply <version> actually performs the control-plane upgrade to the chosen version.
4. Draining cordons the node and evicts its Pods (gracefully, respecting PodDisruptionBudgets) so they're rescheduled onto other nodes before you disrupt/restart the node's kubelet — avoiding running workloads being killed uncleanly. Doing it one node at a time keeps the rest of the cluster serving traffic and preserves capacity/availability throughout the upgrade, so only a small slice is unavailable at any moment rather than the whole fleet at once.

---

Managed cluster upgrade strategies

Theory

On managed Kubernetes (EKS, GKE, AKS), the provider handles the control-plane upgrade for you (often a one-click or one-API-call operation, with the provider managing etcd, API server, etc.). Your responsibility shifts largely to node (data-plane) upgrades and validating that your workloads tolerate the new version.

Managed node-upgrade strategies:

• Surge/rolling node upgrades: the provider (or a managed node group) creates new nodes on the new version, cordons/drains old nodes, and moves workloads over — minimizing disruption (respecting PDBs).
• Blue-green node pools: create a new node pool on the new version, migrate workloads, then delete the old pool — safe and easily reversible.
• Auto-upgrade / release channels: GKE (and others) offer channels (rapid/regular/stable) and can auto-upgrade the control plane and nodes on a maintenance schedule, trading control for less manual effort.

Best practices regardless of provider: upgrade non-prod first, respect PodDisruptionBudgets, use maintenance windows, and test workloads against the new version (check for removed/deprecated APIs). Managed upgrades remove most control-plane toil, but you still own workload compatibility and node rollout safety.

Example

# GKE: upgrade control plane, then node pool (or use release channels for auto)
gcloud container clusters upgrade prod --master --cluster-version 1.30.2
gcloud container clusters upgrade prod --node-pool default-pool --cluster-version 1.30.2

# EKS: control plane, then managed node group (rolling, respects PDBs)
aws eks update-cluster-version --name prod --kubernetes-version 1.30
aws eks update-nodegroup-version --cluster-name prod --nodegroup-name ng-1

Exercises

1. (Beginner) In managed Kubernetes, who handles the control-plane upgrade?
2. (Beginner) What upgrade responsibility remains largely yours on a managed cluster?
3. (Intermediate) Describe the blue-green node pool upgrade strategy.
4. (Interview) What should you do before upgrading production, regardless of managed vs. self-managed, to avoid surprises? (Hint: test non-prod, check deprecated/removed APIs, PDBs, maintenance windows.)

Answers

1. The cloud provider handles the control-plane upgrade (API server, etcd, scheduler, controller-manager).
2. Node/data-plane upgrades and ensuring your workloads are compatible with the new version (node pool rollout, PDBs, testing) — even if the provider offers node auto-upgrade, workload compatibility is yours.
3. Create a new node pool running the new Kubernetes version alongside the old pool, migrate workloads onto the new pool (cordon/drain the old nodes so Pods reschedule to the new ones), verify everything is healthy, then delete the old node pool. It's safe and easily reversible (keep the old pool until confident).
4. Upgrade and test in non-production first; check for deprecated/removed APIs your manifests use (a common breakage across minor versions) and update them; ensure PodDisruptionBudgets are set so drains don't take down too many replicas; schedule during a maintenance window; and verify application health/compatibility on the new version before rolling production. This catches API removals, workload incompatibilities, and disruption issues before they hit prod.

---

Node draining and cordoning

Theory

Before you take a node out of service (for upgrade, maintenance, or decommissioning), you must move its workloads off safely. Two related operations:

• Cordoning (kubectl cordon <node>): marks the node unschedulable — no new Pods will be placed on it, but existing Pods keep running. It's a "stop sending work here" flag.
• Draining (kubectl drain <node>): cordons the node and evicts its existing Pods (gracefully), so they're rescheduled elsewhere, leaving the node empty and ready for maintenance.

Draining uses the Eviction API, which respects PodDisruptionBudgets (PDBs) — a PDB declares the minimum available replicas (or max unavailable) for an application, so drain will block/slow rather than violate it (preventing an eviction from taking a service below its safe replica count). Common flags: --ignore-daemonsets (DaemonSet Pods aren't evicted — they're node-bound), --delete-emptydir-data (acknowledge losing emptyDir data). After maintenance, kubectl uncordon <node> makes it schedulable again. Cordon/drain + PDBs are the mechanism for zero/minimal-disruption node maintenance.

Example

kubectl cordon node-2                     # stop scheduling new Pods here
kubectl drain node-2 --ignore-daemonsets --delete-emptydir-data   # evict Pods safely
#   ... perform maintenance / upgrade ...
kubectl uncordon node-2                    # allow scheduling again

# A PodDisruptionBudget makes drain respect availability:
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: web-pdb }
spec:
  minAvailable: 2                # drain won't evict below 2 available web Pods
  selector: { matchLabels: { app: web } }

Exercises

1. (Beginner) What is the difference between cordoning and draining a node?
2. (Beginner) What does kubectl uncordon do?
3. (Intermediate) How does a PodDisruptionBudget interact with a drain?
4. (Interview) Why does kubectl drain skip DaemonSet Pods, and how do PDBs enable minimal-disruption maintenance? (Hint: node-bound Pods; drain blocks rather than violating availability.)

Answers

1. Cordoning marks the node unschedulable (no new Pods) but leaves existing Pods running; draining cordons and evicts the existing Pods (gracefully) so they reschedule elsewhere, emptying the node.
2. Marks the node schedulable again, so the scheduler can place new Pods on it after maintenance.
3. Drain uses the Eviction API, which honors PDBs: it won't evict Pods if doing so would drop an application below its PDB's minAvailable (or exceed maxUnavailable). Instead the drain waits/blocks until it's safe to evict more, ensuring the service stays above its declared availability threshold.
4. DaemonSet Pods are inherently tied to their node (one per node, providing node-level function) and would just be recreated on the same node, so evicting them is pointless — drain skips them (hence --ignore-daemonsets). PDBs enable minimal-disruption maintenance by declaring how many replicas must stay available; drain respects them, so evictions proceed only as fast as the app can tolerate (e.g., one at a time while keeping minAvailable up), preventing a maintenance operation from taking a service down. Together, cordon/drain move work off a node while PDBs guarantee availability isn't violated during the process.

---

Version skew policy

Theory

Kubernetes components run at potentially different versions during and after upgrades, so Kubernetes defines a version skew policy specifying the maximum allowed version differences between components. Respecting it is what makes staged upgrades safe. The core rules:

• kube-apiserver is the reference. In HA, the API servers may differ by at most 1 minor from each other.
• kubelet may be up to 3 minor versions older than the API server (but never newer). This is why you upgrade the control plane first — the API server must be at least as new as the kubelets.
• kube-controller-manager, kube-scheduler, cloud-controller-manager must be within 1 minor of (and not newer than) the API server.
• kube-proxy matches its node's kubelet (within the same skew of the API server).
• kubectl is supported within ±1 minor of the API server.

The practical implications: upgrade order matters (control plane before nodes, because the API server must lead), you can't skip minors (or you'd violate skew mid-upgrade), and you shouldn't let nodes lag more than 3 minors behind. The skew policy is the formal guarantee that a partially-upgraded cluster still functions correctly, enabling rolling, non-disruptive upgrades.

Example

API server = v1.30 (the reference)
  kubelet:            v1.27 - v1.30 allowed (up to 3 older, never newer)
  controller-mgr/sched: v1.29 - v1.30 (within 1, not newer)
  kube-proxy:         matches node kubelet
  kubectl:            v1.29 - v1.31 (±1)
=> upgrade order: API server FIRST, then the rest, one minor at a time.

Exercises

1. (Beginner) Which component is the reference point for the version skew policy?
2. (Beginner) How many minor versions older than the API server may a kubelet be? Can it be newer?
3. (Intermediate) How does the skew policy explain why you upgrade the control plane before nodes?
4. (Interview) Why does the version skew policy make it impossible to skip minor versions during an upgrade? (Hint: skipping would exceed allowed component differences mid-upgrade.)

Answers

1. The kube-apiserver.
2. Up to 3 minor versions older, and it may never be newer than the API server.
3. Because the kubelet must not be newer than the API server (and components must be within their allowed skew of it), the API server has to be upgraded first so it's at least as new as everything else. Upgrading a node's kubelet ahead of the control plane would make the kubelet newer than the API server, violating the policy.
4. Component versions must stay within the allowed skew of the API server at all times. If you jumped the control plane two minors at once (e.g., 1.29 → 1.31), the still-1.29 kubelets/components could fall outside the supported skew relative to a 1.31 API server during the process, producing an unsupported, potentially broken state. Upgrading one minor at a time keeps every component within the permitted difference throughout, so the cluster remains functional at each step — which is precisely why skipping minors isn't allowed.

---

16.2 Backup and Disaster Recovery

Things fail; the question is whether you can recover. This subchapter covers backing up cluster state and applications and planning for disaster.

---

etcd backup and restore

Theory

Since etcd holds the entire cluster state (Chapter 2), backing it up is the single most critical disaster-recovery task for a self-managed cluster: lose etcd irrecoverably and you lose the whole cluster's state. etcd provides a snapshot mechanism — etcdctl snapshot save captures a consistent point-in-time copy of the datastore, which you store safely (encrypted, off-cluster, ideally off-site).

Key practices:

• Regular, automated snapshots (frequency driven by your RPO — how much state you can afford to lose).
• Store snapshots securely and off the cluster (they contain everything, including Secrets — treat them as sensitive), and test restores (an untested backup isn't a backup).
• Restore with etcdctl snapshot restore, which recreates the etcd data directory from a snapshot; you then point etcd at it and bring the control plane back. In HA etcd, restore involves reinitializing the cluster from the snapshot.

On managed Kubernetes, the provider manages etcd (and its backups), so this is largely their responsibility — but on self-managed clusters, etcd backup/restore is your job and the foundation of cluster DR. (Note: etcd backup restores cluster objects; it doesn't back up PersistentVolume data — that needs separate volume snapshots/Velero.)

Example

# Take a snapshot of etcd (self-managed control plane):
ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-$(date +%F).db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Restore from a snapshot (recreates the data dir):
ETCDCTL_API=3 etcdctl snapshot restore /backup/etcd-2026-07-01.db \
  --data-dir /var/lib/etcd-restored

Exercises

1. (Beginner) Why is backing up etcd the most critical DR task on a self-managed cluster?
2. (Beginner) What command takes an etcd snapshot?
3. (Intermediate) Why must etcd snapshots be treated as sensitive and stored off-cluster, and why test restores?
4. (Interview) Does an etcd backup protect your PersistentVolume data? What else do you need for full application DR? (Hint: etcd = objects, not PV contents; need volume snapshots/Velero.)

Answers

1. Because etcd stores the entire cluster state (all API objects); if it's lost and unrecoverable, the whole cluster's state is gone. A snapshot is the only way to restore that state.
2. etcdctl snapshot save (with the appropriate endpoints/certs).
3. Snapshots contain everything in the cluster — including Secrets — so they're highly sensitive and must be encrypted and stored securely off the cluster (so a cluster/node loss doesn't also lose the backup). Restores must be tested because a backup that has never been verified may be corrupt, incomplete, or unrestorable — you only truly have a backup if you've confirmed you can restore from it.
4. No — an etcd backup captures Kubernetes objects (Deployments, Services, PVCs/PV definitions, Secrets, etc.), but not the actual data inside PersistentVolumes (the disk contents). For full application disaster recovery you also need volume-level backups — CSI volume snapshots and/or a tool like Velero (which backs up cluster objects and orchestrates PV snapshots) — so both the cluster state and the persistent application data can be recovered.

---

Velero for cluster and application backup

Theory

Where etcd snapshots protect the whole cluster's raw state (self-managed only, all-or-nothing), Velero provides application-level, selective, portable backup and restore for Kubernetes. It's the standard tool for backing up namespaces/applications and their data, and works on managed clusters too. Velero backs up two things: Kubernetes API objects (filtered by namespace, label, or resource type) and, via integration, the persistent volume data backing them (using CSI volume snapshots or its file-level backup, Restic/Kopia).

Velero's capabilities make it the go-to for practical DR and migration:

• Selective backup/restore: back up a single namespace or app, not the whole cluster.
• Scheduled backups with retention (TTL).
• PV data: snapshot volumes alongside object backups so restored apps have their data.
• Migration / cloning: restore a backup into a different cluster (e.g., move an app between clusters or clouds, or rebuild after disaster) — because backups are stored in object storage (S3/GCS/Azure Blob) and are portable.

So Velero complements etcd backup: etcd is coarse full-cluster state (self-managed), while Velero gives granular, portable, data-inclusive app backups that work anywhere. Many teams rely on Velero as their primary Kubernetes backup/DR tool.

Example

# Scheduled daily backup of one namespace, including volume data, kept 30 days:
velero schedule create web-daily \
  --schedule="0 2 * * *" --include-namespaces web --ttl 720h --snapshot-volumes

velero backup create web-now --include-namespaces web       # on-demand backup
velero restore create --from-backup web-now                 # restore (same or new cluster)

Exercises

1. (Beginner) What two categories of things does Velero back up?
2. (Beginner) Where does Velero store its backups?
3. (Intermediate) How does Velero differ from an etcd snapshot in granularity and portability?
4. (Interview) How does Velero enable cross-cluster migration and application-level DR that etcd backups cannot? (Hint: selective, portable, object-storage-based, includes PV data, restore to any cluster.)

Answers

1. Kubernetes API objects (selectable by namespace/label/resource) and the persistent volume data backing them (via CSI snapshots or file-level backup like Restic/Kopia).
2. In object storage (e.g., AWS S3, Google Cloud Storage, Azure Blob), which makes backups durable and portable.
3. An etcd snapshot is coarse and all-or-nothing — the entire cluster's state, restorable only to a matching self-managed control plane. Velero is granular (back up/restore individual namespaces/apps or filtered resources), includes PV data, works on managed clusters, and is portable (stored in object storage, restorable to a different cluster).
4. Velero stores portable backups (objects + PV data) in cloud object storage independent of any single cluster, so you can restore into a different cluster — enabling migration between clusters/clouds and rebuilding after a cluster is lost. It's selective (per-namespace/app) and includes the persistent data, so you can recover specific applications with their state. An etcd snapshot only reconstructs the same self-managed cluster's object state (not PV contents) and can't selectively restore apps or move them to another cluster — so Velero covers the application-level, data-inclusive, portable DR/migration scenarios etcd backups cannot.

---

PV snapshot backup strategies

Theory

For stateful applications, backing up the persistent data is as important as the cluster objects. PersistentVolume snapshot strategies protect that data, using the CSI VolumeSnapshot mechanism (Chapter 7) — point-in-time copies of a volume, often orchestrated by Velero or the storage system.

Key strategy considerations:

• Application consistency: a raw volume snapshot is only crash-consistent (like a power-loss image). For databases, you should quiesce/flush the app (or use its backup mode / a pre-snapshot hook) so the snapshot is application-consistent — otherwise restore may require recovery or yield inconsistent data. Velero supports backup hooks (run a command in the Pod, e.g., fsfreeze or DB flush, before/after snapshot).
• Frequency & retention: driven by RPO; keep a rotation of snapshots (and expire old ones).
• Off-site/durability: snapshots stored with the storage provider should be replicated/exported so a regional failure doesn't lose them.
• Test restores: verify you can actually restore data and that the app starts cleanly from it.

The overall approach is to combine volume snapshots (for data) with object backups (Velero, for the Kubernetes resources) — and to ensure snapshots are application-consistent for stateful systems. This is what makes stateful-workload DR trustworthy rather than "we have snapshots but can't reliably restore."

Example

# Velero backup hook: flush/quiesce a database before the volume snapshot
metadata:
  annotations:
    pre.hook.backup.velero.io/command: '["/bin/sh","-c","mysql -e \"FLUSH TABLES WITH READ LOCK\""]'
    post.hook.backup.velero.io/command: '["/bin/sh","-c","mysql -e \"UNLOCK TABLES\""]'

# Or a direct CSI VolumeSnapshot (point-in-time copy of a PVC):
# kind: VolumeSnapshot -> source: { persistentVolumeClaimName: data }  (see Ch.7)

Exercises

1. (Beginner) What mechanism is used to snapshot PersistentVolumes?
2. (Beginner) What is the difference between crash-consistent and application-consistent snapshots?
3. (Intermediate) How does Velero help achieve application-consistent snapshots for a database?
4. (Interview) What combination of backups gives trustworthy DR for a stateful application, and what step is essential to trust it? (Hint: volume snapshots + object backup, app-consistent; test restores.)

Answers

1. CSI VolumeSnapshots (point-in-time copies of a volume/PVC), often orchestrated by Velero or the storage system.
2. A crash-consistent snapshot captures the disk exactly as-is at an instant (like a power-loss image), possibly with in-flight writes/unflushed buffers, so restore may need recovery or be inconsistent. An application-consistent snapshot is taken with the app quiesced/flushed so on-disk data is in a consistent, clean state at snapshot time.
3. Velero supports backup hooks that run commands inside the Pod before and after the snapshot — e.g., flush and lock the database (or fsfreeze the filesystem) pre-snapshot and unlock/thaw post-snapshot — so the volume is captured in an application-consistent state.
4. Combine volume snapshots for the persistent data with object-level backups (e.g., Velero for the Kubernetes resources like PVCs, StatefulSets, Secrets), ensuring the volume snapshots are application-consistent (via quiesce/flush hooks) — and store them durably/off-site with sensible frequency/retention (RPO). The essential step to trust it is to test restores: regularly verify you can actually restore the data and bring the application back up cleanly, since an untested backup may be unrestorable or inconsistent.

---

Recovery time and recovery point objectives

Theory

Disaster recovery planning is quantified by two objectives that shape every backup/HA decision:

• RTO (Recovery Time Objective): the maximum acceptable time to restore service after an incident — "how long can we be down?" It drives choices about failover automation, standby infrastructure, and restore speed.
• RPO (Recovery Point Objective): the maximum acceptable amount of data loss measured in time — "how much recent data can we lose?" It drives backup/replication frequency (an RPO of 1 hour means backing up/replicating at least hourly).

These are business decisions with cost trade-offs: tighter objectives (near-zero RTO/RPO) require more expensive solutions (hot standbys, active-active, continuous replication), while looser objectives tolerate cheaper approaches (periodic backups, slower rebuilds). For Kubernetes: RPO informs how often you snapshot etcd/volumes and replicate; RTO informs whether you keep a warm/standby cluster (fast failover) vs. rebuilding from backups (slower). You design the backup cadence, HA topology (active-active vs. active-passive, Chapter 15), and runbooks to meet the RTO/RPO your business requires — and test that you actually can. Stating RTO/RPO turns vague "we have backups" into a measurable, verifiable DR posture.

Example

RTO = how long to recover (downtime tolerance)
RPO = how much data loss is acceptable (time)

Example targets & implications:
  RPO 5 min  -> replicate/snapshot at least every 5 min (continuous replication)
  RTO 15 min -> need warm standby + automated failover (not "rebuild from scratch")
  RPO 24h / RTO 8h -> nightly backups + documented manual rebuild is acceptable

Exercises

1. (Beginner) What does RTO measure, and what does RPO measure?
2. (Beginner) Which objective drives how frequently you back up or replicate?
3. (Intermediate) If your RPO is 5 minutes, what does that imply about backup/replication frequency?
4. (Interview) How do RTO/RPO targets translate into Kubernetes DR architecture choices, and why are they business (not just technical) decisions? (Hint: standby cluster vs. rebuild; replication cadence; cost trade-offs.)

Answers

1. RTO measures the maximum acceptable time to restore service after an incident (downtime tolerance); RPO measures the maximum acceptable amount of data loss, expressed as a time window.
2. RPO (recovery point objective) — it dictates how frequently you must back up or replicate data.
3. You must capture/replicate state at least every 5 minutes (e.g., continuous or near-continuous replication / very frequent snapshots), because anything older than 5 minutes could be lost in a disaster and would exceed the RPO.
4. Tighter targets demand more capable (and costly) architecture: a low RTO pushes toward warm/hot standby clusters and automated failover (active-active or active-passive with fast promotion) rather than rebuilding from backups; a low RPO pushes toward frequent/continuous replication and snapshots rather than nightly backups. Looser targets allow cheaper periodic backups and manual rebuild runbooks. They're business decisions because they trade cost against acceptable downtime and data loss — the organization must decide how much it's worth to reduce potential downtime/loss, and those decisions then dictate the technical DR design (topology, backup cadence, automation) and the runbooks/tests needed to actually meet them.

---

16.3 Cluster Cost Optimization

Kubernetes makes it easy to over-provision and overspend. This subchapter covers keeping cluster costs under control.

---

Right-sizing workloads

Theory

The most common source of Kubernetes waste is over-requested resources: teams set CPU/memory requests far higher than actual usage "to be safe," so the scheduler reserves capacity that sits idle — you pay for nodes to hold reservations nothing uses. Right-sizing means aligning requests (and limits) with actual observed usage (plus a sensible safety headroom), reclaiming the wasted capacity.

The approach: measure real consumption (Metrics Server for current, Prometheus for historical percentiles like p95/p99), then set requests near typical usage and limits to accommodate peaks (with the CPU/memory caveats from Chapter 8). Tools help: the Vertical Pod Autoscaler (in recommendation mode) suggests right-sized requests from history; cost tools (Kubecost, Goldilocks) surface over/under-provisioned workloads. Right-sizing directly increases bin-packing density (more Pods per node → fewer nodes → lower cost) while avoiding the opposite failure (under-requesting → contention, throttling, OOMs). It's usually the highest-ROI cost optimization because over-provisioning is so pervasive, and it compounds with autoscaling (right-sized requests make Cluster Autoscaler and HPA far more efficient).

Example

Before (over-requested):          After right-sizing (requests ~ real usage):
  requests: cpu 1000m, mem 2Gi      requests: cpu 250m, mem 512Mi
  actual usage: ~200m / ~400Mi      -> 3-4x more Pods fit per node
  => nodes mostly idle, overpaying  => fewer nodes needed, lower cost

kubectl top pods -A                    # observe actual usage vs. requests
# Use VPA recommendations / Goldilocks to derive right-sized requests.

Exercises

1. (Beginner) What is the most common source of Kubernetes resource waste?
2. (Beginner) What does "right-sizing" a workload mean?
3. (Intermediate) What tools help you determine appropriate resource requests?
4. (Interview) Why is right-sizing usually the highest-ROI cost optimization, and what's the risk of over-correcting? (Hint: over-provisioning is pervasive; under-requesting causes contention/OOM.)

Answers

1. Over-requesting resources — setting CPU/memory requests much higher than actual usage, so reserved-but-idle capacity forces you to run (and pay for) more nodes.
2. Aligning a workload's resource requests (and limits) with its actual observed usage plus reasonable headroom, instead of over-reserving.
3. Metrics Server (current usage via kubectl top), Prometheus/Grafana (historical percentiles), the Vertical Pod Autoscaler in recommendation mode, and cost/analysis tools like Kubecost and Goldilocks.
4. Over-provisioning is extremely common (teams pad requests "to be safe"), so reclaiming that idle reserved capacity across many workloads yields large savings with relatively little effort — and it improves bin-packing so fewer nodes are needed, compounding with autoscaling. The risk of over-correcting (setting requests too low) is resource contention, CPU throttling, and OOMKills under load — degrading reliability. So right-sizing must target real usage plus adequate headroom (e.g., based on p95/p99), not the bare minimum.

---

Spot and preemptible nodes

Theory

Cloud providers sell spare capacity at steep discounts (up to ~70–90% off) as spot instances (AWS/Azure) or preemptible VMs (GCP) — with the catch that the provider can reclaim them at short notice (interruption). Running suitable workloads on spot/preemptible nodes is one of the biggest infrastructure cost levers in Kubernetes.

The key is matching workloads to interruption tolerance:

• Good fits: stateless, replicated, fault-tolerant, or restartable workloads — batch/ML jobs, CI runners, stateless services with enough replicas and PDBs to survive node loss.
• Poor fits: singleton stateful workloads, workloads that can't tolerate abrupt termination, or anything requiring guaranteed availability on a single node.

Practical patterns: use separate node pools (spot + on-demand), taint spot nodes so only opted-in (toleration-bearing) workloads run there, spread replicas across nodes/zones (topology spread) so a spot reclamation doesn't take out a whole service, handle termination notices to drain gracefully, and keep a baseline of on-demand nodes for critical/stateful components. Tools like Karpenter and Cluster Autoscaler can mix spot/on-demand intelligently. Done right, spot/preemptible nodes cut compute cost dramatically for the large share of workloads that tolerate interruption.

Example

Cost lever: run interruptible workloads on cheap spot/preemptible nodes

  [on-demand pool]  <- critical/stateful (guaranteed)
  [spot pool (tainted)] <- batch/ML, CI, replicated stateless (tolerate interruption)
     spread replicas across nodes/zones; handle termination notice -> drain

  Savings: up to ~70-90% on the spot portion of compute

Exercises

1. (Beginner) Why are spot/preemptible nodes cheaper, and what's the catch?
2. (Beginner) Name a workload type well-suited to spot nodes and one that isn't.
3. (Intermediate) How do taints/tolerations help you use spot nodes safely?
4. (Interview) What practices make running on spot/preemptible nodes safe despite interruptions? (Hint: fault-tolerant workloads, replicas + PDB + topology spread, termination handling, on-demand baseline.)

Answers

1. They're spare cloud capacity sold at a large discount (up to ~70–90% off), but the provider can reclaim/terminate them at short notice (interruption), so they're not guaranteed to keep running.
2. Well-suited: stateless/replicated/fault-tolerant or restartable workloads (batch/ML jobs, CI runners, stateless services with enough replicas). Not suited: singleton stateful workloads or anything needing guaranteed uninterrupted availability on a single node.
3. Taint the spot nodes so ordinary workloads won't schedule there, and add matching tolerations only to workloads that can tolerate interruption — ensuring critical/stateful workloads stay on on-demand nodes while only opted-in, interruption-tolerant workloads run on spot capacity.
4. Run only interruption-tolerant workloads on spot; ensure enough replicas with PodDisruptionBudgets and topology spread across nodes/zones so a reclamation doesn't take down a service; handle termination notices (the short warning) to drain/reschedule gracefully; use taints/tolerations and separate node pools to keep critical/stateful workloads on on-demand nodes; maintain an on-demand baseline for guaranteed capacity; and let tools (Karpenter/Cluster Autoscaler) mix spot/on-demand and diversify instance types to reduce simultaneous-reclamation risk. These practices absorb interruptions so the cost savings don't compromise availability.

---

Cluster Autoscaler tuning

Theory

The Cluster Autoscaler (Chapter 8) saves money by removing underused nodes and adding nodes only when needed — but its default behavior may be too conservative or too aggressive for optimal cost, so tuning matters. The goal is to keep utilization high (few idle nodes) without harming availability or causing thrash.

Key tuning levers:

• Scale-down parameters: scale-down-utilization-threshold (how empty a node must be before removal) and scale-down-unneeded-time (how long it must stay underused) — lower/shorter values reclaim nodes faster (more savings) but risk removing capacity that's soon needed (thrash) and disrupting workloads.
• Node group configuration: right-size and diversify node groups; mix instance types/spot.
• Respecting workloads: PDBs, safe-to-evict annotations, and avoiding nodes with un-movable Pods (local storage) affect what can be scaled down — misconfiguration leaves nodes stuck.
• Expander strategy: how CA chooses which node group to scale up (e.g., least-waste, priority) to pick cost-effective nodes.

There's also the newer Karpenter, which replaces node-group-centric autoscaling with just-in-time, right-sized node provisioning (picking optimal instance types/sizes per pending Pods), often achieving better bin-packing and cost than a tuned Cluster Autoscaler. Tuning autoscaling is about balancing cost (aggressive scale-down, tight packing) against stability and responsiveness (avoiding thrash and capacity shortfalls).

Example

Cluster Autoscaler tuning trade-off:
  aggressive: low scale-down-utilization-threshold, short scale-down-unneeded-time
     -> more idle nodes removed (cheaper) BUT risk of thrash / capacity gaps
  conservative: opposite -> stable but more idle capacity (costlier)

  expander: least-waste | priority | least-cost -> pick cheapest suitable node group
  Karpenter: provisions right-sized nodes just-in-time per pending Pods (better packing)

Exercises

1. (Beginner) What two actions does the Cluster Autoscaler take to control cost?
2. (Beginner) Name a parameter that controls how aggressively nodes are scaled down.
3. (Intermediate) What risk comes from making scale-down too aggressive?
4. (Interview) How does Karpenter differ from the traditional Cluster Autoscaler, and why can it improve cost/packing? (Hint: node-group scaling vs. just-in-time right-sized node provisioning.)

Answers

1. It adds nodes when there are unschedulable Pods (scale up) and removes underused nodes whose Pods can be rescheduled (scale down), keeping capacity matched to demand.
2. scale-down-utilization-threshold (how empty a node must be to be considered for removal) or scale-down-unneeded-time (how long it must remain underused before removal). (Either is acceptable.)
3. Thrash and capacity shortfalls: removing nodes too eagerly can delete capacity that's needed again moments later (causing repeated add/remove churn and scheduling delays) and can disrupt workloads (evictions), hurting stability and responsiveness even though it maximizes short-term savings.
4. The traditional Cluster Autoscaler scales predefined node groups (fixed instance types) up/down by count. Karpenter instead provisions nodes just-in-time, choosing the optimal instance type and size for the current set of pending Pods (and consolidating/replacing nodes for better packing), rather than being constrained to preconfigured groups. This lets it pick the most cost-effective, best-fitting capacity and bin-pack more tightly, often yielding better utilization and lower cost than a node-group-based autoscaler, with faster and more flexible provisioning.

---

Kubecost and cost visibility tools

Theory

You can't optimize what you can't measure, and Kubernetes makes cost opaque: a shared cluster's bill doesn't tell you which team, namespace, or workload drove the spend. Cost visibility tools — chiefly Kubecost (and OpenCost, its CNCF open-source core, plus cloud-native cost tools) — solve this by allocating cluster cost down to namespaces, workloads, labels, and teams.

What they provide:

• Cost allocation/showback/chargeback: break down spend by namespace/label/team (based on each workload's resource requests/usage and the underlying node/cloud prices), enabling accountability.
• Efficiency insights: identify over-provisioned/idle resources and right-sizing opportunities (unused requests, idle nodes), quantifying waste.
• Recommendations: suggest request right-sizing, spot usage, and autoscaling improvements.
• Budgets/alerts: notify when spend or a namespace exceeds thresholds, and forecast costs.

The value is turning an opaque aggregate bill into actionable, attributable data: teams see their own cost, waste is quantified, and optimization efforts (right-sizing, spot, autoscaler tuning) can be prioritized by impact and verified. Cost visibility is the feedback loop that makes all the other cost optimizations measurable and sustained rather than one-off.

Example

# Kubecost/OpenCost surface cost per namespace/workload/label:
# namespace       cpu-cost  mem-cost  total/mo   efficiency
# team-a          $420      $180      $600       38%  <- over-provisioned
# team-b          $150      $90       $240       81%
# plus: right-sizing recommendations, idle-cost, budgets/alerts

Exercises

1. (Beginner) What problem do Kubernetes cost visibility tools solve?
2. (Beginner) What is the open-source CNCF core underlying Kubecost?
3. (Intermediate) What does "cost allocation" (showback/chargeback) enable?
4. (Interview) Why is cost visibility the feedback loop that makes other optimizations (right-sizing, spot, autoscaler tuning) effective and sustained? (Hint: attribute/quantify waste, prioritize by impact, verify results, accountability.)

Answers

1. Cost opacity — a shared cluster's cloud bill doesn't reveal which namespace/workload/team drove the spend; these tools allocate and attribute cost so it becomes visible.
2. OpenCost (the CNCF open-source cost-allocation core that Kubecost builds on).
3. Breaking down and attributing cluster cost to specific namespaces, labels, teams, or workloads — enabling showback (showing each team its cost) and chargeback (billing teams for their usage), which drives accountability and informed decisions.
4. Optimizations only stick if you can see their target and their effect. Cost visibility quantifies and attributes waste (which workloads over-request, which nodes sit idle, which namespaces cost most), so teams can prioritize the highest-impact fixes (right-sizing the worst offenders, moving suitable workloads to spot, tuning the autoscaler), and then verify that changes actually reduced cost. It also creates accountability (teams own their spend) and can alert on regressions. Without this measurement/feedback loop, optimizations are guesswork and savings erode over time; with it, cost management becomes data-driven, targeted, and continuously maintained.

---

16.4 Troubleshooting

Things break; effective troubleshooting is a core operational skill. This subchapter covers diagnosing the most common Kubernetes failures.

---

Debugging pod failures and CrashLoopBackOff

Theory

The most common Pod problem is a container that keeps crashing, shown as CrashLoopBackOff — the container starts, exits/crashes, Kubernetes restarts it, it crashes again, and the kubelet applies an exponential backoff (increasing delay between restarts) to avoid hammering. CrashLoopBackOff is a symptom (the container won't stay up), not a root cause — your job is to find why it's crashing.

The systematic debugging workflow:

1. kubectl describe pod <name>: read the container state/last state (exit code, reason like Error/OOMKilled) and the Events (image pull errors, probe failures, scheduling issues).
2. kubectl logs <name> --previous: the crashed container's own logs are usually the most direct clue (stack trace, config error, missing dependency).
3. Check the exit code: e.g., 137 = OOMKilled/SIGKILL, 1/2 = application error, 126/127 = command not found/not executable.
4. Common causes: application bug/misconfiguration (bad env/ConfigMap/Secret), missing dependency (a DB not reachable at startup), failing liveness probe (restarting a healthy-but-slow app), insufficient memory (OOM), or a wrong command/entrypoint.
5. For images without a shell, use kubectl debug (ephemeral container) to inspect.

The discipline is describe → logs (--previous) → exit code → hypothesis → fix, treating CrashLoopBackOff as a pointer to dig deeper rather than the answer itself.

Example

kubectl get pod web            # STATUS: CrashLoopBackOff, RESTARTS climbing
kubectl describe pod web       # Events + Last State: Terminated, Reason: Error, Exit: 1
kubectl logs web --previous    # the crashed container's logs -> the real error
# e.g., "FATAL: config key DB_HOST not set"  -> fix the ConfigMap/env, redeploy

Exercises

1. (Beginner) What does CrashLoopBackOff mean, and is it a root cause?
2. (Beginner) Which command shows the logs of the crashed (previous) container?
3. (Intermediate) What does exit code 137 usually indicate?
4. (Interview) Walk through a systematic approach to diagnosing a CrashLoopBackOff. (Hint: describe → previous logs → exit code → common causes → ephemeral debug.)

Answers

1. It means the container repeatedly starts, crashes, and is restarted with increasing (exponential) backoff delays. It's a symptom, not a root cause — you still have to find why the container is exiting.
2. kubectl logs <pod> --previous.
3. The container was killed by SIGKILL (128 + 9) — most commonly OOMKilled (it exceeded its memory limit), though it can be any external SIGKILL.
4. Start with kubectl describe pod to read the container's last state/exit code and the Events (pull errors, probe failures, scheduling). Then kubectl logs --previous to see the crashed container's own output (usually the clearest clue). Interpret the exit code (137 = OOM, 1/2 = app error, 126/127 = bad command). Form a hypothesis from common causes — app misconfig (bad env/ConfigMap/Secret), missing/unreachable dependency at startup, an aggressive liveness probe restarting a slow app, insufficient memory, or a wrong entrypoint. If the image lacks a shell, use kubectl debug (ephemeral container) to inspect the environment/filesystem. Fix the identified cause and redeploy, then confirm the restarts stop.

---

Node Not Ready troubleshooting

Theory

A node in NotReady state can't run new workloads, and after a timeout its Pods get evicted/rescheduled — so diagnosing it quickly matters. NotReady means the node's kubelet has stopped reporting healthy status to the control plane (missed heartbeats) or is reporting a problem condition. The cause is on the node or its connectivity, so you investigate at the node level.

The diagnostic path:

1. kubectl describe node <name>: check the node Conditions — Ready, and pressure conditions like MemoryPressure, DiskPressure, PIDPressure — and Events. These often name the problem (e.g., disk full).
2. Common causes: the kubelet is down/crashed or misconfigured; network issues preventing the kubelet from reaching the API server; resource exhaustion on the node (disk full → DiskPressure, out of memory); container runtime (containerd/CRI-O) down; CNI problems; certificate/expiry issues; or the node/VM itself failed.
3. On the node (SSH): check systemctl status kubelet and its logs (journalctl -u kubelet), the runtime status, disk (df -h) and memory, and connectivity to the API server.
4. Remediate: restart the kubelet/runtime, free disk space, fix networking, or (in cloud/autoscaled setups) let the node be replaced.

The mental model: NotReady = "the control plane can't confirm this node is healthy," so trace kubelet → runtime → node resources → connectivity to find and fix the break.

Example

kubectl get nodes                       # node-3  NotReady
kubectl describe node node-3            # Conditions: DiskPressure=True, or Ready=Unknown
# On the node:
systemctl status kubelet                # is the kubelet running?
journalctl -u kubelet -n 100            # kubelet errors (runtime? certs? network?)
df -h                                    # disk full -> DiskPressure

Exercises

1. (Beginner) What does a NotReady node status fundamentally indicate?
2. (Beginner) Where do you look first to see node conditions and events?
3. (Intermediate) List three common causes of a node going NotReady.
4. (Interview) Describe the chain you'd investigate (from control plane to node internals) to diagnose a NotReady node. (Hint: describe node conditions → kubelet → runtime → resources → connectivity.)

Answers

1. That the node's kubelet is no longer reporting a healthy Ready status to the control plane (missed heartbeats or a failing condition) — the control plane can't confirm the node is healthy, so it won't schedule new Pods there.
2. kubectl describe node <name> — its Conditions (Ready, MemoryPressure, DiskPressure, PIDPressure) and Events.
3. Any three: kubelet down/crashed or misconfigured; network/connectivity failure to the API server; node resource exhaustion (disk full → DiskPressure, out of memory); container runtime (containerd/CRI-O) down; CNI/networking problems; certificate expiry; or the underlying node/VM failing.
4. Start at the control plane: kubectl describe node to read Conditions/Events (they often point at the cause, e.g., DiskPressure). Then go to the node itself: check the kubelet (systemctl status kubelet, journalctl -u kubelet) since NotReady usually means the kubelet isn't reporting healthy; check the container runtime status; check node resources (df -h for disk, memory) for exhaustion; and verify connectivity to the API server (network/DNS/certs). Follow the chain kubelet → runtime → resources → connectivity to isolate the break, then remediate (restart kubelet/runtime, free resources, fix networking) or replace the node.

---

Network connectivity debugging

Theory

Network problems ("Service X can't reach Service Y") are among the trickiest to debug because they span DNS, Services, endpoints, kube-proxy, CNI, and NetworkPolicies. A layered, systematic approach isolates the failing layer:

1. DNS: can the client resolve the target's name? Test from a debug Pod (nslookup <service>.<ns>). DNS failures point at CoreDNS or resolv.conf/NetworkPolicy blocking port 53.
2. Service & endpoints: does the Service exist and have endpoints? kubectl get svc,endpointslices — empty endpoints means the selector matches no ready Pods (label mismatch or Pods not ready).
3. Direct Pod connectivity: can you reach the target Pod IP directly (bypassing the Service)? If Pod-to-Pod works but Service doesn't, suspect kube-proxy/Service config; if Pod-to-Pod fails, suspect CNI/routing.
4. NetworkPolicy: is a policy blocking the traffic? Check for policies selecting either side (remember default-deny once selected, and egress policies breaking DNS). Cilium/Hubble can show dropped flows.
5. Ports/app: is the app actually listening on the expected targetPort?

Tools: a debug Pod (kubectl run -it --rm --image=nicolaka/netshoot or kubectl debug) to run curl, nslookup, nc, dig from inside the cluster network. The discipline is to test each layer in order (DNS → Service/endpoints → Pod IP → policy → app) to localize the fault rather than guessing.

Example

# From a debug Pod inside the cluster network (netshoot):
kubectl run net --rm -it --image=nicolaka/netshoot -- bash
  nslookup payments.default                 # 1) DNS resolves?
  curl -v http://payments.default:80        # 2/3) Service reachable?
  curl -v http://10.244.2.7:8080            # 3) direct Pod IP reachable?
kubectl get svc,endpointslices -n default   # 2) does the Service have endpoints?
kubectl get networkpolicy -A                # 4) any policy blocking traffic?

Exercises

1. (Beginner) What's the first layer to test when a service can't reach another (name-based)?
2. (Beginner) What does an empty EndpointSlice for a Service usually indicate?
3. (Intermediate) If Pod-to-Pod (direct IP) works but Service access fails, where do you look?
4. (Interview) Describe the layered order you'd use to isolate a cross-service connectivity failure, and why order matters. (Hint: DNS → Service/endpoints → Pod IP → NetworkPolicy → app port.)

Answers

1. DNS resolution — verify the client can resolve the target's Service name (e.g., nslookup <service>.<namespace> from a debug Pod).
2. That the Service's selector matches no ready Pods — typically a label mismatch between the Service selector and the Pods, or the backing Pods aren't Ready (failing readiness), so there are no endpoints to route to.
3. At the Service layer/kube-proxy and Service configuration: since the Pods are reachable directly, the problem is in how the Service maps to them — check the Service's endpoints/selector, port/targetPort, and kube-proxy health/rules (iptables/IPVS) on the relevant nodes.
4. Test in layers: (1) DNS (can the name resolve?), (2) Service & endpoints (does the Service exist with ready endpoints?), (3) direct Pod IP (is the target reachable bypassing the Service? — isolates CNI/routing vs. Service issues), (4) NetworkPolicy (is traffic being dropped by policy, including egress/DNS blocks?), and (5) app/port (is the app listening on the expected port?). Order matters because each layer builds on the previous: name resolution must work before Service routing, Service routing depends on endpoints, and confirming direct Pod connectivity distinguishes a network/CNI fault from a Service/kube-proxy fault. Testing sequentially localizes exactly where the path breaks instead of guessing across many possible causes.

---

OOMKilled and resource exhaustion

Theory

OOMKilled (exit code 137) means a container exceeded its memory limit and the kernel's OOM killer terminated it (memory is incompressible, Chapter 8 — you can't throttle it, so the process is killed). It manifests as restarts/CrashLoopBackOff with the last state showing OOMKilled. This is a resource-exhaustion failure, and there are two levels to consider:

• Container-level OOM: the container hit its own memory limit. Fix by raising the limit to fit actual usage (observe with kubectl top/Prometheus), reducing the app's memory use (fix leaks, tune heap/cache — using the Downward API to size to the limit), or right-sizing.
• Node-level memory pressure: the node runs low on memory, triggering MemoryPressure and kubelet eviction of Pods (by QoS order: BestEffort first, then Burstable exceeding requests, Guaranteed last, Chapter 8). Fix by right-sizing requests/limits (so scheduling reflects real usage), spreading load, adding capacity, and giving critical Pods Guaranteed QoS.

Diagnosis: kubectl describe pod (last state OOMKilled, exit 137), check the container's limit vs. its actual usage trend, and look for node MemoryPressure and eviction events. The distinction matters: container OOM = that container's limit is too low (or it leaks); node pressure = the node is oversubscribed (usually from under-set requests). Both point back to correct resource requests/limits and right-sizing as the durable fix.

Example

kubectl describe pod web
#   Last State: Terminated  Reason: OOMKilled  Exit Code: 137
#   Limits: memory: 256Mi     <- container hit its own limit

kubectl top pod web                    # actual memory usage vs. the 256Mi limit
kubectl get events --field-selector reason=Evicted     # node-level memory eviction?
kubectl describe node node-3 | grep MemoryPressure     # node oversubscribed?

Exercises

1. (Beginner) What does OOMKilled mean, and what exit code accompanies it?
2. (Beginner) Why can't Kubernetes just "throttle" memory like it does CPU?
3. (Intermediate) Distinguish a container-level OOM kill from node-level memory-pressure eviction.
4. (Interview) What is the durable fix for recurring OOMKills and memory-pressure evictions, and how does QoS factor in? (Hint: right-size requests/limits; Guaranteed QoS for critical workloads.)

Answers

1. The container exceeded its memory limit and was killed by the kernel's OOM killer; the exit code is 137 (128 + SIGKILL 9).
2. Memory is incompressible — you can't reclaim in-use memory from a running process gracefully the way you can withhold CPU cycles. So when a container exceeds its memory limit, the only enforcement is to kill it (OOM), not slow it down.
3. Container-level OOM: the container hit its own memory limit and was killed (its limit is too low or the app leaks/uses too much). Node-level memory-pressure eviction: the node ran low on memory, so the kubelet evicted Pods (in QoS order) to reclaim memory — a node-oversubscription problem, typically from requests set below real usage.
4. Right-size requests and limits to reflect actual memory usage (observed via metrics), and fix the app's consumption (leaks, oversized heaps/caches — e.g., size to the limit via the Downward API). Correct requests make the scheduler stop oversubscribing nodes (preventing node pressure), and adequate limits stop premature container OOMs. QoS matters because under node memory pressure the kubelet evicts BestEffort first, then Burstable exceeding requests, and protects Guaranteed last — so setting critical workloads to Guaranteed (requests == limits) ensures they're the least likely to be evicted, while proper requests/limits across the board reduce the pressure in the first place.

---

etcd health and performance

Theory

Because etcd is the cluster's brain (Chapter 2), etcd problems degrade or break the entire cluster — a slow or unhealthy etcd makes the API server slow or unavailable, so everything (scheduling, controllers, kubectl) suffers. etcd health/performance is therefore a critical operational concern, especially at scale.

What to watch and why:

• etcd is extremely disk-latency-sensitive: it fsyncs writes to disk for consistency, so slow disks are the #1 cause of etcd problems. Fast SSDs/NVMe and low fsync/WAL sync and backend commit latency are essential; high disk latency shows up as etcd_disk_wal_fsync_duration_seconds spikes and leader elections/warnings.
• Quorum/health: use etcdctl endpoint health/status to check members, the leader, and DB size; a member down or repeated leader elections signal trouble (network or disk).
• Database size: etcd has a space quota (default ~2–8 GB); exceeding it puts etcd into a read-only alarm state (NOSPACE). Mitigate with compaction (of old revisions) and defragmentation to reclaim space, and keep object counts/churn reasonable (e.g., don't store huge/many objects, high event churn).
• Latency between members: etcd needs low-latency networking among members (avoid spreading across high-latency links).

Diagnosis combines etcd metrics (via Prometheus), etcdctl health/status, and control-plane symptoms (slow API responses). The durable fixes: fast dedicated disks, adequate resources, keeping the DB compacted/defragmented and under quota, low-latency member networking, and reliable backups (16.2). Treat etcd as the highest-sensitivity component: keep it fast, healthy, and backed up.

Example

ETCDCTL_API=3 etcdctl endpoint status --write-out=table --endpoints=... 
# shows each member's DB SIZE, LEADER, RAFT TERM  (watch DB size vs. quota)
ETCDCTL_API=3 etcdctl endpoint health --endpoints=...   # member health

# Key Prometheus metric (disk sensitivity):
#   etcd_disk_wal_fsync_duration_seconds  -> high = slow disk = etcd problems
# Reclaim space: compact old revisions, then defragment
etcdctl compact <rev>; etcdctl defrag --endpoints=...

Exercises

1. (Beginner) Why does an unhealthy etcd affect the whole cluster?
2. (Beginner) What hardware characteristic is etcd most sensitive to?
3. (Intermediate) What happens when etcd exceeds its storage quota, and how do you reclaim space?
4. (Interview) What are the key measures to keep etcd healthy and performant at scale? (Hint: fast dedicated disks, low fsync latency, compaction/defrag under quota, low-latency members, backups.)

Answers

1. Because etcd is the sole datastore behind the API server; if etcd is slow or unavailable, the API server becomes slow/unavailable, and since every component works through the API server, scheduling, controllers, and kubectl all degrade or fail — the whole cluster is affected.
2. Disk latency — etcd fsyncs writes to disk for consistency, so slow disks are the primary cause of etcd performance problems; it needs fast, low-latency storage (SSD/NVMe).
3. Exceeding the storage quota triggers a NOSPACE alarm that puts etcd into a read-only/maintenance state (rejecting writes), which effectively breaks cluster changes. You reclaim space by compacting old key revisions and then defragmenting the database to release freed space (and then clear the alarm), plus reduce what's stored/churned.
4. Give etcd fast, dedicated, low-latency disks (SSD/NVMe) and monitor fsync/WAL and backend-commit latency; provision adequate CPU/memory; keep the database under its quota via regular compaction and defragmentation and by limiting object count/size and event churn; ensure low-latency networking between etcd members (co-located/close, odd-numbered for quorum); watch health/leader stability (avoid frequent leader elections); and maintain reliable, tested backups (snapshots) for recovery. Together these keep etcd fast, stable, and recoverable, which keeps the whole control plane healthy.

---