8. Scheduling and Resource Management

How does Kubernetes decide where each Pod runs, and how does it prevent one workload from starving another? This chapter covers the economics and placement of workloads: declaring resource needs, the scheduler's decision process, steering Pods toward or away from nodes, enforcing fairness with quotas, and scaling automatically. Getting this right is the difference between a stable, cost-efficient cluster and one that thrashes or falls over.

---

8.1 Resource Requests and Limits

Requests and limits tell Kubernetes how much CPU and memory a container needs and may use. They drive scheduling, fairness, and stability. This subchapter covers them and the QoS classes they produce.

---

CPU and memory requests

Theory

A request is the amount of CPU or memory a container is guaranteed. It is the primary input to scheduling: the scheduler sums the requests of all Pods on a node and only places a new Pod where the node's allocatable capacity can satisfy the new Pod's requests. Requests are a reservation — that capacity is set aside for the container whether or not it uses it all.

Units: CPU is measured in cores, often as millicores (m): 500m = half a core, 1 = one full core. Memory is in bytes, usually with suffixes: Mi (mebibytes), Gi (gibibytes). Setting requests well is crucial: too low and Pods get over-packed onto nodes and contend; too high and you waste capacity (low utilization, higher cost). Requests are about scheduling and guarantees, distinct from limits (which cap usage).

Example

spec:
  containers:
  - name: app
    image: myapp:1.0
    resources:
      requests:
        cpu: "250m"        # guaranteed 0.25 core; used for scheduling
        memory: "256Mi"    # guaranteed 256 MiB

Node allocatable: 4 CPU, 8Gi
Pods' requests:  A=1CPU/2Gi  B=1CPU/2Gi  C=1.5CPU/3Gi  (sum 3.5CPU/7Gi)
A new Pod requesting 1CPU fits (0.5 CPU left? no -> only 0.5 left, won't fit)

Exercises

1. (Beginner) What does a resource request guarantee, and what scheduling decision does it drive?
2. (Beginner) What does 500m CPU mean?
3. (Intermediate) A node has 4 allocatable CPUs. Pods with requests totaling 3.5 CPU are running. Can a Pod requesting 1 CPU be scheduled there? Why?
4. (Interview) What are the consequences of setting requests too low versus too high across a cluster? (Hint: contention/overpacking vs. wasted capacity/cost.)

Answers

1. It guarantees (reserves) that amount of CPU/memory for the container. It drives scheduling: the scheduler only places a Pod on a node whose remaining allocatable capacity can satisfy the Pod's requests.
2. Half of one CPU core (500 millicores = 0.5 core).
3. No. Only 0.5 CPU of requests remains (4 − 3.5), which is less than the 1 CPU requested, so the scheduler won't place it there (it would go Pending or to another node). Scheduling is based on summed requests vs. allocatable, regardless of actual current usage.
4. Too low: the scheduler over-packs nodes (it thinks Pods need little), leading to CPU contention/throttling and memory pressure/OOM as actual usage exceeds reservations. Too high: capacity is reserved but unused, so nodes appear full while idle — low utilization, fewer Pods per node, more nodes needed, and higher cost. Right-sizing requests balances density against stability.

---

CPU and memory limits

Theory

A limit is the maximum amount of CPU or memory a container may use. Limits cap consumption to protect other workloads and the node. Critically, CPU and memory limits behave very differently when exceeded:

• CPU is compressible: exceeding the CPU limit causes the container to be throttled (slowed down) — it's denied extra CPU cycles but keeps running. No crash.
• Memory is incompressible: exceeding the memory limit causes the container to be OOMKilled (terminated by the kernel's OOM killer) and restarted. You cannot "throttle" memory.

This asymmetry is one of the most important operational facts in Kubernetes. It means an under-provisioned memory limit leads to crash loops (OOMKilled), while an under-provisioned CPU limit leads to sluggishness. Many practitioners recommend setting memory requests = limits (to avoid surprise OOMs) and being cautious or even omitting CPU limits (to avoid unnecessary throttling), depending on workload.

Example

spec:
  containers:
  - name: app
    image: myapp:1.0
    resources:
      requests: { cpu: "250m", memory: "256Mi" }
      limits:   { cpu: "500m", memory: "512Mi" }   # caps; memory overflow = OOMKill

kubectl get pod app -o jsonpath='{.status.containerStatuses[0].lastState}'
# {"terminated":{"reason":"OOMKilled","exitCode":137}}   # exceeded memory limit

Exercises

1. (Beginner) What does a resource limit define?
2. (Beginner) What happens when a container exceeds its CPU limit versus its memory limit?
3. (Intermediate) A container repeatedly shows OOMKilled with exit code 137. What does this indicate and what's the likely fix?
4. (Interview) Explain the "compressible vs. incompressible" distinction between CPU and memory and how it should influence how you set limits. (Hint: throttle vs. kill; memory requests=limits.)

Answers

1. The maximum CPU/memory a container is allowed to consume (an upper cap).
2. Exceeding the CPU limit throttles the container (it's slowed but keeps running); exceeding the memory limit causes it to be OOMKilled (terminated, then restarted).
3. It indicates the container exceeded its memory limit and was killed by the OOM killer (137 = 128 + SIGKILL/9). The fix is to raise the memory limit/request to fit the app's actual usage, and/or reduce the app's memory consumption (fix leaks, tune heap/cache sizes, e.g., via the Downward API).
4. CPU is compressible: the kernel can simply give a container fewer cycles, so going over the limit throttles it without killing it. Memory is incompressible: you can't take memory back from a running process gracefully, so exceeding the limit triggers an OOM kill. This means memory limits are safety-critical — under-setting them causes crashes — so a common practice is to set memory requests equal to limits for predictable behavior, while treating CPU limits more loosely (or omitting them) to avoid needless throttling, since overshoot only slows the app.

---

Quality of Service (QoS) classes

Theory

Based on how a Pod sets requests and limits, Kubernetes assigns it a Quality of Service (QoS) class, which determines its priority when the node runs out of memory and must evict Pods. The three classes:

QoS Class	Condition	Eviction priority
Guaranteed	Every container has requests == limits for both CPU and memory	Evicted last (highest protection)
Burstable	At least one container has a request, but not Guaranteed	Evicted after BestEffort
BestEffort	No requests or limits set on any container	Evicted first (least protection)

When a node is under memory pressure, the kubelet evicts BestEffort Pods first, then Burstable Pods exceeding their requests, and protects Guaranteed Pods the longest. Thus setting requests/limits isn't just about scheduling — it directly affects which workloads survive resource pressure. Critical workloads should be Guaranteed; sacrificial/batch workloads can be BestEffort.

Example

# Guaranteed: requests == limits for cpu AND memory on every container
resources:
  requests: { cpu: "500m", memory: "512Mi" }
  limits:   { cpu: "500m", memory: "512Mi" }
---

# Burstable: has requests, but limits differ / partial
resources:
  requests: { cpu: "250m", memory: "256Mi" }
  limits:   { cpu: "1",    memory: "512Mi" }
---

# BestEffort: no requests or limits at all  -> evicted first
resources: {}

kubectl get pod app -o jsonpath='{.status.qosClass}'   # Guaranteed | Burstable | BestEffort

Exercises

1. (Beginner) What are the three QoS classes?
2. (Beginner) What condition makes a Pod "Guaranteed"?
3. (Intermediate) Under node memory pressure, in what order are Pods evicted by QoS class?
4. (Interview) You have a critical payment service and a best-effort batch job on the same nodes. How should you set resources so the payment service survives memory pressure? (Hint: Guaranteed vs. BestEffort.)

Answers

1. Guaranteed, Burstable, and BestEffort.
2. Every container in the Pod sets requests equal to limits for both CPU and memory (and both are specified).
3. BestEffort first, then Burstable (especially those exceeding their requests), and Guaranteed Pods last (most protected).
4. Make the payment service Guaranteed by setting requests == limits for CPU and memory on all its containers, so it's evicted last and has reserved capacity. Leave the batch job BestEffort (no requests/limits) or low-priority Burstable, so it's sacrificed first under pressure. This ensures the node reclaims memory from the expendable batch job before touching the critical service.

---

LimitRange for namespace defaults

Theory

If developers forget to set requests/limits, Pods become BestEffort and risk over-packing and eviction. A LimitRange is a namespace-scoped policy that automatically applies default requests/limits to containers that don't specify them, and can enforce min/max bounds per container/Pod and default ratios. It runs as an admission controller on Pod creation.

This gives platform teams guardrails: every Pod in a namespace gets sane resource values even if the manifest omits them, and no one can request absurdly large or tiny amounts. Capabilities include default (limits applied if omitted), defaultRequest (requests applied if omitted), max/min (allowed range), and maxLimitRequestRatio (cap how much limits can exceed requests). LimitRange operates per-object; ResourceQuota (next) operates on namespace aggregates.

Example

apiVersion: v1
kind: LimitRange
metadata: { name: defaults, namespace: dev }
spec:
  limits:
  - type: Container
    default:                       # applied as limits if not set
      cpu: "500m"
      memory: "512Mi"
    defaultRequest:                # applied as requests if not set
      cpu: "100m"
      memory: "128Mi"
    max: { cpu: "2", memory: "2Gi" }      # nobody may exceed this
    min: { cpu: "50m", memory: "64Mi" }   # nobody may go below this

Exercises

1. (Beginner) What does a LimitRange do for containers that omit requests/limits?
2. (Beginner) Is a LimitRange scoped to a namespace or the whole cluster?
3. (Intermediate) How can a LimitRange prevent a single Pod from requesting an unreasonably large amount of memory?
4. (Interview) How do LimitRange and ResourceQuota differ in scope, and why might you use both together? (Hint: per-object defaults/bounds vs. namespace aggregate totals.)

Answers

1. It injects default requests and limits (from defaultRequest/default) so those containers aren't left without resource settings (avoiding BestEffort and over-packing).
2. Namespace-scoped.
3. By setting a max for the relevant resource (e.g., max.memory: 2Gi); the admission controller rejects any container/Pod whose request or limit exceeds the maximum.
4. LimitRange governs per-object values: defaults and min/max bounds and ratios for individual containers/Pods. ResourceQuota governs namespace aggregates: the total requests/limits and object counts summed across the namespace. Used together, LimitRange ensures every Pod has sensible, bounded per-Pod resources (and defaults so quota math works), while ResourceQuota caps the namespace's total consumption — per-object hygiene plus overall fairness/capacity control.

---

8.2 Scheduling Concepts

The scheduler decides which node runs each Pod. This subchapter opens up its workflow and the affinity/anti-affinity/spread mechanisms you use to influence placement.

---

Scheduler workflow and phases

Theory

The kube-scheduler turns a Pod with no assigned node into a Pod bound to a specific node. (Introduced in Chapter 2; here we go deeper.) Its decision runs through a scheduling framework of phases, conceptually two stages:

1. Filtering (predicates): eliminate nodes that cannot run the Pod — insufficient allocatable resources, unsatisfied nodeSelector/affinity, untolerated taints, volume/zone conflicts, port conflicts. The output is the set of feasible nodes.
2. Scoring (priorities): rank the feasible nodes by a weighted set of plugins — resource balance (least/most allocated), affinity preferences, topology spread, image locality, inter-Pod affinity. The highest-scoring node wins (ties broken at random).

Then binding writes the chosen node to the Pod. The framework exposes extension points (PreFilter, Filter, Score, Reserve, Permit, Bind, etc.) where plugins — built-in or custom — hook in. If no node is feasible, the Pod stays Pending with an explanatory event.

Example

Pending Pod
   |
   v  FILTER: remove infeasible nodes
[node1 ok][node2 no:insufficient mem][node3 ok][node4 no:taint]
   |
   v  SCORE: rank feasible nodes
[node1: 72][node3: 88]   -> pick node3 (highest)
   |
   v  BIND: set spec.nodeName = node3

kubectl describe pod pending-pod | grep -A3 Events
# Warning  FailedScheduling  0/4 nodes are available: 2 Insufficient memory,
#          1 node(s) had untolerated taint, 1 node(s) didn't match nodeSelector.

Exercises

1. (Beginner) What are the two main stages of the scheduler's decision?
2. (Beginner) What does the scheduler write once it picks a node?
3. (Intermediate) A Pod is Pending with "0/5 nodes available". How do you find the specific per-node reasons?
4. (Interview) The scheduling framework exposes extension points for plugins. Why is a plugin-based scheduler architecture valuable, and what does it enable beyond the default behavior? (Hint: custom scheduling logic without forking.)

Answers

1. Filtering (find feasible nodes) and Scoring (rank them); followed by binding.
2. It writes the chosen node into the Pod's spec.nodeName (via a Binding), after which the kubelet on that node runs the Pod.
3. Run kubectl describe pod <pod> and read the FailedScheduling event, which enumerates per-node reasons (e.g., insufficient memory, untolerated taint, selector mismatch, volume zone conflict).
4. A plugin framework lets you customize or extend scheduling at well-defined extension points (Filter, Score, Reserve, Bind, etc.) without forking the scheduler. It enables custom placement logic — e.g., specialized bin-packing, hardware-aware or topology-aware scheduling, gang/batch scheduling, or third-party schedulers — while reusing the core machinery. Organizations can add their own constraints/priorities or run multiple schedulers, adapting placement to their needs cleanly.

---

NodeSelector

Theory

The simplest way to constrain where a Pod runs is nodeSelector: a map of key-value labels a node must have for the Pod to be scheduled there. The scheduler filters out any node whose labels don't include all the specified pairs. It's a hard requirement (the Pod won't schedule if no node matches) but expressively limited — only exact equality, ANDed together.

This is used for basic targeting: run on SSD nodes (disktype: ssd), on a particular instance type, on GPU nodes, or in a particular pool. Nodes carry both built-in labels (e.g., kubernetes.io/hostname, topology.kubernetes.io/zone, kubernetes.io/arch) and custom labels you add. For anything more nuanced (preferences, OR logic, "in"/"not in" sets), you graduate to node affinity (next).

Example

# Label a node so Pods can target it:
kubectl label node node-3 disktype=ssd

spec:
  nodeSelector:
    disktype: ssd            # Pod only schedules on nodes labeled disktype=ssd
  containers:
  - name: app
    image: myapp:1.0

Exercises

1. (Beginner) What does nodeSelector match against on a node?
2. (Beginner) Is nodeSelector a hard requirement or a preference?
3. (Intermediate) How do multiple key-value pairs in a nodeSelector combine — AND or OR?
4. (Interview) What are the expressiveness limitations of nodeSelector that motivate node affinity? (Hint: only equality/AND, no preferences or set operators.)

Answers

1. Node labels — the node must have all the specified key-value label pairs.
2. A hard requirement: if no node matches, the Pod stays Pending (it won't schedule).
3. AND — all specified label pairs must be present on the node.
4. nodeSelector supports only exact-equality matches ANDed together. It can't express preferences (soft constraints), set operators (In/NotIn/Exists/DoesNotExist), or OR logic across values. Node affinity adds required vs. preferred rules, weighted preferences, and richer match expressions, making it far more flexible for real placement needs.

---

Node affinity and anti-affinity

Theory

Node affinity is the more powerful successor to nodeSelector for attracting Pods to nodes based on node labels. It supports two flavors:

• requiredDuringSchedulingIgnoredDuringExecution: a hard rule (like nodeSelector but expressive) — the Pod only schedules on matching nodes.
• preferredDuringSchedulingIgnoredDuringExecution: a soft rule with a weight — the scheduler prefers matching nodes but will place the Pod elsewhere if needed.

The long names encode semantics: the rule applies during scheduling but is ignored during execution (a Pod already running isn't evicted if node labels later change). Node affinity uses matchExpressions with operators In, NotIn, Exists, DoesNotExist, Gt, Lt — enabling set membership and node anti-affinity (via NotIn/DoesNotExist, i.e., avoid certain nodes). This is how you express "must run on amd64 GPU nodes, preferably in zone us-east-1a."

Example

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:     # hard
        nodeSelectorTerms:
        - matchExpressions:
          - { key: kubernetes.io/arch, operator: In, values: [ amd64 ] }
          - { key: gpu, operator: Exists }
      preferredDuringSchedulingIgnoredDuringExecution:    # soft
      - weight: 80
        preference:
          matchExpressions:
          - { key: topology.kubernetes.io/zone, operator: In, values: [ us-east-1a ] }

Exercises

1. (Beginner) What is the difference between required... and preferred... node affinity?
2. (Beginner) Name three operators node affinity supports that nodeSelector does not.
3. (Intermediate) What does "IgnoredDuringExecution" mean for a Pod already running when node labels change?
4. (Interview) How would you express "the Pod must run on GPU nodes and should preferably avoid spot instances"? Sketch the affinity structure. (Hint: required Exists on gpu; preferred NotIn on a spot label.)

Answers

1. required... is a hard constraint — the Pod only schedules on nodes matching it (or stays Pending). preferred... is a soft, weighted preference — the scheduler favors matching nodes but will still schedule elsewhere if none match.
2. Any three of: In, NotIn, Exists, DoesNotExist, Gt, Lt.
3. The rule is only evaluated at scheduling time. If a running Pod's node later stops matching (labels change), the Pod is not evicted — it keeps running. Affinity influences placement, not ongoing residency.
4. Use a required nodeAffinity with matchExpressions: [{ key: gpu, operator: Exists }] (must be a GPU node), plus a preferred rule with a weight whose preference.matchExpressions is [{ key: <spot-label, e.g. node.kubernetes.io/lifecycle>, operator: NotIn, values: [ spot ] }] (prefer non-spot). The required term enforces GPU; the weighted preferred term steers away from spot nodes without making it mandatory.

---

Pod affinity and anti-affinity

Theory

While node affinity relates Pods to node labels, inter-Pod affinity/anti-affinity relates Pods to other Pods based on their labels, within a topology domain (defined by topologyKey, e.g., hostname, zone). This lets you co-locate or separate Pods:

• podAffinity: schedule this Pod near Pods matching a label (same topology domain) — e.g., place a web server in the same zone as its cache to reduce latency.
• podAntiAffinity: schedule this Pod away from Pods matching a label — e.g., spread replicas of a service across different nodes/zones so a single failure doesn't take them all out.

Both have required (hard) and preferred (soft) variants. The topologyKey is essential: it defines what "near"/"away" means — kubernetes.io/hostname means per-node, topology.kubernetes.io/zone means per-zone. A classic use is anti-affinity on hostname to ensure no two replicas of a critical service land on the same node. (Note: pod affinity can be computationally expensive at large scale.)

Example

spec:
  affinity:
    podAntiAffinity:                          # spread replicas across nodes
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels: { app: web }
        topologyKey: kubernetes.io/hostname   # "away" = different node
    podAffinity:                              # co-locate with cache in same zone
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 50
        podAffinityTerm:
          labelSelector: { matchLabels: { app: cache } }
          topologyKey: topology.kubernetes.io/zone

Exercises

1. (Beginner) What does pod affinity relate Pods to, versus node affinity?
2. (Beginner) What is the role of topologyKey?
3. (Intermediate) Write (in words) an anti-affinity rule ensuring no two web Pods share a node.
4. (Interview) Why is podAntiAffinity on kubernetes.io/hostname a common high-availability pattern, and what is a performance caveat of heavy pod affinity use? (Hint: spread for fault tolerance; scheduling cost at scale.)

Answers

1. Pod affinity relates a Pod to other Pods (by their labels, within a topology domain); node affinity relates a Pod to node labels.
2. It defines the topology domain over which "near"/"away" is measured — e.g., kubernetes.io/hostname (per node), topology.kubernetes.io/zone (per zone). Affinity/anti-affinity is evaluated within these domains.
3. A required podAntiAffinity with a labelSelector matching app: web and topologyKey: kubernetes.io/hostname — meaning a web Pod cannot be scheduled onto a node that already runs a web Pod, forcing one per node.
4. Spreading replicas across nodes (hostname anti-affinity) ensures a single node failure can't take down all replicas of a service, improving availability. The caveat: inter-Pod affinity/anti-affinity requires the scheduler to evaluate relationships against many existing Pods across the cluster, which is computationally expensive and can significantly slow scheduling in large clusters — so it should be used judiciously (often preferred or topology spread constraints are lighter alternatives).

---

Topology spread constraints

Theory

Topology spread constraints are a more modern, declarative way to control how Pods are evenly distributed across topology domains (zones, nodes), addressing the verbosity and performance issues of podAntiAffinity for spreading. You declare how unevenly Pods may be distributed via maxSkew over a topologyKey, and what to do if the constraint can't be met (whenUnsatisfiable: DoNotSchedule (hard) or ScheduleAnyway (soft)).

maxSkew is the maximum allowed difference between the most and least populated domains. For example, maxSkew: 1 over zones keeps replicas balanced across zones to within one. This is the recommended tool for "spread my Deployment's replicas evenly across zones/nodes for resilience and balance" — clearer and more efficient than equivalent anti-affinity rules. labelSelector identifies the group of Pods being balanced.

Example

spec:
  topologySpreadConstraints:
  - maxSkew: 1                                  # at most 1 difference between zones
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule            # hard constraint
    labelSelector:
      matchLabels: { app: web }
  containers:
  - name: web
    image: nginx

3 zones, 6 replicas, maxSkew 1  ->  2 + 2 + 2  (balanced)
If zone-c is full:               ->  3 + 3 + 0 would skew by 3 > 1 -> blocked/Pending

Exercises

1. (Beginner) What do topology spread constraints control?
2. (Beginner) What does maxSkew: 1 mean?
3. (Intermediate) What is the difference between whenUnsatisfiable: DoNotSchedule and ScheduleAnyway?
4. (Interview) Why are topology spread constraints often preferred over podAntiAffinity for evenly distributing replicas across zones? (Hint: declarative evenness, performance, finer control.)

Answers

1. How evenly Pods (matching a label selector) are distributed across topology domains (e.g., zones or nodes).
2. The most populated and least populated topology domains may differ by at most one Pod — keeping the distribution balanced to within one.
3. DoNotSchedule makes the constraint hard — if placing the Pod would violate maxSkew, it stays Pending. ScheduleAnyway makes it a soft preference — the scheduler tries to satisfy it but will still place the Pod even if it skews the distribution.
4. Topology spread constraints express "even distribution" directly via maxSkew, are easier to reason about, and are more scheduler-efficient than emulating spreading with many podAntiAffinity rules (which are computationally expensive at scale). They also offer finer control (soft/hard, per-domain skew, minDomains) and natively target balanced placement across zones/nodes for resilience, rather than the coarse "never co-locate" semantics of anti-affinity.

---

8.3 Taints and Tolerations

Affinity attracts Pods to nodes; taints do the opposite — they repel Pods from nodes unless the Pod explicitly tolerates them. This subchapter covers this complementary mechanism.

---

Tainting nodes

Theory

A taint is applied to a node and repels Pods that do not tolerate it. Taints and tolerations work together as the inverse of affinity: affinity is a Pod saying "I want this kind of node," while a taint is a node saying "keep Pods off me unless they explicitly accept this condition." This lets you reserve nodes for specific purposes.

A taint has a key, optional value, and an effect. You apply it with kubectl taint. Common uses: control-plane nodes are tainted so general workloads stay off them; GPU/specialized nodes are tainted so only GPU workloads (which tolerate the taint) land there; nodes under maintenance get tainted to drain workloads. Kubernetes also applies taints automatically for node conditions (not-ready, unreachable, memory-pressure, disk-pressure). A Pod is only scheduled to a tainted node if it has a matching toleration.

Example

# Reserve nodes for GPU workloads by tainting them:
kubectl taint nodes gpu-node-1 gpu=true:NoSchedule
# Remove a taint (note trailing minus):
kubectl taint nodes gpu-node-1 gpu=true:NoSchedule-

kubectl describe node gpu-node-1 | grep Taints
# Taints: gpu=true:NoSchedule    # only Pods tolerating gpu=true may schedule here

Exercises

1. (Beginner) Is a taint applied to a node or a Pod? What does it do?
2. (Beginner) What three parts make up a taint?
3. (Intermediate) Give two real-world reasons to taint a node.
4. (Interview) Explain how taints (on nodes) and tolerations (on Pods) together enable "dedicated nodes," and contrast this mechanism with node affinity. (Hint: repel-by-default vs. attract; need both taint and toleration to reserve.)

Answers

1. A taint is applied to a node; it repels Pods that don't have a matching toleration (keeping them from being scheduled there).
2. A key, an optional value, and an effect (e.g., NoSchedule).
3. Any two: reserve control-plane nodes (keep workloads off), dedicate GPU/specialized nodes to specific workloads, isolate nodes for a team/tenant, or mark nodes for maintenance/draining. Kubernetes also auto-taints nodes with conditions (not-ready, unreachable, pressure).
4. Tainting a node repels all Pods by default; only Pods carrying a matching toleration may schedule there. To truly dedicate nodes you usually combine a taint (so unrelated Pods are kept off) with node affinity/nodeSelector on the intended Pods (so they're attracted to those nodes) — the toleration alone only permits scheduling, it doesn't force it. Contrast: node affinity is Pod-driven attraction ("I prefer/require these nodes"), while taints are node-driven repulsion ("stay away unless you tolerate me"). Affinity alone can't keep other Pods off a node; a taint can.

---

Toleration syntax and effects

Theory

A toleration on a Pod allows (but does not require) it to schedule onto nodes with a matching taint. A toleration specifies a key, an operator (Equal to match a specific value, or Exists to match any value of that key), an effect, and — for the NoExecute effect — an optional tolerationSeconds.

Matching rules: a toleration tolerates a taint if the keys, effects, and (for Equal) values match. operator: Exists with no key matches all taints (used to tolerate everything, e.g., for critical DaemonSets). Importantly, a toleration only permits scheduling onto a tainted node — it does not attract the Pod there; without a taint, tolerations have no effect. So to pin Pods to dedicated nodes you pair a toleration (to be allowed on the tainted node) with node affinity/selector (to be drawn to it).

Example

spec:
  tolerations:
  - key: "gpu"
    operator: "Equal"
    value: "true"
    effect: "NoSchedule"        # matches taint gpu=true:NoSchedule
  # tolerate ANY taint (e.g., for a cluster-critical agent):
  - operator: "Exists"
  # NoExecute with grace period: stay 300s after node is tainted, then evict
  - key: "node.kubernetes.io/unreachable"
    operator: "Exists"
    effect: "NoExecute"
    tolerationSeconds: 300

Exercises

1. (Beginner) Does a toleration force a Pod onto a tainted node?
2. (Beginner) What is the difference between the Equal and Exists operators in a toleration?
3. (Intermediate) What does a toleration with only operator: Exists (no key) tolerate?
4. (Interview) To dedicate GPU nodes exclusively to GPU workloads (and ensure those workloads actually land there), what combination of mechanisms do you need and why? (Hint: taint + toleration + affinity/selector.)

Answers

1. No — it only permits scheduling onto a node with a matching taint; it does not attract or force the Pod there.
2. Equal matches a taint with the same key and value; Exists matches a taint with the given key regardless of its value (and, with no key, matches any taint).
3. All taints — a keyless operator: Exists toleration tolerates every taint on every node (commonly used for cluster-critical DaemonSets that must run everywhere).
4. Three things: (1) taint the GPU nodes (e.g., gpu=true:NoSchedule) so non-GPU Pods are kept off; (2) add a matching toleration to GPU workloads so they're allowed on those nodes; and (3) add node affinity/nodeSelector (e.g., on a gpu label) to the GPU workloads so they're actually drawn to the GPU nodes rather than scheduled elsewhere. The taint keeps others out, the toleration lets the right Pods in, and the affinity ensures they go there — all three are needed for true dedication.

---

NoSchedule, PreferNoSchedule, NoExecute

Theory

A taint's effect determines what happens to Pods that don't tolerate it:

Effect	Effect on non-tolerating Pods
NoSchedule	New Pods won't be scheduled onto the node; existing Pods stay.
PreferNoSchedule	The scheduler tries to avoid placing Pods here, but may if necessary (soft).
NoExecute	New Pods won't schedule and existing non-tolerating Pods are evicted.

The crucial distinction is NoExecute: unlike the others, it affects already-running Pods, evicting those that don't tolerate the taint. This is how Kubernetes handles node problems — when a node becomes unreachable, the node controller adds a NoExecute taint, and Pods (subject to their tolerationSeconds) are evicted and rescheduled elsewhere. With NoExecute, a Pod's tolerationSeconds sets how long it may remain after the taint appears before eviction (the default for the auto-added not-ready/unreachable taints is 300s).

Example

kubectl taint nodes node-2 maintenance=true:NoExecute   # evicts non-tolerating Pods now
kubectl taint nodes node-3 spot=true:PreferNoSchedule    # soft avoidance

NoSchedule       : keeps NEW pods off; running pods unaffected
PreferNoSchedule : best-effort avoidance for new pods
NoExecute        : keeps new pods off AND evicts running non-tolerating pods

Exercises

1. (Beginner) Which taint effect evicts already-running Pods that don't tolerate it?
2. (Beginner) How does PreferNoSchedule differ from NoSchedule?
3. (Intermediate) When a node becomes unreachable, which effect does Kubernetes apply automatically, and what role does tolerationSeconds play?
4. (Interview) You want to drain a node of workloads immediately for emergency maintenance using taints. Which effect do you use and what happens to the Pods? (Hint: NoExecute eviction and rescheduling.)

Answers

1. NoExecute.
2. NoSchedule is a hard rule — non-tolerating new Pods are never placed on the node. PreferNoSchedule is soft — the scheduler tries to avoid the node but will place Pods there if no better option exists.
3. It applies a NoExecute taint (e.g., node.kubernetes.io/unreachable). tolerationSeconds on the Pods' (auto-added) tolerations determines how long they may remain before eviction — default 300 seconds — giving a grace period in case the node recovers quickly before Pods are rescheduled.
4. Apply a NoExecute taint to the node (e.g., kubectl taint nodes <node> maintenance=true:NoExecute). Non-tolerating running Pods are evicted from the node (per any tolerationSeconds), and their controllers (Deployment/ReplicaSet/StatefulSet) reschedule replacements onto other nodes. (In practice kubectl drain is the standard tool, which cordons and evicts gracefully; the NoExecute taint is the underlying eviction mechanism.)

---

Use cases: dedicated nodes, GPU nodes

Theory

Bringing taints, tolerations, and affinity together, here are the canonical patterns:

• Dedicated nodes (for a team/tenant/workload class): taint the nodes (dedicated=team-a:NoSchedule), add the matching toleration to that team's Pods, and use node affinity/labels so those Pods target the dedicated nodes. Result: only team-a's Pods run there, and they reliably do.
• GPU/specialized hardware nodes: GPU nodes are expensive; you don't want ordinary Pods consuming them. Taint them (nvidia.com/gpu=present:NoSchedule), and only GPU workloads (which tolerate it and request the GPU resource) land there. Often the device plugin and node labels are combined with the taint.
• Spot/preemptible nodes: taint with PreferNoSchedule or a dedicated taint so only fault-tolerant workloads (with tolerations) use the cheaper, interruptible capacity.

The recurring lesson: taint to repel, tolerate to permit, affinity to attract — use them in combination to achieve precise, reliable placement.

Example

# GPU workload: tolerate the GPU taint, request the GPU, and target GPU nodes
spec:
  tolerations:
  - { key: "nvidia.com/gpu", operator: "Exists", effect: "NoSchedule" }
  nodeSelector:
    accelerator: nvidia-tesla       # attract to GPU nodes
  containers:
  - name: trainer
    image: ml-trainer:1.0
    resources:
      limits: { nvidia.com/gpu: 1 } # request 1 GPU (via device plugin)

Exercises

1. (Beginner) Why are GPU nodes commonly tainted?
2. (Beginner) What three mechanisms combine to reliably place a workload on dedicated nodes?
3. (Intermediate) For spot/preemptible nodes, why might you only allow fault-tolerant workloads, and how do taints help?
4. (Interview) Summarize the roles "taint, toleration, affinity" each play in achieving exclusive, reliable placement, and what breaks if you omit any one. (Hint: repel/permit/attract.)

Answers

1. GPUs are scarce and expensive; tainting GPU nodes keeps ordinary Pods off them so the costly hardware is reserved for workloads that actually need (and tolerate + request) GPUs.
2. A taint on the nodes (repel others), a matching toleration on the workload (permit it there), and node affinity/nodeSelector on the workload (attract it to those nodes).
3. Spot/preemptible nodes can be reclaimed at any time, so only workloads that tolerate sudden termination (stateless, restartable, or checkpointed) should run there. Taints (a dedicated taint or PreferNoSchedule) keep non-tolerant workloads off, while fault-tolerant workloads carry the toleration to opt in — protecting critical services from running on interruptible capacity.
4. Taint repels all non-tolerating Pods from the node (exclusivity). Toleration permits a specific Pod onto the tainted node (access). Affinity/selector attracts the Pod to those nodes (reliable placement). Omit the taint → other Pods can also use the nodes (no exclusivity). Omit the toleration → your intended Pod can't schedule there at all. Omit the affinity → your Pod is allowed on the nodes but may be scheduled elsewhere instead, so it isn't reliably placed. All three are needed for exclusive, dependable placement.

---

8.4 Resource Quotas

In multi-tenant clusters, you must stop one namespace from consuming all resources. ResourceQuotas and PriorityClasses provide fairness and prioritization. This subchapter covers them.

---

ResourceQuota for namespaces

Theory

A ResourceQuota caps the aggregate resource consumption within a namespace — total CPU/memory requests and limits, and counts of objects. It's the primary tool for multi-tenancy fairness: give each team a namespace with a quota so no single team can exhaust the cluster. It's enforced at admission time — if a new Pod would push the namespace over its quota, the request is rejected.

A key interaction: once a ResourceQuota constrains CPU/memory requests or limits in a namespace, every Pod created there must specify those requests/limits (otherwise admission can't account for them and rejects the Pod). This is why ResourceQuota is usually paired with a LimitRange that supplies defaults — so Pods without explicit resources still get values that satisfy the quota accounting. ResourceQuota operates on namespace totals; LimitRange operates per object.

Example

apiVersion: v1
kind: ResourceQuota
metadata: { name: team-quota, namespace: team-a }
spec:
  hard:
    requests.cpu: "10"          # total CPU requests across the namespace
    requests.memory: 20Gi
    limits.cpu: "20"
    limits.memory: 40Gi
    pods: "50"                  # max number of Pods

kubectl get resourcequota team-quota -n team-a
# NAME         AGE   REQUEST                              LIMIT
# team-quota   1d    requests.cpu: 6/10, requests.memory: 12Gi/20Gi   ...

Exercises

1. (Beginner) What does a ResourceQuota limit, and at what scope?
2. (Beginner) When is a ResourceQuota enforced?
3. (Intermediate) Why must Pods specify requests/limits once a ResourceQuota on those resources exists, and what helps with that?
4. (Interview) How do ResourceQuota and LimitRange complement each other in a multi-tenant cluster? (Hint: aggregate caps vs. per-object defaults/bounds.)

Answers

1. The aggregate resource consumption of a namespace — total CPU/memory requests and limits and counts of objects (Pods, Services, PVCs, etc.).
2. At admission time — when a new object is created; if it would exceed the namespace's hard quota, the request is rejected.
3. Because the quota tracks total requests/limits, the admission controller must know each Pod's requests/limits to account for them; a Pod that omits them can't be counted against a CPU/memory quota and is rejected. A LimitRange with defaults solves this by injecting requests/limits into Pods that don't specify them, so they satisfy quota accounting.
4. ResourceQuota enforces namespace-wide aggregate caps (fairness/capacity across tenants), while LimitRange enforces per-object defaults and min/max bounds. Together: LimitRange ensures every Pod has sensible bounded resources (and defaults so quota math works), and ResourceQuota ensures the namespace's total stays within its allotment — preventing any one tenant from monopolizing the cluster while keeping individual Pods well-formed.

---

Compute and object count quotas

Theory

ResourceQuota covers two broad categories:

• Compute resource quotas: limit requests.cpu, requests.memory, limits.cpu, limits.memory, and extended/hugepages and even requests.nvidia.com/gpu. These bound how much compute the namespace can reserve/use.
• Object count quotas: limit the number of objects of a kind — pods, services, configmaps, secrets, persistentvolumeclaims, services.loadbalancers (cap costly cloud LBs!), services.nodeports, and even counts of arbitrary resources via the count/<resource>.<group> syntax.

Object counts matter for reasons beyond compute: too many Secrets/ConfigMaps bloat etcd; uncapped LoadBalancer Services rack up cloud costs; runaway object creation can degrade the control plane. Storage quotas also exist (requests.storage, and per-StorageClass via <class>.storageclass.storage.k8s.io/requests.storage). Together these let platform teams bound both the compute footprint and the object/API footprint of each tenant.

Example

spec:
  hard:
    # compute
    requests.cpu: "10"
    limits.memory: 40Gi
    # object counts
    pods: "50"
    services.loadbalancers: "2"      # cap expensive cloud LBs
    persistentvolumeclaims: "20"
    count/deployments.apps: "30"
    # storage (per class)
    gold.storageclass.storage.k8s.io/requests.storage: 100Gi

Exercises

1. (Beginner) Name the two broad categories of quotas a ResourceQuota can enforce.
2. (Beginner) Give one object-count quota that directly limits cloud cost.
3. (Intermediate) Why might you cap the number of Secrets or ConfigMaps in a namespace?
4. (Interview) Beyond compute, why are object-count and storage quotas important for cluster stability and cost? (Hint: etcd bloat, control-plane load, cloud LB/disk spend.)

Answers

1. Compute resource quotas (CPU/memory/GPU requests and limits) and object count quotas (numbers of objects like Pods, Services, PVCs).
2. services.loadbalancers (each LoadBalancer Service provisions a paid cloud load balancer).
3. Every Secret/ConfigMap is stored in etcd and watched; large numbers bloat etcd, increase memory/IO and watch load, and can degrade API performance. Capping them prevents a tenant from inadvertently (or maliciously) overwhelming the datastore.
4. Object counts and storage consume shared, finite resources beyond node CPU/memory: etcd storage and watch/API load (many objects slow the control plane), and real money (each LoadBalancer Service = a cloud LB; each PVC = a provisioned disk). Quotas on these protect control-plane stability (bounded etcd/API footprint) and budget (bounded LB/disk spend), complementing compute quotas to keep the whole platform healthy and cost-controlled.

---

Quota scopes

Theory

Sometimes you want a quota to apply only to a subset of objects in a namespace — for example, "BestEffort Pods may number at most 10" or "terminating (batch) Pods get a separate CPU budget." Quota scopes enable this by restricting which objects a ResourceQuota counts.

Built-in scopes include Terminating / NotTerminating (Pods with an active deadline vs. without), BestEffort / NotBestEffort (by QoS class), and PriorityClass (via scopeSelector, matching Pods of certain priority classes). The modern, expressive form is scopeSelector with operators, letting you, e.g., apply a quota only to Pods with priorityClassName in (high). Scopes let one namespace have differentiated quotas for different workload classes — separating, say, long-running services from short batch jobs, or capping low-QoS workloads separately.

Example

# Quota that ONLY applies to high-priority Pods in the namespace:
apiVersion: v1
kind: ResourceQuota
metadata: { name: high-prio-quota, namespace: team-a }
spec:
  hard:
    requests.cpu: "4"
    requests.memory: 8Gi
  scopeSelector:
    matchExpressions:
    - { scopeName: PriorityClass, operator: In, values: [ "high" ] }

Exercises

1. (Beginner) What do quota scopes let you restrict?
2. (Beginner) Name two built-in quota scopes.
3. (Intermediate) How would you create a quota that only limits BestEffort Pods?
4. (Interview) Why are quota scopes (e.g., by PriorityClass) useful in a namespace that runs both critical services and batch jobs? (Hint: differentiated budgets per workload class.)

Answers

1. Which subset of objects a ResourceQuota applies to/counts (e.g., only Pods of a certain QoS, priority, or termination behavior).
2. Any two: Terminating, NotTerminating, BestEffort, NotBestEffort, PriorityClass (via scopeSelector).
3. Create a ResourceQuota with the BestEffort scope (or a scopeSelector matching it); it will then count/limit only Pods in the BestEffort QoS class, e.g., capping their count or resources.
4. Scopes let one namespace enforce different budgets for different workload classes. For instance, you can give high-priority/critical services a guaranteed CPU/memory budget while separately capping batch or BestEffort jobs, so batch work can't crowd out critical services and each class is bounded appropriately — finer-grained fairness than a single namespace-wide quota.

---

Priority classes and PriorityLevelConfiguration

Theory

Two distinct "priority" mechanisms exist; don't conflate them:

• PriorityClass (scheduling priority): a cluster-scoped object mapping a name to an integer priority. A Pod's priorityClassName sets how important it is for scheduling and preemption. Higher-priority Pending Pods are scheduled before lower-priority ones, and — if preemptionPolicy allows — a high-priority Pod can preempt (evict) lower-priority Pods to make room when the cluster is full. System-critical Pods use built-in classes like system-cluster-critical.
• PriorityLevelConfiguration (API Priority and Fairness, APF): part of the API server's flow control, it classifies and rate-limits API requests so that a flood of low-importance requests can't starve critical ones (e.g., protecting leader election and node heartbeats). Paired with FlowSchema, it ensures fair, bounded concurrency for API traffic.

In short: PriorityClass prioritizes Pods for the scheduler; PriorityLevelConfiguration prioritizes API requests for the API server.

Example

# Scheduling priority (PriorityClass):
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata: { name: high }
value: 1000000
globalDefault: false
description: "Critical user-facing services"
preemptionPolicy: PreemptLowerPriority
---

# Pod opting into it:
spec:
  priorityClassName: high

Exercises

1. (Beginner) What does a PriorityClass affect for a Pod?
2. (Beginner) What is preemption in the context of PriorityClass?
3. (Intermediate) Distinguish PriorityClass from PriorityLevelConfiguration in one sentence each.
4. (Interview) Why is API Priority and Fairness (PriorityLevelConfiguration + FlowSchema) important for control-plane stability under load? (Hint: prevent low-priority request floods from starving critical API traffic.)

Answers

1. Its scheduling importance — the order in which Pending Pods are scheduled and whether it can preempt lower-priority Pods to get resources.
2. When the cluster lacks room for a high-priority Pending Pod, the scheduler can evict (preempt) lower-priority running Pods to free resources for it (subject to preemptionPolicy), then schedule the high-priority Pod.
3. PriorityClass sets a Pod's scheduling/preemption priority for the kube-scheduler. PriorityLevelConfiguration (with FlowSchema) sets the priority/concurrency of API server requests for API Priority and Fairness flow control.
4. The API server has finite concurrency; without flow control, a burst of low-value requests (e.g., a misbehaving controller listing everything repeatedly) could saturate it and starve critical traffic like leader election, node heartbeats, and scheduler/controller calls — risking cascading control-plane failure. APF classifies requests into priority levels with bounded, fair concurrency and queuing, so essential requests are protected and the control plane stays responsive and stable under heavy or abusive load.

---

8.5 Autoscaling

Static replica counts and node pools waste money and risk outages. Kubernetes autoscales at three levels — pods (count), pods (size), and nodes — plus event-driven scaling. This subchapter covers them.

---

Horizontal Pod Autoscaler (HPA)

Theory

The Horizontal Pod Autoscaler (HPA) automatically adjusts the number of replicas of a workload (Deployment/StatefulSet) based on observed metrics — classically CPU utilization, but also memory, custom, and external metrics. It scales out (more Pods) under load and in (fewer Pods) when idle, keeping the target metric near a desired value.

The HPA controller periodically (default every 15s) reads metrics (via the Metrics API / Metrics Server for resource metrics) and computes desired replicas with roughly: desired = ceil(current * (currentMetric / targetMetric)). For example, with a 50% CPU target, if current utilization is 100% across 2 Pods, it scales to ~4. It respects minReplicas/maxReplicas and uses stabilization windows to avoid flapping. The HPA requires resource requests to be set (to compute utilization percentages) and the Metrics Server installed. It is the most common autoscaler and the standard way to handle variable traffic.

Example

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: web }
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target: { type: Utilization, averageUtilization: 50 }   # keep CPU ~50%

kubectl get hpa web
# NAME  REFERENCE        TARGETS   MINPODS  MAXPODS  REPLICAS
# web   Deployment/web   72%/50%   2        10       4         # scaling up

Exercises

1. (Beginner) What does the HPA adjust automatically?
2. (Beginner) What two prerequisites must be in place for CPU-based HPA to work?
3. (Intermediate) With a 50% CPU target, 3 Pods at 90% average utilization, roughly how many replicas will the HPA target?
4. (Interview) Why does the HPA require resource requests to be set, and what role do stabilization windows play? (Hint: utilization is relative to requests; avoid flapping.)

Answers

1. The number of replicas (Pods) of the target workload, scaling out/in based on metrics.
2. Resource requests must be set on the Pods (so utilization percentage can be computed), and the Metrics Server (Metrics API) must be installed to provide CPU/memory metrics.
3. desired ≈ ceil(3 × (90/50)) = ceil(5.4) = 6 replicas (bounded by maxReplicas).
4. CPU/memory "utilization" is expressed as a percentage of the Pod's request, so without a request there's no denominator to compute the percentage — the HPA can't determine utilization. Stabilization windows (and scaling policies) damp rapid oscillations: they make the controller wait/consider recent recommendations before scaling down (and optionally up), preventing "flapping" where transient metric spikes/dips cause constant scale up/down churn that disrupts the workload.

---

Vertical Pod Autoscaler (VPA)

Theory

Where the HPA changes the number of Pods, the Vertical Pod Autoscaler (VPA) changes the size of Pods — automatically recommending or setting CPU/memory requests and limits based on actual historical usage. It's for workloads that don't scale horizontally well (e.g., a single-writer database, or a JVM app) or to right-size requests you'd otherwise have to tune by hand.

The VPA has three modes: Off (only produces recommendations you can read), Initial (sets resources only at Pod creation), and Auto/Recreate (updates running Pods by evicting and recreating them with new resources — disruptive, since changing a running container's requests historically required a restart). A major caveat: VPA and HPA should not both act on the same resource metric (e.g., both on CPU), as they fight each other. VPA is best for vertical right-sizing; HPA for horizontal scaling on load. (In-place Pod resize is reducing VPA's disruptiveness in newer Kubernetes.)

Example

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata: { name: db-vpa }
spec:
  targetRef:
    apiVersion: apps/v1
    kind: StatefulSet
    name: db
  updatePolicy:
    updateMode: "Auto"        # Off | Initial | Auto (evicts to apply new sizes)

kubectl describe vpa db-vpa | grep -A6 "Recommendation"
# Target:  cpu: 850m   memory: 1200Mi   (suggested requests based on usage)

Exercises

1. (Beginner) What does the VPA adjust, in contrast to the HPA?
2. (Beginner) What are the VPA's update modes?
3. (Intermediate) Why has applying VPA recommendations to running Pods traditionally been disruptive?
4. (Interview) Why should you avoid running HPA and VPA on the same metric for the same workload, and how can they be combined safely? (Hint: conflicting actions; HPA on custom metric while VPA on CPU/mem, or VPA Off mode.)

Answers

1. The VPA adjusts the size of Pods — their CPU/memory requests (and limits) — whereas the HPA adjusts the number of Pod replicas.
2. Off (recommendations only), Initial (set resources only at Pod creation), and Auto/Recreate (update running Pods by evicting and recreating them with new resources).
3. Changing a running container's resource requests historically required recreating the Pod, so the VPA in Auto mode evicts and recreates Pods to apply new sizes — causing restarts/disruption. (In-place Pod vertical scaling in newer Kubernetes is making this less disruptive.)
4. If both act on the same metric (e.g., CPU), they conflict: the HPA adds replicas to lower per-Pod CPU while the VPA raises per-Pod CPU requests based on usage, and they oscillate against each other. Combine them safely by having the HPA scale on a different signal (a custom/external metric like requests-per-second or queue depth) while the VPA right-sizes CPU/memory, or run the VPA in Off mode for recommendations only — so the two never contend over the same dimension.

---

Cluster Autoscaler

Theory

The HPA and VPA change Pods, but if there's no room on existing nodes, new Pods stay Pending. The Cluster Autoscaler (CA) adjusts the number of nodes: it watches for Pods that can't be scheduled due to insufficient resources and adds nodes (by scaling the cloud provider's node groups / autoscaling groups), and it removes underutilized nodes whose Pods can fit elsewhere, to save cost.

Key behaviors: CA scales up when there are unschedulable Pods that would fit on a new node of an existing node group; it scales down when a node has been underutilized for a period and its Pods can be safely rescheduled (respecting PodDisruptionBudgets and certain "do-not-evict" annotations). It works per node group and relies on the cloud provider integration. CA complements the HPA: HPA adds Pods, and when those Pods don't fit, CA adds nodes — together they scale the whole stack. (Newer alternatives like Karpenter provision right-sized nodes more flexibly.)

Example

Traffic up -> HPA adds Pods -> some Pods Pending (no node capacity)
                                   |
                                   v
                 Cluster Autoscaler sees unschedulable Pods
                                   |
                                   v
              adds a node to the node group -> Pods schedule

Traffic down -> Pods removed -> node underutilized for N min
              -> CA drains and removes the node -> cost saved

Exercises

1. (Beginner) What does the Cluster Autoscaler adjust?
2. (Beginner) What triggers the Cluster Autoscaler to add a node?
3. (Intermediate) Under what conditions will the Cluster Autoscaler remove a node?
4. (Interview) Explain how the HPA and Cluster Autoscaler work together to handle a traffic surge, and what each is responsible for. (Hint: HPA scales Pods, CA scales nodes; Pending Pods bridge them.)

Answers

1. The number of nodes in the cluster (by scaling cloud node groups up or down).
2. The presence of unschedulable (Pending) Pods that don't fit on current nodes but would fit if a new node of an existing node group were added.
3. When a node is underutilized for a sustained period and all its Pods can be safely rescheduled onto other nodes — respecting PodDisruptionBudgets, Pods that can't be moved (e.g., certain local-storage or non-replicated Pods, or those with do-not-evict annotations). It then drains and removes the node to save cost.
4. On a surge, the HPA detects higher metrics and increases the Deployment's replica count. If existing nodes lack capacity, the new Pods become Pending. The Cluster Autoscaler observes these unschedulable Pods and adds node(s) to a node group; once the node joins, the Pending Pods are scheduled. The HPA owns how many Pods; the CA owns how many nodes — and Pending Pods are the signal that bridges the two so the whole stack scales to meet demand.

---

KEDA: event-driven autoscaling

Theory

The HPA scales on CPU/memory (and custom metrics), but many workloads should scale on event sources the HPA doesn't natively understand: queue length (Kafka, RabbitMQ, SQS), Prometheus query results, cron schedules, database row counts, cloud pub/sub backlog. KEDA (Kubernetes Event-Driven Autoscaling) fills this gap. It's an add-on that provides scalers for dozens of event sources and drives autoscaling — including the ability to scale to zero when there are no events (which the plain HPA cannot do).

Architecturally, KEDA acts as a metrics adapter and operator: you define a ScaledObject referencing your Deployment and a trigger (e.g., "scale based on Kafka topic lag"). KEDA translates the external event metric into something the HPA mechanism consumes (it actually creates/manages an HPA under the hood for the scaling math), and it independently handles activation from/to zero. This makes KEDA the standard tool for queue-driven workers and bursty event processing.

Example

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata: { name: consumer }
spec:
  scaleTargetRef: { name: consumer }     # the Deployment to scale
  minReplicaCount: 0                       # scale to zero when idle
  maxReplicaCount: 30
  triggers:
  - type: kafka
    metadata:
      bootstrapServers: kafka:9092
      consumerGroup: orders
      topic: orders
      lagThreshold: "100"                  # add replicas per 100 messages of lag

Exercises

1. (Beginner) What kind of signals does KEDA let you scale on that the plain HPA cannot?
2. (Beginner) What HPA limitation does KEDA notably overcome regarding idle workloads?
3. (Intermediate) In KEDA, what object do you define to configure event-driven scaling, and what does it reference?
4. (Interview) KEDA creates an HPA under the hood yet adds capabilities. Explain its architecture and why scale-to-zero requires logic beyond the standard HPA. (Hint: external scalers/metrics adapter; HPA min is 1; activation phase.)

Answers

1. External event-source metrics — e.g., message-queue lag/length (Kafka, RabbitMQ, SQS), Prometheus query results, cloud pub/sub backlog, cron schedules, database queries — dozens of scalers beyond CPU/memory.
2. Scale to zero: KEDA can scale a workload down to 0 replicas when there are no events (and back up when events arrive), which the standard HPA cannot do (its minimum is 1).
3. A ScaledObject (or ScaledJob), which references the target workload (scaleTargetRef) and defines one or more triggers (event sources) with thresholds.
4. KEDA runs an operator plus a metrics adapter exposing external-event metrics through the Kubernetes external/custom metrics API. For the 1..N scaling range it creates and manages a standard HPA that consumes those metrics (reusing the HPA's scaling math). But the HPA's minimum is 1 replica, so KEDA itself handles the activation phase: it watches the event source and, when activity crosses a threshold, scales the workload from 0 to 1 (and back to 0 when idle) outside the HPA's purview. Thus scale-to-zero needs KEDA's extra activation logic because the HPA alone can neither read arbitrary event sources nor go to zero.

---

Custom and external metrics

Theory

The HPA's power comes from the metrics APIs it can consume, of which there are three:

• Resource Metrics API (metrics.k8s.io, via Metrics Server): CPU and memory of Pods — the classic HPA input.
• Custom Metrics API (custom.metrics.k8s.io): application/Kubernetes-object metrics exposed by an adapter (e.g., the Prometheus Adapter) — scale on things like requests per second, queue depth as a Pod metric, or error rate. These are metrics associated with Kubernetes objects.
• External Metrics API (external.metrics.k8s.io): metrics from outside the cluster not tied to a Kubernetes object — e.g., a cloud queue length, a SaaS metric, a global business KPI.

Scaling on the right metric matters: CPU is a poor proxy for many workloads (a queue worker may be I/O-bound; a web app's true pressure is RPS or p99 latency). The Custom/External metrics APIs let you autoscale on the signal that actually reflects load. An adapter (Prometheus Adapter, KEDA, cloud adapters) implements these APIs to feed the HPA.

Example

# HPA scaling on a custom per-Pod metric (requests-per-second) via an adapter:
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: web }
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: web }
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: Pods
    pods:
      metric: { name: http_requests_per_second }
      target: { type: AverageValue, averageValue: "100" }   # 100 rps per Pod
  - type: External                                          # external (e.g., queue)
    external:
      metric: { name: queue_messages }
      target: { type: AverageValue, averageValue: "30" }

Exercises

1. (Beginner) Name the three metrics APIs the HPA can use.
2. (Beginner) What component typically implements the Custom Metrics API from Prometheus data?
3. (Intermediate) Give an example where CPU is a poor scaling metric and a custom/external metric is better.
4. (Interview) What is the distinction between a "custom" metric and an "external" metric for HPA purposes, and why does it matter which API a metric comes through? (Hint: tied to a K8s object vs. outside the cluster.)

Answers

1. The Resource Metrics API (metrics.k8s.io), the Custom Metrics API (custom.metrics.k8s.io), and the External Metrics API (external.metrics.k8s.io).
2. The Prometheus Adapter (it exposes Prometheus query results through the custom/external metrics APIs for the HPA to consume).
3. A queue-worker or I/O-bound service: it may sit at low CPU while a huge backlog accumulates, so CPU won't trigger scaling when it's actually overwhelmed. Scaling on queue length (external metric) or requests/second/latency (custom metric) reflects real load and scales correctly. (Likewise a latency-sensitive web app scaling on p99 latency or RPS rather than CPU.)
4. A custom metric is associated with a Kubernetes object (e.g., a per-Pod or per-Service metric like RPS) and comes through custom.metrics.k8s.io. An external metric is not tied to any cluster object — it originates outside the cluster (e.g., a cloud queue's length, a third-party KPI) and comes through external.metrics.k8s.io. It matters because the API determines how the value is identified and scoped (object-associated vs. free-standing), which adapter serves it, and how the HPA references it (type: Pods/Object vs. type: External). Choosing the correct API ensures the HPA can locate and interpret the metric properly.

---