4. Workload Resources

Pods are the atom of execution in Kubernetes, but you rarely create them directly. Instead you use higher-level workload resources — Deployments, StatefulSets, DaemonSets, Jobs — that manage Pods for you, providing self-healing, scaling, ordering, and lifecycle guarantees. This chapter starts at the Pod and works up the abstraction ladder, showing which controller to reach for given the shape of your workload.

---

4.1 Pods

A Pod is the smallest deployable unit in Kubernetes — one or more containers that share a network and storage context and are always scheduled together. This subchapter dissects the Pod inside and out.

---

Pod lifecycle and phases

Theory

A Pod is not simply "on" or "off." It moves through a defined set of phases that summarize where it is in its life. Understanding these phases is the foundation of debugging — most "why isn't my app working?" questions start by reading the phase and the container states beneath it.

The status.phase values are:

Phase	Meaning
Pending	Accepted by the cluster but not yet running — being scheduled, or images are pulling.
Running	Bound to a node; at least one container is running (or starting/restarting).
Succeeded	All containers terminated successfully (exit 0) and will not restart.
Failed	All containers terminated, and at least one failed (non-zero exit) or was killed.
Unknown	The node's state can't be obtained (usually a node/communication failure).

Beneath the phase, each container has its own state: Waiting (with a reason like ContainerCreating, ImagePullBackOff, CrashLoopBackOff), Running, or Terminated (with an exit code and reason like OOMKilled, Completed, Error). The phase is a high-level summary; the container states tell you the real story.

Example

kubectl get pod web -o jsonpath='{.status.phase}'      # -> Running
kubectl get pod web -o jsonpath='{.status.containerStatuses[0].state}'
# {"waiting":{"reason":"CrashLoopBackOff","message":"back-off 40s ..."}}

Typical happy-path progression:
  Pending (scheduling)            <- waiting for a node
  Pending (ContainerCreating)     <- pulling image, setting up network/volumes
  Running                         <- container(s) up
  Succeeded   (for batch Pods)    OR stays Running (for services)

Exercises

1. (Beginner) List the five Pod phases and give a one-line meaning for each.
2. (Beginner) What is the difference between a Pod phase and a container state?
3. (Intermediate) A Pod is in Pending for a long time. Where do you look to find out why, and name two possible causes.
4. (Interview) A Pod shows phase Running but your application is not serving traffic. Explain how that is possible and what you would inspect. (Hint: container restart counts, readiness probes, CrashLoopBackOff under a still-"Running" phase.)

Answers

1. Pending — accepted but not yet running (scheduling/pulling); Running — on a node with at least one container running; Succeeded — all containers exited 0, no restart; Failed — all containers terminated and at least one failed; Unknown — node state cannot be determined.
2. The phase is a coarse, cluster-level summary of the whole Pod's lifecycle position. A container state (Waiting/Running/Terminated, with reasons and exit codes) describes each individual container and carries the detailed diagnostic information.
3. Look at kubectl describe pod <name> (the Events section) and the container statuses. Causes include: no node has sufficient resources; unsatisfiable nodeSelector/affinity or taints; image pull failure (ImagePullBackOff); unbound PersistentVolumeClaim; or no CNI/network setup.
4. The phase Running only means a container process started — not that the app is ready. The container may be crash-looping (restarting repeatedly; check RESTARTS and --previous logs), failing its readiness probe (so it is excluded from Service endpoints despite "running"), or listening on the wrong port. Inspect restart counts, readiness/liveness probe status, logs, and whether the Service's EndpointSlices include the Pod.

---

Single-container vs multi-container pods

Theory

The most common Pod has exactly one container — Kubernetes' "one process per container" default. But a Pod can hold multiple containers that are tightly coupled and must share fate, network, and storage. The key principle: containers go in the same Pod only when they are a single cohesive unit that should always be co-located and co-scheduled; otherwise they belong in separate Pods.

All containers in a Pod share:

• The network namespace — same IP, same port space; they reach each other over localhost.
• Volumes — they can mount the same volume to share files.
• Lifecycle coupling — they are scheduled together onto one node and live/die together as a unit.

Multi-container patterns (named in Google's "Design Patterns for Container-Based Distributed Systems" paper) include sidecar (augment the main app), ambassador (proxy outbound connections), and adapter (transform output). If two containers do not need to share a host, network, or storage, put them in separate Pods so they can scale and schedule independently.

Example

apiVersion: v1
kind: Pod
metadata:
  name: web-with-logger
spec:
  containers:
  - name: app                       # main container writes logs to a shared volume
    image: myapp:1.0
    volumeMounts:
    - name: logs
      mountPath: /var/log/app
  - name: log-shipper               # sidecar reads the same volume and ships logs
    image: fluent-bit:2.0
    volumeMounts:
    - name: logs
      mountPath: /var/log/app       # SAME volume -> shares files with "app"
  volumes:
  - name: logs
    emptyDir: {}                    # shared scratch space, lives with the Pod

Exercises

1. (Beginner) What do all containers in the same Pod share?
2. (Beginner) How do two containers in the same Pod communicate over the network?
3. (Intermediate) Give one example where two containers belong in the same Pod and one where they belong in separate Pods.
4. (Interview) What is the guiding principle for deciding whether to add a container to an existing Pod versus creating a new Pod? (Hint: shared fate, co-scheduling, and independent scalability.)

Answers

1. They share the network namespace (one IP and port space), can share volumes, and share lifecycle/scheduling (placed on the same node, started and stopped together).
2. Over localhost (and distinct ports), since they share the same network namespace and IP.
3. Same Pod: an app container plus a log-shipping sidecar that reads the app's logs from a shared volume (tightly coupled, must be co-located). Separate Pods: a frontend and a backend service that communicate over the network and need to scale independently — they should be different Deployments/Pods fronted by Services.
4. Co-locate in one Pod only when the containers form a single cohesive unit that must share fate, node, network, and/or storage and should always be scheduled and scaled together. If they can or should scale, schedule, upgrade, or fail independently, put them in separate Pods. Over-stuffing a Pod couples lifecycles that ought to be independent.

---

Init containers

Theory

Sometimes a Pod's main container should not start until some setup is done: waiting for a dependency to be ready, downloading config, running database migrations, or setting file permissions. Init containers solve this. They are special containers that run to completion, one at a time, in order, before any of the Pod's normal (app) containers start.

Properties that make them distinct:

• They run sequentially; each must exit 0 before the next starts.
• If an init container fails, the kubelet restarts it (per the Pod's restartPolicy); the app containers do not start until all init containers succeed. This makes the Pod robustly block on prerequisites.
• They can use a different image with tools the main app does not need (e.g., git, curl), keeping the app image minimal.

Example

apiVersion: v1
kind: Pod
metadata:
  name: web
spec:
  initContainers:
  - name: wait-for-db               # block until the database Service resolves/answers
    image: busybox:1.36
    command: ['sh', '-c', 'until nc -z db 5432; do echo waiting; sleep 2; done']
  containers:
  - name: app
    image: myapp:1.0               # only starts AFTER wait-for-db exits 0

Exercises

1. (Beginner) When do init containers run relative to the app containers?
2. (Beginner) In what order do multiple init containers execute?
3. (Intermediate) Why might you use a different (larger, tool-rich) image for an init container than for the app container?
4. (Interview) An init container is stuck failing repeatedly. What happens to the Pod's app containers, and how does this behavior help enforce dependencies? (Hint: app containers don't start; restartPolicy.)

Answers

1. Init containers run (and must all complete successfully) before any of the Pod's normal app containers start.
2. Sequentially and in the order listed; each init container must exit 0 before the next one begins.
3. The init container can carry tools needed only for setup (e.g., git, curl, migration utilities) without bloating or adding attack surface to the app image. The app image stays minimal while setup still has what it needs.
4. The app containers do not start at all until every init container succeeds; a failing init container is restarted per the Pod's restartPolicy, so the Pod effectively blocks (often in Init:Error/Init:CrashLoopBackOff) until the prerequisite is met. This enforces dependencies declaratively — the app can assume its prerequisites are satisfied because Kubernetes guarantees it never starts otherwise.

---

Sidecar containers

Theory

A sidecar is a helper container that runs alongside the main application container in the same Pod to augment or support it — handling cross-cutting concerns like logging, monitoring, proxying, or syncing files, so the main app doesn't have to. The classic examples: a log-shipping agent, a service-mesh proxy (Envoy), or a file-sync container pulling content for a web server.

Historically sidecars were just ordinary additional containers in the Pod, which created a problem for Jobs and startup ordering: a long-running sidecar would keep a Job from completing, and there was no guarantee the sidecar started before the app. Modern Kubernetes (stable in v1.29+) introduces native sidecar containers, declared as init containers with restartPolicy: Always. These start before app containers (like init containers), keep running alongside them (unlike normal init containers), and are shut down after the app containers — fixing both the ordering and Job-completion issues.

Example

apiVersion: v1
kind: Pod
metadata:
  name: web
spec:
  initContainers:
  - name: log-shipper              # a NATIVE sidecar: init container that keeps running
    image: fluent-bit:2.0
    restartPolicy: Always          # <- this makes it a sidecar, not a normal init container
    volumeMounts:
    - name: logs
      mountPath: /var/log/app
  containers:
  - name: app
    image: myapp:1.0
    volumeMounts:
    - name: logs
      mountPath: /var/log/app
  volumes:
  - name: logs
    emptyDir: {}

Exercises

1. (Beginner) What is a sidecar container, in one sentence?
2. (Beginner) How are native sidecar containers declared in modern Kubernetes?
3. (Intermediate) Why did running a sidecar as an ordinary container cause problems for Jobs?
4. (Interview) Compare a native sidecar (init container with restartPolicy: Always) to a plain additional app container. What ordering and lifecycle guarantees does the native sidecar add? (Hint: starts before, stops after, doesn't block Job completion.)

Answers

1. A sidecar is a helper container that runs in the same Pod as the main app to provide supporting/cross-cutting functionality (logging, proxying, syncing, monitoring).
2. As an init container with restartPolicy: Always (stable since v1.29). This signals Kubernetes to start it before the app containers but keep it running alongside them.
3. An ordinary container runs for the life of the Pod, but a Job is considered complete only when its containers terminate. A long-running sidecar never exits on its own, so the Job's Pod never reached Succeeded — the sidecar kept it "running" forever.
4. A native sidecar starts and becomes ready before the app containers (so the app can rely on it, e.g., a proxy being up), keeps running during the app's life, and is terminated after the app containers stop. Crucially, the Pod/Job can complete based on the main containers finishing — the sidecar does not block completion. A plain app container has none of these ordering or completion guarantees.

---

Pod spec anatomy

Theory

Every Pod (and every workload that creates Pods) is described by a Pod spec — the spec section that declares what to run and how. Reading and writing this fluently is essential. The major regions of a Pod spec are:

• containers / initContainers: the images, commands, ports, env, resources, probes, and volume mounts.
• volumes: storage available to the Pod's containers.
• Scheduling controls: nodeSelector, affinity, tolerations, topologySpreadConstraints.
• Security: securityContext (Pod- and container-level), serviceAccountName.
• Lifecycle/behavior: restartPolicy, terminationGracePeriodSeconds, dnsPolicy, hostNetwork.

Within a container, the most-used fields are name, image, command/args, env/envFrom, ports, resources (requests/limits), volumeMounts, and the three probes (livenessProbe, readinessProbe, startupProbe).

Example

apiVersion: v1
kind: Pod
metadata:
  name: anatomy
  labels: { app: anatomy }
spec:
  serviceAccountName: app-sa        # identity for API access
  restartPolicy: Always             # Always | OnFailure | Never
  terminationGracePeriodSeconds: 30 # time for graceful shutdown
  containers:
  - name: app
    image: myapp:1.0
    command: ["/bin/app"]           # overrides image ENTRYPOINT
    args: ["--port=8080"]           # overrides image CMD
    ports:
    - containerPort: 8080
    env:
    - name: LOG_LEVEL
      value: "info"
    resources:                      # scheduling + limits (see Chapter 8)
      requests: { cpu: "100m", memory: "128Mi" }
      limits:   { cpu: "500m", memory: "256Mi" }
    readinessProbe:                 # gate traffic until ready (see Chapter 10)
      httpGet: { path: /healthz, port: 8080 }
    volumeMounts:
    - name: data
      mountPath: /data
  volumes:
  - name: data
    emptyDir: {}

Exercises

1. (Beginner) What is the difference between a container's command/args and the image's ENTRYPOINT/CMD?
2. (Beginner) Name the three valid values of restartPolicy.
3. (Intermediate) Where in a Pod spec do volumes get declared versus mounted, and why are these two separate steps?
4. (Interview) The same Pod spec template appears inside Deployments, StatefulSets, DaemonSets, and Jobs. Why is reusing one Pod template across all controllers a powerful design decision? (Hint: consistency, composability, learn-once.)

Answers

1. command overrides the image's ENTRYPOINT and args overrides its CMD. If omitted, the image's built-in ENTRYPOINT/CMD are used. They let you change what runs without rebuilding the image.
2. Always, OnFailure, and Never.
3. Volumes are declared once under spec.volumes (defining the storage source, e.g., emptyDir, configMap, PVC) and mounted per-container under containers[].volumeMounts (defining where each container sees it). Separating them lets multiple containers mount the same declared volume at different paths and decouples "what storage exists" from "who uses it and where."
4. Every workload controller embeds the identical Pod template, so once you understand the Pod spec you understand how to configure all of them. It composes cleanly — affinity, probes, resources, security all work the same regardless of controller — reducing cognitive load, enabling consistent tooling/validation, and letting controllers focus only on how many Pods and with what lifecycle, not on redefining what a Pod is.

---

Ephemeral containers for debugging

Theory

How do you debug a running Pod whose container image is a minimal, distroless image with no shell, no curl, no ps? You cannot kubectl exec a shell that does not exist. Ephemeral containers solve this: they are temporary containers you inject into a running Pod for debugging, sharing the Pod's namespaces so they can inspect the existing containers' processes, network, and (optionally) filesystem.

Key characteristics:

• They are added to a live Pod via kubectl debug and cannot be specified in the Pod spec at creation time.
• They have no resource guarantees, no probes, and no ports — they are strictly for interactive troubleshooting.
• They do not restart, and removing them is automatic when you are done; they don't alter the original containers.

This is the modern, supported way to debug — far better than baking debug tools into production images.

Example

# Attach a debug container (with a full shell + tools) to a running Pod:
kubectl debug -it web-abc --image=busybox:1.36 --target=app
# --target=app shares the process namespace of container "app" so you can see its PIDs

# Inside, you can inspect the app's network and processes:
/ # wget -qO- localhost:8080/healthz
/ # ps aux

# Or create a copy of the Pod with a debug container (non-disruptive variant):
kubectl debug web-abc -it --image=busybox --copy-to=web-debug

Exercises

1. (Beginner) What problem do ephemeral containers solve for minimal/distroless images?
2. (Beginner) Which kubectl command injects an ephemeral debug container into a running Pod?
3. (Intermediate) Why can't ephemeral containers have resource requests/limits, ports, or probes?
4. (Interview) Compare baking debugging tools into a production image versus using ephemeral containers. What are the security and operational advantages of the ephemeral approach? (Hint: image size, attack surface, on-demand tooling.)

Answers

1. Minimal/distroless images often lack a shell and tools, so kubectl exec has nothing to run. Ephemeral containers inject a temporary, tool-rich container into the running Pod, sharing its namespaces so you can debug the existing containers without modifying their images.
2. kubectl debug (e.g., kubectl debug -it <pod> --image=busybox --target=<container>).
3. They are designed purely for short-lived, interactive debugging and are added to an already-running Pod. Allowing resource requests would affect scheduling/QoS of a Pod that is already placed; probes and ports imply a managed, long-lived service role. Ephemeral containers intentionally have none of these so they never disturb the Pod's guaranteed resources or service behavior.
4. Baking tools into production images increases image size, expands the attack surface (every tool is a potential exploit vector), and means you ship debugging capabilities to every running instance permanently. Ephemeral containers keep production images minimal and hardened, and provide debugging tools on demand only when needed, in a temporary container that is removed afterward — better security posture and smaller, faster images, with no loss of debuggability.

---

4.2 ReplicaSets and Deployments

You almost never create bare Pods in production — they have no self-healing or rollout machinery. Deployments (built on ReplicaSets) give you declarative, versioned, self-healing application management. This subchapter covers both.

---

ReplicaSet purpose and behavior

Theory

A bare Pod is mortal: if it (or its node) dies, it is gone for good. A ReplicaSet is the controller that fixes this. Its single responsibility is to maintain a stable set of replica Pods running at any given time — if you ask for 3 and only 2 exist, it creates one; if 4 exist, it deletes one. This is reconciliation (Chapter 1) applied to Pod count.

A ReplicaSet identifies the Pods it owns through a label selector plus ownerReferences on the Pods. This is important: a ReplicaSet will adopt any existing Pod matching its selector that lacks a controller, and conversely it only manages Pods carrying the matching labels. In practice you rarely create ReplicaSets directly — you create Deployments, which create and manage ReplicaSets for you (adding rollout and rollback on top). Knowing the ReplicaSet exists underneath is essential for understanding how Deployments work.

Example

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: web
spec:
  replicas: 3                  # desired count the controller maintains
  selector:
    matchLabels:
      app: web                 # which Pods this RS owns
  template:                    # the Pod template used to create missing Pods
    metadata:
      labels:
        app: web               # MUST match the selector above
    spec:
      containers:
      - name: web
        image: nginx:1.25

kubectl get rs web
# NAME   DESIRED   CURRENT   READY   AGE
# web    3         3         3       2m
kubectl delete pod -l app=web --field-selector ... # delete one; RS recreates it

Exercises

1. (Beginner) What is the one job of a ReplicaSet?
2. (Beginner) How does a ReplicaSet know which Pods belong to it?
3. (Intermediate) Why do you rarely create ReplicaSets directly, and what do you create instead?
4. (Interview) The Pod template's labels must match the ReplicaSet's selector. What goes wrong if they don't, and why is this constraint enforced? (Hint: orphaned Pods, runaway creation.)

Answers

1. To maintain a specified number of identical replica Pods running at all times — creating or deleting Pods to converge actual count to the desired replicas.
2. Via its label selector (which Pods match) combined with ownerReferences set on Pods it creates. It manages Pods that match the selector and adopts matching controller-less Pods.
3. Because Deployments manage ReplicaSets and add rollout, rollback, and revision history on top. You create a Deployment; it creates/manages ReplicaSets automatically, which is almost always what you want.
4. Kubernetes rejects a ReplicaSet whose template labels don't match its selector (selector does not match template labels). If it were allowed, the Pods the RS creates would not match its own selector, so it would never "see" them as satisfying the count and would create Pods endlessly (runaway creation), while the created Pods would be effectively orphaned. The constraint guarantees the controller can recognize the Pods it produces.

---

Deployment creation and management

Theory

A Deployment is the workhorse controller for stateless applications. It manages ReplicaSets, which manage Pods — a three-level hierarchy. The Deployment adds what a raw ReplicaSet lacks: declarative updates with versioned rollouts and rollbacks. When you change a Deployment's Pod template (e.g., a new image), the Deployment creates a new ReplicaSet and gradually shifts Pods from the old ReplicaSet to the new one, keeping a history of revisions.

You manage a Deployment declaratively with kubectl apply -f, and observe rollouts with kubectl rollout status. The Deployment continuously reconciles: it ensures the right number of replicas of the right version are running, replacing failed Pods and completing in-progress rollouts. For the vast majority of services (web servers, APIs, workers), the Deployment is the correct resource.

Example

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels: { app: web }
  template:
    metadata:
      labels: { app: web }
    spec:
      containers:
      - name: web
        image: nginx:1.25

kubectl apply -f web-deploy.yaml
kubectl get deploy,rs,pods -l app=web   # see the Deploy -> RS -> Pods hierarchy
kubectl rollout status deployment/web   # wait for the rollout to finish
kubectl scale deployment web --replicas=5

Exercises

1. (Beginner) What is the three-level hierarchy a Deployment manages?
2. (Beginner) What does a Deployment add on top of a plain ReplicaSet?
3. (Intermediate) When you change a Deployment's image, what does the Deployment create, and what happens to the old ReplicaSet?
4. (Interview) Why is the Deployment the recommended resource for stateless apps, but not for databases? (Hint: identity, ordering, and storage — foreshadowing StatefulSets.)

Answers

1. Deployment -> ReplicaSet -> Pods. The Deployment manages ReplicaSets, which manage Pods.
2. Declarative, versioned rollouts and rollbacks with revision history (plus pause/resume and rollout strategies), which a bare ReplicaSet does not provide.
3. It creates a new ReplicaSet for the new template and scales it up while scaling the old ReplicaSet down (a rolling update). The old ReplicaSet is kept (scaled to 0) for rollback history, up to revisionHistoryLimit.
4. Deployment Pods are interchangeable and have no stable identity or ordered lifecycle — perfect for stateless apps where any replica is as good as another. Databases need stable network identities, ordered startup/shutdown, and stable per-Pod persistent storage, which Deployments do not provide. Those guarantees come from StatefulSets.

---

Rolling updates and rollbacks

Theory

The whole point of a Deployment is zero-downtime updates. A rolling update replaces old Pods with new ones incrementally: bring up some new Pods, wait until they are ready, then terminate some old Pods, repeating until the new version fully replaces the old. Because old Pods keep serving until new ones are ready, users see no outage.

Two parameters control the pace, both under strategy.rollingUpdate:

• maxUnavailable: how many Pods (count or %) may be unavailable during the update — controls how aggressively old Pods are removed.
• maxSurge: how many extra Pods (count or %) may be created above the desired count — controls how many new Pods come up at once.

If a new version is broken, you roll back to a previous revision — Kubernetes keeps the old ReplicaSets, so rollback is just scaling the old RS back up and the bad one down. kubectl rollout undo does exactly this.

Example

spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1      # never drop below desired-1 ready Pods
      maxSurge: 1            # allow at most 1 extra Pod during the rollout

kubectl set image deployment/web web=nginx:1.26   # trigger a rolling update
kubectl rollout status deployment/web             # watch progress
kubectl rollout history deployment/web            # list revisions
kubectl rollout undo deployment/web               # roll back to previous revision
kubectl rollout undo deployment/web --to-revision=3

Exercises

1. (Beginner) Why does a rolling update avoid downtime?
2. (Beginner) What do maxUnavailable and maxSurge control?
3. (Intermediate) With replicas: 4, maxUnavailable: 1, maxSurge: 1, what is the minimum number of ready Pods and the maximum total Pods during a rollout?
4. (Interview) How is a rollback able to be so fast and safe in Kubernetes? Explain what actually happens at the ReplicaSet level. (Hint: old ReplicaSets are retained, not deleted.)

Answers

1. New Pods are brought up and confirmed ready before old Pods are removed, so a working set of Pods is always serving traffic — there is never a moment with zero available replicas (given sane settings and readiness probes).
2. maxUnavailable caps how many Pods may be unavailable during the update (how fast old Pods are torn down); maxSurge caps how many extra Pods above the desired count may be created (how fast new Pods are added).
3. Minimum ready = replicas - maxUnavailable = 4 - 1 = 3. Maximum total = replicas + maxSurge = 4 + 1 = 5.
4. When a Deployment updates, it does not delete the old ReplicaSet — it scales it to 0 and keeps it in history. A rollback simply scales the previous (known-good) ReplicaSet back up and the current (bad) one down — the same rolling mechanism in reverse. Because the old Pods' exact spec is preserved in the retained ReplicaSet, the rollback is fast, deterministic, and requires no rebuild or re-fetch of configuration.

---

Deployment strategies (Recreate, RollingUpdate)

Theory

A Deployment supports two built-in update strategies, chosen via spec.strategy.type:

• RollingUpdate (the default): incrementally replace old Pods with new ones, as described above. Both versions run simultaneously during the transition. This gives zero downtime but requires your app to tolerate two versions running at once (e.g., compatible database schemas).
• Recreate: terminate all old Pods first, then create the new ones. This causes a brief downtime window (no Pods serving during the switch) but guarantees only one version runs at a time. Use it when two versions cannot coexist — for example, an app that holds an exclusive lock, or a schema migration incompatible with the old code.

More advanced strategies (canary, blue-green) aren't native Deployment types; they're built using multiple Deployments/Services or tools like Argo Rollouts (covered in Chapter 12).

Example

# Recreate: full stop, then full start (accepts downtime, avoids version overlap)
spec:
  strategy:
    type: Recreate
---

# RollingUpdate (default): gradual, zero-downtime, versions overlap
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%

Exercises

1. (Beginner) What are the two built-in Deployment strategy types?
2. (Beginner) Which strategy causes downtime, and which avoids it?
3. (Intermediate) Give a concrete scenario where Recreate is the correct choice despite the downtime.
4. (Interview) RollingUpdate runs two versions of your app simultaneously. What application-level requirement does this impose, and how does it relate to database schema changes? (Hint: backward/forward compatibility.)

Answers

1. RollingUpdate (default) and Recreate.
2. Recreate causes downtime (all old Pods stop before new ones start). RollingUpdate avoids downtime (gradual replacement).
3. When the old and new versions cannot run at the same time — e.g., the app requires an exclusive resource/lock that only one version may hold, or a non-backward-compatible schema migration means the old code would corrupt or fail against the new schema. Recreate guarantees only one version exists at a time.
4. RollingUpdate means old and new Pods serve traffic concurrently, so both versions must be mutually compatible (the new version works with old data/clients and vice versa). For databases this requires backward/forward-compatible, expand-then-contract migrations: first deploy schema changes that both versions tolerate (additive), roll out the new code, then remove old columns/fields in a later step — never a breaking change in a single rollout.

---

Pausing and resuming deployments

Theory

Sometimes you want to make several changes to a Deployment without triggering a separate rollout for each one. By default, every edit to the Pod template kicks off a new rollout immediately. Pausing a Deployment lets you batch changes: while paused, the Deployment records your spec changes but does not act on them. When you resume, it performs a single rollout that incorporates all the accumulated changes.

This is also useful for canary-style manual control: you can pause mid-rollout to observe the new Pods before allowing the rollout to continue, then resume (or roll back) based on what you see. kubectl rollout pause and kubectl rollout resume control this.

Example

kubectl rollout pause deployment/web        # stop reacting to spec changes

# Make several changes — none triggers a rollout yet:
kubectl set image deployment/web web=nginx:1.26
kubectl set resources deployment/web -c=web --limits=cpu=500m,memory=256Mi
kubectl set env deployment/web LOG_LEVEL=debug

kubectl rollout resume deployment/web       # ONE rollout applies all changes
kubectl rollout status deployment/web

Exercises

1. (Beginner) By default, what happens each time you change a Deployment's Pod template?
2. (Beginner) What is the benefit of pausing a Deployment before making several changes?
3. (Intermediate) How can pause/resume be used to manually gate a rollout for observation?
4. (Interview) When a Deployment is paused, where do your changes go, and what guarantees do you have that they will be applied correctly on resume? (Hint: desired state is still recorded in the spec.)

Answers

1. It triggers a new rollout immediately (a new ReplicaSet is created and Pods are rolled).
2. You can batch multiple edits (image, env, resources) and have them applied in a single rollout on resume, instead of one rollout per change — fewer disruptions and faster overall.
3. Trigger a change so a rollout begins, then kubectl rollout pause partway through; observe the new Pods/canary behavior; if healthy, kubectl rollout resume to complete it, or kubectl rollout undo to abort. The pause halts further progression while the partial new version is observable.
4. The changes are written to the Deployment's spec (desired state) and stored in etcd as normal — pausing only stops the Deployment controller from acting on the spec, not from recording it. On resume, the controller reconciles actual state to the now-current desired spec via its normal rolling mechanism, so all accumulated changes are applied together and correctly, with the usual rollout guarantees (readiness gating, surge/unavailable limits).

---

4.3 StatefulSets

Some applications — databases, message queues, clustered stores — need stable identities and storage that Deployments cannot provide. StatefulSets are built for exactly these stateful workloads. This subchapter explains how and when to use them.

---

StatefulSet use cases

Theory

Deployments treat Pods as cattle — interchangeable and disposable. Some workloads need pets — each instance has a distinct, persistent identity. A StatefulSet manages Pods that each require one or more of: a stable, unique network identity, stable persistent storage that follows the Pod across restarts/rescheduling, and ordered, graceful deployment and scaling.

Typical use cases: databases (PostgreSQL, MySQL), distributed datastores (Cassandra, MongoDB, Elasticsearch), message brokers (Kafka, RabbitMQ), and any clustered system where members must know each other by stable names and keep their own data. The defining question: does each replica need its own persistent identity and storage? If yes, StatefulSet; if no (replicas are interchangeable), Deployment.

Example

Deployment Pods (interchangeable):       StatefulSet Pods (stable identity):
  web-7d9c-a1b2  (random suffix)            db-0   <- ordinal identity
  web-7d9c-c3d4                              db-1
  web-7d9c-e5f6                              db-2
  any can be replaced by any other          db-0 always "db-0", own storage

Exercises

1. (Beginner) What three guarantees can a StatefulSet provide that a Deployment cannot?
2. (Beginner) Give two example workloads that are appropriate for a StatefulSet.
3. (Intermediate) What single question best determines whether to use a StatefulSet or a Deployment?
4. (Interview) Explain the "pets vs. cattle" analogy and how it maps onto Deployments vs. StatefulSets. (Hint: identity and replaceability.)

Answers

1. Stable, unique network identity per Pod; stable persistent storage that stays with each Pod across rescheduling; and ordered, graceful deployment, scaling, and deletion.
2. Any two: PostgreSQL/MySQL databases, Cassandra/MongoDB/Elasticsearch clusters, Kafka/RabbitMQ/ZooKeeper.
3. "Does each replica need its own stable identity and persistent storage?" If yes -> StatefulSet; if replicas are interchangeable and stateless -> Deployment.
4. "Cattle" are interchangeable and individually disposable — you don't name them; if one fails you replace it with an identical one. That is a Deployment's Pods. "Pets" have names and individual care — each is unique and not freely replaceable. That is a StatefulSet's Pods, where each has a stable ordinal identity (db-0, db-1) and its own persistent data, so it must be recreated as itself, not swapped for an arbitrary replica.

---

Stable network identities

Theory

A StatefulSet gives each Pod a stable, predictable name and DNS record that persists across restarts and rescheduling. Pods are named <statefulset-name>-<ordinal> — db-0, db-1, db-2 — rather than getting random suffixes. Crucially, if db-1 dies and is recreated (even on another node), it comes back as db-1, not a new random name.

Combined with a headless Service (covered below), each Pod also gets a stable DNS name: <pod-name>.<service-name>.<namespace>.svc.cluster.local, e.g., db-0.cassandra.default.svc.cluster.local. This lets cluster members address each other directly and reliably — essential for systems where nodes must form a quorum, replicate to specific peers, or maintain a membership list. Without stable identities, a restarted Pod would be unrecognizable to its peers.

Example

# StatefulSet Pods have ordinal names, created/addressed predictably:
kubectl get pods -l app=db
# NAME   READY   STATUS    AGE
# db-0   1/1     Running   5m
# db-1   1/1     Running   4m
# db-2   1/1     Running   3m

# Each Pod is directly addressable by stable DNS (via a headless Service "db"):
kubectl run t --rm -it --image=busybox -- nslookup db-0.db.default.svc.cluster.local

Exercises

1. (Beginner) What is the naming pattern for StatefulSet Pods?
2. (Beginner) If db-1 is rescheduled to a different node, what name does it come back with?
3. (Intermediate) Write the full DNS name for Pod db-2 of a StatefulSet fronted by a headless Service named cassandra in the data namespace.
4. (Interview) Why are stable per-Pod DNS names essential for clustered databases, but unnecessary for a stateless web Deployment? (Hint: peer membership, replication targets, quorum.)

Answers

1. <statefulset-name>-<ordinal>, with ordinals starting at 0 (e.g., db-0, db-1, db-2).
2. The same name, db-1 — the identity is stable across rescheduling.
3. cassandra-2... actually the Pod is db-2: db-2.cassandra.data.svc.cluster.local (form: <pod>.<headless-service>.<namespace>.svc.cluster.local).
4. Clustered databases form a membership where nodes must reach specific peers by a durable address — for replication, leader election, and quorum — and must recognize a restarted member as the same node holding the same data. Stable DNS names provide that durable addressing. A stateless web Deployment's replicas are interchangeable and addressed collectively through a single Service ClusterIP/load balancer; no replica needs to find a specific peer, so per-Pod identity adds no value.

---

Ordered pod creation and deletion

Theory

StatefulSets manage Pods in order, which matters for systems with initialization dependencies (e.g., a primary must be up before replicas join). By default:

• Creation/scale-up proceeds in order from ordinal 0 upward: db-0 is created and becomes Ready before db-1 starts, which becomes Ready before db-2, and so on.
• Deletion/scale-down proceeds in reverse order, from the highest ordinal down: db-2 is terminated before db-1, before db-0.
• Updates (RollingUpdate) also go in reverse ordinal order, one Pod at a time.

This ordering can be relaxed with podManagementPolicy: Parallel (start/stop all at once) when your app does not need it, but the default OrderedReady is what provides the dependency guarantees. Ordered, one-at-a-time operations make scaling slower but safe for stateful systems.

Example

spec:
  podManagementPolicy: OrderedReady   # default: one at a time, in order
  # podManagementPolicy: Parallel     # alternative: all at once (no ordering)
  updateStrategy:
    type: RollingUpdate               # updates go highest-ordinal -> lowest

Scale up 0 -> 3:   create db-0 (wait Ready) -> db-1 (wait Ready) -> db-2
Scale down 3 -> 1: delete db-2 -> db-1 -> (keep db-0)

Exercises

1. (Beginner) In what order are StatefulSet Pods created during scale-up?
2. (Beginner) In what order are they deleted during scale-down?
3. (Intermediate) What does podManagementPolicy: Parallel change, and when would you use it?
4. (Interview) Why is ordered, one-at-a-time scaling important for a primary/replica database, even though it is slower than parallel? (Hint: a replica needs the primary to exist before it can join.)

Answers

1. In ascending ordinal order starting at 0, and each Pod must be Running and Ready before the next is created (db-0, then db-1, then db-2).
2. In descending order, highest ordinal first (db-2, then db-1, then db-0).
3. Parallel makes the StatefulSet create and delete all Pods simultaneously instead of waiting for ordering/readiness. Use it when Pods have no startup interdependencies and you want faster scaling, while still wanting stable identities and per-Pod storage.
4. Many stateful systems have initialization dependencies: a replica must connect to an already-running primary to bootstrap and begin replication, and quorum-based systems benefit from members joining in a controlled sequence. Ordered, ready-gated startup guarantees the prerequisite member (e.g., db-0 as primary) exists and is ready before dependents (db-1, db-2) try to join, avoiding race conditions and failed bootstraps. The slower rollout is an acceptable cost for correctness.

---

Persistent storage with StatefulSets

Theory

The feature that truly distinguishes StatefulSets is per-Pod persistent storage via volumeClaimTemplates. Instead of all Pods sharing one volume, the StatefulSet creates a separate PersistentVolumeClaim for each Pod, named deterministically (e.g., data-db-0, data-db-1). Each Pod gets its own volume that stays bound to that ordinal identity for the life of the StatefulSet.

The critical guarantee: if db-1 is deleted and recreated, it re-attaches to the same PVC (data-db-1) and thus the same data. Even on a different node, db-1 keeps its data. Deleting the StatefulSet does not delete the PVCs by default — this protects data from accidental loss, but means you must clean up PVCs manually when you truly want to discard the data. This is exactly the behavior a database needs.

Example

apiVersion: apps/v1
kind: StatefulSet
metadata: { name: db }
spec:
  serviceName: db
  replicas: 3
  selector: { matchLabels: { app: db } }
  template:
    metadata: { labels: { app: db } }
    spec:
      containers:
      - name: db
        image: postgres:16
        volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:               # one PVC PER Pod, auto-created
  - metadata: { name: data }
    spec:
      accessModes: ["ReadWriteOnce"]
      resources: { requests: { storage: 10Gi } }

kubectl get pvc
# NAME        STATUS   VOLUME   CAPACITY   ACCESS MODES
# data-db-0   Bound    pv-...   10Gi       RWO       <- one per ordinal
# data-db-1   Bound    pv-...   10Gi       RWO
# data-db-2   Bound    pv-...   10Gi       RWO

Exercises

1. (Beginner) What field in a StatefulSet creates a separate volume per Pod?
2. (Beginner) When db-2 is recreated, does it get a new empty volume or its old data?
3. (Intermediate) What happens to the PVCs when you delete a StatefulSet, and why is that the default behavior?
4. (Interview) Contrast how storage is handled in a Deployment versus a StatefulSet, and explain why the StatefulSet model is required for databases. (Hint: shared/ephemeral vs. per-identity durable.)

Answers

1. volumeClaimTemplates — the StatefulSet generates one PVC per Pod from the template (named <template>-<statefulset>-<ordinal>).
2. It re-attaches to its existing PVC (data-db-2) and keeps its old data. The volume is bound to the ordinal identity, not to a particular Pod instance.
3. By default the PVCs (and their data) are retained, not deleted, when the StatefulSet is deleted. This protects against accidental data loss for stateful workloads; you must delete the PVCs explicitly to reclaim the storage. (Newer persistentVolumeClaimRetentionPolicy can change this behavior.)
4. A Deployment typically uses ephemeral or shared volumes — all replicas are interchangeable and have no individually durable storage; if a Pod is replaced, its local data is gone. A StatefulSet provisions a dedicated, durable PVC per Pod identity that survives Pod restarts and rescheduling. Databases require each member to keep its own data persistently and reattach to it after any restart; only the per-identity durable storage of a StatefulSet provides that.

---

Headless services for StatefulSets

Theory

A normal Service gives a single ClusterIP that load-balances across all backing Pods — which hides individual Pods. StatefulSets need the opposite: clients must reach specific Pods by name. A headless Service (clusterIP: None) provides this. Instead of a single virtual IP, it creates DNS records for each backing Pod, returning the Pods' individual IPs.

A StatefulSet references a headless Service via spec.serviceName. This is what enables the stable per-Pod DNS names (db-0.db.namespace.svc.cluster.local). The headless Service does no load balancing — it is purely a DNS mechanism for discovering and addressing the individual stable Pods. (You can still create an additional normal Service on top if some clients want load-balanced access to the set, e.g., for read replicas.)

Example

apiVersion: v1
kind: Service
metadata:
  name: db
spec:
  clusterIP: None          # <- headless: no virtual IP, DNS per Pod instead
  selector: { app: db }
  ports:
  - port: 5432

# A headless Service returns each Pod's IP, not a single VIP:
kubectl run t --rm -it --image=busybox -- nslookup db.default.svc.cluster.local
# Address: 10.244.1.5   (db-0)
# Address: 10.244.2.7   (db-1)
# Address: 10.244.3.9   (db-2)

Exercises

1. (Beginner) What field makes a Service "headless"?
2. (Beginner) Does a headless Service load-balance traffic? What does it provide instead?
3. (Intermediate) How does a StatefulSet associate itself with its headless Service?
4. (Interview) Why must a StatefulSet use a headless Service rather than a standard ClusterIP Service to achieve stable per-Pod addressing? (Hint: a single VIP hides individual Pods.)

Answers

1. clusterIP: None.
2. No, it does not load-balance. It provides DNS records that resolve to the individual Pod IPs (and, with a StatefulSet, stable per-Pod DNS names), enabling direct addressing of specific Pods.
3. Through the StatefulSet's spec.serviceName, which names the headless (governing) Service; this drives the per-Pod DNS subdomain (<pod>.<serviceName>.<ns>.svc.cluster.local).
4. A standard ClusterIP Service exposes one virtual IP and load-balances to a random backing Pod, deliberately hiding which Pod you reach — the opposite of what stateful members need. A headless Service has no VIP and instead publishes per-Pod DNS records, so clients (and cluster members) can resolve and connect to a specific stable Pod (e.g., the primary, or a particular replica). That direct, identity-based addressing is required for replication, quorum, and membership.

---

4.4 DaemonSets

When you need exactly one copy of a Pod on every node — a log collector, a node monitor, a CNI agent — a DaemonSet is the tool. This subchapter covers node-level workloads.

---

DaemonSet use cases

Theory

Most controllers answer "how many replicas do I want?" A DaemonSet answers a different question: "I want this Pod running on every node (or every node matching a selector)." As nodes join the cluster, the DaemonSet automatically adds the Pod to them; as nodes leave, their Pods are removed. The count of Pods equals the count of (matching) nodes, not a fixed number.

This is the right model for node-level infrastructure — agents that must run once per machine to do their job:

• Log collectors (Fluentd, Fluent Bit) that gather every node's container logs.
• Node monitoring agents (node-exporter, Datadog agent).
• Networking components (CNI agents like Calico/Cilium, kube-proxy itself).
• Storage daemons (CSI node plugins).

If your workload is "one per node, tied to the node's resources or local data," it is a DaemonSet.

Example

apiVersion: apps/v1
kind: DaemonSet
metadata: { name: node-exporter }
spec:
  selector: { matchLabels: { app: node-exporter } }
  template:
    metadata: { labels: { app: node-exporter } }
    spec:
      containers:
      - name: node-exporter
        image: prom/node-exporter:latest
        ports: [ { containerPort: 9100 } ]

kubectl get daemonset node-exporter
# NAME            DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR
# node-exporter   3         3         3       3            3           <none>
# DESIRED == number of (matching) nodes, not a fixed replica count

Exercises

1. (Beginner) What question does a DaemonSet answer that a Deployment does not?
2. (Beginner) Name two real-world workloads that fit the DaemonSet model.
3. (Intermediate) When three new nodes join the cluster, what does a DaemonSet do automatically?
4. (Interview) Why is a DaemonSet, rather than a Deployment with many replicas, the correct way to run a per-node log collector? (Hint: placement guarantee vs. replica count, and node coverage.)

Answers

1. "On which nodes should this Pod run?" (one per node, automatically tracking node membership) — versus a Deployment's "how many interchangeable replicas do I want?".
2. Any two: log collectors (Fluentd/Fluent Bit), node metrics agents (node-exporter), CNI/network agents (Calico, Cilium), kube-proxy, CSI node plugins.
3. The DaemonSet controller automatically schedules its Pod onto each newly joined (matching) node, so the three new nodes each get a copy without any manual change.
4. A log collector must run on every node to capture that node's container logs. A Deployment only guarantees a count of Pods placed wherever the scheduler likes — it might put two replicas on one node and none on another, leaving nodes uncovered. A DaemonSet guarantees exactly one Pod per node and adapts automatically as nodes are added/removed, which is precisely the coverage guarantee a per-node agent requires.

---

Node-level workloads

Theory

DaemonSet Pods often need deeper access to the node itself than ordinary application Pods, because their job is to observe or manage the host. Such workloads commonly use:

• hostPath volumes to read node directories (e.g., /var/log/containers for logs, /proc and /sys for metrics).
• hostNetwork: true to use the node's network namespace directly (common for networking/monitoring agents that must see host interfaces or bind to host ports).
• Elevated privileges / securityContext (capabilities or privileged: true) when they manipulate kernel or device state (CNI, storage).

Because they run everywhere with elevated access, DaemonSets are powerful and security-sensitive — a compromised DaemonSet runs on every node with host access. They also typically set resource requests carefully and tolerate node taints so they can run even on specialized or control-plane nodes (covered next).

Example

spec:
  template:
    spec:
      hostNetwork: true            # use the node's network stack
      containers:
      - name: agent
        image: monitoring-agent:1.0
        securityContext:
          privileged: true         # node-level access (use sparingly!)
        volumeMounts:
        - name: varlog
          mountPath: /var/log
          readOnly: true
      volumes:
      - name: varlog
        hostPath:
          path: /var/log           # read the node's log directory

Exercises

1. (Beginner) Why do DaemonSet Pods often use hostPath volumes?
2. (Beginner) What does hostNetwork: true give a Pod?
3. (Intermediate) Why are DaemonSets considered more security-sensitive than typical application Deployments?
4. (Interview) A monitoring DaemonSet needs to read /proc and bind to a host port. Which Pod spec features enable this, and what is the security trade-off? (Hint: hostPath, hostNetwork/hostPort, privileges, blast radius.)

Answers

1. Because their purpose is to observe or manage the node, they mount host directories (logs in /var/log, metrics from /proc and /sys, container runtime sockets) directly via hostPath.
2. It places the Pod in the node's network namespace, so it shares the host's interfaces and IP and can see host-level traffic and bind host ports directly (instead of getting its own isolated Pod network).
3. They run on every node (broad coverage) and frequently require elevated access (hostPath, hostNetwork, privileged/capabilities). A compromise of a DaemonSet therefore potentially yields host-level access across the entire cluster — a much larger blast radius than a single app Pod.
4. Reading /proc uses a hostPath volume (or hostPID), and binding a host port uses hostNetwork: true (or hostPort); kernel-level access may need securityContext capabilities or privileged: true. The trade-off: these break Pod isolation and give host-level visibility/control, so a vulnerability in the agent can lead to node compromise on every node. Mitigate by granting the minimum needed (specific capabilities over full privilege, read-only mounts, restricted hostPaths), strict RBAC, and trusted images.

---

DaemonSet update strategies

Theory

Updating a DaemonSet means replacing the Pod on every node with a new version. DaemonSets support two updateStrategy types:

• RollingUpdate (the default): replace Pods gradually, node by node, controlled by maxUnavailable (how many node Pods may be down at once) and optionally maxSurge. This limits how much of your per-node coverage is disrupted simultaneously — important when the DaemonSet is critical infrastructure (e.g., the CNI), where taking all of them down at once would break the whole cluster.
• OnDelete: the controller does not automatically replace Pods on update. New Pods are only created when you manually delete the old ones. This gives operators full manual control over exactly when and where each node's Pod is upgraded — useful for highly sensitive node agents.

Example

spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1     # update one node's Pod at a time
# --- or ---
spec:
  updateStrategy:
    type: OnDelete          # update a node's Pod only when you delete it manually

kubectl rollout status daemonset/node-exporter   # watch a rolling update progress

Exercises

1. (Beginner) What are the two DaemonSet update strategies?
2. (Beginner) Under OnDelete, how does a node's Pod get the new version?
3. (Intermediate) Why does maxUnavailable matter especially for a CNI DaemonSet?
4. (Interview) When would you deliberately choose OnDelete over RollingUpdate for a DaemonSet? (Hint: critical node agents, manual scheduling of disruption, coordination with maintenance windows.)

Answers

1. RollingUpdate (default) and OnDelete.
2. Only when you manually delete the existing Pod on that node — the DaemonSet controller then creates a new (updated) Pod to replace it. Without deletion, the old Pod keeps running.
3. A CNI DaemonSet provides Pod networking on every node; if too many are unavailable simultaneously, networking breaks across those nodes. A small maxUnavailable (e.g., 1) ensures only one node's networking is briefly disrupted at a time, preserving cluster-wide connectivity during the upgrade.
4. Choose OnDelete for critical or fragile node agents where you must control exactly when and on which node a disruption occurs — e.g., coordinating with maintenance windows, validating the new version on one node before proceeding, or where automatic rolling could risk cluster stability. It trades automation for precise operator control over the rollout.

---

Tolerations with DaemonSets

Theory

Many nodes carry taints that repel ordinary Pods — control-plane nodes are tainted so workloads don't land on them, and special nodes (GPU, dedicated) may be tainted too. But node-level DaemonSets often must run even on these nodes (e.g., a monitoring or networking agent should cover the control plane too). To do so, the DaemonSet's Pods carry tolerations that allow them to be scheduled onto tainted nodes.

The DaemonSet controller automatically adds several tolerations to its Pods so they keep running through common node conditions (e.g., node.kubernetes.io/not-ready, unreachable, disk-pressure, memory-pressure) — because you generally want node agents to persist even when a node is unhealthy. To also run on control-plane nodes, you explicitly add a toleration for the control-plane taint. (Taints and tolerations are covered in depth in Chapter 8.)

Example

spec:
  template:
    spec:
      tolerations:
      # run on control-plane nodes despite their taint:
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule
      # tolerate node problems so the agent keeps running on unhealthy nodes:
      - key: node.kubernetes.io/not-ready
        operator: Exists
        effect: NoExecute

Exercises

1. (Beginner) What is a taint's purpose, and what lets a Pod ignore one?
2. (Beginner) Why might a DaemonSet need a toleration for the control-plane taint?
3. (Intermediate) Which tolerations does the DaemonSet controller add automatically, and why?
4. (Interview) A monitoring DaemonSet is missing from your control-plane nodes. What is the most likely cause and the fix? (Hint: taints vs. tolerations.)

Answers

1. A taint marks a node to repel Pods that don't tolerate it (controlling what may schedule there). A matching toleration on a Pod lets that Pod be scheduled onto (or remain on) the tainted node.
2. Control-plane nodes are tainted (e.g., node-role.kubernetes.io/control-plane:NoSchedule) to keep regular workloads off them. A DaemonSet that should cover all nodes (e.g., a metrics/log agent) needs a toleration for that taint to be placed on the control-plane nodes too.
3. It automatically adds tolerations for node-condition taints such as not-ready, unreachable, disk-pressure, memory-pressure, pid-pressure, and unschedulable. This is so node agents continue running even when a node becomes unhealthy or is cordoned — exactly when monitoring/networking is most needed.
4. The control-plane nodes carry a NoSchedule taint and the DaemonSet's Pods lack a matching toleration, so they are never scheduled there. The fix is to add a toleration for the control-plane taint (key: node-role.kubernetes.io/control-plane, operator: Exists, effect: NoSchedule) to the DaemonSet's Pod template.

---

4.5 Jobs and CronJobs

Not every workload runs forever. Batch tasks — migrations, backups, report generation — need to run to completion and stop. Jobs and CronJobs manage these finite and scheduled workloads. This subchapter covers them.

---

Job types: non-parallel, parallel, indexed

Theory

A Job runs one or more Pods until a specified number of them successfully complete, then stops. Unlike a Deployment (which keeps Pods running indefinitely), a Job is about finishing work. Two fields shape its behavior:

• completions: how many successful Pod completions are required for the Job to be done.
• parallelism: how many Pods may run at once.

These combine into three patterns:

Pattern	Settings	Use case
Non-parallel	completions: 1, parallelism: 1	A single one-off task (a migration, a backup).
Parallel with fixed completions	completions: N, parallelism: M	Process N work items, M at a time (a work queue).
Indexed	completionMode: Indexed	Each Pod gets a unique index (0..N-1) so it can process a specific shard.

Indexed Jobs are powerful for partitioned/batch workloads: each Pod reads its JOB_COMPLETION_INDEX to know which slice of work it owns (e.g., process file partition #3).

Example

apiVersion: batch/v1
kind: Job
metadata: { name: process-shards }
spec:
  completions: 5            # need 5 successful completions
  parallelism: 2            # run at most 2 Pods at once
  completionMode: Indexed   # each Pod gets a unique index 0..4
  template:
    spec:
      restartPolicy: OnFailure
      containers:
      - name: worker
        image: shard-processor:1.0
        command: ["sh", "-c", "echo processing shard $JOB_COMPLETION_INDEX"]

Exercises

1. (Beginner) How does a Job differ from a Deployment in terms of when it stops?
2. (Beginner) What do completions and parallelism control?
3. (Intermediate) You must process 100 files, 10 at a time, with each Pod handling a specific file partition. Which Job pattern and settings fit?
4. (Interview) How does an Indexed Job let each Pod know which portion of work to do, and why is this better than coordinating via an external queue for partitioned batch jobs? (Hint: JOB_COMPLETION_INDEX, no external coordination.)

Answers

1. A Deployment keeps Pods running indefinitely (long-lived services). A Job runs Pods until the required number complete successfully, then stops — it is for finite, completing work.
2. completions = number of successful Pod completions needed for the Job to be considered done; parallelism = maximum number of Pods running concurrently.
3. An Indexed parallel Job with completions: 100, parallelism: 10, completionMode: Indexed. Each Pod uses its index (0–99) to select the file partition it should process.
4. In Indexed mode, Kubernetes assigns each Pod a unique completion index exposed via the JOB_COMPLETION_INDEX env var (and pod hostname annotation). Each Pod deterministically maps its index to a work slice (partition N), so no external queue or locking is needed to divide the work. This is simpler and more robust for statically partitionable workloads: no coordination service to run or fail, no risk of two Pods claiming the same item, and the assignment is reproducible.

---

Job completion and backoff policies

Theory

Jobs must handle failure deliberately. Two mechanisms govern this:

• restartPolicy (in the Pod template) must be OnFailure or Never for Jobs (not Always). OnFailure restarts the container in place; Never lets the Pod fail and the Job creates a new Pod.
• backoffLimit: the number of retries before the Job is marked Failed. Each failed Pod increments a counter; retries use exponential backoff (10s, 20s, 40s, … capped at 6 minutes). When retries exceed backoffLimit (default 6), the Job gives up and is marked Failed.

Additional controls: activeDeadlineSeconds caps the total wall-clock time the Job may run (regardless of retries), and ttlSecondsAfterFinished auto-deletes the Job (and its Pods) some time after it finishes, preventing accumulation of completed Jobs. Newer podFailurePolicy lets you treat specific exit codes as retryable or fatal.

Example

apiVersion: batch/v1
kind: Job
metadata: { name: migrate }
spec:
  backoffLimit: 4                 # retry up to 4 times, then mark Failed
  activeDeadlineSeconds: 600      # hard cap: fail if running > 10 minutes total
  ttlSecondsAfterFinished: 3600   # auto-delete 1h after completion
  template:
    spec:
      restartPolicy: Never        # failed Pod -> new Pod (not in-place restart)
      containers:
      - name: migrate
        image: migrator:2.0

Exercises

1. (Beginner) Which two restartPolicy values are valid for a Job's Pods?
2. (Beginner) What does backoffLimit do when reached?
3. (Intermediate) Explain the difference between backoffLimit and activeDeadlineSeconds.
4. (Interview) Why is ttlSecondsAfterFinished valuable in a cluster that runs many Jobs, and what problem does it prevent? (Hint: accumulation of completed Job/Pod objects in etcd.)

Answers

1. OnFailure and Never (not Always).
2. When the number of failed Pod retries exceeds backoffLimit, the Job stops retrying and is marked Failed.
3. backoffLimit caps the number of retries (failures) before the Job is failed; retries use exponential backoff. activeDeadlineSeconds caps the total elapsed time the Job may be active regardless of how many retries have occurred — if exceeded, the Job (and its running Pods) is terminated and marked Failed. One bounds attempts; the other bounds wall-clock duration.
4. Completed/failed Jobs and their Pods persist as API objects until deleted, consuming etcd storage and cluttering kubectl get output. In a cluster running many (e.g., scheduled) Jobs, these accumulate indefinitely without cleanup. ttlSecondsAfterFinished automatically deletes a Job and its Pods a set time after it finishes, preventing unbounded object accumulation and the associated etcd bloat and operational noise.

---

CronJob scheduling syntax

Theory

A CronJob creates Jobs on a repeating schedule — it is Kubernetes' cron. It is the right tool for periodic tasks: nightly backups, hourly report generation, periodic cleanup. A CronJob's spec.schedule uses standard cron syntax: five fields for minute, hour, day-of-month, month, and day-of-week.

 ┌───────────── minute (0–59)
 │ ┌───────────── hour (0–23)
 │ │ ┌───────────── day of month (1–31)
 │ │ │ ┌───────────── month (1–12)
 │ │ │ │ ┌───────────── day of week (0–6, Sun=0)
 │ │ │ │ │
 * * * * *

Examples: 0 2 * * * = every day at 02:00; */15 * * * * = every 15 minutes; 0 0 * * 0 = midnight every Sunday. The CronJob controller checks the schedule and creates a Job (which creates Pods) at each scheduled time. You can also set a timeZone field (stable in recent versions) so schedules aren't tied to the controller's clock zone.

Example

apiVersion: batch/v1
kind: CronJob
metadata: { name: nightly-backup }
spec:
  schedule: "0 2 * * *"          # every day at 02:00
  timeZone: "Europe/Rome"        # interpret schedule in this zone
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
          - name: backup
            image: backup-tool:1.0
            command: ["/backup.sh"]

Exercises

1. (Beginner) What does a CronJob create on each scheduled tick?
2. (Beginner) Write the cron expression for "every day at midnight."
3. (Intermediate) Translate */15 9-17 * * 1-5 into plain English.
4. (Interview) Why does specifying a timeZone on a CronJob matter, and what subtle bug can arise without it? (Hint: controller clock vs. business hours, DST.)

Answers

1. A Job (which in turn creates the Pod(s) that run the task).
2. 0 0 * * *.
3. Every 15 minutes, during the hours 09:00–17:00, Monday through Friday (i.e., every 15 minutes within business hours on weekdays).
4. Without timeZone, the schedule is interpreted in the time zone of the CronJob controller (historically UTC), which may differ from the intended business/local zone — so a "2 AM backup" could run at the wrong local time. It also makes daylight-saving transitions confusing. Setting timeZone pins the schedule to a specific zone so the Job fires at the intended local time consistently, including across DST changes.

---

ConcurrencyPolicy and history limits

Theory

Two practical concerns arise with recurring Jobs: what if a scheduled run is still going when the next is due, and how many old Jobs should be kept?

concurrencyPolicy governs overlap:

• Allow (default): permit concurrent runs — a new Job starts even if the previous is still running.
• Forbid: skip the new run if the previous Job is still active (no overlap).
• Replace: cancel the currently running Job and start a new one.

History limits govern retention of finished Jobs:

• successfulJobsHistoryLimit (default 3) and failedJobsHistoryLimit (default 1) cap how many completed/failed Jobs the CronJob keeps for inspection; older ones are garbage-collected.

Also relevant: startingDeadlineSeconds controls how long a missed run may still be started after its scheduled time (e.g., if the controller was down), and suspend: true pauses a CronJob without deleting it.

Example

spec:
  schedule: "*/5 * * * *"
  concurrencyPolicy: Forbid          # don't start a new run if one is still going
  startingDeadlineSeconds: 120       # allow a late start within 2 minutes
  successfulJobsHistoryLimit: 5       # keep last 5 successful Jobs
  failedJobsHistoryLimit: 3          # keep last 3 failed Jobs
  suspend: false                     # set true to pause scheduling
  jobTemplate: { ... }

Exercises

1. (Beginner) What are the three values of concurrencyPolicy and what does each do?
2. (Beginner) What do the two history-limit fields control?
3. (Intermediate) A backup Job takes 8 minutes but is scheduled every 5 minutes. Which concurrencyPolicy prevents overlapping backups, and what is the consequence?
4. (Interview) Explain startingDeadlineSeconds and a scenario where setting it too low (or leaving it default) causes scheduled runs to be silently skipped. (Hint: controller downtime and missed schedules.)

Answers

1. Allow — permit concurrent runs (default); Forbid — skip the new run if the previous is still active; Replace — terminate the running Job and start the new one.
2. successfulJobsHistoryLimit and failedJobsHistoryLimit cap how many completed successful and failed Jobs (and their Pods) the CronJob retains for inspection before garbage-collecting older ones.
3. Forbid prevents overlap — a new backup won't start while the previous 8-minute backup is still running. The consequence is that scheduled runs which fall during an in-progress backup are skipped, so backups effectively run roughly every ~8–10 minutes instead of every 5, reducing frequency.
4. startingDeadlineSeconds is the window after a scheduled time during which a Job may still be started if it could not start on time (e.g., the CronJob controller was down or busy). If the controller is down longer than this deadline when a run was due, that run is missed and skipped rather than started late. If set too low (or with the default and many missed schedules), runs that the controller couldn't launch in time are silently dropped. Additionally, if more than 100 schedules are missed without a deadline, the controller stops scheduling and logs an error — so a too-restrictive or unset deadline combined with downtime leads to skipped executions.

---