2. Kubernetes Architecture

Every Kubernetes cluster is split into two planes: a control plane that makes global decisions and stores the cluster's desired state, and a set of worker nodes that actually run the containers. This chapter dissects each component, traces how they communicate through the API server, and covers the add-ons that turn a bare cluster into a usable platform. Understanding this architecture is what separates someone who can run kubectl apply from someone who can debug why a Pod will not start.

---

2.1 Control Plane Components

The control plane is the cluster's brain. It exposes the API, persists state, schedules workloads, and runs the control loops that keep reality matching intent. This subchapter examines each control plane process in turn.

---

kube-apiserver

Theory

Imagine a company where every decision — hiring, spending, scheduling — must go through a single front desk that validates the request, records it in the official ledger, and notifies the relevant departments. Nothing is "real" until the front desk writes it down. In Kubernetes, that front desk is the kube-apiserver.

The API server is the only component that talks to etcd (the datastore). Every other component — the scheduler, controllers, kubelets, and your kubectl — communicates through the API server, never directly with each other or with storage. This hub-and-spoke design means the API server is the single point of authentication, authorization, validation, and audit.

Key properties:

• RESTful: resources (Pods, Services, etc.) are exposed as HTTP endpoints you can GET, POST, PUT, PATCH, and DELETE.
• Stateless: the API server holds no state itself; all state lives in etcd. This means you can run multiple API server replicas behind a load balancer for high availability.
• Declarative: clients submit the desired state of an object; the API server validates and stores it.

Example

# kubectl is just an HTTP client for the API server.
# This flag shows the actual REST calls kubectl makes:
kubectl get pods -v=8

# You can talk to the API server directly with curl via a proxy:
kubectl proxy --port=8080 &
curl http://localhost:8080/api/v1/namespaces/default/pods   # list Pods as JSON

# Inspect the API server's own health endpoint:
curl http://localhost:8080/healthz   # returns "ok" when healthy

            +-------------------------------------------+
            |              kube-apiserver               |
 kubectl -->|  authn -> authz -> admission -> validate  |--> etcd
 kubelet -->|        (the ONLY writer to etcd)          |
scheduler ->+-------------------------------------------+

Exercises

1. (Beginner) Which Kubernetes component is the only one that reads from and writes to etcd directly?
2. (Beginner) Why can you run multiple replicas of the kube-apiserver without a leader election, but not multiple active schedulers?
3. (Intermediate) Using kubectl get pods -v=8, identify the HTTP method and path used to list Pods in the default namespace.
4. (Interview) The API server is described as "stateless." Where is cluster state actually stored, and what advantage does keeping the API server stateless provide for availability? (Hint: think about horizontal scaling and load balancing.)

Answers

1. The kube-apiserver is the only component that communicates with etcd directly. All other components go through the API server.
2. The API server is stateless — every replica reads/writes the same etcd backend, so requests can be load-balanced across replicas without coordination. The scheduler and controllers, by contrast, take actions (creating Pods, binding them to nodes); running several active copies would cause duplicate or conflicting actions, so they use leader election to ensure only one is active at a time.
3. The request is GET /api/v1/namespaces/default/pods. The -v=8 output shows a line like GET https://<host>:6443/api/v1/namespaces/default/pods?limit=500.
4. Cluster state is stored in etcd. Because the API server holds no state of its own, any replica can serve any request, and you can place multiple replicas behind a load balancer (active-active). If one fails, traffic simply routes to another with no failover delay or state migration. This is the foundation of control plane high availability.

---

etcd as the cluster store

Theory

If the API server is the front desk, etcd is the official ledger locked in the vault. It is a distributed, consistent key-value store that holds the entire state of the cluster: every Pod, Service, ConfigMap, Secret, and the desired-state specs you submit. If etcd is lost and unrecoverable, the cluster's state is gone.

etcd uses the Raft consensus algorithm to stay consistent across multiple replicas. Raft elects a leader; all writes go through the leader, which replicates them to followers. A write is only acknowledged once a majority (quorum) of members have persisted it. This is why etcd clusters use an odd number of members (3, 5, 7): a 3-node cluster tolerates 1 failure, a 5-node cluster tolerates 2.

Because consistency requires a quorum, etcd prioritizes consistency over availability (CP in CAP terms). If quorum is lost, etcd stops accepting writes rather than risk diverging state.

Example

# etcd stores everything under /registry. Inspect a key directly:
ETCDCTL_API=3 etcdctl get /registry/pods/default/my-pod \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Check cluster health and which member is the leader:
ETCDCTL_API=3 etcdctl endpoint status --write-out=table \
  --endpoints=https://127.0.0.1:2379 ...

Raft quorum math (why odd numbers):
  3 members -> quorum = 2 -> tolerates 1 failure
  5 members -> quorum = 3 -> tolerates 2 failures
  4 members -> quorum = 3 -> tolerates 1 failure  (no better than 3, more cost)

Exercises

1. (Beginner) What kind of data store is etcd, and what does it hold for a Kubernetes cluster?
2. (Beginner) Why are etcd clusters deployed with an odd number of members?
3. (Intermediate) A 5-member etcd cluster loses 3 members simultaneously. Can it still accept writes? Explain using the concept of quorum.
4. (Interview) etcd is a CP system in CAP terms. What does it sacrifice when a network partition occurs, and why is that the correct trade-off for a cluster datastore? (Hint: consider what would happen if two halves of a partition both accepted writes.)

Answers

1. etcd is a distributed, strongly consistent key-value store. It holds the complete state of the cluster: all API objects (Pods, Services, Deployments, Secrets, ConfigMaps, etc.) and their desired/observed state.
2. An odd number maximizes fault tolerance per node. Quorum is (N/2)+1. A 3-node cluster (quorum 2) tolerates 1 failure; a 4-node cluster (quorum 3) also only tolerates 1 failure but costs an extra node. Odd numbers avoid paying for capacity that does not improve tolerance.
3. No. A 5-member cluster has quorum = 3. Losing 3 members leaves only 2 available, which is below quorum. etcd will stop accepting writes (reads may be served stale depending on configuration) until quorum is restored, to avoid split-brain.
4. During a network partition, etcd sacrifices availability on the minority side: the partition without a quorum stops serving writes. This is correct because allowing both sides to accept writes independently would create divergent, conflicting cluster states (split-brain) — e.g., two API servers each believing different Pods should exist. Consistency is non-negotiable for a datastore that is the single source of truth, so refusing writes is safer than diverging.

---

kube-scheduler

Theory

When you create a Pod, it does not immediately run anywhere — it is born "unscheduled," with no node assigned. The kube-scheduler is the matchmaker that watches for these unassigned Pods and decides which node each should run on. It does not start the container itself; it merely writes the chosen node into the Pod's spec.nodeName. The kubelet on that node then takes over.

Scheduling happens in two phases:

1. Filtering (Predicates): eliminate nodes that cannot run the Pod. A node is filtered out if it lacks sufficient CPU/memory, fails to match a nodeSelector, has a taint the Pod does not tolerate, or has port conflicts.
2. Scoring (Priorities): rank the remaining feasible nodes. Factors include spreading Pods across nodes, packing onto fuller nodes, image locality, and affinity preferences. The highest-scoring node wins (ties broken randomly).

Example

# This Pod requests resources and a node label; the scheduler must find
# a node that satisfies BOTH before binding.
apiVersion: v1
kind: Pod
metadata:
  name: web
spec:
  nodeSelector:
    disktype: ssd          # filtering: only nodes labeled disktype=ssd qualify
  containers:
  - name: web
    image: nginx
    resources:
      requests:
        cpu: "500m"        # filtering: node must have >= 0.5 CPU free
        memory: "256Mi"

# Watch the scheduler's decision in the Pod's events:
kubectl describe pod web | grep -A2 Events
# Normal  Scheduled  ...  Successfully assigned default/web to node-2

Exercises

1. (Beginner) After you create a Pod, what does the scheduler actually write to make it run on a node?
2. (Beginner) Name the two phases of the scheduling process and describe what each accomplishes.
3. (Intermediate) A Pod stays in Pending state forever. List three scheduler-related reasons this could happen and how you would diagnose each.
4. (Interview) The scheduler only sets spec.nodeName; it never starts the container. Explain the separation of concerns between the scheduler and the kubelet, and why this decoupling is valuable. (Hint: think about which component owns node-local execution.)

Answers

1. The scheduler writes the chosen node's name into the Pod's spec.nodeName field (via a Binding subresource). It does not launch the container.
2. Filtering (predicates): remove nodes that cannot run the Pod (insufficient resources, taint mismatch, selector mismatch, port conflict). Scoring (priorities): rank the feasible nodes by preference (spreading, affinity, resource balance, image locality); the highest-scoring node is selected.
3. (a) Insufficient resources — no node has enough free CPU/memory; check kubectl describe pod events for "Insufficient cpu/memory". (b) Unsatisfiable nodeSelector/affinity — no node has the required labels; check node labels with kubectl get nodes --show-labels. (c) Taints without tolerations — all nodes are tainted (e.g., control-plane taint); check kubectl describe node for taints. The Pod's events from kubectl describe pod usually state the exact reason.
4. The scheduler owns the global placement decision (which node), while the kubelet owns local execution (pulling images, starting containers, reporting status) on its own node. Decoupling them means: the scheduler can be replaced or extended without touching node execution; node failures are isolated to the kubelet; and the scheduler remains a pure decision-maker working only through the API server, keeping it stateless and replaceable. The kubelet reacts to the bound Pod via the level-triggered watch model.

---

kube-controller-manager

Theory

Kubernetes is built on controllers — independent control loops that each watch a resource type and work to make actual state match desired state. Rather than running dozens of separate processes, Kubernetes bundles the core controllers into a single binary, the kube-controller-manager, for operational simplicity. Inside it run the Node controller, ReplicaSet controller, Deployment controller, Job controller, EndpointSlice controller, ServiceAccount controller, and many more.

Each controller follows the same reconciliation pattern: observe current state via the API server's watch mechanism, compare it to desired state, and act to close the gap. For example, the ReplicaSet controller notices a ReplicaSet wants 3 Pods but only 2 exist, and creates one more.

In a high-availability setup you run multiple controller-manager replicas, but only one is active at a time via leader election — because controllers take actions, and duplicate actions would cause chaos.

Example

kube-controller-manager  (one process, many loops)
 ├── Node controller          watches Nodes, marks NotReady, evicts Pods
 ├── ReplicaSet controller    maintains desired Pod count
 ├── Deployment controller    manages ReplicaSets for rollouts
 ├── Job controller           runs Pods to completion
 ├── EndpointSlice controller keeps Service endpoints current
 └── ServiceAccount controller creates default SAs / tokens

# Leader election is recorded in a Lease object:
kubectl -n kube-system get lease kube-controller-manager
# NAME                      HOLDER                         AGE
# kube-controller-manager   control-plane-1_a1b2c3...      10d

Exercises

1. (Beginner) What is a "controller" in Kubernetes, in one sentence?
2. (Beginner) Why are many controllers packaged into a single kube-controller-manager binary?
3. (Intermediate) Name three controllers that run inside the controller-manager and describe what each one keeps in sync.
4. (Interview) Why do controller-manager replicas use leader election while the API server does not? (Hint: distinguish components that decide/act from components that serve.)

Answers

1. A controller is a control loop that continuously watches a resource and reconciles actual cluster state toward the declared desired state.
2. For operational simplicity: bundling them into one process reduces the number of binaries to deploy, configure, secure, and monitor, while each controller still runs as a logically independent loop.
3. Examples: Node controller keeps node health/status in sync and evicts Pods from dead nodes; ReplicaSet controller keeps the running Pod count equal to the desired replicas; Job controller ensures a Job's Pods run to successful completion. (Also acceptable: Deployment, EndpointSlice, ServiceAccount controllers.)
4. Controllers act on the cluster (create/delete Pods, evict, update endpoints). Two active controller-managers would issue duplicate or conflicting actions, so leader election ensures exactly one is active. The API server only serves and stores requests statelessly, so multiple replicas can run active-active behind a load balancer without conflict.

---

cloud-controller-manager

Theory

Kubernetes needs to interact with the underlying cloud provider for certain operations: provisioning a load balancer when you create a LoadBalancer Service, attaching storage volumes, or learning a node's region/zone. Originally this cloud-specific code lived inside the core components, which made the project hard to maintain and bound to specific clouds. The cloud-controller-manager (CCM) extracts all cloud-provider-specific logic into a separate, pluggable binary.

This separation means cloud vendors can develop and release their integration on their own schedule, and the core Kubernetes binaries stay vendor-neutral. The CCM runs controllers such as:

• Node controller (cloud part): checks the cloud API to confirm whether a node that stopped responding was actually deleted from the cloud.
• Route controller: configures network routes in the cloud's network for Pod traffic.
• Service controller: creates/updates/deletes cloud load balancers for LoadBalancer Services.

On a self-managed bare-metal cluster with no cloud provider, the CCM is simply absent.

Example

# Creating this Service on a cloud cluster triggers the CCM's Service
# controller to provision an actual cloud load balancer (ELB, GLB, etc.).
apiVersion: v1
kind: Service
metadata:
  name: frontend
spec:
  type: LoadBalancer       # CCM sees this and calls the cloud's LB API
  selector:
    app: frontend
  ports:
  - port: 80
    targetPort: 8080

kubectl get svc frontend
# NAME       TYPE           CLUSTER-IP    EXTERNAL-IP        PORT(S)
# frontend   LoadBalancer   10.96.0.40    a1b2c3.elb.aws...  80:31234/TCP
# The EXTERNAL-IP is populated by the CCM after the cloud LB is ready.

Exercises

1. (Beginner) What problem does separating cloud-specific logic into the cloud-controller-manager solve?
2. (Beginner) On a bare-metal cluster with no cloud provider, is the cloud-controller-manager present?
3. (Intermediate) You create a type: LoadBalancer Service on a self-managed bare-metal cluster and the EXTERNAL-IP stays <pending> forever. Explain why, and name a tool that addresses this.
4. (Interview) Why was moving cloud-provider code out of the core kube-controller-manager beneficial for both the Kubernetes project and cloud vendors? (Hint: think about release cadence and code ownership.)

Answers

1. It decouples vendor-specific integration (load balancers, volumes, node metadata, routes) from the core Kubernetes codebase, keeping the core vendor-neutral and letting cloud providers maintain their own integrations independently.
2. No. With no cloud provider configured, there is no cloud-controller-manager — its controllers have nothing to integrate with.
3. There is no cloud provider to fulfill the LoadBalancer request, so no external load balancer is provisioned and EXTERNAL-IP never populates. A tool like MetalLB (or kube-vip) implements LoadBalancer Services on bare metal by assigning IPs from a configured pool and announcing them via ARP/BGP.
4. The core project no longer has to carry, test, and release code for every cloud, reducing maintenance burden and avoiding vendor lock-in. Cloud vendors gain ownership of their integration and can fix bugs or add features on their own release cadence without waiting for a Kubernetes release, improving velocity for both sides.

---

2.2 Worker Node Components

Worker nodes are where containers actually run. Each node runs a small set of components that register the node, execute Pods, wire up networking, and talk to a container runtime. This subchapter covers them.

---

kubelet

Theory

The kubelet is the primary node agent — the on-the-ground supervisor present on every node, including control plane nodes. Its job is narrow but critical: ensure that the containers described in the Pods assigned to its node are running and healthy. It is the bridge between the cluster's desired state and the actual processes on the machine.

The kubelet watches the API server for Pods bound to its node (spec.nodeName matches). For each such Pod, it:

1. Pulls the required container images.
2. Asks the container runtime (via the CRI) to create and start the containers.
3. Runs the Pod's liveness, readiness, and startup probes.
4. Reports the Pod's status and the node's health (heartbeats) back to the API server.

Importantly, the kubelet only manages Pods it knows about from the API server (plus "static Pods" defined by local manifest files). It does not manage arbitrary containers you start manually with docker run.

Example

# Static Pods: the kubelet runs any manifest placed in this directory,
# even without a working API server. Control plane components often run this way.
ls /etc/kubernetes/manifests/
# etcd.yaml  kube-apiserver.yaml  kube-controller-manager.yaml  kube-scheduler.yaml

# The kubelet sends node heartbeats; a missed heartbeat eventually marks
# the node NotReady:
kubectl get nodes
# NAME      STATUS   ROLES           AGE   VERSION
# node-1    Ready    <none>          12d   v1.30.2
# node-2    NotReady <none>          12d   v1.30.2   <- kubelet stopped reporting

Exercises

1. (Beginner) On which nodes does the kubelet run?
2. (Beginner) Does the kubelet manage containers you start manually with docker run? Why or why not?
3. (Intermediate) What is a "static Pod," where is it defined, and why are control plane components often run as static Pods?
4. (Interview) A node shows NotReady, but the application Pods on it appear to keep serving traffic for several minutes. Explain what is happening from the kubelet's and control plane's perspectives. (Hint: heartbeats vs. running containers vs. eviction timeout.)

Answers

1. The kubelet runs on every node in the cluster, including control plane nodes (so they can run static Pods like the API server).
2. No. The kubelet only manages Pods assigned to its node by the API server, plus static Pods from its manifest directory. Containers started directly via the runtime/docker run are outside Kubernetes' knowledge, so the kubelet ignores them.
3. A static Pod is a Pod defined by a manifest file in the kubelet's watched directory (default /etc/kubernetes/manifests). The kubelet runs it directly without involving the scheduler or controllers, and it works even before/without the API server — which is exactly why bootstrap control plane components (apiserver, etcd, scheduler, controller-manager) run as static Pods.
4. The kubelet on the failing node has stopped sending heartbeats (e.g., it crashed or lost network), so the node controller marks the node NotReady. However, the containers themselves may still be running locally and reachable, so traffic continues. The control plane waits for the eviction timeout (default ~5 minutes via the unreachable taint) before marking the Pods for deletion and rescheduling them elsewhere. During that window the Pods exist in both places conceptually, which is why they keep serving until eviction kicks in.

---

kube-proxy

Theory

Pods are ephemeral — they come and go, each with its own IP. A Service gives a stable virtual IP (ClusterIP) that fronts a changing set of Pods. But something has to actually route traffic sent to that virtual IP to a real Pod. That something is kube-proxy, a network agent running on every node.

kube-proxy watches the API server for Services and their EndpointSlices (the current set of healthy backing Pod IPs) and programs the node's networking rules so that traffic to a Service's ClusterIP is load-balanced to one of the backend Pods. It typically operates in iptables mode (writes iptables rules) or IPVS mode (uses the kernel's IP Virtual Server for better performance at scale). It does not proxy packets in userspace in modern modes — it configures the kernel to do the routing.

Example

Request to Service ClusterIP 10.96.0.10:80
        |
        v  (kube-proxy programmed kernel rules on this node)
   DNAT to one of:  10.244.1.5:8080
                    10.244.2.7:8080   <- chosen pod (random/round-robin)
                    10.244.3.9:8080

# In iptables mode, you can see the rules kube-proxy created for a Service:
iptables -t nat -L KUBE-SERVICES -n | grep 10.96.0.10
# Each Service ClusterIP maps to a chain that DNATs to backend Pod IPs.

# Check kube-proxy is running on every node (it's a DaemonSet):
kubectl -n kube-system get pods -l k8s-app=kube-proxy -o wide

Exercises

1. (Beginner) What problem does kube-proxy solve, given that Pod IPs change constantly?
2. (Beginner) Name the two common modes kube-proxy operates in.
3. (Intermediate) Traffic to a Service's ClusterIP is failing, but the backend Pods are healthy and reachable by their Pod IPs. How would you investigate kube-proxy as the cause?
4. (Interview) Modern kube-proxy modes program the kernel (iptables/IPVS) rather than proxying packets in userspace. Why is that a significant performance improvement? (Hint: think about context switches and per-packet handling.)

Answers

1. kube-proxy implements the Service abstraction at the network level: it programs each node so that traffic to a Service's stable virtual IP is forwarded and load-balanced to the current set of healthy backend Pod IPs, insulating clients from Pod churn.
2. iptables mode (programs iptables NAT rules) and IPVS mode (uses the kernel's IP Virtual Server, more efficient with many Services).
3. Verify kube-proxy is running on the relevant node(s) (kubectl -n kube-system get pods -l k8s-app=kube-proxy), check its logs, confirm the Service has populated EndpointSlices (kubectl get endpointslices), and inspect the kernel rules (iptables -t nat -L | grep <clusterIP> or ipvsadm -Ln). Missing rules or empty endpoints point to kube-proxy or selector/label issues.
4. Userspace proxying copies every packet between kernel and a userspace process, incurring a context switch and data copy per packet — expensive and a bottleneck under load. Programming iptables/IPVS lets the kernel route packets directly in the network stack with no userspace round trip, dramatically reducing per-packet overhead and scaling to far more connections and Services.

---

Container runtime interface (CRI)

Theory

The kubelet does not know how to start a container by itself — it delegates that to a container runtime. Early on, Kubernetes had hard-coded support for Docker. To support multiple runtimes cleanly, the project defined the Container Runtime Interface (CRI): a standard gRPC API the kubelet uses to talk to any compliant runtime. This is the same decoupling philosophy as the CRI's cousins, CNI (networking) and CSI (storage).

The CRI defines two gRPC services:

• RuntimeService: lifecycle of Pods (sandboxes) and containers — create, start, stop, remove, exec, list.
• ImageService: pulling, listing, and removing container images.

Because of the CRI, the kubelet is runtime-agnostic. As long as a runtime implements the CRI, the kubelet can use it. This is what allowed Kubernetes to remove the built-in "dockershim" in v1.24 — Docker itself does not implement the CRI directly.

Example

+---------+   CRI (gRPC)   +--------------+   OCI runtime   +-------+
| kubelet | -------------> | containerd / | -------------> | runc  | -> container
|         |   RuntimeSvc   | CRI-O        |                +-------+
|         |   ImageSvc     +--------------+
+---------+

# crictl is a CRI-compliant CLI for debugging at the node level
# (analogous to docker, but speaks CRI directly to the runtime):
crictl ps                 # list running containers known to the runtime
crictl images             # list images
crictl pods               # list Pod sandboxes

Exercises

1. (Beginner) What does the CRI let the kubelet do that hard-coded runtime support did not?
2. (Beginner) Name the two gRPC services the CRI defines and what each handles.
3. (Intermediate) Why was Kubernetes able to remove "dockershim" in v1.24 without breaking the ability to run containers? What did clusters switch to?
4. (Interview) The CRI, CNI, and CSI all follow the same architectural pattern. Describe that pattern and explain why Kubernetes favors it. (Hint: stable interface vs. swappable implementation.)

Answers

1. The CRI lets the kubelet talk to any compliant container runtime through a single standard gRPC API, making the kubelet runtime-agnostic instead of tied to one specific runtime.
2. RuntimeService handles Pod sandbox and container lifecycle (create/start/stop/remove/exec/list); ImageService handles image operations (pull/list/remove).
3. Docker does not implement the CRI natively; the kubelet used a shim ("dockershim") to translate. Maintaining that shim in-tree was a burden, so it was removed. Clusters switched to CRI-native runtimes — most commonly containerd (which Docker already uses under the hood) or CRI-O — so containers still run normally.
4. The pattern is a stable, well-defined interface with swappable implementations: Kubernetes core depends only on the interface (CRI for runtimes, CNI for networking, CSI for storage), while vendors provide pluggable implementations behind it. This keeps the core small and vendor-neutral, lets implementations evolve and ship independently, and allows operators to choose the best implementation for their needs without changing Kubernetes itself.

---

containerd and CRI-O

Theory

With the CRI defining the contract, two runtimes dominate real clusters: containerd and CRI-O. Both implement the CRI and both ultimately use a low-level OCI runtime (usually runc) to actually create the container process via Linux namespaces and cgroups.

• containerd: a general-purpose, industry-standard runtime originally extracted from Docker and donated to the CNCF. It is broader in scope than just Kubernetes (it can be embedded in other tools), supports a plugin model, and is the default runtime in most managed Kubernetes offerings. Its CRI support is provided by a built-in plugin.
• CRI-O: a lightweight runtime built specifically for Kubernetes by Red Hat and the community. It implements only what the CRI requires — nothing more — which keeps it minimal and tightly aligned to Kubernetes releases. It is the default in OpenShift.

In practice both are production-grade; the choice often comes down to distribution defaults and operational preference.

Example

Aspect	containerd	CRI-O
Origin	Extracted from Docker, CNCF graduated	Built by Red Hat for Kubernetes
Scope	General-purpose, embeddable	Kubernetes-only
CRI support	Via built-in CRI plugin	Native, the entire purpose
OCI runtime	runc (default)	runc (default)
Common in	GKE, EKS, AKS, kind, k3s	OpenShift, many on-prem
CLI for debugging	ctr, crictl	crictl

# Find out which runtime a node uses:
kubectl get nodes -o wide
# ...  CONTAINER-RUNTIME
# ...  containerd://1.7.13

Exercises

1. (Beginner) What do both containerd and CRI-O ultimately use to create the container process?
2. (Beginner) Which runtime was built specifically and only for Kubernetes?
3. (Intermediate) How would you determine which container runtime a given node is using?
4. (Interview) containerd has broader scope than CRI-O, while CRI-O is minimal. Discuss the trade-offs of a general-purpose runtime versus a purpose-built one for a Kubernetes platform team. (Hint: think features and reuse vs. surface area and alignment.)

Answers

1. Both use a low-level OCI runtime, typically runc, which sets up the Linux namespaces and cgroups for the container process.
2. CRI-O was built specifically and exclusively for Kubernetes (it implements only the CRI).
3. Run kubectl get nodes -o wide and read the CONTAINER-RUNTIME column (e.g., containerd://1.7.13 or cri-o://1.30.0). On the node itself, crictl version also reports the runtime.
4. A general-purpose runtime (containerd) offers broader functionality, a plugin ecosystem, and reuse outside Kubernetes, but carries more surface area and features a Kubernetes-only platform may never use. A purpose-built runtime (CRI-O) has a smaller footprint, fewer moving parts, and tracks Kubernetes releases tightly — reducing attack surface and version drift — at the cost of not being reusable elsewhere. Teams optimizing for minimalism and tight K8s alignment lean CRI-O; teams valuing ecosystem breadth and broad tooling lean containerd.

---

2.3 Cluster Communication

This subchapter zooms out to how the components actually talk to each other: everything flows through the API server, using an efficient watch mechanism, and every request passes through authentication, authorization, and admission before it is stored.

---

API server as the central hub

Theory

A recurring theme is worth stating explicitly: Kubernetes uses a hub-and-spoke communication model with the API server at the center. No component talks directly to another. The scheduler does not call the kubelet; the controller-manager does not call etcd. Instead, every component reads from and writes to the API server, and reacts to changes it observes there.

This design has profound benefits:

• Single security boundary: authn, authz, admission, and audit happen in exactly one place.
• Loose coupling: components don't need to know about each other's network addresses or even existence; they only know the API server.
• Extensibility: a new controller or operator just watches and updates objects — no changes to existing components required.

The flip side: the API server is on the critical path for everything, so its availability and performance are paramount.

Example

        +--------------------+
        |   kube-apiserver   |<---- kubectl (you)
        +---------+----------+
          ^   ^   ^   ^
          |   |   |   |   (everyone watches/writes here, never each other)
     +----+ +-+ +-+ +----+
     |sched| |cm| |kubelet x N
     +-----+ +--+ +-------+

# Example of the indirection: the scheduler "tells" the kubelet to run a
# Pod by writing spec.nodeName via the API server — it never calls the kubelet.
kubectl get pod web -o jsonpath='{.spec.nodeName}'   # node-2

Exercises

1. (Beginner) In the hub-and-spoke model, what is the hub?
2. (Beginner) Does the scheduler ever directly contact the kubelet? How does its decision reach the kubelet?
3. (Intermediate) Give two concrete benefits of routing all communication through the API server instead of letting components talk peer-to-peer.
4. (Interview) The hub-and-spoke model makes the API server a critical dependency. What architectural measures keep this from being a single point of failure? (Hint: stateless replicas, etcd quorum, load balancing.)

Answers

1. The kube-apiserver is the hub; all other components are spokes.
2. No, the scheduler never contacts the kubelet directly. It writes the binding (spec.nodeName) to the API server; the kubelet on that node observes the change via its watch and acts on it.
3. Any two: a single place to enforce authentication/authorization/admission/audit; loose coupling (components only need to know the API server, not each other); easy extensibility (new controllers just watch/update objects); consistent validation of all writes.
4. Run multiple stateless API server replicas behind a load balancer (active-active), back them with a multi-member etcd cluster that maintains quorum (tolerating member failures), and place the control plane across failure domains/availability zones. Because the API server holds no state, any replica can serve any request, so losing one causes no data loss and minimal disruption.

---

Watch and event-driven updates

Theory

If every component had to constantly poll the API server ("any changes yet? any changes yet?"), the load would be enormous and updates would lag. Instead Kubernetes uses watches: a client opens a long-lived connection and the API server pushes notifications whenever a matching object is created, updated, or deleted. This is the event-driven backbone of the whole system.

Under the hood:

• Each object has a resourceVersion (derived from etcd's revision). A watch can resume from a given resourceVersion, so a client that reconnects does not miss changes.
• Clients typically use an informer (from client-go): it does an initial LIST to build a local cache, then a WATCH to keep the cache updated incrementally. Controllers read from this local cache instead of hammering the API server.
• Watches make the system level-triggered in spirit but efficient in practice: controllers are woken by events but always reconcile against current state.

Example

# Observe the watch stream yourself — this blocks and prints changes live:
kubectl get pods --watch
# NAME    READY   STATUS              AGE
# web     0/1     Pending             0s
# web     0/1     ContainerCreating   1s
# web     1/1     Running             4s   <- pushed as state changes

LIST + WATCH pattern (informer):
  1. LIST all Pods at resourceVersion=1000  -> populate local cache
  2. WATCH from resourceVersion=1000        -> receive deltas (ADD/UPDATE/DELETE)
  3. On disconnect, re-WATCH from last seen resourceVersion (no missed events)

Exercises

1. (Beginner) Why is the watch mechanism preferable to having every component poll the API server?
2. (Beginner) What is the purpose of an object's resourceVersion in the context of watches?
3. (Intermediate) Describe the LIST-then-WATCH pattern an informer uses and why the initial LIST is necessary.
4. (Interview) A controller's watch connection drops for 10 seconds and several objects change during the gap. Explain how the controller avoids missing those changes when it reconnects. (Hint: resourceVersion and re-list/resync.)

Answers

1. Watches push changes to clients over a long-lived connection, so clients learn of changes immediately without repeated polling. Polling would create high, constant load on the API server and introduce latency between a change and its detection.
2. The resourceVersion is a monotonic marker (from etcd) identifying a point in the object's change history. A watch can start/resume from a specific resourceVersion so the client receives all subsequent changes and does not miss or duplicate events across reconnects.
3. The informer first issues a LIST to fetch the full current set of objects and build a local cache (and learn the current resourceVersion). It then opens a WATCH from that resourceVersion to receive incremental deltas. The initial LIST is necessary because a watch only delivers future changes; without it the cache would start empty and the controller would lack a baseline of current state.
4. On reconnect the client resumes the watch from the last observed resourceVersion, and the API server replays the changes that occurred since then (as long as they are still within the server's history window). If the resourceVersion is too old to replay ("too old" error), the informer performs a fresh LIST to resync the cache, then re-establishes the watch. Either way, level-triggered reconciliation against current state ensures correctness even if individual events were missed.

---

Authentication and authorization flow

Theory

Before any request changes the cluster, the API server runs it through a gauntlet. The first two stages are authentication (who are you?) and authorization (are you allowed to do this?).

1. Authentication (authn): the API server verifies the caller's identity using one or more configured methods — X.509 client certificates, bearer tokens, ServiceAccount tokens, or OIDC. The result is a username and group memberships. If no authenticator succeeds, the request is rejected with 401 Unauthorized. Note: Kubernetes has no built-in "user" object — users are external identities.
2. Authorization (authz): given the authenticated identity, the API server asks the configured authorizers (most commonly RBAC) whether this identity may perform this verb (get/list/create/update/delete) on this resource in this namespace. If none allows it, the request is rejected with 403 Forbidden.

Only after passing both does the request continue to admission control.

Example

Request --> [ Authentication ] --> [ Authorization ] --> [ Admission ] --> etcd
              who are you?           are you allowed?       mutate/validate
              401 if fail            403 if fail            422 if rejected

# "Can I do this?" — query the authorization layer without performing the action:
kubectl auth can-i create deployments --namespace dev
# yes

kubectl auth can-i delete nodes
# no

# Check as another identity (impersonation, if you have permission):
kubectl auth can-i list secrets --as=system:serviceaccount:dev:builder -n dev

Exercises

1. (Beginner) What question does authentication answer, and what question does authorization answer?
2. (Beginner) Which HTTP status code indicates an authentication failure, and which indicates an authorization failure?
3. (Intermediate) Use a single kubectl command to check whether the ServiceAccount dev:builder can list Secrets in the dev namespace.
4. (Interview) Kubernetes has no built-in user objects, yet it authenticates users. How is this possible, and what are two mechanisms used to establish user identity? (Hint: certificates and external identity providers.)

Answers

1. Authentication answers "who are you?" (establishes identity). Authorization answers "are you allowed to do this?" (checks permissions for the requested action).
2. Authentication failure -> 401 Unauthorized. Authorization failure -> 403 Forbidden.
3. kubectl auth can-i list secrets --as=system:serviceaccount:dev:builder -n dev.
4. Users are external to Kubernetes — there is no User API object. Identity is asserted by credentials the API server validates: e.g., (a) X.509 client certificates where the certificate's Common Name is the username and Organization fields are groups, signed by a CA the API server trusts; and (b) OIDC tokens from an external identity provider (Google, Okta, Azure AD, Dex), where the API server validates the JWT and extracts username/groups from claims. (Bearer tokens and authenticating webhooks are also valid.) ServiceAccounts, by contrast, are in-cluster objects for in-cluster identities.

---

Admission controllers pipeline

Theory

After authn and authz, but before the object is persisted, the request passes through the admission control pipeline — a chain of plugins that can inspect, modify, or reject the request based on policy. This is where the cluster enforces rules that go beyond "who can do what" into "what is allowed to exist and how."

Admission runs in two ordered phases:

1. Mutating admission: plugins may change the object (e.g., inject a sidecar, set default values, add labels). The MutatingAdmissionWebhook plugin calls external webhooks for this.
2. Validating admission: plugins may accept or reject but not modify (e.g., enforce that no privileged Pods are created). The ValidatingAdmissionWebhook plugin and the newer ValidatingAdmissionPolicy (CEL-based) handle this.

If any admission plugin rejects the request, it never reaches etcd and the client gets an error (typically 422). Built-in controllers include NamespaceLifecycle, LimitRanger, ResourceQuota, and PodSecurity.

Example

            +--------------------- Admission --------------------+
authn/authz | Mutating (can modify) --> Validating (accept/reject)| --> etcd
            +----------------------------------------------------+
   e.g. inject sidecar, set defaults      e.g. block privileged Pods

# A MutatingWebhookConfiguration wires an external webhook into the pipeline:
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: sidecar-injector
webhooks:
- name: inject.example.com
  rules:
  - operations: ["CREATE"]
    apiGroups: [""]
    apiVersions: ["v1"]
    resources: ["pods"]          # intercept Pod creation to inject a sidecar
  clientConfig:
    service:
      name: injector
      namespace: sidecar-system
      path: /mutate
  admissionReviewVersions: ["v1"]
  sideEffects: None

Exercises

1. (Beginner) At what point in request handling do admission controllers run — before or after the object is stored in etcd?
2. (Beginner) What is the key difference between a mutating and a validating admission controller?
3. (Intermediate) Why must the mutating phase run before the validating phase? Give an example where the order matters.
4. (Interview) Admission webhooks let you enforce arbitrary organizational policy. What is a major operational risk of a poorly configured mutating/validating webhook, and how can you mitigate it? (Hint: think about what happens when the webhook endpoint is down and failurePolicy.)

Answers

1. Admission controllers run after authentication and authorization but before the object is persisted to etcd.
2. A mutating controller can modify the incoming object (set defaults, inject containers/labels). A validating controller can only accept or reject it — it cannot change it.
3. Mutating runs first so that validating sees the final, post-mutation object. Example: a mutating webhook injects a sidecar container and security context defaults; a validating webhook (or PodSecurity) then checks the complete Pod against policy. If validation ran first, it would judge an incomplete object and could pass something the mutation later makes non-compliant — or reject something the mutation would have fixed.
4. If a webhook is configured with failurePolicy: Fail and its endpoint becomes unavailable, the API server cannot complete admission and will reject the affected requests, potentially blocking all Pod creation cluster-wide (an outage). Mitigations: scope the webhook narrowly with precise rules and namespaceSelector/objectSelector (exclude kube-system), run the webhook service highly available, set sensible timeoutSeconds, and consider failurePolicy: Ignore for non-critical policies (accepting that some requests bypass the check during outages). Always test webhook downtime scenarios.

---

2.4 Add-ons and Extensions

A freshly bootstrapped cluster is missing pieces most workloads assume exist: in-cluster DNS, metrics, a dashboard, and a network implementation. These are provided by add-ons — themselves running as Kubernetes workloads. This subchapter covers the most important ones.

---

CoreDNS

Theory

Services need names, not just IPs. CoreDNS is the cluster's DNS server, deployed as a Deployment in the kube-system namespace. It gives every Service a stable DNS name so Pods can find each other by name (e.g., payments.default.svc.cluster.local) instead of hard-coding ClusterIPs that could change.

When a Pod makes a DNS query, it is routed (via the Pod's /etc/resolv.conf, configured by the kubelet) to the CoreDNS Service. CoreDNS watches the API server for Services and Endpoints and answers queries from that live data. It is configured via a Corefile stored in a ConfigMap, using a plugin chain (the kubernetes plugin handles cluster records; forward sends external queries upstream).

Example

# Every Service gets a name of the form:
#   <service>.<namespace>.svc.cluster.local
kubectl run test --rm -it --image=busybox -- nslookup payments.default
# Server:    10.96.0.10
# Address:   10.96.0.10:53
# Name:      payments.default.svc.cluster.local
# Address:   10.96.43.12

# CoreDNS Corefile (in the coredns ConfigMap):
.:53 {
    kubernetes cluster.local in-addr.arpa ip6.arpa {  # serve cluster records
        pods insecure
    }
    forward . /etc/resolv.conf   # forward everything else upstream
    cache 30
}

Exercises

1. (Beginner) What does CoreDNS provide to the cluster, and in which namespace does it run?
2. (Beginner) Write the fully qualified DNS name for a Service named cache in the prod namespace.
3. (Intermediate) A Pod cannot resolve payments.default. List three things you would check to diagnose the DNS failure.
4. (Interview) CoreDNS answers from data it watches in the API server rather than a static zone file. Why is this essential in a dynamic cluster? (Hint: Services and endpoints change constantly.)

Answers

1. CoreDNS provides in-cluster DNS — resolving Service (and Pod) names to IPs — and runs in the kube-system namespace (typically as the coredns Deployment fronted by the kube-dns Service).
2. cache.prod.svc.cluster.local.
3. Any three: confirm CoreDNS Pods are running and healthy (kubectl -n kube-system get pods -l k8s-app=kube-dns); check the Pod's /etc/resolv.conf points at the cluster DNS IP; verify the target Service exists and has endpoints (kubectl get svc,endpointslices); test resolution from a debug Pod (nslookup); inspect CoreDNS logs and the Corefile; check NetworkPolicies aren't blocking port 53.
4. Services and the Pods backing them are created, deleted, and rescheduled constantly, changing IPs frequently. A static zone file would be stale within seconds. By watching the API server, CoreDNS always answers with the current set of Services and endpoints, so name resolution stays correct as the cluster changes — which is the whole point of stable names over ephemeral IPs.

---

Kubernetes Dashboard

Theory

The Kubernetes Dashboard is an optional web-based UI for the cluster. It lets you view and manage workloads, inspect logs, see resource usage, and create/edit objects through a browser instead of kubectl. It is useful for newcomers, for quick visual inspection, and for operators who prefer a GUI for some tasks.

The Dashboard is not installed by default and runs as a workload in the cluster (commonly the kubernetes-dashboard namespace). Because it can be a powerful entry point, it must be secured carefully: it authenticates users and acts on their behalf via the API server, respecting their RBAC permissions. Historically, misconfigured Dashboards (granted cluster-admin and exposed publicly) have been a notable attack vector — so it should be exposed via kubectl proxy or an authenticated Ingress, never wide open with broad privileges.

Example

# Install (community manifest) and access securely via the API server proxy:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/.../recommended.yaml
kubectl proxy
# Then browse to:
# http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/

# Create a token for a ServiceAccount to log in (scoped by RBAC):
kubectl -n kubernetes-dashboard create token admin-user

Exercises

1. (Beginner) Is the Kubernetes Dashboard installed by default in a standard cluster?
2. (Beginner) What does the Dashboard use under the hood to read and modify cluster objects?
3. (Intermediate) Describe a safe way to access the Dashboard without exposing it publicly.
4. (Interview) Why is granting the Dashboard's ServiceAccount cluster-admin and exposing it via a public LoadBalancer dangerous? (Hint: think about the blast radius if the endpoint is reached by an attacker.)

Answers

1. No. The Dashboard is an optional add-on that must be installed explicitly.
2. It uses the kube-apiserver (the standard Kubernetes API), acting on behalf of the logged-in identity and constrained by that identity's RBAC permissions.
3. Install it, then access it through kubectl proxy (which tunnels to the API server over your authenticated kubeconfig) on localhost, or place it behind an authenticated Ingress with TLS and SSO. Log in with a token tied to a least-privilege ServiceAccount. Do not expose it via a public LoadBalancer.
4. If the Dashboard's ServiceAccount has cluster-admin and the UI is publicly reachable, anyone who reaches it (or bypasses weak auth) can perform any action in the cluster — read all Secrets, create privileged Pods, exfiltrate data, or take over nodes. The blast radius is the entire cluster. Least privilege (scoped RBAC) plus no public exposure limits damage even if the endpoint is reached.

---

Metrics Server

Theory

To answer "how much CPU and memory is this Pod using right now?" — and to drive the Horizontal Pod Autoscaler — the cluster needs live resource metrics. The Metrics Server is a lightweight add-on that collects CPU/memory usage from each node's kubelet (via the Summary API), aggregates it in memory, and exposes it through the Metrics API (metrics.k8s.io).

Crucially, Metrics Server is for real-time, short-lived metrics used by kubectl top and autoscalers — it is not a monitoring system. It keeps only the latest readings in memory (no historical storage), so for dashboards, trends, and alerting you use Prometheus instead. Without Metrics Server installed, kubectl top and CPU/memory-based HPA will not work.

Example

# Metrics Server powers these commands:
kubectl top nodes
# NAME     CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
# node-1   210m         10%    1200Mi          31%

kubectl top pods -n default
# NAME    CPU(cores)   MEMORY(bytes)
# web     5m           48Mi

# An HPA consumes the Metrics API to scale on CPU:
kubectl autoscale deployment web --cpu-percent=70 --min=2 --max=10

Exercises

1. (Beginner) What two commands stop working if the Metrics Server is not installed?
2. (Beginner) Where does the Metrics Server get its data from?
3. (Intermediate) Why is the Metrics Server unsuitable as your monitoring/alerting solution, and what would you use instead?
4. (Interview) The Horizontal Pod Autoscaler depends on the Metrics API. Trace the path from a CPU spike in a Pod to the HPA adding a replica. (Hint: kubelet -> Metrics Server -> Metrics API -> HPA controller.)

Answers

1. kubectl top nodes and kubectl top pods (and CPU/memory-based HPA scaling) stop working.
2. It scrapes resource usage from each node's kubelet (the kubelet's Summary API, sourced from cAdvisor), then aggregates it.
3. Metrics Server holds only the most recent readings in memory with no historical retention, so it cannot provide trends, long-term dashboards, or alerting. Use Prometheus (with kube-state-metrics, Grafana, and Alertmanager) for monitoring, history, and alerts.
4. The Pod's container CPU usage rises; the node's kubelet/cAdvisor records it. Metrics Server scrapes the kubelet and exposes the aggregated value via the Metrics API (metrics.k8s.io). The HPA controller (in the controller-manager) periodically queries the Metrics API, computes the current CPU utilization against the target (e.g., 70%), calculates the desired replica count, and updates the Deployment's replicas. The Deployment/ReplicaSet controller then creates the new Pod, which the scheduler places on a node.

---

Container Network Interface (CNI) plugins

Theory

Kubernetes defines that every Pod must get its own IP and be able to reach every other Pod, but it deliberately does not implement the networking itself. That job is delegated to a CNI (Container Network Interface) plugin. CNI is a CNCF standard: when the kubelet creates a Pod, it calls the configured CNI plugin to set up the Pod's network namespace, assign an IP, and wire up routes.

This is the same "stable interface, swappable implementation" pattern as CRI and CSI. Different CNI plugins make different trade-offs:

• Flannel: simple overlay networking, easy to set up, minimal features.
• Calico: high performance, supports NetworkPolicy enforcement, can run without an overlay (BGP routing).
• Cilium: eBPF-based, advanced observability, network policy, and service mesh features.

Without a CNI plugin installed, Pods stay stuck in ContainerCreating / Pending because their network cannot be set up — a very common "fresh kubeadm cluster" gotcha.

Example

# A brand-new kubeadm cluster has nodes NotReady until a CNI is installed:
kubectl get nodes
# NAME     STATUS     ROLES           AGE   VERSION
# node-1   NotReady   control-plane   2m    v1.30.2   <- no CNI yet

# Install a CNI (Calico example), then nodes go Ready:
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/.../calico.yaml
kubectl get nodes
# node-1   Ready      control-plane   5m    v1.30.2

CNI config lives on each node, e.g. /etc/cni/net.d/10-calico.conflist
kubelet --(CNI ADD)--> plugin --> assigns Pod IP, sets up veth + routes

Exercises

1. (Beginner) Why doesn't Kubernetes ship with its own built-in Pod networking implementation?
2. (Beginner) What symptom do you see on a fresh cluster that has no CNI plugin installed?
3. (Intermediate) Compare Flannel and Calico in terms of features, particularly around NetworkPolicy.
4. (Interview) The CNI follows the same design philosophy as CRI and CSI. State that philosophy and explain why offloading networking to plugins is strategically smart for the Kubernetes project. (Hint: ecosystem, neutrality, innovation.)

Answers

1. Kubernetes specifies the networking model (every Pod gets an IP; all Pods can reach all Pods) but leaves the implementation to pluggable CNI plugins, so it stays neutral and lets specialized projects provide networking suited to different environments.
2. Nodes stay NotReady and Pods stay Pending/ContainerCreating because the kubelet cannot set up Pod networking without a CNI plugin.
3. Flannel provides simple overlay networking (e.g., VXLAN) and is easy to deploy, but historically does not enforce NetworkPolicy on its own. Calico is higher-performance, can route without an overlay (BGP), and natively enforces Kubernetes NetworkPolicy (plus richer Calico-specific policies). For network segmentation/policy, Calico (or Cilium) is the choice.
4. The philosophy is a standard interface with swappable implementations: Kubernetes depends only on the CNI contract, and vendors supply implementations behind it. This keeps the core small and vendor-neutral, fosters a competitive ecosystem (Calico, Cilium, Flannel, Weave) that can innovate independently (e.g., eBPF in Cilium), and lets operators pick the networking stack that fits their performance, security, and topology needs without forking Kubernetes.

---