6. Networking

Networking is where Kubernetes feels most like magic — and where it most often breaks. Kubernetes defines a simple, opinionated networking model (every Pod gets an IP; all Pods can reach all Pods) and then delegates the implementation to plugins. On top of that base it layers Services for stable access, Ingress for HTTP routing, NetworkPolicies for segmentation, and DNS for discovery. This chapter builds that stack from the bottom up.

---

6.1 Kubernetes Networking Model

Before Services and Ingress make sense, you must understand the foundational model: a flat network where every Pod is directly addressable. This subchapter covers those first principles.

---

Pod-to-pod communication

Theory

In Kubernetes, every Pod gets its own unique IP address, and any Pod can communicate with any other Pod directly using that IP, across nodes, without NAT. This is the core promise of the Kubernetes networking model. From a Pod's perspective, the cluster looks like one big flat network where it can reach any peer as if they were on the same LAN.

This is deliberately different from the Docker default (where containers share the host's IP and you publish ports). It dramatically simplifies application design: services don't need to discover which host a peer is on or deal with port-mapping — they just use Pod IPs (usually via a Service for stability). Containers within a Pod share that single Pod IP and communicate over localhost.

The actual wiring (routes, overlays, encapsulation) is the CNI plugin's job; the model only guarantees the behavior, not the implementation.

Example

Node A                          Node B
+---------------------+         +---------------------+
| Pod 1: 10.244.1.5   |         | Pod 3: 10.244.2.9   |
| Pod 2: 10.244.1.6   |         | Pod 4: 10.244.2.10  |
+---------------------+         +---------------------+
        \____________  direct, no NAT  ____________/
   10.244.1.5 can reach 10.244.2.9 by IP across nodes

# Each Pod has its own IP:
kubectl get pods -o wide
# NAME    READY   STATUS    IP            NODE
# web-1   1/1     Running   10.244.1.5    node-a
# web-2   1/1     Running   10.244.2.9    node-b
# From web-1 you can curl 10.244.2.9 directly (no port mapping needed).

Exercises

1. (Beginner) How many IP addresses does each Pod get, and who shares it?
2. (Beginner) Can a Pod on node A reach a Pod on node B directly by IP? Is NAT involved?
3. (Intermediate) How does Kubernetes' networking model differ from Docker's default host-port-mapping approach, and why is that simpler for app developers?
4. (Interview) The networking model specifies behavior but not implementation. What component provides the implementation, and why is separating the two valuable? (Hint: CNI, pluggability.)

Answers

1. Each Pod gets exactly one unique IP address, shared by all containers in that Pod (they communicate among themselves over localhost).
2. Yes — directly by Pod IP, across nodes, with no NAT. That is a fundamental guarantee of the model.
3. Docker's default gives containers private IPs behind the host and requires publishing/mapping host ports to reach them, so callers must know host+port and avoid port conflicts. Kubernetes gives every Pod a routable IP on a flat network, so apps just talk to Pod/Service IPs without port mapping or host awareness — much simpler service-to-service communication.
4. The CNI plugin (Calico, Cilium, Flannel, etc.) implements the actual routing/overlay. Separating the guaranteed behavior from the implementation lets operators choose a networking solution suited to their environment (overlay vs. BGP, eBPF, policy support, performance) without changing applications or Kubernetes itself — the same stable-interface/swappable-implementation philosophy as CRI and CSI.

---

Flat network model principles

Theory

The Kubernetes network model is governed by a few non-negotiable rules every conformant CNI must satisfy:

1. Every Pod has a unique IP across the whole cluster.
2. Pods on a node can communicate with all Pods on all nodes without NAT.
3. Agents on a node (e.g., the kubelet) can communicate with all Pods on that node.
4. (For Pods using the host network, they use the node's IP.)

This is the flat network model: a single, non-overlapping IP space shared by all Pods, with no address translation between them. The principle deliberately pushes complexity (overlays, encapsulation, BGP) below the model so applications can assume a simple, uniform network. The trade-offs different CNIs make are about how to deliver this flat space — VXLAN overlays (encapsulate packets), native routing (advertise Pod CIDRs via BGP), or eBPF — each with different performance and feature characteristics.

Example

Cluster Pod CIDR: 10.244.0.0/16  (one flat space, carved per node)
  node-a -> 10.244.1.0/24   (Pods 10.244.1.x)
  node-b -> 10.244.2.0/24   (Pods 10.244.2.x)
  node-c -> 10.244.3.0/24   (Pods 10.244.3.x)
No overlap; any Pod IP is unique and routable to any other, no NAT.

Exercises

1. (Beginner) State the core rule of the flat network model regarding Pod-to-Pod traffic.
2. (Beginner) Why must Pod IP ranges per node not overlap?
3. (Intermediate) Name two different techniques CNIs use to implement the flat network, and one trade-off between them.
4. (Interview) Why does Kubernetes deliberately keep NAT out of Pod-to-Pod communication, and what application-level problems does that avoid? (Hint: source IP preservation, peer addressing, simplicity.)

Answers

1. Any Pod can reach any other Pod directly by IP, across nodes, without NAT (a single flat IP space).
2. Because every Pod IP must be unique and routable cluster-wide; overlapping per-node ranges would create ambiguous/duplicate addresses, breaking direct routing and the uniqueness guarantee.
3. Examples: overlay/encapsulation (e.g., VXLAN, as in Flannel) wraps Pod packets to traverse the underlay — easy to deploy on any network but adds encapsulation overhead; native/BGP routing (e.g., Calico) advertises Pod CIDRs so packets route natively — better performance but needs network/router cooperation. (eBPF-based, e.g., Cilium, is another.) Trade-off: overlay = portability/simplicity vs. overhead; native routing = performance vs. infrastructure requirements.
4. NAT rewrites source/destination addresses, which would obscure the real client IP, complicate direct peer-to-peer addressing, and break protocols that embed addresses. By guaranteeing no NAT between Pods, Kubernetes preserves source IPs (important for logging, security policy, and some protocols), lets services address peers by their actual IPs, and keeps the mental model simple — every Pod is just directly reachable, like hosts on a flat LAN.

---

CNI plugin overview (Calico, Flannel, Cilium)

Theory

A CNI plugin implements the flat network model. The three most common — each making different trade-offs — are:

Plugin	Approach	NetworkPolicy	Notable strengths
Flannel	Simple overlay (VXLAN by default)	No (by itself)	Easiest to set up; minimal; good for learning/small clusters.
Calico	Native routing (BGP) or overlay (IPIP/VXLAN)	Yes (rich)	High performance, mature NetworkPolicy, scalable.
Cilium	eBPF-based dataplane	Yes (L3–L7)	eBPF performance/observability, L7 policy, service mesh, Hubble.

Flannel prioritizes simplicity at the cost of features (notably no policy enforcement). Calico is the go-to when you need solid NetworkPolicy and performance, and can run without an overlay for near-native speed. Cilium is the modern, feature-rich choice built on eBPF, offering identity-aware L3–L7 policy, deep observability (Hubble), and even service-mesh capabilities — at the cost of requiring a recent kernel and more conceptual surface area.

Example

# Installing a CNI is a one-time cluster step (examples):
kubectl apply -f https://.../flannel/kube-flannel.yml          # Flannel
kubectl apply -f https://.../calico/calico.yaml                # Calico
cilium install                                                 # Cilium (CLI)

# CNI components run as a DaemonSet (one agent per node):
kubectl -n kube-system get pods -l k8s-app=calico-node -o wide

Exercises

1. (Beginner) Which of the three plugins does not enforce NetworkPolicy on its own?
2. (Beginner) What underlying technology distinguishes Cilium's dataplane?
3. (Intermediate) You need network segmentation (NetworkPolicy) and high throughput on a cluster where you control the network. Which CNI fits and why?
4. (Interview) Cilium uses eBPF instead of iptables for its dataplane. What advantages does this bring as cluster and Service counts grow large? (Hint: iptables rule scaling, observability, L7 awareness.)

Answers

1. Flannel (it provides networking only; you'd pair it with something else for policy).
2. eBPF (it programs the Linux kernel directly via eBPF programs rather than relying on iptables).
3. Calico (or Cilium). Calico provides mature, performant NetworkPolicy and can use native BGP routing without an overlay for near-native throughput when you control the underlying network. Cilium is also a strong fit (eBPF performance + rich policy).
4. iptables-based dataplanes degrade as rules grow — large numbers of Services/endpoints create long, linearly-scanned rule chains that slow updates and packet processing. Cilium's eBPF dataplane uses efficient in-kernel maps with effectively O(1) lookups, scaling better to many Services and high churn. It also enables rich, identity-based and L7-aware policy and deep flow observability (Hubble) that iptables cannot provide — improving performance, scalability, and visibility simultaneously.

---

IP address management (IPAM)

Theory

IPAM (IP Address Management) is the part of networking responsible for allocating and tracking IP addresses — ensuring every Pod gets a unique, valid IP from the cluster's address space and that addresses are reclaimed when Pods die. The CNI plugin includes an IPAM component that does this.

The cluster is configured with a Pod CIDR (e.g., 10.244.0.0/16). Typically this large block is subdivided into per-node CIDRs (e.g., a /24 per node), and each node's IPAM hands out IPs from its slice to Pods scheduled there. This local allocation avoids a central bottleneck. Cloud-native CNIs may instead assign real VPC IPs to Pods (e.g., AWS VPC CNI gives each Pod an ENI/secondary IP from the VPC), trading IP exhaustion concerns for native routability. Key operational concerns: avoiding CIDR overlap with other networks, and not exhausting the address space (which causes Pods to get stuck ContainerCreating with IPAM errors).

Example

Pod CIDR: 10.244.0.0/16  (65,534 usable IPs)
  split into /24 per node => 254 Pods/node, 256 nodes max
  node-a: 10.244.1.0/24  (allocates 10.244.1.2 ... 10.244.1.254)
  node-b: 10.244.2.0/24
IPAM on each node assigns + frees IPs as Pods come and go.

# Symptom of IP exhaustion / IPAM failure:
kubectl describe pod stuck-pod
# Events: ... failed to allocate IP address: no available addresses in range

Exercises

1. (Beginner) What is IPAM responsible for?
2. (Beginner) What is a "Pod CIDR," and how is it typically divided?
3. (Intermediate) What symptom would you expect if a node's IP range is exhausted?
4. (Interview) Cloud CNIs like the AWS VPC CNI assign real VPC IPs to Pods instead of overlay IPs. What does this gain, and what new constraint does it introduce? (Hint: native routability vs. VPC IP exhaustion.)

Answers

1. Allocating unique IP addresses to Pods from the cluster's address space and tracking/reclaiming them as Pods are created and destroyed.
2. The Pod CIDR is the IP range reserved for Pod addresses (e.g., 10.244.0.0/16). It is typically subdivided into smaller per-node blocks (e.g., a /24 per node) so each node allocates IPs locally from its slice.
3. New Pods scheduled to that node fail to start, staying in ContainerCreating/Pending with IPAM errors like "no available addresses" — because the node cannot allocate an IP for the Pod's network namespace.
4. Assigning real VPC IPs makes Pods natively routable within the VPC (no overlay/encapsulation overhead, direct integration with cloud security groups and load balancers, preserved source IPs). The constraint is VPC IP exhaustion: Pods consume actual VPC subnet addresses, so subnet sizing and per-node IP/ENI limits cap Pod density and must be planned carefully to avoid running out of addresses.

---

6.2 Services

Pods are ephemeral and their IPs change; Services provide stable endpoints in front of them. This subchapter covers every Service type and how discovery and affinity work.

---

ClusterIP service

Theory

A Service is an abstraction that provides a stable virtual IP and DNS name in front of a dynamic set of Pods selected by labels. The ClusterIP type — the default — exposes the Service on an internal, cluster-only virtual IP. It is reachable from inside the cluster but not from outside. This is the workhorse for internal service-to-service communication (e.g., a frontend talking to a backend API).

The Service decouples clients from Pod churn: clients connect to the stable ClusterIP (or DNS name), and kube-proxy load-balances each connection to one of the healthy backing Pods (whose membership is tracked in EndpointSlices). When Pods are added, removed, or rescheduled with new IPs, the Service IP stays constant and the endpoint set updates automatically. The set of Pods behind a Service is defined by its selector matching Pod labels.

Example

apiVersion: v1
kind: Service
metadata: { name: backend }
spec:
  type: ClusterIP            # default; internal-only virtual IP
  selector:
    app: backend             # selects Pods labeled app=backend
  ports:
  - port: 80                 # the Service port clients use
    targetPort: 8080         # the container port traffic is sent to

kubectl get svc backend
# NAME      TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)
# backend   ClusterIP   10.96.12.34    <none>        80/TCP
# Reachable in-cluster as: backend.<namespace>.svc.cluster.local:80

Exercises

1. (Beginner) What does a ClusterIP Service provide, and is it reachable from outside the cluster?
2. (Beginner) What is the difference between port and targetPort?
3. (Intermediate) How does a Service know which Pods to send traffic to, and what object tracks the current set?
4. (Interview) Explain how a Service provides stability to clients despite Pods being created and destroyed with changing IPs. (Hint: stable VIP/DNS, selector, EndpointSlices, kube-proxy.)

Answers

1. A stable internal virtual IP (and DNS name) load-balancing to the selected Pods. ClusterIP is internal-only — not reachable from outside the cluster.
2. port is the port the Service listens on (what clients connect to); targetPort is the port on the backing Pods/containers that traffic is forwarded to. They can differ.
3. Via its label selector matching Pod labels. The current set of healthy backing Pod IPs is tracked in EndpointSlices (formerly Endpoints), which the control plane keeps updated as Pods come and go.
4. Clients use the Service's stable ClusterIP or DNS name, which never changes for the life of the Service. Behind it, the selector continuously identifies matching, ready Pods and the EndpointSlice controller updates the endpoint list as Pods churn. kube-proxy programs each node so traffic to the ClusterIP is load-balanced to a current healthy Pod. Thus clients are insulated entirely from Pod creation/deletion and IP changes.

---

NodePort service

Theory

A NodePort Service exposes the Service on a static port on every node's IP, making it reachable from outside the cluster at <any-node-IP>:<nodePort>. It builds on ClusterIP: creating a NodePort also creates a ClusterIP, and the node port simply forwards into it. The port is allocated from a configurable range (default 30000–32767).

NodePort is the simplest way to get external traffic into a cluster without a cloud load balancer, but it has limitations: the ports are high-numbered and non-standard, you must track node IPs (which can change), and there's no built-in load balancing across nodes (clients hit a specific node, though kube-proxy then routes to any Pod). In practice NodePort is used for development, on-prem setups behind an external load balancer, or as the underlying mechanism that LoadBalancer Services build upon.

Example

apiVersion: v1
kind: Service
metadata: { name: web }
spec:
  type: NodePort
  selector: { app: web }
  ports:
  - port: 80
    targetPort: 8080
    nodePort: 30080        # optional; otherwise auto-assigned in 30000-32767

kubectl get svc web
# NAME   TYPE       CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE
# web    NodePort   10.96.5.6     <none>        80:30080/TCP   1m
# Reachable externally at: http://<node-IP>:30080

Exercises

1. (Beginner) Where is a NodePort Service reachable from, and on what kind of port?
2. (Beginner) What is the default node port range?
3. (Intermediate) A NodePort also implicitly creates which other Service type, and why?
4. (Interview) What are the practical drawbacks of exposing production traffic directly via NodePort, and what is typically placed in front of it? (Hint: non-standard ports, node IP tracking, no L7 features; external LB/Ingress.)

Answers

1. From outside the cluster, at any node's IP on a static, high-numbered port (<node-IP>:<nodePort>).
2. 30000–32767.
3. It also creates a ClusterIP Service. The node port forwards incoming external traffic to the ClusterIP, which load-balances to the Pods — so NodePort is a layer on top of ClusterIP.
4. Drawbacks: non-standard high ports (not 80/443), clients must know/track node IPs (which change as nodes are added/removed), no TLS termination, host-based routing, or other L7 features, and limited/awkward load balancing. In production you typically place an external load balancer in front (which a LoadBalancer Service automates) and/or an Ingress controller for HTTP routing and TLS, using NodePort only as the underlying plumbing.

---

LoadBalancer service

Theory

A LoadBalancer Service is the standard way to expose a service to the internet in a cloud environment. It builds on NodePort/ClusterIP and additionally asks the cloud provider (via the cloud-controller-manager) to provision an external load balancer (AWS ELB/NLB, GCP LB, Azure LB) that forwards traffic to the Service. The cloud LB gets a stable external IP/hostname, which Kubernetes reports in the Service's EXTERNAL-IP.

This gives you a single, stable, internet-facing endpoint with the cloud's load balancing, health checks, and (often) integration with TLS and DDoS protection. The catch: each LoadBalancer Service typically provisions its own cloud load balancer, which costs money and doesn't scale well if you have dozens of services — which is exactly the problem Ingress solves (one LB fronting many services). On bare metal there's no cloud LB, so tools like MetalLB implement the type.

Example

apiVersion: v1
kind: Service
metadata: { name: frontend }
spec:
  type: LoadBalancer
  selector: { app: frontend }
  ports:
  - port: 80
    targetPort: 8080

kubectl get svc frontend
# NAME       TYPE           CLUSTER-IP    EXTERNAL-IP        PORT(S)
# frontend   LoadBalancer   10.96.0.40    a1b2.elb.aws.com   80:31234/TCP
# The cloud-controller-manager populated EXTERNAL-IP once the LB was ready.

Exercises

1. (Beginner) What does a LoadBalancer Service ask the cloud provider to create?
2. (Beginner) Which component populates the Service's EXTERNAL-IP?
3. (Intermediate) Why can running many LoadBalancer Services become expensive and unwieldy?
4. (Interview) On a bare-metal cluster, a LoadBalancer Service stays <pending>. Explain why and how to resolve it. (Hint: no cloud CCM; MetalLB.)

Answers

1. An external (cloud) load balancer with a stable public IP/hostname that forwards traffic to the Service's nodes/Pods.
2. The cloud-controller-manager (its Service controller), after the cloud LB is provisioned.
3. Each LoadBalancer Service typically provisions its own cloud load balancer, each of which costs money and consumes cloud quota. With many services this means many LBs to pay for and manage — inefficient compared to a single LB fronting many services, which is what Ingress provides.
4. There is no cloud provider/CCM to fulfill the LoadBalancer request, so no external LB is created and EXTERNAL-IP never populates. Resolve it by installing a bare-metal load-balancer implementation such as MetalLB (or kube-vip), which assigns IPs from a configured pool and announces them via ARP (L2) or BGP, fulfilling LoadBalancer Services on-prem.

---

ExternalName service

Theory

An ExternalName Service is unusual: it has no selector, no Pods, no ClusterIP. Instead it maps a Service name to an external DNS name via a CNAME record. When something in the cluster looks up the Service, CoreDNS returns a CNAME pointing to the external host. It is essentially an in-cluster DNS alias for an external endpoint.

Its purpose is to let in-cluster clients reference an external dependency (a managed database, a third-party API, a legacy system) through a stable internal name, decoupling the client from the actual external address. You could later migrate that dependency into the cluster by changing the Service to a normal ClusterIP type — without touching any client. Note: because it's just DNS aliasing, no traffic flows through Kubernetes, no load balancing happens, and it works at the DNS level only (not for things requiring TLS SNI rewriting, etc.).

Example

apiVersion: v1
kind: Service
metadata: { name: prod-db }
spec:
  type: ExternalName
  externalName: db.prod.example.com   # CNAME target; no selector/ClusterIP

# In-cluster, "prod-db" resolves to the external host via CNAME:
kubectl run t --rm -it --image=busybox -- nslookup prod-db.default.svc.cluster.local
# prod-db.default.svc.cluster.local  canonical name = db.prod.example.com

Exercises

1. (Beginner) What does an ExternalName Service map a Service name to, and via what DNS record type?
2. (Beginner) Does an ExternalName Service have a ClusterIP or select Pods?
3. (Intermediate) Give a scenario where ExternalName helps you migrate an external dependency into the cluster later without changing clients.
4. (Interview) Since ExternalName works purely at the DNS layer, what are its limitations compared to a real Service? (Hint: no load balancing, no port remap, TLS/SNI considerations.)

Answers

1. To an external DNS hostname, via a CNAME record.
2. No — it has no ClusterIP and no selector/Endpoints; it is just a DNS alias.
3. Suppose your app uses an external managed database at db.prod.example.com. Point clients at the internal name prod-db (an ExternalName Service). Later, when you run the database inside the cluster, change prod-db to a normal ClusterIP Service selecting the in-cluster Pods. Clients keep using prod-db and need no changes — the indirection makes the migration transparent.
4. It only returns a DNS alias, so: there's no load balancing or health checking by Kubernetes (that's the external host's concern), no port remapping (it doesn't proxy traffic, so targetPort-style remap doesn't apply), kube-proxy isn't involved, and it can cause TLS/SNI mismatches (the client connects to the external host directly, so certificates must match the real external name, not the Service name). It's a convenience alias, not a traffic-handling Service.

---

Headless services

Theory

A headless Service (clusterIP: None) was introduced in Chapter 4 for StatefulSets, but it's a general Service feature. By setting clusterIP: None, you tell Kubernetes not to allocate a virtual IP and not to load-balance. Instead, a DNS lookup of the Service returns the A/AAAA records of all the backing Pods directly (their individual IPs).

This is useful whenever the client — not kube-proxy — should decide which Pod to talk to, or needs to see all of them: stateful systems addressing specific members, client-side load balancing, or service discovery where the app wants the full endpoint list (e.g., gRPC clients that maintain their own connection pool to each backend). With a selector, the headless Service produces per-Pod records; without a selector, you can manage the Endpoints manually.

Example

apiVersion: v1
kind: Service
metadata: { name: cassandra }
spec:
  clusterIP: None          # headless
  selector: { app: cassandra }
  ports:
  - port: 9042

# Returns ALL Pod IPs, not a single VIP:
kubectl run t --rm -it --image=busybox -- nslookup cassandra.default.svc.cluster.local
# Address: 10.244.1.5
# Address: 10.244.2.7
# Address: 10.244.3.9

Exercises

1. (Beginner) How do you make a Service headless?
2. (Beginner) What does a DNS query for a headless Service return instead of a single virtual IP?
3. (Intermediate) Give two situations where a headless Service is preferable to a normal ClusterIP Service.
4. (Interview) Why might a gRPC client specifically benefit from a headless Service rather than a ClusterIP? (Hint: long-lived connections, client-side load balancing, single VIP pinning.)

Answers

1. Set clusterIP: None in the Service spec.
2. The individual IP addresses (A/AAAA records) of all the backing Pods.
3. Any two: addressing specific members of a stateful system (StatefulSets), client-side load balancing where the client picks the backend, service discovery needing the full endpoint list, or peer-to-peer/clustered apps that must know all members.
4. gRPC uses long-lived HTTP/2 connections; with a ClusterIP, kube-proxy load-balances at connection time, so a client tends to pin a single persistent connection to one backend Pod, defeating load distribution across many requests. A headless Service returns all Pod IPs, letting the gRPC client open and balance requests across connections to each backend itself (client-side load balancing), achieving even distribution and reacting to endpoint changes.

---

Service discovery via DNS

Theory

How does a Pod find a Service? Through DNS, provided by CoreDNS. Every Service automatically gets a DNS A/AAAA record at <service>.<namespace>.svc.cluster.local resolving to its ClusterIP (or, for headless, to the Pod IPs). This is the primary service-discovery mechanism in Kubernetes — apps connect by name, not by IP.

Crucially, DNS supports short names via search domains. The kubelet configures each Pod's /etc/resolv.conf with search suffixes so that within the same namespace you can use just backend, from another namespace backend.other-ns, and the fully qualified name always works. There's also a legacy mechanism — environment variables (BACKEND_SERVICE_HOST/_PORT) injected for Services that existed when the Pod started — but DNS is preferred because it's dynamic (env vars are stale if the Service is created after the Pod).

Example

DNS name structure:
  <service>.<namespace>.svc.cluster.local
  e.g.  backend.payments.svc.cluster.local -> 10.96.12.34

From a Pod in namespace "payments":   curl http://backend          (short name works)
From a Pod in another namespace:       curl http://backend.payments
Always works (FQDN):                   curl http://backend.payments.svc.cluster.local

cat /etc/resolv.conf
# search payments.svc.cluster.local svc.cluster.local cluster.local
# nameserver 10.96.0.10

Exercises

1. (Beginner) What is the fully qualified DNS name pattern for a Service?
2. (Beginner) From a Pod in the same namespace as a Service api, what short name can you use to reach it?
3. (Intermediate) Why is DNS-based discovery preferred over the injected environment-variable mechanism?
4. (Interview) Explain how the search domains in a Pod's /etc/resolv.conf enable both short and cross-namespace name resolution. (Hint: suffix search list ordering.)

Answers

1. <service>.<namespace>.svc.cluster.local.
2. Just api (the same-namespace search suffix resolves it).
3. DNS is dynamic: a Pod can resolve any Service at any time, including Services created after the Pod started. Environment-variable discovery only injects variables for Services that existed when the Pod was created, so it misses later Services and goes stale — and it clutters the environment. DNS avoids these problems and supports short/cross-namespace names.
4. The resolver appends each search suffix in order to an unqualified name until one resolves. With search payments.svc.cluster.local svc.cluster.local cluster.local, looking up api first tries api.payments.svc.cluster.local (same namespace) — resolving short names; looking up api.other tries api.other.svc.cluster.local — resolving cross-namespace; and a full FQDN (ending in the cluster domain) resolves directly. The ordered suffix list is what makes both conveniences work.

---

Session affinity

Theory

By default, a Service load-balances each new connection independently, so successive requests from the same client may hit different Pods. For most stateless apps that's ideal. But some apps keep per-client in-memory state (a session, a cache) and work better if a given client consistently reaches the same Pod. Session affinity provides this "sticky" behavior.

Kubernetes Services support sessionAffinity: ClientIP, which routes requests from the same client source IP to the same backing Pod (within a configurable timeout, default 3 hours). This is L3/L4 affinity based on IP only — it cannot do cookie-based stickiness (that requires an L7 proxy like an Ingress controller). Note that session affinity is generally a workaround; the more robust pattern is to externalize session state (e.g., to Redis) so any Pod can serve any request, keeping the service truly stateless and horizontally scalable.

Example

apiVersion: v1
kind: Service
metadata: { name: web }
spec:
  selector: { app: web }
  sessionAffinity: ClientIP            # stick a client IP to one Pod
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 10800            # affinity duration (default 3h)
  ports:
  - port: 80
    targetPort: 8080

Exercises

1. (Beginner) What does session affinity do?
2. (Beginner) Which value enables IP-based stickiness on a Service, and what is its default timeout?
3. (Intermediate) Why can a Service not do cookie-based session stickiness, and what can?
4. (Interview) Session affinity is often called a workaround. What is the more scalable architectural alternative, and why is it preferable? (Hint: externalize state, statelessness.)

Answers

1. It routes requests from the same client consistently to the same backing Pod ("sticky sessions"), rather than load-balancing each request independently.
2. sessionAffinity: ClientIP; default timeout is 3 hours (10800 seconds).
3. A Service operates at L3/L4 and only sees IP addresses, not HTTP cookies, so it can only stick by client IP. Cookie-based affinity requires an L7 (HTTP-aware) proxy such as an Ingress controller (e.g., NGINX with affinity annotations) that can read/set cookies.
4. The better approach is to externalize session state (store sessions/caches in a shared store like Redis or a database) so the application is truly stateless and any Pod can serve any request. This avoids the fragility of affinity (uneven load, lost sessions when a Pod dies, broken stickiness behind NAT where many clients share an IP) and enables clean horizontal scaling and rolling updates without disrupting user sessions.

---

6.3 Ingress

Exposing many HTTP services each via its own LoadBalancer is wasteful. Ingress provides a single entry point with HTTP-aware routing, TLS, and host/path rules. This subchapter covers it.

---

Ingress resource and controllers

Theory

An Ingress is an API object that defines HTTP/HTTPS routing rules for exposing Services to the outside world — routing by hostname and URL path, terminating TLS, etc. — through a single external entry point. Critically, the Ingress object is just a set of rules; it does nothing on its own. You must run an Ingress controller (a Pod that watches Ingress objects and implements them), such as the NGINX Ingress Controller, Traefik, HAProxy, or a cloud's controller.

This is a deliberate separation: the Ingress resource is the portable, declarative spec; the Ingress controller is the pluggable implementation (the same pattern as CNI/CSI). One Ingress controller (fronted by a single LoadBalancer) can serve many Services across many hostnames and paths — solving the "one LB per service" cost problem of LoadBalancer Services. Note: Ingress is for HTTP(S); for other protocols you use Services or the newer Gateway API.

Example

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: site
spec:
  ingressClassName: nginx          # which controller implements this
  rules:
  - host: shop.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: frontend
            port: { number: 80 }

# The controller runs as a workload and is fronted by one LB:
kubectl -n ingress-nginx get pods,svc
# service/ingress-nginx-controller   LoadBalancer   ...   80:.../443:...

Exercises

1. (Beginner) What does an Ingress resource define, and what protocol(s) is it for?
2. (Beginner) Why does creating an Ingress object alone not route any traffic?
3. (Intermediate) How does Ingress solve the cost problem of using a LoadBalancer Service per application?
4. (Interview) Describe the separation between the Ingress resource and the Ingress controller and why this mirrors Kubernetes' broader extensibility philosophy. (Hint: portable spec vs. swappable implementation.)

Answers

1. It defines HTTP/HTTPS routing rules (host- and path-based) and TLS settings for exposing Services externally. It is for HTTP(S) traffic (L7).
2. The Ingress object is only a declarative set of rules. Without a running Ingress controller to watch those objects and configure an actual proxy/load balancer, nothing implements the rules, so no traffic is routed.
3. A single Ingress controller, fronted by one external load balancer, can route to many backend Services based on host/path rules. So instead of one cloud LB per service, you pay for one LB and define many routes — far cheaper and easier to manage at scale.
4. The Ingress resource is a standard, portable, declarative API describing desired routing; the Ingress controller is a pluggable implementation (NGINX, Traefik, cloud-native, etc.) that realizes those rules. This is the same stable-interface/swappable-implementation pattern as CRI, CNI, and CSI: Kubernetes defines the contract while letting operators choose the implementation best suited to their needs, without changing the resource or applications.

---

NGINX Ingress Controller

Theory

The NGINX Ingress Controller is the most widely used Ingress controller. It runs NGINX (a battle-tested reverse proxy) inside the cluster, watches Ingress objects, and dynamically generates NGINX configuration to route traffic accordingly. It handles TLS termination, path/host routing, rewrites, rate limiting, and many features exposed through annotations.

A common point of confusion: there are two different projects — the community ingress-nginx (maintained by the Kubernetes project) and NGINX Inc.'s nginx-ingress (commercial/F5). They use different annotation prefixes and feature sets. The community ingress-nginx is by far the most common in tutorials and default installs. Operationally, it runs as a Deployment or DaemonSet fronted by a LoadBalancer/NodePort Service, and you select it via ingressClassName: nginx.

Example

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /     # NGINX-specific behavior
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  ingressClassName: nginx
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /v1
        pathType: Prefix
        backend:
          service: { name: api-v1, port: { number: 80 } }

helm install ingress-nginx ingress-nginx/ingress-nginx -n ingress-nginx --create-namespace

Exercises

1. (Beginner) What software does the NGINX Ingress Controller run under the hood?
2. (Beginner) How are NGINX-specific behaviors (like URL rewrites) configured on an Ingress?
3. (Intermediate) Why is it important to know whether you're using community ingress-nginx or NGINX Inc.'s controller?
4. (Interview) The NGINX Ingress Controller regenerates NGINX config as Ingress/Service/endpoint objects change. Explain the watch-and-reconcile loop that makes this dynamic. (Hint: watches API server, templates config, reloads NGINX.)

Answers

1. NGINX (a reverse proxy/load balancer) running inside the cluster.
2. Through annotations on the Ingress object (e.g., nginx.ingress.kubernetes.io/rewrite-target), which the controller translates into NGINX configuration.
3. The two projects (community ingress-nginx vs. F5/NGINX Inc.'s nginx-ingress) use different annotation prefixes, configuration, and feature sets, so annotations and docs for one don't apply to the other. Using the wrong annotations silently does nothing or behaves unexpectedly, so you must know which controller you run.
4. The controller runs a control loop: it watches the API server for Ingress, Service, and EndpointSlice changes; on any change it renders an NGINX configuration from templates reflecting the current desired routing and the current set of healthy backend endpoints; then it applies/reloads that config in the NGINX process. Because it reacts to watch events (level-triggered reconciliation), routing stays continuously in sync with the cluster state as backends scale, fail, or are reconfigured.

---

Path-based and host-based routing

Theory

The core power of Ingress is routing a single external IP to many backends based on the request. Two dimensions:

• Host-based routing: route by the HTTP Host header (the domain). shop.example.com goes to the shop Service; api.example.com goes to the API Service — all on the same IP/controller. This is name-based virtual hosting.
• Path-based routing: route by URL path within a host. /web to the frontend, /api to the backend, /grafana to monitoring.

Paths have a pathType: Prefix (match by URL path prefix, the common choice), Exact (exact match), or ImplementationSpecific. Order and specificity matter, and some controllers need rewrite annotations when the backend doesn't expect the path prefix. Combining host and path rules lets one Ingress controller serve an entire portfolio of applications.

Example

spec:
  ingressClassName: nginx
  rules:
  - host: shop.example.com           # host-based: by domain
    http:
      paths:
      - path: /                      # path-based: by URL path
        pathType: Prefix
        backend: { service: { name: frontend, port: { number: 80 } } }
      - path: /api
        pathType: Prefix
        backend: { service: { name: api, port: { number: 80 } } }
  - host: admin.example.com          # different host -> different backend
    http:
      paths:
      - path: /
        pathType: Prefix
        backend: { service: { name: admin, port: { number: 80 } } }

Exercises

1. (Beginner) What request attribute does host-based routing use, and what does path-based routing use?
2. (Beginner) What is the most commonly used pathType?
3. (Intermediate) Write (in words) the routing for: app.example.com/ to web, app.example.com/api to api.
4. (Interview) When routing /api to a backend that serves its routes at /, what problem arises and how is it typically solved? (Hint: the backend receives /api/...; rewrite-target.)

Answers

1. Host-based routing uses the HTTP Host header (the domain); path-based routing uses the URL path.
2. Prefix.
3. One Ingress rule with host: app.example.com and two paths: path: / (pathType: Prefix) -> Service web; path: /api (pathType: Prefix) -> Service api. Requests to the root go to web; requests under /api go to api.
4. The backend receives the full path including the prefix (e.g., /api/users instead of /users), so its routes don't match and it returns 404s. This is solved with a URL rewrite (e.g., nginx.ingress.kubernetes.io/rewrite-target: / combined with a capture-group path), which strips/rewrites the prefix before forwarding so the backend sees the path it expects.

---

TLS termination at Ingress

Theory

TLS termination means the Ingress controller decrypts incoming HTTPS traffic at the edge, so your backend Services can receive plain HTTP internally. This centralizes certificate management at one place (the Ingress) instead of configuring TLS in every application. You provide the certificate and key in a kubernetes.io/tls Secret and reference it in the Ingress tls section, mapping it to the host(s) it covers.

In practice, certificate provisioning is automated with cert-manager, an add-on that obtains and renews certificates (e.g., free Let's Encrypt certs via ACME) and stores them in TLS Secrets that the Ingress references. The controller then serves HTTPS for the configured hosts, terminating TLS and forwarding plaintext to backends. (If you need encryption all the way to the Pod — "end-to-end" TLS or mTLS — that's a service-mesh concern, Chapter 13.)

Example

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: secure-site
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod   # cert-manager auto-issues
spec:
  ingressClassName: nginx
  tls:
  - hosts: [ shop.example.com ]
    secretName: shop-tls            # kubernetes.io/tls Secret with cert + key
  rules:
  - host: shop.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend: { service: { name: frontend, port: { number: 80 } } }

Exercises

1. (Beginner) What does "TLS termination at the Ingress" mean for backend traffic?
2. (Beginner) What type of Secret holds the certificate and key for an Ingress?
3. (Intermediate) What does cert-manager automate, and why is that valuable?
4. (Interview) TLS terminates at the Ingress, leaving traffic plaintext between the Ingress and the Pods. When is that acceptable, and what would you use if you need encryption end-to-end? (Hint: trusted cluster network vs. mTLS/service mesh.)

Answers

1. The Ingress controller decrypts HTTPS at the edge, so backends receive plain HTTP; TLS is handled centrally at the Ingress rather than in each app.
2. A kubernetes.io/tls Secret (containing tls.crt and tls.key), referenced via secretName in the Ingress tls section.
3. cert-manager automatically obtains and renews TLS certificates (e.g., from Let's Encrypt via ACME) and stores them in TLS Secrets the Ingress uses. This eliminates manual certificate issuance, installation, and renewal — preventing outages from expired certs and enabling free, automated HTTPS.
4. Terminating at the Ingress is acceptable when the in-cluster network between the Ingress and Pods is considered trusted and not subject to your threat model (common in many setups). If you require encryption all the way to the Pod (e.g., zero-trust, compliance, untrusted network), use a service mesh providing automatic mTLS between workloads (Chapter 13), or configure the backend and Ingress for end-to-end/re-encrypted TLS so traffic is never plaintext on the wire.

---

Ingress class and annotations

Theory

A cluster can run multiple Ingress controllers (e.g., an internal one and an external one, or NGINX plus a cloud controller). The IngressClass mechanism tells each Ingress object which controller should handle it. You reference it via spec.ingressClassName (the modern field; the older kubernetes.io/ingress.class annotation is deprecated). An IngressClass can be marked default so Ingresses without an explicit class use it.

Annotations are how you configure controller-specific behavior that the standard Ingress spec doesn't cover — rewrites, rate limits, timeouts, body size limits, auth, canary weights, and more. Because annotations are controller-specific, they are not portable between controllers. The portable, structured successor to all of this is the Gateway API, which moves many annotation-driven behaviors into typed resources (GatewayClass, Gateway, HTTPRoute) — worth knowing as the direction the ecosystem is heading.

Example

apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: nginx
  annotations:
    ingressclass.kubernetes.io/is-default-class: "true"   # default for unspecified
spec:
  controller: k8s.io/ingress-nginx
---

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "50m"    # controller-specific
    nginx.ingress.kubernetes.io/rate-limit-rps: "100"
spec:
  ingressClassName: nginx
  rules: [ ... ]

Exercises

1. (Beginner) What does ingressClassName on an Ingress determine?
2. (Beginner) How do you make an IngressClass the default for Ingresses that don't specify one?
3. (Intermediate) Why are Ingress annotations not portable across different controllers?
4. (Interview) The Gateway API is positioned as the successor to Ingress + annotations. What structural problems with the annotation approach does it aim to fix? (Hint: typed/role-oriented resources vs. unstructured controller-specific annotations.)

Answers

1. Which Ingress controller (via its IngressClass) should implement that Ingress's rules — important when multiple controllers run in the cluster.
2. Mark the IngressClass with the annotation ingressclass.kubernetes.io/is-default-class: "true"; Ingresses without an explicit ingressClassName then use it.
3. Annotations encode controller-specific features and use controller-specific keys/semantics (e.g., nginx.ingress.kubernetes.io/...). A different controller doesn't understand those keys, so the configuration silently doesn't apply — making annotation-based config non-portable.
4. Annotations are unstructured strings with no schema/validation, are controller-specific (non-portable), and overload a single resource with concerns belonging to different roles. The Gateway API replaces them with typed, validated, role-oriented resources (GatewayClass for infra providers, Gateway for cluster operators, HTTPRoute/etc. for app developers), enabling schema validation, clearer separation of responsibilities, portability across implementations, and richer routing expressed as first-class fields instead of opaque annotations.

---

6.4 Network Policies

By default, all Pods can talk to all Pods — which is convenient but insecure. NetworkPolicies let you segment the network and enforce least-privilege connectivity. This subchapter covers them.

---

Default allow vs deny behavior

Theory

A critical security default to internalize: by default, all Pods can communicate with all other Pods in the cluster, with no restrictions. The flat network model is "default allow." This is convenient but means a single compromised Pod can reach every other Pod (lateral movement).

A NetworkPolicy changes this for the Pods it selects. The key rule: as soon as any NetworkPolicy selects a Pod, that Pod becomes "default deny" for the direction(s) the policy covers — only traffic explicitly allowed by a policy is permitted; everything else is dropped. Pods not selected by any policy remain default-allow. A common pattern is to apply an explicit default-deny policy to a namespace (selecting all Pods with empty rules), then add specific allow policies on top — flipping the namespace to least-privilege.

Example

# Default-deny ALL ingress in a namespace: selects every Pod, allows nothing.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: secure
spec:
  podSelector: {}            # {} = all Pods in the namespace
  policyTypes:
  - Ingress                  # no ingress rules listed => deny all inbound

Without any policy:   anyone -> Pod        (default allow)
With this policy:     nothing -> selected Pods, unless another policy allows it

Exercises

1. (Beginner) What is the default Pod-to-Pod communication behavior in a fresh cluster?
2. (Beginner) What happens to a Pod's allowed traffic the moment a NetworkPolicy selects it?
3. (Intermediate) Write (in words) what a policy with podSelector: {} and policyTypes: [Ingress] and no ingress rules achieves.
4. (Interview) Why is "default allow" a security concern, and how does a default-deny-plus-explicit-allow strategy improve posture? (Hint: lateral movement, least privilege.)

Answers

1. Default allow — every Pod can reach every other Pod with no restrictions.
2. It switches from default-allow to default-deny for the covered direction(s): only traffic explicitly permitted by a NetworkPolicy is allowed; all other traffic in that direction is dropped.
3. It selects all Pods in the namespace and, because it lists Ingress as a policy type but specifies no ingress rules, it denies all inbound traffic to every Pod in that namespace (a namespace-wide default-deny for ingress).
4. With default allow, a single compromised Pod can freely reach every other Pod, enabling lateral movement and broad blast radius. Applying a namespace default-deny and then adding only the specific allow rules each workload needs enforces least privilege: Pods can talk only to their legitimate dependencies, so a compromise is contained to the explicitly permitted paths, dramatically limiting lateral movement.

---

Ingress and egress rules

Theory

NetworkPolicies control traffic in two directions, declared in policyTypes:

• Ingress rules: what is allowed to connect to the selected Pods (inbound).
• Egress rules: what the selected Pods are allowed to connect to (outbound).

Each rule specifies allowed peers (from for ingress, to for egress) and optionally ports. Peers can be Pods (by podSelector), namespaces (by namespaceSelector), or IP blocks (ipBlock, for external addresses). Rules are additive (a union — anything matching any rule is allowed) and whitelist-only (you can only allow, never explicitly deny; denial is implicit for anything not allowed once a policy applies).

A frequent gotcha: locking down egress can break DNS — Pods need to reach CoreDNS (UDP/TCP 53) to resolve names, so an egress policy must explicitly allow DNS or name resolution fails.

Example

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: api-policy, namespace: app }
spec:
  podSelector: { matchLabels: { app: api } }
  policyTypes: [ Ingress, Egress ]
  ingress:
  - from:
    - podSelector: { matchLabels: { app: frontend } }   # only frontend may call api
    ports:
    - { protocol: TCP, port: 8080 }
  egress:
  - to:
    - podSelector: { matchLabels: { app: db } }          # api may reach db
    ports:
    - { protocol: TCP, port: 5432 }
  - to:                                                   # allow DNS (critical!)
    - namespaceSelector: {}
    ports:
    - { protocol: UDP, port: 53 }
    - { protocol: TCP, port: 53 }

Exercises

1. (Beginner) What is the difference between an ingress rule and an egress rule?
2. (Beginner) Can a NetworkPolicy explicitly deny a specific source? Why or why not?
3. (Intermediate) Why might adding an egress policy suddenly break your application's ability to resolve Service names?
4. (Interview) NetworkPolicy rules are additive whitelists. Explain how multiple policies selecting the same Pod combine, and the implication for designing a policy set. (Hint: union of allows, no explicit deny.)

Answers

1. Ingress rules govern inbound connections to the selected Pods (who may connect to them); egress rules govern outbound connections from the selected Pods (what they may connect to).
2. No. NetworkPolicies are whitelist-only — you can only specify allowed traffic. Denial is implicit: once any policy selects a Pod for a direction, anything not explicitly allowed is denied. There is no "deny this specific peer" rule.
3. An egress policy makes the Pod default-deny for outbound traffic, so unless you explicitly allow egress to CoreDNS on port 53 (UDP/TCP), DNS queries are dropped and name resolution fails — breaking Service discovery even though the app and Services are fine.
4. When several policies select the same Pod, their allowed peers/ports are unioned — traffic is permitted if any applicable policy allows it. Because there's no explicit deny, you cannot subtract permissions with another policy; you can only widen what's allowed. Implication: design policies as the complete set of things each workload should be allowed to do, and rely on a default-deny baseline to forbid everything else — adding a permissive policy can only loosen, never tighten.

---

Pod selector and namespace selector

Theory

NetworkPolicy peers are chosen with selectors, and understanding how they combine is the crux of writing correct policies:

• podSelector (inside from/to) matches Pods by label within the policy's own namespace (unless combined with a namespaceSelector).
• namespaceSelector matches entire namespaces by their labels — allowing traffic from/to all Pods in those namespaces.
• When podSelector and namespaceSelector appear in the same from/to element (same list item), they are ANDed: "Pods matching X in namespaces matching Y." When they're in separate list items, they are ORed.

This AND/OR distinction is the single most common source of NetworkPolicy bugs. Note also that namespace selection relies on namespace labels (e.g., the built-in kubernetes.io/metadata.name label), so you select namespaces by labeling them, not by name directly in older setups.

Example

  ingress:
  # AND: Pods labeled app=frontend, only from namespaces labeled team=web
  - from:
    - namespaceSelector: { matchLabels: { team: web } }
      podSelector: { matchLabels: { app: frontend } }
  # vs. OR: anything in team=web namespaces, OR any Pod labeled app=frontend anywhere selected
  - from:
    - namespaceSelector: { matchLabels: { team: web } }
    - podSelector: { matchLabels: { app: frontend } }

Exercises

1. (Beginner) By default, podSelector inside a from block matches Pods in which namespace?
2. (Beginner) What does namespaceSelector match on?
3. (Intermediate) Explain the difference in meaning between combining podSelector and namespaceSelector in the same list item versus separate list items.
4. (Interview) A policy intended to allow "frontend Pods in the web namespace" is accidentally allowing all Pods in the web namespace AND all frontend Pods everywhere. What is the likely YAML mistake? (Hint: AND vs. OR via list structure.)

Answers

1. The policy's own namespace (the namespace where the NetworkPolicy is defined), unless a namespaceSelector is also specified.
2. Namespace labels — it selects whole namespaces whose labels match (e.g., kubernetes.io/metadata.name or custom labels), allowing traffic from/to all Pods in those namespaces.
3. In the same from/to list element, podSelector and namespaceSelector are ANDed ("Pods matching the podSelector that are in namespaces matching the namespaceSelector"). In separate list elements they are ORed ("any Pod in matching namespaces" OR "Pods matching the podSelector in the policy's namespace").
4. The two selectors were placed as separate list items (two - entries) instead of as two keys under a single list item. That makes them ORed — allowing all Pods in web namespaces and, separately, all frontend Pods. The fix is to put namespaceSelector and podSelector under the same list item so they're ANDed.

---

Network policy enforcement with Calico/Cilium

Theory

A subtle but vital point: the NetworkPolicy resource is part of the Kubernetes API, but enforcement is done by the CNI plugin — and not all CNIs enforce NetworkPolicy. If your CNI doesn't (e.g., plain Flannel), you can create NetworkPolicy objects and they will be silently ignored, giving a false sense of security. You must run a policy-capable CNI such as Calico or Cilium (or others like Antrea, Weave Net).

Beyond the standard API, these CNIs offer extended policy capabilities via their own CRDs:

• Calico: GlobalNetworkPolicy/NetworkPolicy with ordering, explicit deny/allow, richer selectors, and cluster-wide scope.
• Cilium: CiliumNetworkPolicy/CiliumClusterwideNetworkPolicy with identity-based and L7-aware rules (e.g., allow only GET /api HTTP calls), powered by eBPF.

The takeaway: choose a CNI that enforces policy, and reach for vendor CRDs when standard NetworkPolicy (L3/L4 only) is insufficient.

Example

# Cilium L7 policy: allow frontend to call api ONLY on GET /api/* (HTTP-aware)
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata: { name: l7-api }
spec:
  endpointSelector: { matchLabels: { app: api } }
  ingress:
  - fromEndpoints:
    - matchLabels: { app: frontend }
    toPorts:
    - ports: [ { port: "8080", protocol: TCP } ]
      rules:
        http:
        - { method: "GET", path: "/api/.*" }   # L7 rule, beyond standard NetworkPolicy

Exercises

1. (Beginner) Which component actually enforces NetworkPolicy?
2. (Beginner) What happens if you create NetworkPolicy objects on a CNI that doesn't support them?
3. (Intermediate) Name a capability that Cilium's CiliumNetworkPolicy offers that the standard NetworkPolicy API does not.
4. (Interview) A security audit finds NetworkPolicies defined but traffic flowing freely against them. What is the most likely root cause, and how would you confirm and fix it? (Hint: non-enforcing CNI.)

Answers

1. The CNI plugin (the network dataplane), not the Kubernetes control plane — the API only stores the policy objects.
2. They are accepted and stored by the API server but silently ignored (not enforced), so all traffic flows as if no policy existed — a dangerous false sense of security.
3. L7/HTTP-aware rules (e.g., allow only specific HTTP methods/paths), identity-based policy, DNS-aware egress, and clusterwide policies — capabilities beyond the standard API's L3/L4 (IP/port) scope.
4. The most likely cause is a CNI that does not enforce NetworkPolicy (e.g., plain Flannel). Confirm by checking which CNI is installed and whether it advertises policy support (and test connectivity that a policy should block). Fix by switching to or adding a policy-capable CNI such as Calico or Cilium (or a policy add-on), then verify the previously-defined policies now actually block traffic.

---

6.5 DNS in Kubernetes

DNS underpins service discovery; understanding CoreDNS configuration and the records it serves helps you debug and customize name resolution. This subchapter covers DNS specifics.

---

CoreDNS configuration

Theory

CoreDNS is the cluster DNS server (introduced in Chapter 2). Its behavior is defined by the Corefile, stored in the coredns ConfigMap in kube-system. The Corefile is a chain of plugins processed per query: the kubernetes plugin answers cluster records (Services/Pods) by watching the API server; forward sends non-cluster queries to upstream resolvers; cache caches responses; and others (errors, health, ready, loop, reload) provide operational features.

Customizing CoreDNS is done by editing this ConfigMap — for example, to add stub domains (forward a specific domain to a specific DNS server), rewrite queries, or tune caching. CoreDNS picks up changes automatically (the reload plugin). Because all in-cluster name resolution depends on it, CoreDNS configuration and health are central to cluster operability.

Example

# coredns ConfigMap (Corefile):
.:53 {
    errors
    health
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {   # cluster records
        pods insecure
        fallthrough in-addr.arpa ip6.arpa
    }
    forward . /etc/resolv.conf       # upstream for external names
    cache 30
    loop
    reload                           # auto-reload on ConfigMap change
}
# Custom stub domain example: forward acme.internal to a specific server
acme.internal:53 {
    forward . 10.10.0.53
}

Exercises

1. (Beginner) Where is CoreDNS's configuration stored?
2. (Beginner) What does the forward plugin do?
3. (Intermediate) How would you make CoreDNS resolve a custom internal domain via a specific external DNS server?
4. (Interview) The Corefile is a plugin chain processed per query. Explain how the kubernetes, forward, and cache plugins cooperate to resolve both cluster and external names efficiently. (Hint: cluster records vs. upstream, caching layer.)

Answers

1. In the coredns ConfigMap (the Corefile) in the kube-system namespace.
2. It forwards DNS queries that CoreDNS isn't authoritative for (e.g., external/internet names) to upstream resolvers (commonly the node's /etc/resolv.conf).
3. Add a server block / stub domain to the Corefile (via the coredns ConfigMap), e.g. acme.internal:53 { forward . <dns-server-ip> }, so queries for that domain are sent to the specified server. CoreDNS reloads the change automatically.
4. For each query, the chain is evaluated: the cache plugin first serves a cached answer if present (fast path). On a miss, the kubernetes plugin answers names in the cluster domain (*.svc.cluster.local, Pod records) from its API-server-watched data; anything outside the cluster domain falls through to forward, which queries upstream resolvers. The response is then cached for subsequent queries. This split lets cluster names resolve locally and instantly from live data while external names are delegated upstream, with caching reducing load and latency for both.

---

Service DNS records

Theory

CoreDNS publishes predictable DNS records for Services, which is the backbone of service discovery:

• Normal (ClusterIP) Service: an A/AAAA record at <service>.<namespace>.svc.cluster.local resolving to the Service's ClusterIP.
• Headless Service: A/AAAA records returning the individual Pod IPs (one per ready Pod), plus per-Pod records when backed by a StatefulSet.
• SRV records: for named ports, _<port-name>._<protocol>.<service>.<namespace>.svc.cluster.local lets clients discover port numbers, not just addresses.
• ExternalName Service: a CNAME to the external host.

These records update as Services and endpoints change. The consistent naming scheme means applications and operators can predict any Service's address from its name and namespace — no service registry to query manually.

Example

ClusterIP svc "api" in ns "shop":
  api.shop.svc.cluster.local            A   10.96.10.20

Headless svc "db" in ns "data":
  db.data.svc.cluster.local             A   10.244.1.5
                                        A   10.244.2.7

Named port "grpc" (SRV):
  _grpc._tcp.api.shop.svc.cluster.local SRV 0 100 50051 api.shop.svc.cluster.local

Exercises

1. (Beginner) What does the A record of a normal ClusterIP Service resolve to?
2. (Beginner) How does the DNS result for a headless Service differ from a ClusterIP Service?
3. (Intermediate) What are SRV records used for in Kubernetes DNS?
4. (Interview) Explain how the predictable Service DNS naming scheme eliminates the need for a separate service-registry lookup in application code. (Hint: name+namespace deterministically maps to an address.)

Answers

1. The Service's stable ClusterIP (virtual IP).
2. A ClusterIP Service's A record returns a single virtual IP; a headless Service's A records return the individual Pod IPs of all ready backends (no VIP), enabling direct per-Pod addressing.
3. SRV records expose named ports (and the associated host) so clients can discover the port number/protocol for a service, e.g., _grpc._tcp.<svc>.<ns>.svc.cluster.local, rather than hard-coding ports.
4. Every Service has a deterministic name <service>.<namespace>.svc.cluster.local that CoreDNS resolves to its current address, kept up to date as endpoints change. Applications simply use that name; the platform's DNS is the registry. There's no need to call a discovery API or maintain client-side registries — knowing a service's name and namespace is enough to reach it, and resolution always reflects the live set of healthy backends.

---

Pod DNS records

Theory

Beyond Services, individual Pods can also have DNS records. By default, a Pod gets an A record of the form <pod-ip-with-dashes>.<namespace>.pod.cluster.local (e.g., 10-244-1-5.default.pod.cluster.local) — rarely used directly. More useful are the StatefulSet per-Pod records via a headless Service: <pod-name>.<service>.<namespace>.svc.cluster.local (e.g., db-0.db.data.svc.cluster.local), which give stable, predictable names to individual stateful Pods.

A Pod's own DNS identity can be customized with hostname and subdomain fields: setting subdomain to a headless Service's name yields a stable record <hostname>.<subdomain>.<namespace>.svc.cluster.local. This is mostly relevant for clustered/stateful applications where members must address each other by stable names. For ordinary stateless Pods, you use the fronting Service's DNS, not per-Pod records.

Example

apiVersion: v1
kind: Pod
metadata:
  name: worker
  labels: { app: worker }
spec:
  hostname: worker-1            # sets the Pod's hostname
  subdomain: workers            # must match a headless Service named "workers"
  containers:
  - name: app
    image: myapp:1.0
---

apiVersion: v1
kind: Service
metadata: { name: workers }
spec:
  clusterIP: None              # headless, enables per-Pod records
  selector: { app: worker }
  ports: [ { port: 80 } ]
# Resulting record: worker-1.workers.<namespace>.svc.cluster.local

Exercises

1. (Beginner) What is the default DNS record format for an individual Pod?
2. (Beginner) Which workload type makes per-Pod DNS names genuinely useful, and what is the name format?
3. (Intermediate) How do the hostname and subdomain fields create a stable per-Pod DNS name?
4. (Interview) Why do stateless application Pods rarely need per-Pod DNS records, while clustered databases rely on them? (Hint: interchangeable replicas vs. members addressing specific peers.)

Answers

1. <pod-ip-with-dashes>.<namespace>.pod.cluster.local (e.g., 10-244-1-5.default.pod.cluster.local).
2. StatefulSets (with a headless governing Service), giving stable names of the form <pod-name>.<service>.<namespace>.svc.cluster.local (e.g., db-0.db.data.svc.cluster.local).
3. Setting a Pod's subdomain to the name of a headless Service (and hostname to a desired name) makes CoreDNS publish <hostname>.<subdomain>.<namespace>.svc.cluster.local for that Pod, giving it a stable, resolvable per-Pod DNS name tied to the headless Service.
4. Stateless Pods are interchangeable; clients reach them collectively through a single Service name/VIP, so individual Pod identity is irrelevant and per-Pod DNS adds nothing. Clustered databases have members that must contact specific peers (for replication, quorum, membership) and must recognize a restarted member as the same node — which requires stable, individually addressable DNS names, exactly what per-Pod records (via StatefulSet + headless Service) provide.

---

Custom DNS policies

Theory

Each Pod has a dnsPolicy that controls how its DNS resolution is configured, plus an optional dnsConfig for fine-grained customization. The policies:

dnsPolicy	Behavior
ClusterFirst (default)	Cluster DNS (CoreDNS) first; non-cluster names forwarded upstream.
Default	Inherit the node's /etc/resolv.conf (not cluster DNS).
ClusterFirstWithHostNet	Like ClusterFirst, but required when the Pod uses hostNetwork: true.
None	Ignore defaults entirely; use only what you provide in dnsConfig.

A common gotcha: a Pod with hostNetwork: true and the default ClusterFirst will not use cluster DNS — you must set ClusterFirstWithHostNet for it to resolve Service names. dnsConfig lets you add custom nameservers, searches, and options (e.g., tuning ndots, which affects how short names are expanded and can have performance implications).

Example

spec:
  dnsPolicy: "None"            # use ONLY the dnsConfig below
  dnsConfig:
    nameservers:
    - 10.96.0.10               # custom resolver(s)
    searches:
    - svc.cluster.local
    - example.com
    options:
    - { name: ndots, value: "2" }   # tune short-name expansion
---

spec:
  hostNetwork: true
  dnsPolicy: ClusterFirstWithHostNet   # needed so hostNetwork Pods use cluster DNS

Exercises

1. (Beginner) What is the default dnsPolicy for a Pod?
2. (Beginner) Which dnsPolicy makes a Pod use the node's resolv.conf instead of cluster DNS?
3. (Intermediate) A hostNetwork: true Pod can't resolve Service names. What is the fix?
4. (Interview) What does the ndots option control, and why can a high ndots value cause DNS performance problems for short external names? (Hint: search-domain expansion attempts.)

Answers

1. ClusterFirst.
2. Default (it inherits the node's /etc/resolv.conf).
3. Set dnsPolicy: ClusterFirstWithHostNet. With hostNetwork: true, the default ClusterFirst doesn't use cluster DNS, so Service names fail to resolve; the ...WithHostNet policy restores cluster DNS resolution for host-network Pods.
4. ndots is the threshold of dots in a name below which the resolver tries the search domains before treating the name as absolute. With the typical ndots: 5, a name with fewer than 5 dots (e.g., an external api.example.com, 2 dots) is first tried against each cluster search suffix (...svc.cluster.local, etc.), generating several failing lookups before the absolute query succeeds. A high ndots thus multiplies DNS queries for short external names, adding latency and load; lowering ndots (or using FQDNs with a trailing dot) reduces these wasteful lookups.

---