13. Service Mesh

As applications decompose into many microservices, the network between them becomes the hard part: secure connections, retries, timeouts, load balancing, traffic shaping, and observability — repeated in every service. A service mesh moves these cross-cutting concerns out of application code and into the infrastructure layer. This chapter explains the concept, then surveys the leading implementations: Istio, Linkerd, and Cilium's eBPF-based mesh.

---

13.1 Service Mesh Concepts

This subchapter builds the mental model: why meshes exist, how they're structured, and the patterns that define them.

---

Why service meshes exist

Theory

In a microservices system, every service-to-service call needs the same things: encryption (mTLS), retries and timeouts, load balancing, circuit breaking, and telemetry (metrics/traces/logs). Without a mesh, each team implements these in application code using language-specific libraries — inconsistently, repeatedly, and coupling business logic to networking concerns. Worse, you can't easily enforce a uniform policy (e.g., "all traffic must be encrypted") across services written in different languages.

A service mesh solves this by moving these concerns into a dedicated infrastructure layer that handles all service-to-service communication transparently. Applications just make normal network calls; the mesh intercepts them and applies mTLS, retries, traffic policy, and observability — without changing application code. The benefits: consistent, language-agnostic security and reliability; centralized policy and traffic control; and uniform, automatic observability of all inter-service traffic. The cost: added complexity and per-request latency/resource overhead from the proxies — so a mesh is justified when you have enough services that the consistency and control outweigh the overhead.

Example

Without a mesh: every service re-implements (per language):
  [svc-A: mTLS+retry+metrics lib]  ->  [svc-B: mTLS+retry+metrics lib]

With a mesh: concerns move to the infra layer (proxies):
  [svc-A]->[proxy] ===mTLS/retry/metrics===> [proxy]->[svc-B]
   (app code is unchanged; the mesh handles cross-cutting concerns)

Exercises

1. (Beginner) Name three cross-cutting concerns a service mesh handles.
2. (Beginner) Does adopting a service mesh require changing application code?
3. (Intermediate) Why is implementing these concerns in per-language libraries problematic at scale?
4. (Interview) What is the central trade-off of adopting a service mesh, and when is it justified? (Hint: consistency/control/observability vs. complexity and overhead; enough services to warrant it.)

Answers

1. Any three: mutual TLS encryption, retries, timeouts, load balancing, circuit breaking, traffic shaping/routing, and observability (metrics/traces/logs).
2. No — a mesh handles these concerns transparently at the infrastructure layer; applications make normal network calls and need no (or minimal) code changes.
3. Each service/language reimplements the same logic with different libraries, producing inconsistency (different behaviors/bugs), duplicated effort, coupling of networking to business code, and an inability to enforce uniform cross-language policy (e.g., guaranteeing mTLS everywhere). Upgrades/policy changes must be made in every codebase.
4. The trade-off is gaining consistent, language-agnostic security (mTLS), reliability (retries/timeouts/circuit breaking), centralized traffic control, and uniform observability — at the cost of added architectural complexity and per-request latency/CPU/memory overhead from the proxies and control plane. It's justified when you have enough services that managing these concerns in-app becomes inconsistent and unscalable, and the value of uniform security/observability/traffic control outweighs the operational and performance overhead. For a handful of services, the overhead often isn't worth it.

---

Data plane vs control plane in a mesh

Theory

A service mesh has the same two-plane structure as Kubernetes itself:

• Data plane: the network of proxies that actually carry and act on the traffic. Each service instance has a proxy (typically a sidecar) that intercepts all inbound/outbound traffic and applies the mesh's policies — mTLS, routing, retries, load balancing — and emits telemetry. The data plane is in the request path for every call.
• Control plane: the brain that configures and manages the data plane. It doesn't touch request traffic; instead it distributes configuration (routing rules, security policies, service discovery, certificates) to all the proxies and collects their telemetry. Operators interact with the control plane (via CRDs/APIs), which translates intent into proxy configuration.

This separation mirrors Kubernetes' API-server-vs-kubelet split: you declare desired behavior to the control plane, and the distributed data plane enforces it on live traffic. Istio's Envoy proxies + istiod, and Linkerd's micro-proxies + control plane, are concrete instances. Understanding which plane a component belongs to clarifies where latency, failures, and configuration live.

Example

            Control Plane (configures, no traffic)
            [ policies | routing | certs | discovery ]
                 |        distributes config        ^
                 v                                   | telemetry
   Data Plane:  [proxy]<->[proxy]<->[proxy]  (carries actual request traffic)
                  |         |         |
               [svc-A]   [svc-B]   [svc-C]

Exercises

1. (Beginner) Which plane carries the actual request traffic?
2. (Beginner) Does the control plane sit in the request path?
3. (Intermediate) What does the control plane distribute to the data plane?
4. (Interview) How does the mesh's data/control plane split mirror Kubernetes' own architecture, and why is this separation valuable? (Hint: declare to control plane, distributed enforcement; like API server vs kubelet.)

Answers

1. The data plane (the proxies).
2. No — the control plane configures and manages the proxies and collects telemetry but does not handle the actual request traffic.
3. Configuration and policy: routing rules, security/mTLS policies, certificates, service discovery information — which the proxies then enforce on live traffic (and it collects their telemetry).
4. Like Kubernetes' API server (declarative control) vs. kubelets/components (distributed execution), a mesh separates a control plane where you declare desired behavior (routing, security, policy) from a data plane of distributed proxies that enforce it on real traffic. This separation is valuable because it decouples policy/intent from enforcement: you manage behavior centrally and declaratively while enforcement scales out across every proxy, the control plane can fail without immediately dropping traffic (proxies keep using last config), and concerns (config vs. data path) are cleanly isolated for reasoning about latency, failures, and upgrades.

---

Sidecar proxy pattern

Theory

The classic way a mesh injects its data plane is the sidecar proxy pattern: a proxy container (e.g., Envoy) is added to every application Pod, sharing the Pod's network namespace. Traffic to and from the app container is transparently redirected (via iptables rules or, increasingly, eBPF) through the sidecar, which applies all mesh functions. The app is unaware — it thinks it's talking directly to other services.

Injection is usually automatic via a mutating admission webhook (Chapter 9) triggered by a namespace/Pod label, which adds the sidecar to the Pod spec at creation. Benefits: per-Pod isolation, language-agnostic, and fine-grained control. Drawbacks — which have driven recent innovation — include resource overhead (a proxy per Pod adds CPU/memory across the whole fleet), latency (extra hops), and operational complexity (sidecar lifecycle, startup ordering, upgrades). These costs motivated sidecar-less approaches: Istio's Ambient mode (shared per-node proxies + optional per-workload proxies) and Cilium's eBPF mesh, which reduce or eliminate per-Pod sidecars.

Example

Each Pod gets a sidecar proxy sharing its network namespace:
  +---------------- Pod ----------------+
  |  [app container]  <->  [envoy proxy] |  <- iptables/eBPF redirects traffic
  +-------------------------------------+
        outbound/inbound traffic always flows app <-> proxy <-> network

# Automatic sidecar injection via a namespace label (Istio example):
apiVersion: v1
kind: Namespace
metadata:
  name: app
  labels: { istio-injection: enabled }   # webhook injects the sidecar into Pods

Exercises

1. (Beginner) In the sidecar pattern, where does the proxy run relative to the app?
2. (Beginner) How is the sidecar typically injected into Pods automatically?
3. (Intermediate) Name two drawbacks of the per-Pod sidecar model.
4. (Interview) What sidecar-less approaches have emerged to address the sidecar model's costs, and what do they change? (Hint: Istio Ambient per-node proxies; Cilium eBPF.)

Answers

1. As a separate proxy container inside the same Pod as the app, sharing the Pod's network namespace, with traffic transparently redirected through it.
2. Via a mutating admission webhook (triggered by a namespace/Pod label like istio-injection: enabled) that adds the sidecar container to the Pod spec at creation time.
3. Any two: resource overhead (a proxy per Pod consumes CPU/memory across the whole fleet), added latency (extra in-path hops), and operational complexity (sidecar lifecycle, startup ordering with the app, and coordinated upgrades).
4. Sidecar-less approaches reduce or remove per-Pod proxies: Istio Ambient mode uses shared per-node proxies (ztunnel) for L4/mTLS plus optional per-workload (waypoint) proxies only where L7 features are needed, eliminating a sidecar in every Pod. Cilium's eBPF service mesh pushes much of the data-plane functionality into the kernel via eBPF (and shared per-node proxies for L7), avoiding per-Pod sidecars entirely. Both aim to cut the resource, latency, and operational overhead of injecting a proxy into every Pod while preserving mesh capabilities.

---

mTLS and service-to-service security

Theory

One of the most compelling reasons to adopt a mesh is automatic mutual TLS (mTLS) for all service-to-service traffic. In mTLS, both the client and server present certificates and verify each other's identity — providing encryption (traffic is confidential on the wire) and mutual authentication (each side cryptographically proves who it is). This is the foundation of zero-trust networking: services don't trust the network; they trust verified identities.

The mesh makes this effortless and transparent: the control plane acts as (or integrates with) a certificate authority, issuing short-lived identity certificates to each workload's proxy (often based on its ServiceAccount, following the SPIFFE/SPIFFE ID standard) and automatically rotating them. Proxies establish mTLS between each other without the application doing anything. On top of authenticated identity, the mesh enforces authorization policies ("service A may call service B's /api but not /admin"). The result: encryption-in-transit and strong identity for the whole fleet, configured centrally, with no app changes and no manual certificate management — something extremely hard to achieve consistently otherwise.

Example

mTLS handshake between proxies (both verify identities):
  [proxy A: cert=spiffe://cluster/ns/app/sa/web]
        <===== mutual TLS (encrypt + authenticate both sides) =====>
  [proxy B: cert=spiffe://cluster/ns/app/sa/api]
  control plane issues + auto-rotates these short-lived certs

# Istio: require strict mTLS for all workloads in a namespace
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata: { name: default, namespace: app }
spec:
  mtls: { mode: STRICT }     # only accept mTLS connections

Exercises

1. (Beginner) What two guarantees does mTLS provide?
2. (Beginner) Who issues and rotates the identity certificates in a mesh?
3. (Intermediate) How does mesh mTLS relate to a zero-trust security model?
4. (Interview) Why is achieving consistent, rotating mTLS across all services far easier with a mesh than implementing it in each application? (Hint: control plane CA + auto-rotation + transparent proxies vs. per-app cert management.)

Answers

1. Encryption of traffic in transit (confidentiality) and mutual authentication (both client and server cryptographically prove their identities).
2. The mesh's control plane (acting as or integrating with a certificate authority) issues short-lived workload identity certificates to each proxy and automatically rotates them (often using SPIFFE identities tied to the workload's ServiceAccount).
3. Zero-trust means not trusting the network and instead verifying identity for every connection. Mesh mTLS provides exactly that: each service proves its cryptographic identity on every call and traffic is encrypted, so trust is based on verified workload identity rather than network location — plus authorization policies restrict which identities may call which services.
4. The mesh provides a control-plane CA that automatically issues and rotates short-lived certificates per workload, and the proxies transparently negotiate mTLS on every connection — with no application code or manual cert handling. Doing this in-app would require every service (in every language) to obtain, store, rotate, and validate certificates, implement mutual TLS correctly and consistently, and coordinate a CA — error-prone, inconsistent, and operationally heavy. The mesh centralizes and automates all of it, guaranteeing uniform encryption and identity fleet-wide effortlessly.

---

13.2 Istio

Istio is the most feature-rich service mesh. This subchapter covers its architecture, traffic management, observability, and security.

---

Istio architecture and components

Theory

Istio is the most feature-complete and widely known service mesh. Its architecture follows the data/control plane split:

• Data plane: Envoy proxies — a high-performance, programmable L7 proxy — deployed as sidecars (or, in Ambient mode, as shared node/waypoint proxies). Envoy does the actual mTLS, routing, retries, load balancing, and telemetry.
• Control plane: istiod, a single consolidated component (earlier versions had separate Pilot/Citadel/Galley) that handles configuration distribution (translating Istio CRDs into Envoy config), service discovery, and certificate issuance/rotation (acting as the CA).

You interact with Istio through its CRDs — VirtualService, DestinationRule, Gateway, PeerAuthentication, AuthorizationPolicy, etc. — which istiod compiles into Envoy configuration pushed to the proxies. Istio also provides ingress/egress Gateways (Envoy-based edge proxies) for traffic entering/leaving the mesh. Its strength is breadth and power (rich L7 traffic management, security, and policy); its historical reputation is complexity, which Ambient mode and consolidation into istiod aim to reduce.

Example

Control plane:   [ istiod ]  (config + discovery + CA; compiles CRDs -> Envoy config)
                      |  pushes config / issues certs
Data plane:   [Envoy]<->[Envoy]<->[Envoy]   (sidecars carrying traffic)
Edge:         [Ingress Gateway (Envoy)] -> into the mesh

istioctl install --set profile=demo        # install Istio
kubectl label namespace app istio-injection=enabled
istioctl proxy-status                       # check Envoy/config sync status

Exercises

1. (Beginner) What proxy does Istio use for its data plane?
2. (Beginner) What is istiod responsible for?
3. (Intermediate) How do you configure Istio's behavior, and how does that reach the proxies?
4. (Interview) Istio is powerful but historically complex. What architectural changes have aimed to reduce that complexity? (Hint: consolidation into istiod; Ambient mode.)

Answers

1. Envoy.
2. The control plane functions: distributing configuration (translating Istio CRDs into Envoy config), service discovery, and certificate issuance/rotation (acting as the mesh CA).
3. Through Istio CRDs (VirtualService, DestinationRule, Gateway, PeerAuthentication, AuthorizationPolicy, etc.). istiod watches these, compiles them into Envoy configuration, and pushes that config to the Envoy proxies in the data plane, which enforce it on traffic.
4. Earlier Istio had multiple separate control-plane components (Pilot, Citadel, Galley, Mixer), which were consolidated into a single istiod, simplifying deployment/operations (and Mixer was removed). More recently, Ambient mode removes per-Pod sidecars in favor of shared per-node L4 proxies (ztunnel) plus optional L7 waypoint proxies, reducing resource overhead and operational complexity. Together these reduce Istio's footprint and the burden of managing sidecars everywhere.

---

VirtualService and DestinationRule

Theory

Istio's two core traffic-management CRDs work together:

• VirtualService: defines routing rules — how requests for a service are matched and routed. It can route by URI/header/weight to different subsets or services, enabling canary splits (90/10), header-based routing (A/B testing), URL rewrites, redirects, retries, timeouts, and fault injection. It answers "where should this request go?"
• DestinationRule: defines policies applied after routing to a destination — load balancing algorithm, connection pool limits, outlier detection (circuit breaking), and TLS settings. Critically, it defines subsets (named groups of a service's Pods by labels, e.g., version: v1/v2) that VirtualServices route to. It answers "how should traffic to this destination behave?"

The pattern: a DestinationRule declares the subsets and traffic policy; a VirtualService routes traffic across those subsets with weights/match rules. Together they give precise, declarative control over service traffic — the foundation for canary releases, circuit breaking, and resilience, all without app changes.

Example

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata: { name: reviews }
spec:
  host: reviews
  subsets:                          # define named subsets by labels
  - { name: v1, labels: { version: v1 } }
  - { name: v2, labels: { version: v2 } }
  trafficPolicy:
    connectionPool: { tcp: { maxConnections: 100 } }
    outlierDetection: { consecutive5xxErrors: 5, interval: 30s }  # circuit breaking
---

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata: { name: reviews }
spec:
  hosts: [ reviews ]
  http:
  - route:                          # canary: 90% v1, 10% v2
    - { destination: { host: reviews, subset: v1 }, weight: 90 }
    - { destination: { host: reviews, subset: v2 }, weight: 10 }

Exercises

1. (Beginner) What does a VirtualService control versus a DestinationRule?
2. (Beginner) What are "subsets," and where are they defined?
3. (Intermediate) How would you implement a 90/10 canary split between two versions using these resources?
4. (Interview) Explain how VirtualService and DestinationRule together enable canary releases and circuit breaking without changing application code. (Hint: subsets + weighted routing; outlier detection policy.)

Answers

1. A VirtualService controls routing — matching and directing requests (by weight/URI/header) to destinations/subsets (the "where"). A DestinationRule controls destination policies applied after routing — load balancing, connection pools, outlier detection/circuit breaking, TLS — and defines subsets (the "how").
2. Subsets are named groups of a service's Pods selected by labels (e.g., version: v1 vs version: v2). They're defined in the DestinationRule and referenced by VirtualServices for routing.
3. Define a DestinationRule with two subsets (v1, v2) by version label, then a VirtualService routing to those subsets with weight: 90 for v1 and weight: 10 for v2 — sending 10% of traffic to the new version.
4. The DestinationRule declares subsets (e.g., v1/v2 by label) and traffic policies including outlier detection (eject unhealthy instances = circuit breaking). The VirtualService routes traffic across those subsets with weights/match rules. So a canary is just shifting weights between subsets (90/10 → 50/50 → 100%), and circuit breaking is the DestinationRule's outlier-detection policy automatically removing failing endpoints — all expressed declaratively in mesh config and enforced by the Envoy proxies, with the application completely unaware and unchanged.

---

Traffic management and load balancing

Theory

Beyond canary routing, Istio provides rich traffic management capabilities, all enforced by Envoy without app changes:

• Load balancing algorithms: round-robin, least-request, random, consistent hashing (for session affinity by header/cookie) — configured in the DestinationRule.
• Resilience: timeouts, retries (with per-try timeouts and retry conditions), and circuit breaking/outlier detection (eject endpoints returning errors), making the system tolerant of slow/failing instances.
• Fault injection: deliberately inject delays or aborts (errors) into a percentage of requests to test resilience (chaos testing) — without touching code.
• Traffic mirroring (shadowing): send a copy of live traffic to a new version to test it with real load while the response is discarded — safe pre-production validation.
• Edge control via Gateways: manage ingress/egress traffic at the mesh boundary.

These turn the network into a programmable, resilient layer: you declaratively shape, protect, and test traffic flows. The combination of fine-grained routing + resilience + testing primitives is a major reason teams adopt Istio for complex traffic scenarios.

Example

# Retries, timeout, and fault injection on a route (VirtualService):
spec:
  hosts: [ ratings ]
  http:
  - route: [ { destination: { host: ratings } } ]
    timeout: 2s
    retries: { attempts: 3, perTryTimeout: 1s, retryOn: 5xx,connect-failure }
    fault:
      delay: { percentage: { value: 10 }, fixedDelay: 5s }   # inject latency in 10%
---

# Traffic mirroring: send a copy to v2 while serving from v1
  http:
  - route: [ { destination: { host: app, subset: v1 } } ]
    mirror: { host: app, subset: v2 }
    mirrorPercentage: { value: 20 }

Exercises

1. (Beginner) Name two resilience features Istio provides for traffic.
2. (Beginner) What is traffic mirroring (shadowing) used for?
3. (Intermediate) What is fault injection, and why would you use it?
4. (Interview) How does Istio's traffic management turn the network into a "programmable, resilient layer," and what's the advantage of doing this in the mesh rather than in code? (Hint: declarative retries/timeouts/circuit-breaking/testing; consistent, language-agnostic, no redeploy.)

Answers

1. Any two: timeouts, retries (with per-try timeout/conditions), and circuit breaking/outlier detection.
2. Sending a copy of live production traffic to another version (while discarding its responses) to test the new version against real traffic safely, without affecting users.
3. Deliberately injecting failures — delays (latency) or aborts (error responses) — into a configurable percentage of requests. It's used to test and validate system resilience (chaos/fault testing): verifying timeouts, retries, fallbacks, and circuit breakers behave correctly under failure, without modifying application code.
4. Istio lets you declaratively configure routing, retries, timeouts, circuit breaking, load-balancing strategies, fault injection, and mirroring — all enforced by Envoy on live traffic. This makes the network programmable (shape/route traffic via config) and resilient (automatic handling of slow/failing instances). Doing it in the mesh rather than code means the behavior is consistent and language-agnostic across all services, centrally managed and auditable, changeable at runtime without redeploying applications, and decoupled from business logic — far more uniform and flexible than each team implementing resilience libraries differently.

---

Observability with Istio (Kiali, Jaeger)

Theory

Because every request passes through Envoy proxies, Istio gets observability for free — it can emit consistent metrics, distributed traces, and access logs for all service-to-service traffic without app instrumentation (beyond trace-header propagation). This produces the "golden signals" (latency, traffic, errors, saturation) uniformly across the mesh.

Istio integrates with a standard observability stack:

• Metrics: Envoy exports rich metrics (request rates, error rates, latencies per service) scraped by Prometheus, visualized in Grafana.
• Distributed tracing: Envoy generates spans and reports to Jaeger (or Tempo/Zipkin) — though the application must propagate the trace headers so spans link across hops (the one piece the mesh can't do alone).
• Service-graph visualization: Kiali is Istio's dedicated console that renders a live topology graph of services and their traffic (rates, health, mTLS status), and helps validate/edit Istio config.

So Istio + Prometheus/Grafana + Jaeger + Kiali gives metrics, traces, and a visual service map of the entire mesh largely automatically — a powerful, consistent observability layer that's hard to achieve with per-service instrumentation alone.

Example

Envoy proxies emit telemetry for every call:
  metrics  -> Prometheus -> Grafana   (rates, errors, latency)
  traces   -> Jaeger      (per-request spans; app must propagate headers)
  topology -> Kiali       (live service graph: traffic, health, mTLS)

istioctl dashboard kiali       # open the service-graph console
istioctl dashboard jaeger      # open distributed tracing

Exercises

1. (Beginner) Why does Istio provide observability with little app instrumentation?
2. (Beginner) What does Kiali provide?
3. (Intermediate) For distributed tracing to work across hops in Istio, what must the application still do?
4. (Interview) How does proxy-based telemetry give consistent "golden signals" across a polyglot fleet, and what's its limitation regarding tracing? (Hint: uniform metrics from Envoy; apps must propagate trace context.)

Answers

1. Because all traffic flows through the Envoy proxies, which automatically emit consistent metrics, traces, and access logs for every request — so the mesh produces telemetry without requiring per-application instrumentation.
2. Kiali is Istio's observability console that visualizes the mesh as a live service topology/graph (traffic rates, health, mTLS status between services) and helps inspect, validate, and configure Istio resources.
3. The application must propagate the trace context headers (e.g., W3C traceparent/B3 headers) from incoming requests to outgoing requests, so that spans generated by different proxies link into a single end-to-end trace. The mesh can generate spans but cannot stitch them across hops without the app forwarding the headers.
4. Every service's traffic passes through identical Envoy proxies, which emit the same set of metrics (request rate, error rate, latency, etc.) regardless of the application's language or framework — yielding uniform golden signals across a polyglot fleet automatically, with no per-language instrumentation. The limitation is tracing: while proxies create spans, they can't correlate them across services unless the application forwards the trace headers between inbound and outbound calls — so distributed tracing still requires that small piece of app cooperation (header propagation), even though metrics and topology come for free.

---

Istio security policies

Theory

On top of automatic mTLS (identity + encryption), Istio enforces security policy via two main CRDs:

• PeerAuthentication: controls mTLS mode for workloads — e.g., STRICT (only accept mTLS), PERMISSIVE (accept both mTLS and plaintext, useful during migration), or DISABLE. Scoped mesh-wide, per-namespace, or per-workload.
• AuthorizationPolicy: defines access control — which identities (principals/ServiceAccounts), from which namespaces/sources, may access which workloads, on which paths/methods. It supports ALLOW and DENY actions and rich conditions (source identity, request path, method, headers, JWT claims). This is how you implement least-privilege service-to-service authorization ("only the web ServiceAccount may call payments on POST /charge").

Istio also supports RequestAuthentication for validating end-user JWTs (e.g., from an OIDC provider) at the proxy, enabling end-user auth at the edge/service. Together these give a layered, identity-based security model — encryption (PeerAuthentication mTLS) plus fine-grained authorization (AuthorizationPolicy) plus request-level auth (RequestAuthentication) — all enforced by the proxies, declaratively, without app code.

Example

# Require strict mTLS, then allow only "web" SA to POST to payments
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata: { name: default, namespace: payments }
spec: { mtls: { mode: STRICT } }
---

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata: { name: payments-allow, namespace: payments }
spec:
  selector: { matchLabels: { app: payments } }
  action: ALLOW
  rules:
  - from: [ { source: { principals: ["cluster.local/ns/app/sa/web"] } } ]
    to:   [ { operation: { methods: ["POST"], paths: ["/charge"] } } ]

Exercises

1. (Beginner) What does PeerAuthentication control?
2. (Beginner) What does AuthorizationPolicy let you define?
3. (Intermediate) Why is PERMISSIVE mTLS mode useful during a migration?
4. (Interview) How do PeerAuthentication, AuthorizationPolicy, and RequestAuthentication layer to form an identity-based security model? (Hint: encryption/identity + service authz + end-user JWT, all at the proxy.)

Answers

1. The mTLS mode for workloads — whether they require mTLS (STRICT), accept both mTLS and plaintext (PERMISSIVE), or disable it — scoped mesh-wide, per-namespace, or per-workload.
2. Fine-grained access control: which source identities (ServiceAccounts/principals), namespaces, or other attributes may access which workloads, on which paths/methods, via ALLOW/DENY rules and conditions (including JWT claims).
3. During migration to mTLS, not all clients may be sending mTLS yet. PERMISSIVE mode lets a workload accept both mTLS and plaintext simultaneously, so you can roll out sidecars/mTLS incrementally without breaking not-yet-migrated callers, then switch to STRICT once everything communicates over mTLS.
4. They layer complementary controls all enforced by the proxies: PeerAuthentication establishes encrypted, mutually-authenticated connections (workload identity + confidentiality via mTLS); AuthorizationPolicy uses those verified identities to enforce which services may call which (service-to-service least privilege, by path/method); and RequestAuthentication validates end-user credentials (JWTs from an OIDC provider) at the proxy for request-level/end-user auth. Together they provide encryption + strong workload identity + fine-grained service authorization + end-user authentication — a complete, identity-based, zero-trust security model configured declaratively without modifying application code.

---

13.3 Linkerd

Linkerd is the lightweight, simplicity-focused mesh. This subchapter covers its design and features.

---

Linkerd architecture and design philosophy

Theory

Linkerd is a CNCF-graduated service mesh whose guiding philosophy is simplicity, performance, and operational ease — deliberately the opposite of "kitchen-sink" feature breadth. Where Istio aims for maximum capability (and accepts complexity), Linkerd aims to do the essential mesh functions (mTLS, reliability, observability) with minimal overhead and a gentle learning curve.

Its defining technical choice is the micro-proxy: instead of Envoy, Linkerd uses linkerd2-proxy, an ultralight proxy written in Rust, purpose-built only for mesh sidecar duties. It's far smaller and lower-latency/lower-memory than a general-purpose proxy, which is central to Linkerd's performance claims. The control plane is similarly lean (destination, identity, proxy-injector components). The trade-off versus Istio: Linkerd intentionally offers fewer advanced traffic-management knobs, focusing on what most teams actually need. Teams choosing Linkerd typically value getting a secure, observable, reliable mesh running quickly with low overhead over having every possible feature.

Example

Aspect	Linkerd	Istio
Proxy	linkerd2-proxy (Rust micro-proxy)	Envoy (general-purpose L7)
Philosophy	Simplicity, low overhead	Maximum features/flexibility
Learning curve	Gentle	Steep
Advanced traffic mgmt	Limited (essentials)	Extensive
Resource footprint	Very low	Higher

linkerd install | kubectl apply -f -     # install control plane
linkerd check                            # validate the installation
kubectl annotate ns app linkerd.io/inject=enabled   # enable sidecar injection

Exercises

1. (Beginner) What is Linkerd's core design philosophy?
2. (Beginner) What proxy does Linkerd use, and in what language is it written?
3. (Intermediate) What is the main trade-off of choosing Linkerd over Istio?
4. (Interview) Why does Linkerd's purpose-built Rust micro-proxy matter for its performance and operational story? (Hint: tiny, fast, low-memory vs. general-purpose Envoy.)

Answers

1. Simplicity, performance, and operational ease — doing the essential mesh functions with minimal complexity and overhead, rather than maximizing features.
2. linkerd2-proxy, a purpose-built ultralight micro-proxy written in Rust.
3. Linkerd offers fewer advanced traffic-management/configuration features than Istio in exchange for far greater simplicity, lower resource overhead, and a gentler learning curve. You trade breadth/flexibility for ease of operation and performance.
4. Because the proxy runs in every meshed Pod and is in the path of every request, its size and efficiency dominate the mesh's overhead. A general-purpose proxy (Envoy) is powerful but heavier in memory/CPU/latency. Linkerd's Rust micro-proxy is purpose-built for only sidecar duties, making it extremely small, fast, and memory-light, with Rust providing memory safety without a GC. This yields low per-Pod overhead and latency at scale and fewer moving parts to operate — directly enabling Linkerd's simplicity and performance claims.

---

Installing and configuring Linkerd

Theory

Linkerd emphasizes a fast, validated install experience. The typical flow uses the linkerd CLI: you run linkerd check --pre to validate the cluster is ready, install the CRDs and control plane, and crucially linkerd check after each step — a hallmark feature that runs extensive health/configuration checks and tells you precisely if anything is wrong. This "check-driven" workflow is part of Linkerd's operational-ease philosophy.

Adding workloads to the mesh is done via injection annotations (linkerd.io/inject: enabled) on namespaces or workloads, which trigger the proxy-injector webhook to add the micro-proxy sidecar — usually applied by re-rolling deployments. Configuration is intentionally minimal: sensible defaults mean most teams need little tuning. Optional extensions add capabilities on top (e.g., linkerd viz for the observability dashboard, linkerd multicluster, linkerd jaeger), keeping the core lean while letting you opt into more. The overall experience targets "secure, observable mesh in minutes, with the tool telling you if something's off."

Example

linkerd check --pre                       # validate cluster readiness
linkerd install --crds | kubectl apply -f -
linkerd install | kubectl apply -f -      # install control plane
linkerd check                             # validate everything is healthy

linkerd viz install | kubectl apply -f -  # optional observability extension
kubectl annotate deploy web linkerd.io/inject=enabled && kubectl rollout restart deploy web

Exercises

1. (Beginner) What command validates a Linkerd installation's health?
2. (Beginner) How do you add a workload to the Linkerd mesh?
3. (Intermediate) What is the purpose of Linkerd extensions like linkerd viz?
4. (Interview) How does Linkerd's "check-driven" install workflow reflect its operational-ease philosophy? (Hint: automated validation catches misconfig early, lowers operational risk.)

Answers

1. linkerd check (and linkerd check --pre before installing).
2. By adding the injection annotation linkerd.io/inject: enabled to the namespace or workload (then rolling the workload so the proxy-injector webhook adds the micro-proxy sidecar).
3. Extensions add optional capabilities on top of the lean core — e.g., linkerd viz provides the observability dashboard/metrics, linkerd multicluster enables cross-cluster communication, linkerd jaeger adds tracing integration — so you opt into extra features without bloating the base installation.
4. The linkerd check command runs comprehensive validations of prerequisites, control-plane health, certificates, proxy versions, and configuration at every stage, clearly reporting any problems and how to fix them. This catches misconfigurations early and gives operators confidence the mesh is correctly set up, reducing the trial-and-error and operational risk common with complex meshes — directly embodying Linkerd's goal of being simple and safe to operate.

---

Automatic mTLS

Theory

A flagship Linkerd feature is automatic mTLS that's on by default — the moment a workload is meshed, its traffic to other meshed workloads is mutually authenticated and encrypted, with zero configuration. This reflects Linkerd's "secure by default, simple to operate" stance: you don't write policies to enable encryption; it just happens.

Mechanically, Linkerd's identity control-plane component acts as a CA, issuing each proxy a short-lived certificate (rotated automatically, by default roughly every 24 hours) tied to the workload's ServiceAccount identity. The micro-proxies establish mTLS transparently for TCP traffic between meshed Pods. Because certificates are short-lived and auto-rotated, the security posture is strong without operator effort or the risk of long-lived credentials. (You can layer authorization policies on top via Linkerd's policy resources, but the baseline encryption + identity requires nothing.) This "mTLS for free, by default" experience is a major part of Linkerd's appeal.

Example

# mTLS is automatic for meshed workloads — verify it's happening:
linkerd viz edges deployment -n app
# SRC      DST       CLIENT_ID                       SECURED
# web      payments  web.app.serviceaccount...       √  (mTLS in effect)

# Identity certs are short-lived and auto-rotated (no manual management).

Exercises

1. (Beginner) Does Linkerd require configuration to enable mTLS between meshed workloads?
2. (Beginner) Which component acts as the CA issuing workload certificates?
3. (Intermediate) What workload attribute is a Linkerd proxy's identity tied to, and why does short-lived rotation matter?
4. (Interview) How does "mTLS on by default" exemplify Linkerd's design philosophy, and what security benefit comes from automatic short-lived cert rotation? (Hint: secure-by-default/zero-config; reduced credential exposure.)

Answers

1. No — automatic mTLS is on by default for meshed workloads; no configuration is needed to enable encryption and mutual authentication between them.
2. The Linkerd identity control-plane component (acting as the certificate authority).
3. It's tied to the workload's ServiceAccount identity. Short-lived, auto-rotated certificates (e.g., ~24h) matter because they limit the window a leaked credential is useful, require no manual rotation, and avoid the risks of long-lived certs — strong security with no operator effort.
4. Enabling encryption and identity automatically, with zero configuration, embodies Linkerd's "simple and secure by default" philosophy — operators get a strong baseline (encrypted, mutually authenticated, identity-based traffic) without writing policy or managing certificates. The automatic short-lived rotation means credentials are continuously refreshed and expire quickly, so a compromised certificate is useless almost immediately and there are no long-lived secrets to leak or manually rotate — robust security that's effortless to maintain.

---

Linkerd observability features

Theory

Like Istio, Linkerd gives observability for free because all meshed traffic flows through its proxies — but consistent with its philosophy, it focuses on the golden metrics that matter most: success rate, request rate (RPS), and latency (p50/p95/p99) per service and per route. These are exposed through the linkerd viz extension, which includes a dashboard and powerful CLI commands for live inspection.

Distinctive Linkerd observability tools:

• linkerd viz stat: live golden metrics for deployments/services.
• linkerd viz top: a live, top-like view of requests by route.
• linkerd viz tap: stream individual live requests flowing through a proxy in real time (great for debugging).
• A Grafana/Prometheus integration and a web dashboard showing service health and dependency graphs.

The emphasis is on immediate, low-friction insight into service health — golden metrics and live request inspection out of the box — rather than a vast configurable telemetry pipeline. This makes diagnosing "is service X healthy, and what are its requests doing right now?" extremely fast, fitting Linkerd's simplicity-first ethos.

Example

linkerd viz stat deploy -n app
# NAME      SUCCESS    RPS    LATENCY_P95   LATENCY_P99
# web       99.8%      24.0   12ms          45ms
# payments  100.0%     5.0    8ms           20ms

linkerd viz top deploy/web -n app     # live per-route request view
linkerd viz tap deploy/web -n app     # stream individual live requests

Exercises

1. (Beginner) What three "golden metrics" does Linkerd focus on?
2. (Beginner) Which extension provides Linkerd's dashboard and metrics CLI?
3. (Intermediate) What does linkerd viz tap let you do that aggregate metrics cannot?
4. (Interview) How does Linkerd's observability approach reflect its simplicity-first philosophy compared to Istio's? (Hint: focused golden metrics + live request inspection out of the box vs. extensive configurable telemetry.)

Answers

1. Success rate, request rate (RPS), and latency (e.g., p50/p95/p99).
2. The linkerd viz extension.
3. tap streams individual live requests flowing through a proxy in real time (source, destination, path, response code, latency), letting you inspect actual ongoing traffic for debugging — which aggregate metrics (averages/percentiles) can't show at the per-request level.
4. Linkerd provides the most useful signals — golden metrics and live request inspection (stat/top/tap) — immediately and with minimal setup, focusing on quickly answering "is this service healthy and what are its requests doing?" rather than offering a vast, highly-configurable telemetry system. Istio (via Envoy) exposes a much broader, more configurable set of metrics and integrations, with corresponding complexity. Linkerd's curated, out-of-the-box, low-friction observability matches its overall simplicity-first, operational-ease philosophy.

---

13.4 Cilium Service Mesh

Cilium brings eBPF-based, sidecar-less mesh capabilities. This subchapter covers its distinctive approach.

---

eBPF-based networking

Theory

Cilium is a CNI (Chapter 6) and service mesh built on eBPF — a Linux kernel technology that lets you run sandboxed programs inside the kernel in response to events (network packets, syscalls) without modifying the kernel or loading modules. For networking, this means Cilium can implement routing, load balancing, network policy, and observability in the kernel data path, rather than via userspace proxies or long iptables rule chains.

Why this matters: traditional kube-proxy/iptables-based networking degrades as Services and rules grow (long, linearly-scanned chains), and userspace proxies add hops and overhead. eBPF programs execute efficiently in-kernel with hash-map lookups (effectively O(1)), enabling high-performance service load balancing, policy enforcement, and packet processing with minimal overhead. Cilium can even replace kube-proxy entirely. This eBPF foundation is what underpins Cilium's networking and its sidecar-light service-mesh approach — pushing mesh functionality into the kernel instead of a proxy in every Pod.

Example

Traditional:  packet -> iptables (long rule chains) / userspace proxy -> dest
Cilium/eBPF:  packet -> eBPF program in kernel (hash-map lookup, in-path) -> dest
              (efficient L3/L4 routing, LB, policy, telemetry in the kernel)

cilium install                         # install Cilium as CNI
cilium status                          # check eBPF datapath / kube-proxy replacement
# Cilium can replace kube-proxy (kubeProxyReplacement) using eBPF for Services.

Exercises

1. (Beginner) What is eBPF, in one sentence?
2. (Beginner) What traditional component can Cilium replace using eBPF?
3. (Intermediate) Why does eBPF-based networking scale better than iptables as Services grow?
4. (Interview) How does running networking logic in the kernel via eBPF benefit performance compared to userspace proxies and iptables? (Hint: no userspace hops, O(1) lookups, in-kernel data path.)

Answers

1. eBPF lets you run sandboxed programs inside the Linux kernel, triggered by events (like network packets or syscalls), without changing kernel source or loading kernel modules.
2. kube-proxy (Cilium can provide a kube-proxy replacement for Service load balancing using eBPF).
3. iptables-based service routing creates long rule chains that are scanned largely linearly, so per-packet processing and rule-update cost grow with the number of Services/endpoints. eBPF uses efficient in-kernel hash-map lookups (≈O(1)) for service/endpoint resolution, so performance stays roughly constant as Services scale, and updates are far cheaper.
4. eBPF programs execute in the kernel directly on the packet path, so traffic doesn't need to be copied to and from a userspace proxy (avoiding context switches and extra hops/latency), and service/endpoint resolution uses constant-time hash-map lookups instead of long iptables chains. The result is lower latency, higher throughput, and better scalability for routing, load balancing, and policy enforcement — achieved in the kernel data path rather than in userspace or via large rule sets.

---

Cilium as a CNI and service mesh

Theory

Cilium's distinctive proposition is unifying the CNI (networking) and service mesh layers using the same eBPF foundation — and doing much of the mesh without per-Pod sidecars. Because Cilium already operates in the kernel data path as the CNI, it can provide many mesh capabilities (L3/L4 load balancing, transparent mTLS/encryption, network policy, observability) directly via eBPF, with no sidecar proxy in every Pod. For L7 features (HTTP routing, L7 policy), it uses a shared per-node Envoy proxy rather than one per Pod.

The benefits: significantly lower overhead (no proxy injected into every Pod — saving CPU/memory across the fleet and removing sidecar lifecycle/latency issues) and a single integrated system for networking + security + mesh. This "sidecar-less mesh" is part of the same industry shift as Istio's Ambient mode. The trade-off: Cilium's mesh feature set differs from Istio's mature L7 traffic-management breadth, and it requires a recent kernel for full eBPF capabilities. For teams wanting performance and a unified networking+mesh stack, Cilium is compelling.

Example

Cilium (sidecar-less mesh):
  L3/L4, mTLS, policy, telemetry -> eBPF in kernel (no per-Pod sidecar)
  L7 (HTTP routing/policy)        -> shared per-NODE Envoy proxy (not per-Pod)

vs. sidecar mesh: a proxy injected into EVERY Pod (more overhead)

Exercises

1. (Beginner) What two layers does Cilium unify with one technology?
2. (Beginner) Does Cilium's mesh require a sidecar proxy in every Pod?
3. (Intermediate) How does Cilium handle L7 features if it avoids per-Pod sidecars?
4. (Interview) What are the benefits and trade-offs of Cilium's sidecar-less, eBPF-based mesh versus a traditional sidecar mesh? (Hint: lower overhead/integration vs. feature breadth/kernel requirements.)

Answers

1. The CNI (pod networking) and the service mesh — both built on eBPF in a single integrated system.
2. No — much of the mesh (L3/L4 load balancing, mTLS/encryption, policy, observability) is done in the kernel via eBPF without per-Pod sidecars.
3. It uses a shared per-node Envoy proxy for L7 functions (HTTP-aware routing, L7 policy), rather than injecting a proxy into every Pod — so L7 capability is available without per-Pod sidecar overhead.
4. Benefits: much lower overhead (no proxy in every Pod, saving fleet-wide CPU/memory and eliminating sidecar latency/lifecycle/startup-ordering issues), a unified networking+security+mesh stack on one eBPF foundation, and high performance from in-kernel processing. Trade-offs: the mesh feature set (especially mature, fine-grained L7 traffic management) may be less extensive than Istio's; full eBPF capabilities require a sufficiently recent Linux kernel; and it's a newer/distinct operational model. It suits teams prioritizing performance and an integrated stack over maximal L7 traffic-shaping features.

---

Hubble observability

Theory

Hubble is Cilium's observability layer, built on eBPF. Because Cilium sees all traffic in the kernel data path, Hubble can provide deep network visibility — flow-level observability of every connection (who talked to whom, on what port, allowed or denied by policy, L7 details like HTTP method/path/status) — without sidecars or app instrumentation. It's the eBPF-native answer to "what is happening on my network, right now and historically."

Hubble offers:

• hubble observe (CLI): stream live network flows with rich filtering (by namespace, pod, verdict, protocol).
• Hubble UI: a visual service map and flow viewer showing live service dependencies and traffic, including which flows are dropped by NetworkPolicy (invaluable for debugging policy).
• Metrics: export flow-based metrics to Prometheus/Grafana.

The standout capability is L3–L7 flow visibility tied to identity and policy verdicts — you can see exactly which connections are happening and why they're allowed or blocked, straight from the kernel. This makes Hubble especially powerful for debugging connectivity and NetworkPolicy issues that are otherwise opaque.

Example

hubble observe --namespace app --verdict DROPPED
# app/web -> app/payments:8080  HTTP/POST /charge  DROPPED (policy denied)
# Immediately shows which flows are blocked by NetworkPolicy and why.

hubble observe --namespace app --protocol http
# app/web -> app/api:8080  GET /users  200  (allowed, L7 visible)

Exercises

1. (Beginner) What is Hubble, and what does it provide?
2. (Beginner) Does Hubble require sidecars or app instrumentation for network visibility?
3. (Intermediate) Why is Hubble especially useful for debugging NetworkPolicy issues?
4. (Interview) How does Cilium's eBPF foundation enable Hubble to show identity- and policy-aware L3–L7 flow visibility that's hard to get otherwise? (Hint: kernel sees all traffic + identity + policy verdicts natively.)

Answers

1. Hubble is Cilium's eBPF-based observability layer; it provides deep, flow-level network visibility (connections, identities, L7 details like HTTP method/path/status, and policy verdicts) plus a service map, CLI, and metrics.
2. No — because Cilium observes traffic in the kernel data path, Hubble provides visibility without per-Pod sidecars or application instrumentation.
3. Hubble shows each flow's verdict (allowed/dropped) and the reason, so you can immediately see which connections NetworkPolicy is blocking (and which are permitted), turning otherwise-opaque "why can't A reach B?" policy debugging into a direct observation of dropped flows with their source/destination/identity.
4. Because Cilium operates in the kernel as the CNI, it inherently sees every packet/flow, knows the cryptographic/workload identity of endpoints, and is the component making policy decisions — so it can emit flow records annotated with source/destination identity, L3–L7 details, and the allow/deny verdict directly from the data path. Hubble surfaces this. Achieving the same with traditional tooling would require correlating data from proxies, iptables logs, and app instrumentation across services; Cilium produces it natively and uniformly from one kernel-level vantage point.

---

Cilium Network Policies

Theory

Standard Kubernetes NetworkPolicy is L3/L4 only (IP/port) and namespaced (Chapter 6). Cilium Network Policies (CRDs: CiliumNetworkPolicy and the cluster-wide CiliumClusterwideNetworkPolicy) extend this with eBPF-powered, identity-aware enforcement and richer capabilities:

• L7 policies: allow/deny based on application-layer attributes — e.g., permit only GET /api/* HTTP requests, or specific Kafka topics/gRPC methods — beyond just port access.
• Identity-based: policies are enforced on Cilium's cryptographic workload identity (derived from labels) rather than fragile IPs, which is more robust as Pods churn.
• DNS-aware (FQDN) egress: allow egress to specific domain names (e.g., api.stripe.com) rather than IP ranges — Cilium resolves and enforces by FQDN.
• Clusterwide scope: apply policies across all namespaces at once.

These run on the same high-performance eBPF data path. The result is far more expressive segmentation than standard NetworkPolicy — you can enforce least privilege at the application protocol level and by stable identity/FQDN, which is especially valuable for zero-trust and controlling egress to external services.

Example

# Cilium L7 + identity policy: allow web -> api ONLY for GET /api/*
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata: { name: api-l7, namespace: app }
spec:
  endpointSelector: { matchLabels: { app: api } }
  ingress:
  - fromEndpoints: [ { matchLabels: { app: web } } ]   # by identity (labels)
    toPorts:
    - ports: [ { port: "8080", protocol: TCP } ]
      rules: { http: [ { method: "GET", path: "/api/.*" } ] }   # L7 rule
---

# FQDN egress: allow only to a specific external domain
  egress:
  - toFQDNs: [ { matchName: "api.stripe.com" } ]

Exercises

1. (Beginner) Name two capabilities Cilium Network Policies add over standard NetworkPolicy.
2. (Beginner) What does FQDN-based egress policy allow you to specify?
3. (Intermediate) Why is identity-based (label-derived) enforcement more robust than IP-based?
4. (Interview) How do Cilium's L7 and FQDN-aware policies strengthen a zero-trust and egress-control posture beyond standard NetworkPolicy? (Hint: app-protocol least privilege; control external destinations by name.)

Answers

1. Any two: L7/application-layer rules (e.g., HTTP method/path, Kafka, gRPC), identity-based enforcement (by workload labels/identity rather than IP), DNS/FQDN-aware egress, and cluster-wide policy scope — all beyond standard NetworkPolicy's L3/L4, IP-based, namespaced model.
2. Egress allowed to specific domain names (FQDNs), e.g., api.stripe.com, rather than to raw IP ranges — Cilium resolves and enforces based on the domain.
3. Pod IPs are ephemeral and reused as Pods churn, so IP-based rules are fragile and can become stale or ambiguous. Cilium derives a stable cryptographic identity from workload labels, so policies follow the workload regardless of its current IP — more robust, accurate, and meaningful (you allow "the web service," not "whatever IP it currently has").
4. Standard NetworkPolicy can only allow/deny by IP/port, so it can't restrict what an allowed connection does or control external destinations by name. Cilium's L7 policies enforce least privilege at the application protocol level (e.g., only GET /api/*, specific methods/paths/topics), shrinking what a compromised or misbehaving client can do even on an allowed port. FQDN egress lets you permit traffic only to specific external services by domain (e.g., a payment API) instead of broad IP ranges, tightly controlling and auditing egress. Combined with identity-based enforcement, this delivers fine-grained, identity- and protocol-aware segmentation that substantially advances a zero-trust posture and egress control beyond what standard NetworkPolicy can express.

---