10. Observability

You cannot operate what you cannot see. Observability is the practice of understanding a system's internal state from its external outputs, traditionally organized into three pillars — logs, metrics, and traces — plus health probes, events, and auditing that are specific to Kubernetes. This chapter covers how to collect, aggregate, and act on each, so you can answer not just "is it up?" but "why is it slow/failing, and where?"

---

10.1 Logging

Logs are the most immediate window into what an application is doing. This subchapter covers reading logs, how Kubernetes handles them at the node level, and how to aggregate them cluster-wide.

---

kubectl logs and log streaming

Theory

The simplest observability tool is kubectl logs, which retrieves the stdout/stderr of a container. The Kubernetes logging convention is that applications should write logs to standard output and standard error, not to files — the container runtime captures these streams, and the kubelet exposes them. This is the twelve-factor "logs as event streams" principle.

Practical usage: -f streams (follows) logs live; --previous shows logs from a crashed/restarted container's prior instance (essential for debugging CrashLoopBackOff); -c selects a specific container in a multi-container Pod; --since/--tail bound the output; and -l with --prefix can read logs across Pods matching a label. The key limitation: kubectl logs reads from the node's local log files, so logs are lost when the Pod is deleted or the node is gone — which is exactly why cluster-level aggregation (later) is needed.

Example

kubectl logs web-abc                       # current logs
kubectl logs -f web-abc -c app             # stream container "app"
kubectl logs web-abc --previous            # logs from the crashed prior container
kubectl logs web-abc --since=1h --tail=100 # last hour, last 100 lines
kubectl logs -l app=web --prefix --all-containers   # across all matching Pods

Exercises

1. (Beginner) Where should containerized applications write their logs, by convention?
2. (Beginner) Which flag shows logs from a container that already crashed and restarted?
3. (Intermediate) Why does kubectl logs become useless after a Pod is deleted, and what does that imply?
4. (Interview) Explain the "logs as event streams" principle and why writing to stdout/stderr (rather than files) is the right model in Kubernetes. (Hint: runtime captures streams; decouples app from log routing.)

Answers

1. To standard output (stdout) and standard error (stderr).
2. --previous (kubectl logs <pod> --previous).
3. kubectl logs reads node-local log files for the Pod's containers; when the Pod is deleted (or the node is gone), those files are removed, so the logs disappear. This implies you need a cluster-level log aggregation system to persist logs beyond a Pod's lifetime.
4. The principle treats logs as a continuous stream of time-ordered events the app simply writes to stdout/stderr, leaving collection, routing, and storage to the platform. In Kubernetes the container runtime captures these streams and the kubelet exposes them, so apps don't manage log files, rotation, or shipping. This decouples the application from log infrastructure (a logging agent can collect and forward streams anywhere), keeps containers stateless, and makes log handling uniform across all workloads.

---

Node-level logging architecture

Theory

To understand where logs live, you need the node-level picture. When a container writes to stdout/stderr, the container runtime redirects those streams to log files on the node (typically under /var/log/pods/... and /var/log/containers/..., the latter being symlinks). The kubelet manages these and provides them to kubectl logs. The kubelet also handles log rotation (e.g., rotating at a size limit and keeping a few files) so logs don't fill the disk — which means old log lines are discarded over time.

This architecture has important consequences: logs are ephemeral and node-local — tied to the container's life on that node, rotated away, and lost on Pod/node deletion. There is no built-in cluster-wide log storage. This is by design: Kubernetes provides the raw streams and node-level handling, and leaves durable, searchable, aggregated logging to add-on systems (next topics). Knowing this explains both how kubectl logs works and why you must deploy a logging stack for production.

Example

Container stdout/stderr
   |  (container runtime)
   v
/var/log/pods/<ns>_<pod>_<uid>/<container>/0.log   (rotated by kubelet)
   ^
   |  kubectl logs reads here   |   a logging agent (DaemonSet) also tails here

# On a node, the raw log files the runtime/kubelet manage:
ls /var/log/containers/        # symlinks: <pod>_<ns>_<container>-<id>.log

Exercises

1. (Beginner) What component redirects container stdout/stderr to files on the node?
2. (Beginner) Where on the node are container logs typically stored?
3. (Intermediate) Why are node-level logs considered ephemeral, and name two events that cause log loss.
4. (Interview) Given Kubernetes provides no built-in cluster log storage, what architectural pattern is used to achieve durable, searchable logs? (Hint: agent per node tails files and forwards to a backend.)

Answers

1. The container runtime (with the kubelet managing/rotating the resulting files).
2. Under /var/log/pods/... (with symlinks in /var/log/containers/...).
3. They are stored as node-local files tied to the container instance and are rotated by the kubelet (old lines discarded) — so they don't persist indefinitely. Log loss occurs on log rotation (old entries dropped), Pod deletion, container removal, or node failure/replacement.
4. A node-level logging agent (typically a DaemonSet, e.g., Fluent Bit/Fluentd) runs on every node, tails the container log files (and adds Kubernetes metadata), and forwards them to a centralized backend (Elasticsearch/OpenSearch, Loki, a cloud log service). The backend provides durable storage, indexing, and search across the whole cluster — the standard cluster-level logging pattern that compensates for the absence of built-in central storage.

---

Cluster-level logging patterns

Theory

To make logs durable and queryable cluster-wide, you add a logging pipeline. Three common patterns for collecting application logs:

1. Node-level agent (DaemonSet) — the most common: one logging agent per node tails all containers' log files and forwards them. Efficient (one agent per node, app-agnostic, no app changes) — the default choice.
2. Sidecar container — a logging sidecar in the Pod reads logs (e.g., from a shared volume or app-specific files) and either streams them to stdout or forwards them directly. Used when the app writes to files instead of stdout, or needs per-app processing.
3. Application pushes directly — the app itself sends logs to the backend via a logging library. Simplest infrastructure but couples the app to the backend and is generally discouraged.

A complete pipeline has: collection (agent) → transport/processing (parse, enrich with Kubernetes metadata, filter) → storage/indexing (Elasticsearch/OpenSearch, Loki, cloud) → visualization/query (Kibana, Grafana). The node-agent pattern with a processing layer is the standard production architecture.

Example

Pattern 1 (recommended): node agent
 [app->stdout]->[node log files]->[Fluent Bit DaemonSet]->[Loki/ES]->[Grafana/Kibana]

Pattern 2: sidecar
 [app->file]->[sidecar reads file]->stdout or ->backend

Pattern 3: app pushes
 [app library]----------------------------->[backend]   (tight coupling)

Exercises

1. (Beginner) What is the most common cluster-level logging collection pattern?
2. (Beginner) Name the four stages of a complete logging pipeline.
3. (Intermediate) When would you use a logging sidecar instead of a node-level agent?
4. (Interview) Why is the node-level DaemonSet agent generally preferred over having each application push logs directly to the backend? (Hint: decoupling, no app changes, efficiency, consistency.)

Answers

1. A node-level logging agent deployed as a DaemonSet (one per node) that tails container logs and forwards them.
2. Collection (agent) → transport/processing (parse/enrich/filter) → storage/indexing (backend) → visualization/query (UI).
3. When the application writes logs to files rather than stdout/stderr (so the node agent can't capture them from the standard streams), or when a specific app needs custom per-Pod log processing/parsing before forwarding. The sidecar reads the app's files (often via a shared volume) and emits them to stdout or directly to the backend.
4. The DaemonSet agent collects logs from all containers without modifying applications, works uniformly across every workload, runs one efficient agent per node, and decouples apps from the logging backend (you can change backends without touching apps). Direct app-push couples each application to a specific backend/library, requires code changes in every app, scatters configuration, and makes backend changes painful — so the node agent is the more maintainable, consistent, and efficient choice.

---

Log aggregation with Fluentd, Fluent Bit, Loki

Theory

The concrete tools that implement the pipeline:

• Fluentd (CNCF): a mature, feature-rich log collector/processor with a huge plugin ecosystem for inputs, parsing, filtering, and outputs. Powerful but relatively heavier (Ruby-based) on resources.
• Fluent Bit (CNCF): a lightweight, high-performance collector (C-based) designed for low memory/CPU — ideal as the per-node DaemonSet agent. Often Fluent Bit collects/forwards and (optionally) Fluentd aggregates centrally; many setups now use Fluent Bit alone.
• Loki (Grafana): a log storage/query backend designed to be cost-efficient by indexing only labels (metadata) rather than full log text (unlike Elasticsearch, which does full-text indexing). It pairs naturally with Grafana for querying via LogQL and with Prometheus-style labels.

A very common modern stack is Fluent Bit (collect) → Loki (store) → Grafana (visualize) — lightweight and cost-effective. The classic alternative is the EFK stack (Elasticsearch, Fluentd/Fluent Bit, Kibana) with full-text search. Choice depends on cost, search needs, and existing tooling.

Example

Tool	Role	Characteristic
Fluentd	Collector/aggregator	Feature-rich, many plugins, heavier
Fluent Bit	Collector (agent)	Lightweight, fast, low footprint
Loki	Storage/query	Indexes labels only — cheap; LogQL + Grafana
Elasticsearch	Storage/query	Full-text index — powerful search, heavier/costlier

# Fluent Bit (DaemonSet) tailing container logs and shipping to Loki (conceptual):
[INPUT]   Name tail   Path /var/log/containers/*.log
[FILTER]  Name kubernetes   # enrich with pod/namespace/labels metadata
[OUTPUT]  Name loki   Host loki.logging.svc   Labels job=fluentbit

Exercises

1. (Beginner) Which is more lightweight: Fluentd or Fluent Bit?
2. (Beginner) What does Loki index, and how does that make it cost-efficient?
3. (Intermediate) Contrast Loki with Elasticsearch for log storage.
4. (Interview) Why might a team choose a Fluent Bit + Loki + Grafana stack over the classic EFK stack? (Hint: resource/cost efficiency, label-based indexing, Grafana unification vs. full-text search needs.)

Answers

1. Fluent Bit (it's a lightweight, C-based collector with low memory/CPU footprint, versus the heavier Ruby-based Fluentd).
2. Loki indexes only labels/metadata (not the full log text). By avoiding expensive full-text indexing and storing compressed log chunks in cheap object storage, it greatly reduces storage and compute cost.
3. Loki indexes labels only (cheap, lower resource use, queried via LogQL, integrates with Grafana and Prometheus-style labels) but has weaker arbitrary full-text search. Elasticsearch performs full-text indexing of log content (powerful, fast ad-hoc text search and analytics) at the cost of much higher storage, memory, and operational overhead.
4. The Fluent Bit + Loki + Grafana stack is lighter and cheaper: Fluent Bit uses minimal node resources, Loki's label-only indexing slashes storage/compute costs, and Grafana unifies logs with metrics/traces in one UI (good for correlation). EFK offers superior full-text search and analytics but is heavier and costlier to run and scale. A team that values cost efficiency, a unified Grafana experience, and label-based querying — and doesn't need heavy full-text search — would prefer the Loki stack.

---

10.2 Metrics and Monitoring

Metrics are numeric time-series that reveal trends, capacity, and health. Prometheus and Grafana dominate this space. This subchapter covers the metrics ecosystem.

---

Metrics Server and resource metrics

Theory

The Metrics Server (introduced in Chapter 2) is the lightweight component that provides real-time resource metrics — current CPU and memory usage of Pods and nodes — via the Metrics API (metrics.k8s.io). It powers kubectl top and the Horizontal/Vertical Pod Autoscalers. It scrapes the kubelets' Summary API and keeps only the latest values in memory (no history).

The crucial framing (worth repeating because it's commonly misunderstood): Metrics Server is not a monitoring system. It exists solely to supply current resource metrics for autoscaling and kubectl top. It cannot show trends, retain history, alert, or store custom application metrics. For all of that — historical dashboards, alerting, custom/business metrics — you use Prometheus (next). So a typical cluster runs both: Metrics Server for autoscaling/top, and Prometheus for full monitoring.

Example

kubectl top nodes
# NAME     CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
# node-1   320m         16%    2100Mi          54%

kubectl top pods -n prod --containers
# POD     NAME   CPU(cores)   MEMORY(bytes)
# web-1   app    45m          180Mi

Exercises

1. (Beginner) What two things does the Metrics Server primarily power?
2. (Beginner) Does the Metrics Server store historical metrics?
3. (Intermediate) Why do clusters typically run both Metrics Server and Prometheus?
4. (Interview) A teammate proposes using Metrics Server data to build CPU-usage dashboards and alerts. Why won't that work, and what's the right tool? (Hint: in-memory latest values only; Prometheus for history/alerting.)

Answers

1. kubectl top (node/Pod resource usage) and the autoscalers (HPA/VPA) via the Metrics API.
2. No — it keeps only the latest values in memory, with no historical retention.
3. They serve different purposes: Metrics Server provides lightweight, real-time resource metrics for autoscaling and kubectl top, while Prometheus provides durable historical storage, custom/application metrics, querying (PromQL), dashboards, and alerting. Neither replaces the other, so both run together.
4. Metrics Server only holds the most recent CPU/memory readings in memory with no history, no long-term storage, and no alerting capability — so you can't build time-series dashboards or alert rules from it. The right tool is Prometheus, which scrapes and stores metrics over time, supports PromQL queries, integrates with Grafana for dashboards, and with Alertmanager for alerting.

---

Prometheus operator and ServiceMonitors

Theory

Prometheus is the de facto standard monitoring system for Kubernetes (a CNCF graduated project). It works on a pull model: it periodically scrapes HTTP /metrics endpoints exposed by targets (apps, exporters, Kubernetes components) and stores the results as time series in its own TSDB, queryable with PromQL. It discovers targets dynamically via Kubernetes service discovery.

Managing Prometheus config by hand is tedious, so the Prometheus Operator (often deployed via the kube-prometheus-stack Helm chart, which bundles Prometheus, Grafana, Alertmanager, and exporters) manages it through CRDs. The key one is the ServiceMonitor (and PodMonitor): instead of editing Prometheus's scrape config, you declare a ServiceMonitor that says "scrape Services with these labels on this port/path," and the Operator generates the config automatically. This makes monitoring declarative and self-service — teams add a ServiceMonitor alongside their app to get it scraped.

Example

# Declaratively tell Prometheus (via the Operator) to scrape this app:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: web
  labels: { release: kube-prometheus-stack }   # matched by the Prometheus selector
spec:
  selector:
    matchLabels: { app: web }                   # which Services to scrape
  endpoints:
  - port: metrics                               # the Service port exposing /metrics
    path: /metrics
    interval: 30s

Exercises

1. (Beginner) Does Prometheus use a push or pull model for collecting metrics?
2. (Beginner) What query language does Prometheus use?
3. (Intermediate) What does a ServiceMonitor let you do without editing Prometheus's raw config?
4. (Interview) Explain how the Prometheus Operator and ServiceMonitors make monitoring declarative and self-service in a multi-team cluster. (Hint: CRDs generate scrape config; teams ship monitors with their apps.)

Answers

1. Pull — Prometheus scrapes targets' /metrics endpoints on an interval.
2. PromQL.
3. Declare which Services/endpoints Prometheus should scrape (by label selector, port, path, interval). The Operator translates ServiceMonitors into Prometheus scrape configuration automatically, so you never hand-edit the Prometheus config.
4. The Prometheus Operator watches ServiceMonitor/PodMonitor CRDs and automatically generates and reloads Prometheus's scrape configuration. Each team can ship a ServiceMonitor alongside their application declaring how to scrape it; the Operator picks it up and Prometheus starts scraping — no central config edits or Prometheus restarts. This makes monitoring declarative (config-as-Kubernetes-objects, version-controllable) and self-service (teams onboard their own metrics) while keeping a single managed Prometheus, which scales cleanly across many teams.

---

kube-state-metrics

Theory

There's an important distinction between two kinds of cluster metrics:

• Resource usage metrics (from Metrics Server / cAdvisor / node-exporter): how much CPU/memory is being consumed.
• Object state metrics (from kube-state-metrics): the state of Kubernetes objects themselves — e.g., how many replicas a Deployment desires vs. has available, whether a Pod is in CrashLoopBackOff, how many Jobs failed, whether a node is unschedulable, PVC phase, etc.

kube-state-metrics (KSM) is a service that listens to the Kubernetes API and exposes the status of objects as Prometheus metrics. It does not measure resource consumption — it reflects the desired vs. actual state and conditions of API objects. This is what lets you alert on "a Deployment has fewer available replicas than desired for 10 minutes" or "Pods are restarting frequently." KSM is a standard component of the kube-prometheus stack, complementing node-exporter (node/hardware metrics) and the app metrics.

Example

# Object-state metrics exposed by kube-state-metrics, queried in PromQL:
kube_deployment_status_replicas_available{deployment="web"}
kube_deployment_spec_replicas{deployment="web"}
# Alert when available < desired:
kube_deployment_status_replicas_available < kube_deployment_spec_replicas

kube_pod_container_status_restarts_total{namespace="prod"}   # restart counts

Exercises

1. (Beginner) What does kube-state-metrics expose, as opposed to Metrics Server?
2. (Beginner) Does kube-state-metrics measure CPU/memory consumption?
3. (Intermediate) Give two examples of conditions you can alert on using kube-state-metrics.
4. (Interview) Clarify the roles of node-exporter, Metrics Server, and kube-state-metrics — what distinct question does each answer? (Hint: node/hardware usage; live resource usage for autoscaling; object desired-vs-actual state.)

Answers

1. The state of Kubernetes API objects (e.g., Deployment desired/available replicas, Pod phase/restarts, Job success/failure, node conditions) as Prometheus metrics — not resource usage.
2. No — it reports object state/conditions, not resource consumption.
3. Any two: a Deployment's available replicas below desired for some duration; Pods stuck in Pending or CrashLoopBackOff; high container restart counts; failed Jobs; PVCs stuck unbound; nodes marked unschedulable/NotReady.
4. node-exporter: node/host and hardware-level resource metrics (CPU, memory, disk, network of the machine). Metrics Server: current Pod/node CPU & memory usage exposed via the Metrics API for kubectl top and autoscaling (real-time, no history). kube-state-metrics: the desired-vs-actual state and conditions of Kubernetes objects (replicas, phases, restarts). Roughly: node-exporter = "how is the machine doing?", Metrics Server = "how much is this Pod using right now (for scaling)?", kube-state-metrics = "what does Kubernetes think the state of my objects is?".

---

Grafana dashboards for Kubernetes

Theory

Grafana is the standard visualization layer. It connects to data sources — most commonly Prometheus (metrics), and also Loki (logs) and Tempo/Jaeger (traces) — and renders dashboards of panels (graphs, gauges, tables, heatmaps) driven by queries (PromQL/LogQL). It turns raw time-series into human-readable views of cluster and application health.

Grafana's strengths for Kubernetes: a huge library of prebuilt community dashboards (import by ID) for cluster overview, node/Pod resources, workloads, and many exporters; templating/variables (e.g., a namespace or Pod dropdown) for reusable dashboards; and the ability to unify all three observability pillars (metrics, logs, traces) in one place with cross-links (click a metric spike → jump to the relevant logs/traces). The kube-prometheus-stack ships Grafana pre-configured with Prometheus and a set of Kubernetes dashboards out of the box.

Example

Grafana data sources:
  Prometheus  -> metrics panels (PromQL)
  Loki        -> logs panels (LogQL)
  Tempo/Jaeger-> trace views

Dashboard variables (templating):
  $namespace = label_values(kube_pod_info, namespace)
  Panel query: sum(rate(container_cpu_usage_seconds_total{namespace="$namespace"}[5m])) by (pod)

# Import a community Kubernetes dashboard by its ID in the Grafana UI
# (e.g., "Kubernetes / Compute Resources / Namespace (Pods)").

Exercises

1. (Beginner) What is Grafana's primary role, and name two data sources it commonly uses.
2. (Beginner) How can you quickly get a Kubernetes dashboard without building it from scratch?
3. (Intermediate) What do Grafana template variables enable?
4. (Interview) How does Grafana help correlate across the three observability pillars, and why is that valuable during an incident? (Hint: unified UI, cross-links from a metric spike to logs/traces.)

Answers

1. Visualization of observability data via dashboards/panels. Common data sources: Prometheus (metrics), Loki (logs), Tempo/Jaeger (traces).
2. Import a prebuilt community dashboard by its ID (Grafana has a large library), or use the dashboards bundled with the kube-prometheus-stack.
3. Reusable, parameterized dashboards: variables (e.g., namespace, workload, Pod dropdowns) let one dashboard be filtered/scoped dynamically instead of duplicating dashboards per target, and queries reference the variables.
4. Grafana can display metrics, logs, and traces from their respective backends in one UI and link between them — e.g., from a latency/error spike on a metrics panel you can jump to the corresponding logs (Loki) and distributed traces (Tempo/Jaeger) for the same time window/service. During an incident this drastically shortens diagnosis: you see that something is wrong (metrics), why via detailed errors (logs), and where in the request path (traces), all correlated, instead of manually stitching together separate tools.

---

Alertmanager and alerting rules

Theory

Dashboards are passive; alerting makes monitoring proactive. In the Prometheus ecosystem, alerting rules are PromQL expressions evaluated continuously by Prometheus; when an expression is true for a specified duration (for:), Prometheus fires an alert. These alerts are sent to Alertmanager, a separate component that handles routing and delivery.

Alertmanager's responsibilities are what make alerting usable rather than noisy:

• Grouping: combine related alerts into one notification (e.g., all alerts from one outage).
• Routing: send different alerts to different receivers (Slack, PagerDuty, email) based on labels/severity.
• Silencing: temporarily mute alerts (e.g., during maintenance).
• Inhibition: suppress lower-priority alerts when a higher-priority related alert is firing (e.g., don't page about individual Pods when the whole node is down).
• Deduplication: avoid repeated notifications for the same alert.

With the Prometheus Operator, alerting rules are declared via PrometheusRule CRDs. Good alerting targets symptoms users feel (high error rate, latency) over noisy causes, following SRE practice.

Example

# A PrometheusRule (Operator CRD) defining an alert:
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata: { name: web-alerts, labels: { release: kube-prometheus-stack } }
spec:
  groups:
  - name: web.rules
    rules:
    - alert: HighErrorRate
      expr: sum(rate(http_requests_total{status=~"5..",app="web"}[5m]))
            / sum(rate(http_requests_total{app="web"}[5m])) > 0.05
      for: 10m                               # must hold 10 min before firing
      labels: { severity: page }
      annotations: { summary: "web 5xx error rate > 5% for 10m" }

Exercises

1. (Beginner) What component evaluates alerting rules, and what component handles routing/delivery?
2. (Beginner) What does the for: field in an alerting rule control?
3. (Intermediate) Name three things Alertmanager does to reduce alert noise.
4. (Interview) Why is "inhibition" valuable, and why do SRE practices favor alerting on symptoms over causes? (Hint: suppress redundant downstream alerts; reduce noise/actionability.)

Answers

1. Prometheus evaluates the alerting rules and fires alerts; Alertmanager handles grouping, routing, and delivery to receivers.
2. How long the alert expression must be continuously true before the alert actually fires — preventing flapping/transient spikes from triggering notifications.
3. Any three: grouping (combine related alerts), deduplication, silencing (mute during maintenance), inhibition (suppress lower-priority alerts when a higher-priority related one fires), and routing (send to the right receiver to avoid spamming everyone).
4. Inhibition suppresses redundant downstream alerts when a root-cause alert is already firing — e.g., if a node is down, you don't also want dozens of pages for each Pod on it; you get the one meaningful alert. Symptom-based alerting (high user-facing error rate, latency, unavailability) is favored because symptoms are what actually impact users and are reliably actionable, whereas alerting on every internal cause produces noise and false alarms (many causes don't affect users). Alerting on symptoms keeps pages few, meaningful, and tied to real impact, while metrics/dashboards/traces help diagnose the cause after a symptom alert fires.

---

10.3 Tracing

When a single request flows through many microservices, logs and metrics alone can't show you where time went or which hop failed. Distributed tracing reconstructs the request's journey. This subchapter covers it.

---

Distributed tracing concepts

Theory

In a microservices architecture, one user request may traverse a dozen services. If it's slow or fails, which service is responsible? Distributed tracing answers this by following a single request across all the services it touches, producing a trace.

Core concepts:

• Trace: the end-to-end journey of one request, identified by a trace ID.
• Span: a single unit of work within the trace (e.g., one service handling the request, or one DB call). Each span has a start/end time, a name, and attributes.
• Parent/child relationships: spans nest to form a tree, showing causality and how time is spent across services.
• Context propagation: the trace ID and span context are passed between services (via HTTP headers like W3C traceparent) so spans from different services link into one trace.

A trace is typically visualized as a waterfall/Gantt of spans, instantly revealing which service or call consumed the most time or errored. Tracing is the pillar that provides causality and latency breakdown across service boundaries — something neither logs nor metrics give you.

Example

Trace ID: 4bf92...   (one request)
  ├─ span: gateway            [=================] 250ms
  │    ├─ span: auth-service     [===] 30ms
  │    └─ span: order-service       [============] 200ms
  │         └─ span: db query           [========] 150ms   <- the bottleneck

Exercises

1. (Beginner) What is the difference between a trace and a span?
2. (Beginner) What identifier ties all spans of one request together?
3. (Intermediate) What is "context propagation" and why is it essential for distributed tracing?
4. (Interview) A user request is intermittently slow across a microservices system. Why is tracing better suited than logs or metrics to find the cause? (Hint: per-request causality and latency breakdown across services.)

Answers

1. A trace is the complete end-to-end record of one request across all services; a span is a single unit of work within that trace (one operation/service hop), with its own timing and attributes. A trace is composed of many nested spans.
2. The trace ID (shared by all spans belonging to the same request).
3. Context propagation is passing the trace/span context (e.g., trace ID via headers like W3C traceparent) from one service to the next as the request flows. It's essential because without it, each service's spans would be isolated; propagation links spans across service boundaries into a single coherent trace.
4. Tracing reconstructs the exact path and timing of an individual request across every service and call, so you can see precisely which hop or dependency added the latency (or failed) for the slow requests. Metrics show aggregate trends (latency is up) but not which request/path; logs show per-service events but aren't inherently correlated end-to-end across services. Tracing provides the per-request causality and latency breakdown across boundaries needed to pinpoint an intermittent, cross-service slowdown.

---

OpenTelemetry in Kubernetes

Theory

Historically, each tracing/metrics vendor had its own SDKs and agents, causing lock-in and fragmentation. OpenTelemetry (OTel) — a CNCF project — is the vendor-neutral standard for generating, collecting, and exporting telemetry (traces, metrics, and logs). It provides consistent SDKs/APIs for instrumenting applications and the OpenTelemetry Collector for receiving, processing, and exporting telemetry to any backend.

In Kubernetes, OTel is typically deployed via the OpenTelemetry Collector (as a DaemonSet for node-local collection and/or a Deployment for central processing), often managed by the OpenTelemetry Operator (which can also auto-inject instrumentation into Pods). The Collector receives telemetry (e.g., OTLP), processes it (batching, filtering, enrichment with Kubernetes metadata), and exports it to backends like Jaeger/Tempo (traces), Prometheus (metrics), or Loki (logs). The big win: instrument once with OTel and switch backends freely — no vendor lock-in.

Example

App (OTel SDK) --OTLP--> OpenTelemetry Collector --> exporters:
                                  |--> Tempo / Jaeger (traces)
                                  |--> Prometheus      (metrics)
                                  |--> Loki            (logs)

# OpenTelemetry Collector pipeline (conceptual):
receivers:  [ otlp ]                 # accept telemetry from apps (gRPC/HTTP)
processors: [ batch, k8sattributes ] # batch + enrich with pod/namespace labels
exporters:  [ otlp/tempo, prometheus ]

Exercises

1. (Beginner) What problem does OpenTelemetry solve regarding vendors?
2. (Beginner) What are the two common ways the OpenTelemetry Collector is deployed in Kubernetes?
3. (Intermediate) What does the OpenTelemetry Collector do between applications and backends?
4. (Interview) Why is "instrument once with OTel, export anywhere" valuable, and how does the Collector enable backend flexibility? (Hint: avoid SDK lock-in; Collector decouples instrumentation from backend via exporters.)

Answers

1. Vendor lock-in and fragmentation — instead of vendor-specific SDKs/agents, OTel provides a single vendor-neutral standard and tooling for generating and exporting telemetry to any backend.
2. As a DaemonSet (node-local collection) and/or as a Deployment (centralized processing/gateway), often managed by the OpenTelemetry Operator.
3. It receives telemetry (e.g., via OTLP), processes it — batching, filtering, sampling, and enriching with Kubernetes metadata (k8sattributes) — and exports it to one or more backends through configurable exporters.
4. Applications are instrumented once against the OTel API/SDK and emit standard telemetry; the Collector (via swappable exporters) decides where it goes. To change or add a backend (e.g., switch from Jaeger to Tempo, or also send metrics to Prometheus), you reconfigure the Collector's exporters — no application code changes. This decouples instrumentation from backends, eliminating per-vendor SDK rewrites and lock-in while letting you fan telemetry out to multiple systems.

---

Jaeger and Tempo integration

Theory

Once spans are generated and collected, you need a tracing backend to store, index, and visualize traces. The two leading open-source choices:

• Jaeger (CNCF): a mature distributed tracing system with its own collectors, storage (Elasticsearch/Cassandra), and a feature-rich UI for searching traces by service/operation/tags and viewing dependency graphs. It can index trace attributes for rich querying.
• Tempo (Grafana): a tracing backend optimized for cost and simplicity — it stores traces in cheap object storage and, by design, does not index trace attributes (you look traces up by trace ID, typically discovered via correlated logs/metrics in Grafana). This makes it very cheap to run at scale and integrates tightly with the Grafana + Loki + Prometheus stack for cross-pillar correlation.

The trade-off mirrors Loki vs. Elasticsearch: Jaeger offers rich trace search at higher storage/operational cost; Tempo offers cheap, scalable trace storage with lookup driven by correlation rather than full indexing. Both are commonly fed by the OpenTelemetry Collector.

Example

Backend	Storage	Trace search	Best with
Jaeger	Elasticsearch/Cassandra	Rich (by service/op/tags)	Standalone, detailed trace queries
Tempo	Object storage (cheap)	By trace ID (correlate via logs/metrics)	Grafana + Loki + Prometheus stack

OTel Collector --(OTLP)--> Tempo  --(Grafana)--> view trace by ID
                                  (jump from a Loki log line's traceID to its trace)

Exercises

1. (Beginner) What is the role of a tracing backend like Jaeger or Tempo?
2. (Beginner) How does Tempo keep costs low compared to Jaeger?
3. (Intermediate) If Tempo doesn't index trace attributes, how do you typically find the trace you want?
4. (Interview) Compare Jaeger and Tempo and explain when each is the better fit. (Hint: rich search vs. cheap scalable storage + Grafana correlation.)

Answers

1. To store, index/organize, and visualize traces (spans) so engineers can search and inspect request flows and latency.
2. Tempo stores traces in inexpensive object storage and avoids indexing trace attributes (only trace-ID lookup), eliminating the costly full indexing and heavy storage backends (Elasticsearch/Cassandra) that Jaeger typically uses.
3. By trace ID — usually discovered through correlation: you find a relevant log line (Loki) or exemplar metric in Grafana that carries the trace ID, then jump to that trace in Tempo. (Tempo also offers TraceQL for some search in newer versions, but the core model is lookup/correlation by ID.)
4. Jaeger is the better fit when you need rich, standalone trace search (by service, operation, tags) and dependency analysis, and can afford its heavier storage/operations. Tempo is better when you want low-cost, highly scalable trace storage and you operate within the Grafana ecosystem (Loki + Prometheus), relying on cross-pillar correlation (jump from logs/metrics to a trace by ID) rather than full attribute indexing. Choose Jaeger for query richness, Tempo for cost/scale and unified Grafana correlation.

---

Instrumenting applications for tracing

Theory

Traces don't appear by magic — applications must be instrumented to create spans and propagate context. There are two approaches:

• Automatic instrumentation: language-specific agents/libraries (OpenTelemetry auto-instrumentation, or the OTel Operator's auto-injection) hook into common frameworks (HTTP servers/clients, gRPC, DB drivers) to generate spans and propagate context without code changes. Quick to adopt; captures framework-level spans.
• Manual instrumentation: developers use the OTel SDK to create custom spans around important business logic, add attributes (user ID, order value), and record events/errors. More effort, but captures domain-specific detail auto-instrumentation can't.

The essential requirement either way is context propagation: incoming trace context must be extracted from request headers and injected into outbound calls, so spans link across services. Best practice is to combine both — auto-instrumentation for broad coverage of framework calls, plus targeted manual spans for critical custom operations. In Kubernetes, the OTel Operator can auto-inject instrumentation by annotating Pods.

Example

# Manual instrumentation with the OpenTelemetry SDK (Python):
from opentelemetry import trace
tracer = trace.get_tracer(__name__)

def process_order(order):
    with tracer.start_as_current_span("process_order") as span:   # custom span
        span.set_attribute("order.id", order.id)                  # domain attribute
        span.set_attribute("order.value", order.total)
        charge_payment(order)        # downstream call inherits context automatically

# OTel Operator auto-injection: annotate the Pod, no code change
metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-python: "true"

Exercises

1. (Beginner) What are the two approaches to instrumenting an application for tracing?
2. (Beginner) What must happen with trace context for spans to link across services?
3. (Intermediate) What can manual instrumentation capture that automatic instrumentation typically cannot?
4. (Interview) Why is combining automatic and manual instrumentation considered best practice? (Hint: broad framework coverage plus business-specific spans/attributes.)

Answers

1. Automatic instrumentation (agents/libraries that instrument frameworks without code changes) and manual instrumentation (developers add spans/attributes via the SDK).
2. Trace context must be propagated — extracted from incoming request headers and injected into outbound requests — so child spans in downstream services attach to the same trace.
3. Domain/business-specific detail: custom spans around particular logic, meaningful attributes (e.g., order ID, customer tier, cache hit/miss), and explicit error/event recording — context that generic framework auto-instrumentation doesn't know about.
4. Automatic instrumentation gives broad, low-effort coverage of common framework operations (HTTP, gRPC, DB calls) and handles context propagation, so you get end-to-end traces quickly. Manual instrumentation adds the business-critical spans and attributes that make traces actionable for your domain (where the time/errors matter to the application's logic). Combining them yields both comprehensive coverage and meaningful, domain-rich detail — more complete and useful traces than either alone.

---

10.4 Health Probes

Kubernetes needs to know whether a container is alive, ready for traffic, and finished starting. Probes provide these signals and drive self-healing and traffic routing. This subchapter covers them.

---

Liveness probes

Theory

A liveness probe answers: "is this container still healthy, or is it stuck/deadlocked?" The kubelet periodically runs the probe; if it fails repeatedly (beyond failureThreshold), the kubelet restarts the container. This provides automatic recovery from states where the process is running but no longer functioning — a deadlock, an infinite loop, an exhausted thread pool — situations a simple process-alive check wouldn't catch.

The critical caution: a misconfigured liveness probe is dangerous. If it's too aggressive (short timeout/threshold) or checks something that's slow under load, it can restart healthy-but-busy containers, turning a load spike into a restart storm and making outages worse. Liveness probes should check only that the process itself is fundamentally healthy — not its dependencies (don't fail liveness because a database is down; that just restarts a container that can't fix the DB). Tunable fields: initialDelaySeconds, periodSeconds, timeoutSeconds, failureThreshold.

Example

spec:
  containers:
  - name: app
    image: myapp:1.0
    livenessProbe:
      httpGet: { path: /healthz, port: 8080 }   # "am I fundamentally alive?"
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 1
      failureThreshold: 3        # 3 consecutive failures -> restart container

Exercises

1. (Beginner) What action does the kubelet take when a liveness probe fails repeatedly?
2. (Beginner) What kind of failure does a liveness probe catch that a process-alive check does not?
3. (Intermediate) Why should a liveness probe avoid checking external dependencies like a database?
4. (Interview) How can a misconfigured liveness probe make an outage worse, and what guidelines prevent this? (Hint: restart storms under load; check process health only, sane thresholds.)

Answers

1. It restarts the container.
2. A process that is running but stuck/non-functional — e.g., deadlocked, in an infinite loop, or otherwise unresponsive — which still appears "alive" as a process but cannot do work. The liveness probe detects functional failure and triggers a restart.
3. Because failing liveness restarts the container, but restarting won't fix an external dependency (e.g., a down database). It would needlessly restart an otherwise-healthy container, and if many Pods do this simultaneously it amplifies the problem. Dependency health belongs in readiness (stop sending traffic) not liveness (restart).
4. If the liveness probe is too sensitive (short timeout/period/threshold) or checks something that slows under load, then during a traffic spike healthy-but-busy containers fail the probe and get restarted, dropping capacity exactly when it's needed and causing cascading restart storms that deepen the outage. Guidelines: probe only fundamental process health (a lightweight endpoint), never external dependencies; set generous timeouts and failure thresholds; use initialDelaySeconds/startup probes to avoid premature failures; and keep the check cheap and fast.

---

Readiness probes

Theory

A readiness probe answers a different question: "is this container ready to receive traffic right now?" If it fails, the kubelet does not restart the container — instead, the Pod is removed from the Service's endpoints, so no traffic is routed to it until it passes again. This is the mechanism that prevents requests from hitting a Pod that's still warming up, temporarily overloaded, or waiting on a dependency.

This is the crucial distinction from liveness: liveness failure = restart; readiness failure = stop sending traffic (no restart). Readiness is where checking dependencies is appropriate — if a Pod can't serve correctly because a needed backend is unavailable, it should report not-ready so it's pulled from rotation, then automatically returns when it recovers. Readiness probes also make rolling updates safe: new Pods only receive traffic once ready, ensuring zero-downtime deployments. Same tunables as liveness.

Example

spec:
  containers:
  - name: app
    image: myapp:1.0
    readinessProbe:
      httpGet: { path: /ready, port: 8080 }   # "can I serve traffic now?"
      periodSeconds: 5
      failureThreshold: 3        # 3 failures -> removed from Service endpoints

Exercises

1. (Beginner) What happens to a Pod when its readiness probe fails?
2. (Beginner) Does a readiness probe restart the container? Contrast with liveness.
3. (Intermediate) Why is it appropriate for a readiness probe (but not liveness) to check dependencies?
4. (Interview) How do readiness probes enable zero-downtime rolling updates? (Hint: new Pods get traffic only when ready; old Pods stay until then.)

Answers

1. It is removed from the Service's endpoints (EndpointSlices), so no traffic is routed to it until it becomes ready again; the container keeps running.
2. No — readiness failure does not restart the container (it only gates traffic). Liveness failure does restart the container. Liveness = restart; readiness = remove from traffic.
3. Because the response to "I can't serve correctly because a dependency is down" should be to stop receiving traffic (so requests go to healthy Pods or fail fast), not to restart the container — restarting wouldn't fix the dependency. Readiness gates traffic and automatically restores it when the dependency recovers, which is the correct behavior; liveness restarting on dependency failure would be harmful.
4. During a rolling update, new Pods are started but only added to the Service's endpoints once their readiness probe passes, while old Pods continue serving until then (and are removed only as new ones become ready, subject to surge/unavailable settings). This guarantees traffic is always routed to Pods that are actually ready to handle it, so users experience no errors or downtime as the new version rolls out.

---

Startup probes

Theory

Some applications take a long and variable time to start — large JVM apps, services loading big caches, legacy apps. This creates a dilemma for liveness probes: set initialDelaySeconds long enough for the slow start, and you delay detecting real deadlocks for fast-starting cases; set it short, and the liveness probe kills the container before it finishes starting. The startup probe resolves this.

A startup probe runs first; while it is failing, the kubelet disables the liveness and readiness probes so they can't act prematurely. Only once the startup probe succeeds do liveness and readiness take over. You give the startup probe a generous total budget (failureThreshold × periodSeconds) to allow even worst-case startup. This lets you keep liveness probes aggressive (fast deadlock detection) for the running state while safely accommodating slow startup — the best of both worlds.

Example

spec:
  containers:
  - name: legacy-app
    image: legacy:1.0
    startupProbe:
      httpGet: { path: /healthz, port: 8080 }
      periodSeconds: 10
      failureThreshold: 30        # allow up to 30*10s = 5 minutes to start
    livenessProbe:                # only active AFTER startup succeeds; can be tight
      httpGet: { path: /healthz, port: 8080 }
      periodSeconds: 5
      failureThreshold: 3

Exercises

1. (Beginner) What problem does a startup probe solve?
2. (Beginner) What happens to the liveness and readiness probes while the startup probe is still failing?
3. (Intermediate) How do you give a startup probe a 5-minute total grace period?
4. (Interview) Why does using a startup probe let you keep liveness probes aggressive, and why is that desirable? (Hint: separate slow-start tolerance from steady-state deadlock detection.)

Answers

1. It handles applications with long/variable startup times, preventing the liveness probe from killing a container before it has finished starting (without having to weaken liveness for the running state).
2. They are disabled (held off) — the kubelet does not run liveness/readiness checks until the startup probe succeeds, so they can't prematurely restart or mark the Pod not-ready during startup.
3. Set periodSeconds and failureThreshold so their product is 5 minutes — e.g., periodSeconds: 10 and failureThreshold: 30 (10s × 30 = 300s = 5 minutes).
4. The startup probe absorbs all the slow-start tolerance, so the liveness probe doesn't need a long initialDelaySeconds or lenient thresholds to accommodate startup. Once startup completes, liveness can use short periods/timeouts and low failure thresholds to detect deadlocks quickly. This is desirable because fast deadlock detection improves recovery time in steady state, while slow startup is handled separately — you no longer have to trade off one against the other.

---

Probe types: HTTP, TCP, exec, gRPC

Theory

All three probes (liveness, readiness, startup) can use any of four mechanisms to perform their check:

• httpGet: the kubelet makes an HTTP GET to a path/port; a response code 200–399 is success. The most common for web services (use a dedicated lightweight health endpoint).
• tcpSocket: success if the kubelet can open a TCP connection to the port. Useful for non-HTTP services (databases, message brokers) where "port accepts connections" is a reasonable health signal.
• exec: the kubelet runs a command inside the container; exit code 0 is success. Flexible (any custom check) but heavier (spawns a process each time) and requires the binary to exist in the image.
• grpc: the kubelet uses the standard gRPC health checking protocol (stable since v1.27); ideal for gRPC services, replacing the old workaround of shipping a grpc-health-probe binary with an exec probe.

Choose the lightest mechanism that meaningfully reflects health: prefer httpGet/grpc for those protocols, tcpSocket for simple connectivity, and exec only when nothing else fits.

Example

livenessProbe:                      # HTTP
  httpGet: { path: /healthz, port: 8080 }
---

readinessProbe:                     # TCP connectivity
  tcpSocket: { port: 5432 }
---

livenessProbe:                      # exec a command (exit 0 = healthy)
  exec: { command: ["sh", "-c", "pg_isready -U postgres"] }
---

livenessProbe:                      # native gRPC health check
  grpc: { port: 50051 }

Exercises

1. (Beginner) What HTTP status code range counts as success for an httpGet probe?
2. (Beginner) What defines success for an exec probe?
3. (Intermediate) When would you choose a tcpSocket probe over an httpGet probe?
4. (Interview) Before native gRPC probes, how did teams health-check gRPC services, and why is the built-in grpc probe an improvement? (Hint: grpc-health-probe binary via exec; simpler, lighter, standard.)

Answers

1. 200–399 (a 2xx/3xx response is considered success).
2. The executed command returning exit code 0 (non-zero = failure).
3. For non-HTTP services where you only need to verify the port is open/accepting connections — e.g., a database (PostgreSQL on 5432), a message broker, or any TCP service that doesn't expose an HTTP health endpoint. A successful TCP connection is the health signal.
4. Teams used an exec probe running a separate grpc-health-probe binary that had to be baked into the image, which added image bloat, spawned a process per check, and required maintaining the extra tool. The native grpc probe lets the kubelet speak the standard gRPC health-checking protocol directly — no extra binary, lighter (no process spawn), simpler to configure, and standardized — making gRPC health checks first-class.

---

10.5 Events and Auditing

Beyond logs/metrics/traces, Kubernetes emits events about what's happening to objects, and can audit every API request. This subchapter covers these Kubernetes-native signals.

---

Kubernetes events and their lifecycle

Theory

Events are Kubernetes objects that record what is happening to other objects — they're the cluster's narration of its own actions. When the scheduler can't place a Pod, when an image fails to pull, when a probe fails, when a Pod is killed (OOM), when a Deployment scales — an Event is created describing it (with a reason, message, involved object, type, and count). Events are the first place to look when debugging "why is my Pod not working?" via kubectl describe (which shows an object's recent events) or kubectl get events.

The critical caveat: events are ephemeral. By default they are stored in etcd with a TTL of ~1 hour and then deleted, to avoid bloating the datastore. This means events are great for immediate debugging but disappear quickly — you can't investigate yesterday's incident from events alone. For retention, you export events to a logging/monitoring system (e.g., via an event exporter to Loki/Elasticsearch). Events are also rate-limited/aggregated (repeated events increment a count rather than creating many objects).

Example

kubectl describe pod web-abc          # shows the Pod's recent Events at the bottom
kubectl get events -n prod --sort-by=.lastTimestamp   # cluster events, newest last
kubectl get events --field-selector involvedObject.name=web-abc,type=Warning
# LAST SEEN  TYPE     REASON      OBJECT        MESSAGE
# 2m         Warning  BackOff     pod/web-abc   Back-off restarting failed container
# 5m         Warning  Failed      pod/web-abc   Failed to pull image "myapp:typo"

Exercises

1. (Beginner) What do Kubernetes events record?
2. (Beginner) Roughly how long are events retained by default, and why the limit?
3. (Intermediate) Where do you most commonly view an object's recent events when debugging?
4. (Interview) Given events' short TTL, how do you make them available for investigating past incidents, and why isn't this built in? (Hint: export to log/monitoring backend; etcd bloat avoidance.)

Answers

1. What is happening to cluster objects — significant occurrences like scheduling decisions, image pull results, probe failures, restarts/OOM kills, scaling actions — with a reason, message, type (Normal/Warning), and the involved object.
2. About one hour (default TTL), after which they're deleted. The limit exists to prevent events from accumulating and bloating etcd (and overloading the API/watchers).
3. Via kubectl describe <kind> <name>, which lists the object's recent Events (also kubectl get events, optionally filtered with field selectors).
4. Export events to a durable backend — e.g., run an event exporter/watcher that streams events to a logging/monitoring system (Loki, Elasticsearch, a SIEM) where they're retained and searchable. It isn't built in because keeping all events indefinitely in etcd would bloat the cluster datastore and degrade performance; events are designed to be short-lived, real-time signals, with long-term retention delegated to external systems suited for storage and querying.

---

Audit policy configuration

Theory

For security and compliance, you need a record of who did what in the cluster — every request to the API server. Auditing provides this: when enabled, the API server logs requests according to an audit policy that you define. Since logging everything in full detail would be overwhelming and storage-heavy, the policy lets you control the level of detail recorded per request type.

The audit levels (per matched rule) are:

• None: don't log these requests.
• Metadata: log request metadata (who, when, verb, resource) but not request/response bodies.
• Request: log metadata + the request body.
• RequestResponse: log metadata + request and response bodies (most verbose).

A policy is a list of rules matched in order by users, groups, resources, verbs, and namespaces — letting you, for example, log Secret access at Metadata, mutating writes at RequestResponse, and ignore noisy read-only/system traffic. Each request also passes through stages (RequestReceived, ResponseComplete, etc.) you can choose to record. Tuning the policy balances forensic value against volume.

Example

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: None                         # ignore noisy health checks
  nonResourceURLs: ["/healthz*", "/readyz*"]
- level: Metadata                     # secrets: log access, but not contents
  resources: [ { group: "", resources: ["secrets"] } ]
- level: RequestResponse              # log full detail for writes to deployments
  verbs: ["create","update","patch","delete"]
  resources: [ { group: "apps", resources: ["deployments"] } ]
- level: Metadata                     # default: metadata for everything else

Exercises

1. (Beginner) What does Kubernetes auditing record, and which component produces it?
2. (Beginner) Name the four audit levels from least to most detailed.
3. (Intermediate) Why would you log Secret access at Metadata rather than RequestResponse?
4. (Interview) How does an audit policy balance forensic value against log volume, and give an example rule ordering. (Hint: per-rule levels; ignore noise, detail sensitive writes.)

Answers

1. It records API requests to the kube-apiserver — who made the request, when, what verb/resource, from where, and the outcome (and optionally bodies) — providing a "who did what" trail. The kube-apiserver produces the audit log according to the configured policy.
2. None, Metadata, Request, RequestResponse.
3. Logging Secret access at RequestResponse would write the secret values themselves into the audit log — a serious exposure. Metadata records that someone accessed a Secret (who/when/which) without capturing its contents, giving the security signal you want without leaking the sensitive data.
4. Each rule sets a level for matched requests, so you record high detail only where it matters and little/nothing where it doesn't. A typical ordering: first None for noisy/irrelevant traffic (health/readiness endpoints, certain system reads); then Metadata for sensitive resources like Secrets (access without contents); then RequestResponse for mutating writes to important resources (full forensic detail on changes); and a catch-all Metadata default at the end. This captures actionable, security-relevant detail while keeping overall log volume manageable.

---

Audit log backends

Theory

Once the API server generates audit events per the policy, they must go somewhere. Kubernetes supports two audit backends:

• Log backend: writes audit events to a file on the control-plane node (JSON lines), with rotation options (max age/size/backups). Simple and self-contained; a logging agent then ships the file to central storage. This is the common approach for self-managed clusters.
• Webhook backend: sends audit events to an external HTTP endpoint (a SIEM, a log aggregator, a security platform) in near real time. Good for centralized, tamper-resistant collection and immediate analysis, at the cost of a network dependency.

In managed clusters, the provider integrates auditing with its cloud logging (e.g., EKS → CloudWatch, GKE → Cloud Logging, AKS → Azure Monitor), so you consume audit logs there. Regardless of backend, the goal is to land audit events in a durable, secure, searchable store — ideally isolated so that even a cluster compromise can't erase the evidence. Audit logs are key inputs for incident response, compliance, and detecting suspicious API activity.

Example

# API server flags (self-managed):
--audit-policy-file=/etc/kubernetes/audit-policy.yaml
--audit-log-path=/var/log/kubernetes/audit.log     # log backend (file)
--audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100
# OR webhook backend:
--audit-webhook-config-file=/etc/kubernetes/audit-webhook.yaml

# Managed clusters surface audit logs in cloud logging, e.g. GKE:
gcloud logging read 'resource.type="k8s_cluster" protoPayload.methodName:"delete"'

Exercises

1. (Beginner) What are the two audit backends Kubernetes supports?
2. (Beginner) Where do managed Kubernetes services typically deliver audit logs?
3. (Intermediate) What is an advantage of the webhook backend over the log (file) backend?
4. (Interview) Why should audit logs be stored in a durable, isolated location, and how does that aid incident response? (Hint: tamper-resistance even if the cluster is compromised; forensic integrity.)

Answers

1. The log backend (writes to a file on the control-plane node) and the webhook backend (sends events to an external HTTP endpoint).
2. In the cloud provider's logging service (e.g., AWS CloudWatch for EKS, Google Cloud Logging for GKE, Azure Monitor for AKS).
3. The webhook backend ships audit events to an external system (SIEM/aggregator) in near real time, enabling centralized, immediate analysis/alerting and storage off the node (more tamper-resistant), rather than sitting in a local file that depends on a separate shipping step and is more exposed if the node is compromised.
4. If audit logs are durable and isolated from the cluster (e.g., in a separate account/SIEM with restricted, append-only access), an attacker who compromises the cluster cannot delete or alter them to cover their tracks — preserving forensic integrity. For incident response this means you retain a trustworthy record of every API action to reconstruct what happened, when, and by whom, supporting investigation, scoping the breach, and meeting compliance/legal requirements.

---

Tools: kubeaudit, kube-bench

Theory

Beyond runtime auditing, you should proactively assess your cluster's security posture against best practices. Two widely used open-source scanners:

• kube-bench (Aqua Security): checks a cluster against the CIS Kubernetes Benchmark — a comprehensive set of security configuration recommendations for the control plane, etcd, kubelet, and policies. It reports which checks pass/fail/warn with remediation guidance, helping you harden node and component configuration (file permissions, API server flags, etc.). Often run as a Job/DaemonSet on the nodes.
• kubeaudit (Shopify): audits your workload manifests / running workloads for security issues — containers running as root, missing securityContext settings, privileged Pods, missing resource limits, allowPrivilegeEscalation, etc. It can run against manifests (in CI) or a live cluster and suggests fixes.

Roughly: kube-bench assesses cluster/infrastructure configuration against CIS, while kubeaudit assesses workload security settings. Both complement the runtime detection (Falco) and policy enforcement (Kyverno/Gatekeeper) from Chapter 9, forming a layered, continuous security-assessment practice.

Example

# kube-bench: check the cluster against the CIS Benchmark
kube-bench run --targets master,node
# [PASS] 1.2.1 Ensure --anonymous-auth is set to false
# [FAIL] 1.2.6 Ensure --kubelet-certificate-authority is set ...  (remediation: ...)

# kubeaudit: audit workloads for insecure settings
kubeaudit all -f deployment.yaml
# ERROR  RunAsNonRoot: runAsNonRoot is not set, root user allowed  [container: app]
# ERROR  Capabilities: capability not dropped: ALL                  [container: app]

Exercises

1. (Beginner) What standard does kube-bench check a cluster against?
2. (Beginner) What kind of issues does kubeaudit look for?
3. (Intermediate) Summarize the difference in focus between kube-bench and kubeaudit.
4. (Interview) How do kube-bench and kubeaudit fit alongside Falco and Kyverno/Gatekeeper to form a layered security practice? (Hint: posture assessment vs. runtime detection vs. admission enforcement.)

Answers

1. The CIS Kubernetes Benchmark.
2. Insecure workload/manifest security settings — e.g., containers running as root, missing/weak securityContext (runAsNonRoot, capabilities not dropped, allowPrivilegeEscalation), privileged Pods, missing resource limits.
3. kube-bench assesses cluster/infrastructure configuration (control plane, etcd, kubelet, node settings) against the CIS Benchmark; kubeaudit assesses workload security configurations (Pod/container securityContext and related settings). One targets how the cluster is set up; the other targets how workloads are configured.
4. They cover different layers and phases: kube-bench/kubeaudit provide proactive posture assessment (is the cluster/workload configured securely against best practices?), useful in CI and periodic scans. Kyverno/Gatekeeper enforce policy at admission (prevent insecure resources from being created). Falco provides runtime detection (catch malicious behavior after deployment). Together: assess and remediate configuration before/continuously (bench/audit), block non-compliant deployments (policy engines), and detect attacks in progress (Falco) — defense in depth across the build/deploy/run lifecycle.

---