17. Advanced Topics

This final chapter surveys the frontier of the Kubernetes ecosystem: kernel-level programmability with eBPF, running specialized hardware like GPUs, serverless and event-driven paradigms, AI/ML platforms, and stretching Kubernetes to the edge. These topics push beyond running ordinary services, showing how Kubernetes has become a universal control plane for increasingly diverse workloads and environments.

---

17.1 eBPF in Kubernetes

eBPF is reshaping Kubernetes networking, security, and observability. This subchapter covers the technology and its applications.

---

eBPF fundamentals

Theory

eBPF (extended Berkeley Packet Filter) lets you run small, sandboxed programs inside the Linux kernel in response to events — network packets, syscalls, function entry/exit, tracepoints — without modifying kernel source or loading kernel modules. The kernel verifies each program before loading it (proving it can't crash or hang the kernel — bounded loops, safe memory access), then JIT-compiles it to native code, so it runs safely and efficiently in kernel space.

Why this is transformative for Kubernetes: traditionally, adding kernel-level networking/security/observability meant kernel modules (risky, hard to maintain) or userspace tools (slow, requiring context switches and data copies). eBPF gives the safety of userspace with the performance and vantage point of the kernel — programmable, dynamic behavior at the exact place packets and syscalls are handled. Programs communicate with userspace via eBPF maps (efficient key-value structures). This foundation underpins Cilium (networking/mesh, Chapter 13), Falco (runtime security, Chapter 9), and a wave of observability tools — making eBPF one of the most important recent developments in the cloud-native stack.

Example

eBPF program lifecycle:
  write program -> kernel VERIFIER checks safety -> JIT to native -> attach to a hook
  hooks: network (XDP/tc), syscalls, kprobes/tracepoints, cgroups, ...
  eBPF MAPS <-> userspace (share data efficiently)

  Result: kernel-speed, safe, dynamic programmability (no module, no kernel rebuild)

Exercises

1. (Beginner) What does eBPF let you do, and where do the programs run?
2. (Beginner) What does the kernel do before loading an eBPF program, and why?
3. (Intermediate) How do eBPF programs share data with userspace?
4. (Interview) Why does eBPF give "the safety of userspace with the performance of the kernel," and why does that matter for cloud-native tooling? (Hint: verified sandboxed in-kernel execution vs. modules/userspace; used by Cilium/Falco.)

Answers

1. Run small sandboxed programs in response to kernel events (packets, syscalls, function/tracepoints) — the programs run inside the Linux kernel, without modifying kernel source or loading kernel modules.
2. It runs the program through a verifier that proves it's safe (bounded/terminating, no invalid memory access) before loading, so a buggy or malicious eBPF program can't crash or hang the kernel; verified programs are then JIT-compiled to native code.
3. Via eBPF maps — efficient in-kernel key-value data structures that both the eBPF program and userspace can read/write, enabling data exchange and configuration.
4. eBPF programs are verified and sandboxed like userspace code (they can't crash the kernel), yet they execute in the kernel at the exact hook where events occur, JIT-compiled to native speed — avoiding the context switches, data copies, and latency of userspace processing and the risk/maintenance burden of kernel modules. For cloud-native tooling this means high-performance, dynamic, safe programmability of networking, security, and observability right in the data path — which is why projects like Cilium (CNI/mesh), Falco (runtime security), and many observability tools build on it to do things (efficient service load balancing, syscall-level detection, deep flow visibility) that were previously slow or unsafe.

---

eBPF-based observability

Theory

eBPF is a game-changer for observability because it can instrument the kernel to see everything — network flows, syscalls, function calls, latency, file/process activity — without modifying or instrumenting applications. This "zero-instrumentation" visibility is its key advantage: you attach eBPF programs to kernel hooks and get rich, low-overhead telemetry across all workloads uniformly, regardless of language or whether the app was built with observability in mind.

Applications in Kubernetes:

• Network observability: tools like Hubble (Cilium) and Pixie capture and expose service-to-service flows, HTTP/gRPC/DNS requests, latencies, and drops directly from the kernel.
• Auto-instrumented metrics/tracing: eBPF can generate golden-signal metrics (RPS, latency, errors) and even traces for services without code changes (e.g., Pixie, Grafana Beyla, Cilium/Tetragon), lowering the barrier to observability.
• Continuous profiling: eBPF-based profilers (Parca, Pyroscope) sample stack traces cluster-wide with negligible overhead to find CPU/memory hotspots.

The overarching benefit: deep, low-overhead, application-transparent observability. Where traditional observability requires instrumenting each app (SDKs, agents, code changes), eBPF observes from the kernel's universal vantage point — a major reason eBPF is central to the modern observability story.

Example

eBPF observability (no app changes):
  kernel hooks (sockets, syscalls, uprobes) -> eBPF programs -> maps -> userspace

  Hubble/Pixie:   L3-L7 flows, HTTP/gRPC/DNS, latency, drops  (auto, cluster-wide)
  Beyla/Pixie:    auto RED metrics + traces without instrumenting the app
  Parca/Pyroscope: continuous CPU/mem profiling, low overhead

Exercises

1. (Beginner) What is the key advantage of eBPF-based observability regarding applications?
2. (Beginner) Name an eBPF-based network observability tool.
3. (Intermediate) How can eBPF produce service metrics/traces without code changes?
4. (Interview) Why is kernel-level, zero-instrumentation observability powerful compared to traditional per-app instrumentation? (Hint: universal vantage point, language-agnostic, low overhead, no code changes.)

Answers

1. It provides deep visibility without modifying or instrumenting the applications ("zero-instrumentation") — you observe from the kernel regardless of the app's language or design.
2. Any of: Hubble (Cilium), Pixie, Cilium Tetragon (also Beyla for metrics/traces).
3. eBPF attaches to kernel hooks (sockets/syscalls/uprobes) to observe requests and responses in the data path, so tools can compute golden-signal (RED) metrics — request rate, errors, latency — and reconstruct traces from the observed traffic automatically, without the application emitting them or being recompiled.
4. Traditional observability requires adding SDKs/agents/code to each application (per language, per app, needing changes and maintenance), and misses anything not instrumented. eBPF observes from the kernel's universal vantage point where all network and syscall activity flows, so it captures rich telemetry across every workload uniformly, language-agnostically, with no code changes, and at low overhead (in-kernel, JIT-compiled, efficient maps). This dramatically lowers the effort to get comprehensive visibility and covers workloads you couldn't or didn't instrument — which is why eBPF is central to modern observability.

---

eBPF-based networking and security

Theory

Beyond observability, eBPF powers high-performance networking and runtime security in Kubernetes — often replacing older mechanisms:

• Networking: eBPF implements pod networking, service load balancing (a kube-proxy replacement with O(1) map lookups instead of long iptables chains), and network policy enforcement directly in the kernel data path — the basis of Cilium (Chapter 6/13). This yields lower latency, better scalability, and identity-aware, L7-capable policy.
• Security (runtime enforcement + detection): eBPF can observe syscalls and kernel events to detect malicious behavior (Falco, Chapter 9) and, crucially, to enforce security in-kernel — e.g., Cilium Tetragon can observe and block process executions, file access, or network activity based on policy, at the kernel level, in real time. This enables enforcement that's both fast and hard to evade (it's below the application).

The theme is moving networking and security into the kernel data path for performance and a strong vantage point: policy and enforcement happen where traffic and syscalls actually occur, with the safety guarantees of the eBPF verifier. This is why eBPF-based tools are increasingly the foundation for cloud-native networking (Cilium replacing kube-proxy/iptables) and runtime security (kernel-level detection and enforcement) rather than userspace or module-based approaches.

Example

Networking (Cilium/eBPF):
  service LB + policy in kernel (O(1) maps) -> replaces kube-proxy/iptables
  identity-aware, L3-L7 network policy at kernel speed

Security (Tetragon/eBPF):
  observe syscalls/exec/file/net events -> detect AND enforce (block) in-kernel
  e.g., block an unexpected process exec or file write in real time

Exercises

1. (Beginner) What older mechanism can eBPF-based service load balancing replace?
2. (Beginner) Besides detecting threats, what can eBPF-based security tools like Tetragon also do?
3. (Intermediate) Why is in-kernel enforcement harder to evade than application-level controls?
4. (Interview) What is the common theme behind eBPF-based networking and security, and what advantages does it bring? (Hint: move into kernel data path — performance + vantage point + verifier safety.)

Answers

1. kube-proxy's iptables-based service routing (eBPF provides a kube-proxy replacement using efficient in-kernel maps).
2. Enforce security in-kernel — not just observe/detect, but actively block actions (e.g., prevent a process execution, file access, or network connection) based on policy, in real time.
3. Because enforcement happens in the kernel, below the application: a compromised or malicious process runs in userspace and can't bypass a control that intercepts its syscalls/kernel operations at the kernel level. It can't simply avoid or disable an in-kernel eBPF hook the way it might tamper with a userspace agent or app-level check, and the eBPF program sees the actual kernel-level activity.
4. The common theme is pushing networking and security into the kernel data path (where packets and syscalls actually occur), using eBPF's verified, sandboxed programs. Advantages: high performance (kernel-speed, O(1) lookups, no userspace hops), a superior vantage point (sees and can act on all traffic/syscalls, enabling identity/L7-aware policy and hard-to-evade enforcement), and safety (the verifier prevents kernel crashes) plus dynamism (no kernel rebuild/module). This makes eBPF the foundation for modern CNIs/meshes (Cilium replacing kube-proxy/iptables) and runtime security (kernel-level detection and enforcement).

---

bpftrace and BCC tools

Theory

To use eBPF directly for ad-hoc tracing and debugging (rather than via a platform like Cilium), two toolkits are standard:

• BCC (BPF Compiler Collection): a toolkit and library for building eBPF programs, with a large collection of ready-made performance/tracing tools (e.g., execsnoop — trace new process executions, opensnoop — trace file opens, tcpconnect, biolatency — block I/O latency, profile). Programs are typically written with a C (kernel) + Python/Lua (userspace) structure. BCC is powerful for building custom tools.
• bpftrace: a high-level tracing language (inspired by awk/DTrace) for writing one-liner and short eBPF tracing scripts quickly. It's the go-to for quick, exploratory kernel/application tracing — e.g., counting syscalls, histogramming latencies — without writing a full BCC program.

In Kubernetes, these are invaluable for deep, node-level performance debugging and understanding what a workload is actually doing at the kernel level (syscalls, I/O, latencies) — the low-level complement to the platform-level eBPF observability tools. You typically run them on a node (or via a privileged debug Pod) to diagnose tricky performance or behavior issues. Rule of thumb: bpftrace for quick one-liners/exploration, BCC for richer, reusable tools.

Example

# bpftrace one-liners (quick ad-hoc kernel tracing):
bpftrace -e 'tracepoint:syscalls:sys_enter_openat { @[comm] = count(); }'  # opens by process
bpftrace -e 'kprobe:vfs_read { @bytes = hist(arg2); }'                      # read-size histogram

# BCC ready-made tools:
execsnoop      # trace every new process exec (great for spotting unexpected execs)
biolatency     # histogram of block-device I/O latency
tcpconnect     # trace TCP connections as they happen

Exercises

1. (Beginner) What is bpftrace best suited for?
2. (Beginner) What does BCC provide beyond a single tool?
3. (Intermediate) When would you choose bpftrace over BCC, and vice versa?
4. (Interview) How do bpftrace/BCC complement platform-level eBPF observability (like Hubble) in a Kubernetes context? (Hint: ad-hoc low-level node/kernel debugging vs. continuous service-level visibility.)

Answers

1. Quick, ad-hoc, exploratory kernel/application tracing via short one-liners or small scripts in a high-level language (e.g., counting syscalls, histogramming latencies).
2. A toolkit/library for building eBPF programs plus a large collection of ready-made performance and tracing tools (execsnoop, opensnoop, biolatency, tcpconnect, profile, etc.) for deeper, reusable instrumentation.
3. Choose bpftrace for fast, exploratory investigations and one-liners where you want an answer quickly without writing a full program. Choose BCC when you need richer, more complex, or reusable tools (custom logic, packaged utilities), or want to build a maintained tool rather than an ad-hoc script.
4. Platform tools like Hubble provide continuous, service-level observability (flows, HTTP/gRPC, policy verdicts) across the cluster. bpftrace/BCC are for ad-hoc, low-level, node/kernel-depth debugging — drilling into syscalls, I/O latency, process execs, and kernel behavior on a specific node when you need to diagnose a tricky performance or behavior issue that service-level telemetry can't explain. They complement each other: use platform observability to spot where/what is wrong at the service level, then bpftrace/BCC to investigate the underlying kernel-level why on the affected node.

---

17.2 GPU and Specialized Hardware

Kubernetes can schedule and manage specialized hardware for compute-intensive workloads. This subchapter covers GPUs and accelerators.

---

GPU operator and device plugins

Theory

Kubernetes only natively understands CPU and memory. To use specialized hardware (GPUs, FPGAs, high-performance NICs), it relies on the Device Plugin framework: a device plugin (a DaemonSet from the hardware vendor) runs on each node, discovers the devices, advertises them to the kubelet as extended resources (e.g., nvidia.com/gpu), and handles allocating a device to a container. Pods then request these resources like any other, and the scheduler places them on nodes with available devices.

Setting up GPUs manually involves many moving parts (drivers, container runtime hooks, device plugin, monitoring). The NVIDIA GPU Operator automates all of it: it deploys and manages the NVIDIA driver, container toolkit (runtime hooks so containers can access GPUs), the device plugin, node labeling (via node feature discovery), and DCGM monitoring — as a single operator, so GPU nodes become ready-to-use without manual per-node driver/toolkit installation. This makes GPUs first-class schedulable resources. The pattern generalizes: device plugins (often bundled in vendor operators) are how any specialized hardware is exposed to and scheduled by Kubernetes.

Example

# A Pod requests a GPU as an extended resource (advertised by the device plugin):
spec:
  containers:
  - name: trainer
    image: pytorch/pytorch:latest
    resources:
      limits:
        nvidia.com/gpu: 1        # request 1 GPU; scheduler finds a node with one free

kubectl describe node gpu-node-1 | grep nvidia.com/gpu
#   nvidia.com/gpu: 4            # device plugin advertises 4 GPUs on this node

Exercises

1. (Beginner) What framework does Kubernetes use to support GPUs and other special hardware?
2. (Beginner) How does a Pod request a GPU?
3. (Intermediate) What does the NVIDIA GPU Operator automate?
4. (Interview) How does the device plugin framework make specialized hardware a first-class schedulable resource, and how does the pattern generalize beyond GPUs? (Hint: discover/advertise as extended resources; scheduler places by availability; FPGAs/NICs similarly.)

Answers

1. The Device Plugin framework (device plugins run per node to expose hardware to the kubelet).
2. By requesting the vendor's extended resource in its container resources.limits, e.g., nvidia.com/gpu: 1.
3. The full GPU enablement stack on nodes: installing/managing the NVIDIA driver, the container toolkit/runtime hooks, the device plugin, node feature discovery/labeling, and monitoring (DCGM) — as one operator, so GPU nodes are ready to schedule GPU workloads without manual per-node setup.
4. A device plugin (per-node DaemonSet) discovers the hardware and advertises it to the kubelet as an extended resource (e.g., nvidia.com/gpu) with a count; the scheduler then treats it like CPU/memory — placing Pods that request that resource onto nodes with available units and the plugin allocates a device to the container. This makes special hardware schedulable using the same request/limit model as normal resources. The pattern generalizes: the same framework exposes FPGAs, high-performance NICs/RDMA, TPUs, and other accelerators as their own extended resources (often via vendor operators bundling the driver + plugin), so Kubernetes can schedule any specialized hardware uniformly.

---

NVIDIA GPU scheduling

Theory

Once GPUs are advertised as resources, scheduling GPU workloads has particular characteristics and challenges:

• Whole-GPU allocation (default): by default a GPU is allocated exclusively to one container — GPUs aren't divisible like CPU by default, so requesting nvidia.com/gpu: 1 gives a container a whole GPU. This can be wasteful if a workload doesn't fully use the GPU.
• Placement: GPU nodes are expensive and typically tainted (Chapter 8) so only GPU workloads (with matching tolerations + node affinity/labels) land there, keeping the costly hardware for jobs that need it.
• Topology awareness: for multi-GPU jobs, performance depends on GPU interconnect (NVLink) and NUMA locality; schedulers/operators can be topology-aware to co-locate GPUs that communicate.
• Sharing options (to improve utilization): time-slicing (multiple Pods share a GPU by time-multiplexing — no memory isolation), MPS (Multi-Process Service), and MIG partitioning (next topic) let multiple workloads share a GPU when whole-GPU exclusivity wastes capacity.

The core tension is utilization vs. isolation: exclusive whole-GPU allocation is simple and isolated but often underutilizes expensive hardware, driving the sharing/partitioning techniques. Effective GPU scheduling combines tainting/affinity (right placement), topology awareness (performance), and sharing strategies (utilization/cost).

Example

# GPU workload: tolerate the GPU taint, target GPU nodes, request GPUs
spec:
  tolerations: [ { key: nvidia.com/gpu, operator: Exists, effect: NoSchedule } ]
  nodeSelector: { accelerator: nvidia }
  containers:
  - name: train
    image: ml:latest
    resources: { limits: { nvidia.com/gpu: 2 } }   # 2 whole GPUs (exclusive by default)

Exercises

1. (Beginner) By default, is a GPU shared among containers or allocated exclusively?
2. (Beginner) Why are GPU nodes typically tainted?
3. (Intermediate) Name two techniques for sharing a single GPU across workloads.
4. (Interview) What is the core tension in GPU scheduling, and how do topology awareness and sharing address the goals of performance and utilization? (Hint: exclusive allocation wastes costly hardware; time-slicing/MIG for utilization, NVLink/NUMA locality for performance.)

Answers

1. Exclusively — by default a whole GPU is allocated to a single container (GPUs aren't divided like CPU by default).
2. Because GPU nodes are expensive; tainting them keeps non-GPU workloads off so the costly GPUs are reserved for workloads that actually need (tolerate + request) them.
3. Any two: time-slicing (time-multiplexing a GPU among Pods, no memory isolation), NVIDIA MPS (Multi-Process Service), and MIG partitioning (hardware partitions of a GPU).
4. The core tension is utilization vs. isolation/cost: exclusive whole-GPU allocation gives clean isolation but frequently underuses very expensive hardware. Sharing techniques (time-slicing, MPS, MIG) raise utilization by letting multiple workloads use one GPU (trading off isolation strength — MIG gives hardware isolation, time-slicing gives none). Topology awareness addresses performance: multi-GPU jobs run faster when the scheduler places them on GPUs with fast interconnect (NVLink) and matching NUMA/PCIe locality, minimizing communication overhead. Together, tainting/affinity ensure the right workloads land on GPU nodes, topology-aware placement maximizes performance for multi-GPU jobs, and sharing/partitioning maximizes utilization (and thus cost-efficiency) of the scarce hardware.

---

FPGA and custom accelerators

Theory

GPUs are the most common accelerator, but Kubernetes supports other specialized hardware through the same device plugin framework — the abstraction generalizes to any device:

• FPGAs (Field-Programmable Gate Arrays): reconfigurable chips programmed for specific tasks (e.g., custom signal processing, inference, financial computation, compression). Vendors (Intel, Xilinx/AMD) provide device plugins so FPGAs are advertised and schedulable, sometimes with support for reprogramming the FPGA "bitstream" per workload.
• TPUs (Tensor Processing Units) and other AI accelerators (AWS Inferentia/Trainium, etc.): exposed as extended resources via their plugins/operators for ML workloads.
• SmartNICs / DPUs, RDMA/InfiniBand NICs: high-performance networking hardware also surfaced via device plugins for HPC/low-latency workloads.

The unifying idea: Kubernetes' device plugin model makes it a general control plane for heterogeneous, accelerated compute — any vendor can expose its hardware as a schedulable extended resource, and workloads request it uniformly (vendor.com/device: N). This is why Kubernetes has become the platform of choice for AI/ML and HPC: it schedules not just CPUs but the whole zoo of accelerators through one consistent mechanism, with vendor operators handling the drivers/toolkits/plugins.

Example

# Requesting an FPGA (advertised by a vendor device plugin) — same model as GPUs:
spec:
  containers:
  - name: accel
    image: fpga-workload:latest
    resources:
      limits:
        fpga.intel.com/arria10: 1     # vendor-specific extended resource
        # or xilinx.com/fpga-...:1, aws.amazon.com/neuron: 1 (Inferentia), etc.

Exercises

1. (Beginner) Does Kubernetes support hardware other than GPUs? Via what mechanism?
2. (Beginner) What is an FPGA, briefly?
3. (Intermediate) How does a workload request a non-GPU accelerator like an FPGA or Inferentia chip?
4. (Interview) Why has the device plugin model made Kubernetes a general control plane for heterogeneous/accelerated compute? (Hint: any vendor exposes hardware as a schedulable extended resource, requested uniformly.)

Answers

1. Yes — via the same device plugin framework used for GPUs, any specialized hardware can be exposed and scheduled.
2. A Field-Programmable Gate Array — a reconfigurable chip that can be programmed (its logic/bitstream) to accelerate specific tasks (e.g., custom inference, signal processing, compression), offering hardware-level acceleration tailored to a workload.
3. The same way as a GPU: request the vendor's extended resource in the container's resources.limits (e.g., fpga.intel.com/arria10: 1, aws.amazon.com/neuron: 1), and the scheduler places the Pod on a node where that device is available (the vendor's device plugin advertises and allocates it).
4. Because the device plugin framework lets any vendor expose any hardware as a schedulable extended resource (advertised per node, allocated by the kubelet), and workloads request all of them with the same uniform model (vendor.com/device: N) as CPU/memory. So GPUs, FPGAs, TPUs, Inferentia/Trainium, SmartNICs/DPUs, and RDMA NICs are all scheduled through one consistent mechanism, with vendor operators handling drivers/toolkits. This uniform abstraction over diverse accelerators is what makes Kubernetes a general-purpose control plane for heterogeneous, accelerated compute — a major reason it dominates AI/ML and HPC platforms.

---

Resource slicing and MIG partitioning

Theory

Because whole-GPU allocation often wastes expensive hardware (many workloads — small inference services, notebooks, dev — need only a fraction of a GPU), Kubernetes and NVIDIA support GPU sharing/partitioning to improve utilization:

• Time-slicing: the GPU is time-multiplexed among multiple Pods — they take turns on the full GPU. Simple and increases utilization, but there's no memory or fault isolation (a misbehaving Pod can affect others; they share the whole GPU's memory), and no guaranteed QoS.
• MPS (Multi-Process Service): allows concurrent kernels from multiple processes to share a GPU more efficiently than time-slicing, with limited isolation.
• MIG (Multi-Instance GPU): a hardware feature (on A100/H100-class GPUs) that partitions one physical GPU into multiple fully isolated instances, each with its own dedicated memory, compute units, and cache. MIG instances are exposed as separate schedulable resources (e.g., nvidia.com/mig-1g.5gb), giving strong, hardware-level isolation and predictable performance per slice.

The trade-off: time-slicing maximizes flexibility/utilization with no isolation (fine for trusted, bursty, or dev workloads); MIG provides true isolation and QoS at the cost of fixed partition sizes and requiring capable hardware (good for multi-tenant or production inference where isolation matters). Both let you pack multiple workloads onto costly GPUs instead of dedicating a whole GPU to an underutilizing job — directly addressing GPU cost-efficiency.

Example

Whole GPU (default):  [============ 1 GPU ============]  -> 1 Pod (may underutilize)

Time-slicing:         [ GPU ] shared by turns among Pod A, B, C  (no isolation)

MIG (hardware partition of one A100/H100):
  [ 1g.5gb ][ 1g.5gb ][ 2g.10gb ][ 3g.20gb ]  -> each an isolated schedulable slice
  requested as e.g.  nvidia.com/mig-1g.5gb: 1

Exercises

1. (Beginner) Why is whole-GPU allocation often wasteful?
2. (Beginner) What is the key isolation difference between time-slicing and MIG?
3. (Intermediate) How are MIG instances exposed to Kubernetes scheduling?
4. (Interview) When would you choose time-slicing versus MIG for GPU sharing, and what's the trade-off? (Hint: flexibility/utilization no isolation vs. hardware isolation/QoS with fixed partitions + capable hardware.)

Answers

1. Because many workloads (small inference, notebooks, dev/test) use only a fraction of a GPU's compute/memory, so dedicating a whole expensive GPU to them leaves most of it idle.
2. Time-slicing provides no memory/fault isolation — Pods time-share the full GPU and its memory, so they can interfere with each other. MIG partitions the GPU into hardware-isolated instances, each with dedicated memory and compute, so slices are isolated and have predictable performance.
3. Each MIG partition is advertised as its own schedulable extended resource (e.g., nvidia.com/mig-1g.5gb), so Pods request a specific MIG profile and the scheduler places them onto a node with that MIG instance available — just like requesting a whole GPU, but at partition granularity.
4. Choose time-slicing when you want maximum flexibility and utilization for trusted, bursty, or non-critical workloads (dev, notebooks) and can tolerate no isolation or QoS guarantees — it's simple and works on any GPU. Choose MIG when you need strong isolation and predictable performance/QoS per workload — e.g., multi-tenant environments or production inference where one workload must not affect another — accepting that MIG requires capable hardware (A100/H100-class) and uses fixed partition sizes (less flexible granularity). The trade-off is flexibility/utilization-without-isolation (time-slicing) versus true hardware isolation/QoS with rigid partitions and hardware requirements (MIG).

---

17.3 Serverless on Kubernetes

Serverless brings scale-to-zero and event-driven execution to Kubernetes. This subchapter covers the main frameworks.

---

Knative Serving

Theory

Serverless on Kubernetes means running code without managing servers/scaling explicitly — the platform scales your app up on demand and, crucially, down to zero when idle (so idle workloads cost nothing). Knative Serving is the leading framework that brings this request-driven autoscaling and scale-to-zero to Kubernetes.

Knative Serving abstracts away much of the Deployment/Service/Ingress/HPA boilerplate: you deploy a Knative Service (a single CRD) and it manages Revisions (immutable snapshots of your code+config for each deploy), Routes (traffic routing, including splitting between revisions for canary), and Configuration. Its defining features:

• Scale-to-zero: when there are no requests, Knative removes all Pods; when a request arrives, it rapidly cold-starts a Pod (the request is buffered by the activator until ready). This is what plain HPA can't do (min 1).
• Request-based autoscaling (KPA — Knative Pod Autoscaler): scales on concurrency/RPS, well-suited to spiky, request-driven workloads.
• Traffic splitting across revisions for easy canary/blue-green and instant rollback.

So Knative turns Kubernetes into a serverless platform for HTTP/event-driven services — you get FaaS/serverless ergonomics (deploy code, auto-scale from zero, pay for what you use) on top of standard Kubernetes.

Example

apiVersion: serving.knative.dev/v1
kind: Service
metadata: { name: hello }
spec:
  template:
    spec:
      containers:
      - image: gcr.io/knative-samples/helloworld-go
        env: [ { name: TARGET, value: "world" } ]
  traffic:                          # split traffic across revisions (canary)
  - { revisionName: hello-00002, percent: 10 }
  - { revisionName: hello-00001, percent: 90 }
# Idle -> scales to 0 Pods; a request -> cold-starts a Pod (via the activator).

Exercises

1. (Beginner) What key capability does Knative Serving add that a plain HPA cannot?
2. (Beginner) What is a Knative "Revision"?
3. (Intermediate) How does Knative serve a request that arrives when the app is scaled to zero?
4. (Interview) Why is scale-to-zero + request-based autoscaling well-suited to spiky/event-driven workloads, and what's the main trade-off? (Hint: cost when idle, elastic to bursts; cold-start latency.)

Answers

1. Scale-to-zero — removing all Pods when idle (and scaling back up on demand), which the HPA can't do (its minimum is 1).
2. An immutable snapshot of a particular deployment (the code/image + configuration) at a point in time; each change creates a new Revision, and traffic can be routed/split across Revisions (enabling canary and instant rollback).
3. The request is received by the Knative activator, which buffers it while Knative rapidly cold-starts a Pod (scales from 0 to 1); once the Pod is ready, the buffered request is forwarded — so the request is served despite there being no running Pods when it arrived.
4. Spiky/event-driven workloads are idle much of the time and then burst. Scale-to-zero means you pay nothing (no running Pods) during idle periods, and request-based autoscaling elastically adds Pods to handle bursts based on actual concurrency/RPS — matching resources to demand precisely. The main trade-off is cold-start latency: the first request after scaling from zero must wait for a Pod to start (and the app to initialize), adding latency to that request — a concern for latency-sensitive paths (mitigated by minimum-scale settings, keeping a warm Pod, at the cost of losing pure scale-to-zero savings).

---

Knative Eventing

Theory

Serverless isn't only request/response — much of it is event-driven: run code in reaction to events (a message on a queue, a file uploaded, a cron tick, a cloud event). Knative Eventing provides the infrastructure to build such systems on Kubernetes, based on loosely-coupled, standards-based event delivery (it uses the CloudEvents specification for a common event format).

Its building blocks:

• Sources: produce events from something (a Kafka topic, a database, a timer/PingSource, cloud services) and send them into the mesh.
• Brokers and Triggers: a Broker is an event hub/bus that receives events; Triggers subscribe to it with filters (by event type/attributes) and route matching events to a sink (e.g., a Knative Service). This decouples producers from consumers — producers emit to the broker without knowing who consumes.
• Sinks: anything that receives events (typically a Knative Service that then scales from zero to handle them).

The result is a declarative, decoupled, event-driven architecture: producers, brokers/triggers, and consumers are all Kubernetes resources, events are standardized (CloudEvents), and consumers can scale to zero and spin up on events. Knative Eventing complements Knative Serving (which runs the consuming services), turning Kubernetes into a platform for event-driven serverless applications.

Example

# A Trigger routes filtered events from a Broker to a sink (a Knative Service):
apiVersion: eventing.knative.dev/v1
kind: Trigger
metadata: { name: on-order }
spec:
  broker: default
  filter:
    attributes: { type: com.example.order.created }   # only these events
  subscriber:
    ref: { apiVersion: serving.knative.dev/v1, kind: Service, name: order-handler }
# Producers send CloudEvents to the Broker; order-handler scales from 0 to process them.

Exercises

1. (Beginner) What does Knative Eventing enable, in contrast to Knative Serving?
2. (Beginner) What event format standard does Knative Eventing use?
3. (Intermediate) What roles do Brokers and Triggers play, and how do they decouple producers from consumers?
4. (Interview) How do Knative Eventing and Serving combine to form an event-driven serverless architecture? (Hint: sources→broker/triggers→sink service that scales from zero.)

Answers

1. Building event-driven systems — running code in reaction to events (from queues, timers, cloud services, etc.) — whereas Knative Serving handles request/response HTTP services and their autoscaling.
2. CloudEvents (a common, standardized event format specification).
3. A Broker is a central event hub that receives events; Triggers subscribe to the broker with filters (by event type/attributes) and forward matching events to a subscriber/sink. This decouples producers from consumers: producers just emit events to the broker without knowing who (if anyone) consumes them, and consumers declare via triggers which events they want — so either side can change independently and multiple consumers can subscribe to the same events.
4. Sources produce CloudEvents (from Kafka, timers, cloud services, etc.) into a Broker; Triggers filter and route those events to sinks, which are typically Knative Serving Services. Those consuming services can scale from zero, spinning up only when events arrive and back down when idle. So Eventing provides the decoupled, standardized event pipeline (sources → broker/triggers) and Serving provides the elastic, scale-to-zero compute that processes the events — together forming a fully serverless, event-driven architecture on Kubernetes where you pay only for actual event processing.

---

OpenFaaS on Kubernetes

Theory

OpenFaaS ("Functions as a Service") is a popular framework for running functions and microservices on Kubernetes with a strong focus on developer simplicity. Where Knative is a lower-level, feature-rich serverless platform, OpenFaaS emphasizes an easy, opinionated developer experience for packaging code as functions and deploying them.

Its model and features:

• Functions as containers: you write a function (in many languages via templates), and OpenFaaS packages it into a container behind a standard watchdog process that handles the HTTP/invocation interface — so any code/binary can become a function.
• Simple CLI/workflow: faas-cli scaffolds (new), builds, pushes, and deploys functions; a UI and API gateway manage and invoke them.
• Autoscaling (including scale-to-zero via add-ons) driven by request load, and integration with the Kubernetes ecosystem (it runs on top of Kubernetes, using its scheduling/scaling).
• Async invocations via a queue (NATS) for event-driven/long-running work.

OpenFaaS's niche is making serverless functions approachable: a gentle, batteries-included workflow (write function → faas-cli up → invoke via gateway) that abstracts Kubernetes details. It's a good fit for teams wanting FaaS ergonomics and quick function deployment without the deeper complexity/flexibility of Knative.

Example

faas-cli new hello --lang python3        # scaffold a function from a template
# edit handler.py ...
faas-cli up -f hello.yml                  # build + push + deploy to the cluster
curl http://gateway/function/hello -d "hi" # invoke via the API gateway

OpenFaaS model:  your code -> template + watchdog -> container -> deployed function
                 gateway routes/invokes; autoscaling (incl. scale-to-zero add-on)

Exercises

1. (Beginner) What is OpenFaaS's main focus/niche?
2. (Beginner) What component wraps your code so any language/binary can be a function?
3. (Intermediate) Outline the typical OpenFaaS developer workflow.
4. (Interview) How does OpenFaaS differ in emphasis from Knative, and when might a team prefer it? (Hint: developer-simplicity/opinionated FaaS ergonomics vs. Knative's lower-level richness/flexibility.)

Answers

1. Making serverless functions/microservices simple to build and deploy on Kubernetes — a developer-experience-focused FaaS framework.
2. The watchdog process (a small server that fronts your function and handles the HTTP/invocation interface), so any code or binary packaged with it becomes an invocable function.
3. Scaffold a function from a language template (faas-cli new), implement the handler, then build/push/deploy in one step (faas-cli up), and invoke it through the API gateway (e.g., curl .../function/<name>), managing/monitoring via the CLI/UI.
4. OpenFaaS emphasizes an opinionated, batteries-included, easy developer workflow (templates, CLI, gateway, UI) that abstracts Kubernetes details for quick function deployment. Knative is a lower-level, more feature-rich and flexible serverless platform (fine-grained revisions/traffic splitting, request-based autoscaling, eventing) with more capability but more complexity. A team might prefer OpenFaaS when they want straightforward FaaS ergonomics and fast, simple function deployment without needing Knative's depth/flexibility or wanting to manage its complexity.

---

KEDA for scale-to-zero

Theory

KEDA (Kubernetes Event-Driven Autoscaling) was introduced in Chapter 8 as the event-driven autoscaler; in the serverless context, its standout role is enabling scale-to-zero for ordinary Kubernetes Deployments driven by event sources — bringing serverless elasticity to regular workloads without adopting a full serverless framework like Knative.

Why it fits serverless:

• Scale-to-zero on events: KEDA can scale a Deployment down to 0 replicas when there's no work (e.g., an empty queue) and scale it up from 0 when events arrive (queue depth > threshold) — the core serverless behavior, applied to any Deployment.
• Rich event triggers: it scales on real workload signals — Kafka lag, RabbitMQ/SQS queue length, Prometheus queries, cron, cloud pub/sub, etc. — via dozens of scalers, which is exactly what event-driven/serverless workers need (scale by backlog, not CPU).
• Lightweight & Kubernetes-native: it works with your existing Deployments (via a ScaledObject), reusing the HPA under the hood for the 1..N range and handling activation from/to zero itself — no need to rewrite apps as "functions."

So KEDA is often the pragmatic path to serverless-style, event-driven, scale-to-zero workloads on standard Kubernetes — especially queue-driven workers — complementing (or as a lighter alternative to) Knative when you don't need a full FaaS/serving platform.

Example

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata: { name: worker }
spec:
  scaleTargetRef: { name: worker }     # an ordinary Deployment
  minReplicaCount: 0                    # scale-to-zero when the queue is empty
  maxReplicaCount: 50
  triggers:
  - type: aws-sqs-queue
    metadata: { queueURL: https://sqs.../jobs, queueLength: "5" }  # scale by backlog

Exercises

1. (Beginner) What serverless capability does KEDA bring to ordinary Deployments?
2. (Beginner) What kinds of signals does KEDA scale on?
3. (Intermediate) How does KEDA relate to the HPA when scaling in the 1..N range versus to/from zero?
4. (Interview) When might KEDA be preferable to a full serverless framework like Knative for event-driven workloads? (Hint: scale-to-zero on real Deployments by backlog, no app rewrite, lighter footprint.)

Answers

1. Scale-to-zero (and scale-up-from-zero) driven by events — serverless elasticity applied to standard Kubernetes Deployments.
2. External event-source metrics — e.g., message-queue depth/lag (Kafka, SQS, RabbitMQ), Prometheus query results, cron schedules, cloud pub/sub — via its many scalers (i.e., real workload signals like backlog, not just CPU).
3. For the 1..N range, KEDA creates and uses a standard HPA under the hood to do the scaling math on the event metric. For the 0↔1 activation (scale-to-zero and back), KEDA handles it itself (since the HPA's minimum is 1), watching the event source and activating the workload when work appears and deactivating it when idle.
4. KEDA is preferable when you want serverless-style, event-driven, scale-to-zero behavior for existing Deployments — especially queue/backlog-driven workers — without rewriting apps as functions or adopting a full serving/FaaS platform. It's lightweight, Kubernetes-native, integrates with your normal Deployments via a ScaledObject, and scales on the real workload signal (queue depth) rather than CPU. Choose it over Knative when you don't need Knative's request-serving features (revisions, traffic splitting, HTTP activation) and just want event-driven autoscaling (including to zero) on standard workloads with minimal footprint and change.

---

17.4 AI/ML Workloads on Kubernetes

Kubernetes has become the standard platform for machine learning. This subchapter covers the ML-specific ecosystem.

---

Kubeflow overview and components

Theory

Machine learning involves a whole lifecycle — data prep, experimentation, training, tuning, serving, monitoring — not just running a container. Kubeflow is the leading open-source project for making Kubernetes a complete ML platform, providing an integrated suite of components covering that lifecycle so teams can do ML at scale on Kubernetes with reproducibility and portability.

Key Kubeflow components:

• Notebooks: managed Jupyter notebook servers in the cluster for interactive development.
• Kubeflow Pipelines: build, run, and track ML workflows/pipelines (DAGs of steps like preprocess → train → evaluate → deploy) with experiment tracking and artifact lineage.
• Katib: automated hyperparameter tuning and neural architecture search.
• Training Operators: distributed training for frameworks (next topic).
• KServe: model serving (later topic).
• Central Dashboard & multi-tenancy (profiles) to organize teams/projects.

The value proposition: instead of stitching together bespoke ML infrastructure, Kubeflow offers a cohesive, Kubernetes-native ML platform — reproducible pipelines, scalable distributed training, tuning, and serving — leveraging Kubernetes' scheduling (including GPUs) and portability across clouds/on-prem. It turns Kubernetes from "runs containers" into "runs the ML lifecycle."

Example

Kubeflow = an ML platform on Kubernetes (integrated components):
  Notebooks   -> interactive dev (Jupyter)
  Pipelines   -> reproducible ML workflows (preprocess->train->eval->deploy)
  Katib       -> hyperparameter tuning / NAS
  Training Operators -> distributed training (PyTorch/TF/...)
  KServe      -> model serving (scalable inference)
  Dashboard + profiles -> UI + multi-tenant organization

Exercises

1. (Beginner) What is Kubeflow's purpose?
2. (Beginner) Name three Kubeflow components and what each does.
3. (Intermediate) What does Kubeflow Pipelines provide for the ML workflow?
4. (Interview) Why is running the ML lifecycle on Kubernetes/Kubeflow attractive compared to bespoke ML infrastructure? (Hint: cohesive platform, reproducibility, GPU scheduling, portability, scale.)

Answers

1. To make Kubernetes a complete, integrated machine-learning platform covering the full ML lifecycle (development, pipelines, tuning, distributed training, and serving).
2. Any three: Notebooks (managed Jupyter for interactive dev), Pipelines (build/run/track ML workflows), Katib (hyperparameter tuning/NAS), Training Operators (distributed training for PyTorch/TF/etc.), KServe (model serving), Central Dashboard/profiles (UI and multi-tenancy).
3. A way to define, run, and track ML workflows as DAGs of steps (e.g., data prep → train → evaluate → deploy), with experiment tracking, artifact/lineage recording, and reproducibility — turning ad-hoc scripts into repeatable, versioned pipelines.
4. Kubeflow provides a cohesive, Kubernetes-native platform for the entire ML lifecycle instead of teams stitching together separate bespoke tools. It gives reproducibility (versioned pipelines/experiments), leverages Kubernetes scheduling including GPUs/accelerators and elasticity for scalable distributed training and serving, offers portability across clouds/on-prem (standard Kubernetes), and integrates development, tuning, training, and serving in one place with multi-tenancy. This reduces the effort and fragmentation of building ML infrastructure and lets ML teams operate at scale on the same robust platform used for other workloads.

---

Training operators (PyTorch, TensorFlow)

Theory

Large models are trained across many machines/GPUs in parallel (distributed training), which requires coordinating multiple worker processes, assigning roles (e.g., parameter servers, workers, masters), wiring up their network addresses, and handling failures. Doing this by hand on Kubernetes is complex. Training operators (part of the Kubeflow Training Operator, formerly per-framework operators like tf-operator/pytorch-operator) automate distributed training via CRDs for each framework — e.g., PyTorchJob, TFJob, MPIJob, XGBoostJob.

You declare a training job as a CRD specifying the replica roles and counts (e.g., 1 master + N workers for PyTorch, or workers + parameter servers for TF) and the container/image, and the operator:

• Creates the Pods with the correct roles and injects the environment each framework needs for distributed coordination (e.g., MASTER_ADDR, rank/world-size, TF_CONFIG).
• Manages the job lifecycle — startup ordering, restarts on failure, and completion.
• Integrates with GPU scheduling, gang-scheduling, and topology.

So training operators turn "run a distributed PyTorch/TensorFlow job across a GPU cluster" into a declarative Kubernetes resource — you describe the distributed job's shape and the operator handles the orchestration, making large-scale, multi-node/multi-GPU training practical and reproducible on Kubernetes.

Example

apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata: { name: train }
spec:
  pytorchReplicaSpecs:
    Master: { replicas: 1, template: { spec: { containers: [ { name: pytorch, image: my-train:latest,
              resources: { limits: { nvidia.com/gpu: 1 } } } ] } } }
    Worker: { replicas: 3, template: { spec: { containers: [ { name: pytorch, image: my-train:latest,
              resources: { limits: { nvidia.com/gpu: 1 } } } ] } } }
# The operator creates 1 master + 3 workers, wiring distributed-training env vars.

Exercises

1. (Beginner) What problem do training operators solve?
2. (Beginner) Name two CRDs provided for distributed training.
3. (Intermediate) What does a training operator do beyond just creating Pods?
4. (Interview) How do training operators make large-scale distributed training declarative and reproducible on Kubernetes? (Hint: CRD describes replica roles; operator wires coordination env, lifecycle, GPU/gang scheduling.)

Answers

1. Automating the complex orchestration of distributed training across multiple machines/GPUs — coordinating multiple worker processes, assigning roles, wiring their networking, and handling lifecycle/failures — which is tedious and error-prone to do manually.
2. Any two: PyTorchJob, TFJob, MPIJob, XGBoostJob.
3. It assigns the correct replica roles (e.g., master/workers, or workers/parameter servers), injects the framework-specific coordination environment (e.g., MASTER_ADDR, rank/world-size, TF_CONFIG) so the processes can find each other, manages startup ordering and restarts on failure, tracks job completion, and integrates with GPU/gang scheduling and topology.
4. You describe the distributed job's shape as a CRD — the replica roles, counts, image, and resources (GPUs) — and the operator handles all the orchestration: creating the right Pods, wiring the distributed-training coordination env, managing lifecycle/failures, and scheduling onto GPUs. Because the entire job is a declarative Kubernetes resource (versionable, repeatable) rather than manual, imperative setup, large-scale multi-node/multi-GPU training becomes reproducible and portable — run the same manifest to get the same distributed job — while leveraging Kubernetes scheduling for the accelerators.

---

KServe for model serving

Theory

Training produces a model; serving it — exposing it for low-latency, scalable inference — is a distinct challenge with its own needs (autoscaling with demand, GPU use, canary rollouts of model versions, standardized prediction APIs, and scale-to-zero for cost). KServe (formerly KFServing) is the standard Kubernetes framework for model inference serving, built on Knative (for serverless autoscaling/scale-to-zero) and often Istio.

What KServe provides:

• A simple InferenceService CRD: you point it at a trained model (in storage) and a model framework runtime (TensorFlow, PyTorch/TorchServe, scikit-learn, XGBoost, ONNX, or custom), and it deploys a scalable serving endpoint — no need to build serving containers yourself.
• Serverless inference: leveraging Knative for autoscaling including scale-to-zero (idle models cost nothing; spin up on request), and GPU support for accelerated inference.
• Advanced serving features: canary rollouts and traffic splitting between model versions, explainability and payload logging, transformers (pre/post-processing), and support for model ensembles/pipelines.
• Standardized inference protocol (the V2 inference protocol) for consistent client APIs across frameworks.

So KServe turns "deploy a model for production inference" into a declarative resource with production-grade serving features and serverless economics — the serving counterpart to the training operators, completing the ML lifecycle on Kubernetes.

Example

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata: { name: sklearn-iris }
spec:
  predictor:
    minReplicas: 0                 # scale-to-zero when idle (via Knative)
    model:
      modelFormat: { name: sklearn }
      storageUri: "gs://models/iris/v1"   # point at the trained model
# KServe deploys a scalable inference endpoint with a standardized predict API.

Exercises

1. (Beginner) What problem does KServe address in the ML lifecycle?
2. (Beginner) What CRD do you use to deploy a model with KServe?
3. (Intermediate) What does KServe gain by building on Knative?
4. (Interview) What production serving features does KServe provide beyond just running a model, and why do they matter? (Hint: canary/traffic-split, scale-to-zero, standardized protocol, explainability/transformers.)

Answers

1. Model serving/inference — exposing trained models as scalable, production-grade, low-latency prediction endpoints (the deployment/serving stage, as opposed to training).
2. An InferenceService.
3. Serverless capabilities — autoscaling driven by request load and scale-to-zero (idle models consume no resources, spinning up on demand), plus request-based scaling — leveraging Knative Serving so inference endpoints are elastic and cost-efficient.
4. Beyond running the model, KServe offers: canary rollouts / traffic splitting between model versions (safely test new models with a fraction of traffic, easy rollback); scale-to-zero and autoscaling for cost-efficiency and elasticity; a standardized inference protocol/API across frameworks (consistent clients regardless of model type); transformers for pre/post-processing; explainability and payload logging for trust/monitoring; and support for ensembles/pipelines and GPUs. These matter because production ML serving needs safe model updates, cost control, consistent interfaces, observability, and the ability to scale with real demand — KServe provides them declaratively so teams don't build bespoke serving infrastructure per model.

---

Data pipelines with Argo Workflows

Theory

ML and data work is full of multi-step batch workflows — ETL/data processing, feature engineering, training pipelines, batch scoring — that need orchestration: run steps in a defined order (a DAG), pass artifacts between them, run steps in parallel, and handle retries/failures. Argo Workflows is the leading Kubernetes-native workflow engine for exactly this: it runs each step as a container (Pod) and orchestrates them as a DAG (or step sequence), all defined declaratively as a CRD.

Key features:

• Container-native steps: every step is a container, so any tool/language works, leveraging Kubernetes scheduling (including GPUs) and scaling.
• DAG and step templates: express dependencies and parallelism (fan-out/fan-in), loops, conditionals.
• Artifact passing: outputs of one step (files, data) passed to the next, with artifact storage (S3/GCS) integration.
• Retries, timeouts, and resource control per step, plus scheduling (CronWorkflow) for recurring pipelines.

Argo Workflows is the engine underneath many pipeline systems (Kubeflow Pipelines runs on Argo Workflows). Its niche: general-purpose, Kubernetes-native pipeline/batch orchestration — well-suited to data pipelines, CI/CD jobs, and ML workflows where you need to chain containerized steps reliably at scale. It complements the ML-specific components (training operators, KServe) by handling the surrounding data-processing and orchestration glue.

Example

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata: { generateName: ml-pipeline- }
spec:
  entrypoint: dag
  templates:
  - name: dag
    dag:
      tasks:
      - { name: preprocess, template: run, arguments: {...} }
      - { name: train, template: run, dependencies: [preprocess] }   # DAG ordering
      - { name: evaluate, template: run, dependencies: [train] }
  - name: run
    container: { image: my-ml-step:latest, command: ["python","step.py"] }

Exercises

1. (Beginner) What is Argo Workflows used for?
2. (Beginner) How is each step in an Argo Workflow executed?
3. (Intermediate) What features make Argo Workflows suitable for ML/data pipelines (name two)?
4. (Interview) What is Argo Workflows' niche relative to ML-specific tools like training operators and KServe, and how does it relate to Kubeflow Pipelines? (Hint: general container-native DAG/batch orchestration; Kubeflow Pipelines runs on Argo.)

Answers

1. Orchestrating multi-step, container-based batch workflows/pipelines on Kubernetes — e.g., data processing/ETL, ML training pipelines, batch jobs — defined as DAGs or step sequences.
2. As a container (a Kubernetes Pod) — each step runs in its own container, orchestrated by the workflow engine.
3. Any two: DAG/step definition with dependencies and parallelism (fan-out/fan-in, loops, conditionals); artifact passing between steps (with S3/GCS storage); per-step retries/timeouts/resource control; container-native steps leveraging Kubernetes scheduling (incl. GPUs); and scheduled/recurring runs (CronWorkflow).
4. Argo Workflows is a general-purpose, Kubernetes-native workflow/DAG engine for orchestrating any containerized multi-step pipeline (data, ML, CI/CD), whereas training operators and KServe are ML-specific (distributed training and model serving respectively). Argo handles the surrounding orchestration and data-processing glue that connects such steps. Notably, Kubeflow Pipelines is built on top of Argo Workflows — it uses Argo as its execution engine — so Argo underpins ML pipeline orchestration while remaining a broadly applicable batch/workflow tool beyond ML.

---

17.5 Edge and IoT Kubernetes

Kubernetes is stretching beyond the data center to edge and IoT environments. This subchapter covers that frontier.

---

k3s for edge deployments

Theory

Running Kubernetes at the edge — retail stores, factories, telco sites, vehicles, remote locations — means constrained hardware, limited/intermittent networking, and often many small sites with little on-site expertise. Standard Kubernetes is too heavy for such environments. k3s (Chapter 3) is the lightweight, CNCF-certified distribution purpose-built for exactly this: a single small binary (<100 MB), low memory footprint, bundled components, and SQLite as the default datastore — able to run on modest hardware like a Raspberry Pi or small industrial gateway.

Why k3s fits the edge:

• Small footprint & simplicity: minimal resources and a one-command install make it deployable on constrained devices and manageable at many sites.
• Still real Kubernetes: CNCF-conformant, so the same manifests/tooling/skills apply — you develop against standard Kubernetes and deploy to the edge unchanged.
• Edge-appropriate defaults: bundled essentials (a lightweight ingress, local storage, etc.) reduce the setup burden; can run as a single node or small HA cluster (with embedded etcd).

So k3s brings the Kubernetes API and ecosystem to resource-constrained, distributed edge locations without the overhead of full Kubernetes — the foundation for edge/IoT Kubernetes deployments (and often combined with fleet-management tooling from Chapter 15 to manage many edge clusters).

Example

# One-command install on a small edge device (Raspberry Pi, gateway):
curl -sfL https://get.k3s.io | sh -
# Single binary, ~<100MB, SQLite datastore by default, low memory use.

# Manage many edge k3s clusters centrally with fleet tooling (Rancher Fleet, Ch.15).

Exercises

1. (Beginner) Why is standard Kubernetes often unsuitable for edge devices?
2. (Beginner) What characteristics make k3s a fit for the edge?
3. (Intermediate) Why does k3s being CNCF-conformant matter for edge development?
4. (Interview) What challenges of edge environments does k3s address, and how are many edge clusters typically managed? (Hint: constrained hardware/network/expertise; small footprint + fleet management/GitOps.)

Answers

1. Because edge devices have constrained CPU/memory and standard Kubernetes has a relatively heavy resource footprint and complex setup, which is impractical on modest hardware and hard to operate across many small remote sites.
2. A single small binary (<100 MB), low memory footprint, bundled components, SQLite default datastore, and simple one-command install — able to run on modest hardware (e.g., Raspberry Pi) while remaining full Kubernetes.
3. Because conformance guarantees k3s implements the standard Kubernetes APIs/behaviors, so the same manifests, tooling, and skills work — teams develop against standard Kubernetes and deploy to constrained edge clusters unchanged, without maintaining edge-specific variants.
4. k3s addresses edge constraints: limited hardware (small footprint, low memory, single binary), operational simplicity (easy install, bundled essentials, less expertise needed per site), and running standard Kubernetes on modest/remote devices (with single-node or small HA options). Because edge deployments involve many clusters/sites, they're typically managed centrally with fleet-management and GitOps tooling (e.g., Rancher Fleet, Argo CD ApplicationSet — Chapter 15), which push and reconcile configuration/apps and policy across all the edge clusters consistently despite intermittent connectivity.

---

KubeEdge architecture

Theory

k3s runs a full (small) cluster at each edge site, but some IoT/edge scenarios need something different: extending a central cluster's control out to edge nodes/devices that may be numerous, resource-tiny, and only intermittently connected. KubeEdge (a CNCF project) is built for this — it extends Kubernetes orchestration to the edge while keeping the control plane in the cloud, and is designed to tolerate unreliable networks and to interface with IoT devices.

KubeEdge's architecture splits into:

• CloudCore (runs in the cloud/central cluster): syncs with the Kubernetes API server and manages edge nodes/metadata, communicating with the edge over a reliable messaging channel.
• EdgeCore (runs on edge nodes): includes EdgeD (a lightweight edge agent that manages Pods locally, like a kubelet) and MetaManager (local metadata store) — crucially enabling offline autonomy: edge nodes keep running their workloads and cache state even when disconnected from the cloud, then resync when connectivity returns.
• Device management: a DeviceTwin/device model and MQTT-based mapper support integrating actual IoT devices (sensors/actuators) into Kubernetes as manageable resources.

So where k3s = "a small standalone cluster at the edge," KubeEdge = "one cloud control plane orchestrating many edge nodes/devices, resilient to disconnection, with IoT device integration." It targets large-scale IoT/edge fleets with central management and offline-capable edge execution.

Example

KubeEdge (cloud control plane -> many edge nodes/devices):

  Cloud:  [ K8s API server ] <-> [ CloudCore ] ==reliable channel==>
  Edge:   [ EdgeCore: EdgeD (mini-kubelet) + MetaManager (local cache) ]
             |-> runs Pods locally; AUTONOMOUS when offline; resyncs on reconnect
             |-> DeviceTwin + MQTT mapper -> real IoT sensors/actuators

Exercises

1. (Beginner) How does KubeEdge's model differ from running k3s at each site?
2. (Beginner) What are the two main components (cloud side and edge side) of KubeEdge?
3. (Intermediate) What does "offline autonomy" mean for a KubeEdge edge node?
4. (Interview) What edge/IoT challenges is KubeEdge specifically designed to handle that standard Kubernetes/k3s doesn't emphasize? (Hint: central control of many edge nodes, unreliable networks/offline operation, IoT device integration.)

Answers

1. k3s runs a full standalone (small) Kubernetes cluster at each edge site; KubeEdge instead keeps the control plane centrally (in the cloud) and extends it out to many lightweight edge nodes/devices, orchestrating them from the center rather than each site being its own cluster.
2. CloudCore (cloud side — syncs with the API server, manages edge nodes over a reliable channel) and EdgeCore (edge side — EdgeD mini-kubelet + MetaManager local cache running workloads on the edge node).
3. That an edge node continues running its assigned workloads and serving using locally cached state/metadata even when disconnected from the cloud control plane, and then reconciles/resyncs with the cloud once connectivity is restored — so intermittent network loss doesn't stop the edge from operating.
4. KubeEdge targets: central management of large numbers of edge nodes/devices from a cloud control plane (rather than many independent clusters); resilience to unreliable/intermittent networks via offline autonomy (edge keeps running and caches state when disconnected, resyncing later); running on very resource-constrained edge hardware; and IoT device integration (DeviceTwin/device models with MQTT mappers to manage sensors/actuators as Kubernetes resources). Standard Kubernetes (and k3s) assume relatively reliable connectivity and don't focus on cloud-orchestrated fleets of tiny, sometimes-offline edge nodes or native IoT device management — which is exactly KubeEdge's design focus.

---

Akri for IoT device discovery

Theory

A hard problem at the edge is leaf devices — cameras, sensors, USB peripherals, IP devices — that aren't nodes and can't run an agent, yet you want to discover them and use them from Kubernetes workloads. Akri (a CNCF project, "A Kubernetes Resource Interface" for the edge) solves exactly this: it lets you dynamically discover such devices and expose them as Kubernetes resources that Pods can request, even as devices come and go.

How Akri works:

• You define a Configuration describing what to look for using a discovery protocol/handler (ONVIF for IP cameras, udev for USB, OPC UA for industrial devices, or custom).
• Akri's discovery agents (a DaemonSet) find matching devices on/near each node and, for each discovered device, Akri creates an Instance (a custom resource representing that device) and advertises it via the device plugin framework as a schedulable resource.
• Workloads request the device like any extended resource; Akri can also automatically deploy a "broker" Pod per device to expose it (e.g., a camera's frames) to the rest of the cluster. When a device disappears, Akri cleans up; when new ones appear, they're discovered automatically.

So Akri extends the device plugin concept from node-attached hardware (GPUs) to networked/leaf IoT devices, making dynamic, heterogeneous edge devices first-class, discoverable, schedulable Kubernetes resources — filling a key gap for IoT/edge use cases.

Example

# Akri Configuration: discover ONVIF IP cameras and expose them as resources
apiVersion: akri.sh/v0
kind: Configuration
metadata: { name: onvif-cameras }
spec:
  discoveryHandler: { name: onvif }          # protocol to find devices
  brokerSpec: { ... }                         # optional per-device broker Pod
# Discovered cameras -> Akri Instances -> advertised via device plugin ->
# Pods request them like:  resources: { limits: { akri.sh/onvif-cameras: "1" } }

Exercises

1. (Beginner) What problem does Akri solve at the edge?
2. (Beginner) Give two examples of discovery protocols Akri supports.
3. (Intermediate) How does Akri make a discovered device usable by Pods?
4. (Interview) How does Akri extend the device plugin concept, and why is that valuable for IoT/edge? (Hint: from node-attached hardware to dynamic networked/leaf devices as schedulable resources.)

Answers

1. Discovering leaf/IoT devices (cameras, sensors, USB/IP devices) that can't run a Kubernetes agent themselves, and exposing them as Kubernetes resources that workloads can find and use — handling devices dynamically appearing and disappearing.
2. Any two: ONVIF (IP cameras), udev (USB devices), OPC UA (industrial devices), or custom discovery handlers.
3. Akri's discovery agents find matching devices and create an Instance custom resource per device, advertising each via the device plugin framework as a schedulable extended resource; Pods then request that resource (e.g., akri.sh/<config>: "1"), and Akri can deploy a broker Pod to expose the device's data to the cluster. The device becomes a normal, requestable Kubernetes resource.
4. The device plugin framework was designed for node-attached hardware (like GPUs) that's statically present on a node. Akri extends this to dynamic, networked/leaf devices that aren't nodes and appear/disappear over time — discovering them via pluggable protocols and advertising each as a schedulable resource (with optional broker Pods). This is valuable for IoT/edge because such environments are full of heterogeneous, transient devices (cameras, sensors) that you want to orchestrate from Kubernetes; Akri makes them first-class, discoverable, schedulable resources, so workloads can consume edge devices with the same request model as any other resource, dynamically as the fleet changes.

---

Offline and low-bandwidth cluster operation

Theory

Edge and IoT clusters frequently operate with intermittent, low-bandwidth, or no connectivity to central systems (a ship at sea, a remote site, a factory with locked-down networks). Designing for this — offline/disconnected operation — is a distinct discipline that cuts across the edge stack. The core principle: edge clusters must keep functioning autonomously when disconnected and degrade gracefully, syncing when connectivity returns.

Key techniques and considerations:

• Local autonomy: workloads and the local control/agent (e.g., k3s's local datastore, KubeEdge's EdgeCore offline mode) must run and self-heal without the cloud; cached state and local decision-making are essential.
• Air-gapped images/artifacts: with no registry access, you need local image registries/mirrors and pre-loaded images, plus offline installation bundles (no pulling from the internet at deploy time).
• Efficient/eventual sync: minimize data over constrained links — batch/compress telemetry, sync deltas, tolerate lag; GitOps pull models suit this (edge pulls config when it can, reconciles locally).
• Store-and-forward: buffer data/events locally when offline and forward on reconnect; avoid designs that assume constant cloud availability.
• Resilient management: fleet tooling and GitOps must tolerate clusters being unreachable for periods and reconcile opportunistically.

So operating at the edge means engineering for autonomy, local artifacts, bandwidth efficiency, and eventual synchronization rather than assuming a well-connected data center. Tools like k3s (local datastore), KubeEdge (offline autonomy), local registries, and pull-based GitOps together make disconnected/low-bandwidth Kubernetes operation viable.

Example

Designing for disconnected/low-bandwidth edge:
  autonomy:      edge runs + self-heals offline (k3s local DB / KubeEdge EdgeCore)
  air-gapped:    local registry/mirror + pre-loaded images + offline install bundles
  efficient sync: batch/compress/delta telemetry; tolerate lag
  store-forward: buffer events locally -> forward on reconnect
  management:    pull-based GitOps reconciles when connectivity allows

Exercises

1. (Beginner) What is the core requirement for an edge cluster during a network outage?
2. (Beginner) Why are local image registries/mirrors important for edge/air-gapped clusters?
3. (Intermediate) Why does a pull-based GitOps model suit intermittently-connected edge clusters?
4. (Interview) What design principles make Kubernetes viable in offline/low-bandwidth edge environments? (Hint: local autonomy, air-gapped artifacts, efficient/eventual sync, store-and-forward, resilient management.)

Answers

1. That it keeps functioning autonomously — continuing to run workloads and self-heal using local state/decision-making — despite being disconnected, then resyncing when connectivity returns.
2. Because air-gapped/edge clusters may have no (or poor) access to public registries at deploy time, so images must be available locally — a local registry/mirror and pre-loaded images let Pods start without pulling over the internet, and enable offline installs/upgrades.
3. In a pull model, the edge cluster's agent pulls desired state from Git (and reconciles locally) whenever it has connectivity, rather than depending on a central system to push to it in real time. This tolerates intermittent links — the edge reconciles opportunistically when connected and keeps running from its last-known state when offline — whereas a push model assumes the central system can reach the cluster on demand.
4. Principles: local autonomy (edge runs and self-heals offline via local datastore/agent, e.g., k3s/KubeEdge); air-gapped artifacts (local registries/mirrors, pre-loaded images, offline install bundles so no internet is needed at runtime); bandwidth efficiency and eventual sync (batch/compress/delta data, tolerate lag rather than assuming constant high-bandwidth links); store-and-forward (buffer telemetry/events locally and forward on reconnect); and resilient, pull-based management (GitOps/fleet tooling that reconciles opportunistically and tolerates clusters being unreachable). Together these replace the data-center assumption of constant connectivity with designs that keep the edge operational and eventually consistent under disconnection and constrained networks.

---