15. Multi-Cluster and Federation

A single cluster eventually hits limits — of scale, blast radius, geography, or compliance. Running multiple clusters solves these but introduces new problems: how do services find each other across clusters, how do you deploy consistently everywhere, and how do you manage the fleet's lifecycle? This chapter covers the patterns and tools for multi-cluster architectures, federation, and multi-cluster GitOps.

---

15.1 Multi-Cluster Patterns

This subchapter covers why and how organizations run multiple clusters, and the architectural patterns involved.

---

Reasons for multi-cluster architectures

Theory

A single large cluster is simpler to operate, so multi-cluster should be a deliberate choice driven by real needs. The common drivers:

• Blast radius / isolation: a cluster is a failure and security domain. Multiple clusters contain the impact of an outage, a bad upgrade, a misconfiguration, or a compromise to one cluster instead of everything.
• Geography / latency: placing clusters near users in different regions reduces latency and improves availability; clusters per region are natural.
• Compliance / data residency: regulations may require data to stay in a specific country/region, mandating separate clusters per jurisdiction.
• Scale limits: very large workloads can approach practical limits (etcd size, node count, API server load); splitting across clusters relieves this.
• Environment / tenant separation: separate clusters for prod vs. non-prod, or per business unit/tenant, for stronger isolation than namespaces provide.
• Cloud/hybrid: spanning multiple clouds or on-prem+cloud (avoiding lock-in, leveraging best-of-breed) inherently means multiple clusters.

The trade-off is significant added operational complexity (managing many clusters, cross-cluster networking, consistent config). So adopt multi-cluster when the isolation, locality, compliance, or scale benefits clearly justify that complexity — not by default.

Example

Single cluster:  simpler, but one failure/security/compliance domain for everything

Multi-cluster drivers:
  isolation   -> contain outages/compromise per cluster
  geography   -> clusters near users (low latency)
  compliance  -> data stays in-region (EU cluster, US cluster)
  scale       -> avoid single-cluster limits (etcd/nodes/API)
  hybrid/multicloud -> span clouds/on-prem

Exercises

1. (Beginner) Name three reasons organizations adopt multi-cluster architectures.
2. (Beginner) What is the main downside of running many clusters?
3. (Intermediate) How does multi-cluster limit "blast radius" compared to one large cluster?
4. (Interview) Why should multi-cluster be a deliberate decision rather than a default, and what factors tip the balance? (Hint: complexity cost vs. isolation/latency/compliance/scale benefits.)

Answers

1. Any three: blast-radius/failure-and-security isolation, geographic proximity/latency, compliance/data residency, scale limits of a single cluster, environment/tenant separation, and multi-cloud/hybrid spanning.
2. Significantly increased operational complexity — managing many clusters, cross-cluster networking/discovery, and keeping configuration and policy consistent everywhere.
3. Each cluster is a separate failure and security domain. An outage, faulty upgrade, misconfiguration, or breach affects only its cluster rather than the entire estate, so problems are contained and other clusters keep running — unlike one big cluster where a single incident can take everything down.
4. A single cluster is simpler to operate, and each additional cluster multiplies operational overhead (networking, config drift, tooling, upgrades). So you should add clusters only when specific benefits clearly outweigh that cost. Factors that tip the balance: needing strong isolation/blast-radius containment, users spread across regions (latency/availability), legal data-residency/compliance requirements, approaching single-cluster scale limits, or a genuine multi-cloud/hybrid strategy. Absent such drivers, one well-run cluster (with namespaces/RBAC for tenancy) is usually preferable.

---

Active-active vs active-passive

Theory

For high availability across clusters, two fundamental topologies exist:

• Active-active: multiple clusters all serve live traffic simultaneously (e.g., a cluster per region, each handling its region's users, with global load balancing/DNS distributing traffic). Benefits: full utilization of all clusters, low latency (users hit the nearest), and seamless failover (if one cluster dies, others already serve traffic). Challenges: data must be replicated/synchronized across clusters (hard for stateful/consistency-sensitive systems), and you need global traffic management.
• Active-passive (active-standby): one cluster serves traffic; a second standby cluster is kept ready (data replicated to it) and only takes over on failover. Simpler consistency (one active writer) and a clear DR story, but the passive cluster's capacity is idle most of the time (cost), and failover has a recovery time (detecting failure + promoting standby).

The choice hinges on data consistency needs, cost tolerance, and RTO/RPO (recovery time/point objectives). Stateless or eventually-consistent workloads favor active-active for utilization and instant failover; strongly-consistent stateful systems often use active-passive to avoid multi-writer conflicts. Many real architectures mix both (active-active stateless front ends, active-passive databases).

Example

Active-Active:                         Active-Passive:
  Global LB/DNS                          Traffic -> [Primary] (active)
    /        \                                         | replicate data
 [Cluster A] [Cluster B]  both serve                [Standby] (idle, ready)
  data replicated both ways             failover: promote standby (RTO delay)
  full utilization, instant failover    simpler consistency, idle capacity

Exercises

1. (Beginner) In active-active, how many clusters serve live traffic?
2. (Beginner) What is the main downside of active-passive regarding the standby cluster?
3. (Intermediate) Why is data consistency harder in active-active than active-passive?
4. (Interview) How do RTO/RPO and data-consistency requirements drive the choice between active-active and active-passive? (Hint: multi-writer sync vs. single-writer + failover time.)

Answers

1. All of them — multiple clusters serve live traffic simultaneously.
2. The standby cluster sits mostly idle (reserved capacity that isn't serving traffic), which wastes resources/cost; there's also a failover delay before it takes over.
3. Active-active has multiple clusters accepting writes concurrently, so data must be replicated and kept consistent across them — risking multi-writer conflicts and requiring conflict resolution or complex distributed consistency. Active-passive has a single active writer with data replicated one-way to the standby, avoiding concurrent-write conflicts and making consistency much simpler.
4. Active-active gives near-zero RTO (other clusters already serve traffic, so failover is seamless) and full utilization, but demands multi-cluster data synchronization — feasible for stateless or eventually-consistent data, hard for strongly-consistent stateful systems. Active-passive gives simpler, strong consistency (one writer) and a clean DR model, but incurs a nonzero RTO (detect failure + promote standby) and possible RPO gap (replication lag), plus idle standby cost. So workloads needing instant failover and horizontal locality with tolerant data models choose active-active; strongly-consistent stateful systems that can accept some failover time choose active-passive — and many systems combine both per tier.

---

Cluster segmentation strategies

Theory

If you're going to run multiple clusters, how do you divide workloads among them? Common segmentation strategies (often combined):

• By environment: separate clusters for prod, staging, dev — strong isolation of production from experimentation, different security/RBAC and stability requirements.
• By region/zone: clusters per geographic region for latency, availability, and data residency.
• By tenant/business unit: a cluster per team, customer, or business unit for isolation, cost attribution, and independent lifecycle (stronger than namespace tenancy).
• By workload type/criticality: separate clusters for, say, batch/ML vs. latency-sensitive services, or high-security vs. general workloads (so noisy or risky workloads don't affect critical ones).
• By cloud/infra: clusters per cloud provider or on-prem for hybrid/multi-cloud strategies.

The overarching trade-off is again isolation vs. overhead: finer segmentation gives stronger isolation and clearer boundaries but more clusters to manage and more cross-cluster complexity. The goal is to segment along the axes where isolation matters most for your risk, compliance, and organizational structure — while keeping the number of clusters manageable (which is where fleet-management and multi-cluster GitOss tooling, later, becomes essential).

Example

Segmentation axes (often combined):
  Environment:  [prod] [staging] [dev]
  Region:       [us-east] [eu-west] [ap-south]
  Tenant/BU:    [team-a] [team-b] [customer-x]
  Workload:     [latency-sensitive] [batch/ML] [high-security]
  Infra:        [aws] [gcp] [on-prem]

more segmentation = stronger isolation but more clusters to manage

Exercises

1. (Beginner) Name three axes along which you can segment clusters.
2. (Beginner) What is the recurring trade-off in choosing how finely to segment?
3. (Intermediate) Why might you put batch/ML workloads in a separate cluster from latency-sensitive services?
4. (Interview) How should an organization decide its cluster segmentation strategy? (Hint: segment where isolation matters most for risk/compliance/org, balance against management overhead.)

Answers

1. Any three: by environment (prod/staging/dev), by region/zone, by tenant/business unit, by workload type/criticality, and by cloud/infrastructure.
2. Isolation vs. operational overhead — finer segmentation gives stronger isolation and clearer boundaries but creates more clusters to manage and more cross-cluster complexity.
3. To isolate resource contention and risk: batch/ML jobs are bursty and resource-hungry (and often lower priority), so isolating them prevents them from starving or destabilizing latency-sensitive services; the two also have different scaling, node types (e.g., GPUs), and reliability needs, which are easier to manage in separate clusters.
4. Segment along the axes where isolation delivers the most value for the organization's specific risk (blast radius), compliance (data residency, security tiers), latency/geography, and organizational structure (teams/tenants/cost attribution) — prioritizing separations that matter (e.g., prod vs. non-prod, per-region for residency). Then balance that against management overhead: don't create more clusters than you can operate consistently, and invest in fleet-management/GitOps tooling so the chosen segmentation stays maintainable. The right strategy maximizes needed isolation while keeping the cluster count and cross-cluster complexity manageable.

---

Cross-cluster service discovery

Theory

Once services live in different clusters, they need to find and reach each other — but each cluster has its own DNS (CoreDNS) and its own Service ClusterIPs, isolated from the others. Cross-cluster service discovery bridges this so a service in cluster A can resolve and call a service in cluster B.

Approaches:

• Multi-cluster Services (MCS API): a Kubernetes SIG standard (ServiceExport/ServiceImport) where you "export" a Service in one cluster and it becomes discoverable/reachable (via a clusterset.local domain) in others that import it — implemented by tools like Submariner or cloud/mesh integrations.
• Service mesh multi-cluster: meshes (Istio, Linkerd, Cilium Cluster Mesh) can span clusters, providing unified service discovery, mTLS, and traffic routing across them — often the richest option.
• Global DNS / load balancers: expose services externally and use global DNS or a global LB to route across clusters (coarser, typically for edge/ingress traffic).
• Flat networking: tools that connect cluster Pod networks (Submariner, Cilium Cluster Mesh) so Pods/Services are directly routable across clusters.

The key challenge is that cross-cluster calls cross network and trust boundaries, so discovery is usually paired with secure connectivity (mTLS/encrypted tunnels). Choosing an approach depends on how tightly you need clusters integrated (occasional external calls vs. a seamless multi-cluster mesh).

Example

Multi-cluster Services (MCS API):
  Cluster A:  ServiceExport(payments)  --->  payments.ns.svc.clusterset.local
  Cluster B:  ServiceImport(payments)  --->  resolves + routes to A's payments

Service mesh (e.g., Istio/Cilium Cluster Mesh):
  unified discovery + mTLS + routing across clusters (Pods reachable cross-cluster)

Exercises

1. (Beginner) Why can't a service in cluster A resolve a service in cluster B by default?
2. (Beginner) What Kubernetes standard defines exporting/importing Services across clusters?
3. (Intermediate) Name two approaches to cross-cluster service discovery and how they differ.
4. (Interview) Why is cross-cluster discovery usually paired with secure connectivity, and what makes a service mesh an attractive option for it? (Hint: traffic crosses trust boundaries; mesh adds mTLS + unified routing/discovery.)

Answers

1. Each cluster has its own isolated DNS (CoreDNS) and its own Service ClusterIPs/network; a cluster's DNS only knows its own Services, and Pod/Service networks are separate — so B's names/IPs aren't resolvable or routable from A without additional integration.
2. The Multi-Cluster Services (MCS) API, using ServiceExport and ServiceImport (with the clusterset.local domain).
3. Any two: the MCS API (export/import Services via a SIG standard, implemented by tools like Submariner) provides Kubernetes-native cross-cluster Services; a service mesh spanning clusters (Istio/Linkerd/Cilium Cluster Mesh) adds unified discovery plus mTLS and traffic management; global DNS/load balancers route to externally-exposed services (coarser, edge-level); flat cross-cluster networking (Submariner, Cilium Cluster Mesh) makes Pods/Services directly routable. They differ in integration depth (native Services vs. mesh features vs. external routing) and whether they also provide security/routing.
4. Cross-cluster traffic leaves one cluster's network and trust domain and traverses (often untrusted) networks to another, so it needs encryption and authenticated identity to be safe — hence pairing discovery with mTLS/encrypted tunnels. A service mesh is attractive because it provides both in one system: unified cross-cluster service discovery and automatic mTLS, plus consistent traffic management/observability across clusters — so services get secure, identity-verified, routable connectivity spanning clusters without each app implementing it, making the multi-cluster estate behave more like one integrated mesh.

---

15.2 Federation and Multi-Cluster Tools

This subchapter surveys the tools that manage workloads and connectivity across clusters.

---

Kubefed (Kubernetes Federation v2)

Theory

Kubefed (KubeFed, Kubernetes Cluster Federation v2) was the official attempt to manage multiple clusters as one: from a central host cluster, you define federated versions of resources (FederatedDeployment, FederatedService, etc.) with a template, placement (which clusters), and overrides (per-cluster differences), and Kubefed propagates them to the member clusters. It aimed to give a single control point for deploying and configuring workloads across a fleet.

However, Kubefed struggled with adoption and is now effectively deprecated/archived. Its approach — special federated wrapper types and a central push controller — proved complex, added a new API surface, and didn't align well with how teams actually operate (many preferred GitOps-based propagation). It's important primarily as historical/conceptual context: it illustrates the federation idea (template + placement + overrides propagated to clusters) that later tools (and multi-cluster GitOps like Argo CD ApplicationSet, and projects like Karmada) refined. The lesson: naive "one API to rule all clusters" federation is hard; the ecosystem largely moved to GitOps and purpose-built multi-cluster schedulers instead.

Example

# Kubefed-style federated resource (conceptual/historical):
apiVersion: types.kubefed.io/v1beta1
kind: FederatedDeployment
metadata: { name: web }
spec:
  template: { ... }                 # base Deployment
  placement:
    clusters: [ { name: cluster-a }, { name: cluster-b } ]   # which clusters
  overrides:
  - clusterName: cluster-b
    clusterOverrides: [ { path: "/spec/replicas", value: 5 } ]  # per-cluster diff

Exercises

1. (Beginner) What was Kubefed's goal?
2. (Beginner) What three parts define a federated resource in Kubefed (template + …)?
3. (Intermediate) Why did Kubefed struggle with adoption?
4. (Interview) What conceptual model did Kubefed establish, and what approaches largely replaced it? (Hint: template/placement/overrides; GitOps propagation and tools like Karmada.)

Answers

1. To manage multiple clusters centrally as a federation — defining resources once on a host cluster and propagating them (with per-cluster placement and overrides) to member clusters.
2. Template (the base resource), placement (which clusters to deploy to), and overrides (per-cluster customizations).
3. Its model introduced complex, special "federated" wrapper types and a central push-controller API surface that added significant complexity and didn't match how teams increasingly preferred to operate (GitOps). It saw limited adoption and is now effectively deprecated/archived.
4. Kubefed established the conceptual federation model of template + placement + overrides propagated to member clusters. That idea persists, but the mechanism was largely replaced by GitOps-based multi-cluster propagation (e.g., Argo CD ApplicationSet, Flux across clusters) and purpose-built multi-cluster orchestration/scheduling projects (e.g., Karmada, Admiralty), which achieve fleet-wide deployment with less bespoke API surface and better alignment with declarative, Git-driven workflows.

---

Liqo for transparent multi-cluster

Theory

Liqo takes a strikingly different, "transparent" approach to multi-cluster: instead of federating resource definitions, it makes remote clusters appear as virtual nodes in your local cluster. Using the Virtual Kubelet concept, Liqo represents a peered remote cluster as a big node in the local cluster; when you schedule a Pod "onto" that virtual node, Liqo actually runs it in the remote cluster — transparently, with cross-cluster networking and service reachability handled automatically ("resource borrowing"/offloading).

The appeal is seamlessness: from the local cluster's perspective, you just deploy Pods as usual and they may transparently run elsewhere, effectively pooling capacity across clusters as if they were one. Clusters establish peering relationships (which can be dynamic/on-demand), and Liqo extends networking and service discovery across the peered clusters so offloaded workloads still communicate. It's well-suited to bursting/overflow (borrow capacity from another cluster when yours is full) and treating multiple clusters as a unified resource pool — a fundamentally different model from explicit federation or GitOps propagation, prioritizing transparency of the multi-cluster boundary.

Example

Liqo peering makes a remote cluster look like a local virtual node:

  Local cluster
   [real node][real node][ liqo virtual-node (== remote cluster) ]
                                       |
        Pod scheduled here transparently runs in the REMOTE cluster
        (Liqo handles cross-cluster networking + service reachability)

Exercises

1. (Beginner) How does Liqo represent a remote cluster locally?
2. (Beginner) What underlying concept does Liqo build on to do this?
3. (Intermediate) What multi-cluster use case is Liqo especially well-suited to?
4. (Interview) How does Liqo's "transparent" model differ fundamentally from federation (Kubefed) or GitOps propagation? (Hint: virtual node/offloading vs. explicit per-cluster resource definitions.)

Answers

1. As a virtual node in the local cluster; scheduling a Pod onto that node transparently runs it in the peered remote cluster.
2. The Virtual Kubelet concept (a virtual node backed by another system — here, a remote cluster).
3. Capacity bursting/overflow and pooling resources across clusters — e.g., borrowing capacity from a peered cluster when the local one is full, treating multiple clusters as a unified resource pool.
4. Federation (Kubefed) and GitOps propagation require you to explicitly define/propagate resources per cluster (templates, placement, overrides, or per-cluster manifests) — the multi-cluster boundary is visible and managed. Liqo instead hides the boundary: remote clusters appear as local virtual nodes, so you deploy Pods normally and they may transparently execute in another cluster, with networking/service discovery bridged automatically. It's resource-pooling/offloading (capacity federation) rather than declarative resource federation — you don't manage where things go per cluster; you let the local cluster transparently extend into peers.

---

Admiralty for multi-cluster scheduling

Theory

Admiralty focuses specifically on multi-cluster scheduling — intelligently placing workloads across a set of clusters. Like Liqo, it uses virtual-node (Virtual Kubelet) techniques so that a management/source cluster can offload Pods to target clusters, but its emphasis is on the scheduling decision: given multiple candidate clusters, decide which cluster each Pod should run in based on capacity, constraints, cost, or locality, then delegate execution there.

The model: you submit Pods (or workloads) to a source cluster; Admiralty's cross-cluster scheduler evaluates member clusters and creates "delegate" Pods in the chosen target cluster(s), while the source cluster tracks them via proxy/virtual-node Pods. This enables cluster-spanning scheduling — spreading a Deployment's replicas across clusters for HA, bin-packing across a fleet, or routing workloads to clusters with spare/appropriate capacity — using familiar Kubernetes scheduling semantics (affinities, topology spread) extended to the multi-cluster level. It's a good fit when you want a scheduler-driven way to distribute workloads across clusters rather than manually assigning them.

Example

Admiralty: submit to source cluster -> cross-cluster scheduler picks target(s)

  Source cluster:  Pod (pending) --scheduler evaluates member clusters-->
      creates delegate Pods in chosen target(s):
        [Cluster A: 2 replicas]  [Cluster B: 1 replica]  (spread for HA/capacity)
  source tracks them via virtual-node proxy Pods

Exercises

1. (Beginner) What is Admiralty's primary focus?
2. (Beginner) What technique (shared with Liqo) does Admiralty use to offload Pods?
3. (Intermediate) Give an example of a scheduling goal Admiralty enables across clusters.
4. (Interview) How does Admiralty extend familiar Kubernetes scheduling semantics to the multi-cluster level, and when would you choose it? (Hint: cross-cluster scheduler + delegate Pods; scheduler-driven distribution vs. manual placement.)

Answers

1. Multi-cluster scheduling — intelligently deciding which cluster each workload should run in and placing it there.
2. Virtual-node / Virtual Kubelet techniques (representing target clusters so Pods can be offloaded to them).
3. Any example such as: spreading a Deployment's replicas across multiple clusters for high availability, bin-packing/distributing workloads across a fleet by available capacity, or routing Pods to clusters with appropriate/spare resources or better locality/cost.
4. You submit workloads to a source cluster as usual, and Admiralty's cross-cluster scheduler evaluates member clusters — honoring Kubernetes scheduling constructs (affinities, topology spread, resource requests) at the fleet level — then creates delegate Pods in the chosen target cluster(s) while the source tracks them via virtual-node proxies. So multi-cluster placement is expressed with the same familiar semantics as single-cluster scheduling. Choose Admiralty when you want a scheduler-driven way to automatically distribute workloads across clusters (for HA, capacity, or locality) rather than manually deciding and deploying to each cluster.

---

Submariner for cross-cluster networking

Theory

Most multi-cluster patterns assume workloads in different clusters can actually reach each other over the network — but by default cluster Pod/Service networks are isolated (and may even have overlapping CIDRs). Submariner solves the networking layer: it connects the Pod and Service networks of multiple clusters, enabling direct, encrypted cross-cluster connectivity and cross-cluster Service discovery.

Its components:

• Gateway engines in each cluster establish secure (IPsec/WireGuard) tunnels between clusters, so Pods in one cluster can route to Pods/Services in another.
• Lighthouse provides cross-cluster Service discovery (implementing the MCS API's *.clusterset.local DNS), so services are resolvable across the connected clusters.
• It handles overlapping CIDRs (via Globalnet) when clusters weren't planned with distinct ranges.

Submariner is essentially the connectivity fabric underneath higher-level multi-cluster patterns: things like cross-cluster service calls, multi-cluster meshes, or federated apps need the networks joined, and Submariner provides that flat, secure inter-cluster network and discovery. It's the answer to "my clusters can't talk to each other's Pods/Services — connect them."

Example

Submariner joins cluster networks with secure tunnels + cross-cluster DNS:

  [Cluster A]==(IPsec/WireGuard tunnel via Gateways)==[Cluster B]
   Pods/Services in A  <---- directly reachable ---->  Pods/Services in B
   Lighthouse DNS:  payments.ns.svc.clusterset.local resolves across clusters
   Globalnet handles overlapping Pod CIDRs

Exercises

1. (Beginner) What layer of the multi-cluster problem does Submariner address?
2. (Beginner) How does Submariner secure traffic between clusters?
3. (Intermediate) What does Submariner's Lighthouse component provide, and what problem does Globalnet solve?
4. (Interview) Why is a connectivity fabric like Submariner a prerequisite for many higher-level multi-cluster patterns? (Hint: they assume cross-cluster Pod/Service reachability, which is isolated by default.)

Answers

1. The networking layer — connecting the Pod/Service networks of multiple clusters to enable direct cross-cluster connectivity and discovery.
2. Via encrypted tunnels between per-cluster gateway engines, using IPsec or WireGuard.
3. Lighthouse provides cross-cluster Service discovery (implementing the MCS API *.clusterset.local DNS so Services resolve across connected clusters). Globalnet handles overlapping Pod/Service CIDRs between clusters that weren't planned with distinct ranges, allowing them to interconnect despite address conflicts.
4. Higher-level patterns — cross-cluster service calls, multi-cluster service meshes, federated/offloaded workloads — all assume that a Pod/Service in one cluster can actually reach one in another. By default cluster networks are isolated (and possibly overlapping), so that assumption doesn't hold. Submariner establishes the flat, secure inter-cluster network and cross-cluster DNS that makes cross-cluster reachability real, so the higher-level patterns have the connectivity foundation they depend on. Without such a fabric, multi-cluster discovery/routing has nothing to route over.

---

15.3 Multi-Cluster GitOps

This subchapter covers managing many clusters declaratively with GitOps and fleet tooling.

---

Argo CD ApplicationSet for multi-cluster

Theory

Managing dozens of clusters by hand-writing an Argo CD Application per app-per-cluster doesn't scale. ApplicationSet is an Argo CD controller that templates and generates Applications automatically from a generator, so you define the pattern once and it produces (and keeps in sync) many Applications across clusters/environments.

Generators supply the inputs to fill the Application template:

• Cluster generator: create the app on every registered cluster (or those matching a label) — e.g., "deploy monitoring to all clusters labeled env=prod."
• Git generator: generate Applications from directories/files in a Git repo (e.g., one app per subfolder).
• List / Matrix / Merge generators: explicit lists or combinations (cluster × app) for fine control.

When you add a new cluster (and it matches the generator), ApplicationSet automatically creates the corresponding Applications — no manual per-cluster work. This makes fleet-wide, consistent GitOps deployment practical: declare "these apps on these clusters" as a templated ApplicationSet, and Argo CD keeps every cluster reconciled. It's the standard way to do multi-cluster GitOps with Argo CD.

Example

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata: { name: monitoring, namespace: argocd }
spec:
  generators:
  - clusters: { selector: { matchLabels: { env: prod } } }   # all prod clusters
  template:
    metadata: { name: 'monitoring-{{name}}' }                # per-cluster App
    spec:
      project: default
      source: { repoURL: https://github.com/org/config, path: monitoring, targetRevision: main }
      destination: { server: '{{server}}', namespace: monitoring }
      syncPolicy: { automated: { prune: true, selfHeal: true } }

Exercises

1. (Beginner) What problem does ApplicationSet solve for multi-cluster Argo CD?
2. (Beginner) What does the cluster generator do?
3. (Intermediate) What happens when you register a new cluster matching an ApplicationSet's generator?
4. (Interview) How do ApplicationSet generators enable scalable, consistent fleet-wide GitOps? (Hint: template once, generate many Applications from cluster/Git/list inputs; auto-onboard clusters.)

Answers

1. It removes the need to hand-write a separate Application per app-per-cluster by templating and automatically generating those Applications, so managing many clusters scales.
2. It generates an Application for each registered cluster (optionally filtered by label selector), so an app is deployed across all matching clusters.
3. ApplicationSet automatically generates the corresponding Application(s) for the new cluster and Argo CD deploys/reconciles them — the cluster is onboarded to the relevant apps with no manual per-cluster configuration.
4. Generators feed inputs (clusters, Git directories, explicit lists, or combinations via matrix/merge) into a single Application template, so you declare the deployment pattern once and ApplicationSet produces and continuously maintains all the concrete Applications across the fleet. Adding/removing clusters or apps that match the generators automatically adds/removes Applications, keeping every cluster consistently reconciled to Git without manual effort. This templated, input-driven generation is what makes large-scale, uniform multi-cluster GitOps practical.

---

Cluster API (CAPI) for cluster lifecycle

Theory

GitOps manages what runs on clusters — but who manages the clusters themselves (creating, scaling, upgrading, deleting them)? Cluster API (CAPI) is a Kubernetes SIG project that brings declarative, Kubernetes-style lifecycle management to clusters: you describe your clusters as Kubernetes resources (Cluster, MachineDeployment, MachineSet, Machine, and infrastructure/control-plane provider resources), and CAPI controllers provision and manage the actual clusters and their nodes to match.

The key idea: use Kubernetes to manage Kubernetes. A management cluster runs CAPI, which creates and operates workload clusters on various infrastructures via providers (AWS, Azure, GCP, vSphere, bare metal, etc.). Because clusters are now declarative CRs, you can apply the same practices to fleet lifecycle as to apps: version them in Git, GitOps-reconcile them, scale a node pool by editing a MachineDeployment's replicas, and perform rolling upgrades by changing the Kubernetes version in the spec. CAPI is the foundation for treating a fleet of clusters as cattle — reproducible, declarative, automatable — rather than hand-built pets, and it underpins many higher-level fleet/cluster-management platforms.

Example

# CAPI: declare a cluster and a node pool as Kubernetes resources
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata: { name: prod-eu }
spec:
  controlPlaneRef: { kind: KubeadmControlPlane, name: prod-eu-cp }
  infrastructureRef: { kind: AWSCluster, name: prod-eu-infra }
---

apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata: { name: prod-eu-workers }
spec:
  replicas: 5                       # scale nodes by editing this
  # ... references worker infra + kubeadm bootstrap + version

Exercises

1. (Beginner) What does Cluster API manage, in contrast to GitOps tools like Argo CD?
2. (Beginner) What is the "management cluster" vs. a "workload cluster"?
3. (Intermediate) How would you scale a workload cluster's node pool using CAPI?
4. (Interview) What does "use Kubernetes to manage Kubernetes" mean with CAPI, and why is declarative cluster lifecycle valuable for a fleet? (Hint: clusters as CRs; reproducible, GitOps-able, rolling upgrades.)

Answers

1. Cluster API manages the clusters themselves — their creation, scaling, upgrading, and deletion (and their nodes/machines) — whereas Argo CD/GitOps manage the workloads/config running inside clusters.
2. The management cluster runs CAPI and its controllers, which provision and operate workload clusters (the actual clusters that run your applications) on various infrastructures via providers.
3. Edit the MachineDeployment's replicas field (declaratively) — CAPI's controllers reconcile by adding/removing machines (nodes) in that pool to match the new count.
4. CAPI models clusters and their nodes as Kubernetes custom resources reconciled by controllers running in a management cluster — so you manage clusters with the same declarative, API-driven, controller-reconciled approach Kubernetes uses for Pods (hence "Kubernetes managing Kubernetes"). This is valuable because cluster lifecycle becomes reproducible and automatable: clusters can be version-controlled in Git and GitOps-reconciled, provisioned consistently across providers, scaled by editing specs, and upgraded via rolling changes (bump the version in the spec, controllers roll nodes). It turns a fleet of clusters into declarative "cattle" — consistent, auditable, and easy to recreate — rather than fragile hand-built environments.

---

Fleet management with Rancher Fleet

Theory

Rancher Fleet is a GitOps tool built specifically for managing large fleets of clusters at scale — it's designed from the ground up to deploy and manage applications across many clusters (Rancher cites scaling to a very large number). It follows a hub-and-spoke model: a central Fleet controller (in a management cluster) drives agents in downstream clusters, deploying bundles (Git-sourced sets of manifests/Helm/Kustomize) to clusters selected by labels.

Its fleet-oriented design centers on:

• GitRepo resources pointing at Git repositories of deployments.
• Bundles: the unit of deployment, distributed to targeted clusters via cluster/label selectors and ClusterGroups.
• Per-cluster customization: values/overrides applied per target so the same bundle adapts to each cluster.

Fleet is integrated with Rancher (the broader cluster-management platform) but usable standalone. Its niche versus Argo CD/Flux is scale of clusters and fleet-first ergonomics — targeting bundles across thousands of clusters by label with per-target overrides, rather than being primarily a single-cluster/app-centric GitOps tool. It's a strong choice when your primary challenge is consistently managing app delivery across a very large or edge cluster fleet.

Example

# Fleet GitRepo: deploy a bundle to clusters selected by label
apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata: { name: apps, namespace: fleet-default }
spec:
  repo: https://github.com/org/fleet-apps
  paths: [ "monitoring", "logging" ]
  targets:
  - clusterSelector: { matchLabels: { env: prod } }   # target all prod clusters

Exercises

1. (Beginner) What is Rancher Fleet designed for?
2. (Beginner) What is the unit of deployment in Fleet called, and how are targets chosen?
3. (Intermediate) What is Fleet's niche compared to Argo CD/Flux?
4. (Interview) Why does managing thousands of clusters call for a fleet-first GitOps tool, and what capabilities matter most at that scale? (Hint: label-based targeting, per-cluster overrides, hub-and-spoke agents.)

Answers

1. GitOps-based management and application delivery across large fleets of clusters at scale (including edge/many-cluster scenarios).
2. The bundle (a Git-sourced set of manifests/Helm/Kustomize); targets are chosen via cluster/label selectors and ClusterGroups (deploying the bundle to matching clusters, with per-target customization).
3. Fleet is built fleet-first — optimized for deploying and managing apps across a very large number of clusters with label-based targeting and per-cluster overrides (and integrates with Rancher) — whereas Argo CD/Flux are more app/cluster-centric general GitOps tools (though they also support multi-cluster). Fleet's niche is scale of clusters and fleet ergonomics.
4. At thousands of clusters, per-cluster manual configuration is impossible and you need to express deployments as patterns applied across many clusters. The capabilities that matter: label/selector-based targeting (deploy to groups of clusters by attributes, auto-including new matching clusters), per-cluster overrides (adapt one bundle to each cluster's specifics), a scalable hub-and-spoke agent architecture (central control, downstream agents pulling/applying, resilient to scale), and efficient reconciliation. A fleet-first tool like Fleet is engineered for this scale and these ergonomics, whereas tools centered on individual apps/clusters become unwieldy across a massive fleet.

---

Policy propagation across clusters

Theory

In a fleet, you must enforce consistent policy and configuration everywhere — security baselines, resource quotas, RBAC, network policies, admission rules — not cluster-by-cluster (which drifts). Policy propagation is the practice of defining governance centrally and distributing/enforcing it across all clusters uniformly.

Approaches:

• GitOps as the propagation mechanism: store policies (Kyverno/Gatekeeper policies, RBAC, quotas, NetworkPolicies) in Git and have every cluster's GitOps agent apply them — so policy is version-controlled, reviewed, and reconciled onto all clusters (ApplicationSet/Fleet target them fleet-wide).
• Policy engines with multi-cluster distribution: Kyverno and OPA Gatekeeper policies propagated to all clusters (Kyverno even supports policy-report aggregation; Gatekeeper constraints can be templated across clusters).
• Multi-cluster orchestration (e.g., Karmada): propagate resources including policies with placement rules and per-cluster overrides.
• Managed policy services: cloud/fleet platforms (e.g., Google Config Sync/Policy Controller, Azure Policy for AKS, Rancher) offer built-in fleet-wide policy enforcement and drift/compliance reporting.

The goals are consistency (the same guardrails everywhere), compliance visibility (report which clusters comply), and drift prevention (reconcile policy back if changed). The dominant modern pattern is policy-as-code in Git, propagated via GitOps and enforced by policy engines — turning fleet governance into a reviewable, automated, continuously-reconciled system rather than manual per-cluster configuration.

Example

Central policy-as-code in Git:
  [Kyverno/Gatekeeper policies | RBAC | quotas | NetworkPolicies]
                     |  propagate via GitOps (ApplicationSet / Fleet / Karmada)
       +-------------+-------------+
       v             v             v
   Cluster A     Cluster B     Cluster C   (same guardrails enforced + reconciled)
             |-> compliance/drift reports aggregated back

Exercises

1. (Beginner) What is policy propagation across clusters?
2. (Beginner) Name two kinds of policy you'd want consistent across a fleet.
3. (Intermediate) How does GitOps serve as a policy-propagation mechanism?
4. (Interview) What are the goals of fleet-wide policy propagation, and why is "policy-as-code in Git + policy engines" the dominant pattern? (Hint: consistency, compliance visibility, drift prevention; reviewable, automated, reconciled.)

Answers

1. Defining governance/config centrally and distributing and enforcing it uniformly across all clusters in a fleet (rather than configuring each cluster separately).
2. Any two: security baselines/admission policies (Kyverno/Gatekeeper), RBAC, resource quotas, NetworkPolicies, Pod Security standards, or image/registry policies.
3. Policies are stored as code in Git and each cluster's GitOps agent (Argo CD/ApplicationSet, Flux, Fleet) applies and continuously reconciles them — so the same policies are deployed to every targeted cluster, version-controlled and reviewed, and any drift is corrected. Fleet-targeting mechanisms push them to all (matching) clusters automatically.
4. Goals: consistency (identical guardrails on every cluster), compliance visibility (know which clusters conform), and drift prevention (automatically revert unauthorized changes). "Policy-as-code in Git + policy engines" dominates because Git gives version control, review/approval, and auditability of policy changes; GitOps continuously reconciles the policies onto all clusters (self-healing against drift); and policy engines (Kyverno/Gatekeeper) actually enforce them at admission and report compliance. Together this turns fleet governance into a reviewable, automated, continuously-enforced system that scales across many clusters — far more reliable and auditable than manual, per-cluster policy configuration.

---