12. CI/CD and GitOps

Building images and writing manifests is only half the story — you need a reliable, automated path from a code commit to a running workload. This chapter covers continuous integration/delivery pipelines for Kubernetes, then the GitOps paradigm that makes Git the single source of truth and uses in-cluster agents to continuously reconcile the cluster to it, with deep dives into the two leading GitOps tools, Argo CD and Flux.

---

12.1 CI/CD Pipelines for Kubernetes

This subchapter covers the mechanics of pipelines: building images, deploying from CI, and advanced rollout strategies.

---

Building and pushing container images in CI

Theory

The first job of a Kubernetes CI pipeline is to turn source code into a container image and push it to a registry the cluster can pull from. On a commit, CI checks out the code, builds the image, tags it, and pushes it. The tagging strategy is critical: avoid the mutable :latest tag for deployments; instead tag with an immutable, traceable identifier — the Git commit SHA (and/or a semantic version) — so each image maps to exact source and deployments are reproducible.

Building images inside a containerized/Kubernetes CI environment has a wrinkle: you often can't (or shouldn't) use a privileged Docker daemon ("Docker-in-Docker" needs privileged mode, a security risk). Daemonless builders solve this — Kaniko, BuildKit/buildx, or Buildah build images inside a container without a privileged Docker daemon. The pipeline then authenticates to the registry (via short-lived credentials/OIDC where possible) and pushes the tagged image, optionally signing it (cosign) and scanning it (Trivy) as covered in Chapter 9.

Example

# GitHub Actions: build and push tagged by commit SHA
- uses: docker/build-push-action@v5
  with:
    push: true
    tags: registry.example.com/myapp:${{ github.sha }}   # immutable, traceable

# Daemonless build with Kaniko (no privileged Docker needed), in-cluster:
/kaniko/executor --dockerfile=Dockerfile --context=. \
  --destination=registry.example.com/myapp:$GIT_SHA

Exercises

1. (Beginner) What is the first job of a Kubernetes CI pipeline?
2. (Beginner) Why should you avoid tagging deployment images with :latest?
3. (Intermediate) Why is "Docker-in-Docker" problematic in containerized CI, and what tools avoid it?
4. (Interview) Why is tagging images by Git commit SHA important for reproducibility and rollback? (Hint: immutable mapping image↔source; deterministic deploys.)

Answers

1. To build a container image from the source code and push it to a registry the cluster can pull from.
2. :latest is mutable — it can point to different images over time — so deployments aren't reproducible, rollbacks are ambiguous, and caching/pull behavior is unreliable. Immutable, specific tags (commit SHA / version) are required for deterministic deploys.
3. Docker-in-Docker requires a privileged container (access to the Docker daemon/host), which is a significant security risk and often disallowed in shared CI/Kubernetes runners. Daemonless builders — Kaniko, BuildKit/buildx, Buildah — build images inside an unprivileged container without a Docker daemon, avoiding the privilege requirement.
4. A commit SHA uniquely and immutably identifies the exact source state; tagging the image with it creates a one-to-one, permanent mapping between a running image and the code that produced it. This makes deployments fully reproducible (you always know what's running), enables precise rollback (redeploy the image tagged with the previous good SHA), and supports auditing/debugging (trace a production image back to its commit). Mutable tags break all of these guarantees.

---

Deploying to Kubernetes from CI (GitHub Actions, GitLab CI)

Theory

After the image is pushed, the pipeline deploys it. The traditional push-based CD approach: the CI runner authenticates to the cluster (kubeconfig/credentials) and runs kubectl apply, helm upgrade, or kustomize build | kubectl apply to update the workload — typically by bumping the image tag to the just-built SHA. Platforms like GitHub Actions and GitLab CI provide jobs/runners and secret storage to do this.

The key consideration is credential security and blast radius: push-based CD means your CI system holds credentials that can modify the production cluster — a high-value target. Best practices: use short-lived, scoped credentials (OIDC federation between CI and the cloud/cluster rather than long-lived kubeconfig secrets), least-privilege RBAC for the deploy identity, and separate credentials per environment. This push model is simple and common, but its security drawbacks (external system with cluster write access) are a major motivation for GitOps (next subchapter), where an in-cluster agent pulls changes instead.

Example

# GitHub Actions: push-based deploy via Helm, image tag = commit SHA
deploy:
  steps:
  - uses: azure/setup-helm@v4
  - run: |
      helm upgrade --install web ./chart -n prod \
        --set image.tag=${{ github.sha }} --atomic --wait
    env:
      KUBECONFIG: ${{ secrets.PROD_KUBECONFIG }}   # CI holds cluster credentials

Exercises

1. (Beginner) In push-based CD, what does the CI runner do to deploy?
2. (Beginner) Name two platforms commonly used to run Kubernetes CI/CD pipelines.
3. (Intermediate) What is the main security concern with push-based deployment from CI?
4. (Interview) Why are OIDC-federated short-lived credentials preferred over a stored long-lived kubeconfig for CI deploys? (Hint: no standing secret to leak; scoped, expiring access.)

Answers

1. It authenticates to the cluster (using stored credentials) and applies changes — e.g., kubectl apply, helm upgrade, or kustomize build | kubectl apply — usually setting the new image tag (commit SHA).
2. GitHub Actions and GitLab CI (also Jenkins, CircleCI, Argo Workflows, Tekton — any two).
3. The CI system holds credentials with write access to the (production) cluster. CI is an externally-exposed, high-value target; if its secrets leak or a pipeline is compromised, an attacker gains the ability to modify the cluster — a large blast radius.
4. A stored long-lived kubeconfig is a standing secret: if it leaks it grants persistent cluster access until manually rotated. OIDC federation instead has the CI job exchange its identity for a short-lived, scoped token at runtime — there's no durable credential sitting in CI secrets to steal, access expires quickly, and it can be tightly scoped (specific cluster/role/environment) and audited. This dramatically reduces the risk and blast radius of credential compromise.

---

Rolling deployments in pipelines

Theory

When a pipeline updates a Deployment's image, Kubernetes performs a rolling update (Chapter 4) by default — but a robust pipeline must verify the rollout succeeded and react if it didn't, rather than fire-and-forget. The pipeline should wait for the rollout to complete and fail the job (and ideally roll back) if the new version isn't healthy.

The key tools:

• kubectl rollout status deployment/<name>: blocks until the rollout succeeds or fails (with a timeout) — the pipeline gates on this.
• kubectl rollout undo or helm upgrade --atomic: automatically revert on failure.
• Readiness probes (Chapter 10): essential — the rollout only progresses as new Pods become ready, so well-configured probes are what make the rollout (and the pipeline's success/failure signal) meaningful.

The pattern: apply the change → rollout status --timeout → on failure, roll back and fail the pipeline. This turns Kubernetes' built-in rolling update into a safe, gated CD step with automatic recovery, rather than blindly applying and hoping.

Example

# In the pipeline: apply, then gate on rollout health, roll back on failure
kubectl set image deployment/web web=registry.example.com/myapp:$GIT_SHA -n prod

if ! kubectl rollout status deployment/web -n prod --timeout=300s; then
  echo "Rollout failed — rolling back"
  kubectl rollout undo deployment/web -n prod
  exit 1
fi

Exercises

1. (Beginner) What command makes a pipeline wait for a rollout to finish?
2. (Beginner) Why are readiness probes essential to a meaningful rolling deployment?
3. (Intermediate) Describe the apply → verify → recover pattern for a safe pipeline deploy.
4. (Interview) Why is "fire-and-forget" kubectl apply insufficient for production CD, and what makes a rollout gate reliable? (Hint: no health verification; rollout status + probes + auto-rollback.)

Answers

1. kubectl rollout status deployment/<name> (typically with --timeout).
2. Because a rolling update only proceeds (and is judged successful) as new Pods become ready; readiness probes define "ready." Without them, Kubernetes considers a Pod available as soon as it starts, so the rollout/pipeline could report success even though the app isn't actually serving — defeating the verification.
3. Apply the change (new image), then verify with kubectl rollout status --timeout to wait for the new Pods to become ready; if it fails (times out / Pods unhealthy), recover by rolling back (kubectl rollout undo or helm --atomic) and fail the pipeline. This gates success on actual rollout health and auto-reverts bad deploys.
4. Plain kubectl apply returns success once the object is accepted by the API server, not when the new version is actually running and healthy — so a broken deploy can silently "succeed." A reliable gate combines rollout status (waits for real readiness within a timeout), correctly-configured readiness probes (so "ready" means "serving"), and automatic rollback on failure — making the pipeline's success signal reflect genuine application health and preventing bad versions from staying live.

---

Canary and blue-green deployments

Theory

Rolling updates replace all Pods over time, but two advanced strategies give finer control over risk during releases:

• Blue-green: run two complete environments — blue (current) and green (new). Deploy the new version to green while blue serves all traffic; once green is verified, switch all traffic (e.g., repoint the Service/Ingress) from blue to green instantly. Rollback is instant (switch back). Cost: you run double the capacity during the transition, and the cutover is all-at-once.
• Canary: route a small percentage of traffic (e.g., 5%) to the new version while the rest stays on the old, monitor metrics (errors, latency), and gradually increase the new version's share if healthy (or roll back if not). Limits blast radius and enables data-driven promotion, but requires traffic-splitting and good metrics.

Kubernetes Deployments don't do these natively (they only do rolling/recreate). You implement them with multiple Deployments + Service switching, an Ingress/service-mesh traffic split, or — most practically — progressive-delivery controllers like Argo Rollouts or Flagger, which automate canary/blue-green with metric analysis and automatic promotion/rollback.

Example

Blue-Green:                          Canary:
 [blue v1] <- 100% traffic            [v1] <- 95% ---\
 [green v2] (verify, 0%)              [v2] <-  5% ----> monitor metrics
 switch -> [green v2] <- 100%         healthy? -> 25% -> 50% -> 100% (else rollback)

# Argo Rollouts: a canary strategy with automated steps + analysis
spec:
  strategy:
    canary:
      steps:
      - setWeight: 5
      - pause: { duration: 5m }      # observe metrics
      - setWeight: 25
      - pause: { duration: 5m }
      - setWeight: 50
      - pause: {}                    # manual or analysis-driven promotion

Exercises

1. (Beginner) In a blue-green deployment, how does traffic move to the new version?
2. (Beginner) What does a canary deployment do with traffic initially?
3. (Intermediate) What is a trade-off of blue-green versus canary?
4. (Interview) Native Kubernetes Deployments don't support canary/blue-green directly. What tools provide progressive delivery, and what do they automate? (Hint: Argo Rollouts/Flagger; traffic shifting + metric analysis + auto promote/rollback.)

Answers

1. The new version (green) is deployed alongside the old (blue) with no traffic; once verified, all traffic is switched at once from blue to green (e.g., by repointing the Service/Ingress). Rollback is an instant switch back.
2. It sends only a small percentage of traffic to the new version (the canary) while the majority stays on the old version, then increases the share gradually based on observed health.
3. Blue-green needs double the capacity during the transition and cuts over all-at-once (instant rollback but a big-bang switch and higher cost). Canary uses less extra capacity and limits blast radius via gradual exposure, but is slower to fully roll out and requires traffic-splitting and reliable metrics to drive promotion. (Either contrast acceptable.)
4. Progressive-delivery controllers like Argo Rollouts and Flagger add canary/blue-green to Kubernetes. They automate traffic shifting (via Service/Ingress or a service mesh), run metric analysis (query Prometheus/etc. for error rate/latency at each step), and automatically promote the release when healthy or roll back when not — turning manual, scripted strategies into safe, automated, data-driven rollouts.

---

12.2 GitOps Principles

GitOps reframes deployment around Git as the source of truth and continuous reconciliation. This subchapter covers the core concepts.

---

GitOps core concepts

Theory

GitOps is an operational model where the desired state of your entire system (applications and infrastructure) is declared in Git, and an automated agent continuously makes the live system match what's in Git. It applies the principles that made application code management reliable — version control, review, history — to operations.

The four widely-cited GitOps principles (per OpenGitOps):

1. Declarative: the whole system is described declaratively (Kubernetes manifests, Helm, Kustomize).
2. Versioned and immutable: desired state is stored in Git — versioned, with full history and immutability of past states.
3. Pulled automatically: software agents automatically pull the declared state from Git.
4. Continuously reconciled: agents continuously observe actual state and reconcile it toward the desired state, correcting drift.

The payoff: Git becomes the single source of truth, every change is a reviewed/auditable commit (and revertable), deployments are reproducible, and the cluster self-heals toward the declared state. GitOps is essentially Kubernetes' reconciliation loop (Chapter 1) extended to your whole deployment, driven by Git.

Example

Developer --commit/PR--> Git repo (desired state, manifests)
                              |
                  GitOps agent (in cluster) pulls + watches
                              |
                  reconcile: make cluster == Git  (continuously)
                              |
                          Kubernetes cluster (actual state)

Exercises

1. (Beginner) In GitOps, where is the desired state stored?
2. (Beginner) List the four core GitOps principles.
3. (Intermediate) How does GitOps make deployments auditable and revertable?
4. (Interview) How is GitOps essentially an extension of Kubernetes' own reconciliation model, and what does that buy operationally? (Hint: declared desired state + continuous reconcile; self-healing, single source of truth.)

Answers

1. In Git (a Git repository holds the declarative manifests/config that define the system).
2. Declarative; versioned and immutable (in Git); pulled automatically by agents; continuously reconciled toward desired state.
3. Every change to the system is made as a Git commit (typically via pull request), so you get full version history, review/approval, and attribution for who changed what and when (auditability). Reverting is just reverting the commit — the agent then reconciles the cluster back to the prior declared state — making rollbacks simple and reliable.
4. Kubernetes controllers already work by continuously reconciling actual state toward a declared desired state stored in the API. GitOps moves the authoritative desired state out to Git and adds an agent that reconciles the cluster to it. Operationally this buys a single source of truth (Git), self-healing against drift (manual or accidental changes are reverted toward Git), reproducible and auditable deployments (commit history), easy rollback (git revert), and the same robust level-triggered convergence model applied to your whole deployment, not just individual controllers.

---

Git as the single source of truth

Theory

The central tenet of GitOps is that Git is the single source of truth for the desired state of the system. The cluster should reflect exactly what's in Git — nothing more, nothing less. This has powerful consequences for how teams operate:

• All changes go through Git: you deploy by committing/merging, not by running ad-hoc kubectl apply. This gives every change a review (pull request), an approver, a timestamp, and a revert path.
• No manual drift: direct changes to the cluster (someone kubectl edit-ing a Deployment) are considered drift and will be reverted by the agent (or at least detected) — the cluster is not authoritative, Git is.
• Reproducibility & DR: because the entire desired state lives in Git, you can recreate the cluster's workloads from scratch by pointing an agent at the repo — invaluable for disaster recovery and spinning up new clusters.

This requires discipline: emergency manual fixes must be backported to Git, or they'll be reverted. The benefit is a fully auditable, reproducible, review-gated operational model where Git history is your deployment history.

Example

WRONG (imperative, drift-prone):
  $ kubectl edit deployment web      # ad-hoc change, no record, gets reverted

RIGHT (GitOps):
  edit manifest in repo -> open PR -> review/approve -> merge
       -> agent reconciles cluster to match -> change is live + auditable

Exercises

1. (Beginner) In GitOps, how should you make a change to a deployment?
2. (Beginner) What happens to a manual kubectl edit change under strict GitOps?
3. (Intermediate) How does Git-as-source-of-truth help with disaster recovery?
4. (Interview) Why must emergency manual fixes be backported to Git in a GitOps model, and what's the discipline cost/benefit? (Hint: agent reverts drift; auditable consistency vs. process overhead.)

Answers

1. By committing the change to the Git repository (typically via a reviewed pull request that is merged) — not by running ad-hoc commands against the cluster.
2. It's treated as drift: the GitOps agent detects it and (if auto-sync/self-heal is on) reverts the cluster back to match Git, undoing the manual change. Git, not the live cluster, is authoritative.
3. The complete desired state (all manifests/config) lives in Git, so to recover you point a GitOps agent at the repo and it reconciles a fresh cluster to the declared state — recreating all workloads reproducibly without manual rebuilding. Git serves as a versioned, restorable definition of the whole system.
4. Because the agent continuously reconciles the cluster to Git, any manual emergency fix that isn't reflected in Git will be reverted (and lost), reintroducing the incident. So fixes must be committed back to Git to persist. The cost is process discipline (even urgent changes need to land in Git), but the benefit is consistency and auditability: the cluster always matches a reviewed, version-controlled definition, there's no untracked drift, and history remains a complete, trustworthy record of every change.

---

Pull-based vs push-based deployments

Theory

This contrast is the crux of why GitOps differs from traditional CI/CD:

• Push-based (traditional CI/CD): an external system (the CI pipeline) has credentials to the cluster and pushes changes into it (kubectl apply, helm upgrade). The cluster is passive; the pipeline drives.
• Pull-based (GitOps): an agent running inside the cluster continuously pulls the desired state from Git and applies it. The cluster drives its own state; no external system needs cluster write credentials.

The pull model's advantages:

1. Security: cluster credentials never leave the cluster — CI doesn't hold them, shrinking the attack surface (CI only needs write access to Git, not to the cluster). The agent's permissions stay internal.
2. Drift correction: because the agent continuously reconciles (not just on pipeline runs), it detects and fixes drift automatically.
3. Scalability: many clusters can each run an agent pulling from Git, rather than CI pushing to each (better for multi-cluster).

The trade-off: pull-based requires running and operating the agent, and feedback (did my change deploy?) is slightly more indirect. GitOps tools (Argo CD, Flux) implement the pull model.

Example

Push-based (CI/CD):
  CI (holds cluster creds) --kubectl/helm--> Cluster     [creds leave CI; one-shot]

Pull-based (GitOps):
  CI --> Git (commit)
  Cluster's agent --pull--> Git --reconcile--> Cluster   [creds stay in cluster; continuous]

Exercises

1. (Beginner) In a pull-based model, what runs inside the cluster and what does it do?
2. (Beginner) In push-based CD, who holds the cluster credentials?
3. (Intermediate) Give two advantages of the pull model over the push model.
4. (Interview) Why is the pull model more secure and better suited to multi-cluster than the push model? (Hint: credentials stay in-cluster; each cluster self-pulls vs. CI pushing to all.)

Answers

1. A GitOps agent/controller runs in the cluster; it continuously pulls the desired state from Git and reconciles the cluster to match it.
2. The external CI/CD system (the pipeline) holds credentials that allow it to write to the cluster.
3. Any two: better security (cluster credentials stay inside the cluster; CI only needs Git access); automatic, continuous drift detection/correction (not just on pipeline runs); better multi-cluster scalability (each cluster runs its own agent pulling from Git); and natural disaster recovery (agent rebuilds state from Git).
4. Security: in pull mode the agent's cluster credentials never leave the cluster — external CI doesn't hold cluster write access, so compromising CI doesn't directly grant cluster control (CI only writes to Git). Multi-cluster: instead of CI maintaining credentials for and pushing to every cluster (a growing, fragile, high-privilege fan-out), each cluster independently runs an agent that pulls from Git and self-reconciles. This scales cleanly to many clusters, keeps per-cluster credentials local, and means adding a cluster is just deploying another agent pointed at the repo.

---

Reconciliation and drift detection

Theory

The engine of GitOps is continuous reconciliation: the agent repeatedly compares the desired state (Git) with the actual state (cluster) and acts to eliminate any difference. A difference between the two is drift, which arises when someone changes the cluster directly, when a resource is deleted/modified out-of-band, or when Git is updated.

The agent's loop:

1. Observe actual cluster state and fetch desired state from Git.
2. Diff them to detect drift (per-resource: in-sync, out-of-sync, missing, extra).
3. Reconcile — depending on policy, either report the drift or auto-correct it by applying Git's state (self-healing).

This is the same level-triggered reconciliation pattern as Kubernetes controllers (Chapter 1), now spanning Git→cluster. Drift detection gives visibility ("the cluster no longer matches Git"); self-healing (optional) automatically reverts unauthorized changes. Together they keep the live system continuously, verifiably aligned with the reviewed source of truth — catching both accidental changes and unauthorized tampering, and making the cluster's state predictable and auditable at all times.

Example

Reconciliation loop (continuous):
  desired (Git)  vs  actual (cluster)
        |                 |
        +------ diff ------+
                 |
       in-sync? -> do nothing
       drift?   -> report (OutOfSync) and/or auto-apply Git state (self-heal)

# Argo CD reflects this: a change made directly in the cluster shows as OutOfSync,
# and with self-heal enabled is reverted to match Git automatically.

Exercises

1. (Beginner) What is "drift" in GitOps?
2. (Beginner) What are the two possible responses to detected drift?
3. (Intermediate) How is GitOps reconciliation related to Kubernetes controllers' reconciliation?
4. (Interview) What is the difference between drift detection and self-healing, and when might you want detection-only? (Hint: report vs. auto-revert; caution in some environments.)

Answers

1. A difference between the desired state declared in Git and the actual state of the cluster — i.e., the cluster no longer matches what's defined in Git (from out-of-band changes, deletions, or new Git commits not yet applied).
2. Report it (flag the resource as out-of-sync for humans to act on) or automatically correct it by applying the Git-declared state (self-heal).
3. It's the same level-triggered reconciliation pattern: continuously observe actual state, compare to desired state, and act to close the gap. Kubernetes controllers reconcile actual cluster state toward desired state stored in the API; GitOps extends this so the authoritative desired state lives in Git and the agent reconciles the whole cluster toward it.
4. Drift detection observes and reports differences (visibility) without changing anything; self-healing additionally auto-reverts the cluster to match Git. You might want detection-only when you need a human in the loop before changes (e.g., sensitive production where an automatic revert during an active incident or a legitimate emergency hotfix could be harmful), or while initially adopting GitOps and building trust — getting alerts on drift while controlling when reconciliation actually applies. Once confident, enabling self-heal gives automatic enforcement of the source of truth.

---

12.3 Argo CD

Argo CD is the leading GitOps tool, with a rich UI and application model. This subchapter covers its architecture and features.

---

Argo CD architecture

Theory

Argo CD is a declarative, pull-based GitOps continuous delivery tool for Kubernetes (a CNCF graduated project). It runs in the cluster, continuously monitors Git repositories of manifests, and reconciles the cluster to match — with a strong web UI (and CLI) for visualizing application state, sync status, and resource health.

Its main components:

• API Server: serves the UI/CLI/API, handling auth and operations.
• Repository Server: clones Git repos and renders manifests (running Helm/Kustomize/plain YAML) to produce the desired state.
• Application Controller: the reconciliation engine — compares desired (from repo server) vs. live cluster state, reports sync/health status, and applies changes.
• (Plus Redis for caching and Dex for optional SSO.)

Argo CD supports plain manifests, Helm, Kustomize, and jsonnet, and can manage many clusters from one Argo CD instance. Its hallmark is observability of deployments: a visual tree of every resource, its health, and whether it matches Git — making GitOps approachable and debuggable.

Example

        +-----------------------------------------------+
        |                  Argo CD                       |
  UI/CLI->| API Server |  | Repo Server (renders Git) |  |
        |               | Application Controller        |--> target clusters
        +-----------------------------------------------+
                  ^                                  |
                  | watch/clone                      | reconcile
                Git repos                       Kubernetes cluster(s)

argocd app list                       # list managed applications
argocd app get web                    # show sync + health status
argocd app sync web                   # trigger a sync to match Git

Exercises

1. (Beginner) Where does Argo CD run, and what does it continuously do?
2. (Beginner) Which Argo CD component renders manifests from Git (Helm/Kustomize/YAML)?
3. (Intermediate) What does the Application Controller do?
4. (Interview) What distinguishes Argo CD's user experience, and why is deployment observability valuable in GitOps? (Hint: visual resource tree, sync/health status, debuggability.)

Answers

1. It runs inside the Kubernetes cluster and continuously monitors Git repositories, reconciling the cluster's state to match the declared desired state in Git.
2. The Repository Server (it clones the repos and renders Helm/Kustomize/plain YAML/jsonnet into manifests).
3. It's the reconciliation engine: it compares the desired state (rendered manifests from the repo server) against the live cluster state, determines sync and health status of each resource, and applies changes to converge the cluster toward Git (per the configured sync policy).
4. Argo CD provides a rich web UI (and CLI/API) showing a visual tree of every application's resources with their health and sync status (in-sync/out-of-sync vs. Git). This observability makes GitOps approachable and debuggable — you can immediately see what's deployed, whether it matches Git, what drifted, and why something is unhealthy, and trigger/inspect syncs. In a model where the cluster self-reconciles, this visibility is crucial for understanding and trusting what the system is doing.

---

Application and AppProject resources

Theory

Argo CD is configured through its own CRDs, chiefly Application and AppProject:

• An Application is the core unit: it declares what to deploy and where. It points to a source (a Git repo, path, and revision — plus Helm/Kustomize config) and a destination (a target cluster and namespace), and it carries the sync policy. Argo CD then keeps that destination in sync with that source.
• An AppProject is a grouping and governance boundary for Applications. It restricts what Applications in the project may do: which source repos are allowed, which destination clusters/namespaces are permitted, and which resource kinds can be deployed — plus project-scoped RBAC roles. It's how you enforce multi-tenancy and guardrails (e.g., team A's apps can only deploy to team A's namespaces from team A's repos).

Together: Applications define individual deployments; AppProjects bound and govern groups of them. (The app-of-apps pattern and ApplicationSet, covered in Chapter 15, scale this to many apps/clusters.)

Example

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata: { name: web, namespace: argocd }
spec:
  project: team-a
  source:
    repoURL: https://github.com/org/team-a-config
    path: apps/web/overlays/prod
    targetRevision: main
  destination:
    server: https://kubernetes.default.svc
    namespace: web
  syncPolicy:
    automated: { prune: true, selfHeal: true }
---

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata: { name: team-a, namespace: argocd }
spec:
  sourceRepos: [ "https://github.com/org/team-a-config" ]   # allowed repos
  destinations:
  - { server: https://kubernetes.default.svc, namespace: "web*" }  # allowed targets

Exercises

1. (Beginner) What does an Argo CD Application declare?
2. (Beginner) What is the purpose of an AppProject?
3. (Intermediate) Name three things an AppProject can restrict.
4. (Interview) How do Application and AppProject together enable safe multi-tenancy in a shared Argo CD instance? (Hint: apps define deployments; projects bound allowed repos/destinations/kinds + RBAC.)

Answers

1. What to deploy and where: a source (Git repo, path, revision, and Helm/Kustomize settings) and a destination (target cluster + namespace), plus the sync policy. Argo CD keeps that destination in sync with that source.
2. To group Applications and act as a governance/security boundary — constraining and applying RBAC to the Applications within it (enabling multi-tenancy and guardrails).
3. Any three: which source repositories are allowed, which destination clusters/namespaces are permitted, which resource kinds may be deployed (allow/deny lists), and project-scoped RBAC roles/permissions.
4. In a shared Argo CD instance, each team's Applications belong to an AppProject that limits them to that team's allowed repos, destination clusters/namespaces, and resource kinds, with project-scoped RBAC controlling who can manage them. So even though many teams share one Argo CD, an Application can only deploy approved content from approved sources to approved targets — preventing one tenant from deploying to another's namespace, pulling from arbitrary repos, or creating disallowed resources. Applications provide the per-deployment definition; AppProjects provide the enforced boundaries around groups of them.

---

Sync policies and auto-sync

Theory

A sync policy governs how Argo CD applies Git changes to the cluster. The key choice is manual vs. automated sync:

• Manual sync: Argo CD detects drift/changes and marks the app OutOfSync, but waits for a human to trigger the sync (via UI/CLI). Good for gated production changes.
• Automated sync: Argo CD applies changes automatically when Git changes. Two important sub-options:
  • selfHeal: true — also revert manual cluster changes back to Git (drift correction).
  • prune: true — delete cluster resources that were removed from Git (without prune, deletions in Git aren't reflected — orphaned resources linger).

Additional features: sync waves (annotations to order resource application — e.g., CRDs/namespaces first), sync hooks (PreSync/PostSync jobs like migrations), and sync options (e.g., create namespaces, server-side apply). Choosing the policy is a balance: automated+selfHeal+prune gives full hands-off GitOps enforcement; manual gives control at the cost of automation. Many teams use automated sync for non-prod and gated/manual or PR-driven promotion for prod.

Example

spec:
  syncPolicy:
    automated:
      prune: true            # delete resources removed from Git
      selfHeal: true         # revert out-of-band cluster changes
    syncOptions:
    - CreateNamespace=true
    - ApplyOutOfSyncOnly=true
---

# Sync waves order resources (lower waves first):
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"   # e.g., apply CRDs before workloads

Exercises

1. (Beginner) What is the difference between manual and automated sync?
2. (Beginner) What does prune: true do?
3. (Intermediate) What does selfHeal: true add, and how does it relate to drift?
4. (Interview) Why might a team use automated sync in non-prod but manual/gated sync in prod, and what do sync waves solve? (Hint: control vs. automation; ordering dependent resources.)

Answers

1. Manual sync detects changes/drift and marks the app OutOfSync but waits for a human to trigger applying them; automated sync applies Git changes to the cluster automatically without manual intervention.
2. It deletes cluster resources that have been removed from Git, so the cluster doesn't accumulate orphaned resources that no longer exist in the source of truth.
3. selfHeal: true makes Argo CD automatically revert out-of-band cluster changes (drift) back to the state declared in Git — so manual kubectl edits are undone and the cluster always matches Git, not just on Git changes but on cluster changes too.
4. Non-prod benefits from full automation (fast, hands-off deploys; quick iteration), so automated sync is convenient and low-risk there. Production often needs a human gate/approval (or PR-based promotion) before changes go live, to control timing and reduce risk, so manual or gated sync is preferred. Sync waves solve ordering: they let you control the sequence in which resources are applied (e.g., CRDs, namespaces, and databases before the workloads that depend on them), preventing failures from applying dependent resources before their prerequisites exist.

---

Multi-cluster deployments with Argo CD

Theory

A single Argo CD instance can manage deployments to many clusters — a common "hub-and-spoke" pattern where one central Argo CD (the hub) deploys to multiple workload clusters (spokes). You register external clusters with Argo CD (storing their credentials as Secrets), and then an Application's destination.server targets a specific registered cluster.

This centralizes GitOps across a fleet: one place to see and control all deployments everywhere. Combined with ApplicationSet (Chapter 15), you can template one Application across many clusters automatically (e.g., "deploy this app to every cluster labeled env=prod"). Considerations: the hub Argo CD holds credentials to all managed clusters (a high-value target to secure), and you must decide between one central instance (simpler oversight, bigger blast radius) versus an Argo CD per cluster (more isolation, more to operate). Multi-cluster Argo CD is a foundational pattern for managing many environments/regions consistently from Git.

Example

# Register an external cluster with Argo CD (stores its creds):
argocd cluster add prod-eu-context

# An Application targeting that specific cluster:
#   destination:
#     server: https://prod-eu.example.com   # the registered cluster's API
#     namespace: web
argocd app create web-eu --dest-server https://prod-eu.example.com \
  --repo https://github.com/org/config --path apps/web --dest-namespace web

Exercises

1. (Beginner) Can one Argo CD instance deploy to multiple clusters?
2. (Beginner) What does an Application's destination.server specify?
3. (Intermediate) What is the "hub-and-spoke" Argo CD pattern?
4. (Interview) What is the security/operational trade-off between one central Argo CD managing all clusters versus one Argo CD per cluster? (Hint: blast radius of stored creds vs. isolation and operational overhead.)

Answers

1. Yes — a single Argo CD instance can manage and deploy to many registered clusters.
2. The target cluster's API server (which registered cluster the Application deploys to), alongside the target namespace.
3. A central Argo CD instance (the hub) manages deployments to multiple workload clusters (the spokes) from one place, providing unified GitOps control and visibility across the fleet.
4. A central (hub) Argo CD must store credentials to every managed cluster, so compromising it endangers all clusters (large blast radius), though it offers single-pane oversight and less operational overhead. One Argo CD per cluster isolates credentials and failure/compromise to that cluster (smaller blast radius, stronger tenancy) but means operating and upgrading many Argo CD instances and losing the single unified view. The choice balances centralized convenience/visibility against isolation and the security risk of concentrating fleet-wide credentials.

---

Argo CD RBAC and SSO

Theory

Because Argo CD can deploy to production across clusters, controlling who can do what within Argo CD is critical. Argo CD has its own RBAC layer (separate from Kubernetes RBAC) governing actions on Argo CD objects: who can view, sync, create, or delete Applications, scoped by project. Policies are defined in a CSV-like format mapping subjects/roles to permissions (e.g., role team-a may sync apps in project team-a but only get others).

For identity, Argo CD integrates SSO via Dex (a bundled OIDC broker) or directly with an OIDC provider (Okta, Google, Azure AD). Users log in with corporate SSO, and their groups map to Argo CD RBAC roles — so you manage access centrally and grant least-privilege per team/project. This two-layer model matters: Argo CD RBAC controls operations in Argo CD (e.g., who may trigger a prod sync), while the underlying cluster RBAC still governs what Argo CD's service account can do in the cluster. Securing both is essential since Argo CD is effectively a high-privilege deployment gateway.

Example

# Argo CD RBAC policy (policy.csv) — map SSO groups to scoped permissions
p, role:team-a, applications, sync, team-a/*, allow
p, role:team-a, applications, get,  */*,      allow
g, team-a-oidc-group, role:team-a        # SSO group -> Argo CD role

# SSO via OIDC (argocd-cm) — users authenticate with corporate IdP
oidc.config: |
  name: Okta
  issuer: https://example.okta.com
  clientID: argocd
  requestedScopes: ["openid","profile","email","groups"]

Exercises

1. (Beginner) Is Argo CD's RBAC the same as Kubernetes RBAC?
2. (Beginner) What does Argo CD use to integrate corporate SSO?
3. (Intermediate) How do SSO groups connect to Argo CD permissions?
4. (Interview) Why must you secure both Argo CD's own RBAC and the cluster RBAC of Argo CD's service account? (Hint: Argo CD RBAC = who can operate Argo CD; cluster RBAC = what Argo CD can do in the cluster.)

Answers

1. No — Argo CD has its own RBAC layer governing actions on Argo CD objects (Applications, projects, syncs), separate from (and in addition to) Kubernetes RBAC.
2. Dex (a bundled OIDC broker) or a direct OIDC integration with an external identity provider (Okta, Google, Azure AD, etc.).
3. Users authenticate via SSO/OIDC, which provides their group memberships; Argo CD RBAC policies map those groups (g, <group>, role:<role>) to roles with scoped permissions (e.g., sync apps within a specific project), so access is granted per team/project based on central group membership.
4. They control different things. Argo CD RBAC governs who can perform which operations within Argo CD (e.g., who may sync or delete a production Application) — protecting the deployment control plane. Cluster RBAC governs what Argo CD's service account is permitted to do in the target cluster (what resources it can create/modify). If Argo CD RBAC is weak, an unauthorized user could trigger powerful deployments; if Argo CD's cluster RBAC is over-broad, a compromise of Argo CD (or a malicious sync) could do anything in the cluster. Since Argo CD is effectively a high-privilege gateway to production, both layers must be locked down with least privilege.

---

12.4 Flux CD

Flux is the other major GitOps tool — a set of composable controllers. This subchapter covers Flux v2.

---

Flux v2 architecture

Theory

Flux (Flux v2, "Flux CD") is the other leading CNCF GitOps tool. Unlike Argo CD's more monolithic, UI-centric design, Flux v2 is built as a set of composable controllers called the GitOps Toolkit, each handling one concern and exposed via its own CRDs. This makes Flux modular, API-driven, and a natural building block for higher-level platforms (it's notably more "Kubernetes-native CLI/CRD"-oriented and ships without a built-in UI, though third-party UIs exist).

The GitOps Toolkit controllers:

• Source Controller: fetches artifacts from sources (Git repos, Helm repos, OCI, buckets) and exposes them internally.
• Kustomize Controller: builds and applies Kustomize/plain-YAML from a source, with reconciliation/health/pruning.
• Helm Controller: manages Helm releases declaratively (via HelmRelease).
• Notification Controller: handles inbound events (webhooks to trigger syncs) and outbound alerts (Slack, etc.).
• Image Automation Controllers: update Git with new image versions (covered later).

You bootstrap Flux into a cluster (flux bootstrap), which installs these controllers and configures them to manage themselves from Git. The philosophy: small, focused, interoperable controllers you compose, versus a single integrated application.

Example

GitOps Toolkit controllers (each a CRD-driven concern):
  Source Controller   -> GitRepository / HelmRepository / OCIRepository
  Kustomize Ctrl      -> Kustomization
  Helm Controller     -> HelmRelease
  Notification Ctrl   -> Receiver / Alert / Provider
  Image Automation    -> ImageRepository / ImagePolicy / ImageUpdateAutomation

flux bootstrap github --owner=org --repository=fleet --path=clusters/prod
flux get kustomizations           # status of reconciliations

Exercises

1. (Beginner) How is Flux v2's architecture structured, in contrast to Argo CD?
2. (Beginner) Name three GitOps Toolkit controllers and their concerns.
3. (Intermediate) What does flux bootstrap do?
4. (Interview) What are the implications of Flux's composable, CRD-driven, UI-less design for platform builders versus Argo CD's integrated UI approach? (Hint: modularity/automation/building-block vs. out-of-box visibility.)

Answers

1. Flux v2 is a set of small, composable, single-purpose controllers (the GitOps Toolkit), each with its own CRDs, rather than a more monolithic, UI-centric application like Argo CD.
2. Any three: Source Controller (fetch artifacts from Git/Helm/OCI/bucket sources), Kustomize Controller (build/apply Kustomize/YAML), Helm Controller (manage Helm releases via HelmRelease), Notification Controller (inbound webhooks/outbound alerts), Image Automation controllers (update image versions in Git).
3. It installs the Flux controllers into the cluster and configures them to manage the cluster (and themselves) from a Git repository/path — committing Flux's own manifests to Git so Flux is itself managed via GitOps from the start.
4. Flux's modular, CRD/CLI-driven, UI-less design makes it highly composable and automatable — ideal as a building block embedded in larger platforms/products, scriptable and API-native, with each concern independently usable. The trade-off is no built-in graphical visibility out of the box (you rely on CLI/CRDs or third-party UIs). Argo CD's integrated UI gives immediate, approachable deployment observability and operations for end users, at the cost of being a more monolithic, opinionated application. Platform builders often favor Flux for embedding/automation; teams wanting turnkey visibility/self-service often favor Argo CD.

---

GitRepository, Kustomization, HelmRelease resources

Theory

Flux's declarative model centers on a few CRDs that compose source + apply:

• GitRepository (a source): defines where to fetch from — a Git repo URL, branch/tag/semver, and interval. The Source Controller keeps an internal artifact of that repo up to date. (Siblings: HelmRepository, OCIRepository, Bucket.)
• Kustomization (an apply): references a source and a path within it, and tells the Kustomize Controller to build and apply those manifests to the cluster, with interval, prune, health checks, and dependency ordering (dependsOn).
• HelmRelease (an apply for Helm): declaratively manages a Helm release — referencing a chart (from a HelmRepository/OCI/GitRepository source) plus values — handled by the Helm Controller, which performs installs/upgrades/rollbacks.

The pattern is separation of "where to get it" (source) from "what to do with it" (Kustomization/HelmRelease). One GitRepository source can feed many Kustomizations/HelmReleases. This composability — plus dependsOn for ordering and health gating — is how Flux expresses an entire fleet's desired state declaratively.

Example

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata: { name: fleet, namespace: flux-system }
spec:
  url: https://github.com/org/fleet
  ref: { branch: main }
  interval: 1m
---

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata: { name: web, namespace: flux-system }
spec:
  sourceRef: { kind: GitRepository, name: fleet }
  path: ./apps/web/overlays/prod
  prune: true                      # delete resources removed from Git
  interval: 5m
  dependsOn: [ { name: infra } ]   # order: apply infra first

Exercises

1. (Beginner) What does a GitRepository resource define?
2. (Beginner) What does a Kustomization resource do with a source?
3. (Intermediate) How does Flux separate "where to get manifests" from "what to do with them"?
4. (Interview) How do dependsOn and prune on a Kustomization contribute to safe, ordered reconciliation? (Hint: ordering dependencies; removing deleted resources.)

Answers

1. A source: where to fetch manifests from — a Git repo URL, the ref (branch/tag/semver), and the poll interval; the Source Controller maintains an up-to-date artifact of it.
2. It references a source and a path within it, and instructs the Kustomize Controller to build (kustomize) and apply those manifests to the cluster, reconciling on an interval with optional pruning, health checks, and ordering.
3. Source CRDs (GitRepository/HelmRepository/OCIRepository/Bucket) define where to fetch artifacts; apply CRDs (Kustomization/HelmRelease) reference a source and define what to build/apply and how. One source can be consumed by many Kustomizations/HelmReleases, cleanly decoupling fetching from applying.
4. dependsOn enforces ordering: a Kustomization waits for its dependencies to be successfully applied and healthy before reconciling, so prerequisites (CRDs, namespaces, infrastructure) come up before dependent workloads — preventing failures from missing dependencies. prune: true removes from the cluster any resources that were deleted from Git, keeping actual state aligned with the source of truth and avoiding orphaned leftovers. Together they make reconciliation both correctly ordered and accurately convergent (no stale resources).

---

Image automation with Flux

Theory

A distinctive Flux capability is image update automation: Flux can watch a container registry for new image versions and automatically commit the updated image tag back to Git — closing the loop so that new builds get deployed without manual manifest edits, while still keeping Git as the source of truth (the change is a commit, fully auditable). Argo CD has no built-in equivalent (it's typically paired with external tools like Argo CD Image Updater).

Three CRDs drive it:

• ImageRepository: scans a registry for available tags of an image.
• ImagePolicy: selects which tag is "latest desired" via a policy (semver range, numerical, regex/filter) — e.g., "highest 1.x semver" or "newest by build timestamp".
• ImageUpdateAutomation: writes the selected tag back into the Git manifests (via markers in the YAML) and commits/pushes.

This enables a fully automated path: CI pushes a new image → Flux detects it via ImageRepository/ImagePolicy → ImageUpdateAutomation commits the new tag to Git → the Kustomize/Helm controller deploys it. It's GitOps-native CD that keeps every deployment recorded as a Git commit.

Example

apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImagePolicy
metadata: { name: web, namespace: flux-system }
spec:
  imageRepositoryRef: { name: web }
  policy: { semver: { range: ">=1.0.0 <2.0.0" } }   # pick highest 1.x

# In the deployment manifest, a marker tells Flux what to update:
spec:
  template:
    spec:
      containers:
      - name: web
        image: registry.example.com/web:1.4.2 # {"$imagepolicy": "flux-system:web"}

Exercises

1. (Beginner) What does Flux's image automation do when a new image version appears?
2. (Beginner) Which CRD decides which image tag is the desired one?
3. (Intermediate) Why does writing the new tag back to Git (rather than directly to the cluster) preserve GitOps principles?
4. (Interview) Trace the fully automated flow from a new image build to it running in the cluster using Flux image automation. (Hint: CI push → ImageRepository → ImagePolicy → ImageUpdateAutomation commit → Kustomize/Helm deploy.)

Answers

1. It automatically updates the image tag in the Git manifests (commits the change to Git), so the new version gets deployed without manual edits.
2. ImagePolicy (it applies a policy like a semver range, numeric, or filter to the tags discovered by ImageRepository to select the desired tag).
3. Because the change is made as a Git commit to the manifests, Git remains the single source of truth — the update is versioned, reviewable, auditable, and revertable, and the cluster is still reconciled from Git. Writing directly to the cluster would bypass Git, creating drift and losing the audit/rollback guarantees; committing to Git keeps the whole flow GitOps-compliant.
4. CI builds and pushes a new image tag to the registry → Flux's ImageRepository scans the registry and discovers the new tag → ImagePolicy selects it as the desired version per its policy → ImageUpdateAutomation writes that tag into the Git manifests (at the marked field) and commits/pushes to Git → the Source Controller picks up the new commit and the Kustomize/Helm Controller applies it, deploying the new image to the cluster — all automatically, with the deployment recorded as a Git commit.

---

Multi-tenancy with Flux

Theory

Flux supports multi-tenancy so multiple teams can safely share a cluster (or fleet) under GitOps, each managing their own apps without interfering with others. The model leans on Kubernetes-native primitives plus Flux's structure:

• Namespace isolation + RBAC: each tenant's Flux resources (GitRepository, Kustomization, HelmRelease) live in their own namespace, and Flux applies their manifests using a tenant-scoped ServiceAccount (via spec.serviceAccountName / impersonation), so a tenant's reconciliation can only do what that ServiceAccount's RBAC allows — confining their blast radius.
• Repository structure: a common pattern is a platform/fleet repo (managed by the platform team) that defines each tenant and points to per-tenant repos the tenants control. The platform Kustomization sets up tenants; tenant Kustomizations reconcile their own apps.
• Source restrictions & policies: limit which sources tenants may use, and combine with admission policies (Kyverno/Gatekeeper) for guardrails.

The key mechanism is impersonation/least-privilege ServiceAccounts: Flux doesn't reconcile every tenant with cluster-admin; it reconciles each tenant as a constrained identity, enforcing isolation through Kubernetes RBAC. This makes Flux multi-tenancy as strong as the underlying RBAC.

Example

# A tenant's Kustomization reconciles with a least-privilege ServiceAccount:
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata: { name: team-a-apps, namespace: team-a }
spec:
  serviceAccountName: team-a-reconciler   # impersonate this SA (RBAC-limited)
  sourceRef: { kind: GitRepository, name: team-a-repo, namespace: team-a }
  path: ./apps
  prune: true
  targetNamespace: team-a                 # confined to the tenant's namespace

Exercises

1. (Beginner) What two Kubernetes-native primitives underpin Flux multi-tenancy?
2. (Beginner) What does setting serviceAccountName on a Kustomization achieve?
3. (Intermediate) Describe the platform-repo + per-tenant-repo structure.
4. (Interview) Why is reconciling each tenant with a least-privilege ServiceAccount (impersonation) the cornerstone of Flux multi-tenancy security? (Hint: confines what each tenant's reconciliation can do via RBAC; no shared cluster-admin.)

Answers

1. Namespaces (isolation per tenant) and RBAC (limiting each tenant's permissions), combined with tenant-scoped ServiceAccounts.
2. It makes Flux apply that Kustomization's resources as that ServiceAccount (impersonation), so the reconciliation is constrained to what the SA's RBAC permits — preventing a tenant from creating resources or affecting namespaces beyond their grant.
3. A central platform/fleet repository, controlled by the platform team, defines the tenants and bootstraps a Flux setup (Kustomizations/sources) for each, pointing at per-tenant repositories that the individual teams own and control. The platform repo governs structure and guardrails; tenants manage their own application manifests in their own repos, which Flux reconciles within their scoped boundaries.
4. If Flux reconciled all tenants with a single highly-privileged (e.g., cluster-admin) identity, any tenant's manifests could affect the whole cluster, breaking isolation. By reconciling each tenant as a least-privilege ServiceAccount scoped to their namespace(s) and resources, Flux ensures a tenant's reconciliation can only do what Kubernetes RBAC explicitly allows that identity — so a malicious or mistaken manifest from one tenant cannot touch another's resources or escalate. The strength of the isolation reduces to the strength of that RBAC, making impersonation the security cornerstone of Flux multi-tenancy.

---