11. Helm and Package Management

Raw YAML manifests don't scale: you end up copying near-identical files per environment, with no versioning, templating, or easy rollback. Helm is the de facto package manager for Kubernetes, bundling related manifests into versioned, parameterized charts. Kustomize offers a template-free alternative. This chapter covers both, from consuming charts to authoring and publishing your own.

---

11.1 Helm Fundamentals

This subchapter introduces Helm's architecture and core vocabulary — charts, releases, repositories — and how it compares to Kustomize.

---

Helm architecture and components

Theory

Helm is "the package manager for Kubernetes." Just as apt or npm install packaged software with dependencies, Helm installs charts — packages of templated Kubernetes manifests — into a cluster. It solves the problems of raw YAML: it templates manifests so one chart serves many configurations, versions deployments, tracks them as releases, and supports upgrade/rollback.

A vital historical note: Helm 3 (current) is client-only — there is no server-side component. Helm 2 had a privileged in-cluster server called Tiller, which was a security liability (it held broad cluster permissions); Helm 3 removed it entirely. Now the helm CLI talks directly to the Kubernetes API using your kubeconfig credentials (so it respects your RBAC), and stores release state as Secrets (by default) in the cluster namespace. The architecture is therefore simple: a CLI + chart packages + release metadata stored in-cluster.

Example

Helm 2 (old):  helm CLI --> Tiller (in-cluster, privileged) --> API server   [removed]
Helm 3 (now):  helm CLI ----------------------------------------> API server
                 (uses your kubeconfig + RBAC; release state stored as Secrets)

helm version            # v3.x — client only
helm install web ./mychart        # render chart + apply to cluster as release "web"
kubectl get secret -l owner=helm  # release history stored as Secrets

Exercises

1. (Beginner) What is Helm, by analogy to other ecosystems?
2. (Beginner) What major component did Helm 3 remove from Helm 2, and why?
3. (Intermediate) In Helm 3, how does the CLI authenticate to the cluster, and where is release state stored?
4. (Interview) Why was removing Tiller a significant security improvement, and how does Helm 3 respect existing RBAC? (Hint: no privileged in-cluster server; client uses user credentials.)

Answers

1. Helm is the package manager for Kubernetes — analogous to apt/yum/npm — installing packaged, versioned, parameterized applications (charts) into a cluster.
2. Tiller, the privileged in-cluster server component, because it was a security liability (it typically ran with broad cluster permissions, creating a powerful attack target and privilege-escalation path).
3. The helm CLI uses your kubeconfig credentials to talk directly to the Kubernetes API (so actions are bound by your RBAC). Release state/history is stored in-cluster, by default as Secrets in the release's namespace.
4. Tiller ran in-cluster with wide-ranging permissions, so anyone who could reach it could effectively perform privileged actions regardless of their own RBAC — a major risk. Helm 3 removes Tiller and makes the client act directly as the user: every operation goes through the API server with the user's own kubeconfig/RBAC, so Helm can do exactly what the user is permitted to do — no more, no less. This eliminates the privileged server and aligns Helm's access with standard Kubernetes authorization.

---

Charts, releases, and repositories

Theory

Three core Helm nouns:

• Chart: a package — a directory (or .tgz) of templated manifests plus metadata and default values. It's the definition of an application (like an installer or a recipe). Charts are versioned.
• Release: an instance of a chart deployed to a cluster, with a name and a specific configuration. Installing the same chart twice (different names) yields two independent releases, each with its own version history (revision 1, 2, 3...). This is what enables upgrades and rollbacks.
• Repository: a place where charts are stored and shared — an HTTP server hosting an index of charts (or, increasingly, an OCI registry). You add repos and pull charts from them (e.g., Bitnami's repo).

The relationship: a chart (from a repository) is installed to create a release (a running, versioned instance) in your cluster. This mirrors package managers: a package from a repository becomes an installed instance. Artifact Hub is the central index for discovering public charts.

Example

helm repo add bitnami https://charts.bitnami.com/bitnami   # add a repository
helm repo update
helm search repo bitnami/postgresql                        # find a chart

helm install db1 bitnami/postgresql      # release "db1" from the chart
helm install db2 bitnami/postgresql      # release "db2" — independent instance
helm list                                # show releases and their revisions
helm history db1                         # revision history of release db1

Exercises

1. (Beginner) Define chart, release, and repository in one phrase each.
2. (Beginner) Can you install the same chart multiple times in one cluster? What distinguishes the instances?
3. (Intermediate) What enables Helm's upgrade and rollback capability at the release level?
4. (Interview) Map Helm's chart/release/repository model onto a traditional package manager and explain why this mental model is useful. (Hint: package/installed-instance/repo.)

Answers

1. Chart = a versioned package of templated Kubernetes manifests (the app definition); Release = a named, configured instance of a chart deployed in a cluster; Repository = a store/index where charts are published and fetched.
2. Yes — install it under different release names; each release is an independent instance with its own configuration and revision history.
3. Each release maintains a revision history (Helm stores the rendered manifests/values for each revision as Secrets). Upgrades create a new revision; rollback re-applies a prior stored revision — so versioned release state is what makes upgrade/rollback possible.
4. It maps directly: a repository hosts charts (packages), and installing a chart produces a release (an installed instance) — just like a repo hosts packages that you install as running software, e.g., apt's repo → package → installed program. This is useful because it brings familiar package-management concepts (discovery, versioning, install, upgrade, rollback, multiple instances) to Kubernetes deployments, so practitioners can reason about app lifecycle the same way they do with OS/language package managers.

---

Installing and managing Helm

Theory

Helm is a single CLI binary you install on your workstation/CI (like kubectl). Once installed, the core lifecycle commands are few and consistent:

• helm repo add/update — register and refresh chart repositories.
• helm search — find charts (repo or Artifact Hub).
• helm install <release> <chart> — deploy a chart as a named release.
• helm upgrade <release> <chart> — apply changes/new versions (with --install to create-or-upgrade idempotently).
• helm rollback <release> <revision> — revert to a previous revision.
• helm uninstall <release> — remove a release.
• helm list / helm status / helm history — inspect releases.

Two indispensable habits for safety: helm template renders a chart to plain YAML locally (to inspect what would be applied), and --dry-run --debug simulates an install/upgrade against the API server without changing anything. helm get values/manifest shows what's currently deployed. These let you validate changes before they hit a cluster.

Example

helm install web ./mychart -n prod --create-namespace \
  -f values-prod.yaml --set image.tag=2.0     # install with value overrides

helm upgrade --install web ./mychart -n prod -f values-prod.yaml   # create-or-upgrade
helm rollback web 2 -n prod                    # revert to revision 2

# Validate before applying:
helm template web ./mychart -f values-prod.yaml         # render to YAML locally
helm upgrade web ./mychart --dry-run --debug -n prod    # simulate, no changes

Exercises

1. (Beginner) Which command deploys a chart as a named release?
2. (Beginner) What does helm upgrade --install do that plain helm upgrade does not?
3. (Intermediate) How can you see the YAML a chart will produce without applying it?
4. (Interview) Why are helm template and --dry-run important habits in a production workflow? (Hint: validate/inspect rendered output before changing the cluster.)

Answers

1. helm install <release-name> <chart>.
2. --install makes it idempotent create-or-upgrade: it installs the release if it doesn't exist yet, or upgrades it if it does — whereas plain helm upgrade fails if the release doesn't already exist. This is ideal for CI/CD.
3. Run helm template <release> <chart> [-f values.yaml], which renders the chart's templates to plain Kubernetes YAML locally without contacting the cluster (or helm install/upgrade --dry-run to render via the API server).
4. They let you inspect and validate exactly what will be applied before it touches a live cluster: helm template renders the final manifests locally (catch templating errors, review diffs, run through policy/linting), and --dry-run --debug simulates the operation against the API server (validating against admission/schema) without making changes. This prevents surprises — misrendered values, accidental deletions, invalid resources — from reaching production.

---

Helm vs Kustomize

Theory

Helm and Kustomize are the two dominant configuration-management tools, with fundamentally different philosophies:

• Helm uses templating: charts contain Go-template placeholders filled from values.yaml. It's a full package manager with versioning, releases, dependencies, rollback, and a sharing ecosystem. Powerful, but templated YAML can become complex and hard to read, and it's not valid YAML until rendered.
• Kustomize uses overlays/patching: you start with plain, valid base YAML and apply patches for each variant (environment) — no templating language. It's built into kubectl (kubectl apply -k), keeps manifests as real YAML, and is simpler for straightforward per-environment customization. But it lacks packaging, versioning, dependencies, and a distribution ecosystem.

Rule of thumb: Helm when you need to package and distribute applications (especially third-party software) with parameters and lifecycle management; Kustomize when you want to customize your own plain manifests across environments without a templating language. They're not mutually exclusive — you can render a Helm chart and post-process with Kustomize, and tools like Argo CD/Flux support both.

Example

Aspect	Helm	Kustomize
Mechanism	Go templating + values	Overlays/patches on plain YAML
Packaging/versioning	Yes (charts, releases)	No
Dependencies	Yes (subcharts)	No
Rollback	Yes (release history)	No (rely on Git/kubectl)
Built into kubectl	No	Yes (apply -k)
Best for	Packaging/distributing apps	Customizing own manifests per env

Exercises

1. (Beginner) What is the core mechanism Helm uses versus Kustomize?
2. (Beginner) Which of the two is built into kubectl?
3. (Intermediate) Give one capability Helm has that Kustomize lacks, and one advantage Kustomize has over Helm.
4. (Interview) When would you choose Helm over Kustomize, and can they be used together? (Hint: packaging/distribution vs. plain-YAML overlays; render Helm then patch.)

Answers

1. Helm uses Go-template-based templating driven by a values file; Kustomize uses overlays and patches applied to plain, valid base YAML (no templating language).
2. Kustomize (kubectl apply -k / kubectl kustomize).
3. Helm has packaging/versioning, releases, dependencies (subcharts), and built-in rollback — which Kustomize lacks. Kustomize keeps manifests as plain, valid YAML (easier to read/diff, no templating complexity) and is built into kubectl. (Either direction acceptable.)
4. Choose Helm when you need to package, version, parameterize, and distribute an application — especially installing/maintaining third-party software with dependencies and lifecycle (upgrade/rollback). Choose Kustomize for customizing your own plain manifests across environments without templating. They can be combined: e.g., render a Helm chart (helm template) and then apply Kustomize patches on top, and GitOps tools (Argo CD, Flux) natively support both, even Helm charts post-rendered with Kustomize.

---

11.2 Working with Charts

This subchapter goes inside charts: their structure, values, the templating language, and lifecycle hooks.

---

Chart structure and anatomy

Theory

A Helm chart is a directory with a conventional structure. The key files and folders:

mychart/
  Chart.yaml          # chart metadata: name, version, appVersion, dependencies
  values.yaml         # default configuration values
  templates/          # templated Kubernetes manifests
    deployment.yaml
    service.yaml
    _helpers.tpl      # reusable named template snippets (helpers)
    NOTES.txt         # message shown to the user after install
  charts/             # subcharts (dependencies) live here
  crds/               # CRDs installed before other resources
  .helmignore         # files to exclude from packaging

The two most important are Chart.yaml (identifies the chart and its versions: version is the chart's own version, appVersion is the app it deploys) and values.yaml (the default parameters users override). Everything in templates/ is rendered through Go templating with the values, then applied. Understanding this layout is the foundation for both using and authoring charts.

Example

# Chart.yaml
apiVersion: v2
name: mychart
version: 1.2.0          # the CHART version (bump on chart changes)
appVersion: "2.4.1"     # the version of the APP being deployed
description: My web app
dependencies:
- name: postgresql
  version: "15.x.x"
  repository: https://charts.bitnami.com/bitnami

Exercises

1. (Beginner) What two files are the most important in a chart, and what does each hold?
2. (Beginner) What goes in the templates/ directory?
3. (Intermediate) What is the difference between version and appVersion in Chart.yaml?
4. (Interview) What is the purpose of the crds/ directory and why does Helm treat CRDs specially? (Hint: CRDs must exist before CRs that use them; installed first, limited lifecycle handling.)

Answers

1. Chart.yaml (chart metadata: name, versions, dependencies) and values.yaml (default configuration values users can override).
2. The templated Kubernetes manifests (Deployments, Services, etc.), plus helpers (_helpers.tpl) and NOTES.txt — rendered through Go templating with the values and applied to the cluster.
3. version is the chart's own package version (bump it whenever the chart changes); appVersion is the version of the application the chart deploys (e.g., the container image's app version). They're independent — you can release a new chart version without changing the app, and vice versa.
4. The crds/ directory holds CustomResourceDefinitions that Helm installs before the rest of the chart's resources, so that custom resources relying on them can be created. Helm treats them specially because CRDs are cluster-scoped, foundational schema that must exist first; to avoid race conditions and accidental data loss, Helm installs CRDs from this directory up front and does not manage their upgrade/deletion through the normal templating lifecycle (CRD updates/removal must be handled deliberately).

---

values.yaml and value overrides

Theory

values.yaml holds a chart's default configuration as structured YAML; templates reference these values (e.g., {{ .Values.image.tag }}). The whole point is parameterization: the same chart produces different deployments by supplying different values — this is how one chart serves dev, staging, and prod.

Values can be overridden at install/upgrade time, and there's a clear precedence (later wins):

1. The chart's own values.yaml (defaults).
2. A parent chart's values (for subcharts).
3. User-supplied -f myvalues.yaml files (multiple allowed; later files override earlier).
4. --set key=value flags on the command line (highest precedence).

The common pattern is a base values.yaml plus per-environment files (values-prod.yaml) layered with -f, reserving --set for one-off or sensitive/CI-injected values. Understanding precedence prevents "why isn't my override taking effect?" confusion.

Example

# values.yaml (defaults)
replicaCount: 1
image:
  repository: myapp
  tag: "1.0"
resources:
  limits: { cpu: 500m, memory: 512Mi }

# Override: file then --set (--set wins)
helm upgrade --install web ./mychart \
  -f values-prod.yaml \                    # e.g., replicaCount: 5
  --set image.tag=2.0                       # highest precedence
# Result: replicaCount from values-prod.yaml, image.tag=2.0 from --set

Exercises

1. (Beginner) What is the purpose of values.yaml?
2. (Beginner) Between a -f values.yaml file and a --set flag, which takes precedence?
3. (Intermediate) Describe the typical pattern for managing values across multiple environments.
4. (Interview) Explain Helm's value precedence order and a scenario where misunderstanding it causes a bug. (Hint: defaults < parent < -f files < --set; an override silently ignored.)

Answers

1. To hold the chart's default configuration values, which templates reference — enabling one chart to be parameterized into many different deployments by overriding values.
2. --set takes precedence over -f value files.
3. Keep a base values.yaml with sane defaults, plus per-environment files (e.g., values-dev.yaml, values-prod.yaml) layered via -f at install/upgrade. Use --set sparingly for one-off or CI-injected/sensitive values. Store these in version control for reproducibility.
4. Precedence (lowest to highest): chart values.yaml defaults → parent chart values (for subcharts) → user -f files (later files override earlier) → --set flags. A common bug: you edit values-prod.yaml to change image.tag, but a --set image.tag=... (e.g., left in a CI script) overrides it, so your file change appears ignored. Or two -f files set the same key and the later one silently wins. Knowing the order tells you exactly which source is taking effect.

---

Installing, upgrading, and rolling back releases

Theory

Helm manages the full release lifecycle, and its killer feature over raw kubectl apply is revision history with rollback. Each helm install/upgrade creates a new revision, storing the rendered manifests and values. If an upgrade breaks, helm rollback <release> <revision> restores a prior known-good state precisely — no manual reconstruction.

Important behaviors and flags:

• helm upgrade computes a three-way merge (old chart, new chart, live state) to apply changes.
• --atomic: if the upgrade fails, automatically roll back to the previous revision (prevents being stuck in a broken half-applied state).
• --wait: wait until resources are ready before considering the release successful (often with --timeout).
• helm history lists revisions; helm rollback reverts; helm uninstall --keep-history can retain history.

This lifecycle management — atomic, waited, reversible — is what makes Helm suitable for safe, repeatable production deployments.

Example

helm history web -n prod
# REVISION  STATUS      CHART          APP VERSION  DESCRIPTION
# 1         superseded  mychart-1.0.0  1.0          Install complete
# 2         deployed    mychart-1.1.0  1.1          Upgrade complete

# Safe upgrade: wait for readiness, auto-rollback on failure
helm upgrade web ./mychart -n prod --atomic --wait --timeout 5m -f values-prod.yaml

helm rollback web 1 -n prod        # revert to revision 1 if needed

Exercises

1. (Beginner) What does each helm install/upgrade create that enables rollback?
2. (Beginner) Which command reverts a release to a previous revision?
3. (Intermediate) What do the --atomic and --wait flags do during an upgrade?
4. (Interview) How is Helm's rollback fundamentally safer/easier than rolling back a raw kubectl apply deployment? (Hint: stored rendered revisions vs. manual reconstruction.)

Answers

1. A new revision, with the rendered manifests and values stored as release history.
2. helm rollback <release> <revision>.
3. --wait makes Helm wait until the release's resources report ready (within --timeout) before declaring success. --atomic causes a failed upgrade to automatically roll back to the previous revision, so you're never left in a broken, half-applied state.
4. Helm stores the complete rendered manifests and values for every revision, so a rollback deterministically re-applies an exact prior known-good state with one command. With raw kubectl apply, there's no built-in history of what was applied at each point — to roll back you must locate and re-apply the previous manifests yourself (from Git, hopefully), reconcile drift, and remove resources that were added, which is manual, error-prone, and incomplete. Helm's stored, versioned releases make rollback precise, fast, and reliable.

---

Helm templating with Go templates

Theory

Helm templates are Kubernetes YAML with Go template directives that get rendered using the chart's values and built-in objects. This is the engine that turns one chart into many configurations. Key building blocks:

• .Values — user/chart values; .Release — release info (name, namespace, revision); .Chart — Chart.yaml data; .Capabilities — cluster/API capabilities.
• Actions in {{ }}: substitution ({{ .Values.x }}), conditionals ({{ if }}...{{ end }}), loops ({{ range }}), variables ({{ $x := ... }}).
• Pipelines and functions: {{ .Values.name | upper | quote }} — Helm includes the Sprig function library (string/number/date helpers), plus default, required, toYaml, nindent, tpl, etc.
• Whitespace control: {{- and -}} trim whitespace to keep rendered YAML valid.

The trade-off: templating makes charts flexible but the source is not valid YAML until rendered, and complex logic can hurt readability. Functions like required (fail if a value is missing) and toYaml/nindent (embed structured values with correct indentation) are workhorses.

Example

# templates/deployment.yaml (excerpt)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Release.Name }}-web            # built-in object
spec:
  replicas: {{ .Values.replicaCount | default 1 }}
  template:
    spec:
      containers:
      - name: web
        image: "{{ .Values.image.repository }}:{{ required "image.tag is required" .Values.image.tag }}"
        {{- if .Values.resources }}
        resources:
          {{- toYaml .Values.resources | nindent 10 }}   # embed structured YAML, indented
        {{- end }}

Exercises

1. (Beginner) What built-in object gives you the release's name and namespace?
2. (Beginner) What does the required function do?
3. (Intermediate) Why are toYaml and nindent commonly used together?
4. (Interview) What is the downside of Go templating in charts, and how do whitespace-control and helper functions mitigate template fragility? (Hint: invalid YAML until rendered; {{- -}}, nindent, default/required.)

Answers

1. .Release (e.g., .Release.Name, .Release.Namespace).
2. It returns a value but fails the render with an error message if that value is empty/missing — enforcing that mandatory values are supplied (e.g., required "image.tag is required" .Values.image.tag).
3. toYaml serializes a structured value (like a map of resources) into YAML, and nindent indents that block by the correct number of spaces (with a leading newline) so it nests properly under its parent key. Together they let you inject arbitrary structured config from values into a template while keeping valid, correctly-indented YAML.
4. The downside is that templated source is not valid YAML until rendered, so it's harder to read/lint, and indentation/whitespace mistakes easily produce invalid manifests; complex logic reduces clarity. Mitigations: whitespace-trim markers ({{-, -}}) keep rendered output clean and valid; nindent/indent ensure correct indentation for embedded blocks; default provides fallbacks to avoid empty values; required fails fast with a clear message instead of producing a broken manifest; and helm template/--dry-run let you verify the rendered YAML before applying.

---

Hooks and tests

Theory

Sometimes you need actions to run at specific points in a release's lifecycle — before an upgrade, after an install, before deletion. Helm hooks are ordinary Kubernetes resources (usually Jobs) annotated to run at a lifecycle phase. Common hook events: pre-install, post-install, pre-upgrade, post-upgrade, pre-delete, post-delete. Typical uses: run a database migration before an upgrade, seed data after install, or back up before deletion. Hook ordering is controlled with helm.sh/hook-weight, and cleanup with helm.sh/hook-delete-policy.

Helm tests are a special hook (helm.sh/hook: test) — Jobs/Pods that validate a release works after deployment (e.g., curl the service, check a DB connection). You run them with helm test <release>; success/failure indicates whether the deployed release is functioning. Together, hooks (lifecycle automation) and tests (post-deploy validation) make charts capable of orchestrating real operational workflows, not just applying static manifests.

Example

# A pre-upgrade migration hook (a Job that runs before the upgrade proceeds):
apiVersion: batch/v1
kind: Job
metadata:
  name: {{ .Release.Name }}-migrate
  annotations:
    "helm.sh/hook": pre-upgrade
    "helm.sh/hook-weight": "0"
    "helm.sh/hook-delete-policy": hook-succeeded
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
      - { name: migrate, image: migrator:2.0, command: ["/migrate.sh"] }
---

# A test hook (run via `helm test <release>`):
apiVersion: v1
kind: Pod
metadata:
  name: "{{ .Release.Name }}-test"
  annotations: { "helm.sh/hook": test }
spec:
  restartPolicy: Never
  containers:
  - { name: curl, image: curlimages/curl, command: ["curl","-f","http://{{ .Release.Name }}-web/"] }

Exercises

1. (Beginner) What are Helm hooks typically implemented as?
2. (Beginner) Name two hook events and a use case for each.
3. (Intermediate) How do you control the order in which multiple hooks run?
4. (Interview) What is a Helm test, how do you run it, and how does it differ from a hook? (Hint: test hook validates a release post-deploy via helm test.)

Answers

1. Ordinary Kubernetes resources — most commonly Jobs (or Pods) — annotated with helm.sh/hook to run at a lifecycle phase.
2. Examples: pre-upgrade to run a database migration before the new version rolls out; post-install to seed initial data; pre-delete to back up before removal. (Any two valid events + uses.)
3. With the helm.sh/hook-weight annotation — hooks are sorted by ascending weight (ties broken by name/kind), so lower weights run first. (And hook-delete-policy controls when hook resources are cleaned up.)
4. A Helm test is a resource annotated with helm.sh/hook: test (a Job/Pod that verifies the release works, e.g., by hitting its Service). You run it on demand with helm test <release>; its success/failure reports whether the deployed release is functioning. It differs from lifecycle hooks in when and why it runs: lifecycle hooks fire automatically during install/upgrade/delete to perform operational steps, whereas a test is invoked explicitly (after deployment) purely to validate that the release is healthy.

---

11.3 Creating and Publishing Charts

This subchapter covers authoring your own charts: scaffolding, reusable helpers, dependencies, and distribution.

---

Scaffolding a new chart

Theory

To author a chart, you don't start from scratch — helm create <name> scaffolds a complete, working starter chart with best-practice structure and a functional example (a Deployment, Service, ServiceAccount, HPA, Ingress, and a sensible values.yaml with helpers). You then edit the templates and values to fit your application.

Two essential companion commands during development:

• helm lint: checks the chart for errors and best-practice violations (malformed templates, missing required fields), catching problems before you try to install.
• helm template / helm install --dry-run: render the chart to confirm the output YAML is what you expect.

The scaffolded chart also demonstrates the conventions you should follow: using _helpers.tpl for names/labels, parameterizing everything through values.yaml, and including NOTES.txt. Starting from helm create and iterating with lint + template is the standard authoring loop.

Example

helm create mychart            # scaffold a working starter chart
tree mychart                   # Chart.yaml, values.yaml, templates/, _helpers.tpl, ...

# Edit templates/values, then validate:
helm lint mychart              # check for errors / best-practice issues
helm template mychart          # render to YAML to inspect
helm install demo mychart --dry-run --debug   # simulate against the cluster

Exercises

1. (Beginner) What does helm create give you?
2. (Beginner) What does helm lint do?
3. (Intermediate) Describe the typical develop-validate loop when authoring a chart.
4. (Interview) Why is scaffolding with helm create and following its conventions (helpers, values parameterization) beneficial for maintainability? (Hint: consistent structure, best practices, reusable naming/labels.)

Answers

1. A complete, working starter chart with best-practice structure and a functional example application (Deployment, Service, ServiceAccount, Ingress, HPA, helpers, values, NOTES) that you customize.
2. It validates the chart for errors and best-practice violations (template syntax, missing/invalid fields, conventions), catching issues before installation.
3. Edit templates and values.yaml, run helm lint to catch errors, render with helm template (and/or helm install --dry-run --debug) to inspect the produced YAML, fix issues, and repeat — iterating until the rendered manifests are correct before a real install.
4. helm create provides a consistent, proven structure and demonstrates conventions like centralizing names/labels in _helpers.tpl and parameterizing everything via values.yaml. Following these makes charts predictable to read and maintain, ensures consistent labeling/naming (important for selectors and tooling), reduces duplication (reusable helpers), and incorporates community best practices from the start — so the chart is easier for others to understand, extend, and operate.

---

Named templates and helpers

Theory

Charts often repeat the same snippets — resource names, label sets, selector labels — across many templates. Named templates (a.k.a. helpers, conventionally defined in _helpers.tpl) let you define a reusable block once and call it everywhere, keeping charts DRY and consistent. Files starting with _ aren't rendered into manifests themselves; they only hold definitions.

You define a named template with {{ define "name" }}...{{ end }} and use it with {{ include "name" . }} (passing the current context .). include is preferred over the built-in template action because include's output can be piped (e.g., {{ include "mychart.labels" . | nindent 4 }}), which template cannot. The canonical use is standardized labels and names: define mychart.fullname, mychart.labels, and mychart.selectorLabels once and reference them in every resource, guaranteeing consistent metadata (critical for selectors matching across Deployments/Services).

Example

# templates/_helpers.tpl
{{- define "mychart.fullname" -}}
{{ .Release.Name }}-{{ .Chart.Name }}
{{- end }}

{{- define "mychart.labels" -}}
app.kubernetes.io/name: {{ .Chart.Name }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}

# templates/service.yaml — reuse the helpers
metadata:
  name: {{ include "mychart.fullname" . }}
  labels:
    {{- include "mychart.labels" . | nindent 4 }}    # include can be piped

Exercises

1. (Beginner) Where are named template helpers conventionally defined, and are those files rendered into manifests?
2. (Beginner) Which keyword defines a named template, and which uses it?
3. (Intermediate) Why is include preferred over the template action?
4. (Interview) Why is defining shared label/name helpers important for correctness, not just DRY-ness? (Hint: selectors must consistently match across resources.)

Answers

1. In _helpers.tpl (files beginning with _), which are not rendered into output manifests themselves — they only hold reusable definitions.
2. {{ define "name" }}...{{ end }} defines a named template; {{ include "name" . }} uses it (passing the context).
3. Because include returns a string that can be piped through functions (e.g., | nindent 4, | quote), whereas the built-in template action writes directly to output and cannot be piped — making include far more flexible for embedding helper output with correct indentation/formatting.
4. Labels and selectors must match exactly between related resources (e.g., a Deployment's pod template labels and its selector, and a Service's selector). Defining them once in a shared helper guarantees every resource uses identical label sets, so selectors reliably match. Hand-writing labels in each template risks subtle mismatches that break Service routing or controller ownership — so helpers protect correctness, not just reduce duplication.

---

Subcharts and chart dependencies

Theory

Real applications often depend on other components — a web app needs a database, a cache. Helm models this with dependencies (subcharts): a parent chart declares dependencies in Chart.yaml, and helm dependency update fetches them into the charts/ directory. Installing the parent then deploys the subcharts too. This lets you compose applications from reusable building blocks (e.g., depend on Bitnami's PostgreSQL chart instead of writing your own).

Key mechanics:

• Values flow down: a parent can override a subchart's values by nesting them under the subchart's name in its own values.yaml. There are also global values shared across all subcharts.
• Conditions/tags: dependencies can be enabled/disabled via condition/tags (e.g., only deploy the bundled database in dev, use an external one in prod).
• Aliases: include the same subchart multiple times under different names.

Dependencies enable modular, reusable chart composition — but deep dependency trees can get complex, so use them judiciously.

Example

# Chart.yaml: declare a dependency, conditionally enabled
dependencies:
- name: postgresql
  version: "15.x.x"
  repository: https://charts.bitnami.com/bitnami
  condition: postgresql.enabled        # toggle via values

# Parent values.yaml: override the subchart's values under its name
postgresql:
  enabled: true
  auth:
    database: myapp
    username: app
global:
  storageClass: fast                   # shared with all subcharts

helm dependency update ./mychart       # fetch subcharts into charts/

Exercises

1. (Beginner) Where do you declare a chart's dependencies, and which command fetches them?
2. (Beginner) How does a parent chart override a subchart's values?
3. (Intermediate) How can you conditionally enable or disable a subchart (e.g., a bundled database)?
4. (Interview) What are the benefits and the risks of composing applications from many subcharts? (Hint: reuse/modularity vs. complexity, value-plumbing, version coupling.)

Answers

1. In the parent's Chart.yaml under dependencies; helm dependency update (or helm dependency build) fetches them into the charts/ directory.
2. By nesting overrides under the subchart's name in the parent's values.yaml (e.g., a top-level postgresql: block sets values passed to the postgresql subchart); global values can also be shared across all subcharts.
3. Give the dependency a condition (e.g., postgresql.enabled) or tags in Chart.yaml, then toggle that value (e.g., postgresql.enabled: false in prod to use an external database) to include or exclude the subchart.
4. Benefits: reuse of well-maintained components (don't reinvent a database chart), modular composition, and consistent deployment of an app plus its dependencies in one release. Risks: complexity grows with deep dependency trees (hard to reason about and debug), value-plumbing between parent and subcharts becomes intricate and error-prone, version coupling/compatibility must be managed, and an upstream subchart change can ripple unexpectedly. Use dependencies where they add clear value and keep trees shallow and well-documented.

---

Packaging and publishing to OCI registries

Theory

Once a chart is ready, you package it into a versioned .tgz archive with helm package (the filename encodes name + version). To share it, you publish to a chart repository. Modern Helm supports OCI registries (the same registries that store container images — Docker Hub, GHCR, ECR, Harbor) as first-class chart storage, which is now the recommended approach over the older HTTP index.yaml repositories.

The OCI workflow: helm package to create the archive, then helm push <chart>.tgz oci://<registry>/<repo> to upload it, and consumers use helm install <name> oci://<registry>/<repo>/<chart> --version <v> to pull it. Benefits of OCI: unified storage/auth/tooling with container images (one registry, one set of credentials and access controls), versioning, and content addressing. (The legacy alternative is hosting a static repo with an index.yaml via helm repo index.)

Example

helm package ./mychart                 # -> mychart-1.2.0.tgz

# Publish to an OCI registry (e.g., GitHub Container Registry):
helm push mychart-1.2.0.tgz oci://ghcr.io/myorg/charts

# Consume it:
helm install web oci://ghcr.io/myorg/charts/mychart --version 1.2.0

Exercises

1. (Beginner) What does helm package produce?
2. (Beginner) What is the recommended modern way to store/distribute charts?
3. (Intermediate) Write the commands to push a packaged chart to an OCI registry and install it.
4. (Interview) What advantages do OCI registries offer over the legacy index.yaml HTTP chart repositories? (Hint: unified storage/auth/tooling with images, versioning, access control.)

Answers

1. A versioned chart archive — a .tgz file named <chart>-<version>.tgz (e.g., mychart-1.2.0.tgz).
2. Publishing to an OCI registry (the same registries used for container images), which is now the recommended approach.
3. Push: helm push mychart-1.2.0.tgz oci://<registry>/<repo>. Install: helm install <name> oci://<registry>/<repo>/mychart --version 1.2.0.
4. OCI registries unify chart and container-image storage in one system with shared authentication, access control, and tooling — so you manage charts with the same credentials, RBAC, replication, and scanning infrastructure as images. They provide robust versioning and content addressing, avoid maintaining a separate static index.yaml/web server, and integrate with existing registry ecosystems (Harbor, ECR, GHCR). The legacy HTTP repo required hosting and regenerating an index file and separate auth, which is more operational overhead and less integrated.

---

Chart Museum and Artifact Hub

Theory

Two more pieces of the chart ecosystem:

• ChartMuseum: an open-source chart repository server you self-host. It serves charts over HTTP with an auto-generated index.yaml and supports various storage backends (local disk, S3, GCS). It's the classic way to run a private chart repository for your organization (predating widespread OCI support; OCI registries now often replace it).
• Artifact Hub: a CNCF-hosted central discovery website (artifacthub.io) for finding public Kubernetes packages — Helm charts, but also Operators, Kustomize bases, OPA/Kyverno policies, Falco rules, and more. It's the "search engine/catalog" for the cloud-native ecosystem: you discover a chart and its repository here, then add the repo and install. It does not host charts itself (it indexes repositories).

In short: ChartMuseum is for hosting your own charts (a private repo server); Artifact Hub is for discovering publicly available charts and other artifacts.

Example

# ChartMuseum: run a private repo server, then upload charts to it
helm repo add myrepo http://chartmuseum.internal:8080
curl --data-binary "@mychart-1.2.0.tgz" http://chartmuseum.internal:8080/api/charts

# Artifact Hub: discover public charts at https://artifacthub.io
#   search "postgresql" -> find the Bitnami chart + its repo -> add and install
helm repo add bitnami https://charts.bitnami.com/bitnami

Exercises

1. (Beginner) What is ChartMuseum used for?
2. (Beginner) What is Artifact Hub, and does it host charts itself?
3. (Intermediate) Besides Helm charts, name two other artifact types you can discover on Artifact Hub.
4. (Interview) Distinguish the roles of ChartMuseum and Artifact Hub in the chart ecosystem. (Hint: self-hosted private repo vs. central public discovery/index.)

Answers

1. Running a self-hosted (often private) Helm chart repository server that stores and serves charts over HTTP with an auto-generated index, backed by various storage backends.
2. Artifact Hub is a CNCF-hosted central website for discovering Kubernetes/cloud-native packages. It does not host the charts themselves — it indexes/links to the repositories that do.
3. Any two: Operators (OLM), Kustomize bases, OPA/Gatekeeper or Kyverno policies, Falco rules, Tekton tasks, CoreDNS plugins, container images, etc.
4. ChartMuseum is infrastructure you run to host and serve your own (typically private) charts — a repository server. Artifact Hub is a public discovery catalog/index where users find publicly available charts and other artifacts and learn which repository to add. One is about hosting/distribution within your control; the other is about searchability/discovery across the broader ecosystem. (OCI registries increasingly fulfill ChartMuseum's hosting role.)

---

11.4 Kustomize

Kustomize is the template-free alternative built into kubectl. This subchapter covers its overlay model, patches, generators, and usage.

---

Kustomize overlays and bases

Theory

Kustomize customizes plain Kubernetes YAML without templates, using a base + overlays model:

• A base is a directory of complete, valid manifests plus a kustomization.yaml listing them — the common configuration shared across environments.
• An overlay is a directory with its own kustomization.yaml that references the base (resources: [ ../../base ]) and applies environment-specific modifications (patches, name prefixes, replica counts, images, extra resources).

Because the base is real YAML, you can kubectl apply -f it directly, and overlays only express the differences per environment (dev/staging/prod). This keeps each environment DRY and the differences explicit and reviewable. You build the final manifests with kubectl kustomize <overlay> (render) or kubectl apply -k <overlay> (render + apply). The philosophy is "declarative, template-free customization through composition," contrasting sharply with Helm's templating.

Example

app/
  base/
    kustomization.yaml      # resources: deployment.yaml, service.yaml
    deployment.yaml         # plain, valid YAML (replicas: 1)
    service.yaml
  overlays/
    prod/
      kustomization.yaml    # references ../../base + prod patches
      replicas-patch.yaml   # replicas: 5

# overlays/prod/kustomization.yaml
resources:
- ../../base
namePrefix: prod-
patches:
- path: replicas-patch.yaml
images:
- name: myapp
  newTag: "2.0"

Exercises

1. (Beginner) What is the difference between a base and an overlay in Kustomize?
2. (Beginner) Which command renders a Kustomize overlay, and which renders + applies it?
3. (Intermediate) Why can a Kustomize base be applied directly with kubectl apply -f, unlike a Helm chart's templates?
4. (Interview) How does the base+overlay model keep multi-environment configuration DRY while keeping differences explicit? (Hint: shared base, overlays express only deltas.)

Answers

1. A base is a set of complete, valid manifests representing the shared/common configuration; an overlay references a base and applies environment-specific modifications (patches, prefixes, image tags, extra resources) — expressing only the differences.
2. kubectl kustomize <dir> renders to YAML; kubectl apply -k <dir> renders and applies.
3. Because Kustomize bases are plain, valid Kubernetes YAML (no templating placeholders), they can be applied as-is. Helm chart templates contain Go-template directives and are not valid YAML until rendered with values, so they can't be applied directly.
4. The common configuration lives once in the base; each overlay references that base and contains only the deltas for its environment (e.g., prod sets replicas and image tag). This avoids duplicating full manifests per environment (DRY) while making each environment's differences small, explicit, and easy to review in isolation — you can see exactly how prod differs from the base without scanning entire duplicated files.

---

Patches: strategic merge and JSON6902

Theory

The heart of Kustomize customization is patching existing resources from the base. Two patch styles:

• Strategic Merge Patch: a partial YAML document that merges into the target resource — you specify only the fields you want to change/add, matched by the resource's identity. Intuitive for most edits (change replicas, add an env var, set resources). It understands Kubernetes' merge semantics (e.g., merging list items by key like container name).
• JSON Patch (RFC 6902): a list of explicit operations (op: replace/add/remove, with a path) applied to the resource. More precise and powerful for surgical changes — especially array element manipulation by index, or removing a field — where strategic merge is awkward.

Use strategic merge for the common "override/add a few fields" case; reach for JSON6902 when you need exact operations (delete a specific item, edit by array index). Both are declared under patches: in kustomization.yaml, targeting resources by kind/name (and optionally label selectors).

Example

# Strategic merge patch: change replicas and add an env var
apiVersion: apps/v1
kind: Deployment
metadata: { name: web }
spec:
  replicas: 5
  template:
    spec:
      containers:
      - name: web                 # matched by name; fields merged
        env:
        - { name: LOG_LEVEL, value: debug }
---

# JSON 6902 patch (in kustomization.yaml): surgical operations
patches:
- target: { kind: Deployment, name: web }
  patch: |-
    - op: replace
      path: /spec/replicas
      value: 5
    - op: remove
      path: /spec/template/spec/containers/0/livenessProbe

Exercises

1. (Beginner) What does a strategic merge patch let you specify?
2. (Beginner) What does a JSON 6902 patch consist of?
3. (Intermediate) Give a case where JSON6902 is more suitable than a strategic merge patch.
4. (Interview) How does a strategic merge patch handle list items (like containers) differently from a naive merge, and why does that matter? (Hint: merge-by-key, e.g., container name.)

Answers

1. Only the fields you want to add or change, as a partial version of the target resource; Kustomize merges them into the base resource.
2. An ordered list of explicit operations (e.g., op: add/replace/remove) each with a path (and value where applicable) applied to the resource.
3. When you need surgical/precise changes that merge semantics handle poorly — e.g., removing a specific field, replacing or deleting an element of an array by index, or modifying a list where there's no merge key. JSON6902's explicit path-based operations express these precisely.
4. Strategic merge uses Kubernetes' schema-aware merge semantics: lists with a merge key (e.g., containers keyed by name) are merged by matching that key, so a patch updates the matching container rather than replacing the entire list. This matters because a naive merge would overwrite or duplicate the whole list; merge-by-key lets you change one container's fields (or add a new one) while leaving the others intact, which is what you almost always want.

---

ConfigMap and Secret generators

Theory

A standout Kustomize feature is generators for ConfigMaps and Secrets. Instead of hand-writing a ConfigMap/Secret manifest, you declare a configMapGenerator/secretGenerator that builds it from literals, files, or env files. Crucially, generators append a content-based hash suffix to the generated object's name (e.g., app-config-7f8c9d) and automatically update references to it (in Deployments, etc.).

This elegantly solves the config-change rollout problem from Chapter 5: when the ConfigMap's content changes, its name changes (new hash), which changes the Pod template that references it, which triggers a rolling update — so config changes automatically and safely propagate to running Pods, with no manual restart and no separate checksum-annotation trick. You can disable hashing if you need stable names (generatorOptions: disableNameSuffixHash: true). Generators are a major ergonomic and correctness win unique to the Kustomize workflow.

Example

# kustomization.yaml
configMapGenerator:
- name: app-config
  literals:
  - LOG_LEVEL=info
  files:
  - app.properties
secretGenerator:
- name: db-creds
  literals:
  - password=s3cr3t
# Generated names get a hash suffix, e.g. app-config-7f8c9d2b1a,
# and references to "app-config" in Deployments are rewritten automatically.

Exercises

1. (Beginner) What do Kustomize ConfigMap/Secret generators create, and from what sources?
2. (Beginner) What does Kustomize append to a generated ConfigMap/Secret's name?
3. (Intermediate) How does the name-hash mechanism cause config changes to roll out automatically?
4. (Interview) Compare Kustomize's generator-hash approach to the Helm "checksum annotation" pattern for triggering rollouts on config change. (Hint: both change the Pod template; generators also rename + update references automatically.)

Answers

1. They generate ConfigMaps/Secrets from literals, files, or env files (so you don't hand-write the manifest), and Kustomize manages them as part of the build.
2. A content-based hash suffix on the name (e.g., app-config-7f8c9d), and it rewrites references to use the hashed name.
3. When the generated object's content changes, its hash (and thus name) changes; Kustomize updates the references in workloads to the new name, which alters the Pod template spec. A changed Pod template triggers a Deployment rolling update, so Pods are recreated and pick up the new config automatically and safely.
4. Both work by changing the Pod template so the Deployment performs a rolling update on config change. Helm's pattern requires you to manually add a checksum/config annotation computed from the ConfigMap and keep references in sync. Kustomize's generators do this automatically: they hash content, rename the object, and rewrite all references — no manual annotation or reference plumbing — making safe config rollouts the default behavior with less boilerplate (at the cost of changing object names, which you can disable if needed).

---

Kustomize with kubectl

Theory

A major practical advantage of Kustomize is that it's built into kubectl — no separate tool to install. Two flags expose it:

• kubectl apply -k <dir>: build the kustomization in <dir> and apply the result to the cluster.
• kubectl kustomize <dir>: build and print the rendered YAML to stdout (for inspection, diffing, or piping).

A caveat: the Kustomize version embedded in kubectl can lag behind the standalone kustomize CLI, so for the newest features you may install the standalone binary. Kustomize also integrates deeply with GitOps tools — Argo CD and Flux natively render kustomizations — making it a first-class citizen of declarative, Git-driven workflows (Chapter 12). The combination of "no extra tooling, plain YAML, native GitOps support" is why many teams choose Kustomize for managing their own applications, reserving Helm for packaged/third-party software.

Example

kubectl kustomize overlays/prod          # render the prod overlay to YAML
kubectl apply -k overlays/prod           # render + apply to the cluster
kubectl kustomize overlays/prod | kubectl diff -f -   # preview changes

# For the latest features, the standalone CLI:
kustomize build overlays/prod | kubectl apply -f -

Exercises

1. (Beginner) Which kubectl flag applies a kustomization, and which just renders it?
2. (Beginner) Do you need to install a separate tool to use Kustomize with kubectl?
3. (Intermediate) Why might you install the standalone kustomize CLI even though kubectl includes it?
4. (Interview) Why is Kustomize's combination of "built into kubectl, plain YAML, native GitOps support" appealing for managing your own applications? (Hint: no extra tooling/templating; Argo CD/Flux render it natively.)

Answers

1. kubectl apply -k <dir> applies it; kubectl kustomize <dir> renders it to stdout.
2. No — Kustomize is built into kubectl (-k / kubectl kustomize).
3. The Kustomize version bundled in kubectl often lags the standalone CLI, so to use the latest Kustomize features/fixes you install the separate kustomize binary (kustomize build).
4. It requires no extra tooling (already in kubectl) and no templating language (manifests stay plain, valid, readable YAML that's easy to diff and review), lowering the learning curve and complexity. And because GitOps controllers (Argo CD, Flux) render kustomizations natively, your overlays plug directly into declarative, Git-driven deployment pipelines. For teams managing their own apps across environments, this yields a simple, transparent, Git-friendly workflow without Helm's templating overhead — while Helm remains better for packaging/distributing software.

---