14. Extending Kubernetes

Kubernetes' greatest strength is that it is not a fixed product but an extensible platform. You can teach it about entirely new resource types, write controllers that automate operational knowledge, plug into the API machinery itself, and build all of this with mature client libraries. This chapter covers the extension points — CRDs, the Operator pattern, API extension mechanisms, and the client libraries that power them — that let you make Kubernetes manage anything, not just Pods.

---

14.1 Custom Resource Definitions (CRDs)

CRDs let you add your own API objects to Kubernetes. This subchapter covers defining, validating, versioning, and structuring them.

---

CRD schema and validation

Theory

A Custom Resource Definition (CRD) lets you extend the Kubernetes API with your own resource types, which then behave like built-in resources: you kubectl get/apply them, they're stored in etcd, served by the API server, watchable, and RBAC-controlled. For example, you could define a Database or Certificate kind. The CRD declares the new kind's group, version, names (plural/singular/kind), scope (Namespaced/Cluster), and — crucially — its schema.

The schema uses OpenAPI v3 (under validation.openAPIV3Schema) to define the resource's fields and validation rules: types, required fields, enums, ranges, patterns, and defaults. The API server enforces this schema on every create/update, rejecting invalid objects — so your custom resource gets the same server-side validation as native resources, with no code. You can also add CEL validation rules (x-kubernetes-validations) for cross-field/complex constraints. A CRD alone just adds the type and storage; to make it do something you pair it with a controller (Operators, next subchapter).

Example

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata: { name: backups.example.com }
spec:
  group: example.com
  scope: Namespaced
  names: { plural: backups, singular: backup, kind: Backup, shortNames: [bk] }
  versions:
  - name: v1
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            required: [source, schedule]
            properties:
              source:   { type: string }
              schedule: { type: string, pattern: '^(\S+\s+){4}\S+$' }   # cron
              retain:   { type: integer, minimum: 1, default: 7 }

Exercises

1. (Beginner) What does a CRD let you do, and how do the resulting objects behave?
2. (Beginner) What schema standard does a CRD use to validate its fields?
3. (Intermediate) Does creating a CRD alone make the custom resource do anything? What else is needed?
4. (Interview) How does CRD schema validation give your custom resources the same robustness as built-in types without writing code? (Hint: API server enforces OpenAPI/CEL on every write.)

Answers

1. It adds a new custom resource type to the Kubernetes API. The resulting objects behave like native ones — stored in etcd, served by the API server, manageable with kubectl, watchable, and controlled by RBAC.
2. OpenAPI v3 (under openAPIV3Schema), optionally augmented with CEL rules via x-kubernetes-validations.
3. No — a CRD only defines the type and provides storage/validation/API serving. To make the resource take effect, you need a controller/operator that watches those custom resources and reconciles real-world state to match them.
4. The CRD's OpenAPI v3 schema (plus optional CEL rules) is enforced by the API server on every create/update: it type-checks fields, applies required/enum/range/pattern constraints and defaults, and rejects invalid objects — exactly as it does for built-in resources. So your custom type gets server-side validation, defaulting, and consistent API behavior declaratively, without writing any validation code; you just describe the schema.

---

Versioning and conversion webhooks

Theory

APIs evolve, and CRDs support multiple versions (e.g., v1alpha1, v1beta1, v1) of the same resource, declared in spec.versions. Exactly one version is marked storage: true (the version actually persisted in etcd), and each version can be independently served. This lets you introduce new schema versions while still supporting clients using older ones — essential for backward compatibility as your API matures.

When clients use different versions than the stored one, the API server must convert between them. For trivial changes, the default None conversion strategy (no field changes between versions) suffices. For non-trivial schema changes, you provide a conversion webhook: the API server calls your webhook to convert objects between versions on the fly (e.g., reading a v1 object that's stored as v2, or vice versa). This is what makes safe CRD evolution possible — you can add/rename/restructure fields across versions and the webhook reconciles representations, so old and new clients both work and stored data stays consistent.

Example

spec:
  versions:
  - { name: v1alpha1, served: true,  storage: false }
  - { name: v1,       served: true,  storage: true }   # the version stored in etcd
  conversion:
    strategy: Webhook                # call a webhook to convert between versions
    webhook:
      clientConfig:
        service: { name: conv, namespace: system, path: /convert }
      conversionReviewVersions: ["v1"]

Exercises

1. (Beginner) How many CRD versions can be marked as the storage version?
2. (Beginner) What does the served flag control for a CRD version?
3. (Intermediate) When is a conversion webhook necessary versus the None strategy?
4. (Interview) How do multiple versions plus conversion webhooks enable safe, backward-compatible evolution of a custom API? (Hint: serve old+new, store one, convert on the fly.)

Answers

1. Exactly one (only one version has storage: true — the version persisted in etcd).
2. Whether that version is exposed/served by the API server for clients to read and write (a version can be served without being the storage version, or stop being served when deprecated).
3. None works only when versions are structurally identical (no field changes requiring transformation). A conversion webhook is needed when versions differ non-trivially (added/renamed/restructured fields), so the API server can transform objects between the requested version and the stored version correctly.
4. You can serve several versions simultaneously (so old and new clients each use their version) while persisting just one storage version in etcd. A conversion webhook transforms objects between any served version and the storage version on the fly, so reads/writes in any version are consistent regardless of how data is stored. This lets you introduce a new schema version, migrate clients gradually, and eventually deprecate old versions — all without breaking existing clients or corrupting stored data — enabling safe, backward-compatible API evolution.

---

Structural schemas and pruning

Theory

Modern CRDs (apiextensions.k8s.io/v1) require a structural schema — a complete, well-formed OpenAPI v3 schema that specifies the type of every field (no ambiguous/partial schemas). Structural schemas are the foundation that enables important safety features:

• Pruning: with a structural schema, the API server removes (prunes) any fields not declared in the schema before storing the object. This prevents clients from sneaking unknown/typo'd fields into etcd, which could otherwise cause confusion, bloat, or security issues. (Without pruning, unknown fields would be silently preserved.)
• Defaulting: structural schemas allow server-side default values for fields.
• They also enable proper validation, conversion, and kubectl explain documentation for your CRD.

The practical upshot: define your CRD's schema completely and correctly, and the API server will validate, default, prune, and document your resource just like a native type — and reject or strip anything that doesn't conform. Pruning in particular guards against silent acceptance of malformed/unexpected data, making your custom API strict and predictable.

Example

# A structural schema (every field typed) enables pruning + defaulting:
schema:
  openAPIV3Schema:
    type: object
    properties:
      spec:
        type: object
        properties:
          replicas: { type: integer, default: 1 }   # defaulting
          image:    { type: string }
        # NOT listed here => pruned (removed) on write

kubectl apply -f cr-with-typo.yaml
# A field like "replcias: 3" (typo, not in schema) is PRUNED — silently dropped,
# so the object stores only declared fields. (Use --validate to catch typos earlier.)

Exercises

1. (Beginner) What is required for a modern (v1) CRD's schema?
2. (Beginner) What does "pruning" do to fields not declared in the schema?
3. (Intermediate) Why is pruning a useful safety feature?
4. (Interview) How do structural schemas underpin validation, defaulting, and pruning, and what risk do they mitigate? (Hint: complete typed schema enables strict, predictable handling; prevents unknown fields in etcd.)

Answers

1. A complete structural schema — a well-formed OpenAPI v3 schema that fully specifies the type of every field (no partial/ambiguous schemas).
2. They are removed (pruned) from the object before it's persisted, so only fields declared in the schema are stored.
3. It prevents unknown, misspelled, or unexpected fields from being silently stored in etcd. This avoids confusion (a typo'd field that's silently ignored), data bloat, and potential security/consistency issues from clients injecting arbitrary fields — keeping stored objects strictly conformant to the declared schema.
4. A structural schema fully types every field, which the API server needs to reliably validate values, apply defaults, perform version conversion, and document the type (kubectl explain). It also enables pruning: anything not in the schema is stripped on write. The risk it mitigates is unstructured/unknown data entering the API — without it, typo'd or unexpected fields could be silently preserved, leading to confusing behavior, bloat, or security concerns. With it, the custom resource is handled as strictly and predictably as a native type.

---

CRD status subresource

Theory

Kubernetes objects conventionally separate spec (the user's desired state) from status (the controller's observed/actual state). A CRD can enable the /status subresource, which formalizes this split with important benefits:

• Separation of concerns: users/clients write spec; the controller writes status. With the status subresource enabled, updates to spec and status go through separate API endpoints, so a controller updating status doesn't accidentally clobber a user's spec change (and RBAC can grant status-update separately).
• No spec changes via status updates: updating the status subresource ignores changes to anything but status (and vice versa), preventing conflicts.
• Enables /scale: a CRD can also expose a scale subresource, letting kubectl scale and the HPA operate on your custom resource.

This is the standard pattern for any custom resource managed by a controller: the controller continuously reconciles toward spec and reports progress/conditions in status (e.g., conditions, observedGeneration, phase). Enabling the status subresource is best practice for well-behaved, controller-managed CRDs.

Example

spec:
  versions:
  - name: v1
    served: true
    storage: true
    subresources:
      status: {}                       # enable the /status subresource
      scale:                           # optional: enable kubectl scale / HPA
        specReplicasPath: .spec.replicas
        statusReplicasPath: .status.replicas
    schema: { openAPIV3Schema: { type: object, properties:
      { spec: { type: object }, status: { type: object } } } }

// Controller updates status via the dedicated endpoint (won't touch spec):
r.Status().Update(ctx, &backup)   // controller-runtime: status subresource

Exercises

1. (Beginner) What is the conventional difference between spec and status?
2. (Beginner) Who writes spec and who writes status for a controller-managed CRD?
3. (Intermediate) What problem does enabling the status subresource prevent?
4. (Interview) Why is enabling the status (and optionally scale) subresource best practice for controller-managed CRDs? (Hint: separate endpoints avoid clobbering; enables scale/HPA; clean reconcile reporting.)

Answers

1. spec is the user's declared desired state; status is the controller's reported observed/actual state.
2. Users/clients write spec; the controller writes status.
3. It separates updates to spec and status into distinct API endpoints, so a controller writing status can't accidentally overwrite a concurrent user change to spec (and vice versa) — preventing conflicts/clobbering between the user's desired state and the controller's reported state.
4. Enabling the status subresource enforces the spec/status separation at the API level: controllers update status without risk of clobbering user spec edits, RBAC can grant status updates independently, and reconcile progress (conditions, observedGeneration, phase) is reported cleanly. The scale subresource lets standard tooling (kubectl scale) and the HPA operate on the custom resource by exposing replicas paths. Together they make a custom resource behave like a proper, well-integrated Kubernetes object that controllers and ecosystem tools handle correctly — the expected pattern for any controller-managed CRD.

---

14.2 Operators

Operators combine CRDs with controllers to automate operational knowledge. This subchapter covers the pattern and how to build them.

---

Operator pattern and use cases

Theory

A CRD adds a new type; an Operator makes it do something. The Operator pattern combines a custom resource (CRD) with a custom controller that encodes operational knowledge — the human expertise of running a particular application — as software. The controller continuously watches its custom resources and reconciles the real world to match them, automating tasks a human operator would otherwise perform.

The canonical use case is stateful, complex applications that require domain-specific operational logic: a database operator (PostgreSQL, MySQL) that handles provisioning, configuration, backups, failover, scaling, and version upgrades automatically; a Kafka/Elasticsearch/Prometheus operator; etc. Instead of you running runbooks, you declare a PostgresCluster resource and the operator does the rest — and keeps doing it (self-healing, responding to failures). The pattern's power is encoding the "Day 2" operational know-how (not just install, but ongoing management) into a controller, turning manual expertise into automated, reliable, declarative operations. Many such operators are published on OperatorHub.

Example

Operator = CRD (new API) + Controller (operational logic)

You declare:                    The operator continuously does:
  kind: PostgresCluster           - provision Pods/PVCs/Services
  spec:                           - configure replication
    instances: 3                  - take scheduled backups
    version: "16"                 - detect primary failure -> failover
    backup: { schedule: ... }     - perform safe version upgrades

Exercises

1. (Beginner) What two things does the Operator pattern combine?
2. (Beginner) What kind of applications most benefit from Operators?
3. (Intermediate) Give three "Day 2" operational tasks a database operator might automate.
4. (Interview) Explain how the Operator pattern turns human operational knowledge into software, and why that's valuable. (Hint: encode runbooks as a reconciling controller; reliability, self-healing, declarative ops.)

Answers

1. A custom resource (CRD) and a custom controller that reconciles those resources (encoding operational logic).
2. Complex, stateful applications that require domain-specific operational knowledge to run — e.g., databases, message brokers, search/monitoring systems (PostgreSQL, Kafka, Elasticsearch, Prometheus).
3. Any three: provisioning/configuration, automated backups and restores, failover/high-availability handling, scaling, rolling version upgrades, and monitoring/self-healing.
4. An Operator encodes the expertise a skilled human operator uses to run an application — the runbooks for install, configuration, backup, failover, scaling, and upgrades — into a controller that continuously watches custom resources and reconciles reality to the declared spec. This is valuable because it makes that operational knowledge automated, consistent, and always-on: tasks that would otherwise require manual intervention (and be error-prone, slow, or done only when someone notices) happen reliably and immediately, the system self-heals from failures, and operations become declarative (you state desired outcomes, the operator achieves them). It scales human expertise across many instances and clusters without scaling the humans.

---

Controller reconcile loop

Theory

The heart of any controller/operator is the reconcile loop — the same level-triggered pattern from Chapter 1, applied to your resources. The controller watches its resource(s) and, on any change (create/update/delete) or periodic resync, invokes a Reconcile function for the affected object. Reconcile's job: read the current desired state (the object's spec), observe the actual world, and take actions to converge them — then update status.

Key properties of a well-written reconcile function:

• Idempotent: it may be called many times for the same object (events, resyncs, retries), so it must produce the same correct result each time — it computes desired vs. actual and acts only on the difference, never assuming it's the "first" call.
• Level-triggered: it acts on the current observed state, not on the specific event that triggered it — so missed events don't break convergence; the next reconcile fixes things.
• Returns a result: success, requeue-after-duration, or error (which triggers retry with backoff).

This loop is what gives operators their robustness and self-healing: regardless of what happened (or what was missed), each reconcile drives the system toward the declared state, retrying on failure. Writing correct, idempotent reconcile logic is the core skill of operator development.

Example

func (r *BackupReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var backup examplev1.Backup
    if err := r.Get(ctx, req.NamespacedName, &backup); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)   // deleted: nothing to do
    }
    // Observe actual state, compare to backup.Spec, act on the DIFFERENCE (idempotent):
    if !backupJobExists(&backup) {
        r.createBackupJob(ctx, &backup)        // safe to call repeatedly
    }
    backup.Status.LastRun = metav1.Now()
    r.Status().Update(ctx, &backup)            // report observed state
    return ctrl.Result{RequeueAfter: time.Hour}, nil   // periodic resync
}

Exercises

1. (Beginner) What does a controller's Reconcile function do at a high level?
2. (Beginner) Why must Reconcile be idempotent?
3. (Intermediate) What does it mean that controllers are "level-triggered," and why does it aid robustness?
4. (Interview) How does the reconcile loop give operators self-healing behavior even when events are missed or actions fail? (Hint: act on current state each call; retries/requeue; convergence.)

Answers

1. It reads the desired state (the resource's spec), observes the actual state of the world, takes actions to make actual match desired, and updates the resource's status.
2. Because it can be invoked many times for the same object (from events, periodic resyncs, and retries), so it must not assume it's running for the first time. It should compute desired vs. actual and act only on the difference, producing the same correct outcome regardless of how often it's called — otherwise repeated calls could create duplicates or cause errors.
3. Level-triggered means the controller responds to the current observed state rather than to the specific event that triggered the call. So even if an event is missed or arrives out of order, the next reconcile still observes reality and corrects it — convergence depends on current state, not on processing every event, which makes the controller robust to dropped events, restarts, and races.
4. Each reconcile re-observes the current actual state and drives it toward the declared spec, so any drift (a deleted resource, a failed component, a missed event) is detected and corrected on the next pass. If an action fails or the object isn't yet converged, the controller returns an error or requeue, and the loop retries (with backoff) — repeatedly attempting until the world matches the spec. Because correctness depends only on current state and the loop keeps running/retrying, the operator continuously heals the system toward desired state regardless of intermediate failures or missed events.

---

Operator SDK and Kubebuilder

Theory

Writing an operator from scratch (API types, CRD manifests, controller scaffolding, manager setup) is a lot of boilerplate. Two frameworks generate and standardize this, both built on the controller-runtime library (the Go foundation for controllers):

• Kubebuilder: a CNCF/Kubernetes-SIG project that scaffolds an operator in Go — it generates the project layout, API type definitions, CRD YAML (from Go structs + markers), and controller stubs, and provides the manager/runtime wiring. You define your types and fill in Reconcile.
• Operator SDK (Red Hat, part of the Operator Framework): builds on Kubebuilder for Go operators and additionally supports Ansible and Helm-based operators (where you can turn an existing Helm chart or Ansible playbooks into an operator with little/no Go code), plus integration with OLM (next).

Both let you focus on the business logic (the reconcile function and API design) rather than plumbing. Kubebuilder is the lower-level Go-centric foundation; Operator SDK wraps it with more options (Go/Ansible/Helm) and Operator Framework tooling. They generate CRDs, RBAC, deployment manifests, and test scaffolding via code markers, dramatically accelerating operator development.

Example

# Kubebuilder: scaffold a project, an API/CRD, and a controller
kubebuilder init --domain example.com --repo example.com/backup-operator
kubebuilder create api --group apps --version v1 --kind Backup
#   -> generates: API types (Go), CRD YAML, controller stub (Reconcile), RBAC markers
make manifests        # regenerate CRDs/RBAC from Go markers
make run              # run the controller locally against the cluster

// Markers on the Go type drive CRD/RBAC generation:
//+kubebuilder:validation:Minimum=1
//+kubebuilder:rbac:groups=apps,resources=backups,verbs=get;list;watch;create;update

Exercises

1. (Beginner) What underlying library do both Kubebuilder and Operator SDK build on?
2. (Beginner) Besides Go, what two operator types does Operator SDK support?
3. (Intermediate) What does scaffolding with these tools generate so you can focus on business logic?
4. (Interview) How do Go markers/annotations in Kubebuilder reduce boilerplate, and what's the relationship between Kubebuilder and Operator SDK? (Hint: markers generate CRD/RBAC; SDK builds on Kubebuilder + adds Ansible/Helm/OLM.)

Answers

1. controller-runtime (the Go library for building controllers/operators).
2. Ansible-based and Helm-based operators (in addition to Go).
3. Project scaffolding: the directory layout, API type definitions, generated CRD manifests, controller stubs (with a Reconcile method to implement), RBAC rules, manager/runtime wiring, and test scaffolding — so you implement the reconcile logic and API design rather than the plumbing.
4. Kubebuilder uses special comment markers on Go types/methods (e.g., validation constraints, RBAC rules, subresources) that code generators read to produce CRD YAML, RBAC manifests, and deepcopy code automatically — so you declare intent inline and regenerate boilerplate with make, instead of hand-writing it. Operator SDK builds on top of Kubebuilder for Go operators (sharing controller-runtime and the same scaffolding), and extends it with additional operator types (Ansible, Helm) and Operator Framework integration (like OLM). So Kubebuilder is the Go-focused foundation; Operator SDK wraps it with more options and tooling.

---

Operator Lifecycle Manager (OLM)

Theory

Once operators exist, you need to manage the operators themselves — install them, resolve their dependencies, handle their RBAC/CRDs, and upgrade them safely over time. The Operator Lifecycle Manager (OLM) is a component (part of the Operator Framework) that does exactly this: it's like a package manager for operators, running in the cluster to install and manage operators declaratively.

OLM introduces concepts such as:

• ClusterServiceVersion (CSV): metadata describing an operator version — its CRDs, required permissions, dependencies, and install strategy.
• Subscription: declares that you want a particular operator (from a catalog) installed and kept updated on a chosen update channel (e.g., stable), enabling automatic or manual upgrades.
• CatalogSource / OperatorHub: catalogs of available operators OLM can install from.

OLM handles dependency resolution (an operator needing another), upgrade graphs (moving safely between versions via channels), and lifecycle/permissions — so cluster admins manage operators consistently rather than each operator being installed ad hoc. It's prominent in OpenShift (where it's built in) and available on upstream Kubernetes.

Example

# Subscribe to an operator from a catalog; OLM installs and keeps it updated
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata: { name: my-db-operator, namespace: operators }
spec:
  channel: stable                 # update channel (upgrade graph)
  name: my-db-operator
  source: operatorhubio-catalog   # a CatalogSource
  sourceNamespace: olm
  installPlanApproval: Automatic   # or Manual (gate upgrades)

Exercises

1. (Beginner) What is OLM, by analogy?
2. (Beginner) What does a Subscription declare?
3. (Intermediate) What does a ClusterServiceVersion (CSV) describe?
4. (Interview) Why is lifecycle management of operators (dependency resolution, channels, upgrades) valuable in a cluster running many operators? (Hint: consistent install/upgrade/permissions vs. ad-hoc operator sprawl.)

Answers

1. A package manager for operators — it installs, manages, and upgrades operators in the cluster declaratively (analogous to how Helm/apt manage packages, but for operators).
2. That you want a particular operator installed from a catalog and kept updated on a chosen update channel (with automatic or manual upgrade approval).
3. An operator version's metadata: its CRDs, required RBAC permissions, dependencies, install strategy, and other descriptive information OLM uses to install and manage that operator.
4. With many operators, installing/upgrading each ad hoc leads to inconsistency, version drift, permission sprawl, and dependency conflicts. OLM provides consistent, declarative lifecycle management: it resolves operator dependencies, uses update channels/upgrade graphs to move safely between versions, manages each operator's CRDs and RBAC, and lets admins approve or automate upgrades. This makes running and maintaining a fleet of operators reliable and auditable — operators are managed as first-class, versioned packages rather than scattered manual installs.

---

OperatorHub

Theory

OperatorHub.io is a central registry/catalog of community and vendor operators — the "app store" for Kubernetes operators. Rather than building an operator for a common application yourself, you can browse OperatorHub for an existing, often well-maintained operator (for databases, monitoring, messaging, storage, etc.), review its capabilities and maturity level, and install it.

OperatorHub categorizes operators by a capability level model — from Level 1 "Basic Install" up through Level 5 "Auto Pilot" (full automated operations: auto-scaling, auto-healing, auto-tuning) — so you can judge how much operational automation an operator actually provides. Operators from OperatorHub integrate with OLM for installation and lifecycle management (especially on OpenShift, which surfaces OperatorHub in its console). The value: a large ecosystem of reusable operational automation you can adopt instead of reinventing, with a standard way to assess maturity and install/manage them. (Helm's Artifact Hub also lists operators; OperatorHub is the OLM-centric catalog.)

Example

OperatorHub capability levels (maturity of operational automation):
  1 Basic Install   -> install/configure the app
  2 Seamless Upgrades -> handle version upgrades
  3 Full Lifecycle  -> backups, failover, scaling
  4 Deep Insights   -> metrics, alerts, analysis
  5 Auto Pilot      -> auto-scaling/healing/tuning

# Discover an operator on operatorhub.io, then install it (via OLM):
# create a Subscription (see OLM example) pointing at the catalog entry.

Exercises

1. (Beginner) What is OperatorHub?
2. (Beginner) What does the capability-level model communicate about an operator?
3. (Intermediate) What does OperatorHub integrate with for installation and lifecycle?
4. (Interview) Why might you adopt an operator from OperatorHub instead of building your own, and what should you evaluate before doing so? (Hint: reuse maintained automation; assess maturity/capability level, trust, maintenance.)

Answers

1. A central registry/catalog ("app store") of community- and vendor-provided Kubernetes operators that you can browse and install.
2. How much operational automation the operator provides — from Level 1 (basic install) up to Level 5 (auto-pilot: auto-scaling, healing, tuning) — helping you gauge its maturity and what "Day 2" operations it handles.
3. OLM (the Operator Lifecycle Manager), which installs and manages the operators (with OperatorHub prominently integrated into OpenShift's console).
4. Building and maintaining a robust operator for a complex stateful app is significant, ongoing work; OperatorHub offers existing, often vendor/community-maintained operators that encode that expertise, saving development and maintenance effort. Before adopting one you should evaluate its capability/maturity level (does it automate the operations you need — backups, upgrades, failover?), its trustworthiness and source (vendor vs. community, security posture, RBAC it requires), maintenance activity and support, license, and how well it fits your environment — since you're delegating critical operational logic to third-party software running with cluster permissions.

---

14.3 Kubernetes API Extension Points

Beyond CRDs, Kubernetes exposes deeper extension points in its API machinery. This subchapter covers them.

---

Aggregated API servers

Theory

CRDs are the easy way to add API types, but they have limits: they're stored in etcd as-is, can't do arbitrary custom storage/business logic on reads, and can't easily implement complex behaviors (custom subresources, specialized validation/processing, non-etcd backing). For those cases, Kubernetes offers API Aggregation: you run your own API server (an "extension API server") and register it with the main API server via an APIService object, so requests for your API group/version are transparently proxied to your server.

To clients, the aggregated API looks like a native part of the Kubernetes API (same endpoint, auth, discovery) — but your extension server fully controls how those resources are validated, stored, and served. This is how Metrics Server works (it serves metrics.k8s.io as an aggregated API, computing metrics on the fly rather than storing them in etcd), and how custom/external metrics APIs are implemented. Aggregation is more powerful but more complex than CRDs (you build and operate a real API server). Rule of thumb: use CRDs unless you need custom storage/logic that CRDs can't provide — then reach for an aggregated API server.

Example

# Register an extension API server so /apis/metrics.k8s.io is proxied to it:
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata: { name: v1beta1.metrics.k8s.io }
spec:
  group: metrics.k8s.io
  version: v1beta1
  service: { name: metrics-server, namespace: kube-system }   # your API server
  groupPriorityMinimum: 100
  versionPriority: 100

Exercises

1. (Beginner) What does API aggregation let you add to Kubernetes beyond a CRD?
2. (Beginner) What object registers an aggregated API with the main API server?
3. (Intermediate) Give a real example of an aggregated API server and why it needs aggregation rather than a CRD.
4. (Interview) When should you choose an aggregated API server over a CRD, and what's the cost? (Hint: custom storage/logic/subresources vs. operating a real API server.)

Answers

1. Your own extension API server serving custom API group(s)/version(s) with full control over validation, storage, and request handling — not just a new type stored in etcd like a CRD.
2. An APIService object (apiregistration.k8s.io), which tells the main API server to proxy requests for that group/version to your extension server.
3. The Metrics Server serves metrics.k8s.io as an aggregated API. It needs aggregation (not a CRD) because metrics are computed on the fly and held in memory rather than stored in etcd — a CRD would persist objects in etcd and can't implement that dynamic, non-etcd-backed, computed serving behavior. (Custom/external metrics adapters are similar.)
4. Choose an aggregated API server when you need capabilities CRDs can't provide: custom/non-etcd storage, computed-on-the-fly responses, specialized validation/business logic on read/write, custom subresources, or protocol-level control. The cost is significant complexity — you must build, deploy, secure, scale, and maintain a real API server (handling auth delegation, TLS, availability), whereas a CRD is declarative and managed by the existing API server. Default to CRDs; use aggregation only when their limitations genuinely block your requirements.

---

Admission webhooks deep dive

Theory

Admission webhooks (introduced in Chapter 9) are a primary API extension point: they let you intercept and influence API requests with custom logic, without modifying Kubernetes. Recapping and going deeper:

• Mutating webhooks modify objects (inject sidecars, set defaults); validating webhooks accept/reject (enforce policy). Mutating runs first.
• Configuration: MutatingWebhookConfiguration/ValidatingWebhookConfiguration specify the rules (which operations/resources/groups to intercept), a clientConfig (the webhook endpoint + CA bundle for TLS), namespaceSelector/objectSelector (scoping), failurePolicy (Fail/Ignore), timeoutSeconds, sideEffects, and reinvocationPolicy.
• Operational pitfalls (critical at scale): webhooks are synchronous and on the API critical path, so a slow/down webhook with failurePolicy: Fail can block API operations cluster-wide; intercepting too broadly (e.g., all Pods including kube-system) can cause outages or deadlocks (a webhook that gates its own dependencies); and they require valid TLS (the API server verifies the webhook's cert against the configured CA).

Webhooks are extremely powerful (they underpin service meshes, policy engines, defaulting), but must be scoped narrowly, made highly available and fast, and fail-safe-considered. A newer in-process alternative — ValidatingAdmissionPolicy (CEL) — avoids the webhook network hop for many validation cases.

Example

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata: { name: pod-policy }
webhooks:
- name: validate.example.com
  rules: [ { operations: ["CREATE"], apiGroups: [""], apiVersions: ["v1"], resources: ["pods"] } ]
  clientConfig: { service: { name: policy, namespace: sys, path: /validate }, caBundle: <CA> }
  namespaceSelector: { matchExpressions: [ { key: kubernetes.io/metadata.name, operator: NotIn, values: [kube-system] } ] }
  failurePolicy: Fail            # consider blast radius if the webhook is down!
  timeoutSeconds: 5
  sideEffects: None
  admissionReviewVersions: ["v1"]

Exercises

1. (Beginner) What two kinds of admission webhooks exist, and which runs first?
2. (Beginner) Why do admission webhooks require TLS?
3. (Intermediate) What can go wrong if a webhook with failurePolicy: Fail becomes unavailable?
4. (Interview) What operational precautions make admission webhooks safe at scale, and what in-process alternative reduces the risk? (Hint: scope narrowly, HA, timeouts, exclude kube-system; ValidatingAdmissionPolicy/CEL.)

Answers

1. Mutating webhooks (modify objects) and validating webhooks (accept/reject); the mutating phase runs first.
2. The API server calls the webhook over HTTPS and verifies its certificate against the configured CA bundle (caBundle), so the connection must use valid TLS — this authenticates the webhook and protects the request/response in transit.
3. Since webhooks are synchronous and on the API request critical path, a Fail policy means that if the webhook is slow or unreachable, the API server rejects the affected requests — potentially blocking creation/updates of those resources cluster-wide (e.g., no new Pods), causing an outage. Overly broad scope can even deadlock (the webhook gating resources it itself depends on).
4. Precautions: scope webhooks narrowly with precise rules and namespaceSelector/objectSelector (exclude kube-system and the webhook's own dependencies); run the webhook service highly available and fast; set sensible timeoutSeconds; choose failurePolicy deliberately (Fail = safe-but-can-block, Ignore = available-but-can-bypass); declare sideEffects: None; and test downtime scenarios. The in-process alternative ValidatingAdmissionPolicy (CEL expressions evaluated inside the API server) removes the external webhook network dependency for many validation use cases, eliminating that critical-path risk.

---

Scheduler extenders and plugins

Theory

The default scheduler (Chapter 8) covers most needs, but sometimes you need custom scheduling logic — special hardware awareness, gang/batch scheduling, custom bin-packing, or external constraints. Kubernetes offers several ways to extend or replace scheduling:

• Scheduler plugins (Scheduling Framework): the modern, preferred approach. The scheduler is built as a framework with extension points (PreFilter, Filter, Score, Reserve, Permit, Bind, etc.); you write plugins in Go that hook into these points and compile them into a scheduler binary. This is efficient (in-process) and powerful.
• Scheduler extenders: an older mechanism where the default scheduler calls out to an external HTTP service (a "extender") during filtering/scoring. Simpler to integrate (any language, no rebuild) but slower (network call per scheduling decision) and more limited.
• Multiple schedulers: you can run your own scheduler alongside the default and have Pods opt in via spec.schedulerName — useful for entirely custom schedulers (e.g., batch schedulers like Volcano/YuniKorn).

The trend favors the Scheduling Framework plugins for performance and flexibility; extenders remain for quick, language-agnostic add-ons; and separate schedulers serve fundamentally different scheduling paradigms.

Example

# Run a Pod with a custom scheduler instead of the default:
spec:
  schedulerName: my-custom-scheduler   # a separate scheduler you run
  containers: [ ... ]

# KubeSchedulerConfiguration enabling a custom framework plugin:
apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
- schedulerName: default-scheduler
  plugins:
    score:
      enabled: [ { name: MyCustomScorePlugin } ]   # framework plugin at Score point

Exercises

1. (Beginner) What is the modern, preferred way to add custom scheduling logic?
2. (Beginner) How does a Pod opt into a non-default scheduler?
3. (Intermediate) What is the main drawback of scheduler extenders versus framework plugins?
4. (Interview) Compare the three approaches (framework plugins, extenders, separate schedulers) and when each fits. (Hint: in-process performance vs. language-agnostic HTTP vs. whole alternative paradigm.)

Answers

1. Writing Scheduling Framework plugins that hook into the scheduler's extension points (PreFilter/Filter/Score/etc.), compiled into the scheduler.
2. By setting spec.schedulerName to the name of the custom scheduler you run; only that scheduler will schedule the Pod.
3. Extenders call an external HTTP service during scheduling, adding network latency per decision (slower) and offering fewer/looser integration points than in-process plugins — so they don't scale as well and are more limited, though they're simpler and language-agnostic.
4. Framework plugins: in-process Go plugins at defined extension points — highest performance and tightest integration; best when you need efficient custom filter/score/bind logic and can build/run a scheduler binary. Extenders: external HTTP callouts from the default scheduler — easiest to add in any language without rebuilding the scheduler, good for quick or third-party integrations, but slower and more limited (per-decision network hop). Separate schedulers: run an entirely different scheduler (opt-in via schedulerName) — appropriate when you need a fundamentally different scheduling paradigm (e.g., gang/batch scheduling for ML/HPC via Volcano/YuniKorn) rather than tweaking the default. Choose plugins for performance-sensitive customization, extenders for simple language-agnostic add-ons, and separate schedulers for distinct scheduling models.

---

Custom metrics API

Theory

The HPA (Chapter 8) can scale on custom and external metrics, but Kubernetes itself doesn't produce those — they're served through metrics APIs implemented as aggregated API servers (tying together two extension points). There are three:

• metrics.k8s.io (resource metrics) — served by Metrics Server.
• custom.metrics.k8s.io (custom metrics tied to Kubernetes objects, e.g., requests-per-second on a Pod/Service).
• external.metrics.k8s.io (external metrics not tied to a cluster object, e.g., a cloud queue length).

To make the HPA scale on, say, RPS or queue depth, you deploy an adapter that implements the custom/external metrics API (as an aggregated API server registered via APIService) and translates from a real metrics backend. The most common is the Prometheus Adapter, which exposes PromQL query results through custom.metrics.k8s.io/external.metrics.k8s.io; KEDA and cloud-specific adapters do similar. So "scale on a custom metric" = run an adapter that serves that metrics API from your monitoring data, then reference the metric in the HPA. This is a concrete, common application of API aggregation in the wild.

Example

# Prometheus Adapter rule: expose a PromQL result as a custom metric
rules:
- seriesQuery: 'http_requests_total{namespace!="",pod!=""}'
  resources: { overrides: { namespace: {resource: "namespace"}, pod: {resource: "pod"} } }
  name: { matches: "http_requests_total", as: "http_requests_per_second" }
  metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'
---

# HPA then scales on that custom metric (see Chapter 8 example)

Exercises

1. (Beginner) Are custom/external metrics produced by Kubernetes itself?
2. (Beginner) What component commonly serves custom metrics from Prometheus data?
3. (Intermediate) How is the custom metrics API related to the aggregated API server extension point?
4. (Interview) Trace what you must deploy to make the HPA scale on a Prometheus-based RPS metric, and why this combines two extension points. (Hint: adapter as aggregated API server exposing custom.metrics.k8s.io; HPA references it.)

Answers

1. No — Kubernetes provides the metrics APIs but not the metric data; you must run adapters that supply custom/external metrics from a real backend.
2. The Prometheus Adapter.
3. The custom (and external) metrics APIs are themselves implemented as aggregated API servers: an adapter registers via an APIService so requests for custom.metrics.k8s.io/external.metrics.k8s.io are proxied to it, and it serves metric values computed from a backend. So consuming custom metrics relies directly on the API aggregation extension point.
4. Deploy a metrics backend (Prometheus) scraping your app's metrics, then deploy the Prometheus Adapter configured to translate a PromQL query (e.g., rate of http_requests_total) into a custom metric (e.g., http_requests_per_second); the adapter registers as an aggregated API server serving custom.metrics.k8s.io. Then define an HPA referencing that custom metric/target. This combines two extension points: API aggregation (the adapter is an aggregated API server exposing the custom metrics API) and the HPA's pluggable metrics consumption (it queries that API). Together they let the HPA scale on an application-meaningful signal sourced from your monitoring system rather than just CPU/memory.

---

14.4 Kubernetes Client Libraries

Building controllers, operators, and tools requires programmatic API access. This subchapter covers the client libraries that make it possible.

---

client-go overview

Theory

client-go is the official Go client library for the Kubernetes API — the foundational SDK that virtually all Go-based Kubernetes tooling (including the components of Kubernetes itself, kubectl, controllers, and operators) is built on. It provides everything needed to talk to the API programmatically: authenticated clients, typed and dynamic access to resources, watching, caching, and leader election.

Key pieces of client-go:

• Clientsets: typed Go clients for built-in resources (e.g., clientset.AppsV1().Deployments(ns).Get(...)) — compile-time-checked, ergonomic.
• Dynamic client / discovery: work with arbitrary resources (including CRDs) without compile-time types, using unstructured objects — essential for generic tooling.
• Informers and listers (next topic): efficient cached watching of resources.
• RESTConfig / kubeconfig loading: build a client from in-cluster config or a kubeconfig file.
• workqueue, leaderelection utilities for building controllers.

client-go is the bedrock of controller-runtime (which wraps it for operators). Understanding it — typed vs. dynamic clients, and the informer machinery — is foundational for writing any serious Kubernetes automation in Go.

Example

import ("k8s.io/client-go/kubernetes"; "k8s.io/client-go/tools/clientcmd")

cfg, _ := clientcmd.BuildConfigFromFlags("", kubeconfigPath)  // or rest.InClusterConfig()
cs, _ := kubernetes.NewForConfig(cfg)                         // typed clientset

// Typed access to a built-in resource:
deploy, _ := cs.AppsV1().Deployments("default").Get(ctx, "web", metav1.GetOptions{})
fmt.Println(*deploy.Spec.Replicas)

Exercises

1. (Beginner) What is client-go?
2. (Beginner) What is the difference between a typed clientset and the dynamic client?
3. (Intermediate) When would you need the dynamic client instead of a typed clientset?
4. (Interview) Why is client-go considered foundational for the Kubernetes Go ecosystem, and what core capabilities does it provide for building controllers? (Hint: used by K8s/kubectl/operators; clients, informers, workqueues, leader election.)

Answers

1. The official Go client library for the Kubernetes API — the SDK used to programmatically authenticate to and interact with the cluster, underpinning most Go Kubernetes tooling.
2. A typed clientset provides compile-time-checked, strongly-typed access to known (built-in) resources (e.g., Deployments); the dynamic client works with arbitrary resources (including CRDs) generically via unstructured objects without compile-time types.
3. When you need to handle resources whose types aren't known at compile time — e.g., arbitrary CRDs, generic tooling that operates across many resource kinds, or code that must work with custom resources it doesn't import Go types for. The dynamic client handles them via unstructured data and discovery.
4. Nearly all Go-based Kubernetes software — Kubernetes components, kubectl, controllers, and operators (via controller-runtime, which wraps client-go) — is built on it, making it the common foundation of the ecosystem. For controllers it provides authenticated typed/dynamic clients, informers/listers for efficient cached watching, workqueues for rate-limited reconcile processing, leader election for HA controllers, and config loading (in-cluster or kubeconfig). These are exactly the building blocks needed to implement the watch-cache-reconcile pattern reliably.

---

Informers and work queues

Theory

Naively, a controller could poll the API server or open raw watches — but that's inefficient and fragile at scale. Informers are client-go's solution: an informer maintains a local in-memory cache of a resource type, populated by an initial LIST and kept current by a WATCH (the LIST-WATCH pattern from Chapter 2). Controllers read from this cache (via a lister) instead of hitting the API server repeatedly, and register event handlers (OnAdd/OnUpdate/OnDelete) to react to changes. SharedInformers let many controllers share one cache/watch per resource, avoiding duplication.

Informers pair with work queues: instead of doing work directly in the event handler, the handler simply enqueues the object's key onto a rate-limited work queue; worker goroutines dequeue keys and run reconcile. This decoupling is crucial because it provides:

• De-duplication: multiple rapid events for the same object collapse into one queue entry.
• Rate limiting and retries with backoff: failed items are requeued with exponential backoff.
• Level-triggered processing: workers re-fetch current state from the cache by key (not stale event data), reconciling against reality.

This informer + workqueue architecture is the standard controller pattern — efficient, resilient, and the basis of controller-runtime. It's why controllers scale and self-heal.

Example

SharedInformer (cache) <--LIST/WATCH-- API server
     | OnAdd/OnUpdate/OnDelete: enqueue object KEY
     v
  WorkQueue (dedup + rate-limit + retry-with-backoff)
     |
     v  worker goroutine
  Reconcile(key): lister.Get(key) from cache -> compare desired/actual -> act

informer.AddEventHandler(cache.ResourceEventHandlerFuncs{
  AddFunc:    func(o interface{}) { queue.Add(keyOf(o)) },   // enqueue key, don't do work here
  UpdateFunc: func(_, n interface{}) { queue.Add(keyOf(n)) },
})

Exercises

1. (Beginner) What does an informer maintain, and how is it kept current?
2. (Beginner) Why do controllers read from a lister/cache instead of the API server directly?
3. (Intermediate) Why do event handlers enqueue a key onto a work queue rather than doing the work inline?
4. (Interview) How do informers + work queues together produce an efficient, resilient, level-triggered controller? (Hint: shared cache reduces API load; queue dedups/retries; workers reconcile current state by key.)

Answers

1. A local in-memory cache of a resource type, populated by an initial LIST and kept up to date by a WATCH (LIST-WATCH), with event handlers firing on changes.
2. To avoid hammering the API server: reads served from the local cache are fast and cheap and reduce load on the API server/etcd, while the cache stays current via the watch. (SharedInformers let multiple controllers share one cache/watch.)
3. To decouple event reception from work execution, gaining de-duplication (collapse rapid repeated events for the same object into one key), rate limiting, and retries with exponential backoff on failure. It also keeps processing level-triggered: the worker later fetches the object's current state from the cache by key rather than acting on possibly-stale event payloads.
4. Informers provide a shared, always-current local cache (efficient: one LIST-WATCH feeds many readers, minimizing API load). Event handlers enqueue only the object's key into a work queue that de-duplicates and rate-limits, and retries failures with backoff (resilient). Worker goroutines dequeue keys and reconcile by re-reading current state from the cache and acting on the desired-vs-actual difference (level-triggered) — so missed/duplicate events don't matter and transient failures are retried. The combination yields controllers that are low-overhead, robust to bursts and failures, and continuously convergent — the standard, scalable controller architecture.

---

Python, Java, and other language clients

Theory

While Go (client-go) is the native and dominant language for Kubernetes development, Kubernetes provides official client libraries for many languages — Python, Java, JavaScript/TypeScript, C#/.NET, and more — plus community clients. These let you automate Kubernetes, build tools, or even write controllers/operators in your team's preferred language.

Key points:

• The Python client (kubernetes) is popular for scripting, automation, data/ML workflows (e.g., Kubeflow tooling), and quick operational tools.
• The Java and other clients suit teams with existing stacks building Kubernetes integrations.
• Many official clients are partly auto-generated from the Kubernetes OpenAPI spec, so they stay consistent across versions and cover the full API.
• Trade-off: non-Go clients are great for scripting and applications, but the richest controller/operator tooling (informers, workqueues, controller-runtime, Kubebuilder) lives in the Go ecosystem. You can build controllers in Python/Java (some clients offer watch/informer-like helpers), but Go remains the path of least resistance for serious controllers.

So: pick the client that fits your language/use case for automation and apps; gravitate to Go for production controllers/operators where the ecosystem is deepest.

Example

# Python client: list Pods in a namespace
from kubernetes import client, config
config.load_kube_config()                      # or load_incluster_config()
v1 = client.CoreV1Api()
for pod in v1.list_namespaced_pod("default").items:
    print(pod.metadata.name, pod.status.phase)

Exercises

1. (Beginner) Name three languages with official Kubernetes client libraries.
2. (Beginner) Which language is most common for serious controller/operator development, and why?
3. (Intermediate) Why are many official clients auto-generated, and what benefit does that bring?
4. (Interview) When would you choose the Python client over Go, and what limitations should you keep in mind for controllers? (Hint: scripting/automation/ML fit; controller tooling richest in Go.)

Answers

1. Any three: Python, Java, JavaScript/TypeScript, C#/.NET (also Go natively).
2. Go (client-go), because the deepest, most mature controller/operator tooling — informers, work queues, controller-runtime, Kubebuilder, Operator SDK — is in the Go ecosystem, and Kubernetes itself is written in Go.
3. They're generated from the Kubernetes OpenAPI specification, so they comprehensively and consistently cover the full API and stay in sync across Kubernetes versions with less manual maintenance — giving reliable, complete coverage across languages.
4. Choose Python (or another non-Go client) for scripting, operational automation, glue code, and application/ML workflows (e.g., Kubeflow) where it fits your team's stack and the task is interacting with the API rather than building a heavy controller. Keep in mind that controller-specific machinery (informers/work queues/controller-runtime, scaffolding) is far richer and more battle-tested in Go; while you can write controllers in Python/Java (some clients offer watch/informer helpers), you'll have less tooling and may reimplement patterns that Go gives you for free — so for production-grade operators, Go is usually preferable.

---

controller-runtime library

Theory

controller-runtime is the higher-level Go library (a Kubernetes SIG project) that sits on top of client-go and provides the batteries-included framework for building controllers and operators. It's what Kubebuilder and Operator SDK scaffold against. Where client-go gives you the raw building blocks (clients, informers, workqueues), controller-runtime assembles them into an ergonomic, opinionated framework so you write far less plumbing.

What it provides:

• Manager: wires up shared caches/informers, clients, leader election, metrics, and health checks, and runs your controllers.
• Controller/Reconciler abstraction: you implement a simple Reconcile(ctx, req) method; the framework handles watches, the work queue, requeuing, and calling you with object keys.
• Client with caching: a unified client that reads from the shared cache and writes to the API, plus the Status().Update() for status subresources.
• Builder API: declaratively set up which resources a controller watches and owns (so changes to owned resources, like Pods created by your CR, trigger reconciliation).

The result: you focus almost entirely on reconcile logic and API types, while controller-runtime handles the informer/workqueue/leader-election/caching machinery correctly. It's the de facto standard for operator development in Go.

Example

// Set up a controller with controller-runtime's builder:
ctrl.NewControllerManagedBy(mgr).
    For(&examplev1.Backup{}).        // watch Backup CRs (primary)
    Owns(&batchv1.Job{}).            // also reconcile when owned Jobs change
    Complete(&BackupReconciler{Client: mgr.GetClient()})

// You implement just the reconcile logic:
func (r *BackupReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    // get the Backup, observe Jobs, act on the difference, update status
    return ctrl.Result{}, nil
}

Exercises

1. (Beginner) What does controller-runtime sit on top of, and what do Kubebuilder/Operator SDK use?
2. (Beginner) What does the Manager component do?
3. (Intermediate) What does .Owns(...) accomplish in the controller builder?
4. (Interview) How does controller-runtime let developers focus on reconcile logic, and what plumbing does it handle for them? (Hint: manager wires caches/informers/workqueue/leader election; you implement Reconcile.)

Answers

1. It sits on top of client-go; Kubebuilder and Operator SDK scaffold projects that build against controller-runtime.
2. The Manager wires together and runs the shared caches/informers, clients, leader election, metrics, and health endpoints, and starts the registered controllers — providing the runtime that hosts your reconcilers.
3. .Owns(<kind>) tells the controller it owns resources of that kind (created/managed by its primary resource via ownerReferences), so changes to those owned objects (e.g., a Job the CR created) trigger reconciliation of the owning custom resource — keeping the parent's reconcile in sync with its children.
4. controller-runtime provides a Manager that sets up shared informers/caches (efficient watching), the work queue and requeue/backoff handling, a cached read + write client, leader election for HA, and metrics/health — and a Builder to declare what to watch/own. You only implement the Reconcile(ctx, req) method (and define your API types). All the watch-cache-enqueue-retry-leaderelection machinery that you'd otherwise assemble from client-go is handled by the framework, so development focuses on the business/reconcile logic and API design rather than controller plumbing — which is exactly why it's the standard for Go operators.

---