9. Security

Kubernetes security is layered, and a breach at any layer can compromise the cluster. This chapter walks the request as it enters the API server — authentication (who are you?), authorization (what may you do?), admission control (does this comply with policy?) — then covers what a Pod is allowed to do at runtime (Pod security) and the supply chain that produced its image. The throughline is least privilege and defense in depth.

---

9.1 Authentication

Authentication is the first gate: the API server must establish who (or what) is making each request. This subchapter covers the mechanisms Kubernetes supports.

---

X.509 client certificates

Theory

The most fundamental authentication method is X.509 client certificates. The API server is configured to trust a Certificate Authority (CA); any client presenting a certificate signed by that CA is authenticated. The certificate's Common Name (CN) becomes the username, and its Organization (O) fields become the user's groups. This is how the cluster-admin kubeconfig from kubeadm works — it contains an admin client certificate.

This mechanism is robust and simple but has a significant operational drawback: certificates cannot be easily revoked (Kubernetes doesn't check CRLs/OCSP), so a leaked cert is valid until it expires. This makes long-lived client certs risky for human users; they're better suited to components and short-lived issuance. Certificates can be requested through the CertificateSigningRequest (CSR) API, where the cluster's CA signs an approved request. For human users at scale, OIDC (covered later) is usually preferred over per-user certs.

Example

# A client cert's subject determines identity:
#   CN=alice, O=developers  ->  user "alice" in group "developers"
openssl x509 -in alice.crt -noout -subject
# subject=CN = alice, O = developers

# Request a cert via the CSR API (approver signs it):
kubectl get csr
kubectl certificate approve alice-csr

Exercises

1. (Beginner) In an X.509 client certificate, which field becomes the username and which becomes the groups?
2. (Beginner) What must the API server trust for certificate authentication to work?
3. (Intermediate) Why are leaked client certificates particularly dangerous in Kubernetes?
4. (Interview) Why are long-lived X.509 certs discouraged for human users, and what is generally preferred instead? (Hint: no revocation; OIDC/central identity.)

Answers

1. The Common Name (CN) becomes the username; the Organization (O) field(s) become the user's groups.
2. A Certificate Authority (CA) whose signature on client certs it will accept (configured as the client CA bundle).
3. Kubernetes does not check certificate revocation (no CRL/OCSP), so a leaked certificate remains valid until it expires — there's no easy way to revoke it short of rotating the CA. A long-lived leaked cert thus grants persistent access.
4. Because they can't be revoked and are hard to rotate per user, long-lived certs are a standing risk for humans. OIDC integration with a central identity provider is preferred: it issues short-lived tokens, supports central revocation/MFA/group management, and ties cluster access to existing corporate identity — far more manageable and secure for human users at scale.

---

Bearer tokens and ServiceAccount tokens

Theory

Another authentication method is bearer tokens: the client sends Authorization: Bearer <token> and the API server validates it. The most important kind is the ServiceAccount (SA) token — the identity mechanism for in-cluster workloads (as opposed to human users). Every Pod is associated with a ServiceAccount (the default one if unspecified), and a token for that SA is mounted into the Pod so the application can authenticate to the API server.

Modern Kubernetes uses bound, projected SA tokens: short-lived, audience-scoped JWTs auto-mounted via a projected volume and auto-rotated by the kubelet — a major security improvement over the old long-lived token Secrets (which never expired). A SA's identity is system:serviceaccount:<namespace>:<name> and it belongs to groups like system:serviceaccounts. ServiceAccounts are the in-cluster counterpart to user accounts and are what you grant RBAC permissions to for controllers, operators, and apps that call the API.

Example

apiVersion: v1
kind: ServiceAccount
metadata: { name: app-sa, namespace: prod }
---

spec:
  serviceAccountName: app-sa          # Pod runs as this identity
  containers: [ ... ]

# The projected token is mounted here inside the Pod:
cat /var/run/secrets/kubernetes.io/serviceaccount/token   # a short-lived JWT
# Identity becomes: system:serviceaccount:prod:app-sa

# Mint a short-lived token for an SA on demand:
kubectl create token app-sa -n prod --duration=1h

Exercises

1. (Beginner) What identity do in-cluster workloads use to authenticate to the API server?
2. (Beginner) What is the format of a ServiceAccount's username?
3. (Intermediate) How do modern bound/projected SA tokens improve on the legacy SA token Secrets?
4. (Interview) Why should you avoid using the default ServiceAccount with broad permissions for application Pods? (Hint: least privilege, per-workload identity, blast radius.)

Answers

1. A ServiceAccount (its bearer token), mounted into the Pod.
2. system:serviceaccount:<namespace>:<name>.
3. Bound/projected tokens are short-lived (have an expiry), audience-scoped, and automatically rotated by the kubelet, and they're bound to the Pod's lifetime. Legacy token Secrets were long-lived, non-expiring, stored in etcd, and not audience-bound — so a leak granted indefinite, broad access. The bound model drastically reduces credential lifetime and blast radius.
4. Every Pod gets an identity; if you use the default SA and grant it permissions, all Pods in the namespace inherit them, violating least privilege and widening the blast radius if any Pod is compromised. Instead, create a dedicated ServiceAccount per workload with only the permissions it needs (and disable token automounting where the API isn't used), so each workload's access is minimal and isolated.

---

OIDC integration

Theory

For human users, the recommended approach is OIDC (OpenID Connect) integration. The API server is configured to trust an external identity provider (IdP) — Google, Azure AD/Entra, Okta, Keycloak, Dex. Users authenticate with the IdP (with SSO, MFA, etc.) and obtain an ID token (JWT); they present this token to the API server, which validates its signature and expiry against the IdP and extracts the username and groups from configured claims.

This is superior to per-user certificates because: tokens are short-lived (limiting leak impact), identity and group membership are managed centrally in the IdP (add/remove users in one place), and you get MFA and corporate SSO for free. RBAC rules then reference the usernames/groups from the IdP claims. OIDC is the standard for enterprise human access; it requires no in-cluster user objects — the IdP is the source of truth for identity, while Kubernetes handles authorization.

Example

# API server flags (conceptual):
--oidc-issuer-url=https://accounts.example.com
--oidc-client-id=kubernetes
--oidc-username-claim=email
--oidc-groups-claim=groups

# kubectl uses an OIDC exec/auth plugin to obtain & refresh the ID token,
# then sends it as a bearer token. RBAC then matches on the claims:
#   user: alice@example.com   groups: [ platform-admins ]

Exercises

1. (Beginner) Who is OIDC integration primarily for — human users or in-cluster workloads?
2. (Beginner) What does the API server extract from a validated OIDC token to make authorization decisions?
3. (Intermediate) Name two advantages of OIDC over per-user X.509 certificates.
4. (Interview) In an OIDC setup, where is the source of truth for user identity and group membership, and how does that simplify access management at scale? (Hint: central IdP, no per-user K8s objects.)

Answers

1. Human users.
2. The username and groups (from the configured token claims, e.g., email and groups), which RBAC then uses.
3. Any two: tokens are short-lived (limiting leak impact, unlike non-revocable certs); identity/groups are managed centrally in the IdP (add/remove/disable users in one place); built-in SSO and MFA; easier auditing — all without minting and distributing per-user certs.
4. The external identity provider (IdP) is the source of truth — users and groups live there, not as Kubernetes objects. To grant or revoke access at scale you manage users/groups in the IdP and write RBAC rules against group claims (e.g., bind a Role to group platform-admins). Onboarding/offboarding and MFA are handled centrally, so the cluster needs no per-user provisioning and access stays consistent with corporate identity.

---

Webhook authentication

Theory

When the built-in methods don't fit, webhook token authentication lets you delegate authentication to an external service. The API server is configured with a webhook endpoint; when it receives a bearer token it doesn't otherwise recognize, it sends a TokenReview request to the webhook, which validates the token however it likes and returns whether it's authenticated plus the username/groups.

This provides maximum flexibility — you can integrate a custom or proprietary identity system, a cloud provider's IAM (this is how managed clusters like EKS map AWS IAM identities to Kubernetes users), or any token format. The trade-off is that the webhook is on the critical path for authentication: if it's slow or down, requests using that method fail, so it must be highly available and fast. Webhook auth is a power-user/integration feature; most clusters use certs, SA tokens, and OIDC.

Example

Client --(Bearer <opaque-token>)--> API server
   API server --(TokenReview)--> external webhook (validates token)
   webhook --> { authenticated: true, user: "bob", groups: ["team-x"] }
   API server proceeds to authorization as user "bob"

# TokenReview response shape the webhook returns:
apiVersion: authentication.k8s.io/v1
kind: TokenReview
status:
  authenticated: true
  user:
    username: "bob"
    groups: ["team-x"]

Exercises

1. (Beginner) What does webhook authentication delegate, and to what?
2. (Beginner) What API object does the API server send to the webhook to validate a token?
3. (Intermediate) Give a real-world example of webhook authentication in managed Kubernetes.
4. (Interview) What operational risk does webhook authentication introduce, and how do you mitigate it? (Hint: critical-path dependency; HA, low latency, caching.)

Answers

1. It delegates token authentication to an external service (an HTTP webhook) that validates the token and returns the identity.
2. A TokenReview (the webhook responds with authenticated plus username/groups).
3. EKS maps AWS IAM identities to Kubernetes users via a webhook/authenticator (e.g., the AWS IAM Authenticator), so IAM users/roles authenticate to the cluster. (Other clouds similarly bridge their IAM to Kubernetes auth.)
4. The webhook becomes a critical-path dependency: if it's slow or unavailable, authentications relying on it fail, potentially locking users out. Mitigate by running the webhook highly available and low-latency, setting sensible timeouts, enabling the API server's authentication token cache (TTL) to reduce calls, and ensuring an alternative admin access path (e.g., a cert-based break-glass credential) exists in case the webhook is down.

---

9.2 Authorization

Once identity is established, authorization decides what that identity may do. RBAC is the dominant mechanism. This subchapter covers it and the principle of least privilege.

---

RBAC overview

Theory

RBAC (Role-Based Access Control) is the standard authorization mode in Kubernetes. It answers: "may this identity perform this verb (get, list, create, update, delete, watch...) on this resource (pods, services, secrets...) in this scope (namespace or cluster)?" RBAC is additive and deny-by-default: with no rule granting access, access is denied; you can only grant, never explicitly deny (denial is the absence of a grant).

RBAC is built from four object types, in two pairs:

• Role / ClusterRole: define a set of permissions (rules). Role is namespaced; ClusterRole is cluster-wide.
• RoleBinding / ClusterRoleBinding: bind a Role/ClusterRole to subjects (users, groups, ServiceAccounts), granting them those permissions.

The mental model: a Role says what can be done; a Binding says who can do it. RBAC is evaluated on every request after authentication, and is the primary tool for enforcing least privilege in a cluster.

Example

Permissions model:
  (subject: user/group/SA)  --bound by-->  RoleBinding  --references-->  Role
                                                                          |
                                                          rules: verbs on resources

kubectl auth can-i list pods -n dev                       # check your own access
kubectl auth can-i '*' '*' --all-namespaces               # am I cluster-admin?

Exercises

1. (Beginner) What three things does an RBAC rule combine to define a permission?
2. (Beginner) Is RBAC deny-by-default or allow-by-default? Can you write an explicit deny rule?
3. (Intermediate) What is the difference in responsibility between a Role and a RoleBinding?
4. (Interview) Explain how RBAC's additive, deny-by-default model supports least privilege and why the absence of explicit deny rules is acceptable. (Hint: grant only what's needed; everything else is implicitly denied.)

Answers

1. Verbs (actions like get/list/create/delete), resources (pods, secrets, etc.), and the API groups they belong to — evaluated within a namespace or cluster scope.
2. Deny-by-default. You cannot write explicit deny rules — RBAC only grants; anything not granted is denied.
3. A Role (or ClusterRole) defines a set of permissions (what actions on what resources). A RoleBinding (or ClusterRoleBinding) assigns that Role to subjects (who gets those permissions). One is the capability definition; the other is the grant to identities.
4. Because access is denied unless a rule grants it, you start from zero and add only the specific permissions each identity needs — the essence of least privilege. There's no need for explicit deny rules: anything you don't grant is already denied, so security is enforced by carefully limiting grants rather than enumerating prohibitions. This makes policies simpler to reason about (you audit what's allowed) and fails safe (a forgotten grant means no access, not accidental access).

---

Roles and ClusterRoles

Theory

A Role is a namespaced set of permission rules — it can only grant access to resources within its own namespace. A ClusterRole is the cluster-scoped equivalent, and it serves several purposes a Role cannot:

1. Grant access to cluster-scoped resources (nodes, PersistentVolumes, namespaces themselves).
2. Grant access to non-resource URLs (e.g., /healthz, /metrics).
3. Be reused across namespaces — a single ClusterRole can be bound (via RoleBindings) in many namespaces, avoiding duplicating the same Role everywhere.

Each rule lists apiGroups, resources, and verbs (and optionally resourceNames to restrict to specific named objects). Kubernetes ships default ClusterRoles like view, edit, admin, and cluster-admin — useful building blocks. Choosing Role vs. ClusterRole comes down to scope: namespace-local permissions use Roles; anything cluster-wide or reused broadly uses ClusterRoles.

Example

# Namespaced Role: read pods in the "dev" namespace only
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { namespace: dev, name: pod-reader }
rules:
- apiGroups: [""]            # "" = core API group
  resources: ["pods"]
  verbs: ["get", "list", "watch"]
---

# ClusterRole: read nodes (a cluster-scoped resource)
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata: { name: node-reader }
rules:
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get", "list", "watch"]

Exercises

1. (Beginner) What is the scope difference between a Role and a ClusterRole?
2. (Beginner) Name two things a ClusterRole can grant access to that a Role cannot.
3. (Intermediate) Why might you define a permission set as a ClusterRole even if you'll use it within namespaces?
4. (Interview) Name the default ClusterRoles Kubernetes ships and describe when you'd bind view vs. edit vs. admin. (Hint: read-only, read-write within ns, manage ns incl. RBAC.)

Answers

1. A Role grants permissions only within its own namespace; a ClusterRole grants permissions cluster-wide (and can be applied across namespaces).
2. Any two: cluster-scoped resources (nodes, PersistentVolumes, namespaces), non-resource URLs (/healthz, /metrics), and reuse of one role definition across multiple namespaces.
3. To define the permission set once and reuse it: a single ClusterRole can be bound via a RoleBinding in each namespace, avoiding copying an identical Role into every namespace and keeping the definition DRY and consistent.
4. Defaults: view, edit, admin, cluster-admin. Bind view for read-only access to most namespaced resources (no secrets, no write). Bind edit for read-write on most namespaced resources (can modify workloads/config but not manage RBAC). Bind admin for full control within a namespace including managing Roles/RoleBindings (but not cluster-scoped resources). (cluster-admin is unrestricted superuser — use sparingly.)

---

RoleBindings and ClusterRoleBindings

Theory

A Role/ClusterRole alone grants nothing until bound to subjects. The binding objects:

• RoleBinding: grants the permissions of a Role or a ClusterRole to subjects within a specific namespace. (Yes — a RoleBinding can reference a ClusterRole, in which case the ClusterRole's permissions are limited to that namespace. This is the reuse pattern.)
• ClusterRoleBinding: grants a ClusterRole's permissions to subjects across the entire cluster.

Subjects are users, groups, or ServiceAccounts. The crucial scoping rule: the binding type determines the scope of the grant, not the role type. A ClusterRole bound via a RoleBinding only applies in that namespace; the same ClusterRole bound via a ClusterRoleBinding applies everywhere. A common mistake — granting cluster-wide access by using a ClusterRoleBinding when a namespaced RoleBinding was intended — is a frequent over-privilege bug.

Example

# Bind the ClusterRole "node-reader" cluster-wide to a group:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata: { name: read-nodes }
subjects:
- { kind: Group, name: platform-team, apiGroup: rbac.authorization.k8s.io }
roleRef: { kind: ClusterRole, name: node-reader, apiGroup: rbac.authorization.k8s.io }
---

# Bind a ClusterRole but scope it to ONE namespace via a RoleBinding:
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { namespace: dev, name: dev-edit }
subjects:
- { kind: ServiceAccount, name: ci, namespace: dev }
roleRef: { kind: ClusterRole, name: edit, apiGroup: rbac.authorization.k8s.io }

Exercises

1. (Beginner) What are the three kinds of subjects a binding can reference?
2. (Beginner) Does a RoleBinding determine scope by the Role's type or the binding's type?
3. (Intermediate) Explain what happens when a RoleBinding references a ClusterRole.
4. (Interview) A team intended to give a ServiceAccount edit access in one namespace but accidentally granted it cluster-wide. What was the likely mistake, and how is it fixed? (Hint: ClusterRoleBinding vs. RoleBinding.)

Answers

1. Users, groups, and ServiceAccounts.
2. By the binding's type/scope: a RoleBinding grants within its namespace regardless of whether it references a Role or a ClusterRole.
3. The ClusterRole's permission set is applied, but scoped to the RoleBinding's namespace only. This lets you reuse a single ClusterRole definition across namespaces by binding it with a RoleBinding in each, without granting cluster-wide access.
4. They likely used a ClusterRoleBinding (which grants cluster-wide) instead of a RoleBinding in the target namespace. Fix it by deleting the ClusterRoleBinding and creating a RoleBinding in the intended namespace that references the same role (e.g., the edit ClusterRole) and the ServiceAccount subject — confining the grant to that one namespace.

---

ABAC and Webhook authorization

Theory

RBAC is the default, but Kubernetes supports other authorization modes, often chained (a request is allowed if any enabled authorizer allows it):

• ABAC (Attribute-Based Access Control): an older mode where access rules are defined in a policy file of attributes (user, resource, namespace, verb) on the API server's disk. Changing policy requires editing the file and restarting the API server, which makes it inflexible and largely deprecated in favor of RBAC.
• Webhook authorization: delegates the authorization decision to an external service via a SubjectAccessReview — similar to webhook authentication but for "may they do this?" This enables centralized or custom policy engines and is how some clouds integrate their IAM for authorization.
• Node authorizer: a special-purpose mode that authorizes kubelets to access only the resources their own Pods need.

In practice, clusters run RBAC (plus the Node authorizer), and reach for webhook authorization when external policy integration is required. ABAC is essentially legacy.

Example

--authorization-mode=Node,RBAC[,Webhook]   # authorizers tried in order; any allow = allow

# Webhook authorization: API server asks an external service via SubjectAccessReview
apiVersion: authorization.k8s.io/v1
kind: SubjectAccessReview
spec:
  user: "alice"
  resourceAttributes: { namespace: "prod", verb: "delete", resource: "pods" }
# external authorizer responds: { status: { allowed: false } }

Exercises

1. (Beginner) Why is ABAC considered inflexible compared to RBAC?
2. (Beginner) What review object does webhook authorization use?
3. (Intermediate) When multiple authorization modes are enabled, how is the final allow/deny decided?
4. (Interview) Why might an organization add webhook authorization alongside RBAC, and what is the cost? (Hint: centralized/custom policy; external critical-path dependency.)

Answers

1. ABAC rules live in a policy file on the API server's disk; changing them requires editing the file and restarting the API server, with no dynamic, API-driven management — unlike RBAC, which is managed as live API objects (kubectl apply) without restarts.
2. A SubjectAccessReview (the external authorizer responds allowed/denied).
3. The enabled authorizers are consulted in order; if any of them allows the request, it is allowed. It's denied only if none allow it (and an explicit deny from one authorizer can short-circuit, but generally the union of allows applies).
4. To integrate centralized or custom authorization policy (e.g., a company-wide policy engine or cloud IAM) that RBAC alone can't express, enabling consistent cross-system access decisions. The cost: the webhook is on the authorization critical path — added latency per request and a dependency that, if slow or down, can disrupt access — so it must be highly available and fast, typically with caching.

---

Least privilege principle

Theory

The principle of least privilege (PoLP) is the guiding philosophy for all the above: every identity (user, ServiceAccount, controller) should have only the permissions it needs to do its job — no more. In Kubernetes this means: prefer narrowly-scoped Roles over broad ClusterRoles, prefer RoleBindings (namespaced) over ClusterRoleBindings, grant specific verbs/resources rather than wildcards, and give each workload its own ServiceAccount rather than sharing privileged ones.

Why it matters: Kubernetes is a high-value target, and over-privileged identities massively increase the blast radius of any compromise or mistake. A leaked token for a least-privilege SA can do little; a leaked cluster-admin token owns everything. Practical habits: avoid cluster-admin except for true administrators; never grant secrets read broadly; audit RBAC regularly (tools like kubectl auth can-i --list, rbac-tool, kubeaudit); and disable SA token automounting for Pods that don't call the API. Least privilege is the single highest-leverage security practice in a cluster.

Example

# Least privilege: a dedicated SA that may only read one ConfigMap, nothing else
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { namespace: app, name: read-app-config }
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  resourceNames: ["app-config"]     # restrict to a SPECIFIC object
  verbs: ["get"]

# Don't auto-mount a token if the Pod never calls the API:
spec:
  serviceAccountName: app-sa
  automountServiceAccountToken: false

Exercises

1. (Beginner) State the principle of least privilege in one sentence.
2. (Beginner) Name two concrete RBAC practices that follow least privilege.
3. (Intermediate) How does resourceNames help enforce least privilege?
4. (Interview) Explain "blast radius" in the context of RBAC and why per-workload ServiceAccounts with minimal permissions reduce it. (Hint: compromise of one identity should grant minimal access.)

Answers

1. Grant every identity only the minimum permissions required to perform its function, and nothing more.
2. Any two: use namespaced Roles + RoleBindings instead of cluster-wide grants; specify exact verbs/resources instead of wildcards; give each workload its own ServiceAccount; use resourceNames to restrict to specific objects; avoid cluster-admin; disable token automounting when unused; audit RBAC regularly.
3. resourceNames restricts a rule to specific named object(s) rather than all objects of a type — e.g., allow get on only the app-config ConfigMap instead of all ConfigMaps. This narrows access to exactly what's needed.
4. "Blast radius" is the extent of damage possible if an identity is compromised. A cluster-admin token, if leaked, lets an attacker do anything cluster-wide (read all secrets, run privileged Pods, take over nodes) — huge blast radius. A per-workload ServiceAccount scoped to only the few resources/verbs that workload needs grants an attacker almost nothing if its token leaks — tiny blast radius. Minimizing each identity's permissions, and isolating them per workload, ensures any single compromise is contained, which is the core defensive value of least privilege.

---

9.3 Admission Control

After authn and authz, admission controllers enforce policy on what may be created or how. This subchapter covers built-in controllers, webhooks, and policy engines.

---

Admission controller types

Theory

Admission controllers intercept requests to the API server after authentication and authorization but before persistence (Chapter 2 introduced the pipeline). There are two functional types, run in two phases:

• Mutating admission controllers can modify the object (set defaults, inject sidecars, add labels). They run first.
• Validating admission controllers can only accept or reject (enforce policy), not modify. They run second, after mutation.

Kubernetes ships many built-in (compiled-in) admission controllers enabled by default — e.g., NamespaceLifecycle, LimitRanger, ResourceQuota, ServiceAccount, DefaultStorageClass, PodSecurity, MutatingAdmissionWebhook, ValidatingAdmissionWebhook. Beyond these, the two webhook controllers are the extension points that let you plug in custom mutating/validating logic (next topics). Admission control is where organizational policy ("no privileged Pods," "all images from our registry," "every Pod must have resource limits") is enforced at the cluster's front door.

Example

Request (after authz)
   |
   v   MUTATING phase (can change the object)
[ MutatingAdmissionWebhook, DefaultStorageClass, ServiceAccount, ... ]
   |
   v   VALIDATING phase (accept/reject only)
[ ValidatingAdmissionWebhook, ResourceQuota, PodSecurity, ... ]
   |
   v  persist to etcd (only if not rejected)

# See which admission plugins are enabled (on the API server):
kube-apiserver -h | grep enable-admission-plugins

Exercises

1. (Beginner) What are the two functional types of admission controllers?
2. (Beginner) Which phase runs first, and why?
3. (Intermediate) Name three built-in admission controllers and what each enforces/does.
4. (Interview) Admission control is described as enforcing organizational policy "at the front door." Give two example policies and say whether each needs a mutating or validating controller. (Hint: inject = mutate; forbid = validate.)

Answers

1. Mutating (can modify the object) and validating (can only accept/reject).
2. The mutating phase runs first, so that validating controllers evaluate the final, post-mutation object (e.g., after defaults/sidecars are injected).
3. Examples: ResourceQuota (enforces namespace aggregate quotas), LimitRanger (applies default/min/max resource values), PodSecurity (enforces Pod Security Standards), ServiceAccount (assigns/automounts SA tokens), DefaultStorageClass (sets default class on PVCs), NamespaceLifecycle (prevents creating objects in terminating namespaces).
4. Examples: "Inject a logging sidecar into every Pod" → mutating (modifies the object). "Reject any Pod that runs as privileged" → validating (rejects, doesn't modify). "Set a default resource request if missing" → mutating. "Require all images come from our registry" → validating (reject if not). Generally, adding/changing fields = mutating; allowing/denying based on rules = validating.

---

ValidatingAdmissionWebhook

Theory

A ValidatingAdmissionWebhook lets you enforce custom policy by having the API server call your external webhook to approve or reject a request. You register a ValidatingWebhookConfiguration specifying which operations/resources to intercept and the webhook's endpoint; the API server sends an AdmissionReview, and your service responds allowed: true/false (with an optional message). The webhook cannot modify the object — only accept or reject it.

Use cases: enforce naming conventions, require certain labels/annotations, forbid :latest image tags, block privileged Pods, require resource limits — any rule expressible as "is this object acceptable?" Critical operational concerns: failurePolicy (Fail = reject if the webhook is unreachable, Ignore = allow), scoping via rules/namespaceSelector/objectSelector to avoid intercepting everything (especially kube-system), and timeoutSeconds. A newer, in-process alternative — ValidatingAdmissionPolicy using CEL expressions — avoids running a webhook for many cases.

Example

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata: { name: no-latest-tag }
webhooks:
- name: images.example.com
  rules:
  - { operations: ["CREATE","UPDATE"], apiGroups: [""], apiVersions: ["v1"], resources: ["pods"] }
  clientConfig:
    service: { name: policy-svc, namespace: policy, path: /validate }
  failurePolicy: Fail            # reject if webhook is down (fail closed)
  namespaceSelector:
    matchExpressions:
    - { key: kubernetes.io/metadata.name, operator: NotIn, values: ["kube-system"] }
  admissionReviewVersions: ["v1"]
  sideEffects: None

Exercises

1. (Beginner) Can a ValidatingAdmissionWebhook modify the object it reviews?
2. (Beginner) What does the API server send to the webhook, and what does the webhook return?
3. (Intermediate) What is the effect of failurePolicy: Fail versus Ignore when the webhook is unreachable?
4. (Interview) What in-process alternative to validating webhooks exists, and what advantage does it offer? (Hint: ValidatingAdmissionPolicy with CEL; no external service.)

Answers

1. No — it can only accept or reject; modification is the job of mutating webhooks.
2. The API server sends an AdmissionReview (containing the object and request info); the webhook returns an AdmissionReview response with allowed: true/false and an optional status message.
3. With Fail, if the webhook can't be reached, the request is rejected (fail closed) — safer for policy enforcement but risks blocking operations if the webhook is down. With Ignore, the request is allowed (fail open) — avoids outages but lets requests bypass the policy during webhook downtime.
4. ValidatingAdmissionPolicy (with F+CEL expressions) evaluates policy in-process in the API server, with no external webhook service to deploy, secure, scale, or keep available. It removes the critical-path network dependency and operational burden of a webhook for many validation use cases, while offering declarative CEL-based rules.

---

MutatingAdmissionWebhook

Theory

A MutatingAdmissionWebhook is the same mechanism but for modifying incoming objects before they're stored. Your webhook receives the AdmissionReview and returns a JSON Patch describing changes to apply — e.g., inject a sidecar container, add default labels/annotations, set a default securityContext, or mount a volume. This is the machinery behind service meshes (Istio/Linkerd inject their proxy sidecar via a mutating webhook) and many platform features.

Because it runs before validation, the mutated object is what validation (and the rest of admission) sees. Important nuances: mutating webhooks should be idempotent (Kubernetes may call them more than once, and re-applying must be safe), order among multiple mutating webhooks can matter, and the same failurePolicy/scoping/timeout concerns apply. Mutating webhooks are powerful but riskier than validating ones — a buggy mutation can corrupt every object it touches — so they're used deliberately and tested carefully.

Example

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata: { name: sidecar-injector }
webhooks:
- name: inject.example.com
  rules:
  - { operations: ["CREATE"], apiGroups: [""], apiVersions: ["v1"], resources: ["pods"] }
  clientConfig:
    service: { name: injector, namespace: mesh, path: /mutate }
  reinvocationPolicy: IfNeeded     # may be re-invoked; mutation must be idempotent
  failurePolicy: Ignore
  admissionReviewVersions: ["v1"]
  sideEffects: None
# The webhook returns a JSON patch adding the sidecar container to the Pod.

Exercises

1. (Beginner) What does a mutating webhook do that a validating one cannot?
2. (Beginner) Name a widely-used feature implemented via mutating webhooks.
3. (Intermediate) Why must a mutating webhook be idempotent?
4. (Interview) Why are mutating webhooks considered riskier than validating webhooks, and what precautions apply? (Hint: silently changes every matched object; idempotency, scoping, testing, ordering.)

Answers

1. It can modify the incoming object (e.g., via a JSON patch) — injecting containers, setting defaults, adding labels — whereas a validating webhook can only accept/reject.
2. Service-mesh sidecar injection (Istio/Linkerd inject their Envoy/proxy container via a mutating webhook). (Also: default securityContext injection, automatic label/annotation insertion.)
3. The API server may invoke mutating webhooks more than once (e.g., via reinvocationPolicy after other mutations), so the webhook must produce the same correct result if applied repeatedly — otherwise it could double-inject a sidecar or repeatedly alter fields, corrupting the object.
4. They silently alter every object they match, so a bug can corrupt all affected resources (e.g., break every new Pod) rather than just rejecting some. Precautions: make mutations idempotent, scope tightly with rules/namespaceSelector/objectSelector (exclude kube-system), set sensible failurePolicy/timeouts, consider ordering relative to other mutating webhooks, run the webhook HA, and test thoroughly (including reinvocation) before broad rollout.

---

OPA Gatekeeper

Theory

Writing and operating custom admission webhooks is a lot of work. OPA Gatekeeper is a popular policy engine that provides a ready-made validating (and limited mutating) admission webhook driven by declarative policies, so you don't write webhook code. It's built on the Open Policy Agent (OPA) and its Rego policy language.

Gatekeeper introduces two concepts: a ConstraintTemplate (defines a reusable policy and its Rego logic, exposing it as a new CRD) and Constraints (instances of that template applied with parameters and scope). For example, a template "required labels" plus a constraint "all namespaces must have an owner label." Gatekeeper also supports audit (continuously checking existing objects against policies, not just new ones) and a constraint library of common policies. It's a CNCF project widely used to enforce governance (image registries, resource limits, allowed types) without bespoke webhook development.

Example

# 1) ConstraintTemplate defines the policy logic (Rego), exposed as a CRD:
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata: { name: k8srequiredlabels }
spec:
  crd: { spec: { names: { kind: K8sRequiredLabels } } }
  targets: [ { target: admission.k8s.gatekeeper.sh, rego: "..." } ]
---

# 2) Constraint applies it: every Namespace must have an "owner" label
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata: { name: ns-must-have-owner }
spec:
  match: { kinds: [ { apiGroups: [""], kinds: ["Namespace"] } ] }
  parameters: { labels: ["owner"] }

Exercises

1. (Beginner) What does OPA Gatekeeper save you from writing yourself?
2. (Beginner) What policy language does Gatekeeper (via OPA) use?
3. (Intermediate) Explain the relationship between a ConstraintTemplate and a Constraint.
4. (Interview) What does Gatekeeper's "audit" feature add beyond admission-time enforcement, and why is it valuable? (Hint: finds existing violations created before the policy.)

Answers

1. A custom admission webhook (server code) — Gatekeeper provides the webhook and you supply declarative policies instead of writing/operating your own webhook service.
2. Rego (the Open Policy Agent policy language).
3. A ConstraintTemplate defines reusable policy logic (in Rego) and exposes it as a new CRD kind; a Constraint is an instance of that template that activates the policy with specific parameters and a match scope (which resources it applies to). One template, many constraints.
4. Audit periodically evaluates existing cluster objects against the active constraints, reporting violations even for resources created before the policy existed (or that bypassed admission). This is valuable for discovering and remediating non-compliant resources already in the cluster, giving continuous compliance visibility rather than only blocking new violations at admission time.

---

Kyverno policy engine

Theory

Kyverno is another CNCF policy engine and the main alternative to Gatekeeper. Its defining feature: policies are written as native Kubernetes YAML (no Rego/new language to learn) — making it especially approachable for Kubernetes practitioners. Kyverno runs as an admission webhook and can validate, mutate, and generate resources, plus verify image signatures and run background scans.

Capabilities that distinguish it:

• Validate: reject non-compliant resources (like Gatekeeper).
• Mutate: modify resources (e.g., add default labels/securityContext) — a richer mutation story than Gatekeeper.
• Generate: automatically create related resources (e.g., create a default NetworkPolicy or ResourceQuota whenever a Namespace is created).
• VerifyImages: enforce image signature verification (cosign).

The choice between Kyverno and Gatekeeper often comes down to language preference (YAML vs. Rego) and feature needs (Kyverno's generate/mutate vs. OPA's general-purpose Rego power). Both enforce policy-as-code for governance and security.

Example

# Kyverno policy in plain YAML: require resource limits on all Pods
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata: { name: require-limits }
spec:
  validationFailureAction: Enforce      # Enforce (block) or Audit (report)
  rules:
  - name: check-limits
    match: { any: [ { resources: { kinds: ["Pod"] } } ] }
    validate:
      message: "CPU and memory limits are required."
      pattern:
        spec:
          containers:
          - resources:
              limits: { memory: "?*", cpu: "?*" }   # must be set (non-empty)

Exercises

1. (Beginner) What makes Kyverno's policy authoring approachable compared to Gatekeeper?
2. (Beginner) Name the four kinds of actions Kyverno policies can perform.
3. (Intermediate) Give an example of Kyverno's "generate" capability and why it's useful.
4. (Interview) How would you decide between Kyverno and OPA Gatekeeper for a team? (Hint: YAML vs. Rego, generate/mutate features vs. general-purpose policy logic.)

Answers

1. Kyverno policies are written as native Kubernetes YAML — no separate policy language (like Rego) is required — so they're familiar to anyone who already writes Kubernetes manifests.
2. Validate, mutate, generate, and verifyImages (image signature verification).
3. Example: automatically generate a default NetworkPolicy (or ResourceQuota/LimitRange/RoleBinding) whenever a new Namespace is created. This is useful for enforcing baseline governance/security on every namespace without manual steps, ensuring consistency and reducing the chance of forgetting required defaults.
4. Consider the team's familiarity and needs: Kyverno if you prefer YAML-native policies, want built-in mutate/generate/image-verification, and value a gentle learning curve. Gatekeeper/OPA if you want the full expressive power of Rego (complex logic, reuse across systems beyond Kubernetes) and the OPA ecosystem/constraint library, and the team is comfortable learning Rego. Both provide validation and audit; the decision largely hinges on language preference and whether Kyverno's generation/mutation features matter.

---

9.4 Pod Security

Even with RBAC, a Pod can be dangerous if it runs as root, privileged, or with host access. Pod-level security controls constrain what containers can do. This subchapter covers them.

---

Pod Security Admission (PSA)

Theory

The old PodSecurityPolicy (PSP) was removed in v1.25. Its replacement is Pod Security Admission (PSA) — a built-in admission controller that enforces the Pod Security Standards (next topic) at the namespace level via labels. Instead of complex policy objects, you label a namespace with the standard ("level") to apply and the mode of enforcement.

Three modes, which can be combined:

• enforce: reject Pods that violate the level.
• audit: allow but record violations in the audit log.
• warn: allow but return a user-facing warning.

You also pin a version (e.g., v1.30) for consistent behavior. PSA is simpler than PSP but coarser (per-namespace, three predefined levels) — for finer or custom rules you combine it with a policy engine (Kyverno/Gatekeeper). It's the standard built-in mechanism for baseline Pod hardening.

Example

apiVersion: v1
kind: Namespace
metadata:
  name: prod
  labels:
    pod-security.kubernetes.io/enforce: restricted     # block violators
    pod-security.kubernetes.io/enforce-version: v1.30
    pod-security.kubernetes.io/warn: restricted        # also warn
    pod-security.kubernetes.io/audit: restricted       # and audit

# A Pod violating "restricted" in this namespace is rejected at creation:
kubectl apply -f privileged-pod.yaml -n prod
# Error: violates PodSecurity "restricted:v1.30": privileged (...)

Exercises

1. (Beginner) What did Pod Security Admission replace?
2. (Beginner) At what scope is PSA applied, and how is it configured?
3. (Intermediate) What are the three PSA modes and how do they differ?
4. (Interview) PSA is simpler but coarser than the old PSP. When would you supplement PSA with a policy engine like Kyverno? (Hint: custom/finer-grained rules beyond the three standard levels.)

Answers

1. PodSecurityPolicy (PSP), which was removed in v1.25.
2. At the namespace level, configured via labels on the Namespace (pod-security.kubernetes.io/<mode>: <level> and a pinned version).
3. enforce (reject violating Pods), audit (allow but log violations to the audit log), and warn (allow but return a warning to the user). They can be set simultaneously with different levels.
4. When you need rules beyond the three predefined Pod Security Standards or finer granularity than per-namespace — e.g., enforce specific allowed registries, require particular labels, mandate resource limits, allow narrow exceptions, or apply object-specific logic. A policy engine (Kyverno/Gatekeeper) provides custom, parameterized policies (validate/mutate/generate) that complement PSA's coarse baseline.

---

Pod Security Standards: Privileged, Baseline, Restricted

Theory

The Pod Security Standards (PSS) define three cumulative security levels that PSA enforces:

• Privileged: unrestricted — no constraints. For trusted, system-level workloads (e.g., CNI/CSI agents) that legitimately need host access.
• Baseline: prevents known privilege escalations while staying broadly compatible. Disallows the most dangerous settings — privileged: true, host namespaces (hostNetwork/PID/IPC), hostPath in dangerous ways, adding dangerous capabilities — but is lenient enough for most apps.
• Restricted: the hardened, best-practice level. Requires running as non-root, dropping ALL capabilities, seccompProfile: RuntimeDefault, allowPrivilegeEscalation: false, read-only root filesystem encouraged, no host access. This is the target for security-sensitive workloads.

They are a spectrum from "anything goes" to "locked down." The recommendation: default namespaces to at least Baseline, and aim for Restricted for production workloads, reserving Privileged only for specific trusted system components.

Example

# A Pod that satisfies the "restricted" standard:
spec:
  securityContext:
    runAsNonRoot: true
    seccompProfile: { type: RuntimeDefault }
  containers:
  - name: app
    image: myapp:1.0
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities: { drop: ["ALL"] }

Exercises

1. (Beginner) Name the three Pod Security Standards from least to most restrictive.
2. (Beginner) Which standard would you use for a trusted CNI agent that needs host access?
3. (Intermediate) List three requirements the Restricted level imposes.
4. (Interview) Why is Baseline a pragmatic default for most workloads while Restricted is the goal for production? (Hint: compatibility vs. hardening trade-off.)

Answers

1. Privileged (least restrictive), Baseline, Restricted (most restrictive).
2. Privileged — system components like CNI/CSI agents legitimately require host namespaces/privileges that Baseline/Restricted forbid.
3. Any three: run as non-root (runAsNonRoot: true), drop ALL capabilities, allowPrivilegeEscalation: false, seccompProfile: RuntimeDefault, no host namespaces/privileged, restricted volume types (and read-only root filesystem is encouraged).
4. Baseline blocks the most dangerous, known-escalation settings while remaining compatible with the majority of existing applications, so it's a safe default that rarely breaks workloads. Restricted enforces full hardening best practices (non-root, dropped capabilities, seccomp, no privilege escalation) which materially reduces attack surface but may require app changes (e.g., not running as root). Production should aim for Restricted to minimize risk, using Baseline as the broadly-applicable minimum and Privileged only for vetted system components.

---

SecurityContext: runAsUser, fsGroup, capabilities

Theory

A securityContext is where you actually configure the per-Pod and per-container security settings that the Pod Security Standards check. It can be set at the Pod level (applies to all containers) and container level (overrides for that container). Key fields:

• runAsUser / runAsGroup / runAsNonRoot: the UID/GID the container process runs as; runAsNonRoot: true blocks running as root (UID 0).
• fsGroup: a supplemental group applied to mounted volumes so the container can read/write them (fixes permission issues on persistent volumes).
• capabilities: Linux capabilities to add or drop. Best practice is drop: ["ALL"] then add back only what's strictly needed (e.g., NET_BIND_SERVICE to bind ports < 1024).
• allowPrivilegeEscalation, readOnlyRootFilesystem, privileged, seccompProfile: further hardening.

Setting a strong securityContext (non-root, dropped capabilities, no privilege escalation, read-only root FS) is the concrete implementation of Pod hardening and is required to meet the Restricted standard.

Example

spec:
  securityContext:                 # Pod-level (all containers)
    runAsNonRoot: true
    runAsUser: 1000
    fsGroup: 2000                  # volume files owned by gid 2000 -> writable
  containers:
  - name: app
    image: myapp:1.0
    securityContext:               # container-level overrides/additions
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities:
        drop: ["ALL"]
        add: ["NET_BIND_SERVICE"]  # only what's needed

Exercises

1. (Beginner) What does runAsNonRoot: true enforce?
2. (Beginner) What problem does fsGroup solve?
3. (Intermediate) What is the recommended approach to Linux capabilities in a securityContext?
4. (Interview) Explain how Pod-level and container-level securityContext interact, and give an example where you'd override at the container level. (Hint: Pod sets defaults; container overrides for specific needs.)

Answers

1. It prevents the container from running as the root user (UID 0); if the image would run as root, the container fails to start.
2. Volume permission issues: fsGroup sets a supplemental group owner on mounted (especially persistent) volumes so the container's process can read/write them, instead of being denied access due to root-owned mount permissions.
3. Drop all capabilities (drop: ["ALL"]) and then add back only the specific capabilities the workload truly needs (e.g., NET_BIND_SERVICE), minimizing privileges.
4. The Pod-level securityContext sets defaults for all containers; a container-level securityContext overrides or augments those for that specific container. Example: a Pod sets runAsNonRoot: true and drops all capabilities globally, but one container that must bind to port 80 overrides at the container level to add: ["NET_BIND_SERVICE"] (or a sidecar needs a read-only root FS while the main app needs writable scratch). Container-level settings take precedence for that container, allowing minimal, targeted exceptions to a hardened Pod default.

---

AppArmor and Seccomp profiles

Theory

Beyond capabilities and user IDs, two Linux kernel security mechanisms further confine what a container can do:

• Seccomp (secure computing mode): filters the system calls a process may make. Most applications use only a small subset of the ~300+ syscalls; blocking the rest shrinks the kernel attack surface dramatically. The RuntimeDefault seccomp profile (a sensible blocklist provided by the runtime) is recommended for all workloads and required by the Restricted standard; you can also supply custom profiles.
• AppArmor: a Linux Security Module that confines a program to a set of allowed capabilities and file/resource accesses via a named profile loaded on the node. It restricts what files a container can read/write/execute and what operations it can perform.

Both are configured through the Pod/container spec (seccomp via securityContext.seccompProfile; AppArmor via securityContext.appArmorProfile in recent versions, or annotations in older ones). They embody defense in depth: even if an attacker gains code execution in a container, these confinements limit what they can actually do to the kernel/host.

Example

spec:
  securityContext:
    seccompProfile:
      type: RuntimeDefault          # block dangerous syscalls (recommended baseline)
  containers:
  - name: app
    image: myapp:1.0
    securityContext:
      appArmorProfile:
        type: Localhost             # use a named AppArmor profile loaded on the node
        localhostProfile: k8s-myapp
      # custom seccomp alternative:
      # seccompProfile: { type: Localhost, localhostProfile: profiles/myapp.json }

Exercises

1. (Beginner) What does seccomp filter, and what does AppArmor confine?
2. (Beginner) Which seccomp profile is the recommended baseline for workloads?
3. (Intermediate) Why does restricting syscalls reduce the kernel attack surface?
4. (Interview) How do seccomp and AppArmor exemplify defense in depth alongside RBAC and securityContext? (Hint: limit damage even after a container is compromised.)

Answers

1. Seccomp filters the system calls a process may make to the kernel; AppArmor confines a program's allowed file/resource accesses and operations via a profile (a Linux Security Module).
2. RuntimeDefault (the runtime-provided default seccomp profile).
3. Most applications need only a small fraction of the kernel's hundreds of syscalls. Many kernel vulnerabilities are reached through specific, rarely-needed syscalls; blocking everything outside the app's required set removes those attack paths, so even exploited code can't invoke the dangerous syscalls — shrinking the exploitable surface.
4. Defense in depth layers independent controls so a single failure isn't catastrophic. RBAC limits API access; securityContext limits privileges/identity (non-root, dropped capabilities); seccomp and AppArmor confine the container at the kernel level — restricting syscalls and file/resource access. So even if an attacker compromises the application and bypasses higher layers, seccomp/AppArmor constrain what they can do to the kernel and host (e.g., can't make the syscalls needed for a container escape), containing the breach. Each layer assumes the others might fail.

---

9.5 Secrets and Supply Chain Security

The images you run are only as trustworthy as their provenance. This subchapter covers verifying images, controlling registries, runtime threat detection, and software bills of materials.

---

Image signing and verification (cosign, Notary)

Theory

How do you know the image your cluster pulls is the one your team built, unmodified? By default, you don't — image tags are mutable and registries can be compromised. Image signing addresses this: a trusted party cryptographically signs an image, and the cluster verifies the signature before running it, guaranteeing authenticity (who built it) and integrity (it wasn't tampered with).

The dominant modern tool is cosign (part of the Sigstore project), which signs container images (and other artifacts) and stores signatures in the registry alongside the image. Sigstore enables keyless signing (using short-lived certificates tied to an OIDC identity, removing the burden of managing signing keys). The older Notary / Docker Content Trust (TUF-based) served a similar purpose. Verification is enforced in the cluster via admission policies (next topic) so unsigned or untrusted images are rejected — closing a major supply-chain attack vector.

Example

# Sign an image with cosign (keyless, using OIDC identity):
cosign sign --yes registry.example.com/myapp@sha256:abc123...

# Verify a signature before deploying:
cosign verify --certificate-identity=ci@example.com \
  --certificate-oidc-issuer=https://accounts.google.com \
  registry.example.com/myapp@sha256:abc123...

Exercises

1. (Beginner) What two properties does image signing guarantee?
2. (Beginner) What is the modern tool/project most associated with container image signing?
3. (Intermediate) What problem does Sigstore's "keyless signing" solve?
4. (Interview) Why is verifying image signatures (and pinning by digest) important against supply-chain attacks, given that tags are mutable? (Hint: tag re-pointing, registry compromise, integrity/authenticity.)

Answers

1. Authenticity (the image came from the expected, trusted signer) and integrity (it has not been altered since signing).
2. cosign (part of the Sigstore project). (Notary / Docker Content Trust is the older alternative.)
3. The operational burden and risk of managing long-lived signing keys. Keyless signing uses short-lived certificates issued against an OIDC identity (recorded in a transparency log), so there are no private keys to store, rotate, or leak — signing is tied to a verifiable identity instead.
4. Tags like :latest or :v1 are mutable pointers — an attacker who compromises the registry or pipeline can re-point a tag to a malicious image, and consumers would unknowingly pull it. Signature verification ensures only images signed by a trusted identity run (authenticity) and that the content matches what was signed (integrity), while pinning by immutable digest (@sha256:...) guarantees you run exactly the intended bytes. Together they defeat tag re-pointing and tampering, even if the registry is compromised.

---

Admission policies for image registries

Theory

Signing is only useful if the cluster enforces verification, and you often also want to restrict where images may come from. This is done with admission policies:

• Registry allowlisting: reject Pods whose images aren't from approved registries (e.g., only registry.example.com/*), preventing use of arbitrary public images that could be malicious or unvetted.
• Signature verification at admission: reject images lacking a valid signature from a trusted signer (Kyverno's verifyImages, Connaisseur, Sigstore policy-controller, or Gatekeeper with external data).
• Tag policies: forbid mutable tags like :latest, requiring immutable digests for reproducibility.

These are implemented with the policy engines from §9.3 (Kyverno, Gatekeeper) or purpose-built controllers. The result is a cluster that will only run images that are from trusted sources, properly signed, and pinned — a strong supply-chain control enforced automatically at deploy time.

Example

# Kyverno: require images from an approved registry AND a valid cosign signature
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata: { name: trusted-images }
spec:
  validationFailureAction: Enforce
  rules:
  - name: only-approved-registry
    match: { any: [ { resources: { kinds: ["Pod"] } } ] }
    validate:
      message: "Images must come from registry.example.com"
      pattern: { spec: { containers: [ { image: "registry.example.com/*" } ] } }
  - name: verify-signature
    match: { any: [ { resources: { kinds: ["Pod"] } } ] }
    verifyImages:
    - imageReferences: [ "registry.example.com/*" ]
      attestors: [ { entries: [ { keyless: { issuer: "https://accounts.google.com", subject: "ci@example.com" } } ] } ]

Exercises

1. (Beginner) What does registry allowlisting prevent?
2. (Beginner) What kind of tag do tag policies typically forbid, and what do they require instead?
3. (Intermediate) Which tools enforce image signature verification at admission?
4. (Interview) Describe a layered admission policy that ensures only trusted images run, and what each layer defends against. (Hint: registry source + signature + digest pinning.)

Answers

1. Running images from unapproved/arbitrary sources (e.g., random public registries), which could be malicious, vulnerable, or unvetted — restricting Pods to images from trusted registries only.
2. They forbid mutable tags like :latest and require immutable digests (@sha256:...) for reproducibility and integrity.
3. Policy engines/controllers such as Kyverno (verifyImages), Sigstore's policy-controller, Connaisseur, or OPA Gatekeeper (with external data) — invoked as admission webhooks.
4. A layered policy: (1) registry allowlist — reject images not from approved registries (defends against pulling untrusted/unknown images); (2) signature verification — reject images without a valid signature from a trusted identity (defends against tampered or unauthorized images, even from the right registry); (3) digest pinning / no mutable tags — require @sha256: digests (defends against tag re-pointing and ensures reproducible, exact-content deployments). Together they ensure every running image is from a known source, provably authentic and untampered, and exactly the intended artifact — enforced automatically at admission.

---

Falco for runtime threat detection

Theory

All the prior controls are preventive (block bad things at deploy time). But you also need detective controls for what happens at runtime — because a container can be exploited, or a legitimate image can behave maliciously, after it's running. Falco (a CNCF project) is the leading runtime security/threat-detection tool for Kubernetes.

Falco taps into kernel events (via eBPF or a kernel module) to observe syscalls and behavior in real time, comparing them against a set of rules that describe suspicious activity: a shell spawned inside a container, an unexpected outbound connection, writes to sensitive paths, reads of secrets, privilege escalation, or a process not in the container's normal profile. When a rule matches, Falco emits an alert (to stdout, a webhook, Falcosidekick, SIEM, etc.). It's the runtime analog to an intrusion detection system, enabling you to detect and respond to attacks that slipped past preventive controls — embodying the "assume breach" mindset.

Example

# A Falco rule: alert when a shell is started inside a container
- rule: Terminal shell in container
  desc: A shell was spawned in a container (often a sign of intrusion)
  condition: >
    spawned_process and container
    and shell_procs and proc.tty != 0
  output: "Shell in container (user=%user.name container=%container.name cmd=%proc.cmdline)"
  priority: WARNING

# Falco runs as a DaemonSet; alerts stream from each node:
kubectl -n falco logs -l app.kubernetes.io/name=falco -f
# WARNING Shell in container (user=root container=web cmd=bash)

Exercises

1. (Beginner) Is Falco a preventive or a detective control? What does it observe?
2. (Beginner) How does Falco typically run in a cluster?
3. (Intermediate) Give two examples of suspicious runtime behavior Falco can detect.
4. (Interview) Why are runtime detection tools like Falco necessary even with strong admission and Pod-security controls? (Hint: assume breach; exploits/zero-days/insider behavior at runtime.)

Answers

1. Detective — it detects suspicious activity at runtime (it doesn't block deployment). It observes kernel-level events/syscalls and container behavior in real time.
2. As a DaemonSet (one agent per node), tapping kernel events via eBPF or a kernel module, emitting alerts.
3. Any two: a shell/terminal spawned inside a container, unexpected outbound network connections, writes to sensitive directories (e.g., /etc), reading secret files, package-manager execution in a running container, privilege escalation attempts, or processes outside the container's expected set.
4. Preventive controls reduce but can't eliminate risk: applications have vulnerabilities (including zero-days), legitimate-but-malicious or compromised images can pass admission, and insiders or supply-chain issues can introduce threats that only manifest at runtime. Under an "assume breach" model, you need to detect malicious behavior after a container is running — Falco provides that visibility (e.g., catching an attacker who exploited an app and spawned a shell), enabling alerting and response that preventive admission/Pod-security checks alone cannot offer.

---

SBOM and vulnerability scanning

Theory

You can't secure what you don't know you're shipping. A Software Bill of Materials (SBOM) is a complete inventory of all components, libraries, and dependencies (with versions) inside a container image — like an ingredients label. Generated at build time (tools: Syft, Trivy, in formats like SPDX or CycloneDX), an SBOM lets you instantly answer "are we affected by the new CVE in library X?" by querying which images contain it — crucial during incidents like Log4Shell.

Vulnerability scanning complements the SBOM by checking those components against known-vulnerability databases (CVEs). Scanners (Trivy, Grype, Clair, Snyk) run in CI (scan before push), in the registry (continuous re-scan as new CVEs are disclosed), and via admission (block images above a severity threshold). The combination — know your contents (SBOM) and check them for known flaws (scanning) — is foundational supply-chain hygiene, shifting vulnerability management "left" into the build pipeline and continuously across the image lifecycle.

Example

# Generate an SBOM for an image (Syft):
syft registry.example.com/myapp:1.0 -o spdx-json > sbom.json

# Scan an image for known CVEs (Trivy), failing on High/Critical:
trivy image --severity HIGH,CRITICAL --exit-code 1 registry.example.com/myapp:1.0
# myapp:1.0 (alpine 3.19)
# Total: 2 (HIGH: 1, CRITICAL: 1)
# CVE-2024-XXXX  openssl  3.1.4  ->  3.1.6  (fixed)

Exercises

1. (Beginner) What is an SBOM, in one sentence?
2. (Beginner) What does vulnerability scanning check images against?
3. (Intermediate) During a newly-disclosed critical CVE, how does having SBOMs speed up your response?
4. (Interview) Explain "shift left" for vulnerability management and why scanning in CI alone is insufficient — what additional scanning points are needed? (Hint: new CVEs appear after build; registry/admission re-scanning.)

Answers

1. A Software Bill of Materials is a complete, machine-readable inventory of all components, libraries, and dependencies (and their versions) contained in a software artifact/image.
2. Known-vulnerability databases — i.e., published CVEs (and advisories) for the components present in the image.
3. With SBOMs you can immediately query which images/deployments contain the affected component and version, pinpointing exposure across your fleet in minutes instead of manually inspecting each image — enabling targeted, fast remediation (as was painfully needed during Log4Shell).
4. "Shift left" means moving vulnerability detection earlier, into the build/CI pipeline, so issues are caught before deployment. But CI scanning only reflects vulnerabilities known at build time; new CVEs are disclosed continuously after an image is built and deployed. So you also need continuous re-scanning in the registry (re-evaluate stored images against updated CVE feeds) and admission-time scanning/policy (block deploying images exceeding a severity threshold), plus runtime awareness — covering the whole lifecycle, not just the moment of build.

---