7. Storage

Containers are ephemeral — when a container restarts, anything written to its writable layer is lost. Real applications need durable storage, shared scratch space, and a way to consume cloud disks without hard-coding vendor details. Kubernetes addresses this with a layered storage model: Volumes attach storage to Pods, PersistentVolumes/Claims decouple storage provisioning from consumption, StorageClasses automate provisioning, and the CSI standard makes any storage system pluggable. This chapter builds that model.

---

7.1 Volumes

A Volume is the most basic storage abstraction — a directory accessible to a Pod's containers, backed by some medium. This subchapter covers the common volume types and their lifecycles.

---

Volume types: emptyDir, hostPath, configMap, secret

Theory

A Volume is a directory, possibly with data, that is accessible to the containers in a Pod. Unlike a container's own ephemeral filesystem (lost on restart), a volume can outlive container restarts within the Pod and can be shared between containers in the Pod. The volume's backing medium is what its type defines:

• emptyDir: an empty directory created when the Pod is assigned to a node, used for scratch space or sharing files between containers in the Pod. It is deleted when the Pod is removed. Can be backed by disk or RAM (medium: Memory).
• hostPath: mounts a file or directory from the node's filesystem into the Pod. Powerful but dangerous (ties the Pod to a node, can access host files) — mostly for node-level system Pods (DaemonSets).
• configMap / secret: project a ConfigMap's or Secret's keys as files (covered in Chapter 5).
• (Plus persistentVolumeClaim, projected, downwardAPI, and CSI volumes covered elsewhere.)

Example

spec:
  containers:
  - name: app
    image: myapp:1.0
    volumeMounts:
    - { name: scratch, mountPath: /cache }     # shared scratch
    - { name: cfg, mountPath: /etc/cfg }        # config files
  volumes:
  - name: scratch
    emptyDir: {}                                # ephemeral, lives with the Pod
  - name: cfg
    configMap: { name: app-config }             # keys as files
  - name: hostlogs
    hostPath:                                   # node filesystem (use sparingly)
      path: /var/log
      type: Directory

Exercises

1. (Beginner) What is the lifespan of an emptyDir volume?
2. (Beginner) What does a hostPath volume mount into the Pod, and why is it risky?
3. (Intermediate) Two containers in one Pod need to share files. Which volume type fits, and why does it work across container restarts?
4. (Interview) Why is hostPath discouraged for general application workloads but acceptable for certain DaemonSets? (Hint: node coupling, security, portability vs. node-level agents.)

Answers

1. It is created when the Pod is assigned to a node and exists as long as that Pod runs on the node; it is deleted permanently when the Pod is removed (it does survive container restarts within the Pod).
2. It mounts a file/directory from the node's own filesystem into the Pod. It's risky because it couples the Pod to that specific node, can expose or modify sensitive host files, and breaks portability and isolation.
3. An emptyDir volume mounted into both containers. It is tied to the Pod (not to a single container), so it persists across individual container restarts and both containers see the same directory, enabling file sharing.
4. For general apps, hostPath ties the Pod to a particular node's filesystem (breaking rescheduling/portability) and is a security risk (host access), so it's discouraged. It's acceptable for node-level DaemonSets (log collectors, monitoring, CNI/CSI agents) whose entire purpose is to interact with the node's filesystem (/var/log, /proc, runtime sockets) and which are intentionally pinned one-per-node with elevated access.

---

Volume lifecycle and pod coupling

Theory

The defining characteristic of a (non-persistent) Volume is that its lifecycle is coupled to the Pod, not to individual containers and not durable beyond the Pod. When the Pod is created, its volumes are set up; when containers restart, the volumes persist (so data survives a crash-restart); but when the Pod is deleted, ephemeral volumes (like emptyDir) and their data are gone.

This is the key distinction that motivates PersistentVolumes (next subchapter): if you need data to survive Pod deletion, rescheduling to another node, or to be independent of any Pod, a plain Volume is insufficient — you need a PersistentVolume whose lifecycle is independent of any Pod. Understanding "what survives what" — container restart vs. Pod deletion vs. node failure — is essential to choosing the right storage.

Example

Event                         emptyDir data?     PV (PVC) data?
---------------------------------------------------------------
Container crashes & restarts   Survives           Survives
Pod deleted / rescheduled      Lost               Survives (re-attaches)
Node fails                     Lost               Survives (on network storage)

# Demonstrate emptyDir survives container restart but not Pod deletion:
kubectl exec mypod -c app -- sh -c 'echo hi > /cache/x'
kubectl exec mypod -c app -- kill 1     # container restarts; /cache/x still there
kubectl delete pod mypod                 # recreate -> /cache is empty again

Exercises

1. (Beginner) Does an emptyDir volume survive a container restart? Does it survive Pod deletion?
2. (Beginner) To which object's lifecycle is a plain Volume coupled?
3. (Intermediate) Construct a table of what data survives: container restart, Pod reschedule, node failure — for emptyDir vs. a network-backed PV.
4. (Interview) Why does the Pod-coupled lifecycle of ordinary Volumes motivate the existence of PersistentVolumes? (Hint: independence from any single Pod.)

Answers

1. It survives a container restart (it's tied to the Pod, not the container) but is lost on Pod deletion.
2. The Pod's lifecycle (it is created with the Pod and destroyed when the Pod is removed).
3. emptyDir: container restart = survives, Pod reschedule/deletion = lost, node failure = lost. Network-backed PV: container restart = survives, Pod reschedule = survives (re-attaches to the new Pod), node failure = survives (data lives on external storage, can be re-mounted elsewhere).
4. Because ordinary Volumes vanish when their Pod does, they can't provide durable or relocatable storage. Stateful apps need data that persists across Pod deletion, rescheduling, and node failure, and that exists as a first-class object independent of any Pod. PersistentVolumes provide exactly that decoupling — storage with its own lifecycle, claimed by Pods via PVCs but outliving them.

---

Projected volumes

Theory

A projected volume combines several volume sources into a single directory. Instead of mounting a ConfigMap, a Secret, and Downward API info at three different paths, a projected volume lets you assemble keys from all of them into one unified mount, each at a path you choose. The supported sources are configMap, secret, downwardAPI, and serviceAccountToken.

The most important modern use is the serviceAccountToken projected source: it requests a short-lived, audience-scoped, auto-rotated ServiceAccount token (a "bound" token) and projects it as a file. This is the secure replacement for the old long-lived ServiceAccount token Secrets — the token has an expiry, is automatically refreshed by the kubelet, and is bound to a specific audience and the Pod's lifetime. Projected volumes thus serve both convenience (unify sources) and security (modern token delivery).

Example

spec:
  containers:
  - name: app
    image: myapp:1.0
    volumeMounts:
    - { name: all-in-one, mountPath: /etc/config }
  volumes:
  - name: all-in-one
    projected:
      sources:
      - configMap: { name: app-config }
      - secret: { name: db-creds }
      - downwardAPI:
          items:
          - { path: "labels", fieldRef: { fieldPath: metadata.labels } }
      - serviceAccountToken:               # short-lived, audience-bound token
          path: token
          audience: vault
          expirationSeconds: 3600

Exercises

1. (Beginner) What does a projected volume let you do with multiple sources?
2. (Beginner) Name the four sources a projected volume supports.
3. (Intermediate) Why is a serviceAccountToken projected volume more secure than mounting a legacy ServiceAccount token Secret?
4. (Interview) Describe two distinct benefits (one convenience, one security) that projected volumes provide. (Hint: unified mount; bound, rotated tokens.)

Answers

1. Combine keys/data from several volume sources into a single mounted directory, each item at a chosen path.
2. configMap, secret, downwardAPI, and serviceAccountToken.
3. The projected token is short-lived (has an expirationSeconds), automatically rotated by the kubelet before expiry, and bound to a specific audience (and the Pod's lifetime). A legacy token Secret is long-lived, non-expiring, not audience-scoped, and stored in etcd — so if leaked it grants indefinite, broad access. The bound token drastically limits exposure.
4. Convenience: it unifies multiple configuration/secret/metadata sources into one directory instead of several separate mounts, simplifying the container's view of its config. Security: the serviceAccountToken source delivers short-lived, audience-bound, auto-rotated tokens (instead of permanent token Secrets), shrinking the blast radius and lifetime of credentials.

---

CSI volume plugins

Theory

Originally, support for each storage system (AWS EBS, GCE PD, NFS, Ceph) was compiled into Kubernetes ("in-tree" volume plugins), which bloated the codebase and tied storage features to Kubernetes releases. The Container Storage Interface (CSI) is the standard (like CRI for runtimes and CNI for networking) that lets storage vendors write out-of-tree drivers implementing a common gRPC interface. Kubernetes then talks to any CSI driver generically.

A CSI volume is consumed in a Pod either indirectly (via a PVC bound to a StorageClass that uses a CSI provisioner — the common path, covered later) or directly via the csi volume source for special cases. The migration to CSI is essentially complete: in-tree plugins have been deprecated/removed in favor of CSI drivers. The practical takeaway: nearly all real storage in modern clusters flows through CSI drivers, even when you only ever interact with PVCs and StorageClasses.

Example

# A StorageClass referencing a CSI driver (the usual indirect path):
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata: { name: fast }
provisioner: ebs.csi.aws.com        # a CSI driver, not an in-tree plugin
parameters: { type: gp3 }

# CSI drivers register themselves; list installed ones:
kubectl get csidrivers
# NAME                 ATTACHREQUIRED   PODINFOONMOUNT   AGE
# ebs.csi.aws.com      true             false            30d

Exercises

1. (Beginner) What problem did "in-tree" volume plugins create, and what replaced them?
2. (Beginner) How do you list the CSI drivers installed in a cluster?
3. (Intermediate) In the common case, how does a Pod end up using a CSI driver without referencing it directly?
4. (Interview) CSI follows the same philosophy as CRI and CNI. State that philosophy and explain why moving storage drivers out-of-tree benefits both Kubernetes and storage vendors. (Hint: independent release cadence, leaner core.)

Answers

1. In-tree plugins compiled every storage integration into Kubernetes itself, bloating the codebase and coupling storage features/bug-fixes to Kubernetes release cycles. The CSI standard (out-of-tree drivers) replaced them.
2. kubectl get csidrivers.
3. The Pod uses a PVC; the PVC references a StorageClass whose provisioner is a CSI driver. When the PVC is created, the CSI driver dynamically provisions the volume and Kubernetes attaches/mounts it — so the Pod consumes CSI-backed storage by referencing only the PVC.
4. The philosophy is a standard interface with swappable, out-of-tree implementations. Vendors implement the CSI gRPC contract and ship/version/patch their drivers independently of Kubernetes releases, adding features on their own schedule; Kubernetes keeps a lean, vendor-neutral core that talks to any compliant driver. Both win: vendors get release independence and faster iteration, and Kubernetes avoids carrying and maintaining storage-specific code.

---

7.2 Persistent Volumes

PersistentVolumes and PersistentVolumeClaims decouple how storage is provided from how it is consumed — the central abstraction for durable storage in Kubernetes. This subchapter covers the model in depth.

---

PersistentVolume (PV) resource

Theory

A PersistentVolume (PV) is a piece of storage in the cluster, provisioned either by an administrator (statically) or dynamically by a StorageClass. Crucially, a PV is a cluster-scoped resource with its own lifecycle, independent of any Pod. It represents the actual storage — a cloud disk, an NFS export, a Ceph volume — abstracted as a Kubernetes object with a capacity, access modes, and a reclaim policy.

The PV is the "supply" side of Kubernetes storage. It exists whether or not any Pod is using it, surviving Pod deletion and rescheduling. This independence is exactly what plain Volumes lack and what stateful workloads require. Administrators (or dynamic provisioners) create PVs; users don't reference PVs directly — they request storage via a PersistentVolumeClaim (next), which binds to a suitable PV. This separation is the producer/consumer split at the heart of the storage subsystem.

Example

apiVersion: v1
kind: PersistentVolume
metadata: { name: pv-nfs-100g }
spec:
  capacity: { storage: 100Gi }
  accessModes: [ ReadWriteMany ]
  persistentVolumeReclaimPolicy: Retain
  storageClassName: nfs
  nfs:                              # the actual backing storage
    server: 10.0.0.5
    path: /exports/data

kubectl get pv
# NAME          CAPACITY  ACCESS MODES  RECLAIM POLICY  STATUS     CLAIM
# pv-nfs-100g   100Gi     RWX           Retain          Available  <none>

Exercises

1. (Beginner) Is a PersistentVolume scoped to a namespace or to the cluster?
2. (Beginner) Whose lifecycle is a PV independent of?
3. (Intermediate) Who creates PVs, and how do end users typically obtain storage rather than referencing a PV directly?
4. (Interview) Explain how the PV represents the "supply" side of a producer/consumer model for storage and why that separation is useful. (Hint: admins/provisioners supply, users claim.)

Answers

1. Cluster-scoped (PVs are not namespaced).
2. Independent of any Pod's lifecycle — it persists across Pod deletion and rescheduling.
3. Administrators create PVs statically, or a StorageClass's provisioner creates them dynamically. End users don't reference PVs directly; they create a PersistentVolumeClaim requesting size/access mode/class, which binds to a matching PV.
4. The PV is the supply: administrators or dynamic provisioners create PVs representing real storage, decoupled from any consumer. Users express demand via PVCs (the consumer side), and Kubernetes binds claims to suitable PVs. This separation lets storage be provisioned, sized, and managed independently of workloads; users request abstract storage without knowing backend details, and admins control the available storage pool — clean division of responsibility and portability across environments.

---

PersistentVolumeClaim (PVC) resource

Theory

A PersistentVolumeClaim (PVC) is a request for storage by a user — the "demand" side. It is a namespaced object specifying how much storage is needed, the required access mode(s), and optionally a StorageClass. Kubernetes matches the PVC to a suitable PV (static binding) or triggers dynamic provisioning of a new PV (via the StorageClass). Once bound, the PVC is referenced by a Pod's volume, and the Pod gets the underlying storage.

The PVC is the abstraction that lets application authors request durable storage without knowing the backend: they ask for "20Gi, ReadWriteOnce, fast class," and the platform fulfills it with whatever storage backs that class (EBS, PD, NFS...). This is the user-facing storage primitive — you almost always create PVCs (often via a StatefulSet's volumeClaimTemplates), and the PV is provisioned for you behind the scenes.

Example

apiVersion: v1
kind: PersistentVolumeClaim
metadata: { name: data, namespace: app }
spec:
  accessModes: [ ReadWriteOnce ]
  storageClassName: fast            # triggers dynamic provisioning
  resources:
    requests: { storage: 20Gi }
---

# A Pod consumes the PVC by name:
spec:
  containers:
  - name: app
    image: myapp:1.0
    volumeMounts: [ { name: vol, mountPath: /data } ]
  volumes:
  - name: vol
    persistentVolumeClaim: { claimName: data }

kubectl get pvc -n app
# NAME   STATUS   VOLUME     CAPACITY   ACCESS MODES   STORAGECLASS
# data   Bound    pvc-9f..   20Gi       RWO            fast

Exercises

1. (Beginner) What is a PVC, and is it namespaced?
2. (Beginner) How does a Pod use a PVC?
3. (Intermediate) Describe what happens when you create a PVC referencing a StorageClass and no matching PV exists yet.
4. (Interview) How does the PVC abstraction let application developers request storage without knowing the underlying storage backend? (Hint: declarative request fulfilled by the platform/StorageClass.)

Answers

1. A PersistentVolumeClaim is a namespaced request for storage (size, access modes, optional StorageClass) made by a user/workload.
2. The Pod references the PVC by name in a persistentVolumeClaim volume and mounts it into a container via volumeMounts.
3. The StorageClass's provisioner dynamically creates a new PV that satisfies the PVC's requirements, then binds the PVC to it — the PVC goes from Pending to Bound and the storage becomes usable, with no manual PV creation.
4. The developer declares an abstract need — capacity, access mode, and a StorageClass name (a quality/tier) — without specifying any backend details. The platform (via the StorageClass's provisioner/CSI driver) fulfills that request with whatever storage backs the class (EBS, PD, NFS, Ceph). The same PVC manifest is therefore portable across clusters/clouds: only the StorageClass implementation differs, not the application's request.

---

Access modes: ReadWriteOnce, ReadOnlyMany, ReadWriteMany

Theory

Access modes describe how a volume may be mounted, and they are a frequent source of confusion and failure. The modes:

Mode	Abbrev	Meaning
ReadWriteOnce	RWO	Mounted read-write by a single node (multiple Pods on that node can share it).
ReadOnlyMany	ROX	Mounted read-only by many nodes.
ReadWriteMany	RWX	Mounted read-write by many nodes simultaneously.
ReadWriteOncePod	RWOP	Mounted read-write by a single Pod (strictest; newer).

The critical nuance: RWO is per-node, not per-Pod — historically misread. Most block storage (cloud disks like EBS, GCE PD) supports only RWO because a block device can attach to one node at a time. RWX requires a shared filesystem (NFS, CephFS, EFS, Azure Files) — you cannot get RWX from a plain block disk. Choosing an access mode the backend doesn't support leaves the PVC unbound or the Pod unschedulable.

Example

spec:
  accessModes: [ ReadWriteOnce ]    # one node R/W; typical for EBS/PD block storage
  # accessModes: [ ReadWriteMany ]  # needs NFS/CephFS/EFS — shared filesystem
  resources: { requests: { storage: 10Gi } }

Backend support (typical):
  AWS EBS / GCE PD / Azure Disk  -> RWO only (block device, single node)
  NFS / CephFS / AWS EFS / Azure Files -> RWX (shared filesystem)

Exercises

1. (Beginner) What do the abbreviations RWO, ROX, and RWX stand for?
2. (Beginner) RWO restricts read-write access to a single what — Pod or node?
3. (Intermediate) You need three Pods on three different nodes to write to the same volume. Which access mode and what kind of backend do you need?
4. (Interview) Why can a typical cloud block disk (EBS/PD) not provide ReadWriteMany, and what storage type is required instead? (Hint: block device single-attach vs. shared filesystem.)

Answers

1. ReadWriteOnce, ReadOnlyMany, ReadWriteMany (and the newer ReadWriteOncePod).
2. A single node (multiple Pods on that same node can share it; the newer ReadWriteOncePod restricts to a single Pod).
3. ReadWriteMany (RWX), backed by a shared filesystem such as NFS, CephFS, AWS EFS, or Azure Files — not a plain block disk.
4. A cloud block disk is a block device that can be attached to only one node at a time (single-attach), so it inherently supports only single-node access (RWO). ReadWriteMany requires concurrent multi-node read-write, which needs a shared/networked filesystem (NFS, CephFS, EFS, Azure Files) that coordinates concurrent access — capabilities a raw block volume doesn't provide.

---

Reclaim policies: Retain, Delete, Recycle

Theory

The reclaim policy (persistentVolumeReclaimPolicy) decides what happens to a PV — and its underlying storage — when its PVC is deleted. This governs whether your data is preserved or destroyed, so it's a high-stakes setting:

• Retain: keep the PV and its data after the PVC is deleted. The PV moves to Released (not reusable until an admin manually cleans it up). Safest for important data — nothing is auto-deleted.
• Delete: delete the PV and the underlying storage asset (the actual cloud disk) when the PVC is deleted. Convenient for dynamically provisioned, disposable volumes — but data is gone. This is the default for dynamically provisioned volumes.
• Recycle: deprecated — used to scrub data (rm -rf) and make the PV available again. Replaced by dynamic provisioning.

The crucial operational lesson: dynamically provisioned PVCs default to Delete, so deleting a PVC (or a namespace) can silently destroy production data unless you set Retain for anything important.

Example

# Set Retain so data survives PVC deletion (recommended for stateful data):
apiVersion: v1
kind: PersistentVolume
metadata: { name: pv-db }
spec:
  capacity: { storage: 50Gi }
  accessModes: [ ReadWriteOnce ]
  persistentVolumeReclaimPolicy: Retain     # Delete | Retain
  storageClassName: fast
  csi: { driver: ebs.csi.aws.com, volumeHandle: vol-0abc }

# After deleting the bound PVC, a Retain PV becomes Released (data kept):
kubectl get pv pv-db
# NAME    STATUS     RECLAIM POLICY   CLAIM
# pv-db   Released   Retain           app/data

Exercises

1. (Beginner) When does the reclaim policy take effect?
2. (Beginner) What is the default reclaim policy for dynamically provisioned volumes?
3. (Intermediate) Explain the difference in outcome between Retain and Delete when a PVC is deleted.
4. (Interview) A team deleted a namespace and lost their database's data. How does the default reclaim policy explain this, and how should they have prevented it? (Hint: Delete default; set Retain; backups.)

Answers

1. When the bound PVC is deleted — it determines the fate of the PV and its underlying storage at that point.
2. Delete.
3. With Retain, deleting the PVC leaves the PV (status Released) and the underlying storage intact, preserving the data until an admin manually handles it. With Delete, deleting the PVC also deletes the PV and the actual backend storage asset, destroying the data.
4. Dynamically provisioned PVs default to Delete. Deleting the namespace deleted the PVCs within it, which triggered deletion of the bound PVs and the underlying cloud disks — permanently destroying the database data. Prevention: set the reclaim policy (or StorageClass reclaimPolicy) to Retain for important volumes, protect critical namespaces/PVCs, and maintain independent backups/snapshots (e.g., Velero, volume snapshots) so data loss isn't tied to object deletion.

---

Volume binding modes

Theory

The volume binding mode (a StorageClass field, volumeBindingMode) controls when a PVC is bound to a PV and the backing volume is provisioned:

• Immediate (default): the PV is provisioned and bound as soon as the PVC is created, before any Pod uses it. Problem: in multi-zone clusters, the volume might be created in a zone where the Pod can't be scheduled (a zonal disk in zone A, but the Pod lands in zone B) — causing the Pod to be unschedulable.
• WaitForFirstConsumer: delay binding/provisioning until a Pod that uses the PVC is scheduled. The scheduler then considers the Pod's constraints (node affinity, zone, resources) and the volume is provisioned in the right zone for the Pod. This is the recommended mode for zonal block storage.

This mode is the fix for the classic "volume in the wrong zone" problem and is essential to understand for multi-zone clusters.

Example

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata: { name: fast }
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer   # provision in the Pod's zone
parameters: { type: gp3 }

Immediate:            PVC created -> volume provisioned in zone A immediately
                      Pod later scheduled in zone B -> cannot attach -> Pending

WaitForFirstConsumer: PVC created -> waits
                      Pod scheduled (zone B chosen) -> volume provisioned in zone B

Exercises

1. (Beginner) What does the volume binding mode control?
2. (Beginner) What is the default binding mode?
3. (Intermediate) Describe the multi-zone problem that Immediate binding can cause.
4. (Interview) Explain how WaitForFirstConsumer solves the zone-mismatch problem and why it requires deferring provisioning until scheduling. (Hint: scheduler knows the Pod's node/zone.)

Answers

1. When a PVC is bound to a PV and the underlying volume is provisioned (immediately vs. when first consumed by a scheduled Pod).
2. Immediate.
3. With Immediate, the volume is provisioned (e.g., in some zone) before the Pod is scheduled. If the volume is zonal (like a cloud block disk) and the Pod is later scheduled to a node in a different zone, the volume can't attach there, leaving the Pod stuck Pending/unschedulable.
4. WaitForFirstConsumer delays binding/provisioning until a Pod using the PVC is being scheduled. At that point the scheduler has chosen (or constrained) the Pod's node and zone, so the provisioner creates the volume in the same zone as the Pod, guaranteeing it can attach. It must defer until scheduling because only then are the Pod's placement constraints (node affinity, zone, resources) known — provisioning earlier would be a blind guess about location.

---

7.3 Storage Classes

StorageClasses turn manual PV creation into automated, on-demand provisioning, and let you offer differentiated storage tiers. This subchapter covers dynamic provisioning.

---

Dynamic provisioning with StorageClasses

Theory

Without automation, an admin must pre-create a PV for every storage need — tedious and slow. A StorageClass enables dynamic provisioning: when a PVC requests a StorageClass, Kubernetes automatically creates a matching PV (and the underlying storage) on demand, then binds it. No admin pre-provisioning required.

A StorageClass defines how to provision: a provisioner (which CSI driver to use), parameters (backend-specific options like disk type), a reclaimPolicy, and a volumeBindingMode. It effectively describes a "class" or tier of storage. The workflow: user creates a PVC referencing a StorageClass → the provisioner creates a new PV with the class's settings → PVC binds → Pod uses it. This is how storage works in virtually all real clusters today: users just request PVCs, and StorageClasses fulfill them automatically.

Example

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata: { name: fast }
provisioner: ebs.csi.aws.com         # which CSI driver provisions the volume
parameters:                          # backend-specific
  type: gp3
  iops: "3000"
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

# User just creates a PVC referencing the class; a PV appears automatically:
kubectl get pvc,pv
# pvc/data   Bound   pvc-7a..   20Gi   RWO   fast
# pv/pvc-7a..  20Gi  RWO  Delete  Bound  app/data  fast   <- auto-created

Exercises

1. (Beginner) What does dynamic provisioning eliminate the need for?
2. (Beginner) What does the provisioner field of a StorageClass specify?
3. (Intermediate) Walk through the sequence of events from creating a PVC with a StorageClass to a Pod using the storage.
4. (Interview) How do StorageClasses let a platform team offer differentiated storage tiers (e.g., "fast" vs. "cheap") to application teams? (Hint: classes as named tiers with distinct parameters.)

Answers

1. Manually pre-creating PersistentVolumes — the StorageClass provisions them on demand automatically.
2. Which provisioner/CSI driver to use to create the underlying storage (e.g., ebs.csi.aws.com).
3. The user creates a PVC referencing a StorageClass → the StorageClass's provisioner dynamically creates a new PV (the actual disk) using the class's parameters/reclaim policy → the PVC binds to that PV → a Pod referencing the PVC is scheduled and the volume is attached/mounted into the container.
4. The platform team defines multiple StorageClasses, each a named tier with its own provisioner/parameters — e.g., fast (SSD/high-IOPS, possibly Retain) and cheap (HDD/standard, Delete). App teams simply pick a class name in their PVCs to get the desired performance/cost tier without knowing backend details. The class name becomes the contract for a storage quality level, and the platform controls what each tier actually maps to.

---

Provisioner types

Theory

The provisioner is the engine that actually creates storage for a StorageClass. Provisioners are essentially CSI drivers (the in-tree kubernetes.io/* provisioners are deprecated/migrated). Each cloud and storage system provides one:

Provisioner	Backend
ebs.csi.aws.com	AWS EBS block volumes
pd.csi.storage.gke.io	Google Persistent Disk
disk.csi.azure.com / file.csi.azure.com	Azure Disk / Azure Files
efs.csi.aws.com	AWS EFS (RWX shared)
rook-ceph.rbd.csi.ceph.com	Ceph (on-prem)

The parameters map passes provisioner-specific options (disk type, IOPS, throughput, filesystem, encryption). There's also a special provisioner, kubernetes.io/no-provisioner, used with local PVs (node-attached disks) where there's nothing to dynamically provision — binding is static but the StorageClass coordinates WaitForFirstConsumer scheduling. Choosing the right provisioner and parameters is how you match storage capability (performance, access mode, durability) to workload needs.

Example

# AWS EBS gp3 with custom IOPS/throughput and encryption:
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  iops: "5000"
  throughput: "250"
  encrypted: "true"
---

# Local PV class: nothing to provision dynamically, but coordinates scheduling
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

Exercises

1. (Beginner) What are modern provisioners essentially implemented as?
2. (Beginner) What does the parameters field of a StorageClass carry?
3. (Intermediate) When and why would you use the kubernetes.io/no-provisioner provisioner?
4. (Interview) Given a workload needing shared read-write access across nodes, which AWS provisioner would you choose over EBS, and why? (Hint: RWX needs a shared filesystem like EFS.)

Answers

1. CSI drivers (the in-tree provisioners have been deprecated and migrated to CSI).
2. Provisioner/backend-specific options — e.g., disk type, IOPS, throughput, filesystem, encryption — passed to the driver when creating the volume.
3. For local PersistentVolumes (storage physically attached to specific nodes). There's nothing to provision on demand, so no-provisioner is used; the StorageClass still coordinates WaitForFirstConsumer so Pods are scheduled to the node holding the local volume. Used for high-performance local disks where data locality matters and you accept node-bound storage.
4. Use efs.csi.aws.com (AWS EFS) instead of ebs.csi.aws.com. EBS is a single-attach block device (RWO), so it can't provide cross-node read-write. EFS is a shared NFS-based filesystem supporting ReadWriteMany, allowing multiple Pods on different nodes to read and write the same volume concurrently.

---

Default StorageClass

Theory

A cluster can designate one StorageClass as the default, marked with the annotation storageclass.kubernetes.io/is-default-class: "true". When a PVC is created without an explicit storageClassName, it uses the default StorageClass — so users get sensible storage automatically without specifying a class.

Two important nuances: (1) If no default is set and a PVC omits the class, the PVC stays Pending (nothing provisions it) — a common "why is my PVC stuck?" cause. (2) Specifying storageClassName: "" (empty string) explicitly disables dynamic provisioning for that PVC, forcing it to bind only to a manually created PV — different from omitting the field entirely. Managed clusters usually ship with a default class (e.g., gp2/gp3 on EKS, standard on GKE). Having exactly one default avoids ambiguity.

Example

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"   # the default class
provisioner: pd.csi.storage.gke.io

kubectl get storageclass
# NAME                 PROVISIONER              ...   AGE
# standard (default)   pd.csi.storage.gke.io          40d
# fast                 pd.csi.storage.gke.io          40d
# A PVC with no storageClassName uses "standard".

Exercises

1. (Beginner) How is a StorageClass marked as the default?
2. (Beginner) What StorageClass does a PVC use if it omits storageClassName?
3. (Intermediate) A PVC without a storageClassName is stuck Pending. What cluster condition would explain this?
4. (Interview) Explain the difference between omitting storageClassName and setting it to "" (empty string). (Hint: default class vs. disabling dynamic provisioning.)

Answers

1. With the annotation storageclass.kubernetes.io/is-default-class: "true" on the StorageClass.
2. The cluster's default StorageClass (if one is configured).
3. There is no default StorageClass configured in the cluster, so a PVC that omits the class has nothing to provision it and remains Pending. (Setting a default class, or adding an explicit class to the PVC, resolves it.)
4. Omitting storageClassName makes the PVC use the cluster's default StorageClass (dynamic provisioning via that class). Setting storageClassName: "" explicitly disables dynamic provisioning for the PVC — it will only bind to a pre-existing, manually created PV with no class, and will not trigger any provisioner. They are semantically different: "use the default" vs. "use no class / static binding only."

---

Volume expansion

Theory

Storage needs grow. Volume expansion lets you increase a PVC's size after creation, without recreating the volume or losing data — provided the StorageClass has allowVolumeExpansion: true and the underlying CSI driver supports it. You expand by editing the PVC's resources.requests.storage to a larger value; the provisioner grows the backend disk, and (for filesystem volumes) the filesystem is resized to use the new space.

Important constraints: you can only grow, never shrink, a volume. Some drivers can expand online (while the Pod is running); others require the Pod to be restarted to complete the filesystem resize. And only dynamically provisioned volumes with an expansion-capable class/driver qualify. This avoids the painful old workflow of provisioning a new larger volume and copying data, making capacity growth a simple declarative edit.

Example

# 1) StorageClass must permit expansion:
allowVolumeExpansion: true

# 2) Edit the PVC to request more storage (grow only):
kubectl patch pvc data -p '{"spec":{"resources":{"requests":{"storage":"50Gi"}}}}'

kubectl get pvc data
# NAME   STATUS   CAPACITY   ...   CONDITIONS
# data   Bound    20Gi             FileSystemResizePending   <- may need Pod restart
# (after resize) data  Bound  50Gi

Exercises

1. (Beginner) What StorageClass setting must be enabled to allow expanding a PVC?
2. (Beginner) Can you shrink a volume with volume expansion?
3. (Intermediate) How do you trigger an expansion on an existing PVC, and what might be required to complete the filesystem resize?
4. (Interview) Why is declarative volume expansion a significant improvement over the pre-expansion workflow of provisioning a new volume and copying data? (Hint: downtime, risk, simplicity, no data migration.)

Answers

1. allowVolumeExpansion: true (and the CSI driver must support expansion).
2. No — expansion only grows a volume; shrinking is not supported.
3. Edit/patch the PVC's spec.resources.requests.storage to a larger value. The provisioner grows the backend disk; completing the filesystem resize may require the Pod to be restarted (offline resize) unless the driver supports online expansion (the PVC may show a FileSystemResizePending condition until done).
4. The old approach meant creating a new larger volume, attaching it, copying all data over, switching the workload, and decommissioning the old volume — slow, error-prone, and usually involving downtime and risk of data loss. Declarative expansion is a single edit that grows the existing volume in place (often online), preserving data and identity with minimal or no disruption — far simpler, safer, and faster.

---

7.4 Container Storage Interface (CSI)

CSI is the standard that makes all of the above storage pluggable. This subchapter goes deeper into the driver architecture and advanced capabilities like snapshots.

---

CSI driver architecture

Theory

A CSI driver is the out-of-tree component implementing the CSI gRPC specification so Kubernetes can manage a specific storage backend. Architecturally, a CSI driver typically has two parts deployed in the cluster:

• Controller plugin (a Deployment/StatefulSet, usually one): handles cluster-wide operations — provisioning/deleting volumes, attaching/detaching them to nodes, creating snapshots. It runs alongside Kubernetes sidecar containers (external-provisioner, external-attacher, external-snapshotter, external-resizer) that watch Kubernetes objects (PVCs, VolumeAttachments) and translate them into CSI calls.
• Node plugin (a DaemonSet, one per node): handles node-local operations — mounting/unmounting the volume into the Pod's filesystem and formatting if needed. It runs the node-driver-registrar sidecar to register the driver with the kubelet.

This split mirrors the CSI spec's Controller and Node services. Understanding it helps debug storage issues: provisioning failures point at the controller plugin; mount failures point at the node plugin on the relevant node.

Example

Controller plugin (Deployment, cluster-wide):
  [csi-provisioner][csi-attacher][csi-snapshotter][csi-resizer][CSI driver]
        watch PVCs/VolumeAttachments -> CreateVolume/ControllerPublishVolume

Node plugin (DaemonSet, per node):
  [node-driver-registrar][CSI driver]  -> NodeStageVolume/NodePublishVolume (mount)

kubectl get pods -n kube-system -l app=ebs-csi-controller    # controller plugin
kubectl get pods -n kube-system -l app=ebs-csi-node -o wide  # node plugin (per node)

Exercises

1. (Beginner) What are the two main components of a CSI driver deployment?
2. (Beginner) Which component runs as a DaemonSet, and what does it do?
3. (Intermediate) A volume provisions fine but fails to mount on one specific node. Which CSI component would you investigate, and where?
4. (Interview) Why does CSI split functionality into Controller and Node services, and how do the Kubernetes sidecar containers bridge Kubernetes objects to CSI calls? (Hint: cluster-wide vs. node-local ops; watchers translating to gRPC.)

Answers

1. The Controller plugin (cluster-wide provisioning/attach/snapshot) and the Node plugin (node-local mount/unmount).
2. The Node plugin runs as a DaemonSet (one per node); it handles mounting/unmounting (and formatting) the volume into Pods on that node and registers the driver with the kubelet.
3. The Node plugin (DaemonSet Pod) on that specific node — check its logs/events, since mounting is a node-local operation handled by the node plugin and kubelet. Provisioning working but mounting failing isolates the problem to node-side staging/publishing.
4. CSI separates cluster-wide operations (create/delete/attach/snapshot a volume — done once, centrally) from node-local operations (stage/mount into a particular node's filesystem — done where the Pod runs), matching how storage actually works. Kubernetes provides sidecar containers (external-provisioner, -attacher, -snapshotter, -resizer, node-driver-registrar) that watch Kubernetes API objects (PVCs, VolumeAttachments, VolumeSnapshots) and translate those into the corresponding CSI gRPC calls on the driver. This keeps the driver focused on talking to its backend while the sidecars handle Kubernetes integration.

---

Popular CSI drivers (AWS EBS, GCE PD, NFS)

Theory

In practice you'll work with a handful of common CSI drivers, each with characteristic capabilities:

• AWS EBS CSI (ebs.csi.aws.com): provisions EBS block volumes — RWO, zonal, high-performance; the default block storage on EKS.
• GCE PD CSI (pd.csi.storage.gke.io): Google Persistent Disks — RWO (regional PD variants offer cross-zone replication); default on GKE.
• Azure Disk / Azure Files CSI: block (RWO) and SMB file shares (RWX) respectively.
• NFS CSI (nfs.csi.k8s.io): exposes an existing NFS server as RWX storage — simple shared filesystem, common on-prem.
• Ceph/Rook, Portworx, OpenEBS: software-defined storage for on-prem/hybrid, offering replication, snapshots, and more.

The selection criteria are access mode (block RWO vs. shared RWX), performance, durability/replication, cost, and environment (cloud vs. on-prem). Knowing each driver's access-mode and topology characteristics prevents the common mistake of expecting RWX from a block driver.

Example

Driver	Access modes	Topology	Typical use
ebs.csi.aws.com	RWO	Zonal	Databases, single-writer apps on EKS
efs.csi.aws.com	RWX	Regional	Shared files across Pods/nodes on EKS
pd.csi.storage.gke.io	RWO (regional opt.)	Zonal/Regional	General block storage on GKE
nfs.csi.k8s.io	RWX	Network	On-prem shared storage

Exercises

1. (Beginner) Which AWS CSI driver provides shared ReadWriteMany storage?
2. (Beginner) What access mode do block-storage drivers like EBS and GCE PD typically provide?
3. (Intermediate) You're on-prem and need RWX shared storage for several Pods. Name a suitable CSI driver.
4. (Interview) When selecting a CSI driver for a stateful workload, what characteristics must you match to the workload's needs? (Hint: access mode, topology/zone, performance, durability, environment.)

Answers

1. efs.csi.aws.com (AWS EFS) — it provides RWX via a shared NFS-based filesystem.
2. ReadWriteOnce (RWO) — single-node block attachment.
3. The NFS CSI driver (nfs.csi.k8s.io) backed by an NFS server, or a software-defined storage system like Ceph/Rook, Portworx, or OpenEBS that offers RWX.
4. Match: access mode (does it need single-node RWO or multi-node RWX?), topology (zonal vs. regional/replicated, and zone-aware scheduling), performance (IOPS/throughput tier), durability/replication (snapshots, cross-zone replication, RPO needs), cost, and environment (cloud-native vs. on-prem software-defined). Choosing a driver whose capabilities don't fit (e.g., expecting RWX from EBS) leads to unbound PVCs or unschedulable Pods.

---

Volume snapshots and cloning

Theory

CSI standardizes volume snapshots — point-in-time copies of a volume — through dedicated API objects (provided by CRDs and the external-snapshotter):

• VolumeSnapshotClass: like a StorageClass but for snapshots (which driver, parameters).
• VolumeSnapshot: a request to snapshot a specific PVC.
• VolumeSnapshotContent: the actual snapshot resource (analogous to PV vs. PVC).

You can then restore a snapshot by creating a new PVC with a dataSource referencing the VolumeSnapshot — producing a new volume with the snapshot's data. Cloning is similar but copies directly from an existing PVC (dataSource of kind PersistentVolumeClaim) without an intermediate snapshot. These enable backups, test/dev data copies, and fast environment duplication — provided the CSI driver supports snapshots/cloning. (Snapshots are crash-consistent at the block level; application-consistent backups may need app coordination, e.g., quiescing a database.)

Example

# Snapshot an existing PVC:
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata: { name: db-snap }
spec:
  volumeSnapshotClassName: csi-snap
  source: { persistentVolumeClaimName: data }
---

# Restore it into a new PVC:
apiVersion: v1
kind: PersistentVolumeClaim
metadata: { name: data-restored }
spec:
  storageClassName: fast
  dataSource:                          # restore from snapshot
    name: db-snap
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes: [ ReadWriteOnce ]
  resources: { requests: { storage: 20Gi } }

Exercises

1. (Beginner) What are the three main snapshot-related API objects, and which is analogous to a PV?
2. (Beginner) How do you restore a snapshot into a usable volume?
3. (Intermediate) What is the difference between cloning a PVC and snapshotting then restoring it?
4. (Interview) A CSI snapshot is "crash-consistent." Why might that be insufficient for a database backup, and what extra step ensures application consistency? (Hint: in-flight writes/buffers; quiesce/flush.)

Answers

1. VolumeSnapshotClass, VolumeSnapshot, and VolumeSnapshotContent. VolumeSnapshotContent is analogous to a PV (the actual backing snapshot), while VolumeSnapshot is analogous to a PVC (the request/claim).
2. Create a new PVC with a dataSource referencing the VolumeSnapshot (kind VolumeSnapshot); the provisioner creates a new volume populated with the snapshot's data.
3. Cloning copies data directly from an existing PVC (dataSource kind PersistentVolumeClaim) into a new PVC, with no separate snapshot object. Snapshot+restore first creates a point-in-time VolumeSnapshot, which can be retained/backed up and later restored into one or more new PVCs. Cloning is a one-step copy; snapshots provide a reusable, retainable restore point.
4. Crash-consistent means the snapshot captures the disk exactly as it was at an instant — like the state after a power loss — which may include in-flight transactions and unflushed buffers, so a database might need recovery on restore or have inconsistent state. For application consistency you coordinate with the app: quiesce/flush it (e.g., put the DB in backup/freeze mode or flush and lock tables) so on-disk data is consistent at snapshot time, then take the snapshot, then resume. Tools/operators often automate this freeze-snapshot-thaw sequence.

---

CSI ephemeral volumes

Theory

Usually CSI volumes are durable and managed via PVCs. But sometimes you want a CSI-backed volume whose lifecycle is tied to the Pod — created when the Pod starts and destroyed when it stops — without the PV/PVC machinery. CSI supports two ephemeral patterns:

• Generic ephemeral volumes: defined inline in the Pod spec but using the full StorageClass/PVC machinery under the hood (a PVC is auto-created and deleted with the Pod). Good for scratch space that needs real provisioned storage (size, type) but no persistence.
• CSI inline ephemeral volumes: the CSI driver itself provides a volume inline in the Pod spec (the driver must support this mode). Common for drivers that inject data rather than persistent storage — e.g., the Secrets Store CSI driver mounting secrets, or a driver providing certificates.

The key idea: ephemeral CSI volumes give you driver-backed storage/data with Pod-coupled lifecycle, bridging the gap between trivial emptyDir and full persistent PVCs.

Example

# Generic ephemeral volume: real provisioned storage, deleted with the Pod
spec:
  containers:
  - name: app
    image: myapp:1.0
    volumeMounts: [ { name: scratch, mountPath: /scratch } ]
  volumes:
  - name: scratch
    ephemeral:
      volumeClaimTemplate:
        spec:
          accessModes: [ ReadWriteOnce ]
          storageClassName: fast
          resources: { requests: { storage: 5Gi } }
---

# CSI inline ephemeral (e.g., Secrets Store CSI driver injecting secrets):
  volumes:
  - name: secrets
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes: { secretProviderClass: "vault-db" }

Exercises

1. (Beginner) What lifecycle are CSI ephemeral volumes tied to?
2. (Beginner) Name the two kinds of CSI ephemeral volumes.
3. (Intermediate) When would you use a generic ephemeral volume instead of an emptyDir?
4. (Interview) The Secrets Store CSI driver uses inline ephemeral volumes. Why is an ephemeral, Pod-coupled, driver-provided volume the right model for injecting secrets at runtime? (Hint: no persistence in etcd, lifecycle matches Pod, fetched on demand.)

Answers

1. The Pod's lifecycle — created when the Pod starts and deleted when the Pod is removed.
2. Generic ephemeral volumes (inline, but using StorageClass/PVC machinery auto-managed with the Pod) and CSI inline ephemeral volumes (provided directly by a CSI driver in the Pod spec).
3. When you need real provisioned storage with specific properties (size, type, performance from a StorageClass) for scratch/temporary data, but don't need it to persist beyond the Pod. emptyDir is limited to node-local disk/RAM with no provisioning controls; a generic ephemeral volume gives you proper CSI-backed storage that's still automatically cleaned up with the Pod.
4. Secrets should be fetched on demand and never linger. An inline ephemeral CSI volume is created when the Pod starts (the driver fetches secrets from the external store at mount time), exists only for the Pod's life, and is torn down with it — so the secret isn't stored as a Kubernetes Secret in etcd, isn't persisted beyond use, and its lifecycle exactly matches the consuming Pod. This minimizes exposure and keeps secret material out of the cluster datastore.

---