6. Data Persistence and Storage

Containers are designed to be disposable, but the data they produce (database files, uploads, caches, configuration) usually is not. This chapter explains where a container's files physically live, why anything written to the container filesystem is lost when the container is removed, and how volumes, bind mounts, and tmpfs mounts let you choose exactly where data is stored and how long it survives. It closes with the operational side of state: backing up, restoring, and migrating data, and running databases in containers without losing sleep (or rows).

<a id="61-storage-concepts"></a>

6.1 Storage Concepts

This section explains how a container's own filesystem is built and stored, which is the foundation for understanding why external storage is needed.

Ephemeral Container Filesystem

Why "ephemeral" is a feature. A container is meant to be a reproducible instance of an image: you should be able to delete it and start an identical one at any time, on any host. That is only possible if the container does not accumulate important, unique state of its own. Docker enforces this by tying the container's private filesystem changes to the container's lifetime: when the container is removed, its changes are removed too.

Analogy. A container is like a hotel room. Housekeeping resets it to the same standard state for every guest (the image). You can rearrange the furniture and leave things on the desk during your stay (stopping and starting the container keeps them), but when you check out (docker rm), everything you left behind is cleared. Valuables belong in the safe deposit box at the front desk, which is the role volumes play.

What survives what

Event	Changes in the container filesystem	Data in a volume or bind mount
Process inside restarts (restart policy)	Kept	Kept
docker stop / docker start	Kept	Kept
docker restart, host reboot	Kept	Kept
docker rm	Deleted	Kept
Recreate with a new image version (docker compose up after a pull, rolling update)	Deleted (it is a new container)	Kept
docker run again from the same image	Not visible: a new container starts from the clean image	Visible, if mounted again

The last two rows are the ones that surprise people: upgrading an application always means replacing the container, so any state kept inside it is lost on every deployment.

The three ways out of the container filesystem

                         +-------------------------------+
                         |          container            |
                         |                               |
                         |  /            container layer  |  ephemeral: dies with
                         |  |            (copy-on-write)  |  the container
                         |  +-- /var/lib/db  ------------+-----> volume       (managed by Docker)
                         |  +-- /app/src     ------------+-----> bind mount   (any host path)
                         |  +-- /run/secret  ------------+-----> tmpfs        (host RAM only)
                         +-------------------------------+

Mount type	Stored in	Survives docker rm	Typical use
Volume	Docker-managed area on the host (or a remote driver)	Yes	Databases, uploads, any application state
Bind mount	An existing host path you choose	Yes (it is just a host directory)	Source code in development, host config files
tmpfs	Host memory	No (gone on stop)	Scratch space, secrets, sensitive temporary files

Example: proving the lifecycle

# Write a file into the container filesystem
docker run -d --name eph alpine:3.20 sleep 3600
docker exec eph sh -c 'echo "hello" > /note.txt'

# Stop/start keeps the writable layer
docker stop -t 1 eph && docker start eph
docker exec eph cat /note.txt                                         # prints "hello"

# Remove the container: the file is gone with it
docker rm -f eph
docker run --rm alpine:3.20 cat /note.txt
# cat: can't open '/note.txt': No such file or directory

# Same experiment with a volume: the data outlives both containers
docker run --rm -v notes:/data alpine:3.20 sh -c 'echo "hello" > /data/note.txt'
docker run --rm -v notes:/data alpine:3.20 cat /data/note.txt         # prints "hello"
docker volume rm notes

Design consequence. Treat the container filesystem as read-mostly scratch space. Many teams go further and run containers with --read-only, so any accidental write to the container filesystem fails loudly instead of silently disappearing later (see Read-Only Root Filesystem).

Key takeaways

• Changes to a container's filesystem survive stop, start, and restart, but are deleted with docker rm.
• Every upgrade or redeploy creates a new container, so in-container state is lost on each deployment.
• Persistent data must go to a volume or bind mount; temporary sensitive data can go to tmpfs.
• Treat the container filesystem as disposable; --read-only makes accidental writes visible.

🧪 Practice

1. (Beginner) Start an nginx:alpine container, edit /usr/share/nginx/html/index.html with docker exec, then stop and start it. Is the edit still there? Remove and recreate the container and check again.
2. (Intermediate) Run redis:7 with --save 60 1 (snapshot every 60 seconds if at least one key changed), set a key, wait a minute, and recreate the container with and without a volume at /data. Explain the result in terms of the table above.
3. Interview question: "A service stored user uploads under /app/uploads and they vanished after Friday's deployment, although nobody deleted anything. What happened?" Hint: think about what a deployment does to the existing container and where those files were written.

Writable Container Layer

Why it exists. Applications expect to write files: PID files, logs, caches, temporary files. Image layers are read-only and shared, so each container needs a private place to put its writes. Docker gives every container one thin writable layer on top of the image layers (see Copy-on-Write for the mechanics). This section focuses on the practical side: where that layer lives, how to see what is in it, and why it should stay small.

Where it lives. With the classic overlay2 storage driver, each container has a directory under /var/lib/docker/overlay2/<id>/, and its changes are in the diff/ subdirectory. With the containerd image store, the equivalent is a containerd snapshot under the containerd root. In both cases it is plain files on the host's disk, inside Docker's data root, and it is deleted when the container is removed.

 /var/lib/docker/
 +-- overlay2/<container-layer-id>/
 |   +-- diff/       <- files the container created or modified (the writable layer)
 |   +-- merged/     <- the combined view mounted as the container's /
 |   +-- work/       <- OverlayFS scratch directory
 +-- containers/<container-id>/
     +-- <id>-json.log   <- stdout/stderr logs: NOT part of the writable layer
     +-- config.v2.json, hostconfig.json, hostname, hosts, resolv.conf ...

What is (and is not) in the writable layer

In the writable layer	Not in the writable layer
Files created or modified anywhere under /	Contents of volumes, bind mounts, and tmpfs mounts
Copied-up versions of modified image files	Container logs captured from stdout/stderr
Whiteouts for deleted image files	/etc/hosts, /etc/resolv.conf, /etc/hostname (bind-mounted by Docker)
Package caches, /tmp, log files the app writes itself	The image layers

Inspecting it

docker run -d --name web nginx:alpine
docker exec web sh -c 'echo test > /tmp/x; rm /etc/nginx/conf.d/default.conf'

# What changed relative to the image (A = added, C = changed, D = deleted)
docker diff web
# C /tmp
# A /tmp/x
# C /etc/nginx/conf.d
# D /etc/nginx/conf.d/default.conf
# ...

# Size of the writable layer vs. virtual size (writable + image)
docker ps -s --filter name=web --format '{{.Names}}\t{{.Size}}'
# web   1.1kB (virtual 52.4MB)

# Physical location (overlay2 storage driver)
docker inspect -f '{{.GraphDriver.Data.UpperDir}}' web
docker rm -f web

Why it should stay small

• Data loss. Anything there disappears on docker rm.
• Performance. Writes pass through the union filesystem, and the first modification of a large image file copies the whole file up.
• Disk exhaustion. By default the layer can grow until the host disk (Docker's data root) is full, affecting every container on the host. A per-container quota (--storage-opt size=10G) exists, but with overlay2 it only works when the data root is on XFS mounted with the pquota option.
• Opacity. docker commit and docker export capture the writable layer, which tempts people to "save" containers instead of fixing the Dockerfile. Note that both ignore volume contents.

Common accidental writers. Applications logging to files instead of stdout, package managers run at container start, language caches (~/.cache/pip, ~/.npm), and uploads saved to a working directory. Point these to stdout, a volume, or a tmpfs mount.

Key takeaways

• Each container has exactly one private writable layer on top of the shared, read-only image layers.
• It lives inside Docker's data root on the host disk and is deleted with the container.
• Volumes, bind mounts, tmpfs mounts, and stdout logs are not part of it.
• Use docker diff and docker ps -s to see what a container has written and how much.
• Keep it small: write-heavy or important data belongs elsewhere.

🧪 Practice

1. (Beginner) Run docker diff on a freshly started postgres:16 container (with POSTGRES_PASSWORD set) and on one started with a named volume at /var/lib/postgresql/data. Compare the outputs.
2. (Intermediate) Run a container that writes 500 MB to /tmp with dd, check docker ps -s and docker system df, then repeat with --tmpfs /tmp. Explain where the bytes went each time.
3. Interview question: "Disk usage under /var/lib/docker keeps growing on a production host, but images and volumes have not changed. Where would you look?" Hint: consider writable layers of long-running containers, log files under containers/, and how to find which container is responsible.

Storage Drivers

Why storage drivers exist. The layered image model needs an implementation: something must stack read-only layers, add a writable layer, and handle copy-on-write. Linux offers several technologies that can do this (union filesystems, filesystem snapshots, block-level snapshots), each with different performance and requirements. A storage driver is the pluggable component of the Docker daemon that implements image and container layers on top of one of these technologies.

Analogy. The storage driver is the warehouse system behind a shop. Customers (containers) see the same shelves (a filesystem) no matter whether the warehouse uses forklifts or conveyor belts. The choice still matters for speed, capacity, and which buildings (backing filesystems) it can operate in.

Scope: what a storage driver does not handle. Storage drivers only manage image layers and container writable layers. Volumes and bind mounts bypass the storage driver completely and write directly to the host filesystem (or a volume driver's backend). This is the main reason volumes are faster for write-heavy workloads.

   container process writes to ...
        |
        +-- /            -> storage driver (overlay2, ...) -> /var/lib/docker/overlay2/...
        |
        +-- /data (vol)  -> direct host filesystem I/O      -> /var/lib/docker/volumes/...

Available drivers

Driver	Technology	Status and use
overlay2	OverlayFS (kernel union filesystem)	Default and recommended on all current Linux distributions
fuse-overlayfs	OverlayFS in user space (FUSE)	Rootless Docker on kernels older than 5.11; slower
btrfs	Btrfs subvolumes and snapshots	Only if the data root is on Btrfs
zfs	ZFS datasets and snapshots	Only if the data root is on ZFS; niche
vfs	Full copy of each layer, no copy-on-write	Testing or unsupported filesystems; very slow and disk-hungry
aufs, devicemapper	Legacy union / block-level snapshots	Removed or deprecated; migrate to overlay2

containerd image store and snapshotters. Recent Docker Engine releases (and Docker Desktop) can delegate image and layer storage to containerd. In that mode the classic storage drivers are replaced by containerd snapshotters (overlayfs, native, btrfs, zfs, and lazy-pulling snapshotters such as stargz). The concepts in this chapter are identical; only the on-disk location and some inspect output differ. See containerd Image Store.

Checking and configuring the driver

docker info --format '{{.Driver}}'                   # e.g. overlay2 (or overlayfs with containerd)
docker info | grep -A5 'Storage Driver'
# Storage Driver: overlay2
#  Backing Filesystem: extfs
#  Supports d_type: true
#  Using metacopy: false
#  Native Overlay Diff: true
#  userxattr: false

{
  "storage-driver": "overlay2"
}

The JSON above goes in /etc/docker/daemon.json and requires a daemon restart. Changing the driver does not convert existing data: images and containers stored by the old driver become invisible (they stay on disk until you delete them). Plan a driver change like a migration: push images to a registry and back up volumes first.

Choosing a driver. In practice: use overlay2 unless you have a specific reason not to. Docker selects it automatically when the kernel and backing filesystem support it. Pick btrfs or zfs only if the host is already standardized on that filesystem and you want to use its native snapshots.

Key takeaways

• A storage driver implements image layers and the container writable layer; nothing else.
• Volumes and bind mounts bypass the storage driver, which is why they are faster for heavy writes.
• overlay2 is the default and the right choice on almost every Linux host.
• With the containerd image store, snapshotters play the same role as storage drivers.
• Changing the storage driver hides existing images and containers; treat it as a migration.

🧪 Practice

1. (Beginner) Run docker info and record the storage driver, backing filesystem, and whether d_type is supported. Find the corresponding directories under the data root.
2. (Intermediate) On a disposable VM, start the daemon with "storage-driver": "vfs", pull an image with several layers, and compare the disk usage under the data root with overlay2. Explain the difference.
3. Interview question: "Why does writing a database to a named volume usually perform better than writing to the container filesystem, even though both end up on the same disk?" Hint: follow the I/O path through the storage driver and think about copy-up.

overlay2

Why overlay2 won. Older drivers either were not in the mainline kernel (AUFS) or worked at block level with complex setup and poor density (devicemapper). OverlayFS is part of the mainline Linux kernel, works at the file level on ordinary ext4 or XFS filesystems, shares the page cache between containers that use the same image, and needs no configuration. overlay2 is Docker's driver built on it, and it supports up to 128 lower layers natively.

How Docker maps layers to OverlayFS. Each image layer is a directory. When a container starts, Docker mounts an OverlayFS whose lowerdir is the list of image layer directories, whose upperdir is the container's own layer, and whose mount point (merged) becomes the container's root filesystem.

 /var/lib/docker/overlay2/
 +-- l/                         short symlinks (e.g. l/ABCD1234 -> ../<layer-id>/diff)
 |                              used to keep the mount option string under one memory page
 +-- <layer-id-1>/              base image layer
 |   +-- diff/                  the layer's files
 |   +-- link                   its short name in l/
 +-- <layer-id-2>/              next image layer
 |   +-- diff/  link
 |   +-- lower                  "l/ABCD1234"  (layers below this one, top to bottom)
 +-- <container-id>/            container layer
     +-- diff/                  upperdir: the container's changes
     +-- work/                  workdir: OverlayFS internal scratch space
     +-- merged/                the mount point -> container's /
     +-- lower                  "l/EFGH5678:l/ABCD1234"

 mount -t overlay overlay \
   -o lowerdir=l/EFGH5678:l/ABCD1234,upperdir=<container-id>/diff,workdir=<container-id>/work \
   <container-id>/merged

Example: following a container to its overlay mount

docker run -d --name ov nginx:alpine
docker inspect -f '{{json .GraphDriver}}' ov | python3 -m json.tool
# {
#   "Data": {
#     "LowerDir": "/var/lib/docker/overlay2/<id>-init/diff:/var/lib/docker/overlay2/<layer>/diff:...",
#     "MergedDir": "/var/lib/docker/overlay2/<id>/merged",
#     "UpperDir": "/var/lib/docker/overlay2/<id>/diff",
#     "WorkDir": "/var/lib/docker/overlay2/<id>/work"
#   },
#   "Name": "overlay2"
# }

# The live kernel mount backing the container (run on the Docker host, as root)
mount | grep "$(docker inspect -f '{{.GraphDriver.Data.MergedDir}}' ov)"

# Create a file in the container and find it in the upper directory on the host
docker exec ov sh -c 'echo hi > /hello.txt'
sudo cat "$(docker inspect -f '{{.GraphDriver.Data.UpperDir}}' ov)/hello.txt"   # hi
docker rm -f ov

The <id>-init layer in LowerDir is a small layer Docker creates for each container with files it manages (/etc/hosts, /etc/resolv.conf placeholders), sitting between the image and the container layer.

Requirements and behavior to know

Topic	Detail
Backing filesystem	ext4, or XFS formatted with ftype=1 (d_type support); check "Supports d_type: true" in docker info
Page cache sharing	Containers from the same image share cached file pages, so memory use scales well
Copy-up cost	The first write to a large existing file copies the entire file; noticeable with big files
Inodes	Many small files across many layers can exhaust inodes on ext4 before space runs out
Directory renames	Renaming a directory that exists in a lower layer is handled by copy-up or redirect metadata; some old tools assumed a simpler model
Open file descriptors	A file opened read-only before being copied up keeps pointing to the lower version (older kernels); rarely matters in practice
Disk quotas	--storage-opt size= requires XFS with project quotas (pquota)

Key takeaways

• overlay2 stacks image layer directories (lowerdir) under a per-container upperdir and mounts the result as /.
• Each layer is a diff/ directory under /var/lib/docker/overlay2/; short links in l/ keep mount options short.
• It shares the page cache across containers and needs no setup on ext4 or XFS (ftype=1).
• Copy-up of large files and inode usage are the main performance and capacity concerns.
• docker inspect (GraphDriver) shows the exact directories used by a container.

🧪 Practice

1. (Beginner) Use docker inspect to find the UpperDir of a running container, create a file inside the container, and locate it on the host.
2. (Intermediate) Pull an image with many layers (for example python:3.13), find its layer directories under /var/lib/docker/overlay2, and use the lower files to reconstruct the layer order. Compare it with docker image history.
3. Interview question: "Why does Docker create the l/ directory of short symlinks instead of passing the full layer paths to the mount call?" Hint: think about how mount options are passed to the kernel and what happens with dozens of long layer paths.

<a id="62-volumes"></a>

6.2 Volumes

This section covers volumes, the storage mechanism Docker manages for you and the recommended way to persist container data.

Named Volumes

Why named volumes. You need data that outlives containers, but you do not want to care where it lives on the host, which permissions the directory has, or how the path differs between Linux, macOS, and Windows. A named volume is a storage area created and tracked by Docker under a name you choose. Containers refer to it by name, Docker handles the rest, and the same command works on every platform.

Analogy. A named volume is like a labeled storage unit in a facility. You rent it under a name ("pgdata"), and whichever moving truck (container) shows up can be given the key. The facility (Docker) decides which building and shelf it is on; you never need to know.

How it works

• A named volume is stored by the default local driver in /var/lib/docker/volumes/<name>/_data on the Docker host (inside the VM on Docker Desktop).
• It is mounted into the container at a path you choose, hiding whatever the image had at that path.
• If the volume is empty and the image has content at the mount path, Docker copies that content into the volume on first mount (pre-population), including ownership and permissions.
• It exists independently of containers: it is not deleted by docker rm, only by docker volume rm or a prune.
• If you reference a name that does not exist yet, docker run creates it automatically.

  docker run -v pgdata:/var/lib/postgresql/data postgres:16

  container A  ----mount---->  +--------------------------------------+
  (postgres:16)                | volume "pgdata"                      |
                               | /var/lib/docker/volumes/pgdata/_data |
  container B  ----mount---->  |  (survives A, reused by B)           |
  (postgres:16, later)         +--------------------------------------+

Example: data survives container replacement

docker volume create pgdata

docker run -d --name db -e POSTGRES_PASSWORD=secret \
  -v pgdata:/var/lib/postgresql/data postgres:16
docker exec db sh -c 'until pg_isready -h 127.0.0.1 -U postgres; do sleep 1; done'   # TCP is up only after init
docker exec db psql -U postgres -c "CREATE TABLE t(x int); INSERT INTO t VALUES (42);"

# Replace the container (as an upgrade would) and reuse the volume
docker rm -f db
docker run -d --name db -e POSTGRES_PASSWORD=secret \
  -v pgdata:/var/lib/postgresql/data postgres:16
docker exec db sh -c 'until pg_isready -h 127.0.0.1 -U postgres; do sleep 1; done'
docker exec db psql -U postgres -c "SELECT * FROM t;"     # 42 is still there

docker rm -f db && docker volume rm pgdata

Example: pre-population from the image

# nginx:alpine ships default HTML in /usr/share/nginx/html
docker run --rm -v site:/usr/share/nginx/html nginx:alpine ls /usr/share/nginx/html
# 50x.html  index.html            <- copied into the empty volume on first mount

# Opt out of the copy with volume-nocopy (or ":nocopy" with -v)
docker run --rm --mount type=volume,src=empty,dst=/usr/share/nginx/html,volume-nocopy \
  nginx:alpine ls /usr/share/nginx/html   # (empty)
docker volume rm site empty

Volume subpaths. Since Docker Engine 26, --mount type=volume,src=shared,dst=/data,volume-subpath=app1 mounts only a subdirectory of a volume, which lets several services use separate folders of one volume. The subdirectory must already exist.

Key takeaways

• Named volumes are Docker-managed storage referenced by name, portable across platforms.
• They live outside the container lifecycle and are reused when containers are replaced.
• An empty volume is pre-populated with the image's content at the mount path; nocopy disables this.
• docker run -v name:/path creates the volume if it does not exist, so a typo silently creates a new, empty volume.
• They are the default choice for any persistent application data.

🧪 Practice

1. (Beginner) Create a named volume, write a timestamp to it from one alpine container, and read it from a second container started later.
2. (Intermediate) Mount a fresh volume at /etc/nginx in an nginx:alpine container, change a config file inside the volume, then start a new container with the same volume. Is the image's default config or your edited version used? What happens if you mount the same volume into an image that has different content at that path?
3. Interview question: "After a deployment, the database came up empty although the volume still exists on the host. What would you check?" Hint: compare the volume name and the mount path in the old and new configuration, and remember what -v does with an unknown name.

Anonymous Volumes

Why anonymous volumes exist. Sometimes a path must bypass the container layer (for performance or because the image author marked it with VOLUME), but nobody has decided on a name for it. Docker then creates an anonymous volume: a regular volume with a random 64-character hexadecimal name, attached to that one container.

Where they come from

Source	Example
A VOLUME instruction in the image, with no mount supplied at runtime	docker run postgres:16 creates one for /var/lib/postgresql/data
-v with only a container path	docker run -v /data alpine
--mount without a source	docker run --mount type=volume,dst=/data alpine

Lifecycle rules

Action	Anonymous volume
docker rm <container>	Kept (orphaned, hard to identify later)
docker rm -v <container>	Removed
docker run --rm ...	Removed when the container is removed
docker volume prune	Removed if unused (since Docker 23, prune only removes anonymous volumes unless you add -a)
docker compose up (recreating a service)	Compose re-attaches the previous container's anonymous volumes; -V / --renew-anon-volumes creates fresh ones

Example: an orphan is born

docker run -d --name cache redis:7          # redis image declares VOLUME /data
docker inspect -f '{{range .Mounts}}{{.Type}} {{.Name}} -> {{.Destination}}{{end}}' cache
# volume 3f9c1e...(64 hex chars) -> /data

docker rm -f cache                           # container gone, volume left behind
docker volume ls -f dangling=true            # lists the orphaned 3f9c1e... volume
docker volume prune -f                       # removes unused anonymous volumes

A legitimate use: shadowing a directory inside a bind mount. In development you often bind-mount the project directory but want node_modules (built inside the container for Linux) to stay out of the host folder. An anonymous volume on the subdirectory takes precedence over the bind mount for that path:

# -v "$PWD":/app          project source from the host
# -v /app/node_modules    anonymous volume masks the host's node_modules
docker run --rm -it \
  -v "$PWD":/app \
  -v /app/node_modules \
  -w /app node:22 sh -c "npm ci && npm test"

Key takeaways

• Anonymous volumes are ordinary volumes with random names, created by VOLUME or by -v /path.
• They persist after docker rm unless you use -v or --rm, and pile up as unnamed orphans.
• docker volume prune removes unused anonymous volumes (named ones only with -a).
• Use named volumes for anything you care about; reserve anonymous volumes for scratch data and masking paths.

🧪 Practice

1. (Beginner) Run mysql:8 three times without a volume (set MYSQL_ROOT_PASSWORD), remove the containers without -v, and count the volumes left behind. Clean them up.
2. (Intermediate) Find which of your images declare volumes with docker image inspect -f '{{json .Config.Volumes}}' <image>. For one of them, decide whether you would replace the anonymous volume with a named volume and why.
3. Interview question: "A CI runner's disk fills up every few days even though all containers are removed after each job. What could be the cause, and how would you fix it permanently?" Hint: consider which images the jobs run, how the containers are removed, and which prune variant is scheduled.

Creating and Managing Volumes

Why manage volumes explicitly. Relying on docker run to auto-create volumes is convenient but hides typos, gives no place for metadata, and makes cleanup guesswork. The docker volume command lets you create volumes deliberately with labels and driver options, inspect them, and remove them safely.

Command overview

Command	Purpose
docker volume create NAME	Create a volume (optionally --driver, --opt, --label)
docker volume ls	List volumes; filter with -f dangling=true, -f label=..., -f driver=..., -f name=...
docker volume inspect NAME	Show driver, mount point, labels, options, scope, creation time
docker volume rm NAME...	Remove volumes (fails if a container, even a stopped one, uses it)
docker volume prune	Remove unused anonymous volumes; -a includes unused named volumes
docker system df -v	Show size and number of containers using each volume

Example: a managed lifecycle

# Create with labels for ownership and lifecycle
docker volume create \
  --label app=shop \
  --label env=dev \
  shop-db

docker volume inspect shop-db
# [
#   {
#     "CreatedAt": "2026-10-14T09:12:03Z",
#     "Driver": "local",
#     "Labels": { "app": "shop", "env": "dev" },
#     "Mountpoint": "/var/lib/docker/volumes/shop-db/_data",
#     "Name": "shop-db",
#     "Options": null,
#     "Scope": "local"            # "local" = this host only; "global" = cluster-wide driver
#   }
# ]

# Find volumes by label, or volumes no container references
docker volume ls -f label=app=shop
docker volume ls -f dangling=true -q

# Which containers use a volume?
docker ps -a --filter volume=shop-db --format '{{.Names}}'

# Size per volume
docker system df -v | sed -n '/VOLUME NAME/,/^$/p'

# Removal fails while in use (even by a stopped container)
docker run -d --name tmp -v shop-db:/data alpine sleep 60
docker volume rm shop-db
# Error response from daemon: remove shop-db: volume is in use - [<container-id>]
docker rm -f tmp && docker volume rm shop-db

# Prune only unused volumes with a specific label
docker volume prune -a -f --filter label=env=dev

Naming conventions. Use predictable names such as <project>-<service>-<purpose> (Compose does this automatically as <project>_<volume>). Combine with labels to answer "who owns this?" months later.

Safety rules. docker volume prune -a and docker compose down -v delete data irreversibly. Make sure you are on the right host (docker context show; see Docker Contexts), and filter by label where possible.

Key takeaways

• Create important volumes explicitly with meaningful names and labels.
• inspect shows the driver, mount point, options, and scope; system df -v shows sizes.
• A volume in use by any container, running or stopped, cannot be removed.
• prune removes unused anonymous volumes by default; -a includes named ones, so filter carefully.

🧪 Practice

1. (Beginner) Create two volumes with different env labels, list them with a label filter, and remove only one of them using a filtered prune.
2. (Intermediate) Write a shell one-liner that prints every volume together with the names of the containers using it (or "unused"). Use docker volume ls -q and docker ps -a --filter volume=....
3. Interview question: "How would you design a safe periodic cleanup of volumes on a shared build server?" Hint: consider the difference between dangling anonymous and named volumes, labels to mark disposable data, and a dry-run step.

Volume Drivers

Why volume drivers. The default local driver stores data on the Docker host's disk. That is fine for a single machine, but it ties data to that host: if a container is rescheduled elsewhere, or the disk fails, the data is gone or unreachable. Volume drivers let a volume live somewhere else, such as an NFS server, an SMB share, cloud block storage, or a distributed storage system, while containers keep using the same -v name:/path syntax.

Analogy. The volume name is a phone number; the driver is the telephone network that routes the call. The container dials the same number whether the person answers from a desk phone (local disk) or a mobile on another continent (network storage).

The local driver is more capable than it looks. It accepts the same options as the Linux mount command, so it can mount network filesystems without any plugin:

--opt	Meaning	Example
type	Filesystem type passed to mount	nfs, cifs, tmpfs, ext4, none (bind)
device	What to mount	:/exports/data, //fileserver/share, /dev/sdb1, tmpfs
o	Comma-separated mount options	addr=10.0.0.5,rw,nfsvers=4, size=100m

# NFS share as a volume (mounted lazily when a container first uses it)
docker volume create --driver local \
  --opt type=nfs \
  --opt o=addr=10.0.0.5,rw,nfsvers=4.1 \
  --opt device=:/exports/media \
  media

# SMB/CIFS share (credentials in options are visible in inspect output; prefer a credentials file)
docker volume create --driver local \
  --opt type=cifs \
  --opt device=//fileserver.local/reports \
  --opt o=addr=fileserver.local,username=svc,password=changeme,uid=1000,gid=1000,vers=3.0 \
  reports

# A dedicated block device formatted as ext4
docker volume create --driver local --opt type=ext4 --opt device=/dev/sdb1 fastdisk

docker run --rm -v media:/media alpine ls /media

Plugin drivers. Third-party drivers are installed as Docker plugins and implement a small HTTP API that the daemon calls:

  docker run -v data:/data --volume-driver=<plugin> ...
          |
          v
   dockerd ---- /VolumeDriver.Create   ----> plugin  ----> storage backend
           ---- /VolumeDriver.Mount    ---->  (returns a host path that dockerd
           ---- /VolumeDriver.Unmount  ---->   bind-mounts into the container)
           ---- /VolumeDriver.Remove, .Path, .Get, .List, .Capabilities

# Generic plugin workflow (sshfs is the classic documentation example)
docker plugin install vieux/sshfs                         # grants requested privileges
docker volume create -d vieux/sshfs \
  -o sshcmd=user@host:/remote/path -o password=... remote
docker run --rm -v remote:/data alpine ls /data

Many historic plugins are unmaintained. For clustered environments, Kubernetes CSI drivers (and CSI-based cluster volumes in Swarm) are the modern equivalent. Writing plugins is covered in Volume and Network Plugins.

Things to evaluate before using remote storage

Concern	Why it matters
Latency and fsync behavior	Databases on NFS can be slow or unsafe if locking and sync semantics are weak
Concurrent access	Most block-storage drivers allow only one host to mount a volume at a time
Availability	If the storage server is down, containers using it fail to start or hang
Credentials	Options passed with --opt are stored in the volume metadata and shown by inspect

Key takeaways

• A volume driver decides where a volume's data actually lives; containers use the same syntax regardless.
• The local driver can already mount NFS, CIFS, tmpfs, and block devices through --opt type/device/o.
• Plugin drivers implement a simple Create/Mount/Unmount/Remove API; many are unmaintained, and CSI is the modern path in clusters.
• Remote storage adds latency and failure modes; test databases on it carefully.

🧪 Practice

1. (Beginner) Create a volume with the local driver and --opt type=tmpfs --opt device=tmpfs --opt o=size=50m. Write to it from one container, then read it from a second container started after the first has exited. Explain the result in terms of when the local driver mounts and unmounts the filesystem.
2. (Intermediate) Run an NFS server in a container or VM, create an NFS-backed volume on another machine (or the same one), and mount it into two containers. Measure the write speed with dd compared to a local volume.
3. Interview question: "Your team wants to run a PostgreSQL container on an NFS-backed volume so it can move between hosts. What concerns would you raise?" Hint: consider file locking, fsync guarantees, latency, split-brain if two hosts mount it, and alternatives like replication.

Sharing Volumes Between Containers

Why share volumes. Some designs split work across containers that need the same files: a web server serving assets built by another container, a sidecar shipping log files written by the application, or a one-off job that seeds data for a long-running service. Mounting the same volume into several containers is the simplest way to do this on a single host.

Analogy. A shared volume is a shared folder on a team drive: everybody sees the same files immediately. It also has the same problem: if two people edit the same spreadsheet at once without coordination, someone's work gets overwritten.

Patterns

  Producer / consumer               Sidecar                         Init / seed
  -------------------               -------                         -----------
  [builder] --writes--> (assets)    [app] --writes--> (logs)        [seed job] --writes--> (data)
                           |                            |               (exits)              |
  [nginx]  <--reads(ro)----+        [shipper] <--reads--+           [service] <--reads/writes+

Two ways to share

Method	Syntax	Notes
Same volume name	-v assets:/srv in each container	Explicit, can mount at different paths and with different modes
--volumes-from	--volumes-from other[:ro]	Copies all mounts (volumes and bind mounts) of another container at the same paths; implicit and harder to read

Example: producer and read-only consumer

docker volume create assets

# Producer writes a new file every 2 seconds
docker run -d --name producer -v assets:/out alpine:3.20 \
  sh -c 'while true; do date > /out/now.txt; sleep 2; done'

# Consumer serves the same files read-only
docker run -d --name web -p 8080:80 -v assets:/usr/share/nginx/html:ro nginx:alpine
curl -s localhost:8080/now.txt        # changes every 2 seconds

# --volumes-from: inherit every mount of "producer" (here: /out), read-only
docker run --rm --volumes-from producer:ro alpine:3.20 cat /out/now.txt

docker rm -f producer web && docker volume rm assets

Concurrency is your problem. Docker does no locking or coordination. Sharing is safe when only one container writes, or when writers use separate files, or when the application itself is designed for shared storage (atomic renames, file locks). It is not safe to point two database server instances at the same data directory: at best one refuses to start because of a lock file, at worst the data is corrupted.

Scope. A local volume can only be shared by containers on the same host. Sharing across hosts requires a network-capable volume driver, and then the concurrency concerns become even more important.

Key takeaways

• The same volume can be mounted into many containers, at different paths and with different modes.
• --volumes-from copies all mounts of another container; explicit volume names are clearer.
• Docker does not coordinate concurrent writers; design for a single writer or application-level locking.
• Never let two database servers share one data directory.

🧪 Practice

1. (Beginner) Run the producer/consumer example and confirm that the consumer cannot write to the volume (docker exec web touch /usr/share/nginx/html/x).
2. (Intermediate) Build a log-shipping sidecar: an nginx container writing access logs to a volume (configure access_log /logs/access.log) and a second container running tail -F on the same file. Then switch to --volumes-from and compare readability.
3. Interview question: "Two replicas of a service write uploads to a shared volume. Users occasionally get truncated files. What is likely happening and how would you fix it?" Hint: think about partial writes visible to readers, write-to-temp-then-rename, and whether object storage is a better fit.

Read-Only Volumes

Why read-only. The principle of least privilege applies to storage too. A container that only needs to read configuration, certificates, or static assets should not be able to modify them, whether because of a bug, a compromised process, or a mistaken script. Mounting a volume read-only turns accidental or malicious writes into immediate errors and protects shared data from consumers.

Syntax

Flag	Read-only form
-v	-v name:/path:ro
--mount	--mount type=volume,src=name,dst=/path,readonly (or ro)
--volumes-from	--volumes-from other:ro
Compose	- name:/path:ro or read_only: true in the long syntax

docker volume create config
docker run --rm -v config:/cfg alpine:3.20 sh -c 'echo "level=info" > /cfg/app.conf'

# Mount read-only: reading works, writing fails
docker run --rm --mount type=volume,src=config,dst=/etc/app,readonly alpine:3.20 sh -c '
  cat /etc/app/app.conf
  echo "level=debug" > /etc/app/app.conf'
# level=info
# sh: can't create /etc/app/app.conf: Read-only file system

docker volume rm config

Read-only is per mount, not per volume. The same volume can be mounted read-write in one container (the writer) and read-only in others (the readers). The restriction is enforced by the kernel through the mount flags in that container's mount namespace.

Combining with a read-only root filesystem. The strongest setup makes the whole container read-only and opens only the exact paths that need writing:

# --read-only        container filesystem cannot be written
# appdata            the one persistent writable path
# certs (readonly)   inputs are read-only
# --tmpfs /tmp       scratch space in memory
docker run -d --name hardened \
  --read-only \
  --mount type=volume,src=appdata,dst=/var/lib/app \
  --mount type=volume,src=certs,dst=/certs,readonly \
  --tmpfs /tmp:size=64m \
  myapp:1.0

See Read-Only Root Filesystem for more on hardening.

Key takeaways

• Mount volumes read-only (:ro / readonly) whenever a container only consumes the data.
• The mode applies to one mount: one writer and many read-only readers can share a volume.
• Writes fail with "Read-only file system", which surfaces mistakes early.
• Combine with --read-only and tmpfs mounts for a minimal writable surface.

🧪 Practice

1. (Beginner) Mount a volume read-only into an alpine container and try touch, rm, and mkdir on it. Note the error each time.
2. (Intermediate) Run nginx:alpine with --read-only, observe why it fails to start, and add the minimal set of tmpfs mounts needed (hint: look at the error messages for cache and PID file paths).
3. Interview question: "Is mounting a volume read-only a sufficient security boundary to protect its data from a compromised container?" Hint: consider what a process with CAP_SYS_ADMIN or a privileged container can do with mounts, and what other containers can still write.

<a id="63-mounts"></a>

6.3 Mounts

This section covers the other two mount types, bind mounts and tmpfs mounts, the two syntaxes used to declare any mount, and the permission issues that come with sharing files between host and container.

Bind Mounts

Why bind mounts. Volumes hide their location on purpose, but sometimes the location is exactly the point: you want the container to see your source code as you edit it, read a configuration file that lives at a known host path, or write a report into a folder you can open on the host. A bind mount maps an existing host file or directory directly into the container.

Analogy. A volume is a storage unit managed by the facility; a bind mount is handing the container a key to a specific room in your own house. It is convenient and immediate, but the container now operates directly on your belongings.

How it works. Docker asks the kernel to bind-mount the host path onto the container path (the same mechanism as mount --bind). There is no copy: both sides see the same inode, so changes are visible instantly in both directions.

   Host                                    Container
   /home/ana/project/          bind        /app/
   +-- src/main.py       <=============>   +-- src/main.py     same files,
   +-- config.yaml                         +-- config.yaml     no copy

Example

mkdir -p ~/site && echo '<h1>Hello from the host</h1>' > ~/site/index.html

# Directory bind mount, read-only for the web server
docker run -d --name site -p 8080:80 \
  --mount type=bind,src="$HOME/site",dst=/usr/share/nginx/html,readonly \
  nginx:alpine
curl -s localhost:8080                  # <h1>Hello from the host</h1>

echo '<h1>Edited</h1>' > ~/site/index.html
curl -s localhost:8080                  # <h1>Edited</h1> -> live, no rebuild

# Single-file bind mount (e.g. a config file)
docker run --rm -v "$PWD/nginx.conf":/etc/nginx/nginx.conf:ro nginx:alpine nginx -t

docker rm -f site

Bind mounts vs named volumes

Aspect	Bind mount	Named volume
Location	Any host path you specify	Managed by Docker
Must exist beforehand	Yes with --mount; -v creates a missing directory (owned by root)	No, created on demand
Pre-population from image	No: the host content hides the image content	Yes, if the volume is empty
Portability	Depends on host paths and OS	Same command on every host
Performance on Docker Desktop	Slower (crosses the VM boundary via file sharing)	Native speed inside the VM
Managed by docker volume	No	Yes
Typical use	Dev source code, host config, sockets, output folders	Application and database data

Advanced options

Option	Purpose
readonly / :ro	Read-only mount; on Docker 25+ with kernel 5.12+, submounts are made read-only recursively too (tune with bind-recursive)
bind-propagation	Whether mounts created later under the source appear in the container (rprivate default; rshared, rslave, ...)
:z / :Z (SELinux)	Relabel the host path so the container may access it: z shared between containers, Z private to one. Never use on system directories like /home or /usr.
consistent, cached, delegated	Historical Docker Desktop for Mac performance hints; accepted but ignored today

Pitfalls and security

• Paths are resolved on the daemon's host. With a remote daemon or a Docker context pointing elsewhere, -v ./config:/etc/app refers to a path on the remote machine (see Client-Server Model).
• Mounting over image content hides it. Bind-mounting an empty host directory onto /app makes the application files in the image invisible.
• Sensitive host paths. Mounting /, /etc, or /var/run/docker.sock gives the container control over the host (see Docker Socket Exposure Risks). Mount the narrowest path possible, read-only where you can.

Key takeaways

• A bind mount exposes an existing host path directly inside the container, with no copying.
• Use it when the host location matters: development, host configuration, output folders.
• Unlike volumes, bind mounts are never pre-populated and hide the image content at that path.
• Prefer --mount, which fails if the source is missing instead of silently creating a root-owned directory.
• Mount the narrowest path, read-only when possible; never expose sensitive host directories casually.

🧪 Practice

1. (Beginner) Bind-mount a host directory into an alpine container, create a file from inside the container, and inspect its owner on the host with ls -l.
2. (Intermediate) Run docker run -v /does/not/exist:/data alpine and docker run --mount type=bind,src=/does/not/exist,dst=/data alpine. Compare the outcomes and inspect what was created on the host.
3. Interview question: "Why is bind-mounting /var/run/docker.sock into a container often described as giving it root on the host?" Hint: consider what the Docker API allows a client to start, and which mounts and privileges such a new container could request.

tmpfs Mounts

Why tmpfs. Some data should never touch a disk: session tokens, decrypted secrets, temporary files with personal data. Other data is purely scratch (build intermediates, caches) and writing it to disk only costs I/O. A tmpfs mount gives the container a filesystem backed by the host's memory. It is fast, private to one container, and disappears completely when the container stops.

Analogy. A tmpfs mount is a whiteboard in a meeting room: quick to write on, nothing is archived, and it is wiped clean when the meeting ends.

Properties

Property	tmpfs mount
Backing storage	Host RAM (can be swapped out if the host has swap)
Lifetime	Until the container stops (not just until it is removed)
Sharing	Never shared between containers
Platforms	Linux containers only
Memory accounting	Counts toward the container's memory limit (cgroup)
Default size	Unlimited unless you set a size; always set one

Syntax

# --tmpfs: short form, options after the colon (mount(8) style)
docker run --rm --tmpfs /scratch:rw,size=64m,mode=1777,noexec,nosuid alpine:3.20 df -h /scratch

# --mount: explicit form with tmpfs-specific keys
docker run --rm \
  --mount type=tmpfs,dst=/scratch,tmpfs-size=64m,tmpfs-mode=1770 \
  alpine:3.20 sh -c 'dd if=/dev/zero of=/scratch/f bs=1M count=100'
# dd: error writing '/scratch/f': No space left on device    <- size limit enforced

Example: proving it is gone on stop

docker run -d --name t --tmpfs /secrets:size=1m alpine:3.20 sleep 3600
docker exec t sh -c 'echo token123 > /secrets/token; cat /secrets/token'   # token123
docker restart -t 1 t
docker exec t ls /secrets                                                    # (empty)
docker rm -f t

Typical uses

• Writable paths for a --read-only container: /tmp, /run, /var/cache/nginx.
• Temporary storage for decrypted secrets or tokens fetched at startup.
• Fast scratch space for tests (for example, a throwaway database data directory in CI, where durability does not matter).

Limits. tmpfs is not a secure vault: a process with access to the container or a host administrator can still read it, and memory can be swapped to disk. Large tmpfs mounts compete with the application for the container's memory limit and can trigger the OOM killer (see Memory Limits).

Key takeaways

• tmpfs mounts are memory-backed, per-container, and erased when the container stops.
• Always set a size; usage counts against the container's memory limit.
• Use them for secrets, scratch data, and writable paths in read-only containers.
• They are Linux-only and never shared between containers.

🧪 Practice

1. (Beginner) Start a container with --tmpfs /data:size=10m, write files until it is full, and check df -h /data inside the container.
2. (Intermediate) Run a container with --memory=100m --tmpfs /data:size=200m and write 150 MB into /data. Explain what happens and why.
3. Interview question: "Why might you run a test database in CI with its data directory on tmpfs, and why would you never do that in production?" Hint: compare I/O speed, durability requirements, and what happens on a container restart.

--mount vs -v Syntax

Why two syntaxes. -v (--volume) is the original, compact syntax from Docker's early days. As mount types and options multiplied, packing everything into colon-separated fields became ambiguous and error-prone. --mount was introduced with explicit key=value pairs that support every mount type and option. Both still work, and they produce the same result for the cases they share.

Shape of each syntax

-v  [SOURCE:]DESTINATION[:OPTIONS]
     |         |             +-- comma list: ro, rw, z, Z, nocopy, rshared, ...
     |         +---------------- path inside the container
     +-------------------------- volume name, host path (starts with / or .), or omitted (anonymous)

--mount type=volume|bind|tmpfs,src=...,dst=...,readonly,<type-specific options>
         key order does not matter; "source"/"src" and "destination"/"dst"/"target" are aliases

Equivalent commands

Goal	-v / --tmpfs	--mount
Named volume	-v data:/var/lib/data	--mount type=volume,src=data,dst=/var/lib/data
Anonymous volume	-v /var/lib/data	--mount type=volume,dst=/var/lib/data
Read-only bind mount	-v "$PWD"/cfg:/etc/app:ro	--mount type=bind,src="$PWD"/cfg,dst=/etc/app,readonly
Volume without pre-population	-v data:/srv:nocopy	--mount type=volume,src=data,dst=/srv,volume-nocopy
tmpfs with size	--tmpfs /tmp:size=64m	--mount type=tmpfs,dst=/tmp,tmpfs-size=64m

Behavioral differences

Topic	-v	--mount
Missing bind source	Silently creates the directory on the host (owned by root)	Fails with an error
How the type is chosen	Inferred: absolute or ./ path means bind, a name means volume	Explicit type=
tmpfs	Not supported (use --tmpfs)	type=tmpfs with tmpfs-size, tmpfs-mode
Per-mount volume driver options	Not supported	volume-driver=...,volume-opt=key=value
Newer options (volume-subpath, bind-recursive)	Not available	Available
Readability	Compact, positional	Verbose, self-documenting

Example: a typo that only --mount catches

# Intended a bind mount of ./confg (typo for ./config)
docker run --rm -v "$PWD/confg":/etc/app alpine ls /etc/app
# (empty) -> and a new empty, root-owned ./confg directory now exists on the host

docker run --rm --mount type=bind,src="$PWD/confg",dst=/etc/app alpine ls /etc/app
# docker: Error response from daemon: invalid mount config for type "bind":
#         bind source path does not exist: /home/ana/confg

Compose. The same split exists in Compose: the short syntax (- data:/var/lib/data:ro) mirrors -v, and the long syntax (type:, source:, target:, read_only:) mirrors --mount. See Volumes in Compose.

Recommendation. Use --mount in scripts, documentation, and anything shared with others; -v is fine for quick interactive commands.

Key takeaways

• -v is compact and positional; --mount is explicit key=value and supports every option.
• The biggest practical difference: -v silently creates missing bind sources, --mount fails.
• tmpfs mounts use --tmpfs or --mount type=tmpfs, never -v.
• Prefer --mount for anything you keep; Compose offers the same short/long split.

🧪 Practice

1. (Beginner) Translate these into --mount syntax: -v logs:/var/log/app, -v /etc/ssl/certs:/certs:ro, --tmpfs /run:size=16m.
2. (Intermediate) Start a container using both syntaxes for different mounts, then run docker inspect -f '{{json .Mounts}}' and docker inspect -f '{{json .HostConfig.Binds}}{{json .HostConfig.Mounts}}'. Explain where each syntax is recorded.
3. Interview question: "A colleague's script uses -v $CONFIG_DIR:/etc/app, and in one environment the application starts with an empty configuration instead of failing. Why, and how would you make the failure explicit?" Hint: consider what happens when the variable is empty or the path is missing, and how --mount behaves differently.

File Permissions and Ownership

Why permissions get confusing. Linux file ownership is stored as numbers (UID and GID), not names. A container shares the host kernel, so a file written by UID 0 inside the container is owned by UID 0 (root) on the host, and a file owned by UID 1000 on the host is owned by whichever user has UID 1000 inside the container, if any. User names can differ completely between host and container (/etc/passwd is per image), which makes the same file look like it belongs to different people.

Analogy. UIDs are like employee badge numbers shared between two offices with different staff lists. Badge 1000 is "ana" in the head office and "node" in the branch office. The door locks (the kernel) only check the number, never the name.

  Host (/etc/passwd)          kernel sees only numbers         Container (/etc/passwd)
  ana     uid=1000   <------------  uid 1000  ------------>    node    uid=1000
  root    uid=0      <------------  uid 0     ------------>    root    uid=0
  (none)             <------------  uid 999   ------------>    postgres uid=999

Common problems and causes

Symptom	Cause
Files created through a bind mount are owned by root on the host	The container process runs as root (UID 0)
Non-root container gets "Permission denied" writing to a bind mount	The host directory is owned by a different UID
Non-root container cannot write to a new named volume	The mount path did not exist in the image, so the volume root is owned by root
Works on macOS, fails on Linux	Docker Desktop's file sharing translates ownership; native Linux does not
"Permission denied" despite correct UIDs on Fedora/RHEL	SELinux labels; add :z or :Z to the bind mount

Example: the classic root-owned file

mkdir -p out
docker run --rm -v "$PWD/out":/out alpine:3.20 touch /out/report.txt
ls -ln out/report.txt
# -rw-r--r-- 1 0 0 0 ... report.txt      <- UID 0: you cannot edit or delete it without sudo

# Run as your own UID/GID so files belong to you
docker run --rm -u "$(id -u):$(id -g)" -v "$PWD/out":/out alpine:3.20 touch /out/report2.txt
ls -ln out/report2.txt
# -rw-r--r-- 1 1000 1000 0 ... report2.txt

Solutions

Technique	How	When to use
Run as the host user	-u "$(id -u):$(id -g)"	Tools and builds writing to bind mounts
Prepare ownership in the image	RUN mkdir /data && chown 10001:10001 /data before VOLUME or before mounting	Named volumes for non-root images (pre-population copies ownership)
Match UIDs at build time	ARG UID=1000 + useradd -u $UID app	Development images shared by a team
Fix at startup, then drop privileges	Entrypoint runs as root, chowns the data dir, then exec gosu app "$@" (or su-exec)	Official database images use this pattern
Remap UIDs	User namespace remapping or rootless Docker	Container root should not be host root (see User Namespace Remapping, Rootless Docker)
SELinux relabel	-v "$PWD":/app:z	Fedora, RHEL, CentOS hosts

Example: preparing a named volume for a non-root image

# syntax=docker/dockerfile:1
FROM alpine:3.20
RUN addgroup -g 10001 app && adduser -D -u 10001 -G app app \
 && mkdir -p /data && chown app:app /data      # ownership is copied into empty volumes
USER app
CMD ["sh", "-c", "date >> /data/log && cat /data/log"]

docker build -t perms-demo .
docker run --rm -v perms:/data perms-demo     # works: the volume inherits 10001:10001
docker run --rm -v "$PWD/empty":/data perms-demo
# sh: can't create /data/log: Permission denied   <- bind mounts are not pre-populated

Key takeaways

• The kernel compares numeric UIDs and GIDs; names in /etc/passwd differ between host and container.
• Root in the container writes root-owned files on the host through bind mounts.
• Empty named volumes inherit ownership from the image path; bind mounts keep the host's ownership.
• Fix with -u, image-side chown, entrypoint chown + privilege drop, or UID remapping.
• On SELinux hosts, permission errors may need :z / :Z rather than chown.

🧪 Practice

1. (Beginner) Create a file through a bind mount as root and as -u $(id -u):$(id -g), then compare ls -ln output on the host and ls -l inside an alpine container.
2. (Intermediate) Run postgres:16 with a bind-mounted empty host directory as its data directory and inspect the ownership of the files it creates. Find in the image's entrypoint script (docker-entrypoint.sh) where ownership is changed and privileges are dropped.
3. Interview question: "A developer on macOS says the container works, but on the Linux CI runner it fails with 'Permission denied' writing to a bind-mounted directory. How do you explain and fix this?" Hint: consider how Docker Desktop shares files with its VM versus native Linux, and which UID the container runs as on CI.

Bind Mounts for Development

Why bind mounts shine in development. The edit-build-run loop is the heartbeat of development. Rebuilding an image for every code change takes seconds to minutes and breaks concentration. Bind-mounting the source directory into a container that runs the app with a file watcher (nodemon, uvicorn --reload, Spring DevTools, air for Go) gives you the consistency of a container with the instant feedback of local development.

  editor on host  --save-->  ./src/app.py  ==bind==>  /app/src/app.py  --inotify-->  dev server
                                                                                      reloads

A typical setup (Compose)

services:
  api:
    build:
      context: .
      target: dev                     # a dev stage with the watcher and dev dependencies
    command: uvicorn app.main:app --host 0.0.0.0 --reload
    ports:
      - "8000:8000"
    volumes:
      - type: bind                    # source code from the host, live
        source: ./src
        target: /app/src
      - type: bind
        source: ./pyproject.toml
        target: /app/pyproject.toml
        read_only: true
      - venv:/app/.venv               # dependencies stay in a named volume, built for Linux

volumes:
  venv:

The same idea for Node.js keeps node_modules in a volume, because packages with native code built on macOS or Windows do not run in a Linux container, and syncing thousands of files over a bind mount is slow:

docker run --rm -it -p 3000:3000 \
  --mount type=bind,src="$PWD",dst=/app \
  --mount type=volume,src=myapp-node-modules,dst=/app/node_modules \
  -w /app node:22 sh -c "npm ci && npm run dev"

Pitfalls and remedies

Problem	Remedy
Slow file access on Docker Desktop (macOS, Windows)	Keep dependency and build directories in named volumes; on WSL2 keep the project in the Linux filesystem, not /mnt/c
File watcher does not notice changes	Make sure the watcher listens inside the container; some tools need polling mode (CHOKIDAR_USEPOLLING=1, WATCHFILES_FORCE_POLLING=true) on certain shares
Root-owned files appear in the repository	Run the dev container as your UID (see File Permissions and Ownership)
Host OS artifacts leak in (.DS_Store, Windows line endings in shell scripts)	.gitattributes with eol=lf for scripts; ignore files in watchers
Case-insensitive host filesystem (macOS, Windows)	Test case-sensitive imports in CI on Linux
Dev image differs from the production image	Use a multi-stage Dockerfile with dev and prod targets built from the same base

Alternatives to bind mounts. docker compose watch syncs files into the container (and can rebuild on dependency changes) instead of mounting them, which avoids several of the issues above, especially on Docker Desktop (see Compose Watch for Development). Docker Desktop also offers synchronized file shares for large repositories. Dev Containers build a whole IDE workflow on the same principle (see Dev Containers).

Never in production. Bind-mounting source code into production containers defeats the purpose of images: the running code is no longer the reviewed, versioned artifact, and it depends on files on a specific host. Production images should contain the code (COPY).

Key takeaways

• Bind-mount source code plus a reloading dev server for an instant edit-run loop.
• Keep dependencies (node_modules, virtualenvs) in named volumes, built inside the container.
• On Docker Desktop, bind mounts are slower; minimize the files they cover or use compose watch.
• Run dev containers as your UID to avoid root-owned files in the repository.
• Bind-mounted code is for development only; production images must contain their code.

🧪 Practice

1. (Beginner) Run a small Flask or Express app with its source bind-mounted and a reloading server. Edit a route and confirm the change appears without restarting the container.
2. (Intermediate) Time npm install (or pip install) with node_modules (or the virtualenv) on the bind mount versus in a named volume. On Docker Desktop, repeat with docker compose watch and compare.
3. Interview question: "Why do teams keep node_modules in a separate volume when bind-mounting a Node.js project, and what problem does this introduce when package.json changes?" Hint: think about native modules built for the host OS, performance, and when the volume gets refreshed.

<a id="64-data-management"></a>

6.4 Data Management

This section covers the operational tasks around persistent data: keeping copies of it, getting it back, moving it, and running stateful services responsibly.

Backing Up Volumes

Why back up volumes. A volume protects data from container removal, not from everything else: a disk failure, a mistaken docker volume prune -a, a bad migration script, ransomware, or a host that disappears. Docker has no built-in docker volume backup command on the Engine, so backups are something you must design.

The core technique. A volume can be mounted into any container. To back it up, mount it (read-only) into a short-lived helper container together with a host directory, and archive one into the other:

  +-------------------+           +----------------------------+
  | volume "pgdata"   | --ro-->   | helper container (debian)  |
  +-------------------+           |   tar czf /backup/x.tgz    |
                                  |        -C /data .          |
  host ./backups  <--bind---------+----------------------------+

# 1. For consistency, stop (or pause) the containers writing to the volume
docker stop db

# 2. Archive the volume content into the current host directory
docker run --rm \
  --mount type=volume,src=pgdata,dst=/data,readonly \
  --mount type=bind,src="$PWD",dst=/backup \
  debian:bookworm-slim \
  tar -czf /backup/pgdata-$(date +%F).tar.gz -C /data .   # -C: archive relative to /data

# 3. Restart the service
docker start db

# The classic variant from the Docker docs uses --volumes-from to reach the volumes of "db"
docker run --rm --volumes-from db -v "$PWD":/backup debian:bookworm-slim \
  tar -czf /backup/db-volumes.tar.gz /var/lib/postgresql/data

File copy vs. application-aware backup. Copying the files of a running database can capture a half-written state that will not start or is silently inconsistent. Choose the method by data type:

Method	Consistency	Downtime	Best for
tar of a stopped container's volume	Consistent	Yes, for the duration of the copy	Any data, small to medium size
tar of a live volume	Not guaranteed	None	Static files, append-only logs
Logical dump (pg_dump, mysqldump, mongodump)	Consistent (transactional snapshot)	None	Databases; portable across versions and architectures
Physical database backup (pg_basebackup, Percona XtraBackup)	Consistent	None	Large databases, point-in-time recovery
Filesystem or block snapshot (LVM, ZFS, cloud disk snapshot)	Crash-consistent	None or seconds	Large volumes; combine with the app's freeze/flush mechanism

Example: logical database backup without stopping anything

# -Fc = custom compressed format for pg_restore; no -t, because a TTY would corrupt binary output
docker exec db pg_dump -U postgres -Fc appdb > appdb-$(date +%F).dump

# MySQL / MariaDB equivalent (password provided via the container's environment)
docker exec mysql sh -c 'exec mysqldump --single-transaction -uroot -p"$MYSQL_ROOT_PASSWORD" shop' > shop.sql

Making backups real

• Automate with cron, a systemd timer, or a scheduled container; label volumes that need backup (--label backup=daily) and loop over docker volume ls -f label=backup=daily -q.
• Store copies elsewhere (the 3-2-1 rule: 3 copies, 2 media, 1 off-site), for example by uploading to object storage.
• Protect archives: they contain everything, including credentials stored in the data.
• Test restores regularly; an untested backup is a hypothesis.

Docker Desktop also offers volume export (to a file, an image, or a registry) from its Volumes view, which is convenient for local development data.

Key takeaways

• Volumes survive container removal, not disk failure or human error: back them up.
• The standard technique is a helper container that mounts the volume read-only and runs tar.
• Stop writers or use application-aware tools (pg_dump, mysqldump) for consistent database backups.
• Automate, keep copies off the host, and regularly test restoring.

🧪 Practice

1. (Beginner) Create a volume with a few files, back it up to a .tar.gz on the host with a helper container, and list the archive contents with tar -tzf.
2. (Intermediate) Write a script that backs up every volume labeled backup=daily into a dated directory and deletes archives older than 7 days. Run it against two test volumes.
3. Interview question: "Why is tar-ing the data directory of a running PostgreSQL container a risky backup strategy, and what would you do instead?" Hint: consider in-flight writes, the write-ahead log, and tools that produce a transactionally consistent copy.

Restoring Volumes

Why restore deserves its own procedure. Restoring happens under pressure: something broke, and people are waiting. A rehearsed, scripted procedure avoids making things worse, such as extracting a backup on top of corrupted data, restoring with wrong ownership so the service cannot start, or overwriting the only good copy.

Restore into a new volume. Never extract over the volume you are trying to recover. Restore into a new volume, verify it, then point the service at it (or swap names by copying).

  pgdata-2026-10-13.tar.gz --> [helper: tar -xzf] --> volume "pgdata-restored" --> verify --> start db on it
                                                       (fresh, empty)
  volume "pgdata" (broken) --> keep until the restore is confirmed, then remove

Example: restoring a tar backup

docker volume create pgdata-restored

docker run --rm \
  --mount type=volume,src=pgdata-restored,dst=/data \
  --mount type=bind,src="$PWD",dst=/backup,readonly \
  debian:bookworm-slim \
  tar -xzf /backup/pgdata-2026-10-13.tar.gz -C /data --numeric-owner   # keep original UIDs/GIDs

# Check ownership: the postgres image expects UID 999 for its data directory
docker run --rm -v pgdata-restored:/data debian:bookworm-slim ls -ln /data | head -5

# Start the service on the restored volume and verify the data
docker run -d --name db-restored -e POSTGRES_PASSWORD=secret \
  -v pgdata-restored:/var/lib/postgresql/data postgres:16
docker exec db-restored psql -U postgres -c '\l'

--numeric-owner matters because the helper image's /etc/passwd may not contain the same users as the original image; without it, tar maps owners by name and can assign the wrong UID.

Example: restoring a logical dump

# Start a fresh database (new, empty volume), then load the dump through stdin (-i, no -t)
docker run -d --name db-new -e POSTGRES_PASSWORD=secret -v pgdata-new:/var/lib/postgresql/data postgres:16
docker exec db-new sh -c 'until pg_isready -h 127.0.0.1 -U postgres; do sleep 1; done'
docker exec db-new createdb -U postgres appdb
docker exec -i db-new pg_restore -U postgres -d appdb --no-owner < appdb-2026-10-13.dump

# Plain SQL dumps (pg_dump without -Fc, mysqldump) are loaded with the SQL client
docker exec -i mysql sh -c 'exec mysql -uroot -p"$MYSQL_ROOT_PASSWORD" shop' < shop.sql

Restore checklist

Step	Why
Identify the right backup (date, host, volume)	Restoring the wrong snapshot is a second incident
Restore into a new volume	Keeps the damaged data for analysis and as a fallback
Match image version	Physical (file-level) backups need the same database major version
Check ownership and permissions	Services commonly refuse to start on wrong ownership
Verify application-level data	Row counts, recent records, application health check
Document time-to-restore	Tells you whether your recovery time objective is realistic

Key takeaways

• Restore into a fresh volume and keep the damaged one until the result is verified.
• Use a helper container with tar -x --numeric-owner for file-level backups.
• Load logical dumps through docker exec -i with the database's restore tool; never allocate a TTY for binary streams.
• File-level backups require the same database major version; logical dumps are portable.
• Practice restores and measure how long they take.

🧪 Practice

1. (Beginner) Restore the archive from the previous topic's exercise into a new volume and compare file lists and checksums (sha256sum) with the original.
2. (Intermediate) Back up a postgres:16 database with pg_dump -Fc, drop a table, and restore only that table into a fresh container with pg_restore -t. Measure the total time.
3. Interview question: "During a restore, the database container keeps exiting with a permissions error, although the files are all there. What happened and how do you fix it?" Hint: think about how the archive was extracted, numeric versus name-based ownership, and what UID the database process runs as.

Migrating Data

Why migrations happen. Data rarely stays where it was created. You move services to a new host, switch from a bind mount to a named volume, rename a volume (Docker has no rename command), move from local disk to network storage, change Docker Desktop for a Linux VM, or upgrade a database to a new major version. Each needs a controlled copy of the data with minimal downtime and no surprises.

Common scenarios and techniques

Scenario	Technique
Rename or copy a volume on the same host	Helper container mounting both volumes, cp -a
Move a volume to another host	Stream tar over SSH, or through Docker contexts
Bind mount to named volume (or back)	Helper container mounting both, cp -a
Anonymous volume to named volume	Find the anonymous name in docker inspect, then copy as above
Database major version upgrade	Logical dump and restore, or pg_upgrade in a dedicated container
Different CPU architecture (amd64 to arm64)	Logical dump and restore (database file formats are not guaranteed to be portable)

Example: copying or renaming a volume locally

docker volume create newname
docker run --rm \
  --mount type=volume,src=oldname,dst=/from,readonly \
  --mount type=volume,src=newname,dst=/to \
  debian:bookworm-slim cp -a /from/. /to/     # -a preserves ownership, modes, timestamps, links
# verify, update the service to use "newname", then:
docker volume rm oldname

Example: moving a volume to another host in one stream

# Over SSH: archive on this host, extract on the remote host, no temporary files
docker run --rm -v appdata:/from:ro debian:bookworm-slim tar -C /from -cf - . \
  | ssh deploy@new-host \
    'docker run --rm -i -v appdata:/to debian:bookworm-slim tar -C /to -xf - --numeric-owner'

# The same with Docker contexts (each docker command talks to a different daemon)
docker --context old-host run --rm -v appdata:/from:ro debian:bookworm-slim tar -C /from -cf - . \
  | docker --context new-host run --rm -i -v appdata:/to debian:bookworm-slim tar -C /to -xf - --numeric-owner

-i keeps stdin open for the receiving container, and neither side uses -t, so the binary stream passes through unchanged.

Example: upgrading PostgreSQL across major versions

# Data files of PostgreSQL 16 cannot be opened by PostgreSQL 17: dump with the old version...
docker exec db16 pg_dumpall -U postgres > all.sql

# ...and load into a new container with a new volume
docker run -d --name db17 -e POSTGRES_PASSWORD=secret -v pgdata17:/var/lib/postgresql/data postgres:17
docker exec db17 sh -c 'until pg_isready -h 127.0.0.1 -U postgres; do sleep 1; done'
docker exec -i db17 psql -U postgres < all.sql

Planning a migration

1. Inventory: which volumes and bind mounts a service uses (docker inspect -f '{{json .Mounts}}').
2. Choose consistency: stop writers for a file copy, or use a database-native method.
3. Copy, preserving ownership (cp -a, tar --numeric-owner).
4. Verify: checksums or file counts for files; row counts and smoke tests for databases.
5. Switch the service configuration, keeping the old data read-only until the new setup has run successfully for a while.

Key takeaways

• Docker has no volume rename or move command; migrations are copies through helper containers.
• Stream tar between hosts over SSH or Docker contexts without intermediate files.
• Preserve ownership (cp -a, --numeric-owner) and verify before switching.
• Database major version and architecture changes need logical dumps or the database's upgrade tooling, not file copies.
• Keep the old data until the new setup is confirmed.

🧪 Practice

1. (Beginner) "Rename" a volume by copying it into a new one with a helper container, and verify that file ownership and modes are identical.
2. (Intermediate) Convert a development setup that stores data in a bind mount (./data) to a named volume without losing data. Write down the exact steps, including how you verified the result.
3. Interview question: "You need to move a 200 GB volume used by a busy service to a new host with less than five minutes of downtime. How would you approach it?" Hint: consider an initial bulk copy while the service runs, a final incremental sync (for example rsync) during a short stop, or database replication instead of file copies.

Running Databases in Containers

Why it is debated. Containers were first popular for stateless services, and early storage options were weak, so "never run databases in containers" became common advice. Today the container is not the problem: a database process in a container is the same process with the same kernel I/O path when its data is on a volume. What matters is the operational discipline around state, such as storage, backups, upgrades, and shutdown, which you need with or without containers.

When it fits

Situation	Container database	Managed database service (RDS, Cloud SQL, ...)
Local development, tests, CI	Ideal: disposable, version-pinned, identical for everyone	Overkill
Small production workloads with a capable ops team	Reasonable	Often simpler
Critical production data, high availability, compliance	Possible with an operator on Kubernetes and serious expertise	Usually the safer default: backups, failover, patching included

The essentials

   +-------------------------------------------------------------+
   | database container                                          |
   |   - pinned image version (postgres:16.4, not latest)        |
   |   - data dir on a NAMED VOLUME   ----> survives upgrades    |
   |   - secrets via *_FILE / secrets ----> not in env or image  |
   |   - healthcheck (pg_isready)     ----> dependents wait      |
   |   - enough stop timeout          ----> clean shutdown       |
   |   - memory/shm limits            ----> predictable          |
   |   - port NOT published publicly  ----> private network      |
   +-------------------------------------------------------------+
               |
               +--> scheduled logical backups to off-host storage

Example: a well-configured PostgreSQL service in Compose

services:
  db:
    image: postgres:16.4                 # pin major and minor; upgrade deliberately
    environment:
      POSTGRES_DB: shop
      POSTGRES_USER: shop
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password   # official images support *_FILE variables
    secrets:
      - db_password
    volumes:
      - pgdata:/var/lib/postgresql/data          # the image's data directory (see note for 18+)
      - ./initdb:/docker-entrypoint-initdb.d:ro  # runs ONLY when the data directory is empty
    shm_size: 256m                       # default /dev/shm (64 MB) is too small for parallel queries
    stop_grace_period: 60s               # let PostgreSQL shut down cleanly (default is 10 s)
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U shop -d shop"]
      interval: 5s
      timeout: 3s
      retries: 10
    deploy:
      resources:
        limits:
          memory: 2g
    # no "ports:" -> reachable only from services on the same network

  api:
    image: acme/api:1.8
    depends_on:
      db:
        condition: service_healthy       # start only when the database accepts connections

secrets:
  db_password:
    file: ./secrets/db_password.txt

volumes:
  pgdata:

Data directory paths of common images

Image	Data directory to mount	Notes
postgres (up to 17)	/var/lib/postgresql/data	Declared as VOLUME
postgres (18 and later)	/var/lib/postgresql	Data now in a version-specific subdirectory (/var/lib/postgresql/18/docker) to ease pg_upgrade
mysql, mariadb	/var/lib/mysql
mongo	/data/db	Config database in /data/configdb
redis	/data	Persistence only if RDB snapshots or AOF are enabled

Always check the image documentation: mounting the volume at the wrong path is the most common cause of "the database lost everything after an upgrade".

Pitfalls

• Initialization runs once. Environment variables like POSTGRES_PASSWORD and the scripts in /docker-entrypoint-initdb.d only apply when the data directory is empty. Changing them later does nothing to an existing volume.
• Unclean shutdowns. A short stop timeout leads to SIGKILL and crash recovery on the next start (see Graceful Shutdown and Signal Handling).
• Automatic upgrades. Tags like postgres:latest or watchtower-style auto-updates can jump a major version, and the new server refuses the old data files.
• Shared data directories. Never mount the same data volume into two database containers at once.
• Exposed ports. Publishing 5432:5432 on a server exposes the database to the network, and Docker's port publishing bypasses many host firewall rules. Keep databases on internal networks, or bind to 127.0.0.1:5432:5432.
• Resource limits. A database tunes its caches to the memory it believes it has; set limits and the database's own memory parameters consistently.

Key takeaways

• Containerized databases are excellent for development and tests; production requires the same discipline as any database.
• Always mount a named volume at the image's exact data directory, and pin the image version.
• Initialization variables and init scripts only take effect on an empty data directory.
• Give the database time to shut down, a healthcheck, adequate shared memory, and a private network.
• Back up with database-native tools and practice restores; consider managed services for critical production data.

🧪 Practice

1. (Beginner) Start postgres:16 with a named volume and a healthcheck using docker run flags (--health-cmd, --health-interval). Watch the health status change with docker ps.
2. (Intermediate) Change POSTGRES_PASSWORD for an existing container with data and recreate it. Explain why the old password still works and how to actually change it. Then stop the container with docker stop -t 1 under write load and look for crash recovery messages in the logs on the next start.
3. Interview question: "Would you run your company's primary production database in a Docker container? Justify your answer." Hint: weigh storage and I/O, backup and point-in-time recovery, high availability and failover, upgrades, team expertise, and what a managed service provides out of the box.

---

<a id="7-docker-networking"></a>