10. Docker Security

Containers share the host kernel, so a weak default, an overly generous flag, or a careless image can turn a compromised application into a compromised host. This chapter works through Docker security in layers: first hardening how a container runs (users, namespaces, capabilities, filesystems), then the kernel mechanisms that confine it (seccomp, AppArmor, SELinux) and the flags that undo them, then how to trust what goes into an image (scanning, signing, SBOMs, provenance, base images), and finally how to handle secrets at build time and at runtime. Each layer is cheap on its own; together they turn "one bug away from root on the host" into defense in depth.

<a id="101-container-isolation-and-hardening"></a>

10.1 Container Isolation and Hardening

This section covers the settings that reduce what a process inside a container can do, even after an attacker has taken it over: who it runs as, how its identities map to the host, which privileges it keeps, and whether it can change its own filesystem.

Running as Non-Root

Why it matters. Unless an image says otherwise, the main process of a container runs as root, UID 0. Namespaces change what the process can see, but without user namespaces (covered in the next two topics) they do not change who it is: UID 0 in the container is UID 0 on the host kernel. Docker's default restrictions (dropped capabilities, seccomp, AppArmor or SELinux) make that root much weaker than real root, but every one of these protections is one bug, misconfiguration, or extra flag away from failing. Real container escapes have depended on it. For example, the 2019 runc vulnerability CVE-2019-5736 let a malicious container overwrite the host's runc binary, and it required root inside the container.

Running as root also causes everyday problems that have nothing to do with exploits:

• Files written to bind mounts are owned by root on the host.
• A compromised app can install packages, modify its own code, or read any file in the container.
• Platforms such as Kubernetes (runAsNonRoot) and OpenShift refuse or remap root containers.

Analogy. Giving a hotel guest the master key "because they will only use it on their own room" works until the guest, or someone who steals the key, wanders down the hallway. A non-root user is a key that only opens one room.

What "root by default" looks like

# The default user of most images is root
docker run --rm alpine id
# uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),...

# Files created in a bind mount belong to root on the host
mkdir -p demo
docker run --rm -v "$PWD/demo:/data" alpine touch /data/created-by-root
ls -ln demo
# -rw-r--r-- 1 0 0 0 ... created-by-root      <- UID 0, GID 0 on the host

# Override the user at run time with numeric UID:GID
docker run --rm --user 1000:1000 -v "$PWD/demo:/data" alpine touch /data/created-by-1000
ls -ln demo
# -rw-r--r-- 1 1000 1000 0 ... created-by-1000

# Check which user an image is configured to run as (empty output means root)
docker image inspect --format '{{.Config.User}}' nginx:1.27-alpine
docker image inspect --format '{{.Config.User}}' gcr.io/distroless/static-debian12:nonroot
# 65532

Building a non-root image. Create a dedicated user in the Dockerfile and switch to it with USER (see USER) as late as possible, after the steps that need root, such as package installation.

# syntax=docker/dockerfile:1
FROM python:3.13-slim

# Create a system group and user with a fixed, high UID/GID.
# A fixed UID makes volume permissions predictable; a high one avoids
# colliding with real users on the host.
RUN groupadd --system --gid 10001 app \
 && useradd  --system --uid 10001 --gid app --no-create-home --shell /usr/sbin/nologin app

WORKDIR /app

# Install dependencies while still root
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Application code stays owned by root and is only readable by "app",
# so a compromised process cannot rewrite its own code
COPY . .

# Give the app write access only where it really needs it
RUN mkdir -p /app/uploads && chown app:app /app/uploads

# Numeric form lets Kubernetes verify runAsNonRoot without reading /etc/passwd
USER 10001:10001

EXPOSE 8000
CMD ["gunicorn", "--bind", "0.0.0.0:8000", "app:wsgi"]

On Alpine-based images the user-creation commands differ:

RUN addgroup -S -g 10001 app && adduser -S -u 10001 -G app -H -s /sbin/nologin app

Many official images already ship a non-root user that you only have to activate, such as node in node:* images, nonroot (UID 65532) in distroless :nonroot tags, and dedicated variants such as nginxinc/nginx-unprivileged.

Common problems and their fixes

Symptom	Cause	Fix
EACCES binding to port 80	Ports below 1024 are privileged	Docker 20.10+ sets net.ipv4.ip_unprivileged_port_start=0 inside container network namespaces, so this mostly affects --network host; otherwise listen on a high port such as 8080 and map it with -p 80:8080
Permission denied writing to /app/...	Directory owned by root	chown only the directories that need writes, at build time
Named volume not writable	The volume was created before the image had the directory with correct ownership	Docker copies the image directory's ownership into a new, empty named volume; recreate the volume or fix ownership once with a root helper container
Bind mount not writable	Host directory owned by another UID	Run with --user "$(id -u):$(id -g)" in development, or align the host directory's owner with the image's UID
App tries to apt-get install at startup	Mutable runtime design	Move installation to the build; runtime should not modify the image

Key takeaways

• Containers run as root by default, and without user namespaces that is the host's UID 0.
• Create a dedicated user with a fixed, high numeric UID and switch to it with USER after root-only steps.
• Keep application code root-owned and read-only to the app user; grant write access only to specific directories.
• --user UID:GID overrides the image user at run time; docker image inspect --format '{{.Config.User}}' reveals the default.
• Port binding is rarely a reason to stay root: use a high port and map it.

🧪 Practice

1. (Beginner) Run docker run --rm <image> id for nginx:1.27-alpine, node:22-alpine, postgres:17, and gcr.io/distroless/static-debian12:nonroot. Which run as root by default? For postgres, inspect its entrypoint and explain how it starts as root but runs the server as a different user.
2. (Intermediate) Take a simple web app Dockerfile that runs as root, convert it to run as UID 10001, and make it work with a named volume mounted at /app/data. Verify with docker exec <container> id and by writing a file to the volume.
3. Interview question: "Docker already drops capabilities and applies seccomp. Why bother running as a non-root user inside the container?" Hint: think about defense in depth, what root still owns inside the container, bind-mounted host files, and escapes that required UID 0.

Rootless Docker

Why it exists. Running containers as non-root protects the host from the application. It does nothing about the Docker daemon itself, which normally runs as root. Anyone who can talk to the daemon can start a container that mounts / from the host, so membership in the docker group is effectively root access (see Docker Socket Exposure Risks). Rootless mode removes root from the picture entirely: the daemon, containerd, and every container run inside a user namespace owned by an ordinary user.

Analogy. Rootful Docker is a building manager with a master key who promises to give tenants only the keys they need. Rootless Docker is a tenant who rents their own apartment and sublets rooms: whatever goes wrong inside, nobody ever had a key to the rest of the building.

How it works. A helper called RootlessKit creates a user namespace in which the unprivileged user appears as UID 0. dockerd starts inside it and believes it is root. Containers it starts are nested inside that namespace, so their "root" maps to the user who installed Docker, and their other UIDs map to a range of subordinate IDs reserved for that user in /etc/subuid and /etc/subgid.

 Rootful (default)                         Rootless
 -----------------                         --------
 host UID 0                                host UID 1000 (alice)
  └─ dockerd            (UID 0)             └─ rootlesskit  (alice) creates a user namespace
      └─ container root (UID 0 on host)          └─ dockerd    (UID 0 inside = 1000 on host)
                                                     └─ container root -> host UID 1000
                                                        container UID 1 -> host UID 100000
                                                        container UID 2 -> host UID 100001 ...
                                                        (range taken from /etc/subuid)

Even a full container escape lands the attacker in an account that has no more power than alice.

Installing and using rootless mode (Ubuntu/Debian example)

# Prerequisites: newuidmap/newgidmap and a user systemd session
sudo apt-get install -y uidmap dbus-user-session

# The user needs at least 65,536 subordinate UIDs and GIDs
grep "^$(whoami):" /etc/subuid /etc/subgid
# /etc/subuid:alice:100000:65536
# /etc/subgid:alice:100000:65536

# If a rootful daemon is running, stop it first (optional but avoids confusion)
sudo systemctl disable --now docker.service docker.socket

# Install the rootless daemon as a user service (script ships with Docker packages)
dockerd-rootless-setuptool.sh install

# Start it now and on login; "linger" keeps it running after you log out
systemctl --user enable --now docker
sudo loginctl enable-linger "$(whoami)"

# Point the CLI at the per-user socket (the installer also creates a "rootless" context)
export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/docker.sock
# or: docker context use rootless

# Confirm
docker info --format '{{.SecurityOptions}}'
# [name=seccomp,profile=builtin name=rootless name=cgroupns]

Verifying the mapping

docker run -d --name sleeper alpine sleep 600

# Inside the container the process is root ...
docker exec sleeper id -u                  # 0

# ... on the host it belongs to alice
ps -o user,pid,cmd -C sleep
# USER   PID   CMD
# alice  4211  sleep 600

# The mapping table: container IDs -> host IDs -> range length
docker exec sleeper cat /proc/self/uid_map
#          0       1000          1
#          1     100000      65536

Limitations to know

Area	Rootless behavior	Workaround
Host ports below 1024	Not allowed by default	sudo sysctl net.ipv4.ip_unprivileged_port_start=80, or sudo setcap cap_net_bind_service=ep $(which rootlesskit)
Networking	User-mode network stack (slirp4netns or pasta); slower, and the source IP of incoming connections may be hidden	Choose a faster driver or port driver in the rootless configuration; accept the trade-off for most dev workloads
Resource limits (--memory, --cpus)	Require cgroup v2 with systemd delegation	Use a modern distribution with cgroup v2 (default on current releases)
--privileged	Allowed, but only grants what the user owns; cannot access host devices the user cannot	Use rootful Docker on a dedicated host when real hardware access is needed
Storage	overlay2 needs kernel 5.11+ (older kernels fall back to fuse-overlayfs)	Use a recent kernel
Bind mounts	Files owned by container UID 1+ appear on the host as subordinate UIDs (e.g. 100000)	Run the container as root (which maps to you), or chown from inside with docker run
Separate state	Images and containers live under ~/.local/share/docker, separate from any rootful daemon	Pull images again for the rootless daemon

Related options. Podman is rootless by default with no daemon at all. Docker Desktop already runs the engine inside a VM, and its Enhanced Container Isolation feature (Docker Business) adds user namespaces and blocks dangerous operations such as mounting the Docker socket into containers.

Key takeaways

• Rootless mode runs the daemon and all containers as an unprivileged user inside a user namespace.
• Container root maps to the installing user; other container UIDs map to that user's subordinate range in /etc/subuid.
• It protects against daemon compromise and container escapes, and removes the "docker group equals root" problem.
• Trade-offs include privileged ports, user-mode networking performance, and cgroup v2 requirements for limits.
• The CLI connects through $XDG_RUNTIME_DIR/docker.sock or the rootless context.

🧪 Practice

1. (Beginner) On a Linux VM, install rootless Docker, run docker run -d nginx, and use ps -o user,pid,cmd -C nginx on the host to see which users own the master and worker processes. Explain why the worker processes show a numeric UID such as 100100.
2. (Intermediate) Try to publish a container on host port 80 in rootless mode. Read the error, apply one of the workarounds, and document the security trade-off of the option you chose.
3. Interview question: "What is the difference between running a container as a non-root user and running Docker in rootless mode?" Hint: consider which process each one protects (the application or the daemon), who owns the socket, and what an attacker gains after an escape in each case.

User Namespace Remapping

Why it exists. Rootless mode changes how Docker is installed and operated. User namespace remapping (userns-remap) is a lighter alternative: the daemon keeps running as root, but every container gets a user namespace in which its UID 0 corresponds to an unprivileged, otherwise unused UID on the host. A process that escapes the container, or writes to a bind mount, does so as "nobody special".

How the mapping works. A user namespace has a translation table. Each line says "container IDs starting at X, map to host IDs starting at Y, for N IDs":

 container UID        host UID
 -------------        --------
       0      ----->   165536      (container root is an unprivileged host UID)
       1      ----->   165537
      ...               ...
   65535      ----->   231071

 /proc/<pid>/uid_map:   0   165536   65536
                        ^     ^        ^
                        |     |        +-- length of the range
                        |     +----------- first host UID
                        +----------------- first container UID

Inside the namespace the process still has root's powers over things owned by the namespace (its own files, mounts it created), but to the host kernel it is UID 165536, which owns nothing.

Enabling it

{
  "userns-remap": "default"
}

Put this in /etc/docker/daemon.json and restart the daemon. The value default makes Docker create a user and group called dockremap and add subordinate ranges for it to /etc/subuid and /etc/subgid. You can also name an existing user ("userns-remap": "builder" or "builder:builder"), provided it has entries in both files.

sudo systemctl restart docker

# Docker created the user and its subordinate range
grep dockremap /etc/subuid /etc/subgid
# /etc/subuid:dockremap:165536:65536
# /etc/subgid:dockremap:165536:65536

# Remapped daemons use a separate storage directory, so images must be pulled again
sudo ls /var/lib/docker/
# 165536.165536  buildkit  containers  image  ...

docker info --format '{{.SecurityOptions}}'
# [name=apparmor name=seccomp,profile=builtin name=userns]

Seeing the effect

docker run -d --name sleeper alpine sleep 600
ps -o user,pid,cmd -C sleep
# USER     PID   CMD
# 165536   5120  sleep 600            <- "root" in the container

docker exec sleeper cat /proc/self/uid_map
#          0     165536      65536

# A bind-mounted host file owned by real root is no longer writable by container root
docker run --rm -v /etc:/host-etc alpine sh -c 'echo test >> /host-etc/hosts'
# sh: can't create /host-etc/hosts: Permission denied

# Host files owned by UIDs outside the mapping appear as "nobody" (65534)
docker run --rm -v /etc:/host-etc alpine ls -ln /host-etc/hosts
# -rw-r--r--    1 65534    65534   ...  /host-etc/hosts

Opting out and incompatibilities. Remapping is daemon-wide. A specific container can opt out with --userns=host, which is required for some features:

• --privileged only works together with --userns=host.
• Sharing the host's PID or network namespace (--pid=host, --network=host) is not supported with remapping.
• Volume or storage plugins that are unaware of the mapping may not work.
• Bind mounts need host files owned by the remapped range (for example sudo chown -R 165536:165536 ./data).

Comparison of the three models

Aspect	Default (rootful)	userns-remap	Rootless
Daemon runs as	root	root	Unprivileged user
Container root on the host	UID 0	Unprivileged subordinate UID	The installing user
Docker socket access means	Host root	Host root	That user only
Setup	None	One daemon.json key	Per-user installation
Per-container opt-out	Not applicable	--userns=host	None
Feature compatibility	Full	Most; no host PID/network sharing	Most; networking and port limits

Kubernetes offers the same idea per pod with hostUsers: false, which gives each pod its own user namespace.

Key takeaways

• userns-remap maps container UID 0 to an unprivileged host UID while the daemon remains rootful.
• The mapping is visible in /proc/<pid>/uid_map, and processes appear on the host under the remapped UID.
• Enabling it switches the daemon to a separate storage directory; existing images and containers seem to disappear until you disable it.
• --userns=host opts a container out, and is needed for --privileged.
• It protects against escapes and bind-mount abuse, but not against misuse of the Docker socket.

🧪 Practice

1. (Beginner) Enable userns-remap on a test VM, run alpine sleep 600, and compare the UID shown by id inside the container with the one shown by ps on the host.
2. (Intermediate) With remapping enabled, mount a host directory into a container that writes to it, observe the permission error, and fix it without disabling remapping. Then explain why a container started with --userns=host --privileged defeats the protection.
3. Interview question: "Why is userns-remap not the default in Docker?" Hint: think about bind-mount ownership, the separate image store, features that require host namespaces, and backward compatibility with existing workloads.

Dropping Capabilities

Why it matters. Linux capabilities split root's powers into separate privileges (see Linux Capabilities for the fundamentals). Docker already keeps only 14 of them, but most applications need none at all. Every capability you leave in place is a tool an attacker can use. For example, NET_RAW allows crafting raw packets, including ARP and DNS spoofing against other containers on the same bridge, and CHOWN with DAC_OVERRIDE lets root ignore file permissions inside the container.

Strategy: deny by default. Start from nothing with --cap-drop ALL and add back only what the application proves it needs. This is the same least-privilege principle as a firewall that blocks everything and opens specific ports.

# Default: root in the container can change file ownership (CAP_CHOWN)
docker run --rm alpine chown nobody /tmp && echo "chown worked"
# chown worked

# Without any capabilities, the same root user cannot
docker run --rm --cap-drop ALL alpine chown nobody /tmp
# chown: /tmp: Operation not permitted

# Add back a single capability
docker run --rm --cap-drop ALL --cap-add CHOWN alpine chown nobody /tmp && echo "chown worked"
# chown worked

# Inspect the resulting sets inside a container (bitmask, decode with capsh)
docker run --rm --cap-drop ALL --cap-add NET_BIND_SERVICE alpine grep Cap /proc/self/status
# CapInh: 0000000000000000
# CapPrm: 0000000000000400
# CapEff: 0000000000000400
# CapBnd: 0000000000000400       <- only bit 10 (CAP_NET_BIND_SERVICE) remains

Non-root processes and capabilities. A process that runs as a non-root user normally has an empty effective set even with Docker's defaults. Dropping capabilities still matters because the bounding set limits what the process could ever regain: a setuid-root binary or a file with file capabilities inside the image could otherwise give an attacker those capabilities back. Combine --cap-drop ALL with a non-root user and no-new-privileges.

Finding the minimal set. There is no universal list; it depends on what the entrypoint does.

1. Run with --cap-drop ALL and watch for Operation not permitted errors in the logs.
2. Add one capability at a time and re-test the complete startup and normal operation.
3. Prefer changing the image (for example a non-root variant that does not need SETUID/SETGID to drop privileges) over adding capabilities.

Typical need	Capability	Notes
Entrypoint switches from root to an app user (gosu, su-exec, nginx master)	SETUID, SETGID	Not needed if the image starts as non-root
Entrypoint fixes ownership of volumes at startup	CHOWN (sometimes DAC_OVERRIDE, FOWNER)	Better done at build time
Bind ports below 1024 with host networking	NET_BIND_SERVICE	Usually unnecessary on bridge networks (Docker 20.10+)
Network tools such as tcpdump	NET_RAW, sometimes NET_ADMIN	Use a temporary debug container, not the app container
Unprivileged ping	None	Docker 20.10+ allows ICMP echo sockets without NET_RAW

Capabilities you should treat as red flags

Capability	Why it is dangerous
SYS_ADMIN	Mounts, namespaces, BPF, and dozens of other operations; close to full root and a common escape ingredient
SYS_PTRACE	Read and modify the memory of other processes (secrets, injected code)
SYS_MODULE	Load kernel modules: game over for the host
DAC_READ_SEARCH	Bypass read checks; used by the 2014 "Shocker" escape via open_by_handle_at
NET_ADMIN	Change interfaces, routes, and firewall rules
SYS_RAWIO, SYS_BOOT, SYS_TIME	Raw device I/O, reboot, change the host clock

In Compose

services:
  api:
    image: acme/api:1.4.2
    user: "10001:10001"
    cap_drop:
      - ALL                  # start from zero
    cap_add:
      - NET_BIND_SERVICE     # only if the app binds a port below 1024 with host networking

  proxy:
    image: nginx:1.27-alpine # starts as root, then switches workers to "nginx"
    cap_drop:
      - ALL
    cap_add:                 # what this entrypoint actually needs; verify for your version
      - CHOWN
      - SETUID
      - SETGID
      - NET_BIND_SERVICE

# Audit what a running container was given
docker inspect --format '{{.Name}} drop={{.HostConfig.CapDrop}} add={{.HostConfig.CapAdd}}' $(docker ps -q)

Key takeaways

• Docker's default set has 14 capabilities; most applications need zero.
• Use --cap-drop ALL and add back only proven needs with --cap-add.
• Even for non-root processes, dropping capabilities shrinks the bounding set and blocks regaining privileges.
• Treat SYS_ADMIN, SYS_PTRACE, SYS_MODULE, DAC_READ_SEARCH, and NET_ADMIN as red flags in any configuration review.
• Discover the minimal set by trial with full functional tests, and prefer images that need no capabilities.

🧪 Practice

1. (Beginner) Run docker run --rm --cap-drop ALL nginx:1.27-alpine and read the errors. Add capabilities one at a time until it starts, then confirm it serves a page.
2. (Intermediate) Repeat the exercise with nginxinc/nginx-unprivileged. How many capabilities does it need? Explain why, in terms of what the entrypoint no longer has to do.
3. Interview question: "A vendor's container requires --cap-add SYS_ADMIN. How would you evaluate and reduce that risk?" Hint: find which operation actually needs it (often a mount or FUSE), look for narrower alternatives, isolate the workload, and combine with seccomp, AppArmor/SELinux, and user namespaces.

Read-Only Root Filesystem

Why it matters. An attacker who gains code execution usually wants to write something: download a tool, drop a web shell, modify a binary or config file to persist, or plant a cron job. Containers are meant to be immutable: the image defines the code, and runtime changes are lost on restart anyway. Making the root filesystem read-only enforces that idea: the container can still run, but it cannot change itself.

Analogy. A museum exhibit behind glass: visitors can look at everything, and staff have one marked drawer where they may put things, but nobody can rearrange the display.

How it works. --read-only mounts the container's root filesystem, including its writable layer, as read-only. Anything that must stay writable is mounted explicitly:

• tmpfs mounts for scratch data (/tmp, PID files, caches), held in memory and lost on stop.
• Volumes or bind mounts for data that must persist.

 /                      read-only (image layers + container layer)
 ├── app/               read-only: code cannot be modified
 ├── etc/               read-only: config cannot be modified
 ├── tmp/               tmpfs: writable, in memory, noexec
 ├── run/               tmpfs: writable (PID files, sockets)
 └── data/              named volume: writable, persistent

Examples

# Writes to the root filesystem fail
docker run --rm --read-only alpine touch /hello
# touch: /hello: Read-only file system

# Provide writable scratch space explicitly.
# noexec: files in /tmp cannot be executed (blocks "download and run" attacks)
# nosuid: setuid bits are ignored; size: cap memory usage
docker run --rm --read-only \
  --tmpfs /tmp:rw,noexec,nosuid,size=64m \
  alpine sh -c 'echo ok > /tmp/x && cat /tmp/x'
# ok

# nginx needs its cache directory and a place for its PID file
docker run -d --name web --read-only \
  --tmpfs /var/cache/nginx:rw,size=32m \
  --tmpfs /var/run:rw,size=1m \
  --tmpfs /tmp:rw,noexec,nosuid,size=16m \
  -p 8080:80 nginx:1.27-alpine

Finding where an app writes. Run the container normally, exercise it, then ask Docker what changed in its writable layer. Every path in the output needs a tmpfs, a volume, or a change to the application.

docker run -d --name probe acme/api:1.4.2
# ... send requests, upload a file, trigger background jobs ...
docker diff probe
# C /tmp
# A /tmp/upload-7f3a.bin        A = added, C = changed, D = deleted
# C /var/run
# A /var/run/api.pid
# C /app
# A /app/__pycache__            <- Python bytecode cache; set PYTHONDONTWRITEBYTECODE=1
docker rm -f probe

In Compose

services:
  api:
    image: acme/api:1.4.2
    read_only: true
    tmpfs:
      - /tmp:size=64m,noexec,nosuid
      - /var/run:size=1m
    volumes:
      - uploads:/app/uploads     # persistent, writable
    environment:
      PYTHONDONTWRITEBYTECODE: "1"

volumes:
  uploads:

What stays writable with --read-only

Location	Writable?	Persists after container removal?
Image paths (/app, /etc, /usr, ...)	No	Not applicable
--tmpfs mounts	Yes	No (memory only)
Named volumes and bind mounts	Yes, unless mounted :ro	Yes
/dev/shm	Yes (Docker mounts it separately)	No
/etc/hosts, /etc/resolv.conf, /etc/hostname	Managed by Docker as bind mounts	Not applicable

Key takeaways

• --read-only (Compose read_only: true) makes the container filesystem immutable at runtime.
• Writable paths are opt-in through tmpfs mounts and volumes; mount scratch space with noexec,nosuid.
• docker diff on a normally running container reveals which paths need to be writable.
• A read-only root blocks tool drops, code tampering, and persistence, and catches accidental runtime state.
• It complements, not replaces, non-root users and dropped capabilities.

🧪 Practice

1. (Beginner) Start redis:7-alpine with --read-only and observe whether it works. Use docker diff on a normal Redis container to find the writable path it needs, and fix the read-only run with a volume.
2. (Intermediate) For an application of your own, run it with --read-only --tmpfs /tmp:noexec, then try to download and execute a script into /tmp from docker exec. Explain each failure you see.
3. Interview question: "If a container is ephemeral and restarted from a clean image anyway, why does a read-only filesystem add security?" Hint: consider what an attacker can do between compromise and restart, persistence in volumes versus the root filesystem, and long-running containers.

no-new-privileges

Why it exists. Even a process that starts as a non-root user with few capabilities can gain privileges when it executes certain files:

• setuid/setgid binaries such as sudo, su, passwd, or mount run with the file owner's identity, often root.
• File capabilities (setcap cap_net_raw+ep /usr/bin/ping) grant capabilities on execution.

If the image contains such a binary, perhaps inherited from a base image you never looked at closely, it becomes a privilege-escalation path. The Linux flag PR_SET_NO_NEW_PRIVS closes it: once set on a process, no execve() can ever grant more privileges than the process already has. The flag is inherited by all children and cannot be cleared.

Analogy. A one-way valve: privileges can flow out of the process (dropping from root to a user still works), but never back in.

Demonstration

# syntax=docker/dockerfile:1
FROM debian:bookworm-slim
# Simulate a dangerous setuid-root binary left in an image
RUN cp /usr/bin/id /usr/local/bin/suid-id && chmod u+s /usr/local/bin/suid-id
USER 1000:1000

docker build -t nnp-demo .

# Without the flag, executing the setuid binary yields effective UID 0
docker run --rm nnp-demo suid-id
# uid=1000 gid=1000 euid=0(root) groups=1000

# With the flag, the setuid bit is ignored
docker run --rm --security-opt no-new-privileges nnp-demo suid-id
# uid=1000 gid=1000 groups=1000

# The kernel exposes the flag in /proc
docker run --rm --security-opt no-new-privileges nnp-demo grep NoNewPrivs /proc/self/status
# NoNewPrivs:	1

Enabling it

Scope	Setting
One container	docker run --security-opt no-new-privileges ... (or no-new-privileges:true)
Compose	security_opt: ["no-new-privileges:true"]
Every container on a daemon	"no-new-privileges": true in /etc/docker/daemon.json
Kubernetes	securityContext.allowPrivilegeEscalation: false

Side effects. sudo and su stop working inside the container, which is the point. Entrypoints that start as root and drop privileges with gosu, su-exec, or setpriv keep working, because they reduce privileges rather than gain them. On SELinux hosts the flag also restricts some domain transitions.

Putting 10.1 together. A hardened baseline combines every setting in this section; each one covers a gap the others leave open.

# --user 10001:10001           not root inside the container
# --cap-drop ALL               no capabilities at all
# no-new-privileges            setuid binaries and file caps cannot add any back
# --read-only                  immutable root filesystem
# --tmpfs /tmp:noexec,...      explicit, non-executable scratch space
# --memory, --pids-limit       bound resource abuse (see chapter 11)
docker run -d --name api \
  --user 10001:10001 \
  --cap-drop ALL \
  --security-opt no-new-privileges \
  --read-only \
  --tmpfs /tmp:rw,noexec,nosuid,size=64m \
  --memory 512m --pids-limit 200 \
  -p 8080:8080 \
  acme/api:1.4.2

The Compose equivalent:

services:
  api:
    image: acme/api:1.4.2
    user: "10001:10001"
    cap_drop: [ALL]
    security_opt:
      - no-new-privileges:true
    read_only: true
    tmpfs:
      - /tmp:size=64m,noexec,nosuid
    mem_limit: 512m
    pids_limit: 200
    ports:
      - "8080:8080"

To audit a whole host against the CIS Docker Benchmark, which checks daemon settings, file permissions, and every running container for these options, run Docker Bench for Security. It runs as root on the host but only reads configuration:

git clone https://github.com/docker/docker-bench-security.git
cd docker-bench-security
sudo sh docker-bench-security.sh
# [WARN] 5.x - Ensure that privileged containers are not used
# [WARN] 5.x - Ensure that the container is restricted from acquiring additional privileges
# [WARN] 5.x - Ensure that the container's root filesystem is mounted as read only
# ...

Key takeaways

• no-new-privileges stops setuid/setgid binaries and file capabilities from raising privileges on execve().
• The flag is inherited and irreversible; dropping privileges still works, gaining them does not.
• Enable it per container, in Compose, or daemon-wide with "no-new-privileges": true.
• Combine it with non-root users, --cap-drop ALL, and --read-only for a hardened baseline.
• Docker Bench for Security audits a host against the CIS Docker Benchmark.

🧪 Practice

1. (Beginner) Build the nnp-demo image and reproduce both outputs. Then check /proc/self/status in a container with and without the flag.
2. (Intermediate) Find all setuid and setgid binaries in an image you use with docker run --rm --user 0 <image> find / -xdev -perm /6000 -type f 2>/dev/null. Decide which can be removed in the Dockerfile and which are neutralized by no-new-privileges.
3. Interview question: "Why can an entrypoint still use gosu to switch from root to an app user when no-new-privileges is enabled, while sudo from that app user fails?" Hint: compare the direction of the privilege change, and what the kernel does with the setuid bit on execve() when the flag is set.

<a id="102-kernel-security-mechanisms"></a>

10.2 Kernel Security Mechanisms

This section explains the kernel-level filters and mandatory access control systems Docker applies to every container, how to customize them, and the two most common ways they get bypassed entirely: privileged containers and an exposed Docker socket.

Seccomp Profiles

Why it exists. Every interaction between a program and the kernel is a system call (syscall): open, read, mount, reboot, and roughly 300 to 450 more depending on the architecture. Because containers share the host kernel, every syscall a container can make is attack surface: most kernel privilege-escalation bugs are reached through some obscure syscall that ordinary applications never use. Seccomp (secure computing mode) lets a process install a filter that the kernel checks on every syscall, before any other permission check, and rejects the ones that are not allowed.

Analogy. If capabilities decide which rooms a visitor may enter, seccomp decides which verbs they may even say to the receptionist. A request for "reboot" is refused before anyone looks at their badge.

Where seccomp fits among the other layers

 process in a container calls, e.g., mount("/dev/sda1", "/mnt", ...)
        |
  [1] seccomp       Is this syscall (with these arguments) allowed at all?
        |           -> no: fail immediately with EPERM
  [2] DAC + caps    Do UID/GID, permission bits, and capabilities allow it?
        |
  [3] LSM           Does the AppArmor or SELinux policy allow it?
        |
  [4] namespaces    What does "/mnt" or "/dev/sda1" refer to for this process?
        |
     the kernel performs the operation

Docker's default profile. Docker applies a built-in profile to every container unless told otherwise. It is an allowlist: its default action is to return an error, and it explicitly allows the several hundred syscalls normal applications need. Roughly 40 to 50 are blocked, including:

Blocked syscalls (examples)	Why
mount, umount2, pivot_root	Filesystem manipulation; allowed only if CAP_SYS_ADMIN is added
reboot, kexec_load, init_module, delete_module	Host-wide effects or kernel code loading
unshare and clone with CLONE_NEWUSER	Creating new namespaces is a frequent exploit step
keyctl, add_key, request_key	Kernel keyring is not namespaced
bpf, perf_event_open, userfaultfd	Large kernel attack surface, frequently exploited
open_by_handle_at	The "Shocker" escape primitive
clock_settime, settimeofday	The clock is not namespaced

The profile is capability-aware: adding a capability such as SYS_ADMIN or SYS_PTRACE also allows the matching syscalls.

# Is seccomp active? Mode 2 means "filter"
docker run --rm alpine grep Seccomp /proc/self/status
# Seccomp:          2
# Seccomp_filters:  1

# The daemon reports the default profile
docker info --format '{{.SecurityOptions}}'
# [name=apparmor name=seccomp,profile=builtin name=cgroupns]

# Creating a user namespace is blocked by the default profile
docker run --rm debian:bookworm-slim unshare --user --map-root-user whoami
# unshare: unshare failed: Operation not permitted

# Disabling seccomp removes that layer (for diagnosis only; never in production)
docker run --rm --security-opt seccomp=unconfined debian:bookworm-slim unshare --user --map-root-user whoami
# root

Writing a custom profile. A profile is JSON. The essential fields are a default action, the architectures it applies to, and a list of rules:

Action	Effect
SCMP_ACT_ALLOW	Allow the syscall
SCMP_ACT_ERRNO	Fail it with an error code (errnoRet, default EPERM)
SCMP_ACT_KILL_PROCESS	Kill the process
SCMP_ACT_LOG	Allow it, but log it (useful for building a profile)
SCMP_ACT_TRACE, SCMP_ACT_TRAP	Notify a tracer or send SIGSYS

To learn the format, here is a small deny-list profile that forbids changing file permissions:

{
  "defaultAction": "SCMP_ACT_ALLOW",
  "syscalls": [
    {
      "names": ["chmod", "fchmod", "fchmodat", "fchmodat2"],
      "action": "SCMP_ACT_ERRNO",
      "errnoRet": 1
    }
  ]
}

docker run --rm --security-opt seccomp=./no-chmod.json alpine chmod 777 /tmp
# chmod: /tmp: Operation not permitted

A profile replaces Docker's default rather than adding to it, so this example is far weaker than the default: it allows mount, keyctl, and everything else. Real profiles should be allowlists. The practical way to build one:

1. Copy Docker's default profile (profiles/seccomp/default.json in the moby/moby repository) as a starting point.
2. Remove syscall groups your application never uses, or switch defaultAction to SCMP_ACT_LOG in a test environment and record what the application actually calls (in the kernel audit log), with a full test suite running.
3. Apply the tightened profile and run the tests again.

services:
  api:
    image: acme/api:1.4.2
    security_opt:
      - seccomp=./seccomp/api.json      # path relative to the Compose file

Kubernetes note. For years Kubernetes ran containers with seccomp disabled unless a pod asked for it. Set securityContext.seccompProfile.type: RuntimeDefault (or enable the kubelet's default-seccomp setting) to get the same protection Docker gives by default.

Key takeaways

• Seccomp filters syscalls before any other check, shrinking the kernel attack surface a container can reach.
• Docker's default profile is an allowlist that blocks around 40 to 50 dangerous syscalls and adapts to added capabilities.
• --security-opt seccomp=<file> replaces the default profile; seccomp=unconfined disables the layer entirely.
• Build custom profiles from the default plus observed behavior (SCMP_ACT_LOG), not from scratch deny lists.
• Verify with grep Seccomp /proc/self/status (mode 2 means a filter is active).

🧪 Practice

1. (Beginner) Run the unshare example with and without seccomp=unconfined and compare. Then check Seccomp: in /proc/self/status for both cases.
2. (Intermediate) Download Docker's default profile, remove chmod, fchmod, fchmodat, and fchmodat2 from the allowed names, and verify that chmod now fails while other operations still work. Explain why this is safer than the deny-list example.
3. Interview question: "Your container works with seccomp=unconfined but fails with the default profile. How do you fix it without disabling seccomp?" Hint: identify the blocked syscall (strace, the audit log, SCMP_ACT_LOG), check whether a capability unlocks it, and allow only that syscall in a copy of the default profile.

AppArmor

Why it exists. Seccomp decides which syscalls are allowed, but it barely looks at their arguments, such as which file is being opened. Classic permissions (UID and mode bits) are discretionary: root, or the owner of a file, can change them. AppArmor is a Linux Security Module (LSM) that adds mandatory access control (MAC): an administrator-defined profile states which files, capabilities, and network operations a program may use, and not even root inside the confined program can override it. AppArmor identifies resources by path, which makes its profiles relatively easy to read and write.

AppArmor is the default LSM on Ubuntu, Debian, and SUSE. Hosts based on Fedora and RHEL use SELinux instead (next topic); a host uses one or the other, not both.

Docker's docker-default profile. On AppArmor hosts, Docker generates and loads a profile called docker-default and applies it to every container. Among other things it:

• denies writes to most of /proc/sys, /proc/sysrq-trigger, and /sys;
• denies reading sensitive kernel interfaces such as /proc/kcore;
• denies mount operations;
• prevents a container from changing its own AppArmor profile.

# Which profile confines a container's process?
docker run --rm alpine cat /proc/self/attr/current
# docker-default (enforce)

# Same information from the host
docker run -d --name web nginx:1.27-alpine
docker inspect --format '{{.AppArmorProfile}}' web
# docker-default

# Profiles loaded in the kernel
sudo aa-status | grep -A3 "profiles are in enforce mode"

Writing a custom profile. Suppose an API container should never start a shell or any other program, even when an attacker achieves code execution or someone runs docker exec. AppArmor can deny execution from the system binary directories:

#include <tunables/global>

profile docker-api-noshell flags=(attach_disconnected,mediate_deleted) {
  #include <abstractions/base>

  network inet stream,                 # TCP over IPv4
  network inet6 stream,                # TCP over IPv6
  deny network raw,                    # no raw sockets (packet crafting)
  deny network packet,

  file,                                # allow ordinary file access ...
  deny /bin/** wlx,                    # ... but never write, link, or execute
  deny /sbin/** wlx,                   #     anything in system binary directories
  deny /usr/bin/** wlx,
  deny /usr/sbin/** wlx,
  /app/api ix,                         # the one program allowed to run

  deny mount,                          # no filesystem mounts
  deny @{PROC}/sys/** w,               # no kernel tuning through /proc/sys
  deny /sys/** w,

  capability net_bind_service,         # every capability not listed is denied
}

# Load (or reload) the profile into the kernel; -W writes it to the cache
sudo apparmor_parser -r -W /etc/apparmor.d/containers/docker-api-noshell

# Run a container under it
docker run -d --name api --security-opt apparmor=docker-api-noshell acme/api:1.4.2

# Even docker exec is confined by the same profile
docker exec -it api sh
# OCI runtime exec failed: exec failed: ... permission denied

# Denials are logged by the kernel
sudo journalctl -k | grep 'apparmor="DENIED"'
# ... apparmor="DENIED" operation="exec" profile="docker-api-noshell" name="/bin/busybox" ...

Developing profiles safely. Put a profile in complain mode (flags=(complain, attach_disconnected,mediate_deleted)) to log violations without blocking them, exercise the application, then tighten and switch to enforce. Tools such as aa-logprof can propose rules from the logs.

Options summary

Setting	Effect
(nothing)	docker-default profile
--security-opt apparmor=<name>	Use a profile already loaded in the kernel
--security-opt apparmor=unconfined	No AppArmor confinement (avoid)
--privileged	Also disables AppArmor confinement
Compose security_opt: ["apparmor=<name>"]	Same as the flag

Key takeaways

• AppArmor is path-based mandatory access control that root inside the container cannot override.
• Docker applies the docker-default profile on AppArmor hosts, mainly protecting /proc, /sys, and mounts.
• Custom profiles are loaded with apparmor_parser and selected with --security-opt apparmor=<name>.
• Profiles also confine docker exec sessions, which makes them useful against interactive abuse.
• Use complain mode and kernel logs to develop profiles before enforcing them.

🧪 Practice

1. (Beginner) On an Ubuntu host, check which profile confines a running container from both inside (/proc/self/attr/current) and outside (docker inspect). Then try docker run --rm alpine sh -c 'echo 1 > /proc/sysrq-trigger' and explain the error.
2. (Intermediate) Write a profile that allows an nginx container to run normally but denies writes everywhere except /var/cache/nginx, /var/run, and /tmp. Develop it in complain mode first and use the logs to find missing rules.
3. Interview question: "How do AppArmor and seccomp complement each other?" Hint: compare what each one filters (syscall numbers versus resources such as paths and capabilities), and give an attack that one stops but the other does not.

SELinux

Why it exists. SELinux (Security-Enhanced Linux) is the other major mandatory access control LSM, the default on Fedora, RHEL, CentOS Stream, Rocky, and AlmaLinux. Instead of paths it uses labels: every process and every file carries a security context, and a policy states which process types may access which file types. Because labels travel with the file rather than its path, SELinux is not fooled by hard links, bind mounts, or renamed paths.

Analogy. AppArmor is a guest list by room number. SELinux puts a colored wristband on every guest and a colored sticker on every door, and the rule is "wristband color must match the door color". Moving the door to another hallway does not change its sticker.

Labels used for containers. A context has the form user:role:type:level. For containers the important parts are:

• Type: container processes run as container_t and may only access files labeled container_file_t (and a few read-only system types). Host files such as /etc/shadow (shadow_t) or home directories (user_home_t) are off limits even to container root.
• Level (MCS categories): each container receives a unique pair of categories such as s0:c123,c456. Two containers are both container_t, but a process labeled c123,c456 cannot touch files labeled c789,c812. This isolates containers from each other.

 host process  (unconfined_t)           /etc/shadow        shadow_t
 container A   container_t:s0:c1,c2 --> /var/lib/.../A     container_file_t:s0:c1,c2   allowed
 container A   container_t:s0:c1,c2 -X> /var/lib/.../B     container_file_t:s0:c3,c4   denied (MCS)
 container A   container_t:s0:c1,c2 -X> /home/alice        user_home_t                 denied (type)

Enabling and checking

{
  "selinux-enabled": true
}

Distribution packages of Docker for SELinux hosts usually set this already in /etc/docker/daemon.json. The host must be in enforcing mode (getenforce prints Enforcing).

docker info --format '{{.SecurityOptions}}'
# [name=seccomp,profile=builtin name=selinux name=cgroupns]

docker run -d --name web nginx:1.27-alpine
docker inspect --format '{{.ProcessLabel}} | {{.MountLabel}}' web
# system_u:system_r:container_t:s0:c215,c822 | system_u:object_r:container_file_t:s0:c215,c822

ps -eZ | grep container_t
# system_u:system_r:container_t:s0:c215,c822  4302 ?  00:00:00 nginx

Bind mounts and relabeling. The most common SELinux surprise: a bind-mounted host directory keeps its host label, so the container cannot read it even though the Unix permissions look fine.

mkdir -p ~/site && echo "hello" > ~/site/index.html
ls -Z ~/site/index.html
# unconfined_u:object_r:user_home_t:s0 /home/alice/site/index.html

docker run --rm -v ~/site:/usr/share/nginx/html alpine cat /usr/share/nginx/html/index.html
# cat: can't open '/usr/share/nginx/html/index.html': Permission denied

# :Z relabels the content with THIS container's private label (one container only)
docker run --rm -v ~/site:/usr/share/nginx/html:Z alpine cat /usr/share/nginx/html/index.html
# hello
ls -Z ~/site/index.html
# system_u:object_r:container_file_t:s0:c401,c977 /home/alice/site/index.html

# :z relabels with a shared label (s0, no categories) usable by every container
docker run --rm -v ~/site:/usr/share/nginx/html:z alpine cat /usr/share/nginx/html/index.html

Suffix	Label applied	Use when
:Z	Private: container_file_t with this container's categories	One container owns the data
:z	Shared: container_file_t:s0	Several containers share the data
(none)	Unchanged	The directory is already labeled correctly

Never use :z or :Z on system directories such as /, /etc, /usr, or your whole home directory: Docker relabels the host files recursively, which can break the host itself.

Other options and debugging

# Disable SELinux separation for one container (avoid; --privileged does this too)
docker run --security-opt label=disable ...

# Run with a specific type or level
docker run --security-opt label=type:my_container_t ...
docker run --security-opt label=level:s0:c100,c200 ...

# Find denials and explain them
sudo ausearch -m avc -ts recent
sudo ausearch -m avc -ts recent | audit2why

For workloads that need more than container_t allows, the udica tool generates a tailored SELinux policy from a container's docker inspect output, which is much safer than disabling labeling.

AppArmor and SELinux compared

Aspect	AppArmor	SELinux
Identifies resources by	Path	Label stored on the object
Default on	Ubuntu, Debian, SUSE	Fedora, RHEL and derivatives
Container default	docker-default profile	container_t type + unique MCS categories
Isolates containers from each other	Mostly via other layers	Yes, through MCS categories
Learning curve	Lower	Higher
Typical fix for denied bind mount	Adjust the profile	:z / :Z relabeling

Key takeaways

• SELinux confines containers with labels: processes run as container_t and may only use container_file_t files.
• Unique MCS categories per container isolate containers from one another, not just from the host.
• Bind mounts need relabeling with :Z (private) or :z (shared); never relabel system directories.
• Diagnose with ausearch -m avc; generate tailored policies with udica instead of using label=disable.
• The daemon needs "selinux-enabled": true and the host must be enforcing.

🧪 Practice

1. (Beginner) On a Fedora or RHEL VM, start two containers and compare their ProcessLabel values. Which part differs, and what does that difference prevent?
2. (Intermediate) Reproduce the bind-mount denial, read the AVC message with ausearch, fix it with :Z, and then show that a second container cannot read the same directory until you switch to :z.
3. Interview question: "A colleague fixed a 'permission denied' on a bind mount by setting SELinux to permissive on the host. What would you do instead?" Hint: consider relabeling options, why permissive mode affects every process on the host, and tools that produce narrow policies.

Privileged Containers and Risks

What --privileged really does. The name sounds like "a bit more access". In reality the flag removes almost every protection discussed in this chapter at once:

Protection	Normal container	--privileged
Capabilities	14 default	All of them, including SYS_ADMIN and SYS_MODULE
Devices	Small safe set (/dev/null, /dev/zero, ...)	Every host device: disks, /dev/mem, USB, GPUs
Seccomp	Default profile	Disabled
AppArmor / SELinux	docker-default / container_t	Unconfined
/sys and /proc	/sys read-only, sensitive /proc paths masked	/sys writable, /proc paths unmasked
Cgroup device rules	Restrictive	Allow all

What remains is mostly namespaces, and namespaces alone are not a security boundary against a process that holds every capability and can see every disk.

Why this matters. In practice, a privileged container should be treated as equivalent to giving its processes root on the host. Anything that compromises the application, such as a vulnerable dependency, a malicious image, or a stolen deployment credential, inherits that level of access. The question to ask in a review is never "is this container well behaved?" but "would I give this image, and everyone who can change it, root on this machine?"

Analogy. A normal container is a tenant in a locked apartment. A privileged container is a tenant who also holds the building's master key ring and the codes to the electrical room. The apartment walls are still there, but they no longer matter.

Why people reach for it. --privileged is usually added to make an error go away, not because the workload needs everything. Common legitimate-sounding reasons and narrower alternatives:

Real need	Narrower alternative
Access one device (serial port, USB, video)	--device /dev/ttyUSB0 (optionally with :r or :rw permissions)
GPU access	--gpus all (NVIDIA Container Toolkit) or --device for specific GPUs
Configure networking in the container	--cap-add NET_ADMIN
One blocked syscall	A custom seccomp profile that allows only that syscall
Build images inside CI	Rootless BuildKit, a remote Buildx builder, or a dedicated build service (see Docker-in-Docker vs Socket Mounting)
Run nested containers	Rootless Docker-in-Docker or a runtime designed for nesting, such as Sysbox
Low-level system agents (CNI, storage, monitoring)	Run them on dedicated, tightly controlled hosts, and document why the access is needed

Privileged by another name. Several other flags grant a large part of the same power on their own. A configuration review should flag them all:

Setting	What it gives away
--cap-add SYS_ADMIN (or ALL)	Most administrative kernel operations
--security-opt seccomp=unconfined	The syscall filter
--security-opt apparmor=unconfined / label=disable	Mandatory access control
--pid=host	Visibility of, and signals to, every host process
--network=host	Host interfaces and services bound to localhost
--ipc=host	Host shared memory segments
--userns=host (on a remapped daemon)	The user namespace protection
-v /:/host, -v /etc:/etc, -v /var/run/docker.sock:...	Host files or the Docker API directly
--device /dev/mem, --device /dev/sda	Raw access to host memory or disks

Detecting privileged containers

# List running containers with the flags that matter most
docker ps -q | xargs docker inspect --format \
  '{{.Name}} privileged={{.HostConfig.Privileged}} pid={{.HostConfig.PidMode}} net={{.HostConfig.NetworkMode}} caps={{.HostConfig.CapAdd}} secopt={{.HostConfig.SecurityOpt}}'
# /web     privileged=false pid= net=bridge caps=[] secopt=[]
# /agent   privileged=true  pid=host net=host caps=[] secopt=[]     <- needs justification

# In Compose projects, search the configuration before it runs
docker compose config | grep -nE 'privileged|cap_add|pid: host|network_mode: host|seccomp|apparmor|label|docker.sock'

Preventing them by policy.

• Kubernetes: Pod Security Standards at the baseline or restricted level reject privileged pods, host namespaces, and dangerous capabilities. Admission controllers such as Kyverno or OPA Gatekeeper can enforce custom rules.
• Docker Engine: authorization plugins can reject API requests that set Privileged, host namespaces, or sensitive mounts.
• Docker Desktop: Enhanced Container Isolation (Docker Business) runs containers in a user namespace, so even --privileged containers cannot reach the Desktop VM's kernel resources.
• Process: treat any privileged: true in a pull request like a request for root SSH access, with an owner, a justification, and an expiry.

Key takeaways

• --privileged grants all capabilities and all devices, and disables seccomp, AppArmor/SELinux, and /proc//sys masking.
• Treat a privileged container as root on the host, and treat its image and its deployers as host administrators.
• Almost every use case has a narrower alternative: --device, --gpus, a single --cap-add, or a custom seccomp profile.
• Host namespaces, unconfined security options, and sensitive host mounts are "privileged by another name".
• Detect with docker inspect, prevent with Pod Security Standards, authorization plugins, or Enhanced Container Isolation.

🧪 Practice

1. (Beginner) Compare docker run --rm alpine ls /dev with the same command plus --privileged, and compare grep Cap /proc/self/status in both. Summarize the difference in two sentences.
2. (Intermediate) Write a shell script that audits every running container on a host and prints a warning for each setting in the "privileged by another name" table. Test it against a Compose project that deliberately uses two of them.
3. Interview question: "A team says their monitoring agent must run with --privileged. How do you handle the request?" Hint: find out which kernel interfaces the agent actually reads, map each one to a device, capability, or read-only mount, and consider host isolation and policy exceptions for what remains.

Docker Socket Exposure Risks

Why it matters. The Docker CLI is only a client. Everything happens through the Engine API, which the daemon serves on a Unix socket, /var/run/docker.sock, and optionally on a TCP port. Whoever can talk to that API can create containers with any configuration, including privileged containers and arbitrary host mounts. Because a rootful daemon runs as root, access to the Docker socket is equivalent to root on the host. That has three practical consequences:

1. The docker group is a root group. Adding a user to it so they can skip sudo gives them full control of the machine.
2. Mounting the socket into a container gives that container control of the host. A compromise of the container, or of any image it runs, becomes a host compromise.
3. An API port exposed on the network without authentication gives the host to anyone who can reach it. Internet-wide scans for open Docker API ports are routine, and unprotected hosts are regularly taken over, often to run cryptocurrency miners.

Analogy. The socket is not a window into the building; it is the building's front desk, which hands out keys to any room on request. Mounting it into a container is like installing a copy of that desk inside one apartment.

 container with the socket mounted
 +-------------------------------+
 |  app (compromised)            |
 |    |                          |
 |    | HTTP over /var/run/docker.sock
 +----|--------------------------+
      v
 dockerd (root) ---- creates any container the request describes,
                     with any mounts, devices, or privileges

A common misconception: :ro does not help. Mounting the socket read-only only prevents changing the socket file. Connecting to it and sending API requests, including requests that create containers, works exactly the same.

services:
  dashboard:
    image: example/dashboard
    volumes:
      # Still full API access. ":ro" protects the socket file, not the API.
      - /var/run/docker.sock:/var/run/docker.sock:ro

Tools that commonly ask for the socket. Reverse proxies that discover containers (Traefik), management UIs (Portainer), auto-updaters, CI runners, log and metrics collectors, and Testcontainers. Each one may be justified, but each one makes its image, its plugins, and its own vulnerabilities part of your host's trusted computing base.

Mitigations, from strongest to weakest

Mitigation	How it helps
Do not mount the socket	Use file-based or API-based discovery, a remote builder, or a dedicated service that does not need the Engine API
Rootless Docker	The socket is worth only the unprivileged user that runs the daemon
Socket proxy	A small proxy exposes only selected endpoints (for example, read-only container listing) to the consuming container
Authorization plugin	The daemon checks every API request against a policy (for example, OPA-based), rejecting privileged or host-mount requests
Docker Desktop Enhanced Container Isolation	Blocks socket mounts into containers except for explicitly allowed images
Restrict group membership	Keep the docker group empty or limited to administrators; audit it like sudo

Using a socket proxy. Instead of giving Traefik the raw socket, place a filtering proxy in between. Only the proxy mounts the socket, and the consumer reaches it over an internal network.

services:
  socket-proxy:
    image: tecnativa/docker-socket-proxy     # pin a specific version or digest in practice
    environment:
      CONTAINERS: 1      # allow GET /containers/* (needed for discovery)
      EVENTS: 1          # allow the event stream
      POST: 0            # deny every POST: no create, start, exec, ...
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    networks: [socket]   # internal network, not reachable from outside

  traefik:
    image: traefik:v3.1
    command:
      - --providers.docker.endpoint=tcp://socket-proxy:2375
      - --providers.docker.exposedbydefault=false
    networks: [socket, web]
    ports:
      - "80:80"

networks:
  socket:
    internal: true       # no route to the outside world
  web: {}

Even read-only access leaks information: GET /containers/<id>/json returns each container's environment variables, which often contain secrets. Allow only the endpoints the consumer actually uses.

Remote access done safely. Never expose the API on plain TCP port 2375. For remote management:

# Option 1: SSH. Reuses existing SSH authentication; nothing extra listens on the network
docker -H ssh://admin@build-01.internal ps
docker context create build-01 --docker "host=ssh://admin@build-01.internal"
docker context use build-01

# Option 2: mutual TLS on port 2376. The daemon requires a client certificate
dockerd --tlsverify \
  --tlscacert=/etc/docker/ca.pem \
  --tlscert=/etc/docker/server-cert.pem \
  --tlskey=/etc/docker/server-key.pem \
  -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock

docker --tlsverify --tlscacert=ca.pem --tlscert=cert.pem --tlskey=key.pem \
  -H tcp://build-01.internal:2376 ps

Auditing a host

# Who can use the socket without sudo?
getent group docker
ls -l /var/run/docker.sock
# srw-rw---- 1 root docker 0 ... /var/run/docker.sock

# Which containers have the socket mounted?
docker ps -q | xargs docker inspect --format '{{.Name}}{{range .Mounts}} {{.Source}}{{end}}' | grep docker.sock

# Is the daemon listening on TCP?
sudo ss -ltnp | grep -E ':2375|:2376'

Key takeaways

• Access to the Docker socket on a rootful daemon is equivalent to root on the host, and so is membership in the docker group.
• Mounting the socket into a container, even with :ro, gives that container control of the host.
• Never expose port 2375; use SSH contexts or mutual TLS on 2376 for remote access.
• Prefer no socket access at all; otherwise use rootless Docker, a filtering socket proxy, or an authorization plugin.
• Audit group membership, socket mounts, and listening ports regularly.

🧪 Practice

1. (Beginner) Run the three audit commands on a machine you use. List who is in the docker group, which containers mount the socket, and whether the daemon listens on TCP.
2. (Intermediate) Deploy the socket-proxy Compose example. From a temporary container on the socket network, confirm that listing containers works and that a request to create a container is rejected. Then explain what information the allowed endpoints still expose.
3. Interview question: "Why is adding a developer to the docker group a security decision, and what would you offer instead?" Hint: think about what the API allows a client to request, and compare rootless Docker, Docker Desktop, remote builders, and sudo with auditing.

<a id="103-image-and-supply-chain-security"></a>

10.3 Image and Supply Chain Security

Hardening the runtime limits the damage a compromised container can do. This section is about not running compromised or vulnerable software in the first place: knowing what is inside an image, who built it and how, and whether the image you deploy is the one that was built.

Vulnerability Scanning

Why it matters. A typical image contains hundreds of packages you never chose explicitly: the base distribution, system libraries, a language runtime, and transitive dependencies. New vulnerabilities (CVEs) are published every day, so an image that was clean when it was built can be vulnerable a month later without a single byte changing. Vulnerability scanning answers the question "which known vulnerabilities affect the packages in this image?"

Where scanning fits in the supply chain

 source -> dependencies -> base image -> build (CI) -> registry -> deploy -> runtime
              |               |             |            |          |          |
           scan deps     trusted base   scan + SBOM   rescan     verify   monitor
                         (10.3)         provenance    continuously signature

How scanners work

1. Inventory. The scanner unpacks the image and reads package metadata: the OS package databases (/lib/apk/db/installed on Alpine, /var/lib/dpkg/status on Debian, the RPM database), language manifests and lockfiles (package-lock.json, requirements.txt, go.sum, JAR manifests), and build information embedded in binaries (for example in Go executables).
2. Matching. Each package and version is compared against vulnerability databases: distribution security trackers, the GitHub Advisory Database, OSV, and NVD.
3. Reporting. Each finding lists the CVE, severity, installed version, and the fixed version, if one exists.

Limits to keep in mind

Limit	Consequence
Only known packages are seen	Binaries added with curl + tar, or copied without metadata, are invisible
Only known vulnerabilities are reported	A clean scan does not mean "secure"; it means "no known CVEs in detected packages"
Distributions backport fixes	Scanners that ignore distro trackers report false positives
Severity is not risk	A critical CVE in a library function your app never calls may matter less than a medium one in exposed code
Many findings have no fix yet	Gates that fail on unfixed issues block every build without improving anything

To prioritize, combine severity (CVSS) with exploitability signals: whether a fix exists, the CISA Known Exploited Vulnerabilities (KEV) catalog, EPSS (the probability of exploitation), and whether the vulnerable package is reachable in your application.

Scanning with Trivy

# Full report for an image
trivy image python:3.13-slim
# python:3.13-slim (debian 12.x)
# Total: 87 (UNKNOWN: 0, LOW: 61, MEDIUM: 18, HIGH: 7, CRITICAL: 1)
#
# Library   | Vulnerability  | Severity | Installed | Fixed     | Title
# ----------+----------------+----------+-----------+-----------+------------------------
# libssl3   | CVE-2025-xxxx  | HIGH     | 3.0.x-1   | 3.0.y-1   | openssl: ...
# ...

# CI gate: fail only on high/critical issues that actually have a fix
trivy image --severity HIGH,CRITICAL --ignore-unfixed --exit-code 1 acme/api:1.4.2

# Machine-readable output for dashboards or code-scanning tools
trivy image --format sarif -o trivy.sarif acme/api:1.4.2

# Scan a source directory: dependencies, leaked secrets, and Dockerfile misconfigurations
trivy fs --scanners vuln,secret,misconfig .

Scanning with Grype (often paired with Syft, which generates SBOMs; see Software Bill of Materials (SBOM))

grype acme/api:1.4.2 --fail-on high      # non-zero exit if any high or critical finding
grype sbom:./sbom.spdx.json              # scan an existing SBOM instead of the image

Accepting a finding deliberately. When a vulnerability does not apply, record the decision instead of silently lowering the threshold. Keep the reason and a review date next to it:

# .trivyignore
# CVE-2024-xxxxx: vulnerable function is in the CLI tool, not used by the service.
# Reviewed by the platform team on 2026-09-30; re-evaluate by 2026-12-31.
CVE-2024-xxxxx

A more formal alternative is a VEX (Vulnerability Exploitability eXchange) document, a machine-readable statement such as "not affected, because the vulnerable code is not present", which scanners like Trivy, Grype, and Docker Scout can consume.

Common scanners

Tool	Notes
Trivy	Open source; images, filesystems, repositories, IaC, secrets; very common in CI
Grype (+ Syft)	Open source; strong SBOM workflow
Docker Scout	Built into the Docker CLI, Desktop, and Hub; base image recommendations and policies (next topic)
Snyk Container	Commercial; developer workflow and fix advice
Registry scanners (Harbor, Quay/Clair, ECR, ACR, GCR/Artifact Registry)	Scan on push and continuously while images are stored

When to scan. Scan at every stage, because each catches something different: in the developer loop (fast feedback), in CI (a gate before push), in the registry (continuous rescanning as new CVEs appear), and for running workloads (to know what is deployed right now).

Key takeaways

• Scanners inventory packages in an image and match them against vulnerability databases.
• A clean scan means "no known vulnerabilities in detected packages", not "secure"; software installed without package metadata is invisible.
• Gate CI on fixable high and critical findings; prioritize with KEV, EPSS, and reachability rather than CVSS alone.
• Record accepted risks with reasons and expiry dates (ignore files or VEX documents).
• Rescan stored and running images continuously, because new CVEs affect old images.

🧪 Practice

1. (Beginner) Scan python:3.13, python:3.13-slim, and python:3.13-alpine with Trivy or Grype and tabulate the totals by severity. Explain the differences.
2. (Intermediate) Add a scan step to a CI pipeline that fails on fixable high or critical vulnerabilities and uploads a SARIF report. Introduce an old base image tag to prove the gate fails, then fix it.
3. Interview question: "Your scanner reports 300 vulnerabilities in a production image. How do you decide what to fix first?" Hint: consider fixability, whether the package comes from the base image or your dependencies, exploitation data (KEV, EPSS), reachability, exposure of the service, and how a base image update can remove many findings at once.

Docker Scout

What it is. Docker Scout is Docker's own supply-chain analysis service, integrated into the Docker CLI (docker scout), Docker Desktop, and Docker Hub. It builds an SBOM for an image (or uses an SBOM attestation if the image has one), matches it against vulnerability data, and adds context that generic scanners do not have: which findings come from the base image, which base image update would remove them, and whether the image meets organization-wide policies.

Quick overview of an image

docker scout quickview acme/api:1.4.2
#   Target             | acme/api:1.4.2            |  0C   2H   5M  21L
#     digest           | 4b3c9a...                 |
#   Base image         | python:3.13-slim          |  0C   1H   4M  20L
#   Updated base image | python:3.13-slim (newer)  |  0C   0H   2M  19L
#                                                      ^ C/H/M/L = critical/high/medium/low

Core commands

# Detailed CVE list, filtered
docker scout cves acme/api:1.4.2 --only-severity critical,high
docker scout cves acme/api:1.4.2 --only-fixed          # only findings with an available fix
docker scout cves acme/api:1.4.2 --ignore-base         # only what YOUR layers added

# Use as a CI gate: non-zero exit code if vulnerabilities match the filters
docker scout cves acme/api:1.4.2 --only-severity critical,high --exit-code

# Which base image change would help the most?
docker scout recommendations acme/api:1.4.2

# What changed between two versions (packages and vulnerabilities)?
docker scout compare acme/api:1.4.2 --to acme/api:1.4.1

# Export the SBOM Scout computed
docker scout sbom --format spdx acme/api:1.4.2 > sbom.spdx.json

Scout accepts several kinds of targets through prefixes, so the same command works before an image is pushed, or even before it is built:

Prefix	Analyzes
image:// (default)	A local image, falling back to the registry
local:// / registry://	Only the local image store / only the registry
oci-dir://, archive://	An OCI layout directory or a docker save tarball
fs://	A local directory of source code (dependency manifests)
sbom://	An existing SBOM file

Policies. For repositories enabled for Scout in an organization, Scout evaluates images against policies and shows the results in the CLI, Desktop, and Hub. Built-in policies cover areas such as:

• no fixable critical or high vulnerabilities;
• no packages with disallowed licenses (for example AGPL);
• supply chain attestations present (SBOM and provenance);
• a non-root default user;
• an up-to-date base image, or an approved base image.

docker scout repo enable --org acme acme/api      # turn on continuous analysis in Hub
docker scout policy acme/api:1.4.2 --org acme     # evaluate the image against policies

The number of repositories you can enable depends on your Docker subscription.

In GitHub Actions. The docker/scout-action action runs the same commands in CI and can post a comparison with the production image as a pull request comment:

- name: Docker Scout
  uses: docker/scout-action@v1
  with:
    command: cves,compare
    image: acme/api:${{ github.sha }}
    to: acme/api:production           # compare with what runs today
    only-severities: critical,high
    exit-code: true                   # fail the job on matching findings

Key takeaways

• Docker Scout analyzes images from an SBOM and is built into the Docker CLI, Desktop, and Hub.
• quickview, cves, recommendations, compare, and sbom cover the daily workflow; --exit-code turns cves into a CI gate.
• It separates base-image findings from your own layers and recommends base image updates.
• Policies express organization rules (vulnerabilities, licenses, attestations, non-root, base images).
• Target prefixes let it analyze local images, registries, archives, source directories, and SBOMs.

🧪 Practice

1. (Beginner) Run docker scout quickview and docker scout recommendations on an image you build. Apply the recommended base image, rebuild, and compare the two versions with docker scout compare.
2. (Intermediate) Run docker scout cves with and without --ignore-base on the same image and explain which team (platform or application) should own each set of findings.
3. Interview question: "Trivy and Docker Scout report different vulnerability counts for the same image. Why, and which is right?" Hint: consider package detection, vulnerability data sources, how backported fixes are handled, and whether an SBOM attestation was used.

Image Signing and Verification

Why it matters. A digest (sha256:...) proves that content has not changed, but not who produced it. Tags are mutable pointers: anyone with push access to a repository, or an attacker who has stolen a CI token, can point acme/api:1.4.2 at different content. Signing binds an identity (a key or a verified account such as a CI workflow) to a specific digest. Verification checks that binding before the image is deployed.

Analogy. A digest is a tamper-evident seal on a package: it shows nothing was changed in transit. A signature is the sender's notarized signature on the seal: it shows who sealed it.

The main tools

Tool	Model	Status
Docker Content Trust (Notary v1)	DOCKER_CONTENT_TRUST=1; tag-based signing	Legacy; Docker is retiring it, including for Docker Official Images, and recommends moving to modern tools
Sigstore Cosign	Key pairs, or keyless signing with short-lived certificates tied to an OIDC identity, recorded in the public Rekor transparency log	Widely adopted, CNCF project
Notation (Notary Project)	X.509 certificates and trust policies; integrates with cloud key services	CNCF project; used by several cloud registries

Cosign and Notation store signatures in the registry as OCI artifacts next to the image, so they travel with it when the image is copied between OCI-compliant registries.

Key-based signing with Cosign

# Create a key pair (cosign.key is encrypted with a password; store it in a secret manager)
cosign generate-key-pair

# Always sign the DIGEST you built, never a tag that might move between build and sign
cosign sign --key cosign.key registry.example.com/acme/api@sha256:4b3c9a...

# Verify: succeeds only if a valid signature for that digest exists
cosign verify --key cosign.pub registry.example.com/acme/api:1.4.2
# Verification for registry.example.com/acme/api:1.4.2 --
# The following checks were performed on each of these signatures:
#   - The cosign claims were validated
#   - The signatures were verified against the specified public key

Keyless signing in CI. Managing long-lived private keys is hard. With keyless signing, the CI job proves its identity through OIDC, Sigstore issues a certificate valid for a few minutes, and the signature is recorded in a transparency log. Verifiers check the identity instead of a key: "this image was signed by the release workflow of acme/api".

# .github/workflows/release.yml (excerpt)
permissions:
  contents: read
  packages: write
  id-token: write            # allows the job to request an OIDC token for keyless signing

jobs:
  release:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: sigstore/cosign-installer@v3
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - id: build
        uses: docker/build-push-action@v6
        with:
          push: true
          tags: ghcr.io/acme/api:${{ github.ref_name }}
      - name: Sign the pushed digest
        run: cosign sign --yes ghcr.io/acme/api@${{ steps.build.outputs.digest }}

# Verify that the image was signed by acme/api's release workflow on a version tag
cosign verify ghcr.io/acme/api:v1.4.2 \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-identity-regexp '^https://github.com/acme/api/\.github/workflows/release\.yml@refs/tags/v'

Verification must be enforced. A signature nobody checks protects nothing, and Docker Engine does not verify Cosign or Notation signatures on docker pull or docker run. Enforcement happens elsewhere:

• Kubernetes admission control: Sigstore policy-controller, Kyverno verifyImages rules, or Ratify with OPA Gatekeeper reject pods whose images are unsigned or signed by the wrong identity.
• Deployment pipelines: run cosign verify (or notation verify) and deploy the verified digest, not the tag, so the image cannot change between check and use.

# Verify, then deploy the exact digest that was verified
DIGEST=$(cosign verify --key cosign.pub registry.example.com/acme/api:1.4.2 --output json \
          | jq -r '.[0].critical.image."docker-manifest-digest"')
docker run -d registry.example.com/acme/api@"$DIGEST"

Key takeaways

• Digests prove integrity; signatures prove origin. Tags prove neither.
• Docker Content Trust (Notary v1) is legacy; Cosign and Notation are the modern options.
• Keyless Cosign signing ties an image to a CI identity without long-lived keys, backed by a transparency log.
• Always sign and deploy by digest.
• Signatures only help when verification is enforced, typically by admission controllers or deployment pipelines.

🧪 Practice

1. (Beginner) Push an image to a test registry, sign it with a Cosign key pair, and verify it. Then push a different image to the same tag and show that verification fails.
2. (Intermediate) Add keyless signing to a GitHub Actions workflow and write the cosign verify command that accepts only signatures from that workflow on tags matching v*.
3. Interview question: "If our registry requires authentication and only CI can push, why do we still need image signatures?" Hint: think about compromised CI tokens, registry or mirror compromise, images copied between registries, and proving origin to downstream consumers.

Software Bill of Materials (SBOM)

Why it matters. When a critical vulnerability such as Log4Shell (2021) or the xz backdoor (2024) is announced, the first question is "where are we affected?" Without an inventory, answering means rescanning or manually inspecting every image. A Software Bill of Materials is that inventory: a machine-readable list of every component in an artifact, with names, versions, package identifiers (purls), licenses, and hashes. SBOMs are also increasingly expected by customers and regulators, for example in US federal procurement and under the EU Cyber Resilience Act.

Analogy. An SBOM is the ingredient list on a food package. When a supplier recalls an ingredient, you check the labels instead of testing every product in the warehouse.

Formats

Format	Maintained by	Notes
SPDX	Linux Foundation (ISO/IEC 5962)	Strong on licensing; used by BuildKit SBOM attestations
CycloneDX	OWASP	Strong on security use cases; supports VEX

Both are JSON (or other serializations) and well supported by scanners. An abbreviated SPDX entry:

{
  "SPDXID": "SPDXRef-Package-deb-openssl",
  "name": "openssl",
  "versionInfo": "3.0.15-1~deb12u1",
  "licenseConcluded": "Apache-2.0",
  "externalRefs": [
    {
      "referenceCategory": "PACKAGE-MANAGER",
      "referenceType": "purl",
      "referenceLocator": "pkg:deb/debian/openssl@3.0.15-1~deb12u1?arch=amd64&distro=debian-12"
    }
  ]
}

Generating SBOMs

# 1. At build time, as an attestation attached to the image (BuildKit)
docker buildx build --sbom=true -t registry.example.com/acme/api:1.4.2 --push .

# Read it back from the registry
docker buildx imagetools inspect registry.example.com/acme/api:1.4.2 \
  --format '{{ json .SBOM.SPDX }}' | jq '.packages | length'
# 214

# 2. From an existing image, with Syft
syft registry.example.com/acme/api:1.4.2 -o spdx-json=sbom.spdx.json
syft registry.example.com/acme/api:1.4.2 -o cyclonedx-json=sbom.cdx.json

# 3. With Docker Scout
docker scout sbom --format spdx registry.example.com/acme/api:1.4.2 > sbom.spdx.json

BuildKit scans the final image by default. Components that only existed in build stages (compilers, build tools) or in the build context are not listed unless you ask for them:

# syntax=docker/dockerfile:1
# Also scan the build context (source manifests such as package-lock.json)
ARG BUILDKIT_SBOM_SCAN_CONTEXT=true

FROM golang:1.25 AS build
# Also scan this stage, so the toolchain used to compile appears in the SBOM
ARG BUILDKIT_SBOM_SCAN_STAGE=true
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 go build -o /out/api .

FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /out/api /api
ENTRYPOINT ["/api"]

Where attestations live. BuildKit stores SBOM and provenance attestations in the image index alongside the platform images (they appear with platform unknown/unknown). They are kept when you push to a registry or export to an OCI layout. With the classic local image store, --load drops them; the containerd image store keeps them.

Using SBOMs

# Rescan an SBOM for new vulnerabilities without pulling the image again
grype sbom:./sbom.spdx.json

# "Are we affected?" across many stored SBOMs
grep -l '"name": "xz-utils"' sboms/*.spdx.json

At scale, platforms such as OWASP Dependency-Track ingest SBOMs from every build and alert when a newly published vulnerability matches a component.

Key takeaways

• An SBOM is a machine-readable inventory of an artifact's components: names, versions, purls, licenses, hashes.
• SPDX and CycloneDX are the two standard formats.
• Generate SBOMs at build time with --sbom=true (attached as an attestation), or afterwards with Syft or Docker Scout.
• Use BUILDKIT_SBOM_SCAN_CONTEXT and BUILDKIT_SBOM_SCAN_STAGE to include build context and build stages.
• Store SBOMs centrally so you can answer "where are we affected?" in minutes and rescan without rebuilding.

🧪 Practice

1. (Beginner) Generate SBOMs for one image with Syft in SPDX and CycloneDX formats and compare the package counts and structure.
2. (Intermediate) Build and push a multi-stage image with --sbom=true, inspect the attestation with imagetools, then add BUILDKIT_SBOM_SCAN_STAGE=true to the build stage and identify what new components appear.
3. Interview question: "What is the difference between scanning an image and having an SBOM for it?" Hint: consider when each is produced, whether build-time information is available, reuse when new CVEs are published, and use beyond vulnerabilities (licenses, inventory).

Build Provenance and Attestations

Why it matters. Signing proves who published an image. It does not tell you how it was built: from which repository and commit, with which build definition, on which builder, with which inputs. Several major supply-chain attacks, such as the 2020 SolarWinds compromise, injected malicious code in the build system rather than in the source. Provenance is a verifiable record of how an artifact was produced, so you can check that an image came from the expected source, through the expected pipeline.

Attestations in general. An attestation is a statement about an artifact, in the in-toto format:

 in-toto statement
 ├── subject        which artifact this is about (name + sha256 digest)
 ├── predicateType  what kind of claim it is, e.g.
 │                  https://slsa.dev/provenance/v0.2   (provenance)
 │                  https://spdx.dev/Document          (SBOM)
 └── predicate      the claim itself (build details, SBOM content, test results, ...)

SBOMs, provenance, vulnerability scan results, and test reports can all be expressed as attestations attached to an image.

SLSA. Supply-chain Levels for Software Artifacts (SLSA, pronounced "salsa") is a framework that grades how trustworthy provenance is:

SLSA Build level	Requirement (simplified)	What it protects against
L1	Provenance exists and describes the build	Mistakes; gives visibility
L2	Built on a hosted build platform that generates and signs the provenance	Tampering after the build
L3	Hardened build platform; builds are isolated and provenance cannot be forged by the build itself	Tampering during the build

Provenance with BuildKit

# mode=min (the default when attestations are supported): basic build information
# mode=max: adds full build parameters, Dockerfile source, and detailed materials
docker buildx build --provenance=mode=max --sbom=true \
  -t ghcr.io/acme/api:1.4.2 --push .

# Read the provenance back
docker buildx imagetools inspect ghcr.io/acme/api:1.4.2 \
  --format '{{ json .Provenance.SLSA }}' \
  | jq '{builder: .builder.id, source: .invocation.configSource, materials: [.materials[].uri]}'
# {
#   "builder": "https://github.com/acme/api/actions/runs/123456789",
#   "source": { "uri": "https://github.com/acme/api.git#refs/tags/v1.4.2",
#               "digest": { "sha1": "9f2c1e..." }, "entryPoint": "Dockerfile" },
#   "materials": [ "pkg:docker/python@3.13-slim?platform=linux%2Famd64", ... ]
# }

With mode=max the provenance includes build arguments, which is another reason never to pass secrets as build args (see Avoiding Secrets in Images).

Making provenance verifiable. BuildKit attestations are attached to the image but are not signed by BuildKit itself, so on their own they reach roughly SLSA Build L1: anyone who can push can also push fake provenance. To make them trustworthy, have the build platform sign them:

# GitHub Actions: signed SLSA provenance stored in GitHub and pushed to the registry
permissions:
  contents: read
  packages: write
  id-token: write          # identity for signing
  attestations: write      # store the attestation

steps:
  - id: build
    uses: docker/build-push-action@v6
    with:
      push: true
      tags: ghcr.io/acme/api:${{ github.ref_name }}
  - uses: actions/attest-build-provenance@v2
    with:
      subject-name: ghcr.io/acme/api
      subject-digest: ${{ steps.build.outputs.digest }}
      push-to-registry: true

# Verify that the image was built by a workflow in the acme organization
gh attestation verify oci://ghcr.io/acme/api:v1.4.2 --owner acme

Cosign can create and verify arbitrary signed attestations in the same way:

# Attach a signed SBOM attestation, then verify it
cosign attest --key cosign.key --type spdxjson --predicate sbom.spdx.json \
  ghcr.io/acme/api@sha256:4b3c9a...
cosign verify-attestation --key cosign.pub --type spdxjson ghcr.io/acme/api:v1.4.2

Admission controllers (Kyverno, Sigstore policy-controller) can then require, for example, "signed provenance showing the image was built from github.com/acme/* on a protected branch" before a pod may run.

Key takeaways

• Provenance records how an artifact was built: source, commit, build definition, builder, and inputs.
• Attestations are in-toto statements (subject, predicate type, predicate); SBOMs and provenance are both attestations.
• BuildKit produces provenance with --provenance=mode=min|max, stored in the image index next to the image.
• Unsigned provenance is informational; signed provenance from a hosted build platform (for example GitHub artifact attestations or Cosign) makes it verifiable.
• SLSA levels describe how much tampering the provenance can resist.

🧪 Practice

1. (Beginner) Build and push an image with --provenance=mode=max, then use imagetools inspect to find the source repository, commit, and base images recorded in it.
2. (Intermediate) Add actions/attest-build-provenance to a workflow, push an image, and verify it with gh attestation verify. Then try to verify with a different --owner and explain the failure.
3. Interview question: "What does SLSA Build Level 3 protect against that signing the final image does not?" Hint: think about where the signature is created, what a compromised build step could do before signing, and who generates the provenance.

Trusted Base Images

Why it matters. The base image is usually the largest part of your image and the source of most of its packages and vulnerabilities. Choosing FROM is choosing whom to trust: its publisher, their build process, and how fast they ship security fixes. It also decides how much an attacker finds once inside: a shell, a package manager, and network tools, or almost nothing.

Base image options

Option	Contents	Strengths	Trade-offs
Docker Official Images (python, node, nginx, ...)	Full or slim distribution + runtime	Reviewed, documented, rebuilt regularly	Larger; many packages you may not need
Verified Publisher images	Vendor-maintained	Published by the software's vendor	Quality varies by vendor
Docker Hardened Images	Minimal, continuously patched	Low CVE counts, SBOMs, provenance, signatures	Different tags and conventions from the official images
Distroless (gcr.io/distroless/*)	Runtime + CA certificates, no shell or package manager	Very small attack surface; :nonroot variants	Harder to debug; use :debug variants or debug containers
Chainguard Images (cgr.dev/chainguard/*)	Minimal, frequently rebuilt	Very low CVE counts, signed, SBOMs	Free tier limited to some images and tags
Red Hat UBI	RHEL-based, redistributable	Enterprise support, RPM ecosystem	Larger
scratch	Nothing at all	The smallest possible image	Only for static binaries; you must add CA certificates, time zones, and users yourself

Practices

1. Prefer minimal images for production. Use a full image for building, and a slim, distroless, or hardened image for running (multi-stage builds).
2. Pin by digest, and automate updates. A tag such as python:3.13-slim can change under you; a digest cannot. Pinning without updates, however, freezes vulnerabilities in place, so pair it with Renovate or Dependabot.
3. Rebuild regularly. Fixes in a base image reach you only when you rebuild. Schedule rebuilds (for example weekly) even when your code has not changed.
4. Verify the base. Many minimal-image publishers sign their images; verify signatures and provenance before adopting a new base, following the identity the vendor documents.
5. Control the source. Pull through a private registry or proxy cache, and restrict which registries can be used: Docker Business offers Registry Access Management and Image Access Management, and Kubernetes admission policies can allow only approved registries.
6. Maintain golden images. Larger organizations often publish internal base images (approved distribution, CA certificates, non-root user, monitoring agent) that all teams build on.

A pinned, minimal, non-root image

# syntax=docker/dockerfile:1
# Build stage: full toolchain, pinned by digest (Renovate updates tag and digest together)
FROM golang:1.25@sha256:<digest> AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/api .

# Runtime stage: no shell, no package manager, runs as UID 65532
FROM gcr.io/distroless/static-debian12:nonroot@sha256:<digest>
COPY --from=build /out/api /api
USER nonroot:nonroot
ENTRYPOINT ["/api"]

Automating base image updates

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": ["config:recommended", "docker:pinDigests"]
}

# .github/dependabot.yml
version: 2
updates:
  - package-ecosystem: docker
    directory: /
    schedule:
      interval: weekly

# Ask Scout which base image would reduce vulnerabilities the most
docker scout recommendations acme/api:1.4.2

Key takeaways

• The base image decides most of your packages, vulnerabilities, and post-compromise attack surface.
• Prefer minimal production bases: slim, distroless, hardened, or scratch for static binaries.
• Pin by digest and automate updates with Renovate or Dependabot; rebuild on a schedule.
• Verify signatures and provenance of base images, and restrict which registries images may come from.
• Golden internal base images standardize users, certificates, and patching across teams.

🧪 Practice

1. (Beginner) Build the same Go or Node.js app on a full official image, a slim image, and a distroless image. Compare size and Trivy results, and check whether docker exec <container> sh works in each.
2. (Intermediate) Pin all FROM lines in a repository by digest and configure Renovate or Dependabot to update them. Explain how you ensure updated images are actually rebuilt and redeployed.
3. Interview question: "Distroless images have no shell. How do you debug a production container built on one?" Hint: look at :debug image variants, docker debug, ephemeral debug containers in Kubernetes, and sharing the target's namespaces from a separate tools container.

<a id="104-secrets-management"></a>

10.4 Secrets Management

Passwords, API tokens, private keys, and certificates are the most valuable things an attacker can find in a container or an image. This section covers how secrets leak into images, how to pass them safely to builds and to running containers, and how to connect containers to dedicated secret management systems.

Avoiding Secrets in Images

Why it matters. An image is a distributable artifact. It is pushed to registries, cached on CI runners, pulled to laptops, mirrored, and sometimes made public by mistake. Anything inside it should be considered readable by anyone who can pull it. Secrets end up in images far more easily than most people expect, because images keep every layer and a lot of metadata.

Analogy. Image layers are like a stack of transparent sheets. Writing a password on one sheet and covering it with a "deleted" sticker on the next sheet hides it when you look from the top, but anyone can lift the sticker and read it.

Where secrets leak

Mistake	Where the secret ends up
COPY . . with a .env, id_rsa, or .aws/ directory in the context	A layer of the image
COPY secret.json then RUN rm secret.json	Still in the earlier layer; deletion only hides it
ARG TOKEN used in RUN	Image history (docker history), build logs, mode=max provenance
ENV API_KEY=...	Image config, visible with docker inspect and in every container
RUN git clone https://user:token@host/repo.git	.git/config inside the layer, plus history
RUN npm config set //registry/:_authToken=...	A .npmrc file written into the layer
.git directory copied into the image	Entire repository history, including secrets committed and later "removed"
Registry cache exported with mode=max	Intermediate build-stage layers pushed to the cache registry

Seeing a "deleted" secret. This shows why deleting a file in a later instruction does not help:

# syntax=docker/dockerfile:1
FROM alpine:3.20
# Layer N contains the key
COPY deploy_key /root/.ssh/deploy_key
# Layer N+1 only records that the file was deleted
RUN echo "using the key..." && rm /root/.ssh/deploy_key

docker build -t leaky:1 .
docker run --rm leaky:1 ls /root/.ssh       # empty: looks clean from the top

# But the layer that added it is still part of the image
docker save leaky:1 -o leaky.tar
mkdir leaky && tar -xf leaky.tar -C leaky
for blob in leaky/blobs/sha256/*; do
  tar -tf "$blob" 2>/dev/null | grep -q 'root/.ssh/deploy_key' && echo "found in $blob"
done
# found in leaky/blobs/sha256/5e1f...

# Build arguments are visible in the history of the RUN steps that used them
# (leaky-args:1 is a variant built with "ARG TOKEN" and "RUN npm ci" using it)
docker history --no-trunc leaky-args:1 | grep TOKEN
# |1 TOKEN=ghp_abc123... /bin/sh -c npm ci

Prevention

1. Keep secrets out of the build context with .dockerignore:

   # .dockerignore
   .git
   .env
   .env.*
   *.pem
   *.key
   id_rsa*
   .aws/
   .ssh/
   .npmrc
   secrets/
   

2. Never use ARG or ENV for secrets. BuildKit's build checks warn about this: the SecretsUsedInArgOrEnv check flags ARG and ENV names that look like secrets (TOKEN, PASSWORD, SECRET, ...).

3. Use secret mounts for build-time credentials (next topic) and runtime injection for application secrets (the topic after that).

4. Do not rely on multi-stage builds alone. The final image may be clean, but the build stage still exists in the builder's cache and may be exported with --cache-to ... mode=max.

5. Scan for secrets in source code and images:

   trivy fs --scanners secret .                   # the repository and build context
   trivy image --scanners secret acme/api:1.4.2   # every layer of the built image
   

If a secret leaked: rotate it. Deleting the tag or rebuilding the image does not recall copies already pulled, cached, or mirrored. Treat the secret as compromised, revoke and replace it, then fix the build.

Key takeaways

• Everything in any layer, the image config, or the history is readable by anyone who can pull the image.
• Deleting a file in a later layer does not remove it; ARG values appear in history and provenance; ENV is in the config.
• Use .dockerignore, never ARG/ENV for secrets, and secret mounts for build credentials.
• Multi-stage builds do not protect build-stage secrets from caches; scan images and repositories for secrets.
• A leaked secret must be rotated, not just removed from the image.

🧪 Practice

1. (Beginner) Build the leaky example with a dummy key and recover it from the saved image. Then build a variant that passes a fake token through ARG and find it with docker history --no-trunc.
2. (Intermediate) Run trivy image --scanners secret against both leaky images and against one of your real images. Write a .dockerignore that would have prevented each finding.
3. Interview question: "A developer says the image is safe because the final stage of the multi-stage build does not contain the token. Is that enough?" Hint: think about which stages are cached and where, --cache-to modes, build logs, provenance, and how the token got into the build stage.

Build-Time Secrets

Why it matters. Builds legitimately need credentials: a token for a private package registry, an SSH key for a private Git dependency, cloud credentials to download an artifact. The goal is for the credential to be available to exactly one RUN instruction and to leave no trace in layers, history, cache keys, or metadata. BuildKit's secret and SSH mounts do exactly that; their mechanics are covered in Secret Mounts and SSH Mounts. This topic focuses on using them safely end to end.

How a secret mount behaves

 client                     BuildKit                         RUN container
 ------                     --------                         -------------
 --secret id=npmrc,  --->   requests the value over    --->  /root/.npmrc on a tmpfs,
   src=~/.npmrc             the session when the step        present only while this
                            runs; never stored in a          RUN executes
                            layer or the cache key
                                                             after RUN: gone, layer has
                                                             no trace of it

Patterns

# syntax=docker/dockerfile:1
FROM node:22-alpine AS deps
WORKDIR /app
COPY package.json package-lock.json ./

# Secret as a file, mounted where the tool expects it
RUN --mount=type=secret,id=npmrc,target=/root/.npmrc \
    npm ci --omit=dev

FROM python:3.13-slim AS pydeps
# Secret exposed as an environment variable for this RUN only
RUN --mount=type=secret,id=pip_index_url,env=PIP_INDEX_URL \
    pip install --no-cache-dir private-package==2.3.0

FROM golang:1.25 AS godeps
# Private Git dependencies through the forwarded SSH agent (no key file in the build)
RUN mkdir -p -m 0700 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts
ENV GOPRIVATE=github.com/acme/*
RUN --mount=type=ssh \
    git config --global url."git@github.com:".insteadOf "https://github.com/" \
 && go mod download

# Provide the secrets from a file, an environment variable, and the SSH agent
docker buildx build \
  --secret id=npmrc,src=$HOME/.npmrc \
  --secret id=pip_index_url,env=PIP_INDEX_URL \
  --ssh default \
  -t acme/app:dev .

In Compose

services:
  app:
    build:
      context: .
      secrets:
        - npmrc                    # made available to RUN --mount=type=secret,id=npmrc

secrets:
  npmrc:
    file: ~/.npmrc                 # or: environment: NPM_RC

In GitHub Actions

- uses: docker/build-push-action@v6
  with:
    push: true
    tags: ghcr.io/acme/app:${{ github.sha }}
    secrets: |
      npm_token=${{ secrets.NPM_TOKEN }}

Pitfalls that still leak

Pitfall	Example	Fix
The command prints the secret	cat /run/secrets/token, set -x, verbose tool output	Keep commands quiet; review --progress=plain logs
The tool persists the secret	npm config set ..., git clone https://token@..., docker login inside RUN	Mount config files with target=; use SSH mounts or credential helpers that read from the mount
The secret is copied to a normal path	RUN cp /run/secrets/key /app/key	Use the mount path directly within the same RUN
Expecting cache invalidation on rotation	Secret contents are not part of the cache key, so changing the token does not rerun the step	Fine for credentials; if the output depends on the secret, bust the cache explicitly (--no-cache-filter <stage>)
Secrets in build args "just for CI"	--build-arg TOKEN=$TOKEN	Replace with --secret id=token,env=TOKEN

Verifying a build is clean

docker history --no-trunc acme/app:dev | grep -iE 'token|password|secret' || echo "history clean"
trivy image --scanners secret acme/app:dev

Key takeaways

• Secret mounts expose a credential to a single RUN on a tmpfs, never in layers, history, or cache keys.
• Use target= for tools that read config files, env= for tools that read environment variables, and --mount=type=ssh for Git over SSH.
• Compose (build.secrets) and CI actions (secrets: input) pass secrets to BuildKit the same way.
• Secrets still leak if commands print them, tools persist them, or they are copied to normal paths.
• Secret contents do not affect the cache key; verify finished images with history checks and secret scanning.

🧪 Practice

1. (Beginner) Convert a Dockerfile that uses ARG NPM_TOKEN into one that uses a secret mount. Confirm that docker history --no-trunc no longer shows the token.
2. (Intermediate) Build a Go service that depends on a private GitHub module using --ssh default, both locally and in GitHub Actions. Explain how the CI job provides the SSH agent or a token without writing it to the image.
3. Interview question: "Why are secret contents excluded from BuildKit's cache key, and when could that cause a problem?" Hint: consider what would happen to the cache and to secrecy if they were included, and builds whose output depends on the secret value.

Runtime Secrets

Why it matters. Running applications need secrets too: database passwords, API keys, TLS private keys. How they are delivered decides how many places they can leak from. The most common method, environment variables, is also one of the leakiest.

Why environment variables leak

• docker inspect shows them to anyone with access to the Docker API.
• They are readable from /proc/<pid>/environ by processes with sufficient access.
• Every child process inherits them, including shells, scripts, and third-party tools.
• Crash reporters, debug pages, and "print the configuration" logging often dump the whole environment.
• They end up in shell history, CI logs, and Compose files committed to Git.

docker run -d --name db -e POSTGRES_PASSWORD=s3cret postgres:17
docker inspect --format '{{json .Config.Env}}' db
# ["POSTGRES_PASSWORD=s3cret","PATH=/usr/local/sbin:...", ...]

Prefer files. Delivering a secret as a file, mounted read-only (ideally from memory), avoids most of these channels: it is not inherited by child processes, not shown by docker inspect, and not part of the image. Many official images support a _FILE convention: instead of POSTGRES_PASSWORD, set POSTGRES_PASSWORD_FILE to a path, and the entrypoint reads the value from the file.

Compose secrets

services:
  db:
    image: postgres:17
    environment:
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password   # read from file, not env
    secrets:
      - db_password

  api:
    image: acme/api:1.4.2
    environment:
      DB_PASSWORD_FILE: /run/secrets/db_password
    secrets:
      - db_password                                      # only services that need it

  worker:
    image: acme/worker:1.4.2                             # no access to db_password

secrets:
  db_password:
    file: ./secrets/db_password.txt                      # git-ignored; or environment: DB_PASSWORD

Each listed secret appears in the container at /run/secrets/<name>. In plain Compose this is a read-only bind mount of the source file, so protect the file on the host (chmod 600, git-ignored). In Swarm mode, secrets are encrypted in the cluster store, sent only to nodes running tasks that need them, and mounted on an in-memory filesystem (see Secrets and Configs in Compose and Swarm Secrets and Configs).

Reading secrets in the application. Support both forms, preferring the file, so the same image works locally and in production:

import os
from pathlib import Path


def read_secret(name: str) -> str:
    """Return a secret from NAME_FILE (a mounted file) or, as a fallback, NAME."""
    file_path = os.environ.get(f"{name}_FILE")
    if file_path:
        return Path(file_path).read_text().strip()   # strip the trailing newline
    value = os.environ.get(name)
    if value is None:
        raise RuntimeError(f"secret {name} is not configured")
    return value


DB_PASSWORD = read_secret("DB_PASSWORD")

Without Compose. With docker run, mount a file from a protected host location, ideally one backed by tmpfs:

docker run -d --name api \
  --mount type=bind,src=/run/app-secrets/db_password,dst=/run/secrets/db_password,readonly \
  -e DB_PASSWORD_FILE=/run/secrets/db_password \
  acme/api:1.4.2

--env-file keeps values out of shell history and process listings on the host, but they are still environment variables inside the container.

Delivery methods compared

Method	Visible in docker inspect	Inherited by child processes	At rest	Rotation
-e / environment:	Yes	Yes	In Compose files, scripts, history	Restart the container
--env-file	Yes	Yes	In a file on the host	Restart the container
Bind-mounted file / Compose secrets	No	No	In a file on the host	Replace the file; app re-reads or restarts
Swarm secrets	No	No	Encrypted in the cluster store	New secret version + service update
Fetched from a secret manager at runtime	No	Depends on how the app stores it	In the secret manager	Can be automatic (next topic)

Operational habits. Give each service only the secrets it needs, never log configuration objects without redacting them, avoid docker exec ... env in shared terminals or recordings, and plan rotation from the start (an app that re-reads the file, or a restart procedure).

Key takeaways

• Environment variables leak through docker inspect, /proc, child processes, logs, and crash reports.
• Prefer read-only secret files under /run/secrets, with the _FILE convention many images support.
• Compose secrets mount files per service; Swarm secrets add encryption at rest and in-memory delivery.
• Write applications to read NAME_FILE first and fall back to NAME.
• Scope secrets to the services that need them, and design for rotation.

🧪 Practice

1. (Beginner) Start PostgreSQL once with POSTGRES_PASSWORD and once with POSTGRES_PASSWORD_FILE via Compose secrets. Compare what docker inspect reveals in each case.
2. (Intermediate) Modify a small web app to use the read_secret pattern, deploy it with Compose secrets, then rotate the password: update the file, update the database user, and restart only what is necessary. Write down the order of steps that avoids downtime.
3. Interview question: "Why are files generally safer than environment variables for secrets, and what risks remain with files?" Hint: compare inheritance, inspection, and logging channels, then think about host file permissions, backups, and an attacker with code execution inside the container.

External Secret Managers

Why they exist. Files and environment variables answer how a secret reaches a container, but not where it lives, who may read it, how it is rotated, or who accessed it last Tuesday. A secret manager is a dedicated service that stores secrets encrypted, enforces fine-grained access policies, logs every access, and can rotate secrets or even generate short-lived ones on demand.

Analogy. Taping keys under the doormat (environment variables in a Compose file) versus a key cabinet with a guard who checks ID, writes every checkout in a log, and changes the locks on a schedule.

Common options

Manager	Typical environment	Notable features
HashiCorp Vault / OpenBao	Self-hosted, multi-cloud	Dynamic secrets (database, cloud credentials), PKI, leases, agent
AWS Secrets Manager / SSM Parameter Store	AWS (ECS, EKS, Lambda)	IAM policies, automatic rotation, native ECS integration
Azure Key Vault	Azure	Managed identities, keys and certificates
Google Secret Manager	Google Cloud	IAM, versioning, workload identity
1Password, Doppler, Infisical	Teams, developer workflows	CLI injection for local development and CI

Integration patterns

 (1) App fetches at startup            (2) Entrypoint wrapper
 +-----------+   SDK call   +-----+    +-----------------------+        +-----+
 | app       | -----------> | SM  |    | entrypoint: fetch ->  | -----> | SM  |
 +-----------+              +-----+    | write /run/secrets -> |        +-----+
                                       | exec app              |
                                       +-----------------------+

 (3) Agent / sidecar renders files     (4) Platform injection
 +-----------+   +--------+            +-------------------+  task/pod spec   +-----+
 | app reads |<--| agent  |---> SM     | ECS / Kubernetes  | references -----> | SM  |
 | /secrets  |   | renews |            | injects env/files |  secret ARN/name  +-----+
 +-----------+   +--------+            +-------------------+

Pattern	Pros	Cons
SDK in the app	Can re-fetch and react to rotation	Couples app code to one provider
Entrypoint wrapper	No app changes	Usually fetches once at startup
Agent or sidecar (e.g. Vault Agent)	Renews leases and rewrites files automatically	Extra process or container to operate
Platform injection (ECS secrets, External Secrets Operator, Secrets Store CSI)	Declarative; app stays generic	Tied to the platform; often still env vars

The bootstrap problem ("secret zero"). To fetch secrets, a container must first prove who it is. If that proof is a static token baked into the image or environment, you have just moved the problem. Use identities the platform provides: IAM roles for ECS tasks, workload identity in Kubernetes and on cloud VMs, OIDC tokens in CI, or Vault's platform auth methods (Kubernetes, AWS, Azure, GCP).

Entrypoint wrapper example (AWS)

#!/bin/sh
# docker-entrypoint.sh: fetch secrets with the task's IAM role, then start the app
set -eu

umask 077                                    # secret files readable by owner only
mkdir -p /run/secrets                        # mount a tmpfs here (--tmpfs /run/secrets)

aws secretsmanager get-secret-value \
  --secret-id "prod/api/db-password" \
  --query SecretString --output text > /run/secrets/db_password

export DB_PASSWORD_FILE=/run/secrets/db_password
exec "$@"                                    # replace the shell: the app becomes PID 1

Platform injection example (Amazon ECS task definition excerpt)

{
  "containerDefinitions": [
    {
      "name": "api",
      "image": "123456789012.dkr.ecr.eu-west-1.amazonaws.com/acme/api@sha256:4b3c9a...",
      "secrets": [
        {
          "name": "DB_PASSWORD",
          "valueFrom": "arn:aws:secretsmanager:eu-west-1:123456789012:secret:prod/api/db-password"
        }
      ]
    }
  ]
}

ECS resolves the reference with the task execution role at startup, so no credential appears in the task definition or the image.

Dynamic secrets. Some managers create credentials on demand, valid for minutes or hours, instead of storing one long-lived password:

# Vault database secrets engine: each call returns a new, short-lived database user
vault read database/creds/api-role
# Key                Value
# lease_id           database/creds/api-role/9xKq...
# lease_duration     1h
# password           A1a-Zk3...
# username           v-api-role-8f2d...

A leaked dynamic credential expires on its own, and every credential maps to exactly one consumer in the audit log.

Local development without plaintext files. Secret manager CLIs can inject values into a command's environment just for its lifetime, so .env files never contain real secrets:

# .env.tpl contains references, safe to commit:  DB_PASSWORD=op://Dev/api-db/password
op run --env-file=.env.tpl -- docker compose up

Key takeaways

• Secret managers add encrypted storage, access policies, audit logs, rotation, and dynamic credentials.
• Integrate through an SDK, an entrypoint wrapper, an agent or sidecar, or platform-native injection.
• Solve "secret zero" with platform identities (IAM roles, workload identity, OIDC), not static tokens.
• Dynamic, short-lived credentials limit the impact of a leak and improve auditability.
• CLI injection keeps real secrets out of .env files in local development and CI.

🧪 Practice

1. (Beginner) Run a Vault dev server in a container (hashicorp/vault with server -dev), store a secret with vault kv put, and read it from a second container using the API and a token.
2. (Intermediate) Write an entrypoint wrapper that fetches a secret from Vault (or a cloud secret manager) into a tmpfs-backed /run/secrets and starts the app with exec. Run the container with --read-only --tmpfs /run/secrets and confirm the secret is not visible in docker inspect.
3. Interview question: "How does a container authenticate to a secret manager without already holding a secret?" Hint: consider identities issued by the platform (IAM task roles, Kubernetes service account tokens, cloud metadata, CI OIDC tokens) and how the secret manager verifies them.

---

<a id="11-resource-management-and-performance"></a>