13. Docker in Development and CI/CD

The earlier chapters treated Docker mainly as a way to package and run software. This chapter is about the two places where most engineers actually work with it every day: the inner loop on a developer's machine (edit, run, debug, test) and the outer loop in a CI/CD pipeline (build, test, scan, tag, publish). When both loops use the same images and the same definitions, "works on my machine" turns into "works in the pipeline, therefore works in production", and that is the main reason teams adopt containers in the first place.

<a id="131-development-workflows"></a>

13.1 Development Workflows

This section covers how to make containers pleasant to develop in: putting the whole toolchain in a container, describing it as a reproducible dev container, keeping the edit-and-see loop fast, attaching a debugger, and starting real dependencies from inside tests. It builds on Bind Mounts for Development, Compose Watch for Development, and Local Development Environments.

Containerized Development Environments

Why it matters. A new developer joins the team. The README says "install Python 3.13, PostgreSQL 16, Redis, the AWS CLI, and protoc 28". Two days later they have a working setup, except it uses Python 3.12 from the system package manager, and one test fails only on their machine. A containerized development environment replaces that list with a definition that the computer executes: the toolchain, the services, and their versions live in files in the repository, so every developer, and CI, gets the same environment from the same commit.

Analogy. A traditional setup is like asking each cook to buy their own ingredients from different shops based on a handwritten list. A containerized environment is a meal kit: the same pre-measured ingredients arrive in every kitchen, and if the recipe changes, the kit changes with it.

How much to put in a container. There is a spectrum, and each step trades some convenience for more reproducibility:

Level	What runs in containers	What runs on the host	Typical tool
1. Services only	Databases, caches, brokers	App, toolchain, editor	docker compose up db redis
2. App and services	App process plus services	Editor; code shared via bind mount or sync	Compose with a dev build target
3. Full toolchain	Compilers, linters, CLIs, test runners, app, services	Only the editor UI (attached to the container)	Dev Containers
4. Remote environment	Everything, on a remote machine	A browser or a thin editor client	GitHub Codespaces, cloud workstations

Level 1 is the easiest place to start and already removes the most painful installs. Level 3 removes "install the toolchain" from onboarding entirely.

Pattern 1: one-off toolchain commands. You can use an image as a disposable tool without installing anything on the host. The current directory is mounted, the command runs, and the container disappears:

# Run the Go test suite with an exact Go version, without installing Go
#   -v "$PWD":/src -w /src      -> mount the project and work inside it
#   -v go-mod-cache:/go/pkg/mod -> keep downloaded modules between runs
docker run --rm \
  -v "$PWD":/src -w /src \
  -v go-mod-cache:/go/pkg/mod \
  golang:1.25 go test ./...

# Format code with a pinned tool version, writing files as your own user (Linux)
docker run --rm -u "$(id -u):$(id -g)" \
  -v "$PWD":/work -w /work \
  node:22-alpine npx --yes prettier@3.3.3 --write "src/**/*.ts"

Wrapping these commands in a Makefile or a small script gives the team one stable entry point (make test, make lint) that works the same way locally and in CI.

Pattern 2: a dev stage in the production Dockerfile. Instead of a separate Dockerfile.dev that drifts away from production, add a dev target to the multi-stage Dockerfile. Both targets share the same base, so the operating system, runtime version, and system libraries are identical:

# syntax=docker/dockerfile:1
FROM python:3.13-slim AS base
WORKDIR /app
ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1
COPY requirements.txt .
RUN --mount=type=cache,target=/root/.cache/pip pip install -r requirements.txt

# ---- development: extra tools, runs as the developer's UID ----
FROM base AS dev
ARG UID=1000
ARG GID=1000
RUN groupadd -g "$GID" dev && useradd -m -u "$UID" -g "$GID" dev
COPY requirements-dev.txt .
RUN --mount=type=cache,target=/root/.cache/pip pip install -r requirements-dev.txt   # pytest, debugpy, ruff
USER dev
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--reload"]   # code arrives via bind mount or watch

# ---- production: code baked in, no dev tools ----
FROM base AS runtime
COPY src/ ./src/
RUN useradd -r -u 10001 app
USER app
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0"]

# Build the dev image with your own UID so files created in the container belong to you
docker build --target dev --build-arg UID="$(id -u)" --build-arg GID="$(id -g)" -t api:dev .

Pattern 3: the whole team's stack in Compose. Levels 1 and 2 are usually expressed as a compose.yaml with a development override; the full pattern, including seed data, profiles, and daily commands, is described in Local Development Environments.

Things that go wrong

Problem	Why	Remedy
Root-owned files in the repository (Linux)	The container writes to a bind mount as root	Run as your UID (-u, a UID build arg, or Dev Containers' automatic UID update)
Slow file access on macOS and Windows	Files cross the Docker Desktop VM boundary	Named volumes for dependencies; Compose Watch; on WSL2 keep code in the Linux filesystem
Apple silicon laptop, amd64 production	Different CPU architecture, some images or binaries are missing	Use multi-architecture images; test the production architecture in CI; --platform linux/amd64 only when needed (emulation is slow)
Tool versions drift between developers	Images pulled by a floating tag at different times	Pin image tags (or digests) in the repository; rebuild periodically on purpose
Environment works locally, fails in CI	CI uses a different setup path	Have CI run the same targets and Compose files developers use

Key takeaways

• A containerized dev environment turns setup instructions into versioned, executable definitions in the repository.
• Start by containerizing services (level 1); move the toolchain into containers when onboarding and drift become painful.
• Use images as disposable tools (docker run --rm -v "$PWD":...) to avoid installing toolchains on the host.
• Put a dev target in the production multi-stage Dockerfile so development and production share one base.
• Watch out for file ownership, file-sharing speed on Docker Desktop, and CPU architecture differences.

🧪 Practice

1. (Beginner) Pick a project you work on and run its test suite with docker run --rm -v "$PWD":/src -w /src <language image> <test command>, without installing the language runtime on your host. Add a named volume for the dependency cache and compare the second run's time.
2. (Intermediate) Add a dev target to an existing multi-stage Dockerfile that installs development tools and runs as a UID passed with a build argument. Verify that files created in a bind-mounted directory from inside the container are owned by your host user.
3. Interview question: "Your team wants every developer to run the full application stack in containers. What would you put in containers first, and what would you keep on the host?" Hint: weigh onboarding pain, file-sharing performance on macOS and Windows, editor integration, and how close the result needs to be to production.

Dev Containers

Why it matters. A containerized toolchain is only half of a development environment. Developers also need their editor to see the same compiler, language server, linters, and debugger as the build. If the editor runs on the host while the code runs in a container, autocompletion uses the wrong library versions, and "go to definition" points at packages that do not exist in the container. Dev Containers solve this by running the editor's backend inside the container, using a standard file, devcontainer.json, that any compatible tool can read.

Analogy. Instead of shouting instructions into a workshop through a window, you walk into the workshop. The editor's window stays on your desktop, but everything it does (open files, run commands, start the debugger) happens inside the container.

How it works

 host                                          container (built from devcontainer.json)
 +---------------------------+                 +------------------------------------------+
 | editor UI (VS Code,       |   remote        | editor server + extensions               |
 | JetBrains Gateway, ...)   | <-------------> |   language server, debugger, terminal    |
 +---------------------------+   protocol      | toolchain: python 3.13, node 22, gh ...  |
 | ./my-project  ============|=== bind mount ==|=> /workspaces/my-project                 |
 +---------------------------+                 +------------------------------------------+
                                                     ^ optional: other services via Compose

The specification. The Development Container Specification (containers.dev) is an open standard. It is supported by VS Code (Dev Containers extension), GitHub Codespaces, JetBrains IDEs, DevPod, and the reference devcontainer CLI. The configuration lives in .devcontainer/devcontainer.json.

A minimal configuration

// .devcontainer/devcontainer.json  (JSON with comments is allowed)
{
  "name": "api",
  "image": "mcr.microsoft.com/devcontainers/python:1-3.12-bookworm",  // prebuilt dev image with a non-root "vscode" user

  // Features: reusable, versioned install scripts layered onto the image
  "features": {
    "ghcr.io/devcontainers/features/node:1": { "version": "22" },
    "ghcr.io/devcontainers/features/docker-outside-of-docker:1": {},   // use the host daemon from inside
    "ghcr.io/devcontainers/features/github-cli:1": {}
  },

  "forwardPorts": [8000],                     // reachable at localhost:8000 on the host
  "postCreateCommand": "pip install -r requirements-dev.txt",   // runs once after creation
  "remoteUser": "vscode",                     // user for terminals and the editor server

  "customizations": {
    "vscode": {
      "extensions": ["ms-python.python", "charliermarsh.ruff"],
      "settings": { "python.defaultInterpreterPath": "/usr/local/bin/python" }
    }
  }
}

Building from your own Dockerfile. Teams that already have a dev stage can point the dev container at it, so the editor, Compose, and CI all use one definition:

{
  "name": "api",
  "build": {
    "dockerfile": "../Dockerfile",   // relative to devcontainer.json
    "context": "..",
    "target": "dev"                  // the dev stage from the multi-stage Dockerfile
  },
  "workspaceFolder": "/workspaces/api",
  "updateRemoteUserUID": true        // Linux: change the container user's UID to match yours
}

Using Compose for multi-service setups. The dev container can be one service in a Compose project; the other services (database, cache) start alongside it and are reachable by name:

{
  "name": "shop",
  "dockerComposeFile": ["../compose.yaml", "compose.devcontainer.yaml"],
  "service": "api",                  // the editor attaches to this service
  "runServices": ["api", "db", "redis"],
  "workspaceFolder": "/workspaces/shop",
  "shutdownAction": "stopCompose"
}

Lifecycle hooks. Each hook runs at a specific moment; choosing the right one keeps rebuilds and restarts fast.

Property	Runs	Typical use
initializeCommand	On the host, before the container is created	Generate a .env file, check prerequisites
onCreateCommand	Once, when the container is first created	Heavy, one-time setup
updateContentCommand	After creation and when content changes (used by prebuilds)	Install dependencies
postCreateCommand	Once, after the container is assigned to the user	User-specific setup, pip install, npm ci
postStartCommand	Every time the container starts	Start a background process
postAttachCommand	Every time the editor attaches	Print a welcome message, open a shell

Features. A feature is a small, versioned package (an OCI artifact containing an install script and metadata) that adds a tool to any base image: a language runtime, the Docker CLI, kubectl, a cloud CLI. Features let teams keep using a plain base image instead of maintaining a custom one for every combination of tools.

Using it without an IDE. The reference CLI runs the same configuration in scripts and CI, which guarantees that CI tests run in exactly the developers' environment:

npm install -g @devcontainers/cli

devcontainer up --workspace-folder .                     # build and start the dev container
devcontainer exec --workspace-folder . pytest -q         # run a command inside it
devcontainer build --workspace-folder . \
  --image-name ghcr.io/acme/api-devcontainer:latest --push   # prebuild an image for faster startup

The devcontainers/ci GitHub Action wraps these commands, and GitHub Codespaces uses the same file to create cloud environments, optionally from prebuilt images.

Docker access from inside a dev container. Two features cover the common need to run docker inside the environment:

Feature	How	Trade-off
docker-outside-of-docker	Mounts the host's Docker socket	Fast, shares the host's images; containers you start are siblings, and bind-mount paths refer to the host (see Docker-in-Docker vs Socket Mounting)
docker-in-docker	Runs a separate daemon inside the dev container (requires privileged mode)	Isolated from the host; separate image cache; more resources

Key takeaways

• Dev Containers run the editor's backend inside the container, so tooling sees exactly the runtime the code uses.
• devcontainer.json is an open specification read by VS Code, Codespaces, JetBrains IDEs, DevPod, and the devcontainer CLI.
• Configure the container from an image, your own Dockerfile target, or a Compose service; add tools with features.
• Use the right lifecycle hook: one-time work in onCreateCommand/postCreateCommand, recurring work in postStartCommand.
• The CLI and prebuilt images let CI and cloud environments reuse the exact developer environment.

🧪 Practice

1. (Beginner) Add a .devcontainer/devcontainer.json to a small project using a mcr.microsoft.com/devcontainers/* image, one feature, and a postCreateCommand. Open it in VS Code with the Dev Containers extension and check which python (or node) in the integrated terminal.
2. (Intermediate) Convert the configuration to use your project's Dockerfile dev target plus a Compose file with a database. Run the test suite through devcontainer up and devcontainer exec from a plain terminal to prove it works without an editor.
3. Interview question: "What is the difference between running your app in a container with a bind mount and using a dev container?" Hint: think about where the language server, debugger, terminal, and extensions run, and what they can see.

Hot Reloading

Why it matters. Rebuilding an image and recreating a container for every code change takes seconds to minutes. Developers make hundreds of small changes a day, so this delay destroys the flow of work. Hot reloading means the running application picks up code changes automatically. It works out of the box on a laptop, but inside a container three separate things must line up, and when one is missing the result is the same: you save a file and nothing happens.

Analogy. Hot reloading in a container is a relay race with three runners: the file must be handed into the container, the watcher must notice it, and the server must reload and be reachable. A dropped baton at any stage stops the race, so troubleshooting means finding which runner dropped it.

Three flavors of reloading

Kind	What happens	State preserved?	Examples
Process restart	A watcher kills and restarts the server process	No	node --watch, nodemon, uvicorn --reload, Flask debug reloader, air (Go), dotnet watch
Live reload	Server or bundler rebuilds, the browser refreshes the page	No (page reloads)	Many static site and template tools
Hot module replacement (HMR)	Changed modules are swapped in the running browser app over a WebSocket	Yes (component state survives)	Vite, webpack dev server, Next.js

The three links in the chain

 1. delivery              2. detection                 3. reload and reach
 editor saves file  -->   watcher inside container --> server restarts / HMR push
 on the host              sees the change               browser connects via published port
                                                         
 bind mount or            inotify events, or           server listens on 0.0.0.0;
 compose watch sync       polling as a fallback        app port AND HMR port published

1. Delivery: getting the change into the container. Either bind-mount the source directory (Bind Mounts for Development) or let Compose Watch copy changed files in. Compose Watch is usually faster on Docker Desktop because the container then reads its own native filesystem.

2. Detection: making the watcher notice. On Linux, watchers use inotify events, which work through bind mounts. Docker Desktop on macOS and Windows forwards file change events for its file shares, but some setups do not deliver them:

Situation	Events delivered?	Fix
Linux host, bind mount	Yes	None needed
Docker Desktop (macOS, Windows), bind mount	Yes, in current versions	None needed; update old Desktop versions
WSL2 with the project under /mnt/c and edited by Windows tools	Often not	Move the project into the WSL2 filesystem, or enable polling
Network filesystems, some VMs, Compose Watch on very large trees	Unreliable	Enable polling

Polling checks file timestamps on an interval. It always works but uses CPU, so treat it as a fallback. Common switches: CHOKIDAR_USEPOLLING=true (many Node.js tools), WATCHFILES_FORCE_POLLING=true (uvicorn, watchfiles), DOTNET_USE_POLLING_FILE_WATCHER=true, and server.watch.usePolling in Vite.

3. Reload and reach: serving the result. Dev servers often listen on 127.0.0.1 by default, which inside a container means "only reachable from inside the container". The server must listen on 0.0.0.0, and every port the browser uses, including a separate HMR WebSocket port if the tool uses one, must be published.

Example: a Node.js API and a Vite front end with Compose Watch

# compose.yaml (development)
services:
  api:
    build: { context: ./api, target: dev }
    command: ["node", "--watch", "src/server.js"]   # built-in restart-on-change (Node 22)
    ports: ["127.0.0.1:3000:3000"]
    develop:
      watch:
        - { action: sync, path: ./api/src, target: /app/src }
        - { action: rebuild, path: ./api/package-lock.json }   # new dependencies need a new image

  web:
    build: { context: ./web, target: dev }
    command: ["npx", "vite", "--host", "0.0.0.0", "--port", "5173"]   # listen on all interfaces
    ports: ["127.0.0.1:5173:5173"]    # Vite serves HMR over the same port by default
    develop:
      watch:
        - { action: sync, path: ./web/src, target: /app/src }
        - { action: rebuild, path: ./web/package-lock.json }

// web/vite.config.js
import { defineConfig } from "vite";

export default defineConfig({
  server: {
    host: true,                     // same as --host 0.0.0.0
    port: 5173,
    strictPort: true,               // fail instead of silently moving to another port
    watch: {
      usePolling: process.env.VITE_POLLING === "true",   // opt-in fallback for WSL2 /mnt/c etc.
      interval: 300,
    },
  },
});

Reloaders by ecosystem

Stack	Command inside the container	Notes
Node.js	node --watch src/server.js or nodemon	--watch is built into Node.js 22
Python (ASGI)	uvicorn app.main:app --host 0.0.0.0 --reload	Polling: WATCHFILES_FORCE_POLLING=true
Python (Flask)	flask --app app run --host 0.0.0.0 --debug	Debug mode enables the reloader
Django	python manage.py runserver 0.0.0.0:8000	Reloads on Python file changes
Go	air (rebuilds and restarts the binary)	Keep the Go build cache in a named volume
Java (Spring Boot)	DevTools restart on classpath changes	Recompile on the host or in the container to trigger it
.NET	dotnet watch run --urls http://0.0.0.0:5000	Polling: DOTNET_USE_POLLING_FILE_WATCHER=true

Debugging a reload that does not happen

# 1) Did the file reach the container?
docker compose exec web cat /app/src/App.jsx | head -n 5

# 2) Does the container see change events? (inotify-tools may need installing in the dev image)
docker compose exec web sh -c 'inotifywait -m /app/src'   # then save the file on the host

# 3) Is the server listening on all interfaces?
docker compose exec web sh -c 'netstat -tln 2>/dev/null || ss -tln'
# tcp  0  0 127.0.0.1:5173  ...   <- wrong: only reachable inside the container
# tcp  0  0 0.0.0.0:5173    ...   <- right

Key takeaways

• Hot reloading in a container needs three things: delivery (bind mount or watch sync), detection (inotify or polling), and reachability (0.0.0.0 plus published ports).
• Restart-based reloaders lose state; HMR swaps modules in the browser and keeps state.
• Use polling only where change events do not arrive, such as WSL2 projects stored under /mnt/c.
• Dependency changes need a rebuild; use a rebuild watch rule for lockfiles.
• When reloading fails, check each link in order instead of guessing.

🧪 Practice

1. (Beginner) Run a small Express or FastAPI app in a container with node --watch or uvicorn --reload and a bind mount. Change a response string and confirm the server restarts without recreating the container.
2. (Intermediate) Start a Vite app in a container with Vite's default host setting. Observe that the page is unreachable, fix it, and then make HMR preserve the state of a counter component while you edit its label.
3. Interview question: "A developer on Windows says hot reloading works for their teammates but not for them, even though they use the same Compose file. How would you investigate?" Hint: follow the three links in the chain, and ask where the project files live and which filesystem the watcher is reading.

Debugging Apps Inside Containers

Why it matters. Log statements only get you so far. To understand a tricky bug you want to stop the program at a line, inspect variables, and step through the code. When the application runs in a container, the debugger in your IDE runs on the host and the process runs in a separate PID and network namespace, so the IDE cannot simply "launch" the process. The solution, in every language, is remote debugging: a small debug agent runs inside the container next to the application, listens on a port, and the IDE connects to it over the network.

Analogy. It is like diagnosing a car over a phone call. The mechanic (your IDE) cannot touch the engine, so someone in the garage (the debug agent) relays every question ("what is in this variable?") and executes every command ("pause at line 42"). The phone line is a published port.

The general pattern

 host                                         container
 +---------------------------+                +-------------------------------------+
 | IDE debugger client       |  TCP           | debug agent  (listens 0.0.0.0:PORT) |
 |  "attach" configuration   | -------------> |   controls the app process          |
 |  path mapping:            |  127.0.0.1:    | app code at /app/...                |
 |  ./src  <->  /app/src     |  PORT          |                                     |
 +---------------------------+                +-------------------------------------+
        published with  -p 127.0.0.1:PORT:PORT   (never 0.0.0.0 on the host)

Four things must be configured, whatever the language:

1. Start the agent in the container, listening on 0.0.0.0 (not 127.0.0.1, which is unreachable from outside the container).
2. Publish the debug port, bound to 127.0.0.1 on the host.
3. Attach from the IDE with an "attach" (not "launch") configuration.
4. Map paths between the host source tree and the container's paths, so breakpoints set in ./src/app.py land on /app/src/app.py.

Debug agents by language

Language	Agent and start command (inside the container)	Default port
Node.js	node --inspect=0.0.0.0:9229 src/server.js (--inspect-brk pauses on the first line)	9229
Python	python -m debugpy --listen 0.0.0.0:5678 [--wait-for-client] -m uvicorn app.main:app --host 0.0.0.0	5678
Java	JAVA_TOOL_OPTIONS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005	5005
Go	dlv exec /app/server --headless --listen=:2345 --api-version=2 --accept-multiclient	2345
.NET	vsdbg started through docker exec by the IDE (pipe transport, no port)	n/a

Example: Python with debugpy and Compose

# compose.debug.yaml - layered on top of the normal dev file
services:
  api:
    command:
      - python
      - -m
      - debugpy
      - --listen
      - 0.0.0.0:5678          # agent reachable from outside the container
      - -m
      - uvicorn
      - app.main:app
      - --host
      - 0.0.0.0
      - --port
      - "8000"
      # no --reload here: the reloader spawns child processes the debugger would not follow
    ports:
      - "127.0.0.1:8000:8000"
      - "127.0.0.1:5678:5678" # debugger: localhost only

docker compose -f compose.yaml -f compose.debug.yaml up -d api

// .vscode/launch.json
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Attach: api (Python in container)",
      "type": "debugpy",
      "request": "attach",
      "connect": { "host": "127.0.0.1", "port": 5678 },
      "pathMappings": [
        { "localRoot": "${workspaceFolder}/src", "remoteRoot": "/app/src" }   // host path <-> container path
      ],
      "justMyCode": false
    },
    {
      "name": "Attach: web (Node.js in container)",
      "type": "node",
      "request": "attach",
      "address": "127.0.0.1",
      "port": 9229,
      "localRoot": "${workspaceFolder}/web",
      "remoteRoot": "/app",
      "restart": true          // reattach automatically after node --watch restarts the process
    }
  ]
}

Example: Go with Delve. Compiled languages need a binary built without optimizations, or the debugger cannot map instructions back to lines and variables appear as "optimized away". A dedicated debug stage keeps this out of the production image:

# syntax=docker/dockerfile:1
FROM golang:1.25 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .

# ---- debug: unoptimized binary plus the Delve debugger ----
FROM build AS debug
RUN go install github.com/go-delve/delve/cmd/dlv@latest     # pin a version in a real project
RUN go build -gcflags="all=-N -l" -o /app/server ./cmd/server  # -N: no optimizations, -l: no inlining
EXPOSE 8080 2345
CMD ["dlv", "exec", "/app/server", "--headless", "--listen=:2345", \
     "--api-version=2", "--accept-multiclient", "--continue"]

# ---- production: optimized, stripped, no debugger ----
FROM build AS release
RUN CGO_ENABLED=0 go build -ldflags="-s -w" -o /app/server ./cmd/server
FROM gcr.io/distroless/static-debian12 AS runtime
COPY --from=release /app/server /server
ENTRYPOINT ["/server"]

docker build --target debug -t api:debug .
# Delve uses ptrace; depending on the kernel and seccomp profile it may need these options
docker run --rm -p 127.0.0.1:8080:8080 -p 127.0.0.1:2345:2345 \
  --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \
  api:debug

Example: Java through an environment variable. The JVM reads JAVA_TOOL_OPTIONS automatically, so the same image can run with or without the debugger:

docker run --rm -p 127.0.0.1:8080:8080 -p 127.0.0.1:5005:5005 \
  -e JAVA_TOOL_OPTIONS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005" \
  acme/orders:dev
# Picked up JAVA_TOOL_OPTIONS: -agentlib:jdwp=...
# Listening for transport dt_socket at address: 5005

address=*:5005 is required since Java 9; a bare 5005 binds only to localhost inside the container. Set suspend=y to make the JVM wait for the debugger before running main, which is how you debug startup code.

Alternative: debug from inside the container. With Dev Containers, or by attaching the editor to a running container, the debugger runs inside the container itself. You can then use a normal "launch" configuration with no ports and no path mapping, because the editor and the process see the same filesystem.

When the container keeps crashing. You cannot attach to a process that exits at startup. Start the container with a different command so it stays alive, then run the application by hand under the debugger, or use the techniques in Debugging Crashed Containers and docker debug:

docker run -it --rm --entrypoint sh -p 127.0.0.1:9229:9229 acme/web:dev
# inside the container:
node --inspect-brk=0.0.0.0:9229 src/server.js     # waits for the IDE before executing line 1

Security: a debug port is remote code execution. Anyone who can connect to a Node.js inspector, debugpy, or JDWP port can run arbitrary code in the process. Always bind debug ports to 127.0.0.1 on the host, never enable them in production images, and never expose them on shared or cloud machines.

Common problems

Symptom	Cause	Fix
IDE: "connection refused"	Agent listens on 127.0.0.1 inside the container, or port not published	Listen on 0.0.0.0; check docker port <container>
Breakpoints stay grey or "unverified"	Path mapping wrong	Match localRoot/remoteRoot to the actual paths (docker compose exec api pwd)
Debugger attaches but stepping jumps around (Go, C, Rust)	Optimized binary	Build with debug flags in a debug stage
Debugger disconnects every time code changes	Reloader restarted the process	Disable reloading while debugging, or enable automatic reattach ("restart": true)
Breakpoints in startup code never hit	Code ran before the IDE attached	--inspect-brk, --wait-for-client, or suspend=y

Key takeaways

• Remote debugging runs a debug agent in the container and attaches the IDE to it over a published port.
• The agent must listen on 0.0.0.0 inside the container; the port must be published only on 127.0.0.1 on the host.
• Path mappings connect host source files to container paths; wrong mappings cause unverified breakpoints.
• Build unoptimized binaries in a separate debug stage for compiled languages; keep debuggers out of production images.
• An open debug port allows arbitrary code execution, so treat it like a root shell.

🧪 Practice

1. (Beginner) Run a Node.js app in a container with --inspect=0.0.0.0:9229, publish the port on 127.0.0.1, attach from VS Code or Chrome (chrome://inspect), and pause on a breakpoint triggered by an HTTP request.
2. (Intermediate) Add a debug stage to a Go or Java project's Dockerfile and a compose.debug.yaml file. Make breakpoints work, including in startup code, without changing the production target.
3. Interview question: "Your breakpoints work when the app runs on your laptop but show as unverified when it runs in a container, even though the debugger connects. What is wrong?" Hint: the debugger compares file paths; think about where the code lives on each side and how the IDE translates between them.

Testcontainers

Why it matters. Integration tests are only meaningful if they run against the real thing: the same PostgreSQL version, the same Kafka broker, the same Redis. In-memory substitutes (H2 instead of PostgreSQL, a fake S3) behave differently in exactly the cases that cause production bugs: SQL dialects, transactions, locking, and error codes. A shared test database causes flaky tests, because tests interfere with each other and with whatever state a previous run left behind. Testcontainers is a family of libraries that start real dependencies as throwaway containers from inside the test code, give the test their address, and remove them afterwards.

Analogy. Instead of all tests sharing one communal kitchen that is never quite clean, each test suite gets a brand-new kitchen built just before it starts and demolished right after it finishes.

How it works

 test process (JUnit, pytest, Jest, go test, ...)
   |
   | 1. Docker Engine API (DOCKER_HOST or the default socket)
   v
 Docker daemon
   |-- 2. starts  postgres:16-alpine    published on a RANDOM host port, e.g. 55012
   |-- 3. starts  testcontainers/ryuk   "reaper": removes everything labelled with this session
   |
   | 4. wait strategy: poll until "ready to accept connections" / port open / HTTP 200
   v
 test gets host + mapped port -> runs against a real database -> library stops containers
 (if the test process crashes, Ryuk notices the lost connection and cleans up)

Key ideas:

• Random host ports. Containers publish to free ports chosen by Docker, so parallel test runs and other services on the machine never collide. Tests ask the library for the actual host and port.
• Wait strategies. A container being "started" does not mean the service inside is ready. The library waits for a log line, an open port, an HTTP endpoint, or the image's health check before handing control to the test.
• Modules. Ready-made classes for common services (PostgreSQL, MySQL, MongoDB, Kafka, Redis, LocalStack, Elasticsearch, and many more) know the right image, ports, wait strategy, and connection string format.
• Ryuk, the resource reaper. A small sidecar container removes containers, networks, and volumes created by a test session, even if the test process is killed.

Testcontainers libraries exist for Java, Go, .NET, Node.js, Python, Rust, Ruby, and other languages. Docker acquired AtomicJar, the company behind the project, in 2023; the libraries remain open source.

Example: Python with pytest

# tests/test_users.py
import pytest
import sqlalchemy as sa
from testcontainers.postgres import PostgresContainer

from app.repository import UserRepository


@pytest.fixture(scope="module")                 # one container for all tests in this module
def engine():
    with PostgresContainer("postgres:16-alpine") as pg:   # pinned version = production version
        engine = sa.create_engine(pg.get_connection_url())  # host and random port filled in
        with engine.begin() as conn:
            conn.execute(sa.text(
                "CREATE TABLE users (id serial PRIMARY KEY, email text UNIQUE NOT NULL)"
            ))
        yield engine
    # leaving the "with" block stops and removes the container


@pytest.fixture
def repo(engine):
    with engine.begin() as conn:
        conn.execute(sa.text("TRUNCATE users RESTART IDENTITY"))   # isolate tests from each other
    return UserRepository(engine)


def test_duplicate_email_is_rejected(repo):
    repo.add("ada@example.com")
    with pytest.raises(sa.exc.IntegrityError):  # a real UNIQUE constraint violation from PostgreSQL
        repo.add("ada@example.com")

Example: Java with JUnit 5

// Testcontainers for Java 1.x with the JUnit 5 extension
import org.junit.jupiter.api.Test;
import org.testcontainers.containers.PostgreSQLContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;

import static org.junit.jupiter.api.Assertions.assertEquals;

@Testcontainers                                   // manages @Container fields
class OrderRepositoryTest {

    @Container                                    // static: one container for the whole class
    static PostgreSQLContainer<?> postgres = new PostgreSQLContainer<>("postgres:16-alpine")
            .withInitScript("schema.sql");        // from src/test/resources

    @Test
    void countsOrders() throws Exception {
        try (Connection c = DriverManager.getConnection(
                postgres.getJdbcUrl(),            // jdbc:postgresql://localhost:55012/test
                postgres.getUsername(),
                postgres.getPassword())) {
            c.createStatement().execute("INSERT INTO orders(total) VALUES (42)");
            ResultSet rs = c.createStatement().executeQuery("SELECT count(*) FROM orders");
            rs.next();
            assertEquals(1, rs.getInt(1));
        }
    }
}

Spring Boot (3.1+) can wire such containers into the application context directly with @ServiceConnection, so no connection properties need to be set by hand.

Generic containers. For anything without a module, start any image and describe how to know it is ready:

// Node.js: a custom service image, ready when /health returns 200
import { GenericContainer, Wait } from "testcontainers";

const mock = await new GenericContainer("wiremock/wiremock:3.9.1")
  .withExposedPorts(8080)                                  // container port
  .withWaitStrategy(Wait.forHttp("/__admin/health", 8080)) // poll until healthy
  .start();

const baseUrl = `http://${mock.getHost()}:${mock.getMappedPort(8080)}`;   // random host port
// ... run tests against baseUrl ...
await mock.stop();

Container lifecycle choices

Scope	How	Speed	Isolation
Per test	Container created in each test	Slowest	Perfect
Per class or module	Static field or module-scoped fixture	Good	Reset data between tests (truncate, transactions)
Singleton per test run	One container started in a base class or session fixture	Fast	Same as above, across the whole suite
Reusable across runs (local only)	testcontainers.reuse.enable=true in ~/.testcontainers.properties and .withReuse(true)	Fastest	Lowest; state survives between runs; not for CI

Running in CI. Testcontainers only needs access to a Docker daemon.

CI environment	What to do
GitHub Actions Linux runners, VM-based runners	Works as-is: the runner has a Docker daemon
GitLab CI with a docker:dind service	Set DOCKER_HOST=tcp://docker:2375 (or the TLS variant) and TESTCONTAINERS_HOST_OVERRIDE=docker, because mapped ports are published on the dind service, not on localhost
Job container with the host socket mounted	Works, but see Docker-in-Docker vs Socket Mounting for the security trade-off
No Docker available	Use a remote Docker host or a cloud service such as Testcontainers Cloud

TESTCONTAINERS_RYUK_DISABLED=true turns off the reaper. Only do this where the whole environment is thrown away after the job (ephemeral CI runners), or where Ryuk cannot run (some rootless or restricted setups).

Testcontainers vs Compose for tests

	Testcontainers	Compose (Integration Testing with Compose)
Where the environment is defined	In test code	In YAML outside the tests
Lifecycle control	Per test, class, or suite	Whole stack per run
Ports	Random, discovered by the test	Fixed or none, defined in YAML
Best for	Repository, adapter, and component tests with a few real dependencies	Black-box tests of the whole system, including your own service images
Developer experience	Run tests from the IDE as usual	A wrapper script or docker compose run

Key takeaways

• Testcontainers starts real dependencies as disposable containers from test code, through the Docker API.
• Random host ports, wait strategies, and the Ryuk reaper make tests parallel-safe, reliable, and self-cleaning.
• Pin the same image version as production; that is the whole point of testing against the real service.
• Share containers per class or suite and reset data between tests to keep suites fast; reuse across runs only locally.
• In CI the library just needs a reachable Docker daemon; with dind, set DOCKER_HOST and TESTCONTAINERS_HOST_OVERRIDE.

🧪 Practice

1. (Beginner) Write one test that starts PostgreSQL with Testcontainers in your language, creates a table, inserts a row, and reads it back. While the test runs, watch docker ps and note the random port and the Ryuk container.
2. (Intermediate) Convert the setup to one container per test session, add data cleanup between tests, and measure the suite time before and after. Then run the suite in a CI job and make it pass there too.
3. Interview question: "Why would you use Testcontainers instead of an in-memory database like H2 for repository tests, and what does it cost you?" Hint: think about SQL dialects, constraints, and locking behavior on one side, and startup time, Docker availability, and CI configuration on the other.

<a id="132-continuous-integration-and-delivery"></a>

13.2 Continuous Integration and Delivery

This section moves from the developer's laptop to the pipeline: how to build images on CI machines, how GitHub Actions does it with Docker's official actions, how CI jobs get access to a Docker daemon, how to keep builds fast on throwaway runners, how to tag images so every deployment is traceable, and how to stop vulnerable images before they reach a registry. It builds on Chapter 9 (BuildKit) and Section 10.3 (supply-chain security).

Building Images in CI

Why it matters. An image built on a developer's laptop is a mystery: nobody knows exactly which commit, which uncommitted changes, which base image version, or which credentials went into it. An image built by CI from a specific commit, in a clean environment, with a recorded log, is an artifact you can trust, trace, and reproduce. CI is also where the image is tested and scanned before anyone can deploy it.

Analogy. A home kitchen versus a certified factory line. Both can bake a cake, but only the factory records the batch number, checks every unit at fixed stations, and guarantees that the cake shipped to every store is the same one that passed inspection.

The guiding principle: build once, promote everywhere. Build the image one time per commit. Test that exact image, scan it, and deploy that exact image (identified by its digest) to staging and then production. Rebuilding for each environment produces a different artifact, because base images, package mirrors, and timestamps change, so the thing that passed the tests is not the thing that runs in production.

 commit 3f9a1c2
     |
     v
 +------+   +-------+   +-------------+   +------+   +------+   +---------+   +--------------+
 | lint |-->| build |-->| test (unit, |-->| scan |-->| push |-->| sign /  |-->| deploy by    |
 |      |   |       |   | integration)|   | gate |   |      |   | attest  |   | digest:      |
 +------+   +-------+   +-------------+   +------+   +------+   +---------+   | dev -> stg ->|
  hadolint   one image    against the       fail on    tag sha,   provenance  | prod         |
  shellcheck per commit   built image       fixable    version    SBOM        +--------------+
                                            criticals
              <-------------- the same image digest flows through every stage ------------->

Principles for CI builds

Principle	In practice
Clean, isolated build	Ephemeral runners or builder instances; nothing left over from earlier jobs
Deterministic inputs	Base images pinned by tag (or digest), lockfiles for dependencies, a strict .dockerignore
Fail fast	Lint and unit tests before the expensive steps; tests run against the built image
Traceability	Tags with the commit SHA; OCI labels (org.opencontainers.image.revision, .source); provenance attestations
Least privilege	Pull requests build but do not push; short-lived credentials (OIDC) instead of stored passwords
Full BuildKit features	A docker-container builder for cache export, multi-platform builds, and attestations

Running tests as part of the build. A test stage in the Dockerfile runs the unit tests in the same environment as the build. If the tests fail, RUN fails, and so does the build:

# syntax=docker/dockerfile:1
FROM node:22-slim AS deps
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm npm ci

FROM deps AS test
COPY . .
RUN npm run lint && npm test          # a failing test fails the build

FROM deps AS build
COPY . .
RUN npm run build && npm prune --omit=dev

FROM node:22-slim AS runtime
WORKDIR /app
ENV NODE_ENV=production
COPY --from=build /app/node_modules ./node_modules
COPY --from=build /app/dist ./dist
USER node
CMD ["node", "dist/server.js"]

The runtime stage does not depend on test, so BuildKit skips the test stage when building runtime. Build the test target explicitly in CI, then the runtime target; the shared deps stage is built only once thanks to the cache.

A portable CI build script. Every CI system can run a shell script, so the core steps can be written once and called from any pipeline:

#!/usr/bin/env bash
# ci/build.sh - build, test, scan, and push one image for the current commit
set -euo pipefail

IMAGE="registry.example.com/acme/web"
GIT_SHA="$(git rev-parse HEAD)"
TAG="sha-${GIT_SHA:0:12}"

# A docker-container builder supports cache export, attestations, and multi-platform output
docker buildx create --name ci --driver docker-container --use --bootstrap 2>/dev/null \
  || docker buildx use ci

# 1) Unit tests inside the build
docker buildx build --target test --progress=plain .

# 2) Runtime image, loaded into the local image store for scanning (not pushed yet)
docker buildx build --target runtime --load \
  --tag "$IMAGE:$TAG" \
  --label "org.opencontainers.image.revision=$GIT_SHA" \
  --label "org.opencontainers.image.source=https://github.com/acme/web" \
  --label "org.opencontainers.image.created=$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  .

# 3) Gate: stop here if there are fixable high or critical vulnerabilities
trivy image --severity HIGH,CRITICAL --ignore-unfixed --exit-code 1 "$IMAGE:$TAG"

# 4) Push only what passed
docker push "$IMAGE:$TAG"
docker buildx imagetools inspect "$IMAGE:$TAG" --format '{{json .Manifest.Digest}}'   # record the digest

Where the build runs. CI systems differ in how a job reaches a builder:

Option	How it works	Notes
Docker daemon on the runner VM	The job uses the VM's own daemon (GitHub-hosted runners, most VM-based runners)	Simplest; isolation comes from the VM
Docker-in-Docker or socket mounting	The job runs in a container and needs a daemon	Trade-offs in Docker-in-Docker vs Socket Mounting
Daemonless and rootless builders	BuildKit in rootless mode (buildctl-daemonless.sh), Buildah	Build without a privileged daemon, common on Kubernetes runners
Remote builders	docker buildx create --driver remote tcp://buildkitd:1234, the kubernetes driver, Docker Build Cloud	Shared, persistent cache; native multi-architecture nodes

Kaniko, long a popular daemonless builder on Kubernetes, was archived by Google in 2025; existing pipelines continue on community forks, while new ones usually choose rootless BuildKit or Buildah.

Reproducibility extras. BuildKit honors the SOURCE_DATE_EPOCH build argument to set timestamps in the image, and with the exporter option rewrite-timestamp=true it rewrites file timestamps in layers too. Combined with pinned inputs, this makes two builds of the same commit produce the same digest, which lets anyone verify that an image really came from the claimed source.

Key takeaways

• Build each commit's image once in CI, then test, scan, and deploy that exact digest; never rebuild per environment.
• Order the pipeline to fail fast: lint, build, test, scan, then push, sign, and deploy.
• A test stage in the Dockerfile runs tests in the build environment; build it as an explicit target.
• Use a docker-container (or remote) builder in CI to get cache export, attestations, and multi-platform builds.
• Label images with their source and revision so any running container can be traced back to a commit.

🧪 Practice

1. (Beginner) Add a test stage to a project's Dockerfile and run docker buildx build --target test .. Break a test on purpose and check that the build fails with a non-zero exit code.
2. (Intermediate) Write a ci/build.sh like the one above for one of your projects and run it locally against a local registry (docker run -d -p 5000:5000 registry:2). Confirm that a scan failure prevents the push.
3. Interview question: "Why is it a bad idea to rebuild the image separately for staging and production, even from the same commit?" Hint: list everything that can change between two builds besides your code, and think about what your tests actually verified.

GitHub Actions for Docker

Why it matters. GitHub Actions is one of the most widely used CI systems, and Docker maintains a set of official actions for it. Together they reduce a production-grade image pipeline (multi-platform build, cache, metadata, registry login, SBOM and provenance) to a few dozen lines of YAML. Knowing what each action does, and how they fit together, is enough to read and write most real-world Docker workflows.

Analogy. The official actions are like standard parts for an assembly line. Each one does one job (log in, set up the machine, compute labels, build), and they fit together because they were designed as one set.

Docker's official actions

Action	Job
docker/setup-qemu-action	Registers QEMU emulators so the runner can build for other CPU architectures
docker/setup-buildx-action	Creates a Buildx builder (docker-container driver by default) for the job
docker/login-action	Logs in to Docker Hub, GHCR, ECR, GAR, ACR, or any registry
docker/metadata-action	Computes tags, OCI labels, and annotations from Git context (branch, tag, PR, SHA)
docker/build-push-action	Runs docker buildx build with inputs for every build option
docker/bake-action	Builds several targets from a Bake file
docker/scout-action	Vulnerability reports and comparisons with Docker Scout

A complete workflow

# .github/workflows/image.yml
name: image

on:
  push:
    branches: [main]
    tags: ["v*.*.*"]               # release tags such as v1.4.2
  pull_request:                    # build and test PRs, but never push them

permissions:
  contents: read
  packages: write                  # push to GHCR with GITHUB_TOKEN

env:
  IMAGE: ghcr.io/${{ github.repository }}   # metadata-action lowercases the name for you

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: docker/setup-qemu-action@v3          # only needed for non-native platforms
      - uses: docker/setup-buildx-action@v3

      - uses: docker/login-action@v3
        if: github.event_name != 'pull_request'    # PRs from forks have no secrets anyway
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - id: meta
        uses: docker/metadata-action@v5
        with:
          images: ${{ env.IMAGE }}
          tags: |
            type=ref,event=branch                  # main
            type=ref,event=pr                      # pr-123
            type=semver,pattern={{version}}        # 1.4.2   (from tag v1.4.2)
            type=semver,pattern={{major}}.{{minor}}  # 1.4
            type=sha                               # sha-3f9a1c2

      - name: Unit tests (test stage)
        uses: docker/build-push-action@v6
        with:
          context: .
          target: test
          cache-from: type=gha
          cache-to: type=gha,mode=max

      - id: build
        name: Build and push
        uses: docker/build-push-action@v6
        with:
          context: .
          target: runtime
          platforms: linux/amd64,linux/arm64
          push: ${{ github.event_name != 'pull_request' }}
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}           # org.opencontainers.image.* labels
          annotations: ${{ steps.meta.outputs.annotations }}
          cache-from: type=gha
          cache-to: type=gha,mode=max
          provenance: mode=max                                # SLSA provenance attestation
          sbom: true                                          # SBOM attestation

      - name: Record digest
        if: github.event_name != 'pull_request'
        run: echo "Pushed ${{ env.IMAGE }}@${{ steps.build.outputs.digest }}" >> "$GITHUB_STEP_SUMMARY"

What each event produces

Event	Tags pushed
Push to main	main, sha-3f9a1c2
Push of tag v1.4.2	1.4.2, 1.4, latest, sha-3f9a1c2 (latest comes from the default flavor: latest=auto, which adds it for non-prerelease semver tags)
Pull request #123	Nothing pushed; the image is built and tested with tag pr-123

Details that matter

• Git context vs path context. Without context:, build-push-action builds directly from the Git repository at the current commit (no checkout needed, and untracked files cannot leak in). With actions/checkout and context: ., it builds from the checked-out directory, which is required if earlier steps generate files the build needs.
• Outputs. steps.<id>.outputs.digest, imageid, and metadata feed later steps: signing with cosign, attestations, or deployment by digest.
• Build summaries. Each build adds a summary to the workflow run with the build's duration, cache usage, and a downloadable build record that can be opened in Docker Desktop.
• Build secrets. Pass secrets to RUN --mount=type=secret with the secrets: input (for example secrets: npm_token=${{ secrets.NPM_TOKEN }}), never as build arguments; see Build-Time Secrets.

Faster multi-platform builds on native runners. Emulating arm64 with QEMU can make compilation-heavy builds 5 to 20 times slower. GitHub offers Arm-based Linux runners (ubuntu-24.04-arm), so each platform can build natively in parallel, push by digest only, and a final job combines the digests into one multi-platform tag:

jobs:
  build:
    strategy:
      matrix:
        include:
          - { platform: linux/amd64, runner: ubuntu-24.04 }
          - { platform: linux/arm64, runner: ubuntu-24.04-arm }
    runs-on: ${{ matrix.runner }}
    steps:
      # ... checkout, setup-buildx, login as above ...
      - id: build
        uses: docker/build-push-action@v6
        with:
          platforms: ${{ matrix.platform }}
          # push an untagged single-platform image; it is reachable only by digest
          outputs: type=image,name=ghcr.io/acme/api,push-by-digest=true,name-canonical=true,push=true
          cache-from: type=gha,scope=${{ matrix.platform }}
          cache-to: type=gha,scope=${{ matrix.platform }},mode=max
      - run: mkdir -p digests && touch "digests/${DIGEST#sha256:}"
        env: { DIGEST: "${{ steps.build.outputs.digest }}" }
      - uses: actions/upload-artifact@v4
        with:
          name: digest-${{ strategy.job-index }}
          path: digests/*

  merge:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact@v4
        with: { pattern: digest-*, path: digests, merge-multiple: true }
      # ... setup-buildx, login ...
      - name: Create the multi-platform manifest list
        working-directory: digests
        run: |
          docker buildx imagetools create -t ghcr.io/acme/api:sha-${GITHUB_SHA::7} \
            $(printf 'ghcr.io/acme/api@sha256:%s ' *)

Security of Docker workflows

Risk	Mitigation
A compromised third-party action steals registry credentials	Pin third-party actions to a full commit SHA (a tag can be moved, as in the 2025 tj-actions/changed-files incident); keep permissions: minimal
Pull requests from forks push images or read secrets	Do not push on pull_request; avoid building untrusted code in pull_request_target workflows
Long-lived cloud registry passwords stored as secrets	Use OIDC (id-token: write) with the cloud's login action to get short-lived credentials
Untrusted PRs fill or poison the shared build cache	Rely on GitHub's branch-scoped cache isolation; write cache only from trusted branches if in doubt

Key takeaways

• setup-buildx, login, metadata, and build-push-action are the core of almost every Docker workflow on GitHub.
• metadata-action derives tags and OCI labels from Git events; build-push-action exposes every Buildx option as an input.
• Build on pull requests, push only from trusted events, and record the pushed digest for later steps.
• For multi-platform builds, prefer native runners per platform and merge digests over QEMU emulation.
• Pin third-party actions to commit SHAs, keep permissions minimal, and prefer OIDC to stored credentials.

🧪 Practice

1. (Beginner) Create a workflow that builds an image on every pull request and pushes it to GHCR on pushes to main, using metadata-action for tags. Inspect the configuration and labels of the pushed image with docker buildx imagetools inspect <image> --format '{{json .Image}}'.
2. (Intermediate) Add linux/arm64 with QEMU and note the build time. Then convert the workflow to the native-runner matrix with a merge job and compare durations and the resulting manifest list.
3. Interview question: "Your workflow pushes images using a Docker Hub password stored as a repository secret, and it builds pull requests from forks. What would you change?" Hint: consider which events receive secrets, what a malicious PR could run during the build, and alternatives to long-lived credentials.

Docker-in-Docker vs Socket Mounting

Why it matters. Many CI systems (GitLab CI with the Docker executor, Jenkins agents in containers, Kubernetes-based runners) run each job inside a container. If that job needs to build images or start containers, it needs a Docker daemon, and a container does not have one. There are two classic ways to give it one, and a few modern alternatives. The choice affects security, build speed, caching, and some surprising behaviors around paths and ports.

Analogy. Your job is a worker inside a rented office (the job container). To get things printed, the worker can either install a private printer inside the office (Docker-in-Docker), or run a cable to the building's central print room (socket mounting). The private printer is isolated but starts empty every time; the cable is fast and shared, but anyone with it can print, and reprint, anything in the building.

 Docker-in-Docker (DinD)                      Socket mounting ("Docker outside of Docker")
 +------------------------------------+       +------------------------------------+
 | host dockerd                       |       | host dockerd  <---------------+    |
 |  +------------------------------+  |       |  +--------------------------+ |    |
 |  | dind container (privileged)  |  |       |  | job container            | |    |
 |  |  inner dockerd               |  |       |  |  docker CLI --socket-----+ |    |
 |  |   +--------+  +--------+     |  |       |  +--------------------------+      |
 |  |   | build  |  | test   |     |  |       |  +--------+  +--------+            |
 |  |   +--------+  +--------+     |  |       |  | build  |  | test   |  siblings  |
 |  +------------------------------+  |       |  +--------+  +--------+  on host   |
 +------------------------------------+       +------------------------------------+
  nested: own images, own cache,              shared: host's images and cache,
  needs --privileged                          socket = root on the host

Docker-in-Docker. The official docker:<version>-dind image runs a complete Docker daemon inside a container. It needs --privileged, because a daemon must create namespaces, cgroups, mounts, and network interfaces. The job talks to this inner daemon over TCP (with TLS by default). Example with GitLab CI:

# .gitlab-ci.yml
build:
  image: docker:28                      # job container with the docker CLI
  services:
    - docker:28-dind                    # sidecar running the inner daemon (privileged runner required)
  variables:
    DOCKER_HOST: tcp://docker:2376      # "docker" is the service's hostname
    DOCKER_TLS_CERTDIR: "/certs"        # dind generates TLS certificates here
    DOCKER_TLS_VERIFY: "1"
    DOCKER_CERT_PATH: "/certs/client"   # shared with the job through the runner's volume config
  script:
    - docker info
    - docker build -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA" .
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"
    - docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA"

The runner itself must allow privileged containers (privileged = true in the GitLab Runner's config.toml), and it must share the /certs/client volume with jobs.

Socket mounting. The job container gets the host's /var/run/docker.sock bind-mounted, and its docker CLI talks to the host's daemon. Containers it starts are siblings of the job container, not children:

# The job container sees, and controls, every container on the host
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock docker:28 docker ps

# GitLab Runner config.toml (Docker executor) - socket mounting for all jobs
[runners.docker]
  image = "docker:28"
  privileged = false
  volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]

The bind-mount path surprise. With a mounted socket, the daemon resolves -v paths on the host, not in the job container. The job's working directory usually does not exist at the same path on the host, so the mount silently produces an empty directory:

# Inside a job container whose project lives at /builds/acme/api
docker run --rm -v "$PWD":/src alpine ls /src
# (empty) - the daemon mounted /builds/acme/api from the HOST, where it does not exist

# Fixes: copy files in instead of mounting them, or build them into an image
docker create --name t alpine ls /src && docker cp . t:/src && docker start -a t && docker rm t

The same applies to ports: a container started with -p 8080:80 publishes on the host, so tests must connect to the host's address, and two parallel jobs that publish the same port collide.

Comparison

	Docker-in-Docker	Socket mounting	Rootless / daemonless builders	Remote builder
Privilege needed by the job	--privileged container	Access to host root via the socket	None (rootless BuildKit, Buildah)	Network access and credentials
Isolation between jobs	Good (separate daemons)	None: jobs see each other's containers	Good	Good (separate builds)
Image and layer cache	Empty each job unless persisted or exported	Shared host cache (fast, but shared)	Exported/imported cache	Persistent on the builder
Bind-mount paths	Paths refer to the dind container's filesystem; they work only if the job directory is shared with it	Paths refer to the host	n/a (build only)	n/a (build only)
Can run containers (tests, Testcontainers)	Yes	Yes	No, build only	No, build only
Typical use	GitLab shared runners, nested test environments	Trusted, single-tenant runners	Kubernetes runners without privileges	Large teams, multi-architecture builds

Hardening options. The Sysbox runtime runs DinD-capable "system containers" without --privileged, using user namespaces. Rootless DinD images (docker:dind-rootless) reduce what a compromised inner daemon can reach. For socket mounting, a filtering socket proxy or an authorization plugin limits which API calls a job may make; see Docker Socket Exposure Risks.

How to choose

• Job runs on its own VM that is destroyed afterwards (GitHub-hosted runners, autoscaled VM runners): use the VM's daemon directly; neither trick is needed.
• Job only builds images: prefer a rootless or remote BuildKit builder, which needs no daemon privileges at all.
• Job must run containers on a shared runner: DinD (ideally via Sysbox or rootless) to keep tenants apart.
• Socket mounting: only on trusted, single-tenant runners, where every job is allowed to be root on the host.

Key takeaways

• DinD runs a nested, privileged daemon per job; it isolates jobs but starts with an empty cache.
• Socket mounting shares the host daemon; it is fast but gives every job root-equivalent control of the host.
• With a mounted socket, -v paths and -p ports refer to the host, which breaks naive bind mounts and causes port collisions.
• For build-only jobs, rootless BuildKit, Buildah, or a remote builder avoid both privileged containers and socket access.
• On ephemeral per-job VMs, use the VM's own daemon and avoid both patterns.

🧪 Practice

1. (Beginner) Start a docker:28-dind container with --privileged, connect to it from a docker:28 container on the same network, and show that docker images lists different images from your host.
2. (Intermediate) In a container with the host socket mounted, reproduce the empty bind-mount problem, then make the same command work using docker cp or by building an image. Explain why each fix works.
3. Interview question: "A colleague proposes mounting /var/run/docker.sock into all CI jobs on the shared runners because DinD builds are slow. What do you say?" Hint: think about what a malicious or compromised job could do with the socket, which tenants share the runner, and how to make DinD or a builder faster with caching instead.

CI Build Caching

Why it matters. On a laptop, the second build of an image takes seconds because BuildKit remembers every step. CI runners are usually ephemeral: each job starts on a fresh machine with an empty cache, so every pipeline reinstalls every dependency and recompiles everything, even for a one-line change. A 2-minute local build becomes a 15-minute pipeline, multiplied by every commit and every developer waiting for it. CI caching is about carrying the right state from one job to the next.

Analogy. An ephemeral runner is a chef who starts every shift in an empty kitchen. Caching is the walk-in fridge where yesterday's prepared stocks and sauces are kept: the chef still cooks, but does not start from raw ingredients every morning.

What can be cached, and how each survives the job

What	Example	Survives an ephemeral runner via
Layer cache (results of Dockerfile steps)	The npm ci layer, the compiled binary	--cache-to / --cache-from export (Registry and Inline Cache)
Cache mounts (RUN --mount=type=cache)	~/.npm, ~/.cache/pip, Go build cache	Not exported; needs a persistent builder or a workaround (below)
Base images	node:22-slim	Registry mirror or pull-through cache close to the runners
The builder itself	A long-running BuildKit daemon	Persistent self-hosted runners, remote builders, Docker Build Cloud

Step 0: make the cache usable. No backend helps if the Dockerfile invalidates the cache early. Copy dependency manifests before source code, keep the build context small with .dockerignore, and avoid steps whose inputs change every build (a timestamp build argument near the top of the file invalidates everything below it). See Layer Ordering and Cache Invalidation.

Choosing a cache backend for CI

Backend	Best for	CI-specific notes
gha	GitHub Actions	Uses the Actions cache service; 10 GB per repository by default, least-recently-used entries evicted; use scope= per image so images do not overwrite each other
registry	Any CI that can push to a registry	Stored as a separate cache image (:buildcache); shared by all runners and branches; works with mode=max
local	CI systems with a directory cache (GitLab cache:, Jenkins workspaces)	The exporter never prunes old entries, so rotate the directory (example below)
inline	Simple setups on the docker driver	mode=min only: multi-stage builder steps are not cached
s3, azblob	Self-hosted runners at scale	Same idea as registry, stored in object storage

Example: GitHub Actions with per-image scopes

- uses: docker/build-push-action@v6
  with:
    context: ./api
    tags: ghcr.io/acme/api:${{ github.sha }}
    push: true
    cache-from: type=gha,scope=api                  # read this image's cache
    cache-to: type=gha,scope=api,mode=max           # write all stages, not only the final ones

The gha backend needs the runner's cache service URL and token. docker/setup-buildx-action and build-push-action expose them automatically; if you call docker buildx build in a plain run: step, add an action such as crazy-max/ghaction-github-runtime first.

Example: registry cache with a branch fallback (any CI)

BRANCH_SLUG="${CI_COMMIT_REF_SLUG:-$(git rev-parse --abbrev-ref HEAD | tr '/' '-')}"
CACHE="registry.example.com/acme/api"

docker buildx build \
  --cache-from "type=registry,ref=$CACHE:cache-$BRANCH_SLUG" \
  --cache-from "type=registry,ref=$CACHE:cache-main" \
  --cache-to   "type=registry,ref=$CACHE:cache-$BRANCH_SLUG,mode=max,ignore-error=true" \
  --tag "$CACHE:sha-$(git rev-parse --short=12 HEAD)" --push .
# 1st --cache-from: this branch's own cache (best match)
# 2nd --cache-from: main's cache, so a brand-new branch is not a cold build
# ignore-error=true: a failed cache export does not fail an otherwise good build

Example: local cache in GitLab CI, rotated so it does not grow forever

build:
  image: docker:28
  services: [docker:28-dind]
  cache:
    key: buildx-$CI_COMMIT_REF_SLUG
    paths: [.buildx-cache/]
  script:
    - docker buildx create --driver docker-container --use
    - >
      docker buildx build
      --cache-from type=local,src=.buildx-cache
      --cache-to type=local,dest=.buildx-cache-new,mode=max
      --tag "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA" --push .
    # replace the old cache with the new one; otherwise stale entries accumulate
    - rm -rf .buildx-cache && mv .buildx-cache-new .buildx-cache

Cache mounts on ephemeral runners. Cache mounts are the biggest win locally (package managers reuse their downloads) but are lost when the runner disappears, because cache export only includes layers. Options:

1. Accept it: the layer that runs npm ci is still cached as long as the lockfile does not change, which covers most builds.
2. Use a persistent builder (self-hosted runner with a long-lived BuildKit, a remote builder, or Docker Build Cloud), where cache mounts survive naturally.
3. Use a workaround that saves the cache mount's contents into the CI cache and injects them before the next build (for example the buildkit-cache-dance action on GitHub).

Measuring the result. Do not guess; check which steps hit the cache:

docker buildx build --progress=plain . 2>&1 | grep -E '^#[0-9]+ (CACHED|DONE)' | sort | uniq -c | head
# The GitHub Actions build summary also reports the cache ratio for each build

Pitfalls

Pitfall	Effect	Fix
Two images write to the same gha scope	They evict each other's cache; hit rate near zero	One scope= per image (and per platform in matrix builds)
mode=min on a multi-stage build	Builder stages always rebuild	mode=max
Base image referenced by a moving tag	New digest invalidates every step after FROM	Pin the tag; update it deliberately (for example with Renovate or Dependabot)
Pull requests from untrusted forks write to the shared cache	Possible cache poisoning of later builds	Write cache only from trusted branches, or rely on platform scoping (GitHub restricts each branch to its own and the default branch's caches)
local cache directory grows without limit	CI cache upload and download become slow	Rotate the directory as in the GitLab example

Key takeaways

• Ephemeral runners start cold; export BuildKit's layer cache with --cache-to and import it with --cache-from.
• Pick the backend that matches the CI system: gha on GitHub Actions, registry anywhere, local with directory caches.
• Use mode=max for multi-stage builds, one cache scope per image, and a main-branch fallback for new branches.
• Cache mounts are never exported; use persistent builders or a workaround if they matter.
• Measure cache hits and fix Dockerfile ordering before tuning backends.

🧪 Practice

1. (Beginner) Add cache-from: type=gha and cache-to: type=gha,mode=max to a GitHub Actions build. Run the workflow twice without code changes and compare the duration and the CACHED steps in the logs.
2. (Intermediate) Set up registry caching with a branch cache and a main fallback. Create a new branch, change one source file, and show from the build log that dependency installation was served from main's cache.
3. Interview question: "Builds on your self-hosted runners are fast, but the same pipeline on cloud-hosted runners takes four times as long. Why, and how would you close the gap?" Hint: consider what persists between jobs on each kind of runner (layers, cache mounts, pulled base images) and which of those can be exported.

Image Tagging Strategies

Why it matters. A tag is the name people and deployment tools use to choose an image, but tags are mutable: the same tag can point to different images over time (see Image Tags). In CI, careless tagging causes real incidents: a rollback that deploys the same broken build, two clusters running "1.4" that are actually different images, or nobody being able to say which commit is running in production. A tagging strategy decides which tags CI creates, which ones may move, and what deployments refer to.

Analogy. Think of tags as labels on shelves in a warehouse. A version label (1.4.2) should be a permanent label glued to one box. A label like "today's special" (latest, staging) is a sign that staff move from box to box. Both are useful, but you should never ship an order based only on the movable sign.

Common tag types in CI

Tag	Example	Moves?	Purpose
Commit SHA	sha-3f9a1c2	Never	Exact traceability from image to code; the default identity of a CI build
Semantic version	1.4.2	Never (by policy)	Human-friendly release identity
Floating minor/major	1.4, 1	Yes, on each patch or minor release	Consumers who want compatible updates automatically
Branch	main, feature-login	Yes, on each push	"Latest build of this branch", for development environments
Pull request	pr-123	Yes, on each push to the PR	Preview environments
Environment or channel	staging, prod	Yes, on each promotion	Human-readable "what is deployed where" (never the source of truth)
latest	latest	Yes	Convenience for local use; avoid in deployments
Build metadata	1.4.2-20261014.1, 1.4.2-build.88	Never	Distinguishing rebuilds of the same version (base image updates)

Recommended baseline

1. Tag every CI build with the commit SHA. This tag never moves.
2. On release tags, add the semantic version and the floating major.minor and major tags.
3. Deploy by digest (image@sha256:...), or at least by an immutable tag, never by a branch or environment tag.
4. Enable tag immutability in the registry for version and SHA tags where supported (Amazon ECR, Harbor, Google Artifact Registry; on Azure Container Registry, lock tags).

Build once, promote by retagging. Promotion from staging to production should not rebuild the image. It points a new tag at the existing manifest, which copies no layers and keeps the digest identical:

# Promote the exact image that passed staging to the release tags, server side (no pull/push of layers)
SRC="ghcr.io/acme/api@sha256:9b2e4c...d71a"
docker buildx imagetools create --tag ghcr.io/acme/api:1.4.2 --tag ghcr.io/acme/api:1.4 "$SRC"

# Across registries (for example to a production registry), copy by digest with skopeo or crane
skopeo copy --all docker://ghcr.io/acme/api@sha256:9b2e4c...d71a \
                  docker://prod.registry.example.com/acme/api:1.4.2   # --all keeps every platform

# Verify: both tags resolve to the same digest
docker buildx imagetools inspect ghcr.io/acme/api:1.4.2 --format '{{.Manifest.Digest}}'

 CI build of 3f9a1c2  ->  ghcr.io/acme/api@sha256:9b2e...   tags: sha-3f9a1c2
                                    |
        tests pass in staging       |  retag (no rebuild)
                                    v
                                    same digest            tags: sha-3f9a1c2, 1.4.2, 1.4, 1
                                    |
        deployment manifests ------>  image: ghcr.io/acme/api@sha256:9b2e...   (immutable reference)

Generating tags automatically. Computing tags in shell scripts is error-prone; docker/metadata-action (see GitHub Actions for Docker) or a small script derives them from Git:

- id: meta
  uses: docker/metadata-action@v5
  with:
    images: ghcr.io/acme/api
    flavor: |
      latest=false                                   # never publish "latest" automatically
    tags: |
      type=sha,format=long,prefix=sha-               # sha-3f9a1c2e... (full SHA, never moves)
      type=semver,pattern={{version}}                # 1.4.2
      type=semver,pattern={{major}}.{{minor}}        # 1.4
      type=semver,pattern={{major}},enable=${{ !startsWith(github.ref, 'refs/tags/v0.') }}   # skip "0" for 0.x
      type=ref,event=branch                          # main
      type=ref,event=pr                              # pr-123

Deploying by digest while keeping tags readable. Tools such as Kustomize, Helm, and Argo CD can store both: the tag for humans and the digest for the runtime.

# Kubernetes Deployment (excerpt): the tag is documentation, the digest is what is pulled
containers:
  - name: api
    image: ghcr.io/acme/api:1.4.2@sha256:9b2e4c...d71a   # when both are present, the digest wins

Housekeeping. SHA, branch, and PR tags accumulate quickly. Configure registry retention rules, for example "keep the last 30 sha-* tags and anything with a semantic version tag; delete untagged manifests older than 14 days". Make sure retention never deletes digests that are still deployed.

Key takeaways

• Tags are mutable pointers; give every CI build an immutable commit SHA tag and add version tags on releases.
• Floating tags (1.4, main, latest, staging) are conveniences for humans, not deployment identifiers.
• Deploy by digest, and enable tag immutability in the registry for version and SHA tags.
• Promote by retagging the same digest (imagetools create, skopeo copy, crane), never by rebuilding.
• Automate tag generation and add retention policies so the registry stays manageable.

🧪 Practice

1. (Beginner) Push an image with a sha-<commit> tag to a registry, then add a 1.0.0 tag to the same digest with docker buildx imagetools create and verify that both tags resolve to the same digest.
2. (Intermediate) Configure metadata-action (or a script) to produce SHA, semantic version, and major.minor tags without latest. Create the Git tags v1.0.0 and v1.0.1 and show that 1.0 moved while 1.0.0 did not.
3. Interview question: "Production runs api:stable. After an incident, the team rolls back by redeploying api:stable, and the bug is still there. What happened, and how should the strategy change?" Hint: think about what a moving tag pointed to at deploy time versus at rollback time, and what an immutable reference would have given you.

Automated Scanning in Pipelines

Why it matters. Vulnerability Scanning and Docker Scout explain how scanners find known vulnerabilities in images. Running a scanner by hand now and then does not protect anything: someone forgets, or the results arrive after the image is already in production. Automating scans in the pipeline turns them into gates (a vulnerable image cannot be published) and monitors (images already deployed are rechecked as new CVEs are published every day).

Analogy. Airport security has several checkpoints: luggage is screened at check-in, passengers at the gate, and some bags again on arrival. Each checkpoint catches different things, and none is enough alone. A pipeline scans the Dockerfile, the built image, the registry, and the running workloads for the same reason.

Checkpoints in a pipeline

  Dockerfile & repo        built image (before push)       registry               running workloads
 +-----------------+      +------------------------+     +-------------------+   +-----------------------+
 | hadolint        |      | vulnerability scan     |     | scan on push      |   | scheduled rescans of  |
 | secret scanning |  ->  |   GATE: fail on        | ->  | continuous rescan |-> | deployed digests      |
 | trivy config    |      |   fixable HIGH/CRIT    |     | of stored images  |   | admission policies    |
 +-----------------+      | SBOM + provenance      |     +-------------------+   | (signed? attested?)   |
   fast, every PR         | PR diff vs production  |                             +-----------------------+
                          +------------------------+

Checkpoint	Catches	Typical tools
Lint the Dockerfile	Unpinned bases, ADD from URLs, running as root, missing --no-install-recommends	hadolint, trivy config, docker build --check
Scan the repository	Secrets committed to Git, vulnerable lockfile dependencies	trivy fs, gitleaks, GitHub secret scanning
Scan the built image (gate)	Known CVEs in OS packages and language dependencies	Trivy, Grype, Docker Scout, Snyk
Attach SBOM and provenance	Nothing directly; enables later scanning and verification	sbom: true, provenance: mode=max, Syft
Registry scanning	New CVEs in already-pushed images	Scan-on-push and continuous scanning in ECR, ACR, Artifact Registry, Harbor, Docker Hub with Scout
Scheduled rescans	New CVEs in what is deployed right now	A nightly CI job scanning production digests
Admission control	Unsigned, unscanned, or policy-violating images	Kyverno, OPA Gatekeeper, Sigstore policy controller

Gate design: what should fail the build? A gate that fails too often gets disabled. A gate that never fails is decoration. A typical, sustainable policy:

Finding	Action
Critical or high severity, fix available	Fail the build
Critical or high severity, no fix available yet	Report, do not fail; track and review
Known exploited (CISA KEV) or high EPSS, any severity	Fail the build, or escalate immediately
Medium and low	Report in the PR and dashboards
Accepted risk (not applicable, mitigated)	Suppress with a reason and an expiry date (.trivyignore, VEX document)
Secrets found in the image	Always fail

Example: GitHub Actions with a scan gate before push. The image is built and loaded into the runner's image store, scanned, and pushed only if the scan passes. The second build is almost free because every step is cached:

jobs:
  image:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
      security-events: write          # upload SARIF to GitHub code scanning
    steps:
      - uses: actions/checkout@v4

      - name: Lint Dockerfile
        run: docker run --rm -i hadolint/hadolint < Dockerfile

      - uses: docker/setup-buildx-action@v3

      - name: Build for scanning (not pushed)
        uses: docker/build-push-action@v6
        with:
          context: .
          load: true                   # into the local image store so the scanner can read it
          tags: ghcr.io/acme/api:scan
          cache-from: type=gha
          cache-to: type=gha,mode=max

      - name: Vulnerability gate
        uses: aquasecurity/trivy-action@0.28.0   # pin third-party actions to a full commit SHA in practice
        with:
          image-ref: ghcr.io/acme/api:scan
          severity: CRITICAL,HIGH
          ignore-unfixed: true         # fail only on findings that have a fix
          exit-code: "1"
          trivyignores: .trivyignore   # documented, expiring exceptions
          format: sarif
          output: trivy.sarif
          limit-severities-for-sarif: true

      - name: Upload results to code scanning
        if: always()                   # also upload when the gate failed
        uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: trivy.sarif

      - uses: docker/login-action@v3
        if: github.event_name != 'pull_request'
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Push (only reached if the gate passed)
        if: github.event_name != 'pull_request'
        uses: docker/build-push-action@v6
        with:
          context: .
          push: true
          tags: ghcr.io/acme/api:sha-${{ github.sha }}
          cache-from: type=gha         # every step is cached: this is a fast rebuild
          sbom: true
          provenance: mode=max

load: true works for single-platform images. For multi-platform builds, either scan each platform separately (--platform with load), scan the pushed image from a staging repository before promoting it, or scan the SBOM attestation.

Showing the change in a pull request. Absolute numbers ("212 vulnerabilities") are hard to act on. A comparison with what runs in production answers the reviewer's real question, "does this PR make things worse?":

- uses: docker/scout-action@v1
  if: github.event_name == 'pull_request'
  with:
    command: compare
    image: local://ghcr.io/acme/api:scan     # the image built in this job
    to: registry://ghcr.io/acme/api:prod     # what is deployed now
    only-severities: critical,high
    write-comment: true                      # posts the diff as a PR comment
    github-token: ${{ secrets.GITHUB_TOKEN }}
    dockerhub-user: ${{ vars.DOCKERHUB_USER }}         # Scout requires a Docker account
    dockerhub-password: ${{ secrets.DOCKERHUB_TOKEN }} # a personal access token, not a password

Rescanning deployed images on a schedule. An image that passed the gate last month may be vulnerable today. A nightly job rescans the digests that are actually deployed and alerts (or opens an issue) when something new and fixable appears:

# .github/workflows/rescan.yml
on:
  schedule:
    - cron: "0 5 * * *"                # daily at 05:00 UTC
  workflow_dispatch: {}

jobs:
  rescan:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        image:
          - ghcr.io/acme/api:prod
          - ghcr.io/acme/web:prod
    steps:
      # private images: add a docker/login-action step here first
      - uses: aquasecurity/trivy-action@0.28.0
        with:
          image-ref: ${{ matrix.image }}
          severity: CRITICAL,HIGH
          ignore-unfixed: true
          exit-code: "1"               # a red scheduled run is the alert

The usual fix is not a code change but a rebuild: rebuilding picks up the patched base image and packages. Pairing scheduled rescans with automated base image update PRs (Renovate, Dependabot) closes the loop.

Keeping scans fast and reliable

• Cache the scanner's vulnerability database between runs (Trivy downloads it on each fresh runner); mirror it internally if runners have restricted internet access.
• Scan the image once and reuse the result: generate an SBOM at build time and scan the SBOM in later stages instead of pulling the image again.
• Run the gate on the exact image you will push. Scanning :latest from the registry in a separate job checks a different artifact.

Key takeaways

• Automate scans at several checkpoints: Dockerfile lint, image gate before push, registry scanning, scheduled rescans, and admission policies.
• Gate on fixable high and critical findings (and known exploited ones); report the rest; suppress only with reasons and expiry dates.
• Build, load, scan, then push; the second build is cheap thanks to the cache.
• Show the vulnerability difference against production in pull requests so reviewers can act on it.
• Rescan deployed digests regularly; the fix is usually a rebuild on an updated base image.

🧪 Practice

1. (Beginner) Add hadolint and a Trivy image scan to a pipeline. Introduce an old base image (for example python:3.9) and confirm that the gate fails; then update the base and confirm that it passes.
2. (Intermediate) Restructure the pipeline so that nothing is pushed unless the scan passes, SARIF results appear in the repository's security tab, and a nightly workflow rescans the production image. Add one documented exception to .trivyignore with a comment and review date.
3. Interview question: "After you enable a vulnerability gate, every build fails because of CVEs in the base image that have no fix. Developers ask you to remove the gate. What do you do?" Hint: distinguish fixable from unfixable findings, consider exploitability data and smaller base images, and think about how exceptions should be recorded and expire.

---

<a id="14-orchestration-and-production"></a>