8. Docker Compose

Real applications are rarely one container: a typical stack has a web server, an API, a database, a cache, and a background worker, each with its own image, ports, volumes, networks, and environment. Docker Compose lets you describe that whole stack in one declarative YAML file and manage it as a unit with a handful of commands. This chapter covers the file format, how each docker run option maps to Compose, the features that make one file serve many environments, and the workflows (development, testing, multi-service deployments) where Compose is the everyday tool.

<a id="81-compose-fundamentals"></a>

8.1 Compose Fundamentals

This section explains what Compose is, how it is installed and run, how a Compose file is organized, and the commands that manage an application's lifecycle.

What Is Docker Compose

The problem. Starting a three-tier application by hand means typing a long sequence of commands, in the right order, with exactly the right options:

docker network create shop
docker volume create shop-db
docker run -d --name db  --network shop -v shop-db:/var/lib/postgresql/data \
  -e POSTGRES_PASSWORD=secret postgres:17
docker run -d --name api --network shop -e DATABASE_URL=postgres://postgres:secret@db/postgres \
  shop/api:1.4
docker run -d --name web --network shop -p 8080:80 shop/web:1.4

The commands are easy to get wrong, hard to share, and leave no record of what the "correct" setup is. Tearing the stack down requires another sequence in reverse. Changing one option means remembering which container to recreate.

The solution. Docker Compose is a tool for defining and running multi-container applications. You describe the desired state (services, networks, volumes, configuration) in a YAML file, usually compose.yaml, and Compose talks to the Docker Engine API to make reality match that description.

Analogy. A docker run command is like giving a builder verbal instructions one wall at a time. A Compose file is the architectural blueprint: anyone can read it, it lives in version control, and the builder can construct the whole house (or rebuild one room) from it.

The same stack as above, expressed declaratively:

# compose.yaml
services:
  db:
    image: postgres:17
    environment:
      POSTGRES_PASSWORD: secret
    volumes:
      - db-data:/var/lib/postgresql/data   # named volume declared below

  api:
    image: shop/api:1.4
    environment:
      DATABASE_URL: postgres://postgres:secret@db/postgres   # "db" resolves via Compose DNS
    depends_on:
      - db

  web:
    image: shop/web:1.4
    ports:
      - "8080:80"
    depends_on:
      - api

volumes:
  db-data:

docker compose up -d      # create network, volume, and all three containers
docker compose down       # stop and remove containers and the network (volume is kept)

Key ideas

Concept	Meaning
Project	One application instance managed by Compose. Named after the directory by default; all its resources share the project name as a prefix and carry a com.docker.compose.project label.
Service	A named definition of how to run one component (image, ports, environment...). Compose creates one or more containers from it.
Desired state	The Compose file says what should exist; Compose computes the create/recreate/remove actions.
Idempotency	Running docker compose up again changes nothing if the file did not change. If one service's configuration changed, only that service's containers are recreated.

What Compose is (and is not)

   compose.yaml  --->  docker compose (CLI plugin)  --->  Docker Engine API  --->  containers,
   (desired state)     computes the diff, orders          (same API as            networks,
                       operations by dependencies          docker run)             volumes

• Compose is a client. It has no daemon of its own; it creates ordinary containers, networks, and volumes that you can also see with docker ps, docker network ls, and docker volume ls.
• Compose targets one Docker host (or whatever host the current context points to). It does not schedule containers across machines, reschedule after a node failure, or perform rolling updates. That is the job of orchestrators such as Swarm and Kubernetes (Chapter 14).
• The file format is the open Compose Specification, also implemented by podman compose, nerdctl compose, and Swarm stacks (with differences).

Typical uses

Use case	Why Compose fits
Local development	One command starts the app plus its databases and queues
Automated tests / CI	Reproducible, disposable environments per test run
Demos and onboarding	git clone && docker compose up is the whole setup guide
Single-host production	Small apps, internal tools, home servers, edge devices

Key takeaways

• Compose turns a sequence of imperative docker commands into one declarative, version-controlled YAML file.
• A project groups services, networks, and volumes; resource names are prefixed with the project name.
• Compose is a client of the Engine API, not a separate runtime; everything it creates is a normal Docker object.
• It manages a single host; multi-node scheduling and self-healing belong to orchestrators.

🧪 Practice

1. (Beginner) Translate three docker run commands you have used before (with ports, a volume, and environment variables) into one compose.yaml, start it with docker compose up -d, and confirm with docker ps that the containers are ordinary Docker containers with project labels.
2. (Intermediate) Run docker compose up -d twice in a row, then change one service's environment variable and run it again. Observe which containers are recreated and explain why the others are left untouched.
3. Interview question: "When would you choose Compose over Kubernetes for running an application in production, and when would that choice become a problem?" Hint: think about the number of hosts, failure handling, rolling updates, and operational complexity.

Compose V2 Plugin

A short history. The original Compose (now called Compose V1) was a separate Python program invoked as docker-compose (with a hyphen). It had its own release cycle, its own dependencies, and occasionally lagged behind Engine features. Compose V2 is a rewrite in Go, shipped as a Docker CLI plugin and invoked as docker compose (with a space). V1 reached end of life in mid-2023 and no longer receives updates; all current Docker installations use V2.

How CLI plugins work. When you type docker compose, the Docker CLI looks for an executable named docker-compose in its plugin directories and runs it, passing along the global configuration (current context, TLS settings, config file). This is why Compose automatically follows docker context use.

   docker compose up
      |
      v
   docker CLI ---- finds plugin ----> ~/.docker/cli-plugins/docker-compose          (per user)
                                      /usr/local/lib/docker/cli-plugins/            (system)
                                      /usr/libexec/docker/cli-plugins/              (packages)
                                          |
                                          v
                                      Compose talks to the daemon of the current context

V1 vs V2 differences that still matter

Aspect	Compose V1	Compose V2
Command	docker-compose	docker compose
Language / distribution	Python, standalone binary or pip	Go, CLI plugin
Container names	project_web_1 (underscores)	project-web-1 (hyphens, valid DNS names)
version: key in the file	Required to select a file format	Obsolete; ignored with a warning
File format	Versions 2.x and 3.x	The unified Compose Specification
BuildKit	Optional	Always used (builds go through Buildx)
Status	End of life (2023)	Actively maintained

The container-name change matters for scripts that parse docker ps output or hard-code names such as myapp_db_1; prefer docker compose ps or label filters instead.

Installing and checking

# Docker Desktop: included. Linux with Docker's repository:
sudo apt-get install docker-compose-plugin      # or: sudo dnf install docker-compose-plugin

docker compose version
# Docker Compose version v2.40.0

# Manual install for a single user (pick the right release and architecture)
mkdir -p ~/.docker/cli-plugins
curl -SL https://github.com/docker/compose/releases/download/v2.40.0/docker-compose-linux-x86_64 \
  -o ~/.docker/cli-plugins/docker-compose
chmod +x ~/.docker/cli-plugins/docker-compose

# Old scripts still call "docker-compose"? Prefer updating them; as a stopgap, a small wrapper works:
# printf '#!/bin/sh\nexec docker compose "$@"\n' | sudo tee /usr/local/bin/docker-compose && sudo chmod +x $_

Key takeaways

• Use docker compose (V2, Go, CLI plugin); docker-compose (V1, Python) is end of life.
• As a CLI plugin, Compose respects Docker contexts and CLI configuration automatically.
• V2 names containers project-service-N with hyphens; avoid hard-coding container names in scripts.
• The top-level version: key is obsolete and should be removed from files.

🧪 Practice

1. (Beginner) Run docker compose version and docker info --format '{{json .ClientInfo.Plugins}}' (or docker info and read the plugin list) to find where the Compose plugin is installed on your machine.
2. (Intermediate) Take an old docker-compose.yml with version: "3.8", run it with docker compose config, note the warning, remove the key, and check that the rendered configuration is identical.
3. Interview question: "A CI pipeline broke after an upgrade because a script ran docker exec myapp_db_1 .... What changed and how would you make the script robust?" Hint: compare V1 and V2 naming, then look at docker compose exec and label-based filters.

compose.yaml Structure

Why a fixed structure. A Compose file describes several kinds of resources: the containers to run, the networks connecting them, the volumes storing their data, and the configuration and secrets they consume. Grouping each kind under its own top-level key keeps the file readable and lets services refer to shared resources by name.

File discovery. Compose looks in the current directory (and then parent directories) for the first of these names:

Priority	File name	Note
1	compose.yaml	Canonical name
2	compose.yml
3	docker-compose.yaml	Legacy name, still supported
4	docker-compose.yml	Legacy name, still supported

If a compose.override.yaml (or docker-compose.override.yml) exists next to it, it is merged automatically (see Multiple Compose Files and Overrides). Use -f <file> to choose files explicitly.

Top-level elements

name: shop                    # project name (optional; default: directory name)

include:                      # pull in other Compose files (8.3)
  - ./monitoring/compose.yaml

services:                     # REQUIRED in practice: the containers to run
  api:
    image: shop/api:1.4

networks:                     # custom networks (a "default" network is created if omitted)
  backend:

volumes:                      # named volumes
  db-data:

configs:                      # non-sensitive configuration files mounted into services
  nginx-conf:
    file: ./nginx.conf

secrets:                      # sensitive data mounted under /run/secrets
  db-password:
    file: ./secrets/db-password.txt

x-common: &common             # extension field: ignored by Compose, used for YAML reuse
  restart: unless-stopped

Element	Purpose	Covered in
name	Project name	Below
services	Container definitions	Services
networks	Networks services attach to	Networks in Compose
volumes	Named volumes	Volumes in Compose
configs, secrets	Files injected into containers	Secrets and Configs in Compose
include	Compose files to import	Extends and Include
x-*	Reusable fragments	Below
version	Obsolete	Remove it

The project name. It determines resource names (shop-api-1, shop_default, shop_db-data) and therefore which containers belong to which project. Precedence, highest first: -p flag, COMPOSE_PROJECT_NAME environment variable, top-level name:, the project directory's name. Two checkouts of the same repository in directories with the same name will collide unless one sets a different project name.

Reuse with YAML anchors and extension fields. YAML anchors (&name) label a node, aliases (*name) reuse it, and the merge key (<<:) copies a mapping's keys into another mapping. Compose ignores any top-level key starting with x-, which makes it a safe place to store anchored fragments.

x-logging: &default-logging           # anchor a fragment
  driver: json-file
  options:
    max-size: "10m"
    max-file: "3"

x-app: &app-defaults
  restart: unless-stopped
  logging: *default-logging           # alias: reuse the whole fragment
  environment:
    TZ: Europe/Rome

services:
  api:
    <<: *app-defaults                 # merge all keys of x-app into this service
    image: shop/api:1.4
  worker:
    <<: *app-defaults
    image: shop/api:1.4
    command: ["python", "-m", "worker"]
    environment:                      # CAUTION: replaces the whole merged "environment" map
      TZ: Europe/Rome
      QUEUE: default

The merge key is shallow: a key defined locally replaces the merged one entirely (it is not deep-merged). Run docker compose config to see the fully resolved file.

YAML pitfalls

Pitfall	Example	Fix
Sexagesimal (base-60) numbers in YAML 1.1 parsers	- 22:22 may parse as an integer	Always quote ports: "22:22"
Booleans in YAML 1.1 tools	restart: no may be read as false	restart: "no"
Tabs	Indentation with tabs	Spaces only
$ in values	password: pa$word is interpolated	Escape as $$ (see Variable Interpolation)

Key takeaways

• Compose discovers compose.yaml automatically and merges compose.override.yaml if present.
• Top-level keys: name, services, networks, volumes, configs, secrets, include, and x-* extensions; version is obsolete.
• The project name prefixes every resource; set it with -p, COMPOSE_PROJECT_NAME, or name:.
• Use x- fields with anchors and aliases for reuse, remembering that <<: merges are shallow.
• docker compose config prints the final, resolved model and catches most syntax mistakes.

🧪 Practice

1. (Beginner) Write a compose.yaml with two services, one named volume, and one custom network. Run docker compose config and identify where Compose added defaults you did not write.
2. (Intermediate) Refactor three services that share restart, logging, and labels settings into an x- fragment with anchors. Verify with docker compose config that the output matches the original file.
3. Interview question: "Two developers clone the same repository into ~/work/app and ~/old/app on the same machine and run docker compose up. What happens, and how do you prevent it?" Hint: consider how the default project name is derived and what resources it prefixes.

Services

What a service is. A service is a named template for running one component of the application. It specifies the image (or how to build it) and everything you would otherwise pass to docker run. Compose creates one container per replica of the service (one by default) and names it <project>-<service>-<index>. Other services reach it by its service name.

Analogy. A service is a job description ("API server: uses this image, listens on 8000, needs these settings"); containers are the employees hired to fill it. You can hire more of them (scale) or replace them (recreate) without rewriting the description.

Mapping docker run to a service

docker run option	Service key
IMAGE	image:
--name	container_name: (avoid; prevents scaling)
COMMAND, --entrypoint	command:, entrypoint:
-p	ports:
-e, --env-file	environment:, env_file:
-v, --mount	volumes:
--network, --network-alias	networks:
--restart	restart:
-u, -w, -h	user:, working_dir:, hostname:
--init	init: true
--read-only, --tmpfs	read_only: true, tmpfs:
--cap-drop, --cap-add	cap_drop:, cap_add:
--memory, --cpus	deploy.resources.limits.memory, deploy.resources.limits.cpus (or mem_limit, cpus)
--health-cmd ...	healthcheck:
--stop-timeout, --stop-signal	stop_grace_period:, stop_signal:
-l	labels:
--log-driver, --log-opt	logging:

Example: a well-defined service

services:
  api:
    image: ghcr.io/acme/shop-api:1.4.2       # pin a version, never rely on "latest"
    command: ["gunicorn", "app:create_app()", "--bind", "0.0.0.0:8000"]   # exec form: no shell
    restart: unless-stopped                   # come back after crashes and daemon restarts
    init: true                                # tiny init as PID 1: reaps zombies, forwards signals
    user: "10001:10001"                       # run as an unprivileged user
    read_only: true                           # immutable root filesystem...
    tmpfs:
      - /tmp                                  # ...with a writable scratch area
    cap_drop: [ALL]
    stop_grace_period: 20s                    # time between SIGTERM and SIGKILL on stop
    deploy:
      resources:
        limits:
          cpus: "1.5"
          memory: 512M
    labels:
      com.acme.team: payments

command and entrypoint forms. Both accept a string (run through a shell parser, not a shell; no pipes or variables expanded at runtime) or a list (exec form, recommended). command replaces the image's CMD; entrypoint replaces the image's ENTRYPOINT and also clears its default CMD.

command: npm run dev                          # string: split into ["npm", "run", "dev"]
command: ["sh", "-c", "echo $$HOSTNAME && exec node server.js"]   # need a shell? ask for one; $$ escapes Compose interpolation

Restart policies

Value	Behavior
"no" (default)	Never restart. Quote it so no YAML tool reads it as a boolean.
always	Always restart; a manually stopped container stays stopped until the daemon restarts, then starts again
on-failure[:N]	Restart only on non-zero exit, optionally at most N times
unless-stopped	Like always, but not if you stopped it explicitly; a common choice for long-running services

Why avoid container_name. A fixed name must be unique on the whole host, so the service cannot be scaled and two projects (say, two branches) cannot run side by side. Other services should use the service name for DNS anyway.

Key takeaways

• A service is a template; Compose creates containers named <project>-<service>-<index> from it.
• Nearly every docker run flag has an equivalent service key.
• Prefer exec-form lists for command/entrypoint, pinned image tags, restart: unless-stopped, and init: true for long-running processes.
• Avoid container_name; address services by service name.
• Quote values that YAML could misinterpret, such as restart: "no" and port mappings.

🧪 Practice

1. (Beginner) Convert this command into a service and run it: docker run -d --name cache --restart unless-stopped -p 6379:6379 --memory 256m redis:7-alpine redis-server --appendonly yes.
2. (Intermediate) Harden a service with read_only, tmpfs, cap_drop: [ALL], and a non-root user. Run it, find what breaks (if anything), and fix it with the minimum extra permissions.
3. Interview question: "What is the difference between overriding command and entrypoint in a Compose service, and when would overriding the entrypoint break an official image?" Hint: think about wrapper scripts like docker-entrypoint.sh that initialize state before running CMD.

Compose CLI Commands

The lifecycle at a glance. Compose commands operate on a project (the current directory's by default) and optionally on specific services. Most mirror a familiar docker command, but act on all matching containers at once.

                 docker compose up -d
   compose.yaml ----------------------> [ created + running ]
                                          |      ^
                     docker compose stop  |      |  docker compose start
                                          v      |
                                        [ stopped ]
                                          |
                     docker compose down  v      (also removes networks; -v removes volumes)
                                        [ removed ]

Global options (placed before the subcommand)

Option	Meaning
-f, --file	Compose file(s) to use; repeatable
-p, --project-name	Project name
--project-directory	Base directory for relative paths and .env
--profile	Enable a profile; repeatable
--env-file	Alternative file(s) for interpolation variables

Core commands

Command	What it does	Useful flags
up	Create/recreate and start services (builds or pulls images if needed)	-d, --build, --wait, --force-recreate, --no-deps, --remove-orphans, --scale svc=N, --watch
down	Stop and remove containers and networks	-v (also volumes), --rmi local|all, --remove-orphans
ps	List the project's containers	-a, --services, --format json
logs	Show aggregated, color-prefixed logs	-f, --tail N, --since 10m, -t
exec	Run a command in a running container	-u, -w, -e, --index N, -T (no TTY)
run	Start a new one-off container for a service	--rm, --service-ports, --no-deps, -e, --entrypoint
build	Build images for services with build:	--no-cache, --pull, --build-arg
pull / push	Pull or push service images	--ignore-pull-failures
config	Validate and print the resolved model	--services, --volumes, --profiles, --format json, -q
start / stop / restart	Lifecycle of existing containers	-t (timeout)
kill, pause, unpause, rm	Same as the docker equivalents, per service
top, port, images, events, cp	Inspection and file copy
ls	List all Compose projects on the host	-a
watch	Sync/rebuild on file changes (8.3)	--no-up
scale	Change replica counts
wait	Block until services' containers exit

exec vs run. This distinction trips up many newcomers:

	docker compose exec api sh	docker compose run --rm api sh
Container	The existing running container	A brand-new container from the service definition
Requires service running	Yes	No (starts dependencies unless --no-deps)
Published ports:	n/a	Not published unless --service-ports
Typical use	Inspect live state, run a quick admin command	Migrations, tests, one-off scripts, a REPL

A typical session

docker compose up -d --build --wait     # build changed images, start everything, wait until healthy
docker compose ps
# NAME          IMAGE          COMMAND                  SERVICE   STATUS                   PORTS
# shop-api-1    shop-api       "gunicorn app:create…"   api       Up 12 seconds (healthy)  8000/tcp
# shop-db-1     postgres:17    "docker-entrypoint.s…"   db        Up 18 seconds (healthy)  5432/tcp
# shop-web-1    nginx:1.27     "/docker-entrypoint.…"   web       Up 11 seconds            0.0.0.0:8080->80/tcp

docker compose logs -f --tail 50 api    # follow one service's logs
docker compose exec db psql -U postgres # shell into the running database
docker compose run --rm api flask db upgrade   # one-off migration in a fresh container

docker compose up -d --no-deps api      # recreate only "api" after changing its config
docker compose config --services        # list service names (handy in scripts)
docker compose ls                       # every Compose project on this host

docker compose down --remove-orphans    # remove containers + networks; keep volumes
# docker compose down -v                # ALSO deletes named volumes: data loss!

How up decides what to recreate. Compose stores a hash of each service's resolved configuration in the container label com.docker.compose.config-hash. On up, it compares hashes: unchanged services are left running, changed ones are recreated, and services removed from the file become orphans (warned about, removed with --remove-orphans). A changed image tag counts as a change; a new image pushed under the same tag does not, so run docker compose pull first (or up --pull always).

Key takeaways

• up -d converges the project to the file; down removes containers and networks; down -v also deletes volumes.
• exec enters a running container; run creates a new one-off container (without published ports by default).
• up --wait blocks until services are running and healthy, which is ideal for scripts and CI.
• Only services whose configuration hash changed are recreated; pull first to pick up new images under the same tag.
• config is the first debugging tool: it shows exactly what Compose will do.

🧪 Practice

1. (Beginner) Start a two-service project, then use ps, logs -f, exec, stop, start, and down in turn, noting what each does to the containers and the network.
2. (Intermediate) Use docker compose run --rm to execute a database migration, then compare docker ps -a before and after with and without --rm. Repeat with --service-ports and check which ports are published.
3. Interview question: "You pushed a fixed image under the same tag and ran docker compose up -d on the server, but the bug is still there. Why?" Hint: consider what Compose compares to decide whether a container must be recreated, and when it pulls images.

<a id="82-configuring-services"></a>

8.2 Configuring Services

This section covers the service keys you will write most often: where the image comes from, how traffic reaches the service, how it is configured, where it stores data, which networks it joins, and how startup is coordinated with health checks.

build vs image

Two ways to get an image. A service needs an image. Either it already exists in a registry (an official database, a release of your own app) and Compose should pull it, or it must be built from source in the repository. image: and build: express these two cases, and combining them expresses a third: "build it, and give the result this name".

The three combinations

Keys present	What Compose does	Resulting image name	Typical use
image only	Pulls the image if it is not present locally	The given name	Third-party services, deployments of released versions
build only	Builds from the context	<project>-<service> (for example shop-api)	Local-only development images
image + build	Uses a local image with that name if present; otherwise tries to pull it and builds if the pull fails. docker compose build always builds and tags it with that name	The given name	Build locally, then docker compose push to a registry

services:
  db:
    image: postgres:17.2                     # pull only

  api:
    build: ./api                             # short form: just the context directory

  web:
    image: ghcr.io/acme/shop-web:${TAG:-dev} # name/tag for the built image
    build:
      context: ./web                         # files sent to the builder
      dockerfile: docker/Dockerfile          # relative to the context
      target: runtime                        # stop at this multi-stage target
      args:                                  # Dockerfile ARG values
        NODE_VERSION: "22"
      cache_from:
        - type=registry,ref=ghcr.io/acme/shop-web:buildcache
      platforms: [linux/amd64, linux/arm64]  # multi-platform (needs a capable builder)
      secrets:
        - npm_token                          # build secret, see 8.4
      additional_contexts:
        shared: ../shared                    # usable as COPY --from=shared ...
    pull_policy: build                       # always build on "up", never pull

Build keys at a glance

Key	Equivalent docker build option
context	Positional context (path or Git URL)
dockerfile / dockerfile_inline	-f / Dockerfile content embedded in YAML
args	--build-arg
target	--target
cache_from / cache_to	--cache-from / --cache-to
ssh, secrets	--ssh, --secret
platforms	--platform
tags, labels	-t (extra tags), --label
no_cache, pull	--no-cache, --pull

When does Compose rebuild? docker compose up does not rebuild an image that already exists just because source files changed; it only builds missing images. Use docker compose up --build (or docker compose build first) after code changes, or let Compose Watch do it.

pull_policy values

Value	Behavior
missing (default)	Pull only if the image is not available locally
always	Pull on every up
never	Never pull; fail if absent
build	Always build (requires build:)
daily, weekly, every_<duration>	Pull again if the local copy is older than that

Pattern: one file, two sources. In development, build from source; in production, pull a released tag. The base file holds image: and an override adds build: (see Multiple Compose Files and Overrides):

# compose.yaml (used everywhere)
services:
  api:
    image: ghcr.io/acme/shop-api:${TAG:-latest}

# compose.override.yaml (loaded automatically on dev machines, absent on servers)
services:
  api:
    build:
      context: ./api
      target: dev

Key takeaways

• image alone pulls; build alone builds and names the image <project>-<service>; both together build and tag with the image name.
• build accepts most docker build options: context, dockerfile, args, target, caches, secrets, SSH, platforms.
• up does not rebuild existing images; use up --build, pull_policy: build, or Compose Watch.
• Splitting image (base file) and build (dev override) lets one project build in development and pull in production.

🧪 Practice

1. (Beginner) Create a service with only build: ., run docker compose up -d, and find the resulting image name with docker compose images. Then add image: myapp:dev and repeat.
2. (Intermediate) Use a multi-stage Dockerfile with dev and runtime targets. Configure Compose so that docker compose up builds dev locally and docker compose -f compose.yaml up (no override) pulls a released image.
3. Interview question: "A developer edits source code, runs docker compose up -d, and sees the old behavior. Explain why and list three ways to fix the workflow." Hint: consider when Compose builds images, --build, bind mounts, and file syncing.

Ports and Expose

Who needs to reach the service? There are two different audiences. Other services in the project reach each other over the project network by service name and container port; no publishing is needed. Clients outside Docker (your browser, another machine) need a port on the host forwarded into the container. ports: serves the second audience; expose: is documentation for the first.

   Browser ---> host:8080 ---(ports: "8080:80")---> web:80
                                                     |
                                    project network  |  api:8000  (no "ports" needed)
                                                     v
                                                    api ---------> db:5432 (no "ports" needed)

Short syntax. [HOST_IP:][HOST_PORT:]CONTAINER_PORT[/PROTOCOL], always quoted:

services:
  web:
    image: nginx:1.27-alpine
    ports:
      - "8080:80"                  # all host interfaces, port 8080 -> container 80
      - "127.0.0.1:8443:443"       # only reachable from the host itself
      - "53:53/udp"                # UDP
      - "9000-9002:9000-9002"      # range
      - "80"                       # random host port -> container 80 (see "docker compose port")

Long syntax. More explicit and easier to merge or generate:

    ports:
      - name: http                 # optional label
        target: 80                 # container port
        published: "8080"          # host port (string allows ranges like "8080-8081")
        host_ip: 127.0.0.1
        protocol: tcp
        app_protocol: http         # informational hint
        mode: host                 # "host" (direct) or "ingress" (Swarm routing mesh)

expose. Lists container ports that the service listens on, without publishing them on the host. It adds metadata (visible in docker inspect) but does not restrict anything: on a user-defined network, every port a container listens on is already reachable by other containers. Think of it as a comment the tooling can read.

  api:
    build: ./api
    expose:
      - "8000"                     # documented for other services; not on the host

Comparison

	ports	expose
Reachable from the host / other machines	Yes	No
Reachable from other services on the network	Yes (via the container port)	Yes (it always was)
Equivalent docker run flag	-p	--expose
Affects scaling	A fixed host port can be bound by only one replica	No

Rules of thumb

• Publish only what humans or external systems need. Databases and internal APIs usually need no ports in production.
• In development, bind tools to localhost: "127.0.0.1:5432:5432". Without a host IP, a published port listens on all interfaces, and on Linux Docker's NAT rules take effect before host firewalls such as ufw (see iptables and Docker in Chapter 7).
• Services always talk to each other on the container port (db:5432), never the published host port.

docker compose port web 80       # which host port maps to web:80?
# 0.0.0.0:8080
docker compose ps --format '{{.Service}} {{.Ports}}'

Key takeaways

• ports publishes container ports on the host; expose only documents ports for other services.
• Inter-service traffic uses service names and container ports, never published ports.
• Quote port mappings and bind development-only ports to 127.0.0.1.
• A fixed host port prevents running more than one replica of a service.

🧪 Practice

1. (Beginner) Run nginx with "8080:80", then change it to "127.0.0.1:8080:80" and verify from another machine (or another network interface) that it is no longer reachable.
2. (Intermediate) Remove all ports from an API service, add expose: ["8000"], and prove with docker compose exec web wget -qO- http://api:8000 that another service can still reach it. Then remove expose and repeat.
3. Interview question: "Your team published PostgreSQL with ports: ['5432:5432'] on a cloud VM protected by ufw, and the database was found by an internet scanner. How did this happen?" Hint: look at where Docker inserts its NAT and filter rules relative to the host firewall's chains.

Environment and env_file

Why environment variables. Following the twelve-factor approach, the same image should run in development, staging, and production; only the configuration differs. Environment variables are the simplest channel for that configuration: database URLs, log levels, feature flags. Compose offers two ways to set them in a container: inline with environment: and from files with env_file:.

environment: map or list syntax

services:
  api:
    image: shop/api:1.4
    environment:                 # map syntax (preferred: one key per line, easy to merge)
      LOG_LEVEL: info
      WORKERS: "4"               # quote numbers and booleans to keep them strings
      FEATURE_X: "true"
      SENTRY_DSN:                # no value: passed through from the shell running compose (unset if absent)

  worker:
    image: shop/api:1.4
    environment:                 # list syntax (equivalent)
      - LOG_LEVEL=debug
      - SENTRY_DSN               # pass-through, same as above

env_file: load variables from files

services:
  api:
    env_file:
      - ./config/common.env      # loaded first
      - path: ./config/local.env # later files override earlier ones
        required: false          # do not fail if the file is missing (Compose 2.24+)

# config/common.env
LOG_LEVEL=info
DATABASE_URL="postgres://app@db:5432/app"   # quotes are removed by Compose's parser
CACHE_URL=redis://cache:6379/${REDIS_DB:-0} # interpolation is supported in env_file values
# comments and blank lines are ignored

Unlike docker run --env-file, Compose's parser understands quotes and ${VAR} expansion inside env_file values.

Two different "env files": the crucial distinction

	.env in the project directory	env_file: in a service
Read by	Compose itself, when parsing compose.yaml	Compose, to set variables inside the container
Purpose	Values for ${VAR} interpolation in the YAML	Container runtime configuration
Reaches the container?	No, unless referenced, e.g. LOG_LEVEL: ${LOG_LEVEL}	Yes, every variable in the file
Selected with	Automatic, or --env-file on the CLI	The env_file: key

   .env  ---(interpolation)--->  compose.yaml  ---(environment / env_file)--->  container env
   shell ---(interpolation)--->      ^
                                     |  env_file: app.env  ------------------->  container env

Precedence inside the container (highest wins)

Rank	Source
1	docker compose run -e KEY=value
2	environment: in the service
3	env_file: (later files override earlier ones)
4	ENV in the image's Dockerfile

Values in ranks 2 and 3 may themselves be interpolated from the shell or .env.

Verifying the result

docker compose config --environment        # variables available for interpolation
docker compose config | sed -n '/api:/,/image:/p'   # resolved environment of "api"
docker compose exec api env | sort         # what the process actually sees
docker compose run --rm -e LOG_LEVEL=debug api env | grep LOG_LEVEL
# LOG_LEVEL=debug

Secrets are not configuration. Environment variables are visible in docker inspect, in /proc/<pid>/environ, in crash dumps, and are inherited by child processes. Use them for non-sensitive settings and prefer Compose secrets for passwords and keys.

Key takeaways

• environment: sets variables inline; env_file: loads them from files; both end up in the container.
• The project's .env file feeds interpolation of the Compose file, not the container, unless a value references it.
• Precedence: run -e > environment > env_file > image ENV.
• A key without a value passes the variable through from the shell running Compose.
• Use config --environment, config, and exec ... env to see what is really set; keep secrets out of environment variables when possible.

🧪 Practice

1. (Beginner) Set LOG_LEVEL in the Dockerfile, in an env_file, and in environment, then print it in the container. Remove the sources one by one and record the value each time.
2. (Intermediate) Put DB_PASSWORD=secret in .env but not in the service definition. Show that the container does not see it, then make it visible with interpolation and, separately, with env_file: .env, and explain the security implication of the second option.
3. Interview question: "What is the difference between the .env file and the env_file attribute in Compose, and why do people confuse them?" Hint: ask who reads each file and at which stage: parsing the YAML or creating the container.

Volumes in Compose

Why declare storage in Compose. Container filesystems are disposable; anything that must survive docker compose up recreating a container (database files, uploads) belongs in a volume, and anything you edit on the host and want to see live in the container (source code in development) belongs in a bind mount. Compose lets you declare both alongside the services that use them, and manages named volumes' lifecycle with the project. (Volume and bind-mount mechanics are covered in Chapter 6.)

Short syntax. SOURCE:TARGET[:MODE]. Compose decides the type from the source:

services:
  db:
    image: postgres:17
    volumes:
      - db-data:/var/lib/postgresql/data         # name -> named volume (must be declared below)
      - ./initdb:/docker-entrypoint-initdb.d:ro  # path (./, ../, /, ~) -> bind mount, read-only

  api:
    build: ./api
    volumes:
      - ./api/src:/app/src                       # bind mount for live code editing
      - /app/node_modules                        # target only -> anonymous volume (shields image content)

volumes:
  db-data:                                       # Compose creates "<project>_db-data"

Relative host paths are resolved from the project directory (normally the directory of the first Compose file), not the current shell directory, so the file works from wherever you run it.

Long syntax. Explicit fields and options the short form cannot express:

    volumes:
      - type: volume
        source: db-data
        target: /var/lib/postgresql/data
        volume:
          nocopy: true            # do not pre-populate an empty volume from the image
          subpath: pg17           # mount only a subdirectory of the volume (Engine 26+)
      - type: bind
        source: ./config
        target: /etc/app
        read_only: true
        bind:
          create_host_path: false # fail if ./config is missing (short syntax silently creates it)
      - type: tmpfs
        target: /tmp
        tmpfs:
          size: 64m               # in-memory, discarded on stop

Top-level volumes: configuring named volumes

volumes:
  db-data:                        # defaults: local driver, name "<project>_db-data"

  shared-media:
    name: media-store             # exact name, no project prefix (shared across projects)

  legacy-data:
    external: true                # created outside Compose; never created or deleted by it
    name: prod_db_data_2019

  nfs-data:
    driver: local
    driver_opts:                  # mount an NFS export as a volume
      type: nfs
      o: "addr=10.0.0.20,rw,nfsvers=4"
      device: ":/exports/app"
    labels:
      com.acme.backup: "daily"

Lifecycle

Command	Containers	Named volumes	External volumes	Anonymous volumes
up (recreating a service)	Replaced	Kept and re-attached	Kept	Re-attached (fresh with -V)
down	Removed	Kept	Kept	Kept (dangling)
down -v	Removed	Deleted	Kept	Deleted

down -v is the most common cause of accidental data loss with Compose. Treat it as "reset my database".

The node_modules trick explained. Bind-mounting ./api over /app hides the node_modules directory installed in the image (and on macOS/Windows the host may have incompatible binaries). Adding an anonymous volume at /app/node_modules places a more specific mount over that path, so the container keeps the image's dependencies while the rest of /app comes from the host.

   /app                <- bind mount from ./api (source code, editable)
   /app/node_modules   <- anonymous volume, seeded from the image (Linux-built dependencies)

Key takeaways

• Short syntax: a name means named volume, a path means bind mount, a lone target means anonymous volume.
• Named volumes must be declared under top-level volumes; Compose prefixes them with the project name unless name: is set.
• external: true marks volumes Compose must neither create nor delete.
• down keeps named volumes; down -v deletes them.
• Use the long syntax for read-only mounts, create_host_path: false, tmpfs sizes, and volume subpaths.

🧪 Practice

1. (Beginner) Run PostgreSQL with a named volume, insert a row, run docker compose down and up again, and confirm the row survives. Then run down -v and confirm it does not.
2. (Intermediate) Build a Node.js service with a bind mount for source code and an anonymous volume for node_modules. Add a dependency to package.json and work out what you must do for the container to see it (and why up --build alone may not be enough; look at -V).
3. Interview question: "How would you share a volume between two separate Compose projects, and what are the risks?" Hint: consider external: true versus name:, which project is allowed to delete it, and concurrent writers.

Networks in Compose

The default you get for free. Without any network configuration, Compose creates one user-defined bridge network called <project>_default and attaches every service to it. Each service is reachable by its service name through Docker's embedded DNS, which is why db:5432 "just works". For many projects, this is all you need.

Why define networks explicitly. Larger applications benefit from segmentation: the reverse proxy should reach the API, but nothing on the public side should reach the database directly. Separate networks enforce that boundary, the same way a building's network separates the guest Wi-Fi from the server room. Explicit networks also let you connect to resources from other projects, set subnets, or cut a network off from the internet.

                    frontend network                      backend network (internal)
   Internet --> [ proxy ] ------------ [ api ] ------------------- [ db ]
     :443                               | both networks             no internet access,
                                        |                           unreachable from proxy

services:
  proxy:
    image: nginx:1.27-alpine
    ports: ["443:443"]
    networks: [frontend]

  api:
    build: ./api
    networks:
      frontend:                      # reachable as "api" from proxy
      backend:
        aliases:
          - api.internal             # extra DNS name on this network only

  db:
    image: postgres:17
    networks: [backend]              # proxy cannot resolve or reach "db"

networks:
  frontend: {}                       # "<project>_frontend", bridge driver
  backend:
    internal: true                   # no route to the outside world

Once a service lists networks:, it is attached only to those networks, not to default.

Top-level network options

Option	Purpose
driver, driver_opts	Driver (bridge default; overlay, macvlan...) and its options
internal: true	No external connectivity (no gateway to the host's outside network)
attachable: true	Allow standalone containers to join (relevant for overlay networks)
ipam	Subnet, IP range, gateway
enable_ipv6	IPv6 support
name	Exact name without the project prefix
external: true	Use an existing network; Compose neither creates nor removes it
labels	Metadata

Per-service network options

services:
  dns:
    image: coredns/coredns:1.12.0
    networks:
      lan:
        ipv4_address: 172.30.0.53    # static IP (requires an ipam subnet on the network)
        aliases: [resolver]

networks:
  lan:
    ipam:
      config:
        - subnet: 172.30.0.0/24
          gateway: 172.30.0.1

Connecting projects together. A common pattern is a shared reverse proxy (Traefik, Caddy, nginx) in one project serving apps from several others. Create the network once and declare it external everywhere:

docker network create edge

# In each application project
services:
  app:
    image: acme/blog:2.1
    networks: [edge, default]        # edge: reachable by the proxy; default: its own db

networks:
  edge:
    external: true                   # must already exist, or "up" fails

Network modes. network_mode replaces network attachment entirely and cannot be combined with networks:.

Value	Effect
host	Use the host's network stack (Linux)
none	Loopback only
service:<name>	Share another service's network namespace (sidecars)
container:<name>	Share a specific container's namespace

Key takeaways

• Compose creates <project>_default and registers each service name as a DNS name on it.
• Explicit networks segment the application; internal: true blocks outbound internet access.
• A service with a networks: list is not attached to default unless listed.
• external: true connects to networks managed outside the project, enabling shared proxies.
• Aliases add DNS names per network; ipv4_address sets static IPs on networks with an IPAM subnet.

🧪 Practice

1. (Beginner) Start a two-service project without any network configuration and use docker network inspect <project>_default to see both containers and their IPs. Resolve one service's name from the other.
2. (Intermediate) Implement the proxy/api/db segmentation above and prove three things: proxy reaches api, proxy cannot resolve db, and db cannot reach the internet.
3. Interview question: "Two Compose projects each define a service called db, and both are attached to a shared external network. What does db resolve to from a third service on that network?" Hint: think about how Docker DNS handles several containers answering to the same name, and how per-network aliases or unique names avoid ambiguity.

depends_on and Startup Order

The problem. The API crashes on startup if the database is not accepting connections yet. Starting containers in the right order is part of the answer, but not all of it: a container being started (its process launched) is not the same as being ready (listening and initialized). PostgreSQL may need several seconds after its process starts, longer on the first run when it initializes the data directory.

Short syntax: order only.

services:
  api:
    depends_on:
      - db           # create and start db before api; stop api before db
      - cache

This guarantees start order and reverse stop order, and makes docker compose up api also start db and cache. It does not wait for db to be ready.

Long syntax: conditions.

services:
  api:
    build: ./api
    depends_on:
      db:
        condition: service_healthy                  # wait until db's healthcheck passes
        restart: true                               # restart api when Compose restarts db
      migrate:
        condition: service_completed_successfully   # wait until the job exits with code 0
      cache:
        condition: service_started                  # same as the short syntax
      tracing:
        condition: service_started
        required: false                             # warn instead of failing if tracing is absent or disabled

  migrate:
    build: ./api
    command: ["flask", "db", "upgrade"]             # one-shot job
    depends_on:
      db:
        condition: service_healthy

  db:
    image: postgres:17
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      retries: 10

Condition	Waits until
service_started	The dependency's container has started (default)
service_healthy	Its healthcheck reports healthy (requires a healthcheck)
service_completed_successfully	It has run to completion with exit code 0

The resulting startup timeline:

   t=0s   db       started ......... starting ......... healthy (t=6s)
   t=6s   migrate                                       started ... exited 0 (t=9s)
   t=9s   api                                                       started

If a dependency becomes unhealthy or the job exits non-zero, up fails with an error such as dependency failed to start: container shop-db-1 is unhealthy.

What depends_on does not do

• It is evaluated only when Compose starts the stack. If the database restarts later (crash, upgrade, network blip), nothing waits; the API must handle lost connections itself (connection retries with backoff, connection pool validation). restart: true only covers restarts performed by Compose commands.
• It does not apply across projects or to resources outside Compose.
• Swarm stacks ignore depends_on; the same application must tolerate any start order there.

Defense in depth. Use depends_on with service_healthy to make local and CI startup deterministic, and make the application retry its connections. The first makes the common case fast and clean; the second makes the system resilient.

Key takeaways

• Short depends_on controls start/stop order and pulls in dependencies, but does not wait for readiness.
• condition: service_healthy waits for a healthcheck; service_completed_successfully sequences one-shot jobs such as migrations.
• restart: true restarts dependents when Compose restarts the dependency; required: false makes a dependency optional.
• Applications must still retry connections, because dependencies can fail after startup.

🧪 Practice

1. (Beginner) Make an API depend on PostgreSQL with the short syntax and an app that connects immediately at startup. Observe the failure on a fresh volume, then fix it with a healthcheck and condition: service_healthy.
2. (Intermediate) Add a migrate job using service_completed_successfully. Make the migration fail on purpose and observe how up reports it and which services start.
3. Interview question: "Why isn't depends_on: condition: service_healthy enough to guarantee the API never sees a database connection error?" Hint: consider events that happen after startup and which component is responsible for them.

Healthchecks in Compose

Why health matters to Compose. A running process may still be useless: a web server stuck in a deadlock, a database still replaying its log. A healthcheck is a command Docker runs periodically inside the container to answer "is this service actually working?". Compose uses the answer for depends_on: condition: service_healthy and for docker compose up --wait, and shows it in docker compose ps. (The Dockerfile HEALTHCHECK instruction and the health state machine are covered in Chapter 5.)

Syntax

services:
  db:
    image: postgres:17
    environment:
      POSTGRES_USER: app
      POSTGRES_DB: app
      POSTGRES_PASSWORD_FILE: /run/secrets/db-password
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]  # $$: evaluated in the container
      interval: 10s          # time between checks once running
      timeout: 3s            # a check taking longer counts as a failure
      retries: 5             # consecutive failures before "unhealthy"
      start_period: 30s      # grace period: failures do not count while starting
      start_interval: 2s     # check frequency during start_period (Engine 25+)

test forms

Form	Example	Notes
CMD list	["CMD", "redis-cli", "ping"]	Executed directly; no shell features
CMD-SHELL list	["CMD-SHELL", "curl -fsS http://localhost:8000/health || exit 1"]	Run with /bin/sh -c; pipes, ||, variables
String	"pg_isready -U app"	Same as CMD-SHELL
["NONE"] or disable: true		Disables a healthcheck inherited from the image

Exit code 0 means healthy and 1 means unhealthy (other codes are reserved; map failures to 1).

The $$ trap. Compose interpolates ${VAR} in the YAML using your shell's environment before creating the container. Writing $${POSTGRES_USER} escapes it so that the literal ${POSTGRES_USER} reaches the container shell, where the container's variable is expanded.

Common healthchecks

Service	test
PostgreSQL	["CMD-SHELL", "pg_isready -U postgres"]
MySQL / MariaDB	["CMD", "mysqladmin", "ping", "-h", "localhost"] / ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"]
Redis	["CMD", "redis-cli", "ping"]
HTTP (image has curl)	["CMD", "curl", "-fsS", "http://localhost:8000/health"]
HTTP (Alpine/BusyBox)	["CMD", "wget", "-q", "--spider", "http://localhost:8000/health"]
Minimal images (distroless)	A small health binary shipped in the image, e.g. ["CMD", "/app/healthcheck"]

The command runs inside the container, so the tool must exist in the image. Many slim and distroless images have no curl or shell; check before writing the test.

Designing a good health endpoint

• Check what the service needs to do its job (can it answer requests, is its own database pool working?), but avoid deep chains: if every service's check calls every dependency, one slow database marks the whole stack unhealthy.
• Keep it fast and cheap; it runs every interval for the lifetime of the container.
• Set start_period to cover the slowest realistic startup (first-run migrations, JIT warmup).

Observing health

docker compose up -d --wait --wait-timeout 120   # exits non-zero if something is not healthy in time
docker compose ps
# NAME         SERVICE  STATUS
# shop-db-1    db       Up 40 seconds (healthy)
# shop-api-1   api      Up 25 seconds (health: starting)

docker inspect --format '{{json .State.Health}}' shop-db-1 | jq '.Status, .Log[-1]'
# "healthy"
# { "Start": "...", "End": "...", "ExitCode": 0, "Output": "/var/run/postgresql:5432 - accepting connections\n" }

What health does not do. Plain Docker and Compose do not restart a container because it is unhealthy. restart: policies react to the process exiting, not to health status. Automatic replacement of unhealthy containers is an orchestrator feature (Swarm, Kubernetes); on a single host, make the process exit on fatal errors or use an external watchdog.

Key takeaways

• A Compose healthcheck overrides the image's HEALTHCHECK; disable: true removes it.
• interval, timeout, retries, start_period, and start_interval tune detection speed versus false alarms.
• Escape container-side variables with $$ so Compose does not interpolate them.
• The test runs inside the container: the tool must exist in the image.
• Health gates depends_on and up --wait, but an unhealthy container is not restarted automatically.

🧪 Practice

1. (Beginner) Add healthchecks to PostgreSQL and Redis services and watch the status change from starting to healthy in docker compose ps.
2. (Intermediate) Write a healthcheck for an HTTP service built on a distroless image (no shell, no curl). Compare two solutions: a dedicated health binary copied in a multi-stage build, and a check implemented in the application's own language.
3. Interview question: "A service's healthcheck calls the database, the cache, and two downstream APIs. During a short cache outage, the whole stack becomes unhealthy and up --wait fails in CI. What is wrong with the design?" Hint: distinguish liveness ("am I working?") from readiness of dependencies, and think about cascading failures.

<a id="83-advanced-compose"></a>

8.3 Advanced Compose

This section covers the features that let one Compose setup serve many environments and team members: variables, layered files, optional services, reuse across files, multiple replicas, and automatic syncing of code changes.

Variable Interpolation

Why interpolation. Hard-coding an image tag, a port, or a hostname in compose.yaml forces you to edit the file for every environment or release. Interpolation lets the file contain placeholders such as ${TAG} that Compose replaces with values from the environment when it reads the file. The file stays generic and committed; the values change per machine, per CI run, or per deployment.

Where values come from (highest priority first)

Rank	Source
1	Variables in the shell that runs docker compose
2	Files passed with --env-file (repeatable; later files override earlier ones), or if none is given, the .env file in the project directory
3	Defaults written in the expression itself (${TAG:-latest})

Interpolation happens once, on the client, while Compose parses the file. The container never sees these variables unless a value passes them on (see Environment and env_file).

Syntax

Expression	Result
$VAR or ${VAR}	Value of VAR; empty string (with a warning) if unset
${VAR:-default}	default if VAR is unset or empty
${VAR-default}	default only if VAR is unset
${VAR:?message}	Error with message if VAR is unset or empty
${VAR?message}	Error if VAR is unset
${VAR:+alt}	alt if VAR is set and non-empty, otherwise empty
${VAR+alt}	alt if VAR is set, otherwise empty
${A:-${B:-x}}	Nested defaults
$$	A literal $ (no interpolation)

Bash-only features such as ${VAR/old/new}, ${#VAR}, or arithmetic are not supported.

Example

# compose.yaml
name: ${PROJECT:-shop}

services:
  api:
    image: ghcr.io/acme/shop-api:${TAG:?TAG must be set, e.g. TAG=1.4.2}   # fail fast in CI
    ports:
      - "${API_BIND:-127.0.0.1}:${API_PORT:-8000}:8000"
    environment:
      LOG_LEVEL: ${LOG_LEVEL:-info}            # shell/.env value forwarded into the container
      SENTRY_ENVIRONMENT: ${ENVIRONMENT:-dev}
      DEBUG_FLAGS: ${DEBUG:+--verbose --trace} # only when DEBUG is set
      PASSWORD_HINT: "starts with $$"          # literal "$"

# .env (project directory; usually git-ignored, with a committed .env.example)
TAG=1.4.2
API_PORT=8081
COMPOSE_PROFILES=debug          # Compose's own settings can live here too

Checking the result. Always render the final file before trusting it:

docker compose config | grep -A3 'image:'
#     image: ghcr.io/acme/shop-api:1.4.2

TAG= docker compose config
# error while interpolating services.api.image: required variable TAG is missing a value: TAG must be set, e.g. TAG=1.4.2

docker compose --env-file .env --env-file .env.staging config   # layered env files
docker compose config --environment     # list variables Compose sees for interpolation

Common mistakes

Symptom	Cause	Fix
WARN The "X" variable is not set. Defaulting to a blank string.	Variable missing	Add it to .env, or use :- / :?
A value in .env is ignored	The shell exports the same variable (shell wins)	unset VAR or check env | grep VAR
A password with $ is truncated	$ was interpolated	Escape as $$, or load it from a secret file
A shell variable in command or healthcheck is empty	Compose interpolated it using the host's environment	Use $$VAR so the container shell expands it
.env not found	It is resolved from the project directory, not the current directory	Use --project-directory or --env-file

Key takeaways

• ${VAR} placeholders are replaced by Compose on the client, from the shell environment first, then .env (or --env-file).
• :- supplies defaults, :? makes variables mandatory, :+ substitutes only when set; $$ escapes a literal dollar.
• .env feeds interpolation (and COMPOSE_* settings), not the container directly.
• docker compose config shows the fully interpolated file and is the quickest way to debug variables.

🧪 Practice

1. (Beginner) Parameterize an image tag and a host port with defaults, then override them with .env, then with a shell variable, and record which value wins in each case with docker compose config.
2. (Intermediate) Make a CI-oriented file in which TAG is mandatory with :?, ports are optional with :-, and a debug flag appears only with :+. Write a short script that tests all three behaviors using docker compose config -q and its exit code.
3. Interview question: "A developer's database password Pa$sw0rd works with docker run -e but authentication fails under Compose. What is happening?" Hint: think about which program sees the $ first and what it does with $sw0rd.

Multiple Compose Files and Overrides

Why layer files. Development, CI, and production share most of a stack's definition but differ in details: development builds from source and mounts code, CI adds a test runner, production pins image tags, adds resource limits, and removes debug ports. Copying the whole file per environment leads to drift. Instead, Compose merges several files: a common base plus a small environment-specific layer.

Analogy. Think of transparent overhead sheets stacked on a projector: the base sheet draws the whole picture, and each sheet on top adds or covers specific details.

How files are selected

Method	Files used
No -f	compose.yaml + compose.override.yaml (if present), automatically
-f a.yaml -f b.yaml	Exactly these, merged in order (later wins); the automatic override is not loaded
COMPOSE_FILE=a.yaml:b.yaml	Same as -f (separator : on Linux/macOS, ; on Windows, configurable with COMPOSE_PATH_SEPARATOR)

   compose.yaml            base: services, networks, volumes (shared by all environments)
      + compose.override.yaml   dev: build from source, bind mounts, debug ports   (automatic)
      + compose.ci.yaml         CI: test runner, tmpfs database                     (-f)
      + compose.prod.yaml       prod: pinned tags, limits, restart policies         (-f)

Example

# compose.yaml (base)
services:
  api:
    image: ghcr.io/acme/shop-api:${TAG:-latest}
    environment:
      LOG_LEVEL: info
    depends_on: [db]
  db:
    image: postgres:17
    volumes:
      - db-data:/var/lib/postgresql/data
volumes:
  db-data:

# compose.override.yaml (development; loaded automatically)
services:
  api:
    build: ./api
    ports: ["127.0.0.1:8000:8000"]
    volumes:
      - ./api/src:/app/src
    environment:
      LOG_LEVEL: debug              # merged by key: replaces only LOG_LEVEL
  db:
    ports: ["127.0.0.1:5432:5432"]

# compose.prod.yaml
services:
  api:
    restart: unless-stopped
    deploy:
      resources:
        limits: { cpus: "2", memory: 1G }
    environment:
      SENTRY_DSN: ${SENTRY_DSN:?required in production}

docker compose up -d                                          # dev: base + override
docker compose -f compose.yaml -f compose.prod.yaml up -d     # prod: base + prod (no dev override)
docker compose -f compose.yaml -f compose.prod.yaml config    # always preview the merge

Merge rules

Kind of field	Examples	Merge behavior
Single values	image, restart, user, build.target	Later file replaces
Maps	environment, labels, deploy, build.args, healthcheck	Merged key by key
Mount lists	volumes, devices, secrets, configs	Merged by container path (target); later wins on conflict
Other lists	ports, expose, dns, cap_add	Combined
Commands	command, entrypoint, healthcheck.test	Replaced entirely

Removing or replacing instead of merging. Since lists like ports are combined, an override cannot remove a port by omission. Compose provides two YAML tags for this:

# compose.prod.yaml
services:
  db:
    ports: !reset []               # remove all ports defined in earlier files
  api:
    environment: !override         # replace the whole map instead of merging it
      LOG_LEVEL: warn

Path resolution. Relative paths in every file are resolved from the project directory, which by default is the directory of the first file. Keep all Compose files in the same directory, or set --project-directory, to avoid surprises.

Key takeaways

• Without -f, Compose merges compose.yaml and compose.override.yaml automatically; with -f, only the listed files are used, in order.
• Single values are replaced, maps merged by key, mounts merged by target, ports combined, commands replaced.
• !reset removes a value from earlier files; !override replaces it instead of merging.
• Paths are relative to the first file's directory; docker compose config shows the merged result.

🧪 Practice

1. (Beginner) Create a base file and a compose.override.yaml that adds a port and a bind mount. Compare docker compose config with docker compose -f compose.yaml config.
2. (Intermediate) Write compose.prod.yaml that removes all published ports from the database with !reset, replaces the API's environment with !override, and adds resource limits. Verify each change in the merged output.
3. Interview question: "Why might a production server accidentally run with development settings, and how would you structure the files and commands to prevent it?" Hint: think about the automatic override file, what happens when it exists on the server, and making the production command explicit (-f or COMPOSE_FILE).

Profiles

The problem. Some services are not always needed: an admin UI for the database, a mail catcher, a load generator, a debugging toolbox, or a heavy search engine that only one team uses. Keeping them in a separate file fragments the configuration; leaving them in the main file starts them for everyone. Profiles solve this by tagging optional services so they start only on request.

Rules

• A service without profiles is always enabled.
• A service with profiles is enabled only when one of its profiles is active.
• Activate profiles with --profile <name> (repeatable) or COMPOSE_PROFILES=a,b (in the shell or .env). --profile "*" enables all.
• Naming a service explicitly on the command line (docker compose up adminer, docker compose run seed) enables that service even if its profile is not active, but not other services in that profile.

services:
  api:                                   # core: always started
    build: ./api
    depends_on: [db]

  db:
    image: postgres:17

  adminer:                               # database UI, only for those who want it
    image: adminer:4
    ports: ["127.0.0.1:8081:8080"]
    profiles: [tools]

  mailpit:                               # fake SMTP server for local email testing
    image: axllent/mailpit:v1.21
    ports: ["127.0.0.1:8025:8025"]
    profiles: [tools, email]

  seed:                                  # one-off job: load demo data
    build: ./api
    command: ["python", "-m", "scripts.seed"]
    depends_on: [db]
    profiles: [seed]

  k6:                                    # load testing
    image: grafana/k6:0.55.0
    command: ["run", "/scripts/load.js"]
    volumes: ["./load:/scripts:ro"]
    profiles: [load]

docker compose up -d                         # api, db
docker compose --profile tools up -d         # api, db, adminer, mailpit
COMPOSE_PROFILES=email docker compose up -d  # api, db, mailpit
docker compose run --rm seed                 # explicitly targeted: runs seed (and its dependency db)
docker compose config --profiles             # list all profiles defined in the file
# email
# load
# seed
# tools

docker compose --profile tools down          # include profiled services when tearing down

Dependencies and profiles. A service that is always enabled must not depends_on a service that is only in a profile (unless the dependency has required: false), or Compose reports that the dependency is disabled. Dependencies should point "inward": optional services depend on core services, not the other way around.

Profiles vs separate files

Need	Use
Optional extra services in the same environment	Profiles
Different settings for the same services per environment	Override files
Separate, independently owned stacks	Separate projects or include

Key takeaways

• Profiles mark optional services; services without profiles always start.
• Activate with --profile or COMPOSE_PROFILES; targeting a service by name enables it.
• Core services must not depend on profiled services.
• Remember to pass the same profiles to down, or those containers are left running.

🧪 Practice

1. (Beginner) Add an adminer service under a tools profile and check that docker compose up -d does not start it while docker compose --profile tools up -d does.
2. (Intermediate) Create a seed job in a profile, run it with docker compose run --rm seed, and then deliberately make a core service depend on it. Read the error and fix the design.
3. Interview question: "Your Compose file has grown to 25 services and new developers complain that it takes 10 minutes and 12 GB of RAM to start. How would you restructure it without splitting the repository?" Hint: group optional services into profiles, choose a sensible default set, and consider COMPOSE_PROFILES in a personal .env.

Extends and Include

Two kinds of reuse. As projects grow, two needs appear. First, several services share most of their configuration (an API and its workers use the same image, environment, and logging), and you want to define it once. Second, an application is built from parts owned by different teams (the payment service, the monitoring stack), and you want to compose them without copying their files. extends addresses the first; include addresses the second.

extends: inherit a service definition

# common.yaml
services:
  python-base:
    image: ghcr.io/acme/shop-api:${TAG:-latest}
    environment:
      DATABASE_URL: postgres://app@db/app
      LOG_FORMAT: json
    restart: unless-stopped

# compose.yaml
services:
  api:
    extends:
      file: common.yaml            # omit "file" to extend a service in the same file
      service: python-base
    command: ["gunicorn", "app:app", "-b", "0.0.0.0:8000"]
    ports: ["8000:8000"]

  worker:
    extends:
      file: common.yaml
      service: python-base
    command: ["celery", "-A", "app", "worker"]
    environment:
      CONCURRENCY: "4"             # merged with the inherited environment

  db:
    image: postgres:17

extends copies the referenced service's configuration and applies the local keys on top using the same merge rules as override files. It copies only the service: networks, volumes, and other top-level resources the base service refers to are not imported and must be declared in the extending file. Compared with YAML anchors, extends works across files, and its merge is deep rather than shallow.

include: import whole Compose files

# compose.yaml (application team)
include:
  - ./infra/db/compose.yaml                     # short syntax: path
  - path: ../monitoring/compose.yaml            # long syntax
    project_directory: ../monitoring            # resolve its relative paths from there
    env_file: ../monitoring/.env                # its own interpolation variables

services:
  api:
    build: .
    depends_on: [db]                            # "db" is defined in the included file

Each included file is loaded as its own Compose application, with its own directory for relative paths and its own .env, and its services, networks, and volumes are then added to the including project. Everything runs as a single project. If an included resource has the same name as a local one, Compose reports a conflict instead of silently merging them; use an override file if you need to modify included services.

Comparison of reuse mechanisms

Mechanism	Scope	Relative paths	Typical use
YAML anchors / x- fields	Within one file	n/a	Small repeated fragments (logging, labels)
extends	One service, across files	From the file being processed	Base service templates
Multiple -f files	Whole model, merged	From the first file	Per-environment overrides
include	Whole files, kept separate	From each included file's own directory	Composing stacks owned by different teams

Key takeaways

• extends inherits one service's configuration (from the same or another file) and merges local keys on top.
• extends does not import top-level networks or volumes; declare them in the extending file.
• include imports whole Compose files as self-contained units with their own paths and .env, then runs them as one project.
• Name conflicts with included resources are errors, not merges.

🧪 Practice

1. (Beginner) Define a base service and use extends to create api and worker services that differ only in command. Inspect the result with docker compose config.
2. (Intermediate) Move a monitoring stack (Prometheus and Grafana with bind-mounted configuration) into its own directory with its own compose.yaml, then include it from the main project. Verify that its relative paths still work and that docker compose ps shows all services in one project.
3. Interview question: "When would you choose include over multiple -f files?" Hint: consider who owns each file, where relative paths are resolved from, and whether you want to modify or just reuse the imported services.

Scaling Services

Why run several containers of one service. A background worker processing a queue goes faster with more consumers; a stateless API can handle more concurrent requests with more processes; testing how an application behaves with several instances catches bugs (sessions in memory, local file uploads) before production. Compose can run N replicas of a service on a single host.

How to scale

services:
  worker:
    build: ./api
    command: ["celery", "-A", "app", "worker"]
    deploy:
      replicas: 3                 # default number of containers for this service

docker compose up -d                          # starts worker-1, worker-2, worker-3
docker compose up -d --scale worker=5         # adds worker-4 and worker-5
docker compose up -d --scale worker=2         # removes the highest-numbered containers
docker compose scale worker=4                 # change replicas without other "up" side effects
docker compose ps worker
docker compose exec --index 2 worker celery -A app inspect active   # target replica #2
docker compose logs worker                    # interleaved, prefixed with worker-1, worker-2...

What prevents scaling

Setting	Problem	Fix
container_name: x	Names must be unique	Remove it
ports: ["8000:8000"]	Only one container can bind host port 8000	Remove the host port, use a range ("8000-8004:8000"), or put a proxy in front
Writable bind mount or volume shared by replicas	Concurrent writes, lock files	Use a database or object storage for shared state
State in the container (sessions, uploads)	Requests hit different replicas	Externalize state (Redis, database, S3)

Load balancing on a single host. All replicas answer to the same service name. Docker's embedded DNS returns all their IPs, so clients that resolve the name for each new connection spread the load roughly round-robin. This is simple but not health-aware, and many clients cache DNS results.

                      DNS "api" -> 172.20.0.4, 172.20.0.5, 172.20.0.6
   proxy:80 ---------------+----------------+----------------+
                           v                v                v
                       api-1:8000       api-2:8000       api-3:8000

A reverse proxy in front is the usual solution. Note that nginx resolves upstream names once at startup unless told otherwise, so it would miss replicas added later:

# nginx.conf (inside the "proxy" service)
resolver 127.0.0.11 valid=10s;          # Docker's embedded DNS; re-resolve every 10 s
server {
    listen 80;
    location / {
        set $upstream http://api:8000;  # a variable forces runtime resolution
        proxy_pass $upstream;
    }
}

Proxies with Docker integration (Traefik, Caddy with plugins) discover replicas through labels and the Docker API instead.

Limits of Compose scaling. All replicas run on one machine, so they share its CPU, memory, and failure fate. There is no rolling update (recreating a scaled service restarts its containers) and no automatic rescheduling. For more than one host or zero-downtime deployments, use Swarm or Kubernetes (Chapter 14).

Key takeaways

• Scale with deploy.replicas, up --scale svc=N, or docker compose scale.
• Remove container_name and fixed host ports from services you want to scale.
• Replicas share the service's DNS name; DNS round-robin is basic and not health-aware.
• Scaled services must be stateless; put a DNS-aware or Docker-aware proxy in front for HTTP.
• Compose scaling is single-host only, with no rolling updates.

🧪 Practice

1. (Beginner) Scale a traefik/whoami service to three replicas and call it repeatedly from another container with wget -qO- http://whoami, observing the different hostnames in the responses.
2. (Intermediate) Put nginx in front of a scaled API. Scale from 2 to 4 replicas and check whether nginx sends traffic to the new ones, first with a static proxy_pass http://api:8000 and then with the resolver configuration above.
3. Interview question: "Your team scaled a Compose service to three replicas and users started getting logged out randomly. What is the likely cause?" Hint: consider where session state lives and how requests are distributed among replicas.

Compose Watch for Development

The problem with rebuild loops. The inner development loop is "edit code, see the result". Rebuilding an image and recreating the container for every change is far too slow. Bind mounts are the traditional answer, but they have drawbacks: on Docker Desktop they cross a VM boundary and can be slow, file-change events are not always delivered, host and container may disagree about file ownership, and a bind-mounted directory hides what the image installed (hence tricks like the anonymous node_modules volume). Compose Watch takes a different approach: it watches files on the host and, depending on rules you define, copies changed files into the running container, restarts it, or rebuilds the image.

Configuration. Rules live under develop.watch in a service:

services:
  web:
    build: .
    command: ["npm", "run", "dev"]          # dev server with its own hot reload
    ports: ["127.0.0.1:3000:3000"]
    develop:
      watch:
        - action: sync                       # copy changed files into the container
          path: ./src
          target: /app/src
          ignore:
            - "**/*.test.ts"                 # patterns relative to "path"
        - action: rebuild                    # dependency changes need a new image
          path: package.json
        - action: rebuild
          path: package-lock.json
        - action: sync+restart               # copy, then restart the container
          path: ./config/app.yaml
          target: /app/config/app.yaml
        - action: sync+exec                  # copy, then run a command in the container
          path: ./templates
          target: /app/templates
          exec:
            command: ["kill", "-HUP", "1"]   # e.g. ask the server to reload

Action	What happens on change	Use for
sync	Files copied into the running container	Source code reloaded by the framework (Vite, nodemon, Flask/Django reload)
rebuild	Image rebuilt and container recreated (like up --build)	Dependency manifests, Dockerfile-relevant files
sync+restart	Files copied, then the container restarted	Configuration read only at startup
sync+exec	Files copied, then a command executed in the container	Triggering a reload without a restart

Running it

docker compose watch               # start services (if needed) and watch for changes
docker compose up --watch          # same, with aggregated logs in the foreground
# watch enabled
# Syncing service "web" after 1 changes were detected
# Rebuilding service(s) ["web"] after changes were detected...

Requirements and behavior

• The service needs build: (watch is for code you build), and the image must include basic tools used for syncing (stat, mkdir, rm).
• The container's USER must be able to write to the target paths; chown the application directory in the Dockerfile.
• Paths matched by .dockerignore are ignored, as well as ignore: patterns; version-control directories are skipped automatically.
• Files are synced one way only (host to container). Files generated in the container (for example a lock file written by npm install) do not come back to the host.

Bind mounts vs Compose Watch

	Bind mount	Compose Watch
Mechanism	Same files shared live	Changed files copied on demand
Direction	Two-way	Host to container
Docker Desktop performance	Can be slow for large trees	Fast (container uses its native filesystem)
Image content under the path	Hidden by the mount	Preserved (e.g. node_modules from the image)
Dependency changes	Manual rebuild	rebuild action handles it
Generated files visible on host	Yes	No

Key takeaways

• develop.watch rules map host paths to actions: sync, rebuild, sync+restart, sync+exec.
• Use sync for code with framework hot reload, rebuild for dependency manifests, sync+restart for startup-only configuration.
• Start it with docker compose watch or docker compose up --watch.
• Watch avoids many bind-mount problems but is one-way; the container user must be able to write the targets.

🧪 Practice

1. (Beginner) Add a sync rule to a Node.js or Python web app with hot reload, run docker compose watch, edit a source file, and confirm the change appears without a rebuild.
2. (Intermediate) Add rebuild for the dependency manifest and sync+restart for a configuration file. Add a new dependency and change the configuration, then verify from the watch output that the correct action ran for each change.
3. Interview question: "Your team's Compose setup on macOS uses bind mounts and the dev server takes 20 seconds to notice changes. What are your options?" Hint: compare file-sharing performance and change notification across the VM boundary, Compose Watch, and limiting what is mounted.

<a id="84-compose-workflows"></a>

8.4 Compose Workflows

This section puts the previous features together into the workflows where Compose is used every day: development environments, complete multi-service applications, automated integration tests, and safe handling of secrets and configuration files.

Local Development Environments

Goals. A good Compose-based development environment gives every developer the same stack with one command, keeps the edit-and-see loop fast, and stays close enough to production that "works on my machine" means something. Concretely:

Goal	Compose technique
One-command onboarding	git clone && cp .env.example .env && docker compose up
Fast feedback	Bind mounts or Compose Watch plus the framework's hot reload
Production parity	Same base file and images as production; differences isolated in compose.override.yaml
Realistic data	Seed scripts and init directories for databases
Debuggability	Debugger ports, log levels, admin tools under profiles
No conflicts between projects	Unique project names, ports bound to 127.0.0.1, no container_name

A typical repository layout

shop/
├── compose.yaml              # base: shared by dev, CI, production
├── compose.override.yaml     # dev-only additions (loaded automatically)
├── .env.example              # documented variables, committed
├── .env                      # personal values, git-ignored
├── api/
│   ├── Dockerfile            # multi-stage: dev and runtime targets
│   └── src/
├── web/
│   ├── Dockerfile
│   └── src/
└── db/
    └── init/                 # *.sql / *.sh run on first database start
        └── 01-schema.sql

The development override

# compose.override.yaml
services:
  api:
    build:
      context: ./api
      target: dev                                   # stage with dev dependencies and tools
    command: ["python", "-m", "debugpy", "--listen", "0.0.0.0:5678",
              "-m", "flask", "run", "--host", "0.0.0.0", "--reload"]
    ports:
      - "127.0.0.1:8000:5000"                       # app
      - "127.0.0.1:5678:5678"                       # debugger (attach from the IDE)
    environment:
      FLASK_DEBUG: "1"
      LOG_LEVEL: debug
    develop:
      watch:
        - { action: sync, path: ./api/src, target: /app/src }
        - { action: rebuild, path: ./api/requirements.txt }

  web:
    build: { context: ./web, target: dev }
    ports: ["127.0.0.1:3000:3000"]
    develop:
      watch:
        - { action: sync, path: ./web/src, target: /app/src }
        - { action: rebuild, path: ./web/package.json }

  db:
    ports: ["127.0.0.1:5432:5432"]                  # connect with a desktop SQL client
    volumes:
      - ./db/init:/docker-entrypoint-initdb.d:ro   # runs only when the data volume is empty

  mailpit:
    image: axllent/mailpit:v1.21
    ports: ["127.0.0.1:8025:8025"]
    profiles: [tools]

Daily commands

docker compose up --watch                     # start everything and sync code changes
docker compose logs -f api                    # follow one service
docker compose exec api pytest -x             # run tests inside the running container
docker compose run --rm api flask db migrate  # generate a migration (files appear via bind mount or "docker compose cp")
docker compose down -v && docker compose up -d   # reset the database (re-runs db/init scripts)

Pitfalls

Problem	Cause	Remedy
Files created in the container are owned by root on the host (Linux)	Container runs as root on a bind mount	Run as your UID (user: "${UID}:${GID}" with values exported or set in .env) or use watch
Port already in use	Another project or a local service uses the port	Parameterize ports with ${API_PORT:-8000}
Changes to dependencies not picked up	Image not rebuilt	rebuild watch action or up --build
Init scripts did not run	Data volume already initialized	down -v to reset, or run the script manually
Slow on macOS/Windows	Bind mount across the VM boundary	Compose Watch, smaller mounts, named volumes for dependencies

Key takeaways

• Keep a production-like base file and put development conveniences in compose.override.yaml.
• Combine a dev build target, hot reload, and bind mounts or Compose Watch for a fast loop.
• Bind development ports to 127.0.0.1 and parameterize them to avoid conflicts.
• Use init directories and seed jobs for data; down -v resets the environment.
• Put optional tools behind profiles so the default stack stays light.

🧪 Practice

1. (Beginner) Create a development setup for a small web API with PostgreSQL, where docker compose up gives a working app with hot reload and a database initialized from a SQL file.
2. (Intermediate) Add remote debugging (debugpy for Python or --inspect=0.0.0.0:9229 for Node.js), attach from your IDE, and hit a breakpoint triggered by a browser request.
3. Interview question: "How do you keep a Compose development environment from drifting away from production over time?" Hint: think about shared base files, multi-stage Dockerfiles with shared stages, the same image versions, and running CI against the production configuration.

Multi-Container Applications

Thinking in services. A multi-container application splits responsibilities into processes that each do one job and can be replaced, scaled, and configured independently. Compose is the place where the relationships between them become explicit: who talks to whom (networks), who stores what (volumes), who waits for whom (dependencies and health), and who is exposed to the outside (ports).

Reference architecture

                       +-----------+
   Internet :80/:443 ->|   proxy   |  TLS, routing, static files
                       +-----+-----+
                frontend     |  network
              +--------------+---------------+
              v                              v
        +-----------+                  +-----------+
        |    web    |                  |    api    |---------------+
        +-----------+                  +-----+-----+               |
                                 backend     |  network (internal) |
                     +-----------------------+---------+           |
                     v                       v         v           v
               +-----------+           +-----------+  +-----------+
               |    db     |<----------|  worker   |  |   cache   |
               +-----------+           +-----------+  +-----------+
               volume: db-data         (no ports)     volume: cache-data

The Compose file

name: shop

x-app: &app                                    # shared settings for api and worker
  image: ghcr.io/acme/shop-api:${TAG:?set TAG}
  restart: unless-stopped
  environment:
    DATABASE_HOST: db
    DATABASE_USER: shop
    DATABASE_PASSWORD_FILE: /run/secrets/db-password   # the app reads the password from a file
    REDIS_URL: redis://cache:6379/0
  secrets: [db-password]
  networks: [backend]
  logging:
    driver: json-file
    options: { max-size: "10m", max-file: "5" }

services:
  proxy:
    image: caddy:2.9-alpine
    restart: unless-stopped
    ports: ["80:80", "443:443"]
    configs:
      - source: caddyfile
        target: /etc/caddy/Caddyfile
    volumes:
      - caddy-data:/data                       # TLS certificates
    networks: [frontend]
    depends_on:
      api: { condition: service_healthy }
      web: { condition: service_started }

  web:
    image: ghcr.io/acme/shop-web:${TAG:?set TAG}
    restart: unless-stopped
    networks: [frontend]

  api:
    <<: *app
    networks: [frontend, backend]              # reachable by proxy, can reach db/cache
    healthcheck:
      test: ["CMD", "wget", "-q", "--spider", "http://localhost:8000/health"]
      interval: 10s
      start_period: 20s
    depends_on:
      migrate: { condition: service_completed_successfully }
      cache: { condition: service_healthy }

  worker:
    <<: *app
    command: ["celery", "-A", "shop", "worker", "--concurrency", "4"]
    depends_on:
      migrate: { condition: service_completed_successfully }
      cache: { condition: service_healthy }

  migrate:
    <<: *app
    restart: "no"                              # one-shot job: the inherited unless-stopped would re-run it forever
    command: ["flask", "db", "upgrade"]
    depends_on:
      db: { condition: service_healthy }

  db:
    image: postgres:17.2
    restart: unless-stopped
    environment:
      POSTGRES_USER: shop
      POSTGRES_PASSWORD_FILE: /run/secrets/db-password
    secrets: [db-password]
    volumes: [db-data:/var/lib/postgresql/data]
    networks: [backend]
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U shop"]
      interval: 5s
      retries: 10

  cache:
    image: redis:7.4-alpine
    restart: unless-stopped
    command: ["redis-server", "--appendonly", "yes"]
    volumes: [cache-data:/data]
    networks: [backend]
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]

networks:
  frontend: {}
  backend:
    internal: true                             # db, cache, worker: no internet access

volumes:
  db-data:
  cache-data:
  caddy-data:

configs:
  caddyfile:
    file: ./deploy/Caddyfile

secrets:
  db-password:
    file: ./secrets/db-password.txt

Design checklist

Concern	Decision in the example
Exposure	Only proxy publishes ports
Segmentation	backend is internal; web cannot reach db
Persistence	Named volumes for database, cache, and certificates
Startup	Database healthy, then migrations, then app and worker
Configuration	Shared x-app fragment; tags required via :?
Secrets	Password files under /run/secrets, not environment variables
Resilience	restart: unless-stopped; app retries connections
Logs	Rotated json-file logs

If the worker needs to call an external API, attach it to a non-internal network as well; internal: true blocks all outbound traffic.

Deploying and updating on a single host

# From your machine, against a remote host via a Docker context
docker context create prod --docker "host=ssh://deploy@shop.example.com"

export TAG=1.5.0
docker --context prod compose pull                  # fetch new images first (less downtime)
docker --context prod compose up -d --wait          # recreate changed services, wait for health
docker --context prod compose ps
docker --context prod image prune -f                # clean old images

file: paths for configs and secrets are read on the machine running the Compose CLI and must also make sense to the Docker host; with a remote context, bind mounts and file-based configs refer to paths on the remote host. Many teams therefore run Compose directly on the server from a checked-out deployment repository.

Key takeaways

• Model each process as a service and make relationships explicit: networks for reachability, volumes for state, health-gated dependencies for order.
• Expose only the entry point; keep databases and caches on an internal network.
• Use one-shot services for migrations and gate the application on their success.
• Share common configuration with x- fragments; require release tags with :?.
• For updates, pull then up -d --wait; remember that Compose recreates containers, so expect brief interruptions.

🧪 Practice

1. (Beginner) Run the reference architecture with simple placeholder images (for example traefik/whoami for web and api) and verify the network boundaries: which services can resolve and reach which.
2. (Intermediate) Add a second worker type that consumes a different queue and needs outbound internet access, without giving the database internet access. Explain your network changes.
3. Interview question: "Walk me through what happens, in order, when you run docker compose up -d on this stack for the first time." Hint: follow networks and volumes creation, image pulls, the dependency graph with its conditions, and what Compose waits for at each step.

Integration Testing with Compose

Why Compose for tests. Unit tests mock the database; integration tests must not. They need the real PostgreSQL version, the real message broker, and the real service images, started in a clean state, every time, on every developer's machine and in CI. Compose provides exactly this: a reproducible, disposable environment described in a file next to the code.

The lifecycle of a test run

   1. up (fresh project name)  ->  2. wait for health  ->  3. run tests  ->  4. collect logs/results  ->  5. down -v
                                                                                  (always, even on failure)

Pattern A: the test runner is a service. Tests run inside the Compose network, so they reach services by name.

# compose.test.yaml (used together with compose.yaml)
services:
  db:
    tmpfs:
      - /var/lib/postgresql/data          # in-memory database: fast, nothing persists
    ports: !reset []                      # no host ports: parallel runs cannot collide

  tests:
    build:
      context: ./api
      target: test                        # stage with test dependencies
    command: ["pytest", "-q", "--junitxml=/reports/junit.xml", "tests/integration"]
    environment:
      API_URL: http://api:8000
      DATABASE_HOST: db
    volumes:
      - ./reports:/reports                # test results back to the host
    depends_on:
      api: { condition: service_healthy }

docker compose -f compose.yaml -f compose.test.yaml -p "it-${CI_JOB_ID:-local}" \
  up --build --exit-code-from tests      # implies --abort-on-container-exit
# ...
# tests-1 exited with code 0
echo $?                                  # exit code of the "tests" container -> CI pass/fail

--exit-code-from tests stops all containers when any container exits and returns the test container's exit code. This is simple, but a crashing unrelated service also ends the run, and the stack is stopped but not removed.

Pattern B: start the stack, test from outside. More flexible: tests run on the CI host (or in a run container), and the script controls each step.

#!/usr/bin/env bash
set -euo pipefail
export COMPOSE_PROJECT_NAME="it-${CI_JOB_ID:-$$}"        # unique per run: parallel-safe
export COMPOSE_FILE="compose.yaml:compose.test.yaml"

cleanup() {
  docker compose logs --no-color > compose-logs.txt || true   # keep evidence for failed runs
  docker compose down -v --remove-orphans                     # always remove everything
}
trap cleanup EXIT

docker compose up -d --build --wait --wait-timeout 180        # fail fast if anything is unhealthy
docker compose run --rm tests                                 # exit code propagates to the script

	Pattern A (--exit-code-from)	Pattern B (up --wait + run)
Simplicity	One command	Short script
Cleanup	Manual down -v	trap guarantees it
Several test suites	Awkward	Several run commands
Logs on failure	Mixed into output	Captured to a file
Health gating	Via depends_on of the runner	--wait for the whole stack

Making tests reliable

Practice	Why
Unique project name per run (-p)	Parallel jobs on one CI runner do not share containers, networks, or volumes
No fixed host ports in test files	Avoids "port already allocated" between parallel runs
tmpfs or fresh volumes, down -v	Each run starts from a known, clean state
Health checks and --wait instead of sleep	Deterministic and faster
Pinned image versions	The same test today and next month
Always collect logs before down	Failures are debuggable after the containers are gone

Compose vs Testcontainers. Compose describes the environment outside the test code and suits black-box tests of the whole system. Libraries such as Testcontainers start containers from within the test code and suit tests that need per-test isolation or programmatic control (see Testcontainers in Chapter 13). Many teams use both.

Key takeaways

• Add a test layer (compose.test.yaml) with a test runner, in-memory or disposable storage, and no host ports.
• up --exit-code-from tests is the one-command pattern; up --wait + run --rm tests with a cleanup trap is the more robust one.
• Use a unique project name per run so parallel CI jobs never collide.
• Gate on health, never on sleep; always capture logs and run down -v.

🧪 Practice

1. (Beginner) Write a compose.test.yaml with a test service that calls the API's health endpoint, and run it with --exit-code-from. Make the test fail on purpose and check the exit code.
2. (Intermediate) Convert the setup to Pattern B with a cleanup trap and run two copies of the script in parallel on the same machine. Confirm there are no conflicts and that nothing remains afterwards (docker compose ls -a, docker volume ls).
3. Interview question: "Integration tests pass locally but fail intermittently in CI with connection refused errors in the first test. How would you investigate and fix it?" Hint: compare machine speed and startup timing, look for sleep-based waiting, and check whether the services have healthchecks that reflect real readiness.

Secrets and Configs in Compose

Why not just environment variables and bind mounts. Passwords in environment variables leak easily: they appear in docker inspect, in docker compose config output, in process listings inside the container, in crash reports, and in child processes. Configuration files bind-mounted with ad hoc paths are scattered across service definitions. Compose provides two dedicated top-level elements: secrets for sensitive data and configs for non-sensitive configuration files. Both are declared once and granted to services explicitly, and both are mounted as files.

Secrets

services:
  db:
    image: postgres:17
    environment:
      POSTGRES_PASSWORD_FILE: /run/secrets/db-password   # official images read *_FILE variables
    secrets:
      - db-password                                      # short syntax: /run/secrets/db-password

  api:
    image: shop/api:1.4
    secrets:
      - source: db-password
        target: database_password                        # -> /run/secrets/database_password
      - api-key

secrets:
  db-password:
    file: ./secrets/db-password.txt                      # content from a file (git-ignored!)
  api-key:
    environment: API_KEY                                 # content from a variable in Compose's environment

Only services that list a secret can read it: web in this project has no access to db-password at all. In plain Compose (not Swarm), a file-based secret is a read-only bind mount of the host file, so it is not encrypted; the benefits are scoping, keeping values out of the environment and image, and a standard location. Protect the source files on the host and make sure the container user can read them.

Reading secrets in your own application. Follow the convention used by official images: accept either X or X_FILE.

import os
from pathlib import Path

def secret(name: str) -> str:
    """Return NAME_FILE's content if set, else the NAME environment variable."""
    path = os.environ.get(f"{name}_FILE")
    if path:
        return Path(path).read_text().strip()   # strip the trailing newline editors add
    return os.environ[name]

db_password = secret("DATABASE_PASSWORD")

Build secrets. A secret can also be passed to the image build without being stored in any layer (see Build-Time Secrets in Chapter 10):

services:
  web:
    build:
      context: ./web
      secrets: [npm-token]           # made available to RUN --mount=type=secret
secrets:
  npm-token:
    environment: NPM_TOKEN

# web/Dockerfile
RUN --mount=type=secret,id=npm-token \
    NPM_TOKEN="$(cat /run/secrets/npm-token)" npm ci   # token exists only during this RUN

Configs. Configs work like secrets but are meant for non-sensitive files (proxy configuration, application settings). They are mounted at /<config-name> by default, or at an explicit target.

services:
  proxy:
    image: nginx:1.27-alpine
    configs:
      - source: nginx-conf
        target: /etc/nginx/conf.d/default.conf
      - source: app-settings
        target: /usr/share/nginx/html/settings.json

configs:
  nginx-conf:
    file: ./deploy/nginx.conf            # from a file
  app-settings:
    content: |                           # inline content, interpolated by Compose
      { "apiUrl": "${PUBLIC_API_URL:-http://localhost:8000}", "env": "${ENVIRONMENT:-dev}" }
  feature-flags:
    environment: FEATURE_FLAGS_JSON      # from a variable

Comparison

	Environment variable	Bind mount	Config	Secret
Intended content	Small settings	Any files, often code	Non-sensitive configuration files	Passwords, keys, tokens
Visible in docker inspect env	Yes	No	No	No
Default location	Process environment	Any path	/<name>	/run/secrets/<name>
Declared once, granted per service	No	No	Yes	Yes
Sources	Literal, .env, env_file	Host path	file, content, environment	file, environment
Encrypted at rest / in transit	No	No	No	Swarm only
Same syntax in Swarm stacks	Yes	Partly	Yes	Yes (backed by the encrypted Raft store)

Hygiene

• Add secrets/ and .env to .gitignore; commit an .env.example and a script or README explaining how to create the secret files.
• Never paste secrets into compose.yaml itself, and remember that docker compose config prints interpolated values, so be careful with it in CI logs.
• For production, generate secret files from a secret manager at deploy time (Vault, AWS Secrets Manager, 1Password CLI, SOPS) rather than storing them on disk permanently. Chapter 10 covers Runtime Secrets and External Secret Managers.

Key takeaways

• Top-level secrets and configs declare files once; services receive only those they list.
• Secrets mount at /run/secrets/<name>, configs at /<name> or a custom target.
• Sources: file, environment, and (configs only) inline content with interpolation.
• In plain Compose, secrets are bind-mounted files, not encrypted; their value is scoping and keeping data out of environment variables and images.
• Use the *_FILE convention in applications, and build.secrets with RUN --mount=type=secret for build-time credentials.

🧪 Practice

1. (Beginner) Move a PostgreSQL password from POSTGRES_PASSWORD to a Compose secret with POSTGRES_PASSWORD_FILE, then confirm with docker inspect that the password no longer appears in the container's environment.
2. (Intermediate) Serve an nginx configuration and a JSON settings file through configs, with the JSON generated inline from interpolated variables. Change a variable, run docker compose up -d, and check whether the container picks up the new content; if it does not, find out what forces it to.
3. Interview question: "Compose secrets in a non-Swarm setup are just bind-mounted files. Are they still worth using, and what would you add for a production deployment?" Hint: compare the exposure surface of environment variables with files, consider who can read files on the host, and think about where the secret files come from at deploy time.

---

<a id="9-advanced-builds-with-buildkit"></a>