9. Advanced Builds with BuildKit

BuildKit is the build engine behind every modern docker build. It replaces the old step-by-step builder with a graph-based solver that runs independent work in parallel, caches very precisely, and gives Dockerfiles abilities the classic builder never had. This chapter explains how BuildKit and its client Buildx are put together, the Dockerfile features that only BuildKit offers (cache, secret, and SSH mounts, heredocs, versioned syntax), and how to build images for several CPU architectures and share build cache across machines. These are the tools that turn a slow CI build that leaks credentials into a fast, safe, reproducible one.

<a id="91-buildkit-and-buildx"></a>

9.1 BuildKit and Buildx

This section covers BuildKit's internal design, how to make sure it is active, the docker buildx command, and the builders and drivers that decide where and how a build runs.

BuildKit Architecture

Why a new builder. The original Docker builder (now called the legacy builder) worked like a shell script: it executed each Dockerfile instruction in order, committing a container to a layer after each one. That design had hard limits:

• Steps ran strictly one after another, even when two stages had nothing to do with each other.
• The entire build context was sent to the daemon as a tarball on every build, even if only one file had changed.
• Every stage was built, even stages the final image never used.
• There was no safe way to give a step a secret, an SSH agent, or a persistent cache directory.
• The output was always an image in the local store; you could not export files, tarballs, or push several platforms at once.

BuildKit (Moby project, introduced in Docker 18.09, default since Docker Engine 23.0) was designed from scratch to fix these problems. It treats a build as a dependency graph to be solved, not a list of commands to run.

Analogy. The legacy builder is a single cook following a recipe line by line. BuildKit is a kitchen manager who reads the whole menu first, sees that the sauce and the dough do not depend on each other, gives them to different cooks at the same time, skips dishes nobody ordered, and reuses anything already prepared earlier in the day.

The main components

Component	Role
Client (docker buildx, buildctl)	Sends the build request, streams the context and progress, receives the result
Session	A gRPC connection from the daemon back to the client, used to pull only the files it needs from the context, and to forward secrets, SSH agents, and registry credentials on demand
Frontend	Converts a build definition (usually a Dockerfile) into LLB. The Dockerfile frontend is one of many possible frontends
LLB (low-level build definition)	A content-addressable graph of simple operations: source (image, Git, HTTP, local files), exec (run a command), file (copy, mkdir, rm), merge, diff
Solver	Walks the LLB graph, computes a cache key for every vertex, runs uncached vertices in parallel, and skips vertices the requested result does not depend on
Worker	Actually executes operations, using runc or containerd and a snapshotter (for example overlayfs)
Cache manager	Stores results by cache key; can import and export cache to registries, directories, and CI services
Exporter	Turns the final result into an output: an image in the local store, a registry push, an OCI or Docker tarball, or plain files on disk

  docker buildx build .                                buildkitd (in dockerd or a container)
  +-------------------+        gRPC session        +----------------------------------------+
  | client            | <========================> |  frontend: Dockerfile ---> LLB graph   |
  |  - build context  |  files on demand (only the |                              |         |
  |  - secrets, SSH   |  ones referenced, only the |  solver: cache keys, parallel run,     |
  |  - registry auth  |  ones that changed)        |          prune unused vertices         |
  |  - progress UI    |                            |                              |         |
  +-------------------+                            |  worker: runc/containerd + snapshotter |
            ^                                      |                              |         |
            |        result (files / tarball)      |  exporter: image | registry | local    |
            +--------------------------------------|            | oci | docker | tar        |
                                                   +----------------------------------------+

Graph solving in practice. Consider a multi-stage Dockerfile with a frontend and a backend stage. BuildKit sees two independent branches and runs them concurrently, then joins them in the final stage:

# syntax=docker/dockerfile:1
FROM node:22-alpine AS web          # branch A
WORKDIR /web
COPY web/ .
RUN npm ci && npm run build

FROM golang:1.25 AS api             # branch B, independent of A: runs in parallel
WORKDIR /api
COPY api/ .
RUN CGO_ENABLED=0 go build -o /out/api .

FROM alpine:3.20 AS debug-tools     # never referenced by "final": never built
RUN apk add --no-cache curl strace

FROM gcr.io/distroless/static AS final
COPY --from=api /out/api /api       # join point: waits for both branches
COPY --from=web /web/dist /static
ENTRYPOINT ["/api"]

            [node:22-alpine]          [golang:1.25]        [alpine:3.20]
                  |                         |                    |
             COPY web/                 COPY api/            RUN apk add  (skipped:
                  |                         |                     not needed)
             RUN npm ci ...           RUN go build
                   \                       /
                    \                     /
                [distroless/static] + COPY --from=api + COPY --from=web
                                  |
                              exporter

Seeing BuildKit at work. The progress output numbers each graph vertex (#1, #2, ...) and shows several running at once. Builds are also recorded so you can inspect them afterwards:

# Plain progress shows every vertex, timing, and CACHED markers
docker buildx build --progress=plain -t demo .
# #9 [api 4/4] RUN CGO_ENABLED=0 go build -o /out/api .
# #10 [web 4/4] RUN npm ci && npm run build      <- interleaved: running in parallel
# #9 DONE 12.4s
# #10 DONE 18.1s

# Build history kept by the builder (Buildx 0.12+)
docker buildx history ls                   # recent builds, status, duration
docker buildx history inspect              # details of the latest build
docker buildx history logs                 # full logs of the latest build

Key takeaways

• BuildKit models a build as a graph (LLB); a frontend such as the Dockerfile parser produces that graph.
• The solver runs independent vertices in parallel and skips anything the requested target does not need.
• A session lets the daemon request only the context files it needs, plus secrets and SSH agents, from the client on demand.
• Exporters decouple building from output: local image, registry, tarball, or plain files.
• The legacy builder executed instructions sequentially; BuildKit replaced it as the default in Docker Engine 23.0.

🧪 Practice

1. (Beginner) Build the multi-stage example with --progress=plain and identify in the output which vertices ran in parallel and which stage was never executed.
2. (Intermediate) Create a 500 MB file in the build context that no COPY instruction references. Build twice and compare the "transferring context" size in the output. Then add COPY . . and compare again. Explain the difference in terms of the session.
3. Interview question: "Why can BuildKit skip a stage that the legacy builder would always execute?" Hint: think about what the solver knows before it starts running anything, and how the final target defines which vertices are reachable.

Enabling BuildKit

Why this still matters. On current Docker versions BuildKit is already on, but you will meet older hosts, CI images pinned to old Docker releases, documentation that sets DOCKER_BUILDKIT=1, and Windows container builds. Knowing which builder actually ran your build explains why a RUN --mount works on one machine and fails with "unknown flag" on another.

Which builder you get

Environment	Default builder	How to switch
Docker Engine 23.0+ on Linux	BuildKit (docker build is an alias of docker buildx build)	DOCKER_BUILDKIT=0 selects the deprecated legacy builder
Docker Desktop (Mac, Windows, Linux)	BuildKit	Same as above
Docker Engine 18.09 to 22.x	Legacy builder	DOCKER_BUILDKIT=1 per command, or the daemon feature flag
Windows containers	Historically legacy only; BuildKit support for Windows containers is newer and more limited	Check the Docker and BuildKit release notes for your version
Docker 23.0+ without the Buildx plugin installed	Falls back to the legacy builder with a deprecation warning	Install docker-buildx-plugin

Turning it on for older engines

# Per command (client side)
DOCKER_BUILDKIT=1 docker build -t app .

# For a whole shell session or CI job
export DOCKER_BUILDKIT=1

{
  "features": { "buildkit": true }
}

The JSON above goes in /etc/docker/daemon.json and makes BuildKit the daemon default; restart the daemon afterwards (sudo systemctl restart docker). An explicit DOCKER_BUILDKIT environment variable on the client overrides the daemon setting.

How to tell which builder ran. The two produce very different output:

# Legacy builder                      # BuildKit
Step 1/4 : FROM alpine:3.20           #1 [internal] load build definition from Dockerfile
 ---> 91ef0af61f39                    #1 transferring dockerfile: 112B done
Step 2/4 : RUN echo hi                #2 [internal] load metadata for docker.io/library/alpine:3.20
 ---> Running in 4c2e8f0b1a7d         #3 [1/2] FROM docker.io/library/alpine:3.20@sha256:...
hi                                    #4 [2/2] RUN echo hi
Removing intermediate container ...   #4 CACHED

# Confirm the plugin and the active builder
docker buildx version              # github.com/docker/buildx v0.29.1 ...
docker buildx ls                   # the builder marked with * is used by "docker build"

# Choose how progress is displayed
docker build --progress=plain .    # full logs, good for CI and debugging
docker build --progress=tty .      # compact live view (default in a terminal)
docker build --progress=quiet .    # print only the image ID

Common symptoms of the wrong builder

Symptom	Cause
the --mount option requires BuildKit	Legacy builder ran a Dockerfile using RUN --mount
Output shows Step 1/N and ---> Running in	Legacy builder is active
# syntax= line is silently ignored	Legacy builder treats it as a comment
--secret, --ssh, --platform with several values, --output rejected	Legacy builder or a very old CLI

Key takeaways

• Docker Engine 23.0+ and Docker Desktop use BuildKit by default; docker build is docker buildx build.
• On older engines enable it with DOCKER_BUILDKIT=1 or "features": {"buildkit": true} in daemon.json.
• DOCKER_BUILDKIT=0 forces the legacy builder, which is deprecated and lacks every feature in this chapter.
• The #N [...] progress format identifies BuildKit; Step N/M identifies the legacy builder.
• Without the Buildx plugin, recent Docker CLIs fall back to the legacy builder.

🧪 Practice

1. (Beginner) Run the same build with DOCKER_BUILDKIT=0 and without it. Compare the output formats and the total build time for a two-stage Dockerfile.
2. (Intermediate) Write a Dockerfile that uses RUN --mount=type=cache. Build it with the legacy builder, read the error, and then make the build work on an older engine without changing the Dockerfile.
3. Interview question: "A build passes on a developer laptop but fails in CI with an error about --mount. What do you check first?" Hint: consider the Docker version in the CI image, the presence of the Buildx plugin, and environment variables set by the pipeline.

docker buildx

What it is. docker buildx is a Docker CLI plugin and the full-featured client for BuildKit. docker build gives you the common subset; docker buildx exposes everything: choosing and managing builders, multi-platform builds, custom outputs, cache import and export, attestations, Bake, and build history. Since Docker 23.0, docker build simply calls docker buildx build with the default builder.

Command map

Command	Purpose
docker buildx build	Build with all BuildKit options
docker buildx bake	Build many targets from a declarative file (see Bake Files)
docker buildx create / use / ls / inspect / rm / stop	Manage builder instances
docker buildx du / prune	Show and clean the build cache of a builder
docker buildx imagetools inspect / create	Inspect manifests in a registry; assemble multi-platform indexes
docker buildx history	List, inspect, and export past builds

Important build options beyond docker build

Option	Meaning
--builder <name>	Use a specific builder instead of the current one
--platform linux/amd64,linux/arm64	Build for one or more target platforms
--output / -o	Choose the exporter: type=image, type=registry, type=docker, type=oci, type=tar, type=local
--push	Shorthand for --output type=registry
--load	Shorthand for --output type=docker (put the result in the local image store)
--cache-from / --cache-to	Import and export build cache
--secret / --ssh	Expose secrets or an SSH agent to RUN --mount
--build-context name=path	Add a named context usable by FROM and COPY --from
--attest, --sbom, --provenance	Attach supply-chain attestations (see Build Provenance and Attestations)
--metadata-file f.json	Write the resulting digest and other metadata to a file (useful in CI)
--call check / outline / targets	Ask the frontend to lint, list build arguments and secrets, or list stages instead of building
--allow network.host,security.insecure	Grant entitlements that some RUN options need

Exporting files instead of an image. One of the most practical BuildKit features: a build can produce just files. This turns a Dockerfile into a portable, reproducible build environment for binaries, documentation, or test reports.

# syntax=docker/dockerfile:1
FROM golang:1.25 AS build
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 go build -o /out/mytool ./cmd/mytool

# An empty stage containing only the artifacts to export
FROM scratch AS artifact
COPY --from=build /out/ /

# Write the contents of the "artifact" stage to ./dist on the host
docker buildx build --target artifact --output type=local,dest=./dist .
ls dist/            # mytool

# Same thing as a single tarball
docker buildx build --target artifact --output type=tar,dest=mytool.tar .

# Export a full image as an OCI layout tarball without touching the image store
docker buildx build --output type=oci,dest=app-oci.tar -t app:1.0 .

# Record the digest of a pushed image for later steps of a pipeline
docker buildx build --push -t registry.example.com/app:1.0 --metadata-file meta.json .
jq -r '."containerimage.digest"' meta.json     # sha256:...

# Lint without building, and list the build arguments the Dockerfile accepts
docker buildx build --call check .
docker buildx build --call outline .

Key takeaways

• docker buildx is the complete BuildKit client; docker build is an alias for docker buildx build with the current builder.
• --output chooses where results go; --push and --load are shorthands.
• A FROM scratch stage plus --output type=local turns Docker into a build tool for plain files.
• --metadata-file and docker buildx history make builds auditable and scriptable.
• --call runs frontend requests (lint, outline, targets) without building.

🧪 Practice

1. (Beginner) Run docker buildx --help and docker buildx build --help. List three options that docker build on a legacy-builder system would not accept.
2. (Intermediate) Write a Dockerfile that generates a static website (for example with a Markdown-to-HTML tool) and export only the generated HTML to ./public using --output type=local. Make sure no image is created in docker images.
3. Interview question: "How would you use Docker to produce release binaries for a CLI tool without requiring the toolchain on developers' machines?" Hint: combine a builder stage, a scratch stage, and a local exporter.

Builder Instances

What a builder is. A builder is a named configuration that tells Buildx which BuildKit to talk to and how. It records a driver (where BuildKit runs), one or more nodes (BuildKit endpoints), the platforms each node supports, and optional BuildKit configuration. Every build runs on exactly one builder.

Why have more than one. The default builder is convenient but restricted (see Build Drivers). You create additional builders to:

• build multi-platform images and export every cache type,
• pin a specific BuildKit version or configuration (registry mirrors, insecure registries, garbage collection limits),
• isolate cache between projects or teams,
• use remote machines, including native ARM servers, as build nodes.

Analogy. Builders are like printers configured on your computer. "Print" (build) always works the same way, but you choose the printer: the small local one, the color one in the office, or a remote print service. Each has its own capabilities and its own paper tray (cache).

Lifecycle

# List builders; * marks the current one
docker buildx ls
# NAME/NODE        DRIVER/ENDPOINT     STATUS    BUILDKIT   PLATFORMS
# default*         docker
#  \_ default       \_ default         running   v0.25.1    linux/amd64, linux/386

# Create a builder running BuildKit in a container, make it current, start it now
docker buildx create --name ci --driver docker-container --use --bootstrap

docker buildx inspect ci          # driver, nodes, platforms, BuildKit version
docker buildx use default         # switch back
docker buildx build --builder ci -t app .   # use a builder for one build only
BUILDX_BUILDER=ci docker build -t app .     # or via environment variable

docker buildx stop ci             # stop the BuildKit container, keep cache
docker buildx rm ci               # remove the builder and its state (cache included)

Customizing BuildKit. The docker-container, kubernetes, and remote drivers accept a buildkitd.toml configuration file and driver options:

# buildkitd.toml
[worker.oci]
  max-parallelism = 4                      # limit concurrent steps on small machines

[registry."docker.io"]
  mirrors = ["mirror.internal.example.com"] # pull Docker Hub images through a mirror

[registry."registry.local:5000"]
  http = true                              # plain-HTTP registry for a lab setup

[[worker.oci.gcpolicy]]
  keepDuration = "168h"                    # keep cache entries for 7 days
  maxUsedSpace = "20GB"                    # hard cap on cache size

# --driver-opt image=... pins the BuildKit version used by the builder
docker buildx create --name tuned \
  --driver docker-container \
  --driver-opt image=moby/buildkit:v0.25.1 \
  --buildkitd-config ./buildkitd.toml \
  --use --bootstrap

Multi-node builders. A builder can have several nodes, each serving different platforms. Buildx sends each platform to a node that supports it natively, which is far faster than emulation:

# Docker contexts pointing at an x86 machine and an ARM machine
docker context create amd64-box --docker host=ssh://build@amd64.example.com
docker context create arm64-box --docker host=ssh://build@arm64.example.com

# One builder, two native nodes
docker buildx create --name native --driver docker-container \
  --platform linux/amd64 amd64-box
docker buildx create --name native --append \
  --platform linux/arm64 arm64-box

docker buildx build --builder native --platform linux/amd64,linux/arm64 \
  -t registry.example.com/app:1.0 --push .

                      docker buildx build --platform linux/amd64,linux/arm64
                                         |
                                  builder "native"
                          +--------------+--------------+
                          |                             |
                 node 1: amd64-box             node 2: arm64-box
                 linux/amd64 (native)          linux/arm64 (native)
                          |                             |
                          +-------------+---------------+
                                        |
                            one image index pushed to the registry

Key takeaways

• A builder = driver + nodes + platforms + configuration; each build runs on one builder.
• default uses the BuildKit embedded in dockerd and cannot be customized much.
• docker buildx create --driver docker-container --use --bootstrap is the standard way to unlock all features.
• Each builder has its own cache; docker buildx rm deletes it.
• Multi-node builders (--append) give native multi-platform builds without emulation.

🧪 Practice

1. (Beginner) Create a builder named lab with the docker-container driver, run a build on it, and find the BuildKit container with docker ps. Remove the builder and check that the container disappears.
2. (Intermediate) Configure a builder with a buildkitd.toml that limits cache to 2 GB and parallelism to 2. Build a large multi-stage project and observe the effect with docker buildx du and the progress output.
3. Interview question: "Your team shares one CI host and builds from different projects keep evicting each other's cache. How could builders help?" Hint: think about per-project builders, garbage collection policies, and what state a builder owns.

Build Drivers

What a driver is. The driver decides where BuildKit runs and how Buildx connects to it. It is chosen when you create a builder and determines which features are available.

The four drivers

Driver	Where BuildKit runs	Typical use
docker	Embedded in the Docker daemon	Default; simple local builds that should appear in docker images immediately
docker-container	A moby/buildkit container managed by Buildx	Local or CI builds that need multi-platform, cache export, or a specific BuildKit version
kubernetes	BuildKit pods in a Kubernetes cluster	Shared, scalable build farms; CI running on Kubernetes
remote	A buildkitd you run and manage yourself, reached over TCP (with TLS) or a socket	Dedicated build servers, custom setups

Docker also offers a managed cloud driver for Docker Build Cloud, which runs builders on Docker's infrastructure with a shared team cache.

Feature comparison

Feature	docker	docker-container	kubernetes	remote
Result appears in docker images automatically	Yes	No (use --load)	No (use --load)	No (use --load)
Cache export (--cache-to)	Inline only*	All backends	All backends	All backends
Multi-platform images	Only with the containerd image store	Yes	Yes	Yes
Tarball / local output	Yes	Yes	Yes	Yes
Custom BuildKit version and buildkitd.toml	No (tied to the engine)	Yes	Yes	Managed by you
Extra setup	None	Docker only	Cluster + kubeconfig	You run buildkitd

* With the containerd image store enabled (the default for new installations of recent Docker Desktop and Engine releases), the docker driver gains multi-platform builds and more cache exporters, which narrows the gap considerably.

Why --load exists. A docker-container builder has its own storage, separate from the Docker image store. A build there produces an image inside the BuildKit container; nothing appears in docker images until you export it. --load streams the result into the local Docker image store, --push sends it to a registry. Without either, Buildx warns that the result stays only in the build cache.

# docker-container driver
docker buildx create --name dc --driver docker-container --use --bootstrap
docker buildx build -t app:dev .
# WARNING: No output specified with docker-container driver.
#          Build result will only remain in the build cache.
docker buildx build -t app:dev --load .          # now visible in docker images

# kubernetes driver: 2 BuildKit replicas in namespace "buildkit"
docker buildx create --name k8s --driver kubernetes \
  --driver-opt namespace=buildkit,replicas=2,requests.cpu=2,requests.memory=4Gi \
  --use --bootstrap

# remote driver: connect to a buildkitd started elsewhere with TLS
docker buildx create --name farm --driver remote \
  --driver-opt cacert=./ca.pem,cert=./cert.pem,key=./key.pem \
  tcp://buildkitd.internal.example.com:1234

Choosing a driver

Need results in "docker images" only, single platform, no remote cache?
        |-- yes --> docker (default)
        |-- no
             |
Building on one machine (laptop, CI runner)?
        |-- yes --> docker-container
        |-- no
             |
Builds run inside a Kubernetes cluster?
        |-- yes --> kubernetes
        |-- no  --> remote (self-managed buildkitd) or a managed cloud builder

Key takeaways

• The driver fixes where BuildKit runs: in dockerd, in a container, in Kubernetes pods, or on a remote daemon.
• docker is the simplest but the most limited; docker-container unlocks every feature on a single host.
• Non-docker drivers keep results in their own store: add --load or --push.
• The containerd image store removes many of the docker driver's restrictions.
• kubernetes and remote turn BuildKit into a shared build service.

🧪 Practice

1. (Beginner) Try docker buildx build --platform linux/amd64,linux/arm64 -t test . on the default builder and on a docker-container builder. Record each result or error message and explain it.
2. (Intermediate) Using a docker-container builder, build an image three ways: with no output flag, with --load, and with --output type=oci,dest=img.tar. For each, determine where the image ended up and how you would run it.
3. Interview question: "When would you choose the kubernetes driver over running Docker-in-Docker in CI pods?" Hint: compare privileges required, cache sharing between jobs, and scaling.

<a id="92-advanced-build-features"></a>

9.2 Advanced Build Features

This section covers the Dockerfile capabilities that exist only with BuildKit: mounts that give a RUN step temporary access to caches, secrets, and SSH agents without storing them in layers, inline heredoc scripts and files, and the versioned frontend that delivers these features.

Cache Mounts

The problem. The layer cache is all or nothing. If requirements.txt changes by one line, the RUN pip install -r requirements.txt layer is invalidated and every package is downloaded again. Package managers already have their own download caches (~/.cache/pip, ~/.npm, ~/.m2, /go/pkg/mod), but in a normal build those caches are thrown away, or worse, baked into the image and bloat it.

The solution. RUN --mount=type=cache mounts a persistent directory, owned by the builder, into the container for the duration of one RUN step. The package manager finds the files it downloaded in earlier builds and only fetches what is new. The directory is never part of the image and survives across builds on the same builder.

Analogy. The layer cache is a photocopy of a finished page: if anything on the page changes, you must rewrite the whole page. A cache mount is a toolbox kept next to the workbench: even when you build something new, the tools you already own do not need to be bought again.

Syntax and options

RUN --mount=type=cache,target=<path>[,id=<id>][,sharing=shared|private|locked][,uid=..,gid=..,mode=..][,ro] <command>

Option	Meaning	Default
target	Where the cache appears inside the container (required)	-
id	Identifies the cache; steps with the same id share it	The target path
sharing	shared: concurrent builds use it together; private: a new copy for each concurrent writer; locked: concurrent builds wait for each other	shared
uid, gid, mode	Ownership and permissions of the cache directory, needed when the step runs as a non-root user	0, 0, 0755
from, source	Seed the cache from a stage or image path	Empty directory
ro / readonly	Mount read-only	Read-write

Examples for common ecosystems

# syntax=docker/dockerfile:1

# ---- Python (pip) ----
FROM python:3.13-slim AS py
WORKDIR /app
COPY requirements.txt .
# Do NOT pass --no-cache-dir here: the whole point is to keep pip's cache
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install -r requirements.txt

# ---- Node.js (npm) ----
FROM node:22-alpine AS node
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm \
    npm ci

# ---- Go: module downloads and the compiler's build cache ----
FROM golang:1.25 AS go
WORKDIR /src
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod \
    go mod download
COPY . .
RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    go build -o /out/app .          # incremental compile even after COPY . . changed

# ---- Java (Maven) ----
FROM maven:3.9-eclipse-temurin-21 AS java
WORKDIR /src
COPY pom.xml .
COPY src ./src
RUN --mount=type=cache,target=/root/.m2 \
    mvn -q package -DskipTests

apt needs extra care. Debian and Ubuntu images ship a docker-clean hook that deletes downloaded packages after every install, which would leave the cache empty. Remove it, and use sharing=locked because apt cannot handle two processes writing its cache at once:

# syntax=docker/dockerfile:1
FROM debian:bookworm-slim
RUN rm -f /etc/apt/apt.conf.d/docker-clean && \
    echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/keep-cache
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
    --mount=type=cache,target=/var/lib/apt,sharing=locked \
    apt-get update && apt-get install -y --no-install-recommends build-essential
# No "rm -rf /var/lib/apt/lists/*" needed: those directories are mounts, not image content

Non-root steps. If the step runs as an unprivileged user, give the cache the right owner, or the tool will fail with "permission denied":

USER 1000
RUN --mount=type=cache,target=/home/app/.cache/pip,uid=1000,gid=1000 \
    pip install --user -r requirements.txt

The other RUN --mount types

Type	Purpose	Covered in
cache	Persistent directory reused across builds	This topic
bind	Read-only (by default) view of context or stage files, without creating a layer	Below
tmpfs	In-memory scratch space for one step	-
secret	A secret file or environment variable for one step	Secret Mounts
ssh	Access to the client's SSH agent	SSH Mounts

A bind mount avoids creating a layer for files that are only needed during one step, such as a lockfile used to install dependencies:

# requirements.txt is visible during this RUN only; no COPY layer is created
RUN --mount=type=bind,source=requirements.txt,target=/tmp/requirements.txt \
    --mount=type=cache,target=/root/.cache/pip \
    pip install -r /tmp/requirements.txt

Limits to remember

• Cache mounts live in the builder. A fresh CI runner or a new builder starts with an empty cache mount.
• They are not exported by --cache-to; remote cache only carries layers. (CI-specific workarounds exist that save and restore the cache mount directory, for example with GitHub Actions.)
• They are a performance optimization only: the build must still be correct with an empty cache.
• Inspect and clean them like other cache records:

docker buildx du --verbose --filter type=exec.cachemount   # list cache mount records
docker buildx prune --filter type=exec.cachemount        # remove only cache mounts

Key takeaways

• RUN --mount=type=cache,target=... keeps package-manager caches between builds without putting them in the image.
• Even when the layer cache misses, a cache mount makes the step incremental instead of starting from zero.
• Use sharing=locked for tools that cannot share a cache concurrently (apt), and uid/gid for non-root steps.
• Cache mounts are local to the builder and are not part of exported remote cache.
• type=bind provides files to a single step without adding a layer.

🧪 Practice

1. (Beginner) Build a Python image that installs ten packages, add one more package to requirements.txt, and time the rebuild with and without a cache mount for pip.
2. (Intermediate) Convert an apt-get install step to use cache mounts correctly, including the docker-clean fix. Prove that the second build after changing the package list downloads only the new package (watch the Get: lines with --progress=plain).
3. Interview question: "What is the difference between the layer cache and a cache mount, and why can a cache mount speed up a step that the layer cache must rerun?" Hint: consider what is keyed by the instruction and its inputs versus what persists as a directory independent of keys.

Secret Mounts

The problem. Builds often need credentials: a token for a private package registry, cloud credentials to download an artifact, a license key. The intuitive solutions all leak:

Approach	Where the secret ends up
COPY .npmrc . then RUN rm .npmrc	In the layer created by COPY; deleting it later only hides it
ARG NPM_TOKEN + RUN ... $NPM_TOKEN	In the image history and build metadata
ENV NPM_TOKEN=...	In the image config, visible to anyone running docker inspect

Anyone who can pull the image can extract any of these.

The solution. RUN --mount=type=secret makes a secret available to one RUN step, as a file on a temporary in-memory mount or as an environment variable. The secret is never written to a layer, never stored in the image history, and is not part of the cache key.

Analogy. It is like a visitor badge handed to a contractor at the door, valid only while they are in one room, and collected again as they leave. The building's records never contain a copy of the key.

Passing secrets to the build

# From a file on the client
docker buildx build --secret id=npmrc,src=$HOME/.npmrc -t app .

# From an environment variable on the client
export GITHUB_TOKEN=ghp_xxx
docker buildx build --secret id=gh_token,env=GITHUB_TOKEN -t app .

Using them in the Dockerfile

# syntax=docker/dockerfile:1
FROM node:22-alpine
WORKDIR /app
COPY package.json package-lock.json ./

# 1) As a file at a chosen path (default would be /run/secrets/npmrc)
RUN --mount=type=secret,id=npmrc,target=/root/.npmrc \
    npm ci

# 2) As a file at the default location, read explicitly
RUN --mount=type=secret,id=gh_token \
    GITHUB_TOKEN="$(cat /run/secrets/gh_token)" ./scripts/fetch-private-assets.sh

# 3) Directly as an environment variable for this step only (Dockerfile 1.10+)
RUN --mount=type=secret,id=gh_token,env=GITHUB_TOKEN \
    ./scripts/fetch-private-assets.sh

# 4) Fail loudly if the secret was not provided, instead of continuing without it
RUN --mount=type=secret,id=license,required=true \
    ./install-licensed-sdk.sh --license-file /run/secrets/license

Option	Meaning	Default
id	Name of the secret passed with --secret id=...	Base name of target
target / dst	File path inside the container	/run/secrets/<id>
env	Expose the secret as this environment variable instead of (or in addition to) a file	-
required	Fail the build if the secret is missing	false (missing secret = absent file)
mode, uid, gid	Permissions and ownership of the secret file	0400, 0, 0

Verifying nothing leaked

docker history --no-trunc app | grep -i token      # nothing
docker image inspect app --format '{{json .Config.Env}}'
docker run --rm app ls /run/secrets               # No such file or directory

Pitfalls

• Your command can still leak it. If a step writes the secret into the filesystem (echo "$TOKEN" > ~/.npmrc, or a tool that caches credentials in a config file), that file lands in the layer. Keep secrets on the mount path or in the step's environment.
• Secrets do not invalidate the cache. Changing a token's value does not rerun the step. This is usually what you want; use --no-cache-filter <stage> if a step's output genuinely depends on the secret.
• Build arguments are not secrets. BuildKit's build checks warn about ARG/ENV names that look like secrets (SecretsUsedInArgOrEnv).
• In Compose, the equivalent is build.secrets (see Secrets and Configs in Compose); the security policy around build-time credentials is discussed in Build-Time Secrets.

Key takeaways

• Pass secrets with --secret id=...,src=... or --secret id=...,env=...; consume them with RUN --mount=type=secret.
• The secret exists only during that RUN step: not in layers, history, or image config.
• Default location is /run/secrets/<id>; target changes the path and env exposes it as a variable.
• Use required=true so a missing secret fails the build instead of silently producing a broken image.
• Commands that copy the secret into the filesystem defeat the protection.

🧪 Practice

1. (Beginner) Build an image that uses ARG TOKEN in a RUN, then find the token with docker history --no-trunc. Rewrite it with a secret mount and confirm the token is gone.
2. (Intermediate) Install a dependency from a private npm or PyPI registry using a secret mount for the credentials file. Run the build once without passing --secret and make sure it fails with a clear error.
3. Interview question: "Why is COPY credentials.json . followed by RUN rm credentials.json not safe, even in a multi-stage build where the final stage does not copy that layer?" Hint: think about who can access intermediate stages, the build cache, and pushed cache images, and compare it with the final-stage-only case.

SSH Mounts

The problem. Some dependencies are fetched over SSH: private Git repositories, Go modules from private hosts, Git submodules. Copying a private key into the build (COPY id_ed25519 /root/.ssh/) has the same leak problem as any secret, and it hands the build a long-lived credential it does not need.

The solution. RUN --mount=type=ssh forwards the SSH agent socket from the client into one RUN step. Commands inside can authenticate with the agent's keys, but the private key itself never leaves the client machine. The agent signs authentication challenges; the build only sees a socket.

 client machine                                   build container (one RUN step)
 +------------------------------+                 +--------------------------------+
 | ssh-agent (holds private key)| <--- session -- | git clone git@github.com:...   |
 |   SSH_AUTH_SOCK=/tmp/agent   |  sign requests  |   SSH_AUTH_SOCK=/run/buildkit/  |
 +------------------------------+                 |     ssh_agent.0 (socket only)  |
          key never leaves                        +--------------------------------+

Example: cloning a private repository

# syntax=docker/dockerfile:1
FROM alpine:3.20 AS source
RUN apk add --no-cache git openssh-client
# Trust GitHub's host key; without this, ssh refuses or prompts
RUN mkdir -p -m 0700 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts
# The agent socket is available ONLY during this step
RUN --mount=type=ssh \
    git clone --depth 1 git@github.com:acme/private-lib.git /src/private-lib

# Make sure the agent is running and has your key
eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_ed25519

# Forward the default agent (uses $SSH_AUTH_SOCK)
docker buildx build --ssh default -t app .

# Or give a key file directly (Buildx loads it into a temporary agent)
docker buildx build --ssh default=$HOME/.ssh/id_ed25519 -t app .

Several identities. Give each agent an ID and select it in the mount:

RUN --mount=type=ssh,id=github  git clone git@github.com:acme/app-lib.git
RUN --mount=type=ssh,id=gitlab  git clone git@gitlab.example.com:ops/tools.git

docker buildx build --ssh github=$HOME/.ssh/github_key --ssh gitlab=$SSH_AUTH_SOCK .

Example: private Go modules. Go downloads over HTTPS by default; rewrite GitHub URLs to SSH and mark the module path as private:

# syntax=docker/dockerfile:1
FROM golang:1.25 AS build
ENV GOPRIVATE=github.com/acme/*
RUN git config --global url."git@github.com:".insteadOf "https://github.com/" && \
    mkdir -p -m 0700 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts
WORKDIR /src
COPY go.mod go.sum ./
RUN --mount=type=ssh \
    --mount=type=cache,target=/go/pkg/mod \
    go mod download

Option	Meaning	Default
id	Which --ssh <id>=... to use	default
target	Socket path inside the container	/run/buildkit/ssh_agent.<N>
required	Fail if the agent was not forwarded	false
mode, uid, gid	Socket permissions and ownership (for non-root users)	0600, 0, 0

SSH_AUTH_SOCK is set automatically inside the step. The same --ssh flag also lets BuildKit fetch a private Git repository used as the build context itself (docker buildx build --ssh default git@github.com:acme/app.git) or as an ADD source.

Key takeaways

• RUN --mount=type=ssh forwards the client's SSH agent socket; the private key is never copied into the build.
• Build with --ssh default (current agent) or --ssh <id>=<key or socket>.
• Add the server's host key to known_hosts in an earlier step, or SSH will refuse to connect.
• Only steps with the mount can use the agent; everything else in the build has no access.
• Prefer SSH mounts over secret mounts when the tool natively speaks SSH (Git, Go modules, rsync).

🧪 Practice

1. (Beginner) Clone a private repository in a Dockerfile using --mount=type=ssh. Then remove --ssh default from the build command and read the error.
2. (Intermediate) Build a Go or Node.js project that depends on a private Git repository, combining an SSH mount with a cache mount. Confirm with docker history that no key material or .ssh private key exists in the image.
3. Interview question: "Why is forwarding an SSH agent safer than mounting the private key as a build secret, and what risk remains?" Hint: consider what a malicious build step could do while the socket is available, versus what it could do with the key file.

Heredocs in Dockerfiles

The problem. Long RUN instructions become chains of && \ that are hard to read, easy to break (a missing backslash silently ends the command), and hostile to comments. Creating a small configuration file inside the image required either a separate file in the context or awkward echo and printf escaping.

The solution. Since Dockerfile syntax 1.4, RUN and COPY accept heredocs, the same <<EOF ... EOF notation used in shell scripts. A heredoc lets you write a multi-line script or the contents of a file inline.

Heredocs with RUN

# syntax=docker/dockerfile:1
FROM debian:bookworm-slim

# Before: fragile line continuations
RUN apt-get update && \
    apt-get install -y --no-install-recommends curl ca-certificates && \
    rm -rf /var/lib/apt/lists/*

# After: one script, one layer, readable, commentable
RUN <<EOF
set -eux                      # stop at the first error and print each command
apt-get update
# curl is needed by the healthcheck
apt-get install -y --no-install-recommends curl ca-certificates
rm -rf /var/lib/apt/lists/*
EOF

Always add set -e. A heredoc is executed as a script, and a shell script continues after a failing command unless told otherwise. Without set -e, the step's exit status is the status of the last command only, so a failed apt-get update would be ignored.

Other interpreters. A shebang line selects the program that runs the script:

RUN <<EOF
#!/usr/bin/env python3
import json, platform
with open("/build-info.json", "w") as f:
    json.dump({"python": platform.python_version()}, f)
EOF

Heredocs with COPY: inline files

# syntax=docker/dockerfile:1
FROM nginx:1.27-alpine

# Create a file without needing it in the build context
COPY <<EOF /etc/nginx/conf.d/default.conf
server {
    listen 8080;
    location / { root /usr/share/nginx/html; }
    location /health { return 200 "ok\n"; }
}
EOF

# Several files in one instruction: each heredoc name becomes the file name
COPY <<robots.txt <<humans.txt /usr/share/nginx/html/
User-agent: *
Disallow: /admin
robots.txt
Team: platform@example.com
humans.txt

Variable expansion and quoting

Form	Behavior
<<EOF in COPY	ARG/ENV variables such as ${VERSION} are expanded by the builder
<<"EOF" or <<'EOF' in COPY	No expansion: the text is copied literally (useful for scripts containing $)
<<EOF in RUN	The script is run by the shell, which expands variables at run time like any script
<<-EOF	Leading tab characters are stripped, so the body can be indented

ARG VERSION=1.4.2
COPY <<EOF /app/version.txt
${VERSION}
EOF
# /app/version.txt contains: 1.4.2

COPY <<"EOF" /usr/local/bin/entrypoint.sh
#!/bin/sh
exec "$@"
EOF
# The script keeps "$@" literally instead of an expanded empty string

Key takeaways

• Heredocs (<<EOF ... EOF) work in RUN and COPY with Dockerfile syntax 1.4+.
• A RUN heredoc is one script and one layer; start it with set -eux so failures stop the build.
• A shebang line runs the heredoc with another interpreter (Python, Bash, ...).
• COPY heredocs create files inline; quote the delimiter to avoid variable expansion.
• Keep large or reusable files in the context; heredocs suit short configs and setup scripts.

🧪 Practice

1. (Beginner) Rewrite a RUN instruction that uses five && \ continuations as a heredoc. Confirm that the number of layers is the same.
2. (Intermediate) Write a heredoc RUN without set -e where the first command fails and the last succeeds. Observe that the build passes. Add set -e and rerun. Then create an entrypoint script with COPY <<"EOF" and verify that $@ survived unexpanded.
3. Interview question: "When would you keep a configuration file in the build context instead of embedding it with a COPY heredoc?" Hint: think about size, reuse across images, editor tooling, and reviewing changes.

Dockerfile Frontends and Syntax Directive

Why frontends. BuildKit itself does not understand Dockerfiles. It understands LLB graphs. A frontend translates some input format into LLB. The Dockerfile frontend is the most common one, but because the interface is generic, the Dockerfile language can evolve independently of the Docker Engine: new instructions and flags ship as a new frontend image, not as a new engine release.

The # syntax directive. The first line of a Dockerfile can name the frontend image to use:

# syntax=docker/dockerfile:1
FROM alpine:3.20

When BuildKit sees this directive, it pulls docker/dockerfile:1 (a small image containing the parser), runs it, and lets it produce the LLB for the rest of the file. Without the directive, the frontend built into your Docker Engine is used, which may be older.

 Dockerfile --> first line "# syntax=docker/dockerfile:1" ? --yes--> pull docker/dockerfile:1 --> run it --> LLB
                                     |
                                     no
                                     +--> built-in frontend of this engine version ----------------> LLB

Rules for parser directives

• Directives must appear at the very top of the file. As soon as BuildKit sees a comment, a blank line, or an instruction, it stops looking for directives; a # syntax= line after that is just a comment.
• Each directive may appear only once.
• Other directives: # escape= changes the escape character (often ` on Windows) and # check= configures build checks.

Choosing a version

Reference	Meaning	When to use
docker/dockerfile:1	Latest stable 1.x release, updated automatically	Default recommendation for most projects
docker/dockerfile:1.10	Latest patch of that minor version	Freeze features while accepting bug fixes
docker/dockerfile:1.10.0	Exact release	Strict reproducibility
docker/dockerfile:1@sha256:...	Pinned by digest	Supply-chain-sensitive builds; update via a dependency bot
docker/dockerfile:labs or :1-labs	Stable features plus experimental ones	Trying new features; avoid in production

The 1.x line is backwards compatible, so :1 will not break a working Dockerfile, but a feature you use must exist in the version you select:

Feature	Minimum Dockerfile syntax
RUN --mount (cache, secret, ssh, bind, tmpfs)	1.2
Heredocs, COPY --link, named build contexts in FROM/COPY --from	1.4
Build checks and the # check= directive	1.8
env= option for secret mounts	1.10

Asking the frontend questions. Because the frontend is a program, Buildx can ask it for information without running a build:

docker buildx build --call targets .     # list stages and their descriptions
docker buildx build --call outline .     # build args, secrets, and SSH IDs the build uses
docker buildx build --call check .       # run build checks (linting) only

# syntax=docker/dockerfile:1
# check=error=true

# A comment directly above a stage becomes its description in "--call targets"
# Builds the production image
FROM alpine:3.20 AS release

# check=error=true turns build-check warnings into errors, which is useful in CI.

Overriding and alternative frontends

# Override the frontend without editing the Dockerfile
docker buildx build --build-arg BUILDKIT_SYNTAX=docker/dockerfile:1-labs .

Any image that implements the BuildKit frontend API can be used in # syntax=. Some projects provide frontends for other build formats (for example YAML-based build specs), and tools such as Dagger and Earthly use BuildKit as their execution engine with their own languages. For Dockerfiles, docker/dockerfile is the frontend you will use in practice.

Key takeaways

• A frontend converts a build definition to LLB; the Dockerfile frontend is just one implementation.
• # syntax=docker/dockerfile:1 gets the latest stable Dockerfile features regardless of the engine version.
• Directives must be the very first lines, before any comment or blank line.
• Pin by minor version, exact version, or digest when reproducibility matters; use labs only for experiments.
• --call targets|outline|check queries the frontend without building.

🧪 Practice

1. (Beginner) Add # syntax=docker/dockerfile:1 to a Dockerfile, build with --progress=plain, and find the step where the frontend image is resolved.
2. (Intermediate) Pin the frontend to docker/dockerfile:1.3 and use a heredoc. Read the error, then raise the version to the minimum that works. Finally run --call outline and --call check on the file.
3. Interview question: "Why did Docker move the Dockerfile parser into a separately versioned image, and what trade-off does # syntax=docker/dockerfile:1 introduce?" Hint: consider release cycles of engines versus language features, and what happens when a build pulls an image you did not pin.

<a id="93-multi-platform-and-remote-caching"></a>

9.3 Multi-Platform and Remote Caching

This section covers building one image for several CPU architectures, the manifest structures that make that possible, the three ways to produce non-native binaries (emulation, native nodes, cross-compilation), how to share build cache between machines, and Bake for orchestrating many builds.

Multi-Architecture Images

Why it matters. Containers share the host kernel and run native machine code, so an image built for x86-64 cannot run on an ARM CPU. ARM is everywhere: Apple Silicon laptops, AWS Graviton and other ARM cloud instances, Raspberry Pi and edge devices. Running the wrong architecture fails immediately:

exec /usr/local/bin/app: exec format error

A multi-platform image is one image name (for example nginx:1.27) that contains a separate variant for each platform. Docker automatically pulls the variant matching the host, so the same docker run command works everywhere.

Platform strings. A platform is written os/architecture[/variant]:

Platform	Typical hardware
linux/amd64	Most servers and PCs (Intel, AMD)
linux/arm64 (linux/arm64/v8)	Apple Silicon (inside Docker Desktop's VM), AWS Graviton, Raspberry Pi 4/5 with 64-bit OS
linux/arm/v7	32-bit ARM boards (older Raspberry Pi OS)
linux/arm/v6	Raspberry Pi Zero / 1
linux/386	32-bit x86
linux/ppc64le, linux/s390x	IBM POWER, IBM Z mainframes
linux/riscv64	RISC-V boards
windows/amd64	Windows containers

Building one

# Requires a builder that supports multi-platform (docker-container driver,
# or the docker driver with the containerd image store)
docker buildx create --name multi --driver docker-container --use --bootstrap

# Build both variants and push them under one tag
docker buildx build \
  --platform linux/amd64,linux/arm64 \
  -t registry.example.com/acme/app:1.0 \
  --push .

# Verify what was pushed
docker buildx imagetools inspect registry.example.com/acme/app:1.0
# Name:      registry.example.com/acme/app:1.0
# MediaType: application/vnd.oci.image.index.v1+json
# Manifests:
#   Name:      ...@sha256:3f1c...   Platform: linux/amd64
#   Name:      ...@sha256:9ab2...   Platform: linux/arm64
#   Name:      ...@sha256:7e4d...   Platform: unknown/unknown   <- attestation (provenance)

# Run a specific variant explicitly (uses emulation if it is not native)
docker run --rm --platform linux/arm64 registry.example.com/acme/app:1.0

Multi-platform results normally go straight to a registry with --push. Loading them into the local store with --load only works with the containerd image store; the classic store can hold one platform per tag.

Platform-aware Dockerfiles. BuildKit defines automatic build arguments for every stage; declare them with ARG (no value) to use them:

Argument	Example value	Meaning
TARGETPLATFORM	linux/arm64	Platform being built for
TARGETOS, TARGETARCH, TARGETVARIANT	linux, arm64, ``	Its components
BUILDPLATFORM	linux/amd64	Platform of the machine doing the build
BUILDOS, BUILDARCH, BUILDVARIANT	linux, amd64, ``	Its components

# syntax=docker/dockerfile:1
FROM alpine:3.20
ARG TARGETARCH
# Download the right prebuilt binary for each architecture
RUN wget -qO /usr/local/bin/kubectl \
      "https://dl.k8s.io/release/v1.31.0/bin/linux/${TARGETARCH}/kubectl" && \
    chmod +x /usr/local/bin/kubectl

Three strategies for non-native platforms

Strategy	How	Speed	Effort
Emulation	QEMU runs foreign binaries on the build host (see QEMU Emulation)	Slow, especially for compilation	None in the Dockerfile
Native nodes	A builder with one node per architecture (see Builder Instances)	Fast	Need machines of each architecture
Cross-compilation	Compile on the build platform for the target platform (see Cross-Compilation)	Fast	Dockerfile changes; depends on toolchain support

Base images must exist for every target platform. Most official images (alpine, debian, python, node, golang, nginx) are multi-platform; check with docker buildx imagetools inspect <image> before choosing one.

Key takeaways

• Containers run native code; an image for the wrong architecture fails with exec format error.
• A multi-platform image is one tag with one variant per os/arch[/variant]; clients pull the matching one.
• Build with docker buildx build --platform a,b --push on a multi-platform-capable builder.
• TARGETARCH and friends let one Dockerfile adapt to each platform.
• Choose between emulation, native nodes, and cross-compilation based on speed and effort.

🧪 Practice

1. (Beginner) Inspect python:3.13-slim and nginx:1.27-alpine with docker buildx imagetools inspect and list the platforms each supports. Run docker run --rm --platform linux/arm64 alpine uname -m and explain the output.
2. (Intermediate) Write a Dockerfile that downloads a prebuilt release binary for the correct architecture using TARGETARCH, build it for linux/amd64,linux/arm64 into a local registry (registry:2 on port 5000), and verify both variants run.
3. Interview question: "A developer on an Apple Silicon Mac builds and pushes an image, and production on x86 servers fails with exec format error. What happened and how do you prevent it?" Hint: consider the default target platform of a build, explicit --platform, and CI-built multi-platform images.

Image Manifests and Manifest Lists

Why it matters. Multi-platform images work because of one extra level of indirection in the registry. Understanding it explains how one tag can point to many images, why digests differ between docker pull on different machines, and how to assemble or repair multi-platform images by hand. The underlying OCI objects (manifest, config, layers, descriptors) were introduced in OCI Image Specification; this topic focuses on the multi-platform layer.

Two kinds of top-level objects

Object	Docker media type	OCI media type	Contains
Image manifest	application/vnd.docker.distribution.manifest.v2+json	application/vnd.oci.image.manifest.v1+json	One config + ordered layers, for one platform
Manifest list / image index	application/vnd.docker.distribution.manifest.list.v2+json	application/vnd.oci.image.index.v1+json	A list of manifest descriptors, each with a platform

"Manifest list" is Docker's original name; "image index" is the OCI name for the same idea. BuildKit produces OCI indexes by default; registries and clients accept both.

What an index looks like

docker buildx imagetools inspect --raw registry.example.com/acme/app:1.0 | jq .

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:3f1c...",                 // amd64 image manifest
      "size": 1053,
      "platform": { "architecture": "amd64", "os": "linux" }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:9ab2...",                 // arm64 image manifest
      "size": 1053,
      "platform": { "architecture": "arm64", "os": "linux" }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:7e4d...",                 // attestation manifest (SBOM, provenance)
      "size": 840,
      "platform": { "architecture": "unknown", "os": "unknown" },
      "annotations": {
        "vnd.docker.reference.type": "attestation-manifest",
        "vnd.docker.reference.digest": "sha256:3f1c..."   // which image it describes
      }
    }
  ]
}

How a pull resolves

docker pull acme/app:1.0   (on an arm64 host)
   |
   1. GET /v2/acme/app/manifests/1.0         -> image index (digest sha256:aaa...)
   2. pick the entry with platform linux/arm64
   3. GET /v2/acme/app/manifests/sha256:9ab2 -> arm64 image manifest
   4. GET config blob + missing layer blobs   -> image ready

The repo digest shown by docker images --digests is the digest of the index, so it is the same on every machine; the image ID and layers differ per platform. Pinning acme/app@sha256:<index digest> pins all platforms at once.

Assembling an index from separate builds. When each architecture is built in its own CI job (often on native runners), push them under per-platform tags and then combine them. imagetools create works entirely in the registry, without pulling layers:

# Job 1 (amd64 runner) and job 2 (arm64 runner)
docker buildx build --platform linux/amd64 -t acme/app:1.0-amd64 --push .
docker buildx build --platform linux/arm64 -t acme/app:1.0-arm64 --push .

# Job 3: create a multi-platform tag that references both
docker buildx imagetools create -t acme/app:1.0 -t acme/app:latest \
  acme/app:1.0-amd64 acme/app:1.0-arm64

# Copy a full multi-platform image to another registry (all platforms, no pull)
docker buildx imagetools create -t mirror.example.com/acme/app:1.0 acme/app:1.0

The older docker manifest create / docker manifest push commands do the same for Docker-format manifest lists, but they are less capable (no OCI indexes, no attestations); prefer imagetools.

Key takeaways

• An image manifest describes one platform; an index (manifest list) maps platforms to manifests.
• Clients fetch the index, pick their platform, then fetch that manifest and its blobs.
• The index digest is identical everywhere; per-platform manifests and image IDs differ.
• Attestation manifests appear in the index with platform unknown/unknown.
• docker buildx imagetools inspect --raw shows the structure; imagetools create assembles or copies indexes without pulling layers.

🧪 Practice

1. (Beginner) Use docker buildx imagetools inspect --raw alpine:3.20 | jq '.manifests[].platform' to list the platforms. Then fetch the raw manifest of the linux/arm64 entry by its digest.
2. (Intermediate) Build linux/amd64 and linux/arm64 variants as two separate tags in a local registry, combine them with imagetools create, and pull the combined tag with --platform set to each architecture. Compare the image IDs.
3. Interview question: "Two engineers pull acme/app:1.0 on different machines and report different image IDs but the same digest. Is something wrong?" Hint: distinguish the digest of the index, the digest of a platform manifest, and the image ID derived from the config.

QEMU Emulation

Why it matters. Emulation is the easiest way to build for a foreign architecture: no Dockerfile changes and no extra hardware. Docker Desktop has it preconfigured, which is why building linux/arm64 on an Intel Mac or linux/amd64 on Apple Silicon "just works". On Linux hosts you set it up once.

How it works. Two pieces combine:

• QEMU user-mode emulation translates the machine instructions of a single foreign program into native ones and forwards its system calls to the host kernel. No virtual machine is involved; it is the same kernel.
• binfmt_misc is a Linux kernel feature that maps the "magic bytes" at the start of an executable to an interpreter. Once an ARM64 ELF header is registered with qemu-aarch64, the kernel transparently runs any ARM64 binary through QEMU. The F (fix-binary) flag makes the kernel load the interpreter at registration time, so it works inside containers that do not contain QEMU themselves.

 RUN apt-get install ...  inside an arm64 build container on an amd64 host
        |
   kernel execve("/usr/bin/apt-get")       ELF header says: aarch64
        |
   binfmt_misc: aarch64 -> /usr/bin/qemu-aarch64 (registered with flag F)
        |
   qemu-aarch64 translates instructions, passes syscalls to the amd64 kernel

Setting it up on Linux

# Show which emulators are registered
docker run --privileged --rm tonistiigi/binfmt
# {"supported": ["linux/amd64", "linux/386"], "emulators": [] }

# Register emulators for all supported architectures (or a list: arm64,riscv64)
docker run --privileged --rm tonistiigi/binfmt --install all

ls /proc/sys/fs/binfmt_misc/            # qemu-aarch64  qemu-arm  qemu-riscv64 ...
cat /proc/sys/fs/binfmt_misc/qemu-aarch64   # interpreter path and flags (includes F)

# Test: an arm64 container on an amd64 host
docker run --rm --platform linux/arm64 alpine uname -m     # aarch64

# Remove the registrations
docker run --privileged --rm tonistiigi/binfmt --uninstall 'qemu-*'

Registrations do not survive a reboot unless your distribution's qemu-user-static/binfmt-support packages or a startup job reinstalls them. In GitHub Actions, the docker/setup-qemu-action step does this for you before docker/setup-buildx-action.

Costs and limits

Aspect	Reality
Speed	Every instruction is translated: CPU-heavy steps (compiling C/C++/Rust, npm installs with native addons) can be many times slower than native
Correctness	Rare unsupported system calls or CPU features can crash programs (for example segfaults in JIT-based runtimes) only under emulation
Scope	Ideal for light steps (installing packages, copying files) and for smoke-testing images
Alternatives	Native nodes or cross-compilation for heavy or flaky steps

A practical rule: if the multi-platform build takes much longer than the native one, move the expensive work to a stage that runs on $BUILDPLATFORM and cross-compiles (next topic).

Key takeaways

• QEMU user-mode emulation + binfmt_misc lets the host kernel run binaries of other architectures transparently.
• docker run --privileged --rm tonistiigi/binfmt --install all registers emulators on Linux; Docker Desktop includes them.
• Emulation needs no Dockerfile changes but is slow for CPU-intensive steps and occasionally unreliable.
• Verify with docker run --platform linux/arm64 alpine uname -m.
• Use emulation for light steps and testing; use native nodes or cross-compilation for heavy builds.

🧪 Practice

1. (Beginner) On a Linux host, list /proc/sys/fs/binfmt_misc/, install the arm64 emulator, and run an arm64 Alpine container. Uninstall it and run the container again to see the failure.
2. (Intermediate) Time the build of a small C or Rust project for the native platform and for a foreign platform under emulation. Calculate the slowdown factor and identify which step dominates.
3. Interview question: "How can a container of a different architecture run on the same kernel without a virtual machine?" Hint: separate what the CPU must understand (instructions) from what the kernel provides (system calls), and explain the role of binfmt_misc.

Cross-Compilation

The idea. Instead of emulating the target CPU, run the compiler natively on the build machine and ask it to produce binaries for the target platform. Compilers for Go, Rust, C/C++ (clang), .NET, and others support this. The build runs at full native speed, and only the small final image is assembled for the target platform.

The key Dockerfile pattern. FROM --platform=$BUILDPLATFORM forces a stage to use the build machine's platform, even when the build targets another one. The target is passed to the compiler through TARGETOS/TARGETARCH:

# syntax=docker/dockerfile:1

# Runs natively on the build machine (e.g. amd64) for every target platform
FROM --platform=$BUILDPLATFORM golang:1.25 AS build
ARG TARGETOS TARGETARCH                 # filled per target: linux/amd64, linux/arm64 ...
WORKDIR /src
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod go mod download
COPY . .
RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    CGO_ENABLED=0 GOOS=$TARGETOS GOARCH=$TARGETARCH \
    go build -o /out/app .              # Go cross-compiles natively, no emulation

# No --platform: resolved for the TARGET platform (arm64 base for the arm64 variant)
FROM gcr.io/distroless/static
COPY --from=build /out/app /app
ENTRYPOINT ["/app"]

docker buildx build --platform linux/amd64,linux/arm64,linux/ppc64le \
  -t registry.example.com/acme/app:1.0 --push .

 build host: linux/amd64
 +----------------------------------------------------------+
 | stage "build" (golang, amd64, native speed)               |
 |   GOARCH=amd64 -> /out/app (amd64)                        |
 |   GOARCH=arm64 -> /out/app (arm64)     one native stage   |
 |   GOARCH=ppc64le -> /out/app (ppc64le) per target         |
 +----------------------------------------------------------+
        |                 |                   |
  distroless amd64  distroless arm64   distroless ppc64le     final stages: only COPY,
        |                 |                   |               no foreign code executed
        +---------- one image index ----------+

Because the final stage only copies files, no foreign-architecture code runs at all, so QEMU is not even required.

Languages that need a C toolchain. When the build links C code (Go with CGO_ENABLED=1, Rust crates with C dependencies, C/C++ projects), you need a cross-compiler and target-architecture libraries. The tonistiigi/xx helper image provides wrappers that configure this from TARGETPLATFORM:

# syntax=docker/dockerfile:1
FROM --platform=$BUILDPLATFORM tonistiigi/xx AS xx

FROM --platform=$BUILDPLATFORM golang:1.25-alpine AS build
COPY --from=xx / /                      # adds xx-apk, xx-go, xx-clang, xx-verify ...
RUN apk add --no-cache clang lld        # native cross-compiler (runs on build platform)
ARG TARGETPLATFORM
RUN xx-apk add --no-cache musl-dev gcc  # C libraries for the TARGET architecture
WORKDIR /src
COPY . .
ENV CGO_ENABLED=1
RUN xx-go build -o /out/app . && \
    xx-verify /out/app                  # fails if the binary is not for TARGETPLATFORM

FROM alpine:3.20
COPY --from=build /out/app /usr/local/bin/app
ENTRYPOINT ["app"]

Other ecosystems

Ecosystem	Approach
Go (pure)	GOOS/GOARCH with CGO_ENABLED=0; trivial
Rust	rustup target add + a linker for the target (or xx-cargo); cargo build --target
Java, Kotlin, Scala	Bytecode is platform-independent: build the JAR on $BUILDPLATFORM, copy it into a multi-platform JRE image
.NET	dotnet publish -r linux-arm64 (runtime identifier from TARGETARCH) on $BUILDPLATFORM
Node.js, Python	Pure code needs no compilation; dependencies with native extensions must be installed for the target (prebuilt wheels or binaries, or emulation for that one step)

Key takeaways

• Cross-compilation runs the compiler natively and targets another architecture: fast and emulation-free.
• FROM --platform=$BUILDPLATFORM pins the builder stage to the host platform; TARGETOS/TARGETARCH tell the compiler what to produce.
• Leave --platform off the final stage so its base image matches each target.
• tonistiigi/xx handles cross toolchains and target libraries for C-linked code.
• Platform-independent artifacts (JARs, pure Python/JS) need no cross-compilation, only correct native dependencies.

🧪 Practice

1. (Beginner) Build the pure-Go example for linux/amd64,linux/arm64 with --progress=plain. Confirm that the go build steps run on your native platform (look at the stage labels) and use file on exported binaries to verify their architecture.
2. (Intermediate) Take a Java or .NET application and restructure its Dockerfile so the compile stage uses --platform=$BUILDPLATFORM while the runtime stage is multi-platform. Compare build times before and after for a two-platform build.
3. Interview question: "In a cross-compiling Dockerfile, why must the final stage not use --platform=$BUILDPLATFORM?" Hint: think about which base image variant ends up in each platform's manifest and what happens when it runs on the target host.

Registry and Inline Cache

The problem. BuildKit's cache lives inside the builder. CI runners are usually ephemeral, so every pipeline starts with an empty cache and rebuilds everything, even when only one line of code changed. Teams with several CI machines also cannot share cache between them. (How cache keys are computed was covered in Build Cache.)

The solution. BuildKit can export its cache to external storage at the end of a build (--cache-to) and import it at the start of the next one (--cache-from). Any builder with access to that storage can then reuse the work, matching by cache key exactly as with the local cache.

Cache backends

Backend	Where cache is stored	Notes
inline	Inside the pushed image's config	Simplest; only mode=min; works with the docker driver
registry	A separate cache image in a registry (ref=...:buildcache)	Supports mode=max; cache and image are decoupled
local	A directory (dest= / src=)	For CI systems that persist directories between jobs
gha	GitHub Actions cache service	Native integration in GitHub Actions
s3, azblob	Object storage buckets	Self-hosted CI at scale

mode=min versus mode=max

Mode	What is exported	Trade-off
min (default)	Only layers that are part of the final image	Small cache, but builder stages (compilers, dependency installs) are not cached
max	Layers of all steps in all stages	Larger export, but multi-stage builds get near-complete cache hits

For multi-stage builds, where the expensive work happens in builder stages that are not in the final image, mode=max is usually what you want.

Inline cache

# Embed cache metadata into the image itself and push it
docker buildx build \
  --cache-to type=inline \
  -t registry.example.com/acme/app:latest --push .

# Next build (any machine): use the previous image as a cache source
docker buildx build \
  --cache-from type=registry,ref=registry.example.com/acme/app:latest \
  -t registry.example.com/acme/app:latest --push .

With the plain docker build command on the docker driver, the equivalent is --build-arg BUILDKIT_INLINE_CACHE=1.

Registry cache (recommended for CI)

docker buildx build \
  --cache-from type=registry,ref=registry.example.com/acme/app:buildcache \
  --cache-to   type=registry,ref=registry.example.com/acme/app:buildcache,mode=max \
  -t registry.example.com/acme/app:${GIT_SHA} \
  --push .

 CI run N                                         CI run N+1 (fresh runner)
 +---------------------------+                    +---------------------------+
 | empty local cache         |                    | empty local cache         |
 | build all steps           |                    | --cache-from: import keys |
 | --cache-to  ----------+   |                    |   most steps: CACHED      |
 +-----------------------|---+                    | only changed steps run    |
                         v                        +-------------^-------------+
              registry: acme/app:buildcache  -------------------+

Practical patterns

# Several sources: the branch's own cache first, then main's as a fallback
docker buildx build \
  --cache-from type=registry,ref=reg.example.com/app:buildcache-feature-x \
  --cache-from type=registry,ref=reg.example.com/app:buildcache-main \
  --cache-to   type=registry,ref=reg.example.com/app:buildcache-feature-x,mode=max \
  -t reg.example.com/app:pr-123 --push .

# Registries that reject index-style cache manifests (e.g. Amazon ECR)
--cache-to type=registry,ref=<acct>.dkr.ecr.<region>.amazonaws.com/app:cache,mode=max,image-manifest=true,oci-mediatypes=true

# GitHub Actions cache, one scope per image
--cache-from type=gha,scope=api --cache-to type=gha,scope=api,mode=max

# Do not fail the build if the cache cannot be written (e.g. forks without push rights)
--cache-to type=registry,ref=reg.example.com/app:buildcache,mode=max,ignore-error=true

Pitfalls

• Exporting cache requires a builder that supports it: the docker driver only supports inline (unless the containerd image store is enabled), so use docker-container in CI.
• Cache mounts (--mount=type=cache) are not included in exported cache.
• Exporting mode=max takes time and space; monitor the cache image size and prune old branch caches.
• Cache keys depend on base image digests and file checksums, so a moving base tag or non-deterministic COPY input reduces hit rates.

Key takeaways

• --cache-to exports and --cache-from imports BuildKit cache, making ephemeral CI runners fast.
• inline stores minimal cache inside the image; registry stores a separate cache artifact and supports mode=max.
• mode=max caches every stage and is usually right for multi-stage builds.
• Multiple --cache-from sources allow branch-specific cache with a main-branch fallback.
• Use a docker-container builder for full cache export support; cache mounts are never exported.

🧪 Practice

1. (Beginner) Push an image with --cache-to type=inline, delete all local cache with docker buildx prune -af, and rebuild with --cache-from pointing at the image. Count the CACHED steps.
2. (Intermediate) For a multi-stage Dockerfile with an expensive builder stage, compare cache hits on a fresh builder after exporting with type=registry,mode=min versus mode=max. Explain the difference using which layers each mode exported.
3. Interview question: "Your CI builds take 12 minutes even though most commits only change application code. How would you speed them up?" Hint: start with layer ordering and cache mounts, then consider where cache lives between runs, which backend and mode to export, and how branches share cache.

Bake Files

The problem. Real projects rarely have one docker build command. A monorepo might build an API, a web frontend, and a worker, each for two platforms, with tags, build arguments, cache settings, and secrets, in development and release variants. Encoding that as shell scripts in CI leads to long, duplicated command lines that drift apart between developers and pipelines.

The solution. docker buildx bake reads a declarative file describing targets (one build each) and groups (sets of targets), then builds them all, in parallel, in one BuildKit session, sharing the cache and deduplicating common work. Think of it as "Compose for builds": Compose describes how to run a set of containers; Bake describes how to build a set of images.

File formats and discovery. Bake accepts HCL (most expressive), JSON, and Compose files. Without -f, it looks for compose.yaml/docker-compose.yml and docker-bake.hcl/docker-bake.json (plus docker-bake.override.*) in the current directory and merges them. Without a target name it builds the group named default.

Core concepts

Concept	Purpose
target	One build: context, dockerfile, tags, platforms, args, cache, secrets, output, ...
group	A named set of targets built together
variable	A parameter with a default, overridable by an environment variable of the same name
inherits	Reuse settings from other targets (a common base target)
matrix	Generate many targets from combinations of values
contexts	Named build contexts, including target:<name> to feed one target's result into another
function	Small helper functions written in HCL

Example: a monorepo

# docker-bake.hcl
variable "REGISTRY" { default = "registry.example.com/acme" }
variable "TAG"      { default = "dev" }          # override with TAG=1.4.0 in the environment

group "default" {
  targets = ["api", "web"]                       # "docker buildx bake" builds these
}

group "release" {
  targets = ["api-release", "web-release"]
}

# Shared settings, inherited by every image (a leading _ is just a naming convention)
target "_common" {
  labels = {
    "org.opencontainers.image.source" = "https://github.com/acme/shop"
  }
  args = { BUILD_VERSION = TAG }
}

target "api" {
  inherits   = ["_common"]
  context    = "./services/api"
  tags       = ["${REGISTRY}/api:${TAG}"]
  cache-from = ["type=registry,ref=${REGISTRY}/api:buildcache"]
  cache-to   = ["type=registry,ref=${REGISTRY}/api:buildcache,mode=max"]
  secret     = ["id=npm_token,env=NPM_TOKEN"]      # read from the NPM_TOKEN env var
}

target "web" {
  inherits = ["_common"]
  context  = "./services/web"
  tags     = ["${REGISTRY}/web:${TAG}"]
  contexts = { shared = "./libs/shared" }       # COPY --from=shared in the Dockerfile
}

# Release variants: same images, multi-platform, with attestations
target "api-release" {
  inherits  = ["api"]
  platforms = ["linux/amd64", "linux/arm64"]
  attest    = ["type=provenance,mode=max", "type=sbom"]
}

target "web-release" {
  inherits  = ["web"]
  platforms = ["linux/amd64", "linux/arm64"]
}

docker buildx bake                       # build group "default" (api + web) in parallel
docker buildx bake api                   # one target
docker buildx bake --print release       # show the fully resolved JSON, build nothing
TAG=1.4.0 docker buildx bake --push release   # variables from the environment

# Override any attribute from the command line
docker buildx bake --set api.platform=linux/arm64 --set '*.output=type=docker' api
docker buildx bake -f docker-bake.hcl -f docker-bake.ci.hcl release   # layered files

Matrix targets. Generate one target per combination of values:

target "app" {
  name     = "app-${variant}"                    # unique name per generated target
  matrix   = { variant = ["release", "debug"] }
  target   = variant                             # Dockerfile stage to build
  tags     = ["acme/app:${variant}"]
}
# docker buildx bake app            -> builds app-release and app-debug

Sharing a built stage between targets. A contexts entry of the form target:<name> feeds the result of one target into another as a named context. The base is built once and both consumers use it, without pushing an intermediate image:

target "base" {
  dockerfile = "base.Dockerfile"
}

target "api" {
  contexts   = { base = "target:base" }          # FROM base in api's Dockerfile
  dockerfile = "api.Dockerfile"
}

Bake with Compose. If a compose.yaml already describes the services' build: sections, docker buildx bake can build them directly, and an x-bake extension adds Bake-only attributes:

services:
  api:
    build:
      context: ./services/api
      x-bake:                       # ignored by Compose, read by Bake
        platforms: [linux/amd64, linux/arm64]
        cache-to: ["type=registry,ref=registry.example.com/acme/api:buildcache,mode=max"]
    image: registry.example.com/acme/api:dev

Bake compared with the alternatives

Aspect	Shell script of docker build	docker compose build	docker buildx bake
Parallel builds	Manual (&)	Yes	Yes, in one session with shared cache
All Buildx options (cache export, attestations, outputs)	Yes, verbosely	Subset via build: keys	Yes
Variables, inheritance, matrices	Shell logic	Interpolation only	Yes
Inspect resolved config	No	docker compose config	--print
Primary purpose	Ad hoc	Building images for running stacks	Build orchestration (CI, release)

Key takeaways

• Bake builds many targets from a declarative HCL, JSON, or Compose file, in parallel and in one session.
• Targets hold build settings; groups bundle targets; default runs when no target is named.
• Variables are overridable from the environment; inherits removes duplication; matrix generates targets.
• --print shows the resolved configuration and --set overrides attributes from the command line.
• The same file serves developers and CI, so builds stop drifting between laptops and pipelines.

🧪 Practice

1. (Beginner) Write a docker-bake.hcl with two targets and a default group, run docker buildx bake --print, and then build both. Override the tag with an environment variable.
2. (Intermediate) Convert a CI script that runs three docker buildx build commands with registry cache and multi-platform settings into a Bake file using a common inherited target and a release group. Use --set to build only linux/amd64 locally with --load.
3. Interview question: "When would you use Bake instead of docker compose build, and can they coexist in one repository?" Hint: compare their purpose (running versus building), the build options each exposes, and how Bake reads Compose files and x-bake extensions.

---

<a id="10-docker-security"></a>