15. Docker Internals and Extensibility

Most of the time Docker works as a single command, but underneath it is a set of cooperating daemons, sockets, content-addressed stores, and well-defined extension points. This chapter follows a container from the API call down to the kernel, shows how images are stored and identified on disk, and then covers the interfaces you can use to extend Docker: the Engine API and SDKs, daemon plugins, CLI plugins, Docker Desktop extensions, and alternative OCI runtimes. Knowing these internals is what lets you debug failures that error messages alone do not explain, build tooling on top of Docker, and choose the right isolation level for a workload.

<a id="151-engine-internals"></a>

15.1 Engine Internals

This section opens up the components introduced in containerd and runc and follows the data and control flow between them in detail.

containerd Shims

Why it matters. A container's main process needs a parent. The parent receives the exit status, keeps the stdio pipes open, and owns the process from the kernel's point of view. If that parent were dockerd or containerd, restarting or upgrading either daemon would kill every container on the host. The shim is a tiny per-container process whose only job is to be that parent, so the daemons above it can come and go.

Analogy. Think of a hotel (containerd) and its guests (containers). The front desk staff can change shifts or the reception software can be upgraded, but each guest keeps their room key and their room. The shim is the room: it belongs to the guest for the whole stay, independently of who is working at the desk.

What a shim does

Responsibility	How
Starts the container	Calls the OCI runtime (runc create, then runc start) with the bundle prepared by containerd
Stays the parent	Marks itself a child subreaper (PR_SET_CHILD_SUBREAPER), so when runc exits, the container process is re-parented to the shim instead of PID 1
Keeps stdio alive	Holds the FIFOs or the PTY master connected to the container's stdin, stdout, and stderr
Reports exits	Reaps the process with waitpid() and publishes a TaskExit event with the exit status
Serves the task API	Exposes a small ttrpc API (a lightweight gRPC variant for local sockets) used by containerd: Create, Start, Kill, Exec, Pids, Pause, Resume, Stats, Update, ResizePty, Wait, Delete, Shutdown
Survives daemon restarts	Writes its socket address into the bundle directory, so a restarted containerd can reconnect

How containerd finds the shim binary. A container is created with a runtime type in the form io.containerd.<name>.<version>. containerd converts it to a binary name and looks it up on the PATH:

Runtime type	Shim binary	Low-level runtime behind it
io.containerd.runc.v2	containerd-shim-runc-v2	runc (or crun, via the shim's BinaryName option)
io.containerd.runsc.v1	containerd-shim-runsc-v1	gVisor
io.containerd.kata.v2	containerd-shim-kata-v2	Kata Containers (a lightweight VM)
io.containerd.wasmtime.v1	containerd-shim-wasmtime-v1	WebAssembly runtime (runwasi project)

This naming convention is the shim v2 API: anyone can write a shim and plug a completely different execution technology into containerd, as long as it implements the task API. Docker has supported shim v2 runtimes directly since Engine 23 (see Alternative Runtimes (gVisor, Kata Containers)).

The startup handshake. The shim is started in two phases, which is why its parent is PID 1 and not containerd:

 containerd              shim ("start" mode)        shim (long-running)        runc
     |                          |                          |                     |
     |-- exec shim ... start -->|                          |                     |
     |                          |-- fork, setsid --------->|  becomes subreaper  |
     |<-- prints ttrpc address -|  (exits; long-running    |                     |
     |                          |   shim is re-parented    |                     |
     |                          |   to PID 1)              |                     |
     |-- ttrpc Create(bundle) ------------------------------>|-- runc create ----->| clone() + namespaces,
     |                                                      |                     | cgroups, mounts;
     |                                                      |<-- runc exits ------| "runc init" waits
     |-- ttrpc Start --------------------------------------->|-- runc start ------>| init exec()s the
     |<-- TaskStart event -----------------------------------|                     | user's command
     |                          ...container runs...        |                     |
     |<-- TaskExit event (exit code) ------------------------|  waitpid() returns  |

One shim per container, or per pod. Docker starts one shim per container. Kubernetes (through containerd's CRI plugin) can group all containers of a pod under one shim, which saves memory on nodes running hundreds of containers. The shim API allows both models.

Example: looking at a shim and its bundle

docker run -d --name web nginx:alpine
CID=$(docker inspect -f '{{.Id}}' web)

# The shim process: parent is PID 1, it knows its namespace (moby) and container ID
ps -o pid,ppid,rss,cmd -C containerd-shim-runc-v2
#   PID  PPID   RSS CMD
#  4821     1  9800 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 9f1c... -address /run/containerd/containerd.sock

# The OCI bundle containerd prepared for this container (namespace "moby")
sudo ls /run/containerd/io.containerd.runtime.v2.task/moby/$CID
# address  config.json  init.pid  log  log.json  options.json  rootfs  runtime  shim-binary-path  work
sudo cat /run/containerd/io.containerd.runtime.v2.task/moby/$CID/address
# unix:///run/containerd/s/3f0a...      <- the shim's ttrpc socket; containerd reconnects here

# runc is not running, but it keeps per-container state in a namespace-specific root
sudo runc --root /run/containerd/runc/moby list
# ID         PID    STATUS    BUNDLE                                                   ...
# 9f1c...    4842   running   /run/containerd/io.containerd.runtime.v2.task/moby/9f1c...

# Restart containerd: the container survives because the shim, not containerd, is its parent
PID_BEFORE=$(docker inspect -f '{{.State.Pid}}' web)
sudo systemctl restart containerd
PID_AFTER=$(docker inspect -f '{{.State.Pid}}' web)
echo "$PID_BEFORE $PID_AFTER"          # same PID: the process was never touched

Things that go wrong

Symptom	Likely cause
docker ps shows a container that has no process, or cannot be stopped	Shim crashed or was killed; the state in containerd is stale (restarting containerd and then dockerd usually resynchronizes it)
Orphaned containerd-shim-runc-v2 processes after a crash	Containers removed while containerd was down; the shims can be killed once the container is confirmed gone
High memory use on dense hosts	Each shim costs a few MB of RSS; thousands of containers means hundreds of MB just for shims

Key takeaways

• The shim is a per-container process that is the parent of the container's main process; it makes daemon restarts and upgrades safe.
• It uses a two-phase start (a short-lived start invocation and a long-running daemon) and marks itself a child subreaper.
• containerd talks to shims over ttrpc and maps runtime types such as io.containerd.runc.v2 to binaries such as containerd-shim-runc-v2.
• The shim v2 API is the extension point that lets gVisor, Kata, and WebAssembly runtimes plug into containerd and Docker.
• Bundle and state directories under /run/containerd/ are the place to look when a container's runtime state seems inconsistent.

🧪 Practice

1. (Beginner) Start three containers and list their shims with ps -o pid,ppid,rss,cmd -C containerd-shim-runc-v2. Match each shim's -id argument to docker ps --no-trunc and add up their memory use.
2. (Intermediate) Run a container with docker run -d --name t alpine sleep 600. Kill its shim with sudo kill -9 <shim pid> and observe what happens to the sleep process, to docker ps, and to docker inspect t. Explain the result in terms of who the parent of sleep was.
3. Interview question: "Why is the parent process of a Docker container's main process neither dockerd nor containerd?" Hint: think about what happens to children when their parent exits, and what an operator expects when the daemon is upgraded.

Container Creation Flow

Why it matters. docker run looks atomic, but it is a pipeline that crosses four programs and dozens of kernel operations. When it fails, the error message usually names only the last component that complained (for example "OCI runtime create failed"). Knowing the stages lets you tell immediately whether a problem is in the client, the daemon, the network setup, or the kernel.

The stages of docker run -d -p 8080:80 --name web nginx

 docker CLI                dockerd                     containerd           shim / runc / kernel
     |                        |                             |                      |
 (1) |-- POST /containers/create ->|                        |                      |
     |<-- 404 No such image -------|                        |                      |
     |-- POST /images/create (pull)->|-- resolve, fetch, ---->| store content,      |
     |<-- progress stream ---------|   unpack layers         | snapshots           |
     |-- POST /containers/create ->| (2) validate config,    |                      |
     |                             |  allocate ID and name,  |                      |
     |                             |  prepare RW layer,      |                      |
     |<-- 201 {"Id": "..."} -------|  write config.v2.json   |                      |
 (3) |-- POST /containers/{id}/start ->|                      |                      |
     |                             | (4) mount volumes,      |                      |
     |                             |  set up network sandbox,|                      |
     |                             |  generate OCI spec      |                      |
     |                             |-- (5) create container + task -->|-- start shim -->|
     |                             |                         |               runc create:
     |                             |                         |               namespaces, cgroups,
     |                             |                         |               rootfs, pivot_root
     |                             |-- (6) start task ------>|-------------->| runc start: exec nginx
     |<-- 204 No Content ----------|<-- TaskStart event -----|               |
     |  (prints the container ID)  | (7) update state, emit "start" event, apply restart policy on exit

What happens at each stage

Stage	Component	Work done
1. Create request	CLI	Parses flags into a Config (image, command, env) and a HostConfig (ports, mounts, limits). If the daemon answers "No such image", the CLI pulls and retries; the API itself never pulls implicitly
2. Create	dockerd	Validates the configuration, generates the 64-hex ID and the name, prepares the writable layer (graph driver or containerd snapshot), and writes /var/lib/docker/containers/<id>/config.v2.json and hostconfig.json. No process exists yet: this is the state shown as Created
3. Start request	CLI	POST /containers/{id}/start. With docker run (without -d) the CLI first attaches to the container's streams so no output is lost
4. Prepare	dockerd	Mounts the root filesystem and volumes, creates the network sandbox and endpoints through libnetwork (veth pair, IP address, iptables/nftables rules for -p), writes /etc/hosts, /etc/resolv.conf, and /etc/hostname, then builds the OCI runtime spec: process args, env, capabilities, seccomp and AppArmor profiles, cgroup limits, namespaces, mounts
5. Create task	containerd, shim, runc	containerd stores a container object with the spec, starts a shim, and the shim runs runc create: new namespaces, cgroup, root filesystem, pivot_root, dropped capabilities, seccomp filter. The process is now a paused runc init waiting on a FIFO
6. Start task	shim, runc	runc start signals runc init, which execve()s the user command. From here the process is the container's PID 1
7. Monitor	containerd, dockerd	Events flow back up: TaskStart now, and later TaskExit with the exit code, which dockerd uses for docker wait, health checks, and the restart policy

docker create and docker start are the two halves. docker run equals docker create followed by docker start (plus pull and attach). After docker create the container exists only in dockerd's metadata; containerd does not know about it until it is started. A stopped container usually no longer exists in containerd either, which is why ctr -n moby containers ls lists only running containers.

Example: observing the flow

# Terminal 1: daemon events, in order
docker events --filter container=web --format '{{.Time}} {{.Type}} {{.Action}}'

# Terminal 2: the two halves of "docker run", one at a time
docker create --name web -p 8080:80 nginx:alpine
# Terminal 1 prints:  ... container create
docker inspect -f '{{.State.Status}} pid={{.State.Pid}}' web      # created pid=0
sudo ctr -n moby containers ls | grep -c "$(docker inspect -f '{{.Id}}' web)"   # 0: containerd does not know it yet

docker start web
# Terminal 1 prints:  ... network connect
#                     ... container start
docker inspect -f '{{.State.Status}} pid={{.State.Pid}}' web      # running pid=5127

# The generated OCI spec, as runc received it
CID=$(docker inspect -f '{{.Id}}' web)
sudo jq '{args: .process.args, caps: .process.capabilities.bounding | length,
          namespaces: [.linux.namespaces[].type]}' \
  /run/containerd/io.containerd.runtime.v2.task/moby/$CID/config.json
# { "args": ["/docker-entrypoint.sh","nginx","-g","daemon off;"],
#   "caps": 14,
#   "namespaces": ["mount","network","uts","pid","ipc","cgroup"] }

docker rm -f web
# Terminal 1 prints:  ... container kill / die / stop, network disconnect, container destroy

With "debug": true in daemon.json, the daemon log also shows each API call as it arrives (Calling POST /v1.52/containers/create), which is a quick way to see what any client, not only the CLI, is asking for.

Where an error comes from

Error message (shortened)	Stage	Component
Unable to find image 'x' locally followed by pull access denied	1	CLI pull, registry authentication
Conflict. The container name "/web" is already in use	2	dockerd
Bind for 0.0.0.0:8080 failed: port is already allocated	4	dockerd / libnetwork
error while creating mount source path	4	dockerd (bind mount source missing or not allowed)
OCI runtime create failed: ... exec: "foo": executable file not found in $PATH	5	runc (the command does not exist in the image)
exec format error	6	Kernel (binary built for another CPU architecture, or a script without a shebang)
OCI runtime create failed: ... permission denied	5	runc and the kernel (LSM policy, seccomp, or a read-only path)

Key takeaways

• docker run is a sequence of API calls (create, optionally pull, attach, start), and the API never pulls images on its own.
• dockerd turns the high-level configuration into an OCI runtime spec; runc turns the spec into kernel objects.
• A created container exists only as metadata; containerd and the shim become involved at start.
• docker events, daemon debug logs, and the bundle's config.json let you watch each stage.
• The wording of an error ("OCI runtime", "port is already allocated", "exec format error") tells you which stage failed.

🧪 Practice

1. (Beginner) Run docker events in one terminal and docker run --rm alpine echo hi in another. List the events in order and map each one to a stage in the table above.
2. (Intermediate) Create a container with --cap-drop ALL --cap-add NET_BIND_SERVICE --memory 64m --read-only, start it, and find each setting in the bundle's config.json. Which JSON fields does each flag change?
3. Interview question: "A user reports OCI runtime create failed: exec: "/app/start.sh": no such file or directory, but the file is in the image. What could be wrong?" Hint: the kernel reports "no such file" when the script's interpreter (the shebang line, including a Windows line ending) or a dynamic loader is missing.

Image Store and Content Addressing

Why it matters. Images are shared between many containers, pulled from untrusted networks, and stored for months. Docker needs to deduplicate layers, detect corruption or tampering, and refer to exactly the same bytes on every machine. Content addressing solves all three: every piece of data is named by the cryptographic hash of its content.

Analogy. A library that shelves books by title can end up with two different books under the same title. A library that shelves books by a fingerprint of every page cannot: the same fingerprint always means the same book, a copy with one changed page gets a different place on the shelf, and two donations of the same book automatically land on the same spot.

Properties of a content-addressable store

Property	Consequence
Name = sha256(content)	Identical data is stored once (deduplication), no matter how many images reference it
Content cannot change without changing its name	Objects are immutable; "updating" means writing a new object
Anyone can recompute the hash	Integrity is verifiable after download and at rest
References form a tree (index -> manifest -> config and layers)	The top-level digest pins every byte below it, like a Git commit

Three identifiers for one layer. A layer has different identities depending on where it is:

Identifier	Computed over	Used for
Distribution digest	The compressed blob (.tar.gz or .tar.zst) as stored in the registry	Pulling and pushing; listed in the manifest's layers
DiffID	The uncompressed tar	Listed in the image config's rootfs.diff_ids; independent of compression
ChainID	The DiffID of this layer combined with the ChainIDs of all layers below it	Identifies a stack of layers; used as the on-disk key of a layer in the classic store

The ChainID exists because the same layer content can sit on top of different parents. A file added by COPY app.py . produces the same DiffID in two images, but if the layers below it differ, the resulting filesystems differ, so the stacks must be stored separately. The definition is:

ChainID(L1)  = DiffID(L1)
ChainID(Ln)  = sha256( ChainID(Ln-1) + " " + DiffID(Ln) )

The image ID. In Docker's classic store, the image ID is the digest of the image config JSON (which includes the DiffIDs). That is why an image keeps the same ID after it is pushed and pulled, while its repository digest (the manifest digest) can differ between registries if layers are recompressed. With the containerd image store (next topic), the ID shown is the digest of the manifest or index instead.

The classic store on disk. With the overlay2 storage driver, metadata and filesystem data are kept apart:

/var/lib/docker/
  image/overlay2/
    repositories.json                  # tags and digests -> image ID   ("nginx:alpine" -> sha256:...)
    imagedb/content/sha256/<image-id>  # the image config JSON; file name = sha256 of its content
    layerdb/sha256/<chain-id>/
      diff                             # the layer's DiffID
      parent                           # ChainID of the layer below (absent for the base layer)
      cache-id                         # random ID of the directory under /var/lib/docker/overlay2
      size                             # uncompressed size
      tar-split.json.gz                # tar headers, to rebuild the byte-exact tar when pushing
    distribution/
      diffid-by-digest/sha256/...      # compressed digest -> DiffID (skip downloads of known layers)
      v2metadata-by-diffid/sha256/...  # DiffID -> digests in registries (skip uploads on push)
  overlay2/
    <cache-id>/diff/                   # the unpacked files of one layer (not content-addressed)
    l/                                 # short symlinks to keep overlay mount options short

Only the metadata is content-addressed; the unpacked directories use random cache-ids because the same layer can be unpacked differently by different drivers. tar-split is a clever detail: Docker throws away the original tar after extracting it, but keeps its headers, so it can regenerate a byte-identical tar (and therefore the same DiffID) when you push.

Example: verifying the identifiers by hand

docker pull python:3.13-slim

# The image ID is the hash of the config JSON (classic store)
ID=$(docker image inspect -f '{{.Id}}' python:3.13-slim)
echo "$ID"                                                         # sha256:4f1b...
sudo sha256sum /var/lib/docker/image/overlay2/imagedb/content/sha256/${ID#sha256:}
# 4f1b...   <- same hash: the file is named after its own content

# DiffIDs, bottom layer first
docker image inspect -f '{{json .RootFS.Layers}}' python:3.13-slim | jq -r '.[]' > diffids.txt

# Compute the ChainIDs and check that each exists in layerdb
chain=""
while read -r diff; do
  if [ -z "$chain" ]; then
    chain="$diff"                                                   # ChainID(L1) = DiffID(L1)
  else
    chain="sha256:$(printf '%s %s' "$chain" "$diff" | sha256sum | cut -d' ' -f1)"
  fi
  sudo test -d "/var/lib/docker/image/overlay2/layerdb/sha256/${chain#sha256:}" \
    && echo "found $chain" || echo "missing $chain"
done < diffids.txt

# Follow one layer to its files
TOP=${chain#sha256:}
CACHE=$(sudo cat /var/lib/docker/image/overlay2/layerdb/sha256/$TOP/cache-id)
sudo ls /var/lib/docker/overlay2/$CACHE/diff                        # the files this layer added

Key takeaways

• Content addressing names data by its hash, which gives deduplication, immutability, and integrity checks for free.
• A layer has a distribution digest (compressed), a DiffID (uncompressed), and a ChainID (the layer plus everything below it).
• In the classic store, the image ID is the digest of the config; the repository digest is the digest of the manifest.
• /var/lib/docker/image/<driver>/ holds content-addressed metadata, while /var/lib/docker/<driver>/ holds unpacked layers under random IDs.
• Never edit these directories by hand; use docker image rm and docker system prune so references stay consistent.

🧪 Practice

1. (Beginner) Pull two images that share a base (for example python:3.13-slim and python:3.13-slim-bookworm, or two of your own builds from the same base). Compare their .RootFS.Layers lists and count how many layers are stored only once.
2. (Intermediate) Run the ChainID script above for an image you built yourself. Then rebuild the image after changing a file in a middle layer and explain which ChainIDs changed and which stayed the same.
3. Interview question: "Why can the same image have one image ID but different digests in two registries?" Hint: consider which identifiers are computed over compressed data and which over uncompressed data.

containerd Image Store

Why it matters. Docker's classic image store predates multi-platform images, attestations, and the OCI image index. It can hold only one platform per tag, it drops provenance and SBOM attestations on pull, and on a host that also runs Kubernetes, the same image is stored twice (once by Docker, once by containerd). The containerd image store makes Docker use containerd's content store and snapshotters directly, the same storage that Kubernetes nodes use.

How it is organized. containerd separates three things that the classic store mixes together:

                      dockerd (namespace "moby")
                                |
                 +--------------+---------------+
                 v              v               v
           metadata DB     content store     snapshotter
          (bolt database)  (immutable blobs) (unpacked, mountable layers)
          images: name ->  blobs/sha256/<d>  overlayfs: snapshots/<n>/fs
          index digest     index, manifests,  parent chain by ChainID
          leases, labels   configs, layers    (also: native, stargz, nydus, soci)
                           (compressed)

Part	Location (default)	Holds
Metadata	/var/lib/containerd/io.containerd.metadata.v1.bolt/meta.db	Image names, containers, snapshot references, labels, leases; partitioned by namespace
Content store	/var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/	Every blob exactly as received from the registry: indexes, manifests, configs, compressed layers, attestations
Snapshotter	/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/	Unpacked layers, ready to mount with copy-on-write

Garbage collection. Objects are kept alive by references (labels from an index to its manifests, from a manifest to its layers, from an image to its snapshots). Anything unreferenced is removed by containerd's garbage collector. Leases protect objects that are still being downloaded or built from being collected halfway through.

Classic store compared with the containerd store

Aspect	Classic (graph driver) store	containerd image store
Multi-platform images	One platform per tag locally	Keeps the whole index; several platforms can be stored and loaded locally
Attestations (SBOM, provenance)	Discarded on pull	Kept with the image
Image ID shown by docker images	Config digest	Manifest or index digest
Compressed layers on disk	Discarded after unpacking	Kept in the content store (more disk use, faster re-push)
Storage backend	Graph drivers (overlay2, btrfs, ...)	Snapshotters (overlayfs, native, and lazy-pulling ones such as stargz)
Shared with Kubernetes on the same host	No	Same content store, separate namespaces (moby, k8s.io)
docker build --platform a,b --load	Not possible	Works

Enabling it. It is the default for fresh installations of recent Docker Engine and Docker Desktop releases. On an existing host it is controlled by a feature flag in daemon.json:

{
  "features": {
    "containerd-snapshotter": true
  }
}

Switching stores does not migrate anything. Images and containers from the old store become invisible (they are not deleted), so plan to re-pull images and recreate containers, and switch back temporarily if you need to export something from the old store.

Example: confirming and exploring the store

docker info -f '{{.Driver}} {{json .DriverStatus}}'
# overlayfs [["driver-type","io.containerd.snapshotter.v1"]]    <- containerd image store
# (the classic store would print: overlay2 [["Backing Filesystem","extfs"], ...])

# Pull two platforms of the same tag and see both stored under one name
docker pull --platform linux/amd64 alpine:3.22
docker pull --platform linux/arm64 alpine:3.22
docker image ls --tree alpine:3.22        # one index, two platform manifests (recent CLIs)

# The same objects from containerd's point of view (namespace "moby")
sudo ctr -n moby images ls -q | grep alpine
sudo ctr -n moby content ls | head        # DIGEST  SIZE  AGE  LABELS (gc references)
sudo ctr -n moby snapshots ls | head      # KEY  PARENT  KIND (Committed for layers, Active for containers)

# Build and load a multi-platform image: only possible with this store
docker buildx build --platform linux/amd64,linux/arm64 -t demo:multi --load .

Key takeaways

• The containerd image store makes Docker use containerd's metadata database, content store, and snapshotters.
• It keeps whole image indexes and attestations, so multi-platform images and supply-chain metadata survive locally.
• Compressed blobs are kept alongside unpacked snapshots, so disk usage is higher.
• Switching stores is not a migration: existing images and containers become invisible until you switch back.
• docker info (driver type io.containerd.snapshotter.v1) tells you which store a daemon uses.

🧪 Practice

1. (Beginner) Run docker info on your machine and determine which image store it uses. If it is the containerd store, list the moby namespace with sudo ctr -n moby images ls.
2. (Intermediate) On a test VM with the classic store, pull an image, enable containerd-snapshotter, restart Docker, and run docker images again. Pull the same image, compare docker system df before and after, and explain the difference in disk usage.
3. Interview question: "Your team builds multi-platform images and wants to test the arm64 variant locally before pushing. Why does --load fail on some machines and not on others?" Hint: compare what each image store can hold for a single tag.

Daemon Socket and API Versions

Why it matters. Every interaction with Docker, from the CLI, Compose, an IDE, a CI runner, or a monitoring agent, goes through the daemon's API socket. Who can reach that socket decides who controls the host, and which API version they speak decides whether old tools keep working after an upgrade. The basics of the API were introduced in Docker REST API; this topic covers how the socket is provided and how versioning works.

The sockets on a Docker host

Socket	Server	Protocol	Clients
/run/docker.sock (also reachable as /var/run/docker.sock, because /var/run links to /run)	dockerd	HTTP/1.1 REST	CLI, Compose, SDKs, tools
$XDG_RUNTIME_DIR/docker.sock	Rootless dockerd	HTTP REST	The user's CLI with DOCKER_HOST set
~/.docker/run/docker.sock	Docker Desktop (macOS, Linux)	HTTP REST (proxied into the VM)	Desktop's desktop-linux context
/run/containerd/containerd.sock	containerd	gRPC	dockerd, ctr, nerdctl, kubelet
/run/containerd/s/<hash>	One shim per container	ttrpc	containerd
/run/docker/plugins/<name>.sock	Docker plugins	HTTP JSON-RPC	dockerd

How dockerd gets its socket. On systemd distributions, dockerd is usually started with -H fd://, which means "use the sockets systemd already opened for me". The socket is defined in docker.socket, which also sets its owner and group:

systemctl cat docker.socket
# [Socket]
# ListenStream=/run/docker.sock
# SocketMode=0660
# SocketUser=root
# SocketGroup=docker          <- members of "docker" are effectively root

ls -l /run/docker.sock
# srw-rw---- 1 root docker 0 Oct 14 09:12 /run/docker.sock

A common mistake is setting "hosts" in daemon.json while the systemd unit also passes -H fd://: the daemon refuses to start because the same setting is given twice. Override the unit's ExecStart with a drop-in if you need different listeners. For remote access, prefer an SSH context (docker context create --docker host=ssh://user@host) over opening a TCP port.

API versioning. Each Engine release adds or changes endpoints and raises the API version (for example Engine 28.3 serves API 1.51, Engine 29.0 serves 1.52). A daemon supports a range of versions, from a minimum up to its current one, and behaves as the requested version for any request whose path starts with /v1.xx/.

 client supports up to 1.47          daemon supports 1.44 ... 1.52
         |                                      |
         |-- GET /_ping ----------------------->|
         |<-- header "Api-Version: 1.52" -------|
         |   negotiated = min(1.47, 1.52) = 1.47
         |-- GET /v1.47/containers/json ------->|   daemon answers in 1.47 format

Situation	Result
Older client, newer daemon (version within the daemon's range)	Works: the daemon emulates the older version
Newer client, older daemon	The client negotiates down to the daemon's version; features newer than that are unavailable
Client version below the daemon's minimum	Rejected with client version 1.43 is too old. Minimum supported API version is 1.44
Client pinned with DOCKER_API_VERSION	No negotiation; useful for testing, risky in production

Docker Engine 29 raised the minimum supported API version to 1.44, so very old SDKs and tools that hard-code an old version stopped working after the upgrade. The fix is to update the tool or let it negotiate; the daemon's DOCKER_MIN_API_VERSION environment variable can lower the minimum temporarily while you migrate.

Example: inspecting versions

docker version --format 'client={{.Client.APIVersion}} server={{.Server.APIVersion}} min={{.Server.MinAPIVersion}}'
# client=1.52 server=1.52 min=1.44

# The ping response headers carry the version information that clients negotiate with
curl -sI --unix-socket /run/docker.sock http://localhost/_ping
# HTTP/1.1 200 OK
# Api-Version: 1.52
# Docker-Experimental: false
# Ostype: linux
# Server: Docker/29.0.1 (linux)

# Ask for an older API version explicitly
curl -s --unix-socket /run/docker.sock http://localhost/v1.44/version | jq .ApiVersion
DOCKER_API_VERSION=1.44 docker ps          # the CLI pins the version instead of negotiating

Restricting API access. Mounting the socket read-only (-v /var/run/docker.sock:/var/run/docker.sock:ro) does not make the API read-only; :ro only protects the socket file, not the requests sent through it. To give a tool limited access, put a filtering proxy in front of the socket and point the tool at the proxy:

services:
  docker-proxy:
    image: tecnativa/docker-socket-proxy:latest   # pin a released version tag in production
    environment:
      CONTAINERS: 1          # allow /containers/* endpoints
      EVENTS: 1              # allow /events
      POST: 0                # deny every write request (create, start, exec, ...)
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    networks: [docker-api]

  dashboard:
    image: ghcr.io/example/container-dashboard:2.1
    environment:
      DOCKER_HOST: tcp://docker-proxy:2375   # the tool never sees the real socket
    networks: [docker-api]

networks:
  docker-api:
    internal: true           # the proxy is not reachable from outside

Key takeaways

• The Docker API socket is owned by root and the docker group; access to it is equivalent to root on the host.
• On systemd hosts, docker.socket creates the socket and dockerd receives it through -H fd://.
• A daemon serves a range of API versions; clients negotiate through /_ping and prefix requests with /v1.xx/.
• Upgrades can raise the minimum API version (1.44 in Engine 29), which breaks tools that pin very old versions.
• :ro on the socket mount restricts nothing; use a filtering proxy or an authorization plugin to limit API access.

🧪 Practice

1. (Beginner) Run docker version and curl -sI --unix-socket /run/docker.sock http://localhost/_ping and identify the current and minimum API versions of your daemon.
2. (Intermediate) Call /v1.44/containers/json and /v1.52/containers/json (or your daemon's current version) for the same container and compare the JSON fields. Then try a version below the minimum and record the error.
3. Interview question: "A monitoring tool needs to list containers and read their stats. How do you give it what it needs without giving it root on the host?" Hint: think about filtering at the HTTP level and about which endpoints and methods the tool actually uses.

<a id="152-extending-docker"></a>

15.2 Extending Docker

Docker is designed to be extended at several levels: programs can drive it through its API, the daemon can load plugins for storage, networking, logging, and authorization, the CLI can gain new subcommands, Docker Desktop can host graphical extensions, and the low-level runtime itself can be swapped.

Extension point	Runs where	Extends	Typical examples
Engine API and SDKs	Anywhere with API access	Automation of anything the CLI can do	CI tools, Testcontainers, dashboards, autoscalers
Daemon (managed) plugins	Host, as special containers managed by dockerd	Volumes, networks, IPAM, logging, authorization	Storage vendor drivers, OPA authorization
CLI plugins	Client machine	docker <new-command>	buildx, compose, scout, in-house tools
Docker Desktop extensions	Docker Desktop (UI, VM, host)	Desktop's graphical interface	Log explorers, disk usage tools, security scanners
Alternative OCI runtimes	Host, behind containerd	How containers are isolated	gVisor, Kata Containers, crun, WebAssembly shims

Docker Engine API and SDKs

Why it matters. Everything the Docker CLI does is an API call, so anything you can do by hand you can automate in a program: start test databases from a test suite, clean up resources on a schedule, react to container events, or build a custom dashboard. Using the API directly instead of running docker commands and parsing their text output gives you structured data, proper error handling, and no dependency on CLI output formats.

The shape of the API. The Engine API is a REST-style HTTP API described by an OpenAPI (Swagger) specification that ships with every release. The specification is the authoritative reference: it lists every endpoint, parameter, and response field per API version, and it can be used to generate clients for languages without an SDK.

Endpoint group	Examples	CLI equivalent
/containers	create, {id}/start, {id}/logs, {id}/stats, {id}/archive	docker container ...
/exec	POST /containers/{id}/exec, POST /exec/{id}/start, GET /exec/{id}/json	docker exec
/images, /build	images/create (pull), images/{name}/push, build	docker image ..., docker build (legacy builder; BuildKit uses /session and gRPC)
/networks, /volumes	networks/create, volumes/prune	docker network ..., docker volume ...
/system	_ping, version, info, events, system/df	docker version, docker info, docker events
Swarm	/swarm, /services, /nodes, /tasks, /secrets, /configs	docker swarm, docker service, ...
/plugins	plugins/pull, plugins/{name}/enable	docker plugin ...

Four kinds of responses. Most endpoints return a single JSON document, but some of the most useful ones do not, and SDKs exist mainly to hide these differences:

Kind	Endpoints	Format
Single JSON document	Most endpoints (/containers/json, /info, ...)	One JSON value
JSON stream	/events, /images/create (pull progress), /build, /containers/{id}/stats?stream=true	A sequence of JSON objects sent with chunked transfer encoding
Multiplexed raw stream	/containers/{id}/logs and attach for containers without a TTY	Frames with an 8-byte header: [stream type, 0, 0, 0, size as 4-byte big-endian]; type 1 = stdout, 2 = stderr
Hijacked connection	attach, exec/{id}/start	HTTP upgrade to a raw bidirectional TCP stream, used for interactive stdin

Example: decoding the multiplexed log stream (Python, standard library only)

import http.client
import json
import socket
import struct

class UnixHTTPConnection(http.client.HTTPConnection):
    """HTTPConnection that connects to a Unix socket instead of a TCP host."""
    def __init__(self, path: str):
        super().__init__("localhost")          # Host header only; no TCP connection is made
        self.unix_path = path

    def connect(self):
        self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
        self.sock.connect(self.unix_path)

def api(method: str, path: str, body=None) -> tuple[int, bytes]:
    conn = UnixHTTPConnection("/var/run/docker.sock")
    headers = {"Content-Type": "application/json"} if body is not None else {}
    conn.request(method, path, json.dumps(body) if body is not None else None, headers)
    resp = conn.getresponse()
    return resp.status, resp.read()

def demux(raw: bytes):
    """Split a multiplexed stream into (stream name, text) frames."""
    frames, i = [], 0
    while i + 8 <= len(raw):
        stream, size = struct.unpack(">BxxxL", raw[i:i + 8])   # 1 byte type, 3 padding, uint32 size
        frames.append(({1: "stdout", 2: "stderr"}.get(stream, "stdin"),
                       raw[i + 8:i + 8 + size].decode()))
        i += 8 + size
    return frames

status, body = api("POST", "/v1.44/containers/create?name=mux-demo",
                   {"Image": "alpine", "Cmd": ["sh", "-c", "echo to-stdout; echo to-stderr >&2"]})
cid = json.loads(body)["Id"]
api("POST", f"/v1.44/containers/{cid}/start")
api("POST", f"/v1.44/containers/{cid}/wait")
_, raw = api("GET", f"/v1.44/containers/{cid}/logs?stdout=1&stderr=1")
for stream, text in demux(raw):
    print(f"[{stream}] {text}", end="")
# [stdout] to-stdout
# [stderr] to-stderr
api("DELETE", f"/v1.44/containers/{cid}?force=1")

SDKs. For real programs, use an SDK. It handles transport selection (DOCKER_HOST, TLS, SSH), version negotiation, streams, and typed request and response objects.

Language	Library	Maintained by
Go	github.com/moby/moby/client (earlier releases: github.com/docker/docker/client)	Docker (official; the CLI itself uses it)
Python	docker (docker-py)	Docker (official)
Java	docker-java	Community
Node.js	dockerode	Community
.NET	Docker.DotNet	Community
Rust	bollard	Community

Example: the Go SDK

The example below uses the widely deployed github.com/docker/docker module (v28 API). Since Engine 29 the client is published as the separate modules github.com/moby/moby/client and github.com/moby/moby/api, with similar concepts but some changed signatures, so check that module's documentation when you start a new project.

package main

import (
	"context"
	"io"
	"log"
	"os"

	"github.com/docker/docker/api/types/container"
	"github.com/docker/docker/api/types/image"
	"github.com/docker/docker/client"
	"github.com/docker/docker/pkg/stdcopy"
)

func main() {
	ctx := context.Background()

	// FromEnv honours DOCKER_HOST, DOCKER_TLS_VERIFY, DOCKER_CERT_PATH, and DOCKER_API_VERSION.
	// WithAPIVersionNegotiation pings the daemon and uses the highest version both support.
	cli, err := client.NewClientWithOpts(client.FromEnv, client.WithAPIVersionNegotiation())
	if err != nil {
		log.Fatal(err)
	}
	defer cli.Close()

	// The API never pulls implicitly. The pull only completes once the progress stream is drained.
	progress, err := cli.ImagePull(ctx, "docker.io/library/alpine:3.22", image.PullOptions{})
	if err != nil {
		log.Fatal(err)
	}
	io.Copy(io.Discard, progress)
	progress.Close()

	created, err := cli.ContainerCreate(ctx,
		&container.Config{
			Image:  "alpine:3.22",
			Cmd:    []string{"sh", "-c", "echo to-stdout; echo to-stderr >&2"},
			Labels: map[string]string{"created-by": "sdk-demo"}, // label everything you create
		},
		&container.HostConfig{},
		nil, nil, "sdk-demo") // network config, platform, container name
	if err != nil {
		log.Fatal(err)
	}
	defer cli.ContainerRemove(ctx, created.ID, container.RemoveOptions{Force: true})

	if err := cli.ContainerStart(ctx, created.ID, container.StartOptions{}); err != nil {
		log.Fatal(err)
	}

	// Wait returns two channels: one for the exit status, one for errors.
	statusCh, errCh := cli.ContainerWait(ctx, created.ID, container.WaitConditionNotRunning)
	select {
	case err := <-errCh:
		log.Fatal(err)
	case st := <-statusCh:
		log.Printf("exit code: %d", st.StatusCode)
	}

	logs, err := cli.ContainerLogs(ctx, created.ID, container.LogsOptions{ShowStdout: true, ShowStderr: true})
	if err != nil {
		log.Fatal(err)
	}
	defer logs.Close()
	stdcopy.StdCopy(os.Stdout, os.Stderr, logs) // demultiplexes the 8-byte framed stream
}

Good practices for API clients

• Let the SDK negotiate the API version instead of pinning one; pin only in tests that check behavior for a specific version.
• Label every object you create and use label filters (label=created-by=sdk-demo) for listing and cleanup, so your tool never touches resources it did not create.
• Prefer events (/events with filters) over polling /containers/json in a loop.
• Drain or close every stream; an unread pull or build stream blocks the operation.
• Give the tool the least access possible (see the socket proxy in Daemon Socket and API Versions).

Key takeaways

• The Engine API is the single interface behind every Docker client; its OpenAPI specification is the reference for each version.
• Besides plain JSON, the API returns JSON streams, multiplexed raw streams, and hijacked connections, which SDKs decode for you.
• Official SDKs exist for Go and Python; community SDKs cover Java, Node.js, .NET, Rust, and others.
• Robust tools negotiate versions, label what they create, react to events, and always drain streams.

🧪 Practice

1. (Beginner) Using the Python SDK (docker.from_env()), write a script that prints the name, image, and status of every container that has a given label.
2. (Intermediate) Write a small program in Go or Python that subscribes to /events filtered on type=container and event=die, and prints the container name and exit code each time a container exits with a non-zero code.
3. Interview question: "Why do docker logs for a container started with -t look different on the raw API than for one started without -t?" Hint: with a TTY, stdout and stderr are merged into one terminal stream, so there is nothing to multiplex.

Docker Plugins

Why it matters. The Docker daemon ships with drivers for local volumes, bridge and overlay networks, and a set of logging targets. Real environments need more: storage arrays, cloud block devices, custom network fabrics, centralized logging, and access policies for the API. Instead of compiling all of these into dockerd, Docker defines plugin interfaces: small HTTP APIs that the daemon calls when it needs a volume, a network, an IP address, a log destination, or an authorization decision.

Analogy. A plugin is like a power socket standard. The daemon is the building's wiring: it defines the socket shape (the protocol) and calls into whatever is plugged in. Any vendor can build a device for that socket without opening the walls.

How the daemon talks to a plugin. Every plugin is an HTTP server, usually on a Unix socket. The daemon sends POST requests with JSON bodies (content type application/vnd.docker.plugins.v1+json). The first call is always an activation handshake in which the plugin declares which interfaces it implements:

 dockerd                                         plugin (HTTP over /run/docker/plugins/x.sock)
    |-- POST /Plugin.Activate ---------------------->|
    |<-- {"Implements": ["VolumeDriver"]} -----------|
    |-- POST /VolumeDriver.Create {"Name": "v1"} --->|
    |<-- {"Err": ""} --------------------------------|     empty Err = success

Plugin types

Interface	Implements	Selected with
VolumeDriver	Creating, mounting, and removing volumes	docker volume create -d <plugin>, --mount ...,volume-driver=<plugin>
NetworkDriver	Creating networks and connecting containers to them	docker network create -d <plugin>
IpamDriver	Address pools and IP address allocation	docker network create --ipam-driver <plugin>
LogDriver	Receiving container logs	--log-driver <plugin> or log-driver in daemon.json
authz	Allowing or denying each API request	authorization-plugins in daemon.json
MetricsCollector	Receiving the daemon's Prometheus metrics	Enabled automatically when installed

Two generations of plugins

	Legacy plugins	Managed plugins (plugin v2)
Distribution	Any binary or container you start yourself	An image-like artifact on a registry, installed with docker plugin install
Discovery	Socket in /run/docker/plugins/, or a .spec/.json file in /etc/docker/plugins/ or /usr/lib/docker/plugins/	Managed by dockerd; the plugin's config.json declares its interface and socket
Lifecycle	Your responsibility (systemd unit, container)	docker plugin enable, disable, upgrade, rm; started with the daemon
Isolation	Whatever you give it	Runs through containerd (namespace plugins.moby) with only the capabilities, mounts, and devices it declares
Recommended for	Development and testing	Distribution to other hosts

Managing plugins

# Install: Docker shows the privileges the plugin requests and asks for confirmation
docker plugin install --alias sshfs vieux/sshfs DEBUG=1     # KEY=value sets plugin settings
# Plugin "vieux/sshfs" is requesting the following privileges:
#  - network: [host]
#  - mount: [/var/lib/docker/plugins/]
#  - device: [/dev/fuse]
#  - capabilities: [CAP_SYS_ADMIN]
# Do you grant the above permissions? [y/N]

docker plugin ls                  # ID, NAME, DESCRIPTION, ENABLED
docker plugin inspect sshfs       # config, settings, the socket and rootfs paths

# Settings can only be changed while the plugin is disabled
docker plugin disable sshfs
docker plugin set sshfs DEBUG=0
docker plugin enable sshfs

# Upgrade in place (volumes created by the plugin keep working)
docker plugin disable sshfs
docker plugin upgrade sshfs vieux/sshfs:next
docker plugin enable sshfs

docker plugin disable sshfs
docker plugin rm sshfs            # refused while volumes or networks still use it

The managed plugin definition. A managed plugin is a root filesystem plus a config.json that declares what it needs. Docker grants nothing beyond what is declared, and the user sees the privileged parts at install time:

config.json field	Meaning
entrypoint, args, env	The plugin process and its user-settable configuration
interface.types	Interfaces with versions, such as docker.volumedriver/1.0, docker.networkdriver/1.0, docker.authz/1.0, docker.logdriver/1.0
interface.socket	Socket file name the plugin listens on, inside /run/docker/plugins
network.type	host for plugins that must manipulate the host network stack
mounts, propagatedMount	Host paths mounted into the plugin; a propagated mount makes mounts created by the plugin visible to the host (needed by volume plugins)
linux.capabilities, linux.devices, linux.allowAllDevices	Kernel privileges, such as CAP_SYS_ADMIN for mounting filesystems

Authorization plugins. An authz plugin sees every API request before the daemon executes it, and every response before it is returned, which makes it a policy enforcement point for the API (for example "no privileged containers", "no bind mounts of /"). Open Policy Agent's opa-docker-authz plugin is a common implementation.

 client -- POST /containers/create --> dockerd -- AuthZPlugin.AuthZReq --> authz plugin
                                          |   {"User": "alice", "RequestMethod": "POST",
                                          |    "RequestURI": "/v1.52/containers/create",
                                          |    "RequestBody": <base64 JSON>, ...}
                                          |<-- {"Allow": false, "Msg": "privileged containers are not allowed"}
 client <-- 403 "authorization denied by plugin ...: privileged containers are not allowed"

{
  "authorization-plugins": ["opa-docker-authz"]
}

Important limitations: the User field is only filled when clients authenticate with TLS client certificates (over the local socket every caller is anonymous), large or non-JSON request bodies are not forwarded, and bypass vulnerabilities have been found in the past (for example CVE-2024-41110). Treat authz plugins as one layer of defense, keep the Engine patched, and still restrict who can reach the socket.

Key takeaways

• Plugins are HTTP servers that implement Docker-defined interfaces; the daemon activates them with /Plugin.Activate and then calls interface-specific endpoints.
• Interfaces exist for volumes, networks, IPAM, logging, authorization, and metrics.
• Managed plugins are distributed through registries, run isolated through containerd, and declare their privileges in config.json.
• Plugin settings can only be changed while the plugin is disabled.
• Authorization plugins can enforce API policies but have limits; they complement socket access control rather than replace it.

🧪 Practice

1. (Beginner) Install a managed volume plugin on a test VM, read the privileges it requests, and use docker plugin inspect to find its config.json settings, socket, and root filesystem path.
2. (Intermediate) List all plugins and drivers known to your daemon with docker info --format '{{json .Plugins}}' | jq. For each plugin type, identify which drivers are built in and which come from installed plugins.
3. Interview question: "Your security team wants to prevent developers from starting privileged containers on shared build hosts. What options do you have, and what are the weaknesses of an authorization plugin as the only control?" Hint: consider who can reach the socket, what the plugin can see about the caller and the request, and alternatives such as rootless Docker or separate hosts.

Volume and Network Plugins

Why it matters. Volume and network plugins are the two plugin types you are most likely to build or debug. Volume Drivers showed how to use them; this topic shows the protocol from the plugin's side, so you can write one, test it without Docker, and package it as a managed plugin.

The volume plugin protocol

Endpoint	Request	Response	Called when
/VolumeDriver.Create	{"Name", "Opts"}	{"Err"}	docker volume create, or first use of a named volume
/VolumeDriver.Get	{"Name"}	{"Volume": {"Name", "Mountpoint", "Status"}, "Err"}	docker volume inspect, lookups
/VolumeDriver.List	{}	{"Volumes": [...], "Err"}	docker volume ls
/VolumeDriver.Mount	{"Name", "ID"}	{"Mountpoint", "Err"}	A container using the volume starts; ID identifies the caller
/VolumeDriver.Path	{"Name"}	{"Mountpoint", "Err"}	Docker needs the host path of a mounted volume
/VolumeDriver.Unmount	{"Name", "ID"}	{"Err"}	That container stops
/VolumeDriver.Remove	{"Name"}	{"Err"}	docker volume rm
/VolumeDriver.Capabilities	{}	{"Capabilities": {"Scope": "local"}}	Activation; global means the volume is visible from every node (cluster storage)

The key idea: the plugin does not put anything inside the container. Mount returns a host path, and dockerd bind-mounts that path into the container. Everything the plugin does (mounting NFS, attaching a cloud disk, decrypting a filesystem) happens on the host before it returns that path. Because several containers can use the same volume, Mount and Unmount are called once per container, and a real driver must reference-count them: only the last Unmount may detach the storage.

Example: a directory-backed volume plugin in Go

This plugin stores each volume as a subdirectory. It is stateless (everything is derived from the directory listing), so it survives restarts without losing track of its volumes. It uses the go-plugins-helpers library, which implements the HTTP side of the protocol.

package main

import (
	"fmt"
	"log"
	"os"
	"path/filepath"

	"github.com/docker/go-plugins-helpers/volume"
)

// All volumes are subdirectories of root. As a managed plugin, root must be the
// plugin's propagatedMount so that dockerd can see the directories.
const root = "/var/lib/dirvol"

type dirDriver struct{}

// pathFor maps a volume name to its directory and rejects names such as "../etc".
func pathFor(name string) (string, error) {
	if name == "" || name == "." || name == ".." || name != filepath.Base(name) {
		return "", fmt.Errorf("invalid volume name %q", name)
	}
	return filepath.Join(root, name), nil
}

func (dirDriver) Create(r *volume.CreateRequest) error {
	p, err := pathFor(r.Name)
	if err != nil {
		return err
	}
	return os.MkdirAll(p, 0o755) // r.Options holds "-o key=value" flags; ignored here
}

func (dirDriver) Get(r *volume.GetRequest) (*volume.GetResponse, error) {
	p, err := pathFor(r.Name)
	if err != nil {
		return nil, err
	}
	if _, err := os.Stat(p); err != nil {
		return nil, fmt.Errorf("volume %s not found", r.Name)
	}
	return &volume.GetResponse{Volume: &volume.Volume{Name: r.Name, Mountpoint: p}}, nil
}

func (dirDriver) List() (*volume.ListResponse, error) {
	entries, err := os.ReadDir(root)
	if err != nil {
		return nil, err
	}
	var vols []*volume.Volume
	for _, e := range entries {
		if e.IsDir() {
			vols = append(vols, &volume.Volume{Name: e.Name(), Mountpoint: filepath.Join(root, e.Name())})
		}
	}
	return &volume.ListResponse{Volumes: vols}, nil
}

func (dirDriver) Path(r *volume.PathRequest) (*volume.PathResponse, error) {
	p, err := pathFor(r.Name)
	if err != nil {
		return nil, err
	}
	return &volume.PathResponse{Mountpoint: p}, nil
}

// Mount is called once per container (r.ID). A network-backed driver would mount the
// remote filesystem here on the first call and count callers until the last Unmount.
func (dirDriver) Mount(r *volume.MountRequest) (*volume.MountResponse, error) {
	p, err := pathFor(r.Name)
	if err != nil {
		return nil, err
	}
	return &volume.MountResponse{Mountpoint: p}, nil
}

func (dirDriver) Unmount(r *volume.UnmountRequest) error { return nil }

func (dirDriver) Remove(r *volume.RemoveRequest) error {
	p, err := pathFor(r.Name)
	if err != nil {
		return err
	}
	return os.RemoveAll(p)
}

func (dirDriver) Capabilities() *volume.CapabilitiesResponse {
	return &volume.CapabilitiesResponse{Capabilities: volume.Capability{Scope: "local"}}
}

func main() {
	if err := os.MkdirAll(root, 0o755); err != nil {
		log.Fatal(err)
	}
	h := volume.NewHandler(dirDriver{})
	// Listens on /run/docker/plugins/dirvol.sock; dockerd finds the plugin by that name.
	log.Fatal(h.ServeUnix("dirvol", 0))
}

Testing it as a legacy plugin, with and without Docker

go build -o dirvol . && sudo ./dirvol &

# Talk to the plugin directly, exactly as dockerd would
SOCK=/run/docker/plugins/dirvol.sock
curl -s --unix-socket $SOCK -X POST http://localhost/Plugin.Activate
# {"Implements":["VolumeDriver"]}
curl -s --unix-socket $SOCK -X POST -d '{"Name":"probe"}' http://localhost/VolumeDriver.Create
# {"Err":""}
curl -s --unix-socket $SOCK -X POST -d '{"Name":"../etc"}' http://localhost/VolumeDriver.Create
# {"Err":"invalid volume name \"../etc\""}

# Now through Docker
docker volume create -d dirvol notes
docker run --rm -v notes:/data alpine sh -c 'date > /data/created'
sudo cat /var/lib/dirvol/notes/created
docker volume ls --filter driver=dirvol

Packaging it as a managed plugin. A managed plugin is created from a directory with a rootfs/ and a config.json:

# Dockerfile: produces the plugin's root filesystem
FROM golang:1.25 AS build
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 go build -o /dirvol .

FROM alpine:3.22
COPY --from=build /dirvol /dirvol

{
  "description": "Directory-backed volume plugin (demo)",
  "documentation": "https://example.com/dirvol",
  "entrypoint": ["/dirvol"],
  "interface": {
    "types": ["docker.volumedriver/1.0"],
    "socket": "dirvol.sock"
  },
  "network": { "type": "none" },
  "propagatedMount": "/var/lib/dirvol",
  "linux": { "capabilities": ["CAP_SYS_ADMIN"] }
}

docker build -t dirvol-rootfs .
mkdir -p plugin/rootfs
docker export "$(docker create dirvol-rootfs)" | tar -x -C plugin/rootfs   # unpack the image filesystem
cp config.json plugin/

docker plugin create example/dirvol:0.1 ./plugin      # register locally (disabled)
docker plugin enable example/dirvol:0.1
docker volume create -d example/dirvol:0.1 reports
docker plugin push example/dirvol:0.1                 # publish to a registry for other hosts

Network and IPAM plugins. A network plugin follows the same pattern with a richer lifecycle. The daemon asks the IPAM driver for addresses, then asks the network driver to create the plumbing; the driver creates an interface on the host, and Docker moves it into the container's network namespace and renames it:

 docker network create -d mynet --ipam-driver myipam n1
     dockerd -> IpamDriver.RequestPool      {"Pool": "10.50.0.0/24"}  -> {"PoolID", "Pool"}
     dockerd -> IpamDriver.RequestAddress   (gateway)                 -> {"Address": "10.50.0.1/24"}
     dockerd -> NetworkDriver.CreateNetwork {"NetworkID", "IPv4Data", "Options"}

 docker run --network n1 alpine
     dockerd -> IpamDriver.RequestAddress                             -> {"Address": "10.50.0.2/24"}
     dockerd -> NetworkDriver.CreateEndpoint {"EndpointID", "Interface": {"Address": ...}}
     dockerd -> NetworkDriver.Join           {"EndpointID", "SandboxKey": "/var/run/docker/netns/..."}
             <- {"InterfaceName": {"SrcName": "veth9a1", "DstPrefix": "eth"}, "Gateway": "10.50.0.1"}
     dockerd moves veth9a1 into the container's namespace and renames it eth0

 container stops: NetworkDriver.Leave -> DeleteEndpoint -> IpamDriver.ReleaseAddress

NetworkDriver endpoint	Purpose
GetCapabilities	Declares Scope (local or global) and ConnectivityScope
CreateNetwork / DeleteNetwork	Set up or tear down network-wide state (bridges, tunnels, VLANs)
CreateEndpoint / DeleteEndpoint	Prepare one container's attachment (interface, addresses)
Join / Leave	Hand the interface to Docker for the container's namespace, plus gateway and static routes
ProgramExternalConnectivity / RevokeExternalConnectivity	Apply port publishing and external access rules
EndpointOperInfo	Report details shown by docker inspect

In practice, few new Docker network plugins are written today: most third-party networking moved to Kubernetes, which uses the different CNI plugin model. Docker network plugins remain useful for special environments (custom VLANs, appliances, legacy SDNs) and are a good way to understand how libnetwork is structured.

Key takeaways

• A volume plugin returns a host path from Mount; dockerd bind-mounts it into the container.
• Mount and Unmount are called per container, so real drivers must reference-count before detaching storage.
• Plugins can be tested without Docker by sending the protocol's JSON requests to their socket with curl.
• A managed plugin is a rootfs/ plus a config.json, created with docker plugin create and published with docker plugin push; volume plugins need a propagatedMount.
• Network plugins pair with IPAM plugins and hand an interface to Docker in Join; Kubernetes uses CNI instead.

🧪 Practice

1. (Beginner) Run the dirvol plugin as a legacy plugin, create two volumes, use them from containers, and verify with docker volume inspect that the Mountpoint is the plugin's directory.
2. (Intermediate) Extend dirvol with an option -o quota=100M that is stored in a metadata file and reported in Status by Get. Then package it as a managed plugin and confirm that existing volumes are still listed after docker plugin disable and enable.
3. Interview question: "Two containers on the same host use a volume from a network storage plugin. The first container stops and the second one suddenly gets I/O errors. What is the likely bug in the plugin?" Hint: look at how often Unmount is called and what the plugin does on each call.

CLI Plugins

Why it matters. Teams often wrap Docker in scripts: a cleanup routine, a login helper for an internal registry, a command that prints the running services of a project. Shipping these as CLI plugins makes them feel native (docker cleanup instead of ./scripts/docker-cleanup.sh), shows them in docker --help, and lets them follow the user's Docker configuration. Docker itself uses the same mechanism for buildx, compose, and scout, as described in Compose V2 Plugin.

How the CLI finds and runs a plugin. A CLI plugin is any executable named docker-<name> placed in a plugin directory. The name must match ^[a-z][a-z0-9]*$, and a plugin cannot replace a built-in command.

Directory	Scope
~/.docker/cli-plugins/	Current user
/usr/local/lib/docker/cli-plugins/, /usr/local/libexec/docker/cli-plugins/	Locally installed, all users
/usr/lib/docker/cli-plugins/, /usr/libexec/docker/cli-plugins/	Distribution packages
Paths in cliPluginsExtraDirs in ~/.docker/config.json	Custom locations (for example a team's shared tools directory)

 docker cleanup 12h
    |
    |  1. discovery: for every docker-* executable in the plugin directories,
    |     run "docker-<name> docker-cli-plugin-metadata" and read its JSON
    |     (this is how "docker --help" and "docker info" list plugins)
    |
    |  2. execution: run the plugin with the original arguments,
    |     the plugin name included:  docker-cleanup cleanup 12h
    v
 ~/.docker/cli-plugins/docker-cleanup

The metadata contract. When called with the single argument docker-cli-plugin-metadata, the plugin must print a JSON object and exit successfully:

Field	Meaning
SchemaVersion	Must be "0.1.0"
Vendor	Who maintains the plugin (required)
Version	Plugin version, shown by docker info
ShortDescription	One line shown in docker --help
URL	Optional link to documentation

A plugin that does not answer correctly is listed as invalid in docker info and cannot be run.

Example: a shell-script plugin

#!/usr/bin/env bash
# Install: save as ~/.docker/cli-plugins/docker-cleanup and run chmod +x on it
set -euo pipefail

if [[ "${1:-}" == "docker-cli-plugin-metadata" ]]; then
  cat <<'EOF'
{
  "SchemaVersion": "0.1.0",
  "Vendor": "Example Corp Platform Team",
  "Version": "0.2.0",
  "ShortDescription": "Remove stopped containers and dangling images older than a given age",
  "URL": "https://wiki.example.com/docker-cleanup"
}
EOF
  exit 0
fi

# The CLI passes its arguments through; skip everything up to and including our own name
while [[ $# -gt 0 && "$1" != "cleanup" ]]; do shift; done
shift || true

if [[ "${1:-}" == "-h" || "${1:-}" == "--help" ]]; then
  echo "Usage: docker cleanup [AGE]   (default AGE: 24h)"
  exit 0
fi

age="${1:-24h}"
docker container prune --force --filter "until=${age}"
docker image prune --force --filter "until=${age}"

chmod +x ~/.docker/cli-plugins/docker-cleanup

docker --help | grep cleanup
#   cleanup*    Remove stopped containers and dangling images older than a given age (Example Corp Platform Team, 0.2.0)
docker info --format '{{range .ClientInfo.Plugins}}{{.Name}} {{.Version}}{{"\n"}}{{end}}'
# buildx v0.29.1
# compose v2.40.0
# cleanup 0.2.0
docker cleanup 48h

Script plugins compared with Go plugins

Aspect	Script plugin	Go plugin using github.com/docker/cli/cli-plugins/plugin
Effort	A few lines	A Go project with Cobra commands
Metadata subcommand	Implemented by hand	Generated by the library
Global flags (--context, --host, --config) and TLS	Must be handled by the script; a nested docker call only sees the config file and environment	Parsed automatically; the plugin receives a fully configured CLI object and API client
Help, completion, output formatting	By hand	Same conventions as built-in commands
Distribution	Copy one file	One static binary per OS and architecture

Key takeaways

• A CLI plugin is an executable named docker-<name> in a plugin directory; docker <name> runs it.
• It must answer the docker-cli-plugin-metadata subcommand with a JSON object containing at least SchemaVersion and Vendor.
• The plugin receives the original arguments, including its own name, so argument parsing must account for that.
• Script plugins are quick to write; Go plugins built on the official library integrate fully with contexts, flags, and TLS.
• Official tools such as buildx, compose, and scout are CLI plugins, which is why they can be upgraded separately from the CLI.

🧪 Practice

1. (Beginner) Install the docker-cleanup plugin above and confirm it appears in docker --help and docker info. Break the metadata JSON on purpose and observe how docker info reports the plugin.
2. (Intermediate) Write a plugin docker ports that prints a table of every running container with its published host ports, using docker ps --format '{{json .}}' and jq. Make it honor a --context flag passed after the plugin name.
3. Interview question: "Why did Docker move Compose and Buildx out of the CLI binary into plugins?" Hint: think about release cycles, independent upgrades, and the size and dependencies of the core CLI.

Docker Desktop Extensions

Why it matters. Docker Desktop is the daily environment for many developers, and teams often want tools right next to their containers: a log explorer, a database browser, a disk usage analyzer, a vulnerability report, a tunnel to share a local service. Docker Desktop extensions add such tools as new tabs in the Desktop dashboard, installed from the Extensions Marketplace or from any image registry.

What an extension is. An extension is a regular container image with special labels and a metadata.json file. It can contain up to three parts, each running in a different place:

                Docker Desktop (host OS: macOS, Windows, Linux)
  +-----------------------------------------------------------------------+
  |  Dashboard window                                                     |
  |   +-------------------------------+                                   |
  |   | Extension UI (HTML/JS, usually |  @docker/extension-api-client    |
  |   | React), shown as a new tab     |---------------------------------+ |
  |   +-------------------------------+                                 | |
  |        |                        |                                    | |
  |        | ddClient.extension.    | ddClient.extension.host.cli.exec   | |
  |        | vm.service.get(...)    v                                    | |
  |        |                  Host binaries (optional; per-OS tools,     | |
  |        |                  e.g. a CLI that needs host access)         | |
  +--------|-------------------------------------------------------------|-+
           v                                                             v
  +-------------------------- Docker Desktop VM ------------------------------+
  |  Backend service (optional): a container or Compose project, reachable    |
  |  from the UI over a socket declared in metadata.json                      |
  |                                Docker Engine  <-- ddClient.docker.* calls |
  +---------------------------------------------------------------------------+

Part	Runs in	Use it for
UI	Desktop's dashboard (a web view)	The interface; can call the Docker API and the other two parts
Backend (vm)	A container in the Desktop VM	Long-running logic, state, background work, anything that needs a server
Host binaries (host)	The user's operating system	Tools that must run outside the VM (opening files, using the host's credentials)

The extension image

FROM node:22-alpine AS ui
WORKDIR /ui
COPY ui/package.json ui/package-lock.json ./
RUN npm ci
COPY ui/ .
RUN npm run build                              # produces /ui/build

FROM alpine:3.22
LABEL org.opencontainers.image.title="Container Ages" \
      org.opencontainers.image.description="Shows the oldest containers and prunes stopped ones" \
      org.opencontainers.image.vendor="Example Corp" \
      com.docker.desktop.extension.api.version=">= 0.3.4" \
      com.docker.desktop.extension.icon="https://example.com/icon.svg" \
      com.docker.extension.categories="utility-tools"
COPY metadata.json docker.svg ./
COPY --from=ui /ui/build ui

{
  "icon": "docker.svg",
  "ui": {
    "dashboard-tab": {
      "title": "Container Ages",
      "root": "/ui",
      "src": "index.html"
    }
  }
}

A backend is added with a "vm" section (an image or a Compose file, plus "exposes": {"socket": "backend.sock"}), and host tools with a "host": {"binaries": [...]} section listing one path per operating system.

The UI side. The extension SDK gives the UI a client object with access to Docker, the extension's own backend, and Desktop's interface:

import { createDockerDesktopClient } from "@docker/extension-api-client";

const ddClient = createDockerDesktopClient();

interface ContainerSummary { Id: string; Names: string[]; Created: number; State: string }

// Same data as GET /containers/json?all=1, obtained through Docker Desktop
export async function oldestContainers(limit = 5) {
  const containers = (await ddClient.docker.listContainers({ all: true })) as ContainerSummary[];
  const now = Date.now() / 1000;
  return containers
    .sort((a, b) => a.Created - b.Created)            // Created is a Unix timestamp in seconds
    .slice(0, limit)
    .map((c) => ({
      name: c.Names[0].replace(/^\//, ""),            // the API returns names with a leading "/"
      state: c.State,
      ageDays: Math.floor((now - c.Created) / 86400),
    }));
}

// Any docker CLI command can be run, with stdout and stderr returned
export async function pruneStopped() {
  const result = await ddClient.docker.cli.exec("container", ["prune", "--force"]);
  ddClient.desktopUI.toast.success(result.stdout.trim() || "Nothing to prune");
}

// HTTP call to the optional backend container, over the socket declared in metadata.json
export async function backendHealth() {
  return ddClient.extension.vm?.service?.get("/health");
}

Development workflow

docker extension init container-ages          # scaffold: React UI, Go backend, Dockerfile, metadata.json
cd container-ages
docker build -t example/container-ages:0.1 .
docker extension install example/container-ages:0.1

# Hot reload: serve the UI from a local dev server instead of the image
(cd ui && npm start)                          # e.g. http://localhost:3000
docker extension dev ui-source example/container-ages:0.1 http://localhost:3000
docker extension dev debug example/container-ages:0.1      # open Chrome DevTools for the UI
docker extension dev reset example/container-ages:0.1      # back to the files in the image

docker extension validate example/container-ages:0.1       # checks labels and metadata before publishing
docker extension update example/container-ages:0.2
docker extension rm example/container-ages

Security and governance. An extension can call the Docker API, run containers in the VM, and execute binaries on the host, so installing one is a trust decision similar to installing any developer tool. Organizations can disable extensions entirely or restrict them to an allowlist (a private marketplace) through Docker Desktop's centrally managed settings.

Key takeaways

• A Desktop extension is a container image with labels and a metadata.json, installed with docker extension install or from the Marketplace.
• It can have a UI (dashboard tab), a backend running in the Desktop VM, and host binaries.
• The UI uses @docker/extension-api-client to call Docker, the backend, host binaries, and Desktop UI functions such as toasts.
• docker extension dev ui-source and dev debug give a fast development loop with hot reload and DevTools.
• Extensions run with significant privileges; organizations can disable them or allow only approved ones.

🧪 Practice

1. (Beginner) Install an extension from the Marketplace, then find its image with docker extension ls and docker image inspect, and read its labels and metadata.json.
2. (Intermediate) Scaffold an extension with docker extension init, change the UI to show the five oldest containers with their age (as in the example above), and add a button that prunes stopped containers. Use dev ui-source for hot reload while you work.
3. Interview question: "When would you build a Docker Desktop extension instead of a CLI plugin or a standalone web tool?" Hint: consider who the users are, whether they need a graphical view, which platforms they use, and how the tool is distributed.

Alternative Runtimes (gVisor, Kata Containers)

Why it matters. With runc, every container shares the host kernel. A kernel vulnerability reachable through a system call can let a process escape its container, and seccomp, capabilities, and LSMs only reduce that attack surface. When you run code you do not trust (customer workloads, CI jobs for external pull requests, online code execution), you may need a stronger boundary. Sandboxed runtimes keep the container interface (images, docker run, Kubernetes pods) but change what sits between the process and the host kernel. The OCI runtime layer was introduced in Container Runtimes Overview; this topic covers how the sandboxed ones work and how to use them with Docker.

Analogy. runc is a set of apartments in one building: separate doors and walls, but shared foundations, plumbing, and wiring (the kernel). gVisor puts a full-time concierge between each tenant and the building's systems: every request goes through the concierge, who handles most of it personally. Kata Containers gives each tenant a small separate house (a lightweight virtual machine) on the same plot.

Where the system calls go

       runc                    gVisor (runsc)                    Kata Containers
   +-----------+            +-----------+                     +-----------+
   |    app    |            |    app    |                     |    app    |
   +-----+-----+            +-----+-----+                     +-----+-----+
         | syscalls               | syscalls intercepted            | syscalls
         |                        v                                 v
         |                 +--------------+                  +--------------+
         |                 |   Sentry     |  user-space      | guest kernel |  real Linux kernel
         |                 | (kernel      |  kernel in Go    | in a light-  |  inside a VM
         |                 |  written in  |                  | weight VM    |
         |                 |  Go)         |                  +------+-------+
         |                 +------+-------+                         | virtio devices
         |                        | small, filtered set             v
         |                        | of host syscalls         +--------------+
         |                        |                          | hypervisor   |  QEMU, Cloud Hypervisor,
         v                        v                          | (+ KVM)      |  Firecracker
   +-------------------------------------------------------------------------+
   |                              host kernel                                |
   +-------------------------------------------------------------------------+

gVisor. gVisor implements the Linux system call interface in a user-space program called the Sentry. The application's system calls never reach the host kernel directly; the Sentry handles them (it has its own memory management, file system layer, and network stack) and uses only a small set of host system calls itself. File access goes through a separate component (the Gofer, or direct host file access in newer versions) with its own restrictions. The way system calls are intercepted is called the platform: systrap (the default, works everywhere) or kvm (faster on bare metal, uses hardware virtualization for address space isolation).

Kata Containers. Kata starts a lightweight virtual machine for each container (or each Kubernetes pod), with its own minimal guest kernel and an agent that creates the container inside the VM. The isolation boundary is hardware virtualization, the same one public clouds use between customers. The cost is a VM per sandbox: more memory, a slower start, and a requirement for virtualization support (bare metal or nested virtualization).

Comparison

	runc (and crun)	gVisor (runsc)	Kata Containers
Isolation boundary	Namespaces, cgroups, seccomp, LSM on a shared kernel	User-space kernel; host kernel exposure greatly reduced	Hardware virtualization; separate guest kernel
Startup time	Fastest (around 100 ms)	Close to runc	Slower (VM boot, typically sub-second to a few seconds)
Memory overhead	Negligible	Moderate (Sentry per sandbox)	Highest (guest kernel and agent per sandbox)
System-call-heavy and I/O-heavy workloads	Native speed	Noticeably slower	Near native for CPU, some I/O overhead
Compatibility	Full	Most applications; some system calls, /proc details, and devices are not supported	Very high (a real Linux kernel); some host-level Docker features such as --network host do not apply
Requirements	Any Linux host	Any Linux host (systrap); KVM optional	KVM (bare metal or nested virtualization)
Typical use	Trusted workloads	Untrusted code on shared hosts, CI sandboxes, multi-tenant SaaS	Strong tenant isolation, compliance, running custom kernels

Other runtimes plug into the same layer: crun (a faster, lighter runc replacement written in C), youki (Rust), Sysbox (system containers that can run Docker or systemd inside without --privileged; it powers Docker Desktop's Enhanced Container Isolation), and WebAssembly shims from the containerd runwasi project.

Registering runtimes with Docker. Runtimes are declared in daemon.json and selected per container with --runtime:

{
  "runtimes": {
    "runsc": {
      "path": "/usr/local/bin/runsc",
      "runtimeArgs": ["--platform=systrap"]
    },
    "crun": {
      "path": "/usr/bin/crun"
    },
    "kata": {
      "runtimeType": "io.containerd.kata.v2"
    }
  },
  "default-runtime": "runc"
}

Key	How Docker uses it
path (+ runtimeArgs)	An OCI runtime with a runc-compatible command line; Docker runs it through the standard containerd-shim-runc-v2
runtimeType	A containerd shim v2 runtime (see containerd Shims); Docker starts that runtime's own shim, such as containerd-shim-kata-v2
default-runtime	The runtime used when --runtime is not given

gVisor can write its own entry with sudo runsc install. After editing the file, restart Docker (sudo systemctl restart docker) and check docker info --format '{{json .Runtimes}}'.

Example: seeing the difference

# runc: the container sees the host's kernel
docker run --rm alpine uname -r
# 6.8.0-45-generic

# gVisor: the Sentry reports its own emulated kernel version and boot log
docker run --rm --runtime=runsc alpine uname -r
# 4.4.0
docker run --rm --runtime=runsc alpine dmesg | head -3
# [    0.000000] Starting gVisor...
# [    0.412785] Digging up root...
# [    0.588102] Mounting deweydecimalfs...

# Kata: a different, real kernel, the guest kernel of the VM
docker run --rm --runtime=kata alpine uname -r
# 6.12.x  (the guest kernel shipped with Kata, not the host's)

# Runtimes known to the daemon, and the one a container used
docker info --format '{{json .Runtimes}}' | jq 'keys'
docker inspect -f '{{.HostConfig.Runtime}}' <container>

# A shim v2 runtime can also be named directly, without a daemon.json entry (Engine 23+)
docker run --rm --runtime io.containerd.kata.v2 alpine uname -r

Choosing a runtime per workload, not per host. A common pattern is to keep runc as the default and use a sandboxed runtime only where needed: in Compose with runtime: runsc on a specific service, in Kubernetes with a RuntimeClass (see Kubernetes Container Runtime Interface), and in CI by running jobs from external contributors with the sandboxed runtime. Always test the application under the sandbox: compatibility gaps (an unsupported system call, a missing /proc file, a device that cannot be passed through) appear at run time, not at build time.

Key takeaways

• runc isolates with kernel features on a shared kernel; sandboxed runtimes add a second boundary for untrusted code.
• gVisor intercepts system calls in a user-space kernel (the Sentry); Kata runs each container or pod in a lightweight VM with its own kernel.
• Stronger isolation costs performance, memory, startup time, or compatibility, so choose per workload.
• Docker registers runtimes in daemon.json with path (runc-compatible binaries) or runtimeType (containerd shim v2), and selects them with --runtime.
• uname -r and dmesg inside the container are quick ways to verify which runtime is in use.

🧪 Practice

1. (Beginner) Install gVisor on a Linux VM, register it with sudo runsc install, and compare the output of uname -r, dmesg, and cat /proc/cpuinfo | head under runc and under runsc.
2. (Intermediate) Benchmark the same workload under runc and runsc: for example dd if=/dev/zero of=/tmp/f bs=1M count=500, a pgbench run against PostgreSQL, or a test suite of your own. Record the slowdown for each and explain which kinds of operations are affected most.
3. Interview question: "Your company wants to let customers run their own code in containers on shared infrastructure. Is runc with a strict seccomp profile enough? What alternatives would you evaluate?" Hint: compare the size of the attack surface each option exposes to untrusted code, and weigh it against performance, density, and hardware requirements.