2. Docker Architecture and Setup

This chapter opens the "black box" of Docker: which programs make up the Docker Engine, how a single docker run travels from the CLI through the daemon down to the kernel, and why that client-server design matters in practice. It then covers installing and configuring Docker on Linux, macOS, Windows, and WSL2, including rootless operation and daemon configuration. Finally, it introduces the core objects (images, containers, volumes, networks, registries, contexts) that every later chapter works with.

<a id="21-docker-architecture"></a>

2.1 Docker Architecture

This section breaks Docker into its components and follows a request from the command line to a running container.

Docker Engine

Why it matters. "Docker" is used as the name of a company, a product, a CLI, and a file format. When something breaks (a container will not start, the daemon is unreachable, a build is slow), you need to know which part of Docker is responsible. The Docker Engine is the collection of programs that actually builds and runs containers.

What the Docker Engine is. Docker Engine is the open-source container platform developed in the Moby project. It is not a single binary but a stack of cooperating components, each with one job:

Component	Role	Runs as
docker (CLI)	Turns commands into API requests	Short-lived client process
dockerd (daemon)	Exposes the API; manages images, builds, networks, volumes, logging	Long-running service (root by default)
BuildKit	Executes image builds (embedded in dockerd, driven by docker buildx)	Inside the daemon
containerd	Supervises container lifecycles, stores content and snapshots	Long-running service
containerd-shim-runc-v2	Stays alive as the parent of each container	One per container
runc	Creates the container (namespaces, cgroups, rootfs) and exits	Short-lived per container
docker-init (tini)	Optional minimal init as PID 1 (--init)	Inside the container

Analogy. Think of a restaurant. The CLI is the waiter who takes your order, dockerd is the head chef who plans the work and manages the pantry (images, volumes, networks), containerd is the line cook who supervises each dish, and runc is the specialist who plates the dish and then steps away. You only talk to the waiter, but each role is separate and replaceable.

The layered stack

        docker CLI   docker compose   docker buildx   SDKs / IDEs     (clients)
              \            |               |            /
               +-----------+-------+-------+-----------+
                                   |
                     REST API over /var/run/docker.sock
                                   v
   +-------------------------------------------------------------+
   | dockerd   API server | images | BuildKit | networks |        |
   |           volumes | logging drivers | events | plugins       |
   +------------------------------+------------------------------+
                                  | gRPC over /run/containerd/containerd.sock
                                  v
   +-------------------------------------------------------------+
   | containerd   container lifecycle | content store | snapshots |
   +------------------------------+------------------------------+
                                  | one shim per container
                                  v
         containerd-shim-runc-v2 ---> runc ---> container process
                                     (exits)    (namespaces + cgroups)
                                  |
                          Linux kernel (shared)

Editions and related products

Name	What it is
Moby	Upstream open-source project from which Docker Engine is assembled
Docker Engine (Community / CE)	Free, open-source engine packaged for Linux distributions
Docker Desktop	Desktop application that runs Docker Engine inside a managed Linux VM, plus GUI and extra tools; requires a paid subscription for larger companies
Mirantis Container Runtime	Commercial successor of the former Docker Enterprise Edition

Example: see every component and its version

# "Client" is the CLI; "Server" lists the daemon and the components below it
docker version
# Client: Docker Engine - Community
#  Version:           28.5.1          <- your versions will differ
#  API version:       1.51
#  ...
# Server: Docker Engine - Community
#  Engine:
#   Version:          28.5.1
#   API version:      1.51 (minimum version 1.24)
#  containerd:
#   Version:          v1.7.28
#  runc:
#   Version:          1.3.0
#  docker-init:
#   Version:          0.19.0

# Host-wide view: storage driver, cgroup version, runtimes, number of containers/images
docker info

Key takeaways

• Docker Engine is a stack: CLI, dockerd, BuildKit, containerd, shims, and runc.
• Each layer has a narrow responsibility, and the lower layers are OCI-standard components shared with other tools.
• docker version shows client and server separately; a missing "Server" section means the CLI cannot reach the daemon.
• Docker Engine (open source, Linux) and Docker Desktop (VM-based desktop product) are different products.

🧪 Practice

1. (Beginner) Run docker version and docker info. Write down the versions of the engine, containerd, and runc, plus the storage driver and cgroup version.
2. (Intermediate) Stop the daemon (sudo systemctl stop docker docker.socket) and run docker version again. Explain which part of the output disappears and why. Restart the daemon afterwards.
3. Interview question: "What actually happens, component by component, when you type docker run nginx?" Hint: walk down the stack from the CLI to the kernel, and note which component pulls the image and which one exits early.

Docker Daemon (dockerd)

Why a daemon exists. Containers outlive the command that starts them, images must be cached and shared between containers, and networks need firewall rules that persist. Something has to own this shared, long-lived state. In Docker, that is dockerd, a background service that every client talks to.

What dockerd does

Responsibility	Details
API server	Listens on a Unix socket (and optionally TCP or SSH) and serves the Docker Engine API
Image management	Pulls, pushes, stores, and deletes images; runs builds through the embedded BuildKit
Networking	Creates bridges (docker0), virtual Ethernet pairs, IP allocation, embedded DNS, and iptables/nftables rules for published ports
Volumes	Creates and tracks named volumes and volume plugins
Container orchestration	Translates container configuration into containerd calls
Logging and events	Collects container stdout/stderr through logging drivers; emits an event stream
Swarm	Optional built-in cluster orchestrator

Where it keeps its state. By default everything lives under the data root /var/lib/docker: image layers (overlay2/), container metadata and logs (containers/), volumes (volumes/), networks, and the BuildKit cache. Deleting this directory deletes all Docker data.

How it is started. On systemd distributions, docker.service starts dockerd, usually with socket activation through docker.socket. The daemon is configured with command-line flags in the unit file and, preferably, with /etc/docker/daemon.json (see daemon.json Configuration).

What happens if the daemon stops. By default, stopping dockerd stops all containers. With "live-restore": true, containers keep running while the daemon restarts (for example during an upgrade), because they are parented by containerd shims rather than by dockerd itself.

Example: inspecting the daemon

# Service status and the exact command line the daemon was started with
systemctl status docker
systemctl cat docker | grep ExecStart
# ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
#            -H fd:// -> receive the listening socket from systemd (docker.socket)

# Daemon logs (errors at startup, failed pulls, network setup problems)
journalctl -u docker --since "10 min ago"

# The data root
sudo ls /var/lib/docker
# buildkit  containers  engine-id  image  network  overlay2  plugins  runtimes  swarm  tmp  volumes

# Live stream of what the daemon is doing (run a container in another terminal)
docker events --filter type=container
# 2026-10-14T10:02:11 container create 3f2a... (image=alpine, name=quirky_hopper)
# 2026-10-14T10:02:11 container start  3f2a... (image=alpine, name=quirky_hopper)
# 2026-10-14T10:02:11 container die    3f2a... (exitCode=0, ...)

# Troubleshooting: run the daemon in the foreground with debug output
sudo systemctl stop docker docker.socket
sudo dockerd --debug          # Ctrl+C to stop, then: sudo systemctl start docker

Key takeaways

• dockerd is the long-running service that owns images, containers, networks, volumes, logs, and the API.
• Its state lives in the data root, /var/lib/docker by default.
• It runs as root by default, so anyone who can talk to its socket controls the host.
• It delegates actually running containers to containerd; live-restore keeps containers alive across daemon restarts.

🧪 Practice

1. (Beginner) Run docker events in one terminal and docker run --rm alpine echo hi in another. List every event emitted and the order in which they appear.
2. (Intermediate) Check sudo du -sh /var/lib/docker/* before and after pulling postgres:16. Identify which subdirectory grows and explain what it contains.
3. Interview question: "Why do teams enable live-restore on production hosts, and when does it not help?" Hint: think about daemon upgrades versus host reboots, and about which process is the parent of a container.

Docker CLI

Why a separate CLI. The CLI holds no containers, images, or state of its own. It is a thin client that turns human-friendly commands into API calls. This separation means the same CLI can control a local daemon, a remote server, or a VM, and other clients (Compose, IDEs, CI tools) can use the same API.

Command structure. Modern Docker groups commands by the object they act on: docker <object> <verb>. Older top-level shortcuts still work and are widely used.

Management command	Legacy shortcut	Purpose
docker container ls	docker ps	List containers
docker container run	docker run	Create and start a container
docker image ls	docker images	List images
docker image rm	docker rmi	Remove images
docker image build	docker build	Build an image
docker container exec	docker exec	Run a command in a running container

Other management commands: docker volume, docker network, docker context, docker system, docker builder, docker plugin.

Global options and environment. Options placed before the subcommand apply to the whole invocation: --context, -H/--host, --config, --log-level, -D/--debug. The same settings can come from environment variables such as DOCKER_HOST, DOCKER_CONTEXT, and DOCKER_CONFIG.

Client configuration. The CLI stores its settings in ~/.docker/config.json: registry credentials (or the name of a credential helper), the current context, and default output formats. CLI plugins such as compose and buildx are separate binaries found in ~/.docker/cli-plugins/ or /usr/libexec/docker/cli-plugins/.

{
  "auths": { "ghcr.io": {} },
  "credsStore": "desktop",
  "currentContext": "default",
  "psFormat": "table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}"
}

Here credsStore delegates secret storage to an OS keychain helper (docker-credential-desktop), so auths only records which registries you are logged in to, and psFormat changes the default docker ps columns.

Example: productive CLI usage

# Object-oriented and legacy forms are equivalent
docker container ls -a
docker ps -a

# Custom columns with Go templates
docker ps --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}'

# -q prints only IDs, --filter narrows the result; combine to act on many objects
docker ps -aq --filter status=exited
docker rm $(docker ps -aq --filter status=exited)   # remove all exited containers

# Extract a single field from the full JSON description of an object
docker inspect -f '{{.State.Status}} {{.Config.Image}}' my-container

# Every command documents itself
docker container run --help

# Installed CLI plugins
docker info --format '{{range .ClientInfo.Plugins}}{{.Name}} {{.Version}}{{"\n"}}{{end}}'
# buildx v0.29.1
# compose v2.40.0

Key takeaways

• The CLI is a stateless API client; all state lives in the daemon.
• Prefer docker <object> <verb>; legacy shortcuts like docker ps are equivalent.
• --format, -q, and --filter make the CLI scriptable.
• Client settings and credentials live in ~/.docker/config.json; Compose and Buildx are CLI plugins.

🧪 Practice

1. (Beginner) Translate these legacy commands into their management-command form: docker ps -a, docker images, docker rmi alpine, docker exec -it web sh.
2. (Intermediate) Write a one-liner that prints the name and IP address of every running container on the default bridge network using docker ps -q and docker inspect --format.
3. (Intermediate) Set psFormat in your ~/.docker/config.json so that docker ps shows only names, status, and ports. Then run DOCKER_CONFIG=/tmp/empty docker ps and explain why the custom format is gone.

Docker REST API

Why an API. If the only way to control Docker were a CLI, every tool (Compose, CI systems, IDEs, monitoring agents, Kubernetes integrations) would have to shell out and parse text. Instead, dockerd exposes the Docker Engine API, a versioned HTTP/JSON API. The CLI is just one client of it.

How it is exposed

Transport	Address	Notes
Unix socket (default, Linux)	unix:///var/run/docker.sock	Protected by file permissions (root and the docker group)
Named pipe (Windows)	npipe:////./pipe/docker_engine	Default on Windows hosts
SSH	ssh://user@host	Tunnels to the remote socket; uses SSH authentication
TCP with TLS	tcp://host:2376	Requires mutual TLS certificates
TCP without TLS	tcp://host:2375	Unauthenticated: never expose it on a network

Versioning. Every endpoint can be prefixed with an API version, such as /v1.47/containers/json. Clients and daemons negotiate a version both support, which is why an older CLI can talk to a newer daemon. docker version shows both the API version and the minimum the daemon accepts.

Security. Access to the API is equivalent to root on the host: anyone who can call it can start a privileged container that mounts /. Treat the socket like a root password, and be careful when mounting /var/run/docker.sock into a container.

Example: driving Docker with curl

SOCK=/var/run/docker.sock

# Health check and version information
curl --unix-socket $SOCK http://localhost/_ping            # OK
curl --unix-socket $SOCK http://localhost/version | jq .ApiVersion

# Equivalent of "docker ps": list running containers as JSON
curl --unix-socket $SOCK http://localhost/containers/json | jq '.[].Names'

# "docker run" is several API calls. Step 1: pull the image (the API does not auto-pull)
curl --unix-socket $SOCK -X POST "http://localhost/images/create?fromImage=alpine&tag=latest"

# Step 2: create the container
curl --unix-socket $SOCK -X POST -H "Content-Type: application/json" \
  -d '{"Image": "alpine", "Cmd": ["echo", "hello from the API"]}' \
  "http://localhost/containers/create?name=api-demo"
# {"Id":"1c6594faf5d0...","Warnings":[]}

# Step 3: start it, wait for it to exit, read its logs
curl --unix-socket $SOCK -X POST http://localhost/containers/api-demo/start
curl --unix-socket $SOCK -X POST http://localhost/containers/api-demo/wait   # {"StatusCode":0}
curl --unix-socket $SOCK --output - "http://localhost/containers/api-demo/logs?stdout=1"
# (each line is prefixed by an 8-byte stream header because no TTY was allocated)

# Clean up
curl --unix-socket $SOCK -X DELETE http://localhost/containers/api-demo

Example: the official Python SDK

import docker  # pip install docker

client = docker.from_env()  # honours DOCKER_HOST; defaults to the local socket

print(client.version()["ApiVersion"])

# Pulls the image if needed, runs it, waits, and returns the output
output = client.containers.run("alpine", ["echo", "hello from Python"], remove=True)
print(output.decode().strip())

for container in client.containers.list(all=True):
    print(container.short_id, container.name, container.status)

Key takeaways

• dockerd exposes a versioned HTTP/JSON API; the CLI, Compose, and SDKs are all clients of it.
• The default transport is a Unix socket; remote access should use SSH or TLS, never plain TCP.
• High-level commands like docker run are compositions of several API calls.
• API access equals root access on the host.

🧪 Practice

1. (Beginner) Use curl --unix-socket to list your local images (/images/json) and print only their tags with jq.
2. (Intermediate) Start curl --unix-socket /var/run/docker.sock http://localhost/events and, in another terminal, run and remove a container. Compare the JSON stream with the output of docker events.
3. Interview question: "A CI job mounts /var/run/docker.sock into a build container. What is the security impact?" Hint: consider what a process with API access can ask the daemon to create, and with which privileges.

containerd and runc

Why Docker split itself up. Early Docker was one monolithic binary that did everything, including running containers. That made it hard to upgrade the daemon without killing containers, and impossible for other platforms to reuse just the runtime part. In 2016 (Docker 1.11), Docker extracted the low-level pieces: runc (donated to the OCI) and containerd (later donated to the CNCF). Kubernetes and other platforms now use these same components directly.

Division of labour

	containerd	containerd shim	runc
Level	High-level runtime	Per-container helper	Low-level OCI runtime
Lifetime	Long-running daemon	As long as the container	Exits once the container is started
Responsibilities	Container and task lifecycle, image content store, snapshots (layer filesystems), API over gRPC	Parent of the container process; keeps stdio open; reports exit status	Reads an OCI bundle (config.json + rootfs), sets up namespaces, cgroups, mounts, capabilities, then executes the process
Standard	CNCF graduated project	containerd shim API	OCI runtime spec reference implementation

Why the shim matters. runc exits immediately after starting the container, and containerd should be restartable without killing workloads. The shim fills the gap: it becomes the container's parent, holds its stdin/stdout/stderr, and collects its exit code. This is what makes daemonless restarts and live-restore possible. Shim internals are covered in 15.1 Engine Internals.

Docker inside containerd. Docker stores its containers in the containerd namespace moby, separate from Kubernetes (k8s.io) or tools like nerdctl (default). Recent Docker releases can also use containerd's image store instead of Docker's own (the containerd-snapshotter feature, enabled by default on new installations of recent versions).

Example: the process tree of a running container

docker run -d --name web nginx:alpine

ps -ef --forest | grep -E 'containerd|dockerd|nginx' | grep -v grep
# root  1201     1  /usr/bin/containerd
# root  1350     1  /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
# root  4821     1  /usr/bin/containerd-shim-runc-v2 -namespace moby -id 9f1c... -address /run/containerd/containerd.sock
# root  4842  4821   \_ nginx: master process nginx -g daemon off;
# 101   4890  4842       \_ nginx: worker process
#
# Note: the shim's parent is PID 1, not dockerd or containerd, and runc does not appear at all

# Look at the same container from containerd's point of view
sudo ctr namespaces ls
# NAME LABELS
# moby
sudo ctr -n moby containers ls      # container IDs match "docker ps --no-trunc"
sudo ctr -n moby tasks ls           # TASK = container ID, PID = 4842, STATUS = RUNNING

# Which low-level runtime does Docker use?
docker info --format '{{.DefaultRuntime}}'     # runc

Example: running a container with runc alone

mkdir -p ~/runc-demo/rootfs && cd ~/runc-demo

# Use Docker only to obtain a root filesystem
docker export "$(docker create alpine)" | tar -C rootfs -xf -

runc spec                 # writes a default OCI config.json (args: ["sh"], terminal: true)
sudo runc run demo        # interactive shell as PID 1 in new namespaces; "exit" to leave

No daemon, no image, no network: only the OCI bundle and runc. Everything above runc in the Docker stack exists to produce such a bundle and manage it over time.

Key takeaways

• containerd is the long-running supervisor; runc is a short-lived tool that creates a container and exits.
• A shim process per container keeps it alive independently of dockerd and containerd.
• Docker's containers live in the containerd namespace moby; the same containerd also powers Kubernetes nodes.
• runc consumes an OCI bundle; alternative runtimes (crun, gVisor, Kata) can replace it via --runtime.

🧪 Practice

1. (Beginner) Start a container and find its shim process with ps. Confirm that the container's main process is a child of the shim.
2. (Intermediate) Enable "live-restore": true, restart dockerd, and check that the container's host PID (docker inspect -f '{{.State.Pid}}') has not changed. Repeat without live-restore and compare.
3. Interview question: "Why does Docker need both containerd and runc instead of having dockerd create containers directly?" Hint: consider upgrades without downtime, reuse by Kubernetes, and the OCI separation between high-level and low-level runtimes.

Client-Server Model

Why it matters. Because the CLI and the daemon communicate over an API, they do not have to run on the same machine. This is powerful (manage a remote build server from your laptop) but also a common source of confusion: file paths, ports, and environment variables in a command refer to the daemon's machine, not the client's.

The three parties

   CLIENT HOST                          DOCKER HOST                       REGISTRY
 +---------------+   API request   +----------------------+   pull/push  +-------------+
 |  docker CLI   | --------------> |  dockerd             | <----------> | Docker Hub, |
 |  compose      |  unix / ssh /   |   +- images          |              | GHCR, ECR,  |
 |  SDKs         |  tls            |   +- containers      |              | Harbor ...  |
 +---------------+ <-------------- |   +- volumes         |              +-------------+
                     JSON / stream |   +- networks        |
                                   |  containerd + runc   |
                                   +----------------------+

What runs where

Item	Lives on
CLI config, credentials, contexts	Client
Images, containers, volumes, networks	Docker host
Bind mount source paths (-v /src:/app)	Docker host
Published ports (-p 8080:80)	Docker host's interfaces
Build context (docker build .)	Client, uploaded to the daemon (filtered by .dockerignore)
Container logs	Docker host, streamed to the client on request

Example: using a remote daemon over SSH

# One-off command against a remote host (requires SSH access and Docker on that host)
docker -H ssh://deploy@build01.example.com info

# Point the whole shell session at the remote daemon
export DOCKER_HOST=ssh://deploy@build01.example.com

docker run -d -p 8080:80 nginx       # port 8080 is opened on build01, not on your laptop
docker run --rm -v "$PWD":/src alpine ls /src
#   $PWD is resolved on the client, but /src is mounted from that path on build01:
#   you see build01's directory (or an empty dir), not your local files

docker build -t myapp .              # the local "." is packed and sent to build01

unset DOCKER_HOST                    # back to the local daemon

Key takeaways

• Docker is a client-server system: the CLI sends API requests, the daemon does the work.
• The client and daemon can be on different machines, connected over SSH or TLS.
• Paths, ports, volumes, and networks are resolved on the daemon's host; only the build context is uploaded from the client.
• Docker Desktop is itself a client-server setup: the CLI runs on your OS, the daemon inside a VM.

🧪 Practice

1. (Beginner) Explain why docker build prints "transferring context" and why a large node_modules directory without a .dockerignore slows every build.
2. (Intermediate) If you have access to a second Linux machine (or VM), run docker -H ssh://user@host run -d -p 8080:80 nginx and determine on which machine curl localhost:8080 succeeds.
3. Interview question: "A developer runs docker run -v ./config:/etc/app myimage against a remote daemon and the container sees an empty directory. Why?" Hint: identify on which machine the bind mount source is resolved.

<a id="22-installation-and-configuration"></a>

2.2 Installation and Configuration

This section covers getting a working, secure, and well-configured Docker installation on each major platform.

Installing Docker on Linux

Why the install method matters. Linux is the only platform where Docker runs natively, without a VM. Most distributions ship their own Docker package, but these are often older, packaged differently, or missing the Compose and Buildx plugins. Docker's own repositories provide current releases with all components versioned together.

Installation options

Method	Pros	Cons	Use for
Docker's apt/dnf repository	Current, signed, updates with the system package manager	One-time repository setup	Recommended for most hosts
Distribution package (docker.io, moby-engine)	No extra repository	Often older; plugin set varies	Environments that only allow distro packages
Convenience script (get.docker.com)	One command	Runs a remote script as root; no version pinning	Throwaway test VMs only
Static binaries	No package manager needed	Manual updates, no service files	Unusual or minimal systems

Example: Ubuntu (Debian is analogous)

# 1. Remove conflicting unofficial packages (safe if none are installed)
for pkg in docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc; do
  sudo apt-get remove -y $pkg
done

# 2. Add Docker's GPG key
sudo apt-get update
sudo apt-get install -y ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# 3. Add the repository; packages must be signed by the key above
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] \
https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" \
  | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# 4. Install the engine, CLI, containerd, and the Buildx/Compose plugins
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# 5. Verify (on Debian/Ubuntu the service starts automatically)
sudo docker run --rm hello-world

Example: Fedora (RHEL-family distributions are similar)

sudo dnf -y install dnf-plugins-core
sudo dnf config-manager addrepo --from-repofile https://download.docker.com/linux/fedora/docker-ce.repo
sudo dnf install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# On Fedora/RHEL the service is not started automatically
sudo systemctl enable --now docker
sudo docker run --rm hello-world

Packages installed

Package	Provides
docker-ce	dockerd and the systemd units
docker-ce-cli	The docker CLI
containerd.io	containerd, the shim, and runc
docker-buildx-plugin	docker buildx (BuildKit client)
docker-compose-plugin	docker compose (Compose v2)

Uninstalling. Removing the packages leaves data in place. To remove everything, also delete /var/lib/docker and /var/lib/containerd.

Key takeaways

• Install from Docker's official repository to get current, signed packages with Buildx and Compose.
• Remove distribution-provided Docker packages first to avoid conflicts.
• Five packages make up a full install: engine, CLI, containerd (with runc), Buildx, Compose.
• Verify with docker run hello-world; enable the service manually on RHEL-family systems.

🧪 Practice

1. (Beginner) Install Docker on a fresh Linux VM using the official repository and run hello-world. Record the versions shown by docker version.
2. (Intermediate) Use apt-cache madison docker-ce (or dnf list docker-ce --showduplicates) to install a specific older version, then upgrade it. Check whether running containers survive the upgrade.
3. Interview question: "Why is piping curl https://get.docker.com | sh discouraged for production servers?" Hint: think about reproducibility, version pinning, and trusting remote code executed as root.

Docker Desktop for Windows and macOS

Why a VM is needed. Containers share the host's Linux kernel, but macOS and Windows do not have one. Docker Desktop solves this by running a small, managed Linux VM and placing Docker Engine inside it. The CLI on your desktop talks to that engine through a forwarded socket, so it feels native.

Architecture

   macOS / Windows host
 +---------------------------------------------------------------+
 |  docker CLI, compose, IDE  ---socket / named pipe---+         |
 |  Docker Desktop app (GUI, settings, updates)        |         |
 |                                                     v         |
 |   +-------------------------------------------------------+   |
 |   | Linux VM  (Apple Virtualization framework on macOS,   |   |
 |   |            WSL2 or Hyper-V on Windows)                |   |
 |   |   dockerd -> containerd -> runc -> containers         |   |
 |   +-------------------------------------------------------+   |
 |  file sharing (VirtioFS / 9P) and port forwarding to host     |
 +---------------------------------------------------------------+

What is included. Docker Engine, CLI, Compose, Buildx, a single-node Kubernetes cluster (optional), Docker Scout, the extensions marketplace, a GUI for containers/images/volumes, and automatic port forwarding so -p 8080:80 is reachable on localhost of the host OS.

Practical considerations

Topic	What to know
Licensing	Free for personal use, education, non-commercial open source, and small businesses (fewer than 250 employees and less than $10M annual revenue); larger organizations need a paid subscription
Resources	CPU, memory, and disk for the VM are set in Settings > Resources (on Windows with WSL2, via .wslconfig)
File sharing	Bind mounts from the host cross the VM boundary and are slower than native Linux; keep heavy I/O inside volumes
Host access	Containers reach services on the host via host.docker.internal
CPU architecture	On Apple silicon, images are arm64; amd64 images run through emulation (Rosetta or QEMU), more slowly

Alternatives. Colima and OrbStack (macOS), Rancher Desktop, and Podman Desktop also provide a Linux VM with a Docker-compatible API.

Example: confirming you are talking to a VM

docker context ls
# NAME              DESCRIPTION                               DOCKER ENDPOINT
# default           Current DOCKER_HOST based configuration   unix:///var/run/docker.sock
# desktop-linux *   Docker Desktop                            unix:///Users/me/.docker/run/docker.sock

docker info --format '{{.OperatingSystem}} / {{.Architecture}}'
# Docker Desktop / aarch64

# The kernel reported inside containers is the VM's Linux kernel, not macOS/Windows
docker run --rm alpine uname -r
# 6.10.14-linuxkit

# Open a shell in the VM itself: join PID 1's namespaces from a privileged container
docker run --rm -it --privileged --pid=host alpine nsenter -t 1 -m -u -n -i sh

Key takeaways

• Docker Desktop runs Docker Engine inside a Linux VM, because containers need a Linux kernel.
• The CLI runs on the host OS and reaches the engine through a forwarded socket (desktop-linux context).
• Bind-mount I/O and non-native CPU architectures are the main performance pitfalls.
• Check licensing before using Docker Desktop in a larger company; alternatives exist.

🧪 Practice

1. (Beginner) On Docker Desktop, run docker run --rm alpine uname -a and explain why the output mentions Linux even though the host is macOS or Windows.
2. (Intermediate) Time npm install (or any I/O-heavy task) in a container using a bind mount from the host, then using a named volume. Explain the difference.
3. Interview question: "Why do Docker images built on an Apple silicon Mac sometimes fail on a production x86 server?" Hint: look at the image's platform and the --platform flag.

Docker on WSL2

Why WSL2 matters for Docker. WSL2 (Windows Subsystem for Linux 2) runs a real Linux kernel in a lightweight, tightly integrated VM. That makes it the best way to run Linux containers on Windows: fast startup, low overhead, and direct access to Linux filesystems and tools from a Windows desktop.

Two ways to use Docker with WSL2

	Docker Desktop with WSL2 backend	Docker Engine installed inside a WSL2 distro
Where dockerd runs	In Docker Desktop's own docker-desktop WSL distro	Directly in your distro (e.g. Ubuntu)
Access from distros	"WSL integration" setting exposes the CLI and socket in chosen distros	Only in that distro (unless exposed explicitly)
Access from Windows (PowerShell)	Yes	Not by default
GUI, Kubernetes, extensions	Yes	No
License	Docker Desktop terms	Open source (Docker Engine)
Setup	Installer	Linux install steps plus systemd enabled

Prerequisite: systemd in the distro (for a native engine). Docker Engine is managed by systemd, which WSL2 distros run only if enabled in /etc/wsl.conf.

# /etc/wsl.conf inside the distro
[boot]
systemd=true

# %UserProfile%\.wslconfig on Windows: limits for the whole WSL2 VM
[wsl2]
memory=8GB        # without a limit, WSL2 may take a large share of host RAM
processors=4

Example: native Docker Engine inside Ubuntu on WSL2

# PowerShell on Windows
wsl --install -d Ubuntu      # install WSL2 and an Ubuntu distro
wsl -l -v                    # VERSION column must show 2
wsl --shutdown               # restart the WSL VM after editing wsl.conf / .wslconfig

# Inside Ubuntu: confirm systemd is PID 1, then follow the normal Linux install
ps -p 1 -o comm=             # systemd
# ... install docker-ce as shown in "Installing Docker on Linux" ...
sudo systemctl enable --now docker
docker run --rm hello-world

Performance rule: keep files in the Linux filesystem. Paths under /mnt/c/... cross a network-like file protocol between Windows and Linux and are much slower for bind mounts and builds. Keep projects in the distro's home directory (~/projects) and open them from Windows via \\wsl$\Ubuntu\home\<user>.

Avoid mixing both setups. If Docker Desktop's WSL integration is enabled for a distro that also runs its own dockerd, both compete for /var/run/docker.sock. Use one or the other per distro.

Key takeaways

• WSL2 provides a real Linux kernel on Windows, which is what Docker needs.
• Use Docker Desktop's WSL2 backend for an integrated experience, or install Docker Engine natively in a distro with systemd enabled.
• Keep source code inside the Linux filesystem, not /mnt/c, for good performance.
• Limit WSL2 memory and CPU in .wslconfig.

🧪 Practice

1. (Beginner) Run wsl -l -v and identify which distros run WSL2. If Docker Desktop is installed, find its docker-desktop distro in the list.
2. (Intermediate) Clone the same repository into /mnt/c/tmp/repo and ~/repo, then run a Docker build with each as the context. Compare the build times and explain the gap.
3. (Intermediate) Set a memory limit in .wslconfig, restart WSL, and verify the new limit with docker info --format '{{.MemTotal}}'.

Post-Installation Steps

Why post-installation matters. A fresh install works, but it is not yet comfortable or production-ready: every command needs sudo, the daemon may not start at boot, container logs grow without limit, and published ports may bypass your firewall. A few minutes of setup avoids these problems.

1. Start Docker at boot

sudo systemctl enable --now docker.service containerd.service

2. Use Docker without sudo (the docker group)

sudo groupadd docker            # usually already created by the package
sudo usermod -aG docker "$USER" # add yourself to the group
newgrp docker                   # apply to the current shell (or log out and back in)
docker run --rm hello-world     # no sudo needed now

The socket /var/run/docker.sock is owned by root:docker with mode 660, so group members can use it. Membership in the docker group is equivalent to root access, because a member can run docker run -v /:/host --privileged .... Grant it only to trusted users, or use rootless mode (next topic).

3. Understand the hello-world output. It describes the full client-server flow covered in 2.1:

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

4. Configure log rotation. The default json-file logging driver never rotates logs, so a chatty container can fill the disk. Set limits in /etc/docker/daemon.json (see daemon.json Configuration).

5. Check for warnings and firewall behaviour

docker info 2>&1 | grep -i warning
# e.g. "WARNING: No swap limit support" means swap limits are unavailable on this kernel

Docker manages its own iptables/nftables rules for published ports, and these are evaluated before typical ufw or firewalld rules. A container started with -p 5432:5432 is reachable from the network even if ufw denies port 5432. Bind development services to localhost instead:

docker run -d -p 127.0.0.1:5432:5432 -e POSTGRES_PASSWORD=secret postgres:16

6. Optional conveniences. Shell completion (installed with the package for bash; for zsh/fish use docker completion zsh), and docker login to raise Docker Hub pull rate limits.

Key takeaways

• Enable docker and containerd services so they start at boot.
• The docker group removes the need for sudo but grants root-equivalent power.
• Configure log rotation before containers accumulate logs.
• Published ports can bypass host firewalls; bind to 127.0.0.1 for local-only services.

🧪 Practice

1. (Beginner) Add your user to the docker group and verify that docker ps works without sudo in a new login shell.
2. (Intermediate) As a docker group member (not using sudo), read /etc/shadow by running a container with the host root mounted. Explain what this proves about the group.
3. Interview question: "You enabled ufw and denied all incoming traffic, yet a database container is reachable from the internet. Why?" Hint: compare where Docker inserts its NAT/forwarding rules with where ufw filters traffic.

Running Docker Without Root

Why rootless. By default, dockerd runs as root, and any vulnerability in the daemon or a container escape lands an attacker as root on the host. Rootless mode runs the daemon and all containers as an unprivileged user. Root inside a container maps to your user ID on the host, so a breakout gains only that user's permissions.

How it works. Rootless Docker relies on user namespaces: inside the namespace your user appears as UID 0, and a range of subordinate IDs (from /etc/subuid and /etc/subgid) covers other users in the container. rootlesskit sets up the namespace, and networking is provided in user space (by slirp4netns or pasta) because an unprivileged user cannot create bridges or iptables rules on the host.

Comparing the options

Approach	Daemon runs as	Container root maps to	Protection
sudo docker ...	root	root	None: only who can run sudo
docker group	root	root	None: group members are root-equivalent
userns-remap in daemon.json	root	Unprivileged subordinate UID	Protects against container escapes, not daemon bugs
Rootless mode	Your user	Your user's subordinate UIDs	Protects against both

Example: setting up rootless mode on Ubuntu

# Prerequisites: newuidmap/newgidmap and the rootless extras package
sudo apt-get install -y uidmap docker-ce-rootless-extras

# Your user needs a subordinate ID range (usually created with the user)
grep "^$USER:" /etc/subuid /etc/subgid
# /etc/subuid:alice:100000:65536
# /etc/subgid:alice:100000:65536

# Recommended: disable the system-wide rootful daemon to avoid confusion
sudo systemctl disable --now docker.service docker.socket

# Install and start the per-user daemon (run as your normal user, without sudo)
dockerd-rootless-setuptool.sh install

# Point the CLI at the per-user socket (the tool also creates a "rootless" context)
export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/docker.sock   # e.g. /run/user/1000/docker.sock
docker context use rootless

# Manage it as a user service; keep it running after logout
systemctl --user enable --now docker
sudo loginctl enable-linger "$USER"

# Verify
docker info --format '{{.SecurityOptions}}'   # includes name=rootless
docker run --rm alpine id                     # uid=0(root) inside...
docker run -d --name sleeper alpine sleep 300
ps -o user,pid,cmd -C sleep                   # ...but your user on the host

Limitations to know

Limitation	Workaround
Cannot publish ports below 1024	sudo sysctl net.ipv4.ip_unprivileged_port_start=80, or use high ports
Resource limits (--memory, --cpus) need cgroup v2 and systemd delegation	Use a modern distro with cgroup v2
Network throughput is lower (user-space networking)	Use pasta, or rootful Docker for network-heavy workloads
Some features unavailable (e.g. overlay networks, AppArmor, --net=host to the real host)	Use rootful Docker where required
Data lives in ~/.local/share/docker	Plan disk space in the home partition

Key takeaways

• Rootless mode runs both the daemon and containers as an unprivileged user using user namespaces.
• It is the strongest protection of the options; the docker group offers none.
• It needs uidmap, subordinate ID ranges, and works best with cgroup v2 and systemd.
• Expect limits on privileged ports, networking performance, and some features.

🧪 Practice

1. (Beginner) In rootless mode, run docker run --rm alpine id and then look at the same process from the host with ps. Explain the two different user IDs.
2. (Intermediate) Try docker run -p 80:80 nginx in rootless mode, read the error, and fix it with the appropriate sysctl. Explain why the restriction exists.
3. Interview question: "What is the difference between rootless Docker and userns-remap?" Hint: consider which user the daemon itself runs as, and what an attacker gains by exploiting the daemon in each case.

daemon.json Configuration

Why a configuration file. The daemon has dozens of settings: storage location, logging, registries, networking, security. Passing them as flags in the systemd unit is fragile and gets overwritten on upgrades. daemon.json keeps the daemon's configuration in one versionable JSON file.

Where it lives

Setup	Path
Linux (rootful)	/etc/docker/daemon.json
Linux (rootless)	~/.config/docker/daemon.json
Windows (Docker Engine)	C:\ProgramData\docker\config\daemon.json
Docker Desktop	Settings > Docker Engine (JSON editor)

Example: a practical baseline

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  },
  "live-restore": true,
  "data-root": "/srv/docker",
  "default-address-pools": [
    { "base": "10.200.0.0/16", "size": 24 }
  ],
  "registry-mirrors": ["https://mirror.internal.example.com"],
  "features": {
    "containerd-snapshotter": true
  }
}

JSON does not allow comments, so here is what each key does:

Key	Effect
log-driver, log-opts	Default logging driver; rotate each container's log at 10 MB, keep 3 files (applies to new containers only)
live-restore	Keep containers running while the daemon restarts
data-root	Move /var/lib/docker to another disk
default-address-pools	Subnets used for new networks; avoids clashes with corporate or VPN ranges (default bridge uses 172.17.0.0/16)
registry-mirrors	Pull-through cache for Docker Hub
insecure-registries	Registries allowed over plain HTTP (avoid outside labs)
dns	DNS servers given to containers
userns-remap	Remap container root to an unprivileged user
features.containerd-snapshotter	Use the containerd image store
debug	Verbose daemon logs

Validating and applying changes

# Check syntax and option names before restarting
sudo dockerd --validate --config-file /etc/docker/daemon.json
# configuration OK

# Some options (debug, labels, live-restore, registry mirrors, insecure-registries)
# can be reloaded without stopping anything (SIGHUP)
sudo systemctl reload docker

# Everything else (data-root, storage driver, address pools, ...) needs a restart
sudo systemctl restart docker

# Confirm the effective settings
docker info --format '{{.LoggingDriver}} {{.DockerRootDir}} {{.LiveRestoreEnabled}}'

A common pitfall: options set twice. A setting may come from a flag or from daemon.json, never both. The systemd unit already passes -H fd://, so adding "hosts" to daemon.json makes the daemon refuse to start:

unable to configure the Docker daemon with file /etc/docker/daemon.json:
the following directives are specified both as a flag and in the configuration file: hosts

The fix is a systemd drop-in (sudo systemctl edit docker) that clears ExecStart and restarts dockerd without -H.

Key takeaways

• daemon.json is the single place to configure dockerd; keep it under version control.
• Always set log rotation; change default-address-pools if Docker's subnets clash with your network.
• Validate with dockerd --validate before restarting; some options reload with SIGHUP, others need a restart.
• An option set both as a flag and in the file prevents the daemon from starting.

🧪 Practice

1. (Beginner) Add log rotation to daemon.json, restart Docker, start a new container, and confirm its settings with docker inspect -f '{{.HostConfig.LogConfig}}'. Check an older container too and explain the difference.
2. (Intermediate) Configure default-address-pools with a custom range, create a new network, and verify its subnet with docker network inspect.
3. (Intermediate) Move the data root to a different directory without losing existing images. Describe the steps (stop, copy, configure, start) and verify with docker info.

<a id="23-docker-objects-overview"></a>

2.3 Docker Objects Overview

This section introduces the objects the daemon manages; each one has its own management command (docker image, docker volume, ...) and is covered in depth in later chapters.

Images

Why images exist. To run the same software everywhere, you need a portable, immutable package of the application and everything it depends on. An image is that package: a read-only template from which containers are created.

What an image is. An image is a stack of read-only filesystem layers plus a JSON config (default command, environment variables, exposed ports, working directory, architecture). Each layer records the file changes made by one build step. Layers are content-addressed by their SHA-256 digest, so identical layers are stored and downloaded only once, even when shared by many images.

Naming: tags versus digests. A tag (nginx:1.27) is a human-friendly, mutable pointer: the publisher can move it to a new build. A digest (nginx@sha256:...) identifies exact content and never changes. Use digests when you need reproducibility.

   nginx:1.27-alpine                     (tag -> manifest digest)
   +------------------------------------+
   | config: CMD ["nginx","-g",...]     |
   |         ExposedPorts: 80/tcp       |
   +------------------------------------+
   | layer 4: nginx config files        |  sha256:9a1c...
   | layer 3: nginx binaries            |  sha256:51b2...
   | layer 2: packages                  |  sha256:e4f0...
   | layer 1: alpine base filesystem    |  sha256:3d7a...  (shared with every alpine image)
   +------------------------------------+

Example

docker image pull nginx:1.27-alpine
docker image ls
# REPOSITORY   TAG            IMAGE ID       CREATED       SIZE
# nginx        1.27-alpine    1ff4bb4faebc   2 weeks ago   47.9MB

docker image history nginx:1.27-alpine           # one row per layer, with the instruction that created it
docker image inspect -f '{{.Config.Cmd}} {{.Architecture}}' nginx:1.27-alpine
docker image inspect -f '{{index .RepoDigests 0}}' nginx:1.27-alpine   # the immutable digest

docker image tag nginx:1.27-alpine myweb:v1      # a second name for the same image ID
docker image rm myweb:v1                         # removes only the tag; layers stay while referenced

Images are covered fully in 4. Docker Images and building them in 5. Building Images with Dockerfiles.

Key takeaways

• An image is an immutable stack of layers plus a config describing how to run it.
• Layers are content-addressed and shared between images.
• Tags are mutable names; digests are immutable identifiers.

🧪 Practice

1. (Beginner) Pull python:3.13-slim and python:3.13-alpine. Compare their sizes and number of layers with docker image ls and docker image history.
2. (Intermediate) Pull two images built on the same base (for example nginx:alpine and redis:alpine) and observe in the pull output which layers are reported as "Already exists". Explain why.
3. Interview question: "Why should production deployments reference images by digest rather than by tag such as latest?" Hint: consider what happens if the tag is moved between your test run and your deployment.

Containers

Why containers are separate from images. One image must be able to serve many independent instances, each with its own state, network address, and configuration. A container is such an instance: the image's read-only layers plus a thin writable layer, wrapped in namespaces and cgroups, and running a process.

What defines a container. Its image, its writable layer, its configuration (command, environment variables, ports, mounts, resource limits, restart policy), and its state. Containers move through a lifecycle:

            docker create          docker start            process exits / docker stop
   image ---------------> created -------------> running ---------------------------> exited
                                                  |   ^                                   |
                                     docker pause |   | docker unpause                    | docker rm
                                                  v   |                                   v
                                                 paused                                (removed)

Ephemeral by design. Changes written inside a container live only in its writable layer and disappear when the container is removed. Containers should be disposable: anything worth keeping goes into a volume (next topic) or a new image.

Example

docker container run -d --name web -p 8080:80 nginx:alpine   # create + start, detached
docker container ls                                          # running containers
docker container exec web sh -c 'echo hi > /tmp/note'        # write into the writable layer
docker container diff web                                    # A /tmp/note (plus nginx's own changes)
docker container stop web                                    # state: exited, data still present
docker container ls -a --filter name=web                     # STATUS: Exited (0) ...
docker container start web && docker exec web cat /tmp/note  # hi  (survives stop/start)
docker container rm -f web                                   # writable layer is deleted

Container operations are covered in 3. Working with Containers.

Key takeaways

• A container is a runtime instance of an image with its own writable layer and configuration.
• States: created, running, paused, exited (plus restarting and removing).
• Data in the writable layer survives stop/start but not removal.

🧪 Practice

1. (Beginner) Create a container with docker create, check its state, then start it. Describe what differs between the two states in docker inspect.
2. (Intermediate) Run three containers from the same image with different environment variables and published ports. Show that each has its own IP address and writable layer.
3. Interview question: "Why is it considered bad practice to docker exec into a production container and patch files?" Hint: think about what happens on the next redeploy and about reproducibility.

Volumes

Why volumes exist. A database in a container that loses its data whenever the container is replaced is useless. Volumes store data outside a container's writable layer, so it survives container removal, can be shared between containers, and performs better than the copy-on-write layer.

Storage options at a glance

Type	Managed by	Source	Typical use
Named volume	Docker	/var/lib/docker/volumes/<name>/_data	Databases, persistent app data
Anonymous volume	Docker	Random name, created for a VOLUME in the image	Scratch data tied to a container
Bind mount	You	Any path on the host	Source code during development, config files
tmpfs mount	Kernel (RAM)	Memory only	Secrets or temporary data that must not hit disk

Example: data survives the container

docker volume create pgdata

docker run -d --name db -e POSTGRES_PASSWORD=secret \
  -v pgdata:/var/lib/postgresql/data postgres:16      # mount the volume at the data directory

docker exec db psql -U postgres -c "CREATE TABLE notes(t text); INSERT INTO notes VALUES ('kept');"
docker rm -f db                                       # container gone ...

docker run -d --name db2 -e POSTGRES_PASSWORD=secret \
  -v pgdata:/var/lib/postgresql/data postgres:16      # ... new container, same volume
sleep 3; docker exec db2 psql -U postgres -c "SELECT * FROM notes;"   # kept

docker volume ls
docker volume inspect pgdata -f '{{.Mountpoint}}'     # /var/lib/docker/volumes/pgdata/_data
docker rm -f db2 && docker volume rm pgdata

Storage is covered in 6. Data Persistence and Storage.

Key takeaways

• Volumes keep data independent of any container's lifecycle.
• Prefer named volumes for application data and bind mounts for development source code.
• Removing a container does not remove its named volumes; clean them up explicitly.

🧪 Practice

1. (Beginner) Create a named volume, write a file into it from one alpine container, and read it from another.
2. (Intermediate) Run docker run -d redis twice and inspect the anonymous volumes created. Explain where they come from and how docker run --rm treats them.
3. Interview question: "When would you choose a bind mount over a named volume, and vice versa?" Hint: compare portability, who manages the path, and performance on Docker Desktop.

Networks

Why networks exist. Containers need to talk to each other (an app to its database) and to the outside world, while staying isolated from unrelated containers. Docker networks provide virtual networks with their own subnets, DNS, and isolation boundaries.

Default networks. Every installation has three:

Network	Driver	Behaviour
bridge	bridge	Default for new containers; private subnet behind NAT on docker0; no DNS resolution by container name
host	host	Container shares the host's network stack; no isolation, no port mapping needed
none	null	Loopback only; fully isolated

User-defined networks. Networks you create (docker network create) add an embedded DNS server, so containers reach each other by name, and they isolate groups of containers from one another. Other drivers include overlay (multi-host, Swarm), macvlan/ipvlan (containers appear directly on the physical LAN).

   user-defined bridge "appnet" (10.200.1.0/24, DNS via 127.0.0.11)
   +-----------------------------------------------------------+
   |   +-----------+        "db:5432"        +-----------+     |
   |   |   api     | ----------------------> |    db     |     |
   |   +-----+-----+                         +-----------+     |
   +---------|-------------------------------------------------+
             | -p 8080:8000 (NAT on the host)
        host network / internet

Example

docker network ls
docker network create appnet

docker run -d --name db  --network appnet -e POSTGRES_PASSWORD=secret postgres:16
docker run --rm --network appnet alpine ping -c 2 db        # resolves "db" via Docker DNS

docker run --rm alpine ping -c 1 db                         # default bridge: "bad address 'db'"

docker network inspect appnet -f '{{range .Containers}}{{.Name}} {{.IPv4Address}}{{"\n"}}{{end}}'
docker rm -f db && docker network rm appnet

Networking is covered in 7. Docker Networking.

Key takeaways

• Default networks: bridge, host, none.
• Always create user-defined networks for multi-container apps: they provide name-based DNS and isolation.
• Publishing ports (-p) exposes a container to the outside; container-to-container traffic on the same network needs no publishing.

🧪 Practice

1. (Beginner) Create two user-defined networks and one container in each. Verify that they cannot reach each other, then connect one container to both networks with docker network connect.
2. (Intermediate) Run docker run --rm --network host nginx on a Linux host and explain why curl localhost:80 works without -p.
3. Interview question: "Why can containers on a user-defined bridge resolve each other by name, but not on the default bridge?" Hint: look into the embedded DNS server and the historical --link mechanism.

Registries

Why registries exist. Images must travel from where they are built to every machine that runs them. A registry is a server that stores and distributes images over the OCI distribution API, much like a package repository for containers.

Vocabulary. A registry (e.g. ghcr.io) hosts many repositories (e.g. acme/api), and each repository holds many tags pointing to image manifests.

     ghcr.io  /  acme/api  :  1.4.2    @sha256:5b0b...
     -------     --------     -----     -----------
     registry   repository     tag     digest (optional, immutable)

   Shorthand expansion on Docker Hub:
     nginx            ->  docker.io/library/nginx:latest
     bitnami/redis    ->  docker.io/bitnami/redis:latest

Common registries

Registry	Type	Notes
Docker Hub (docker.io)	Public / private SaaS	Default registry; official images; pull rate limits, lower for anonymous users
GitHub Container Registry (ghcr.io)	SaaS	Tied to GitHub repositories and permissions
Amazon ECR, Google Artifact Registry, Azure ACR	Cloud	Integrated with each cloud's IAM
Harbor, GitLab Registry, registry:2	Self-hosted	Private, on-premises control

Example: push to a local registry

docker run -d --name registry -p 5000:5000 registry:2   # a local registry on localhost:5000

docker pull alpine:3.20
docker tag alpine:3.20 localhost:5000/tools/alpine:3.20  # the name decides where it is pushed
docker push localhost:5000/tools/alpine:3.20

curl -s http://localhost:5000/v2/_catalog                # {"repositories":["tools/alpine"]}
docker image rm localhost:5000/tools/alpine:3.20
docker pull localhost:5000/tools/alpine:3.20             # pulled back from the local registry

# Authenticated registries: credentials go to ~/.docker/config.json or a credential helper
docker login ghcr.io -u <github-user>                    # paste a token when prompted
docker rm -f registry

Registries are covered in 4.3 Image Registries.

Key takeaways

• A registry stores repositories; a repository stores tagged image manifests.
• The image name encodes the registry; without a hostname, Docker Hub (docker.io/library/) is assumed.
• docker login stores credentials on the client; prefer credential helpers over plain-text storage.

🧪 Practice

1. (Beginner) Write the fully qualified reference for postgres, grafana/grafana:11.2.0, and ghcr.io/acme/api.
2. (Intermediate) Push an image to a local registry:2, then use curl against /v2/<repo>/tags/list and /v2/<repo>/manifests/<tag> (with an Accept: application/vnd.oci.image.index.v1+json header) to explore what was stored.
3. Interview question: "Your CI pipeline starts failing with HTTP 429 errors when pulling base images. What is happening and how would you fix it?" Hint: consider Docker Hub rate limits, authentication, and mirrors or pull-through caches.

Docker Contexts

Why contexts exist. Many developers work with more than one daemon: Docker Desktop locally, a rootless daemon, a remote build server, a staging host. Switching by exporting DOCKER_HOST is error-prone. A context is a named, saved connection profile (endpoint plus TLS settings) that you can switch between with one command.

Analogy. Contexts are like saved Wi-Fi networks: you store the connection details once, then pick by name.

Which daemon a command targets. The CLI resolves the endpoint in this order:

Priority	Source
1	--context or -H/--host flag on the command
2	DOCKER_HOST environment variable
3	DOCKER_CONTEXT environment variable
4	currentContext in ~/.docker/config.json (set by docker context use)
5	The default context (local socket)

Example

docker context ls
# NAME        DESCRIPTION                               DOCKER ENDPOINT
# default *   Current DOCKER_HOST based configuration   unix:///var/run/docker.sock

# Save a remote daemon reachable over SSH
docker context create staging \
  --description "Staging server" \
  --docker "host=ssh://deploy@staging.example.com"

docker --context staging ps        # one-off command against staging
docker context use staging         # make it the default for every new shell
docker ps                          # now lists staging containers
docker context use default         # switch back

# Per-shell override without changing the global setting
DOCKER_CONTEXT=staging docker compose up -d

docker context inspect staging     # full endpoint configuration as JSON
docker context rm staging

If DOCKER_HOST is set, it overrides the selected context, and docker context ls warns about it. Unset it when contexts seem to be ignored.

Key takeaways

• A context is a named daemon endpoint; docker context use switches the default one.
• Contexts work with local sockets, SSH, and TLS endpoints, and Docker Desktop and rootless mode create their own.
• Flags beat DOCKER_HOST, which beats DOCKER_CONTEXT, which beats the saved current context.

🧪 Practice

1. (Beginner) Run docker context ls and identify which context is active and which socket it uses.
2. (Intermediate) Create a context for a remote host (or for a rootless daemon on the same machine), run a container through it, and verify with docker --context default ps that the local daemon does not see it.
3. Interview question: "A teammate ran docker compose down -v and accidentally wiped volumes on a shared server instead of their laptop. How could contexts both cause and prevent this?" Hint: think about persistent docker context use, environment overrides, and making the active context visible in your shell prompt.

---

<a id="3-working-with-containers"></a>