1. Containerization Fundamentals

This chapter explains what containers are, where they came from, and which Linux kernel features make them possible. It then covers the open standards (OCI) and the runtimes that implement them, so that Docker can be understood as one tool in a larger ecosystem rather than a black box. Every later chapter builds on these ideas: image layers, isolation, resource limits, and security all come straight from these kernel primitives.

<a id="11-introduction-to-containers"></a>

1.1 Introduction to Containers

This section defines what a container is, how it differs from a virtual machine, and why containers became the standard way to package and run software.

What Is a Container

Why containers exist. Software rarely runs alone. An application depends on a language runtime, system libraries, configuration files, environment variables, and sometimes specific OS packages. When any of these differ between a developer's laptop, the CI server, and production, the result is the classic "it works on my machine" problem. Before containers, teams tried to solve this with long setup documents, configuration management scripts, or full virtual machines, which were slow and heavy.

The analogy. Before standardized shipping containers, cargo was loaded piece by piece, and every ship, truck, and crane needed to handle each item differently. The standard steel container changed that: whatever is inside, the outside has the same shape, so any ship or crane can move it. Software containers do the same: whatever language or stack is inside, any container runtime can start, stop, move, and run it the same way.

What a container actually is. A container is an ordinary Linux process (or group of processes) that the kernel isolates and limits:

• It has its own view of the system: its own filesystem root, process tree, network interfaces, hostname, and users. This is done with namespaces.
• Its use of CPU, memory, I/O, and process count can be capped. This is done with control groups (cgroups).
• Its filesystem comes from an image, a read-only, layered package containing the application and everything it needs, plus a thin writable layer on top.
• It shares the host kernel. There is no guest operating system and no hardware emulation.

Image vs container. An image is the template; a container is a running (or stopped) instance of that template. The relationship is like a class and its objects: one image can back many independent containers.

                 +--------------------------+
                 |   Image: nginx:alpine    |   read-only template
                 |  (layers: OS libs, nginx,|   (filesystem + metadata)
                 |   config, default CMD)   |
                 +------------+-------------+
                              |
          +-------------------+-------------------+
          |                   |                   |
   +------v------+     +------v------+     +------v------+
   | container A |     | container B |     | container C |   each has its own
   | writable    |     | writable    |     | writable    |   writable layer,
   | layer       |     | layer       |     | layer       |   namespaces, cgroup
   +-------------+     +-------------+     +-------------+
          \                   |                   /
           +------------------+------------------+
                              |
                    shared host Linux kernel

Example: a container is just a process on the host

# Start an nginx container in the background (detached)
docker run -d --name web nginx:alpine

# Inside the container, nginx appears as PID 1 of its own process tree
docker exec web ps
# PID   USER     TIME  COMMAND
#     1 root      0:00 nginx: master process nginx -g daemon off;
#    30 nginx     0:00 nginx: worker process

# On the host (native Linux, not Docker Desktop) the same processes are visible
# with different, host-level PIDs
docker inspect -f '{{.State.Pid}}' web    # e.g. 48213 = host PID of container's PID 1
ps -o pid,cmd -p "$(docker inspect -f '{{.State.Pid}}' web)"

# Clean up
docker rm -f web

On Docker Desktop (macOS/Windows), containers run inside a hidden Linux VM, so the host ps command will not show them. The principle is the same; the "host" is simply that VM.

Key takeaways

• A container is an isolated, resource-limited process that shares the host kernel; it is not a lightweight VM.
• Isolation comes from namespaces, limits from cgroups, and the filesystem from a layered image.
• An image is an immutable template; a container is a runtime instance of it with its own writable layer.
• Containers solve environment drift by packaging the application together with its dependencies.

🧪 Practice

1. (Beginner) Run docker run --rm alpine cat /etc/os-release on a non-Alpine host. Explain why the output says Alpine even though the host runs a different distribution.
2. (Intermediate) Start two containers from the same image, create a file in one of them with docker exec, and verify that it does not appear in the other. Explain where the file physically lives.
3. Interview question: "If a container is just a process, what makes it different from any other process on the host?" Hint: think about what the process can see versus what it can use, and which kernel features control each.

Containers vs Virtual Machines

Why compare them. Virtual machines were the dominant isolation technology before containers, and both are used to run multiple workloads on the same hardware. Knowing the trade-offs explains when to pick one, the other, or both.

How a VM works. A hypervisor (KVM, VMware ESXi, Hyper-V, Xen) virtualizes hardware. Each VM boots a complete guest operating system with its own kernel, drivers, and init system. Isolation is strong because each guest only sees virtual hardware, but each VM pays the cost of a full OS: gigabytes of disk, hundreds of megabytes of RAM, and boot times of tens of seconds.

How a container works. A container runtime asks the host kernel to isolate a process. There is only one kernel, shared by all containers. Startup is essentially the time it takes to start a process, typically well under a second, and the image contains only user-space files.

Analogy. VMs are separate houses, each with its own foundation, plumbing, and electrical system. Containers are apartments in one building: each unit has its own locked door and private space, but they share the foundation and utilities (the kernel).

        Virtual Machines                          Containers
+-----------+-----------+-----------+   +-----------+-----------+-----------+
|  App A    |  App B    |  App C    |   |  App A    |  App B    |  App C    |
| Bins/Libs | Bins/Libs | Bins/Libs |   | Bins/Libs | Bins/Libs | Bins/Libs |
| Guest OS  | Guest OS  | Guest OS  |   +-----------+-----------+-----------+
| (kernel)  | (kernel)  | (kernel)  |   |   Container runtime (Docker)      |
+-----------+-----------+-----------+   +-----------------------------------+
|           Hypervisor              |   |   Host OS  (one shared kernel)    |
+-----------------------------------+   +-----------------------------------+
|       Host OS / bare metal        |   |          Infrastructure           |
+-----------------------------------+   +-----------------------------------+
|          Infrastructure           |
+-----------------------------------+

Aspect	Virtual Machine	Container
Unit of isolation	Virtualized hardware	Isolated process
Kernel	One per VM (guest kernel)	Shared host kernel
Typical size	GBs (full OS)	MBs to hundreds of MBs
Startup time	Tens of seconds to minutes	Milliseconds to seconds
Density per host	Tens	Hundreds to thousands
Run a different OS family	Yes (Windows on Linux host, etc.)	No; Linux containers need a Linux kernel
Security boundary	Strong (hardware virtualization)	Weaker (kernel attack surface is shared)
Resource overhead	Higher (guest OS, emulated devices)	Near-native
Typical use	Multi-tenant isolation, different OSes, legacy systems	Application packaging, microservices, CI/CD

They are often combined. Cloud providers run containers inside VMs for tenant isolation. Docker Desktop runs a Linux VM to host Linux containers on macOS and Windows. Projects such as Kata Containers and Firecracker blur the line by giving each container its own lightweight VM.

Example: the kernel is shared

# Kernel version on the host
uname -r
# 6.8.0-45-generic

# Kernel version reported inside containers built from very different distributions
docker run --rm alpine uname -r          # 6.8.0-45-generic  (same kernel)
docker run --rm debian:12 uname -r       # 6.8.0-45-generic  (same kernel)
docker run --rm fedora:40 uname -r       # 6.8.0-45-generic  (same kernel)

# But the user space (distribution) is different in each
docker run --rm alpine cat /etc/alpine-release
docker run --rm debian:12 cat /etc/debian_version

Key takeaways

• VMs virtualize hardware and run a full guest kernel; containers virtualize the OS view and share the host kernel.
• Containers are smaller, faster to start, and denser; VMs offer stronger isolation and can run different OS kernels.
• A container's "distribution" is only its user-space files; the kernel always belongs to the host.
• In practice the two are complementary: containers commonly run inside VMs.

🧪 Practice

1. (Beginner) Run uname -r on your host and inside an alpine and an ubuntu container. Record the results and explain them.
2. (Intermediate) Measure startup time with time docker run --rm alpine true and compare it to the boot time of a VM you have access to (or a published figure). Explain where the difference comes from.
3. Interview question: "Why can't you run a Windows container natively on a Linux host, yet you can run a Linux VM on a Windows host?" Hint: consider what each technology needs from the underlying kernel versus the underlying hardware.

Benefits of Containerization

Why it matters. Containers did not become popular because they are a new kind of isolation (the kernel features existed for years). They became popular because they made packaging, shipping, and running software repeatable and fast. Each benefit below maps to a concrete pain point that existed before.

Benefit	Problem it solves	How containers achieve it
Portability	App runs differently across machines	Image bundles app + dependencies; runs anywhere a compatible runtime exists
Consistency (dev/prod parity)	Drift between environments	The same immutable image is promoted from dev to test to prod
Isolation	Dependency conflicts (two apps need different library versions)	Each container has its own filesystem and libraries
Efficiency	VMs waste resources on duplicate OSes	Shared kernel, near-native performance, small footprint
Speed	Slow provisioning and deployments	Containers start in milliseconds; images are cached and layered
Reproducibility	"Snowflake" servers configured by hand	Environment is defined as code (Dockerfile) and versioned
Scalability	Hard to add capacity quickly	Start more identical instances; orchestrators automate this
Immutability	In-place patching leads to unknown state	Replace containers with new ones built from a new image

Environment as code. The key shift is that the environment is described in a file that lives next to the source code, gets reviewed, and is rebuilt the same way every time:

# The full runtime environment of the app, versioned with the code
FROM python:3.12-slim                         # exact language runtime version
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt   # pinned dependencies
COPY . .
CMD ["python", "app.py"]                      # how the app starts

Isolation of conflicting dependencies. Different versions of the same tool can run side by side without installing anything on the host:

docker run --rm python:3.9  python --version   # Python 3.9.x
docker run --rm python:3.13 python --version   # Python 3.13.x
docker run --rm node:18 node --version         # v18.x
docker run --rm node:22 node --version         # v22.x

The trade-offs. Containers are not free of cost, and a balanced view is important:

Limitation	Explanation
Weaker isolation than VMs	A kernel vulnerability can affect every container on the host
Persistent data needs planning	The container filesystem is ephemeral; volumes are required for state
Operational complexity at scale	Networking, orchestration, observability, and security tooling are needed
Kernel dependency	Linux containers need a Linux kernel (hence a VM on macOS/Windows)
Image hygiene	Images can carry outdated packages and vulnerabilities if not rebuilt regularly

Key takeaways

• The main value of containers is repeatability: the same image behaves the same everywhere.
• Environment-as-code (Dockerfiles) replaces manual setup and makes environments reviewable and versioned.
• Containers isolate dependencies, so conflicting versions can coexist on one host.
• They trade some isolation strength and add operational concerns (state, networking, security) in exchange for speed and efficiency.

🧪 Practice

1. (Beginner) Run two different major versions of a language runtime (for example node:18 and node:22) side by side and print their versions. Confirm nothing was installed on your host.
2. (Intermediate) Take a small application you have written and list every dependency needed to run it on a fresh machine (runtime, packages, environment variables, system libraries). Then write a Dockerfile that captures all of them.
3. Interview question: "Containers are said to provide dev/prod parity. What can still differ between environments even when the same image is used?" Hint: think about what is injected at runtime rather than baked into the image.

History of Containers

Why history matters. Docker did not invent isolation; it packaged decades of operating-system work into a usable developer experience. Knowing the lineage explains why containers rely on specific Linux features and why standards like OCI exist.

The timeline

1979  chroot (Unix V7)        change a process's root directory -> filesystem isolation
 |
2000  FreeBSD Jails           chroot + separate process, network and user space
 |
2001  Linux-VServer           OS-level virtualization patches for Linux
 |
2002  Mount namespace         first Linux namespace (kernel 2.4.19)
 |
2004  Solaris Zones           full OS-level virtualization on Solaris
 |
2005  OpenVZ                  commercial-grade Linux containers (patched kernel)
 |
2006  "Process containers"    developed at Google; renamed cgroups and
2008  cgroups merged          merged into Linux 2.6.24
 |
2008  LXC                     first complete container manager using only
 |                            upstream kernel features (namespaces + cgroups)
2013  Docker released         dotCloud open-sources Docker: images, layers,
 |                            Dockerfile, registry -> containers for developers
2013  User namespaces done    Linux 3.8 completes the main namespace set
 |
2014  libcontainer            Docker replaces LXC with its own Go library
2014  Kubernetes announced    Google open-sources a container orchestrator
2014  rkt / appc (CoreOS)     competing runtime and image format
 |
2015  OCI founded             Open Container Initiative standardizes image
 |                            and runtime formats; Docker donates runc
2015  CNCF founded            Kubernetes becomes its first project
 |
2016  cgroups v2              unified hierarchy (Linux 4.5)
2016  containerd spun out     Docker extracts its core runtime
2017  containerd -> CNCF      Docker becomes Moby + Docker CE/EE
2017  OCI 1.0                 image-spec and runtime-spec 1.0 released
 |
2020  dockershim deprecated   Kubernetes moves to CRI runtimes (containerd, CRI-O)
2022  dockershim removed      Kubernetes 1.24

The key turning point: 2013. LXC already provided working containers in 2008, but using them was complex and there was no standard way to package and share an application. Docker's contribution was the developer experience:

Before Docker (LXC era)	Docker's contribution
Containers configured as mini-servers	One application per container
No portable packaging format	Layered images with a simple build file (Dockerfile)
No standard distribution	Public registry (Docker Hub) with push/pull
Manual, multi-step setup	One command: docker run

From one tool to an ecosystem. As containers grew, the industry split Docker's monolith into standards and reusable components: OCI specifications for formats, runc for low-level execution, containerd for lifecycle management, and the Kubernetes CRI for orchestrator integration. Section 1.3 covers these in detail.

Key takeaways

• Container concepts date back to chroot (1979), FreeBSD Jails (2000), and Solaris Zones (2004).
• Linux containers are built on namespaces (starting 2002) and cgroups (merged 2008).
• Docker (2013) made containers mainstream through images, Dockerfiles, and registries rather than new isolation technology.
• Standardization (OCI, 2015) and componentization (runc, containerd) turned Docker's internals into a shared ecosystem.

🧪 Practice

1. (Beginner) For each of chroot, FreeBSD Jails, and LXC, write one sentence describing what it added compared to its predecessor.
2. (Intermediate) Explain why Kubernetes removed dockershim, and why Docker-built images still run on Kubernetes clusters that use containerd or CRI-O.
3. Interview question: "Docker is often credited with inventing containers. Is that accurate?" Hint: separate the isolation technology from the packaging and distribution workflow.

Common Use Cases

Why it matters. Knowing where containers shine (and where they are a poor fit) helps you decide when to reach for them. Most use cases exploit one of three properties: fast startup, reproducible environments, or isolation of dependencies.

Use case	Why containers fit	Typical example
Microservices	Each service is packaged, versioned, and scaled independently	An orders API in Java and a payments API in Go on the same cluster
CI/CD pipelines	Every build runs in a clean, identical environment	Running tests inside node:22 on every pull request
Local development	Spin up databases and queues without installing them	docker compose up starts Postgres, Redis, and the app
Integration testing	Disposable dependencies created and destroyed per test run	Testcontainers starting a real database for a test suite
Running CLI tools	Use a tool without installing it or its runtime	Running a linter or formatter from an image
Batch and scheduled jobs	Start, process, exit; no idle servers	Nightly report generation as a Kubernetes CronJob
Machine learning	Reproducible stacks (CUDA, Python, libraries)	Training jobs pinned to an exact framework version
Legacy application packaging	Freeze an old runtime without polluting hosts	An app needing an old Java version on a modern host
Edge and IoT	Small footprint, consistent remote updates	Pushing a new image to devices in the field

Example: disposable infrastructure for development

# A throwaway PostgreSQL instance for local development or testing
# -e sets the password required by the official postgres image
# -p exposes the database on localhost:5432
docker run -d --name devdb \
  -e POSTGRES_PASSWORD=secret \
  -p 5432:5432 \
  postgres:16

# ... use it from your application at localhost:5432 ...

# Remove it completely when done; nothing is left installed on the host
docker rm -f devdb

Example: running a tool without installing it

# Run Prettier on the current directory using Node from an image
# -v mounts the current directory into the container, -w makes it the working directory
docker run --rm \
  -v "$PWD":/work \
  -w /work \
  node:22 npx --yes prettier --check .

Where containers are a weaker fit

• Workloads needing a different kernel or kernel modules (use a VM).
• Strong multi-tenant isolation of untrusted code without additional sandboxing (use VMs, gVisor, or Kata).
• GUI desktop applications, which are possible but awkward.
• Very stateful, latency-critical systems where operational simplicity of bare metal wins.

Key takeaways

• Containers are ideal where reproducibility, fast startup, or dependency isolation matter.
• Development, CI, and testing benefit as much as production deployment.
• Disposable containers replace locally installed services and tools.
• Choose VMs or sandboxed runtimes when you need a different kernel or strong isolation of untrusted code.

🧪 Practice

1. (Beginner) Start a Redis container, connect to it with docker exec -it <name> redis-cli, set and get a key, then remove the container.
2. (Intermediate) Pick a CLI tool you use regularly (for example a linter or terraform) and write a shell alias that runs it from a container with the current directory mounted.
3. Interview question: "Your team wants to run customer-submitted code in containers on shared hosts. What concerns would you raise?" Hint: consider what all containers on a host have in common and what alternatives strengthen that boundary.

<a id="12-linux-foundations-of-containers"></a>

1.2 Linux Foundations of Containers

This section opens the black box: containers are assembled from a handful of Linux kernel features, and each one can be used directly from the shell.

Namespaces

Why namespaces exist. A normal Linux process can see every other process, every network interface, every mount point, and the machine's hostname. To make a process believe it is alone on a machine, the kernel must give it a private view of these global resources. That is exactly what a namespace does.

Analogy. Think of a namespace as a pair of tinted glasses. The world (the kernel's resources) has not changed, but each process wearing a particular pair only sees the part of the world that its glasses allow.

What a namespace is. A namespace wraps a global system resource so that processes inside it get their own isolated instance of that resource. Every process belongs to exactly one namespace of each type. A container is simply a process placed in a fresh set of namespaces.

Namespace	Clone flag	Isolates	Effect inside a container
Mount (mnt)	CLONE_NEWNS	Mount points	Own filesystem tree and mounts
PID	CLONE_NEWPID	Process IDs	Own process tree; the main process is PID 1
Network (net)	CLONE_NEWNET	Interfaces, routes, firewall rules, ports	Own eth0, IP address, and port space
UTS	CLONE_NEWUTS	Hostname and domain name	Own hostname
IPC	CLONE_NEWIPC	System V IPC, POSIX message queues	Cannot use other containers' shared memory
User	CLONE_NEWUSER	User and group IDs	Root inside can map to an unprivileged user outside
Cgroup	CLONE_NEWCGROUP	Cgroup root view	Sees its own cgroup as the root
Time	CLONE_NEWTIME	Boot and monotonic clocks	Own clock offsets (Linux 5.6+)

How they are created. Three system calls manage namespaces:

• clone() creates a new process in new namespaces.
• unshare() moves the calling process into new namespaces.
• setns() joins an existing namespace (this is how docker exec enters a running container).

Example: inspecting and creating namespaces by hand

# Each namespace of the current shell is a symlink; the number is the namespace ID
ls -l /proc/$$/ns
# lrwxrwxrwx ... mnt -> 'mnt:[4026531841]'
# lrwxrwxrwx ... net -> 'net:[4026531840]'
# lrwxrwxrwx ... pid -> 'pid:[4026531836]'
# ...

# New UTS namespace: change the hostname without affecting the host
sudo unshare --uts /bin/bash
hostname container-demo
hostname          # container-demo
exit
hostname          # original host name is unchanged

# New PID + mount namespace: --fork makes bash PID 1,
# --mount-proc remounts /proc so ps only sees this namespace
sudo unshare --pid --fork --mount-proc /bin/bash
ps aux
# USER  PID ... COMMAND
# root    1 ... /bin/bash
# root    8 ... ps aux
exit

# List all namespaces on the system and which process owns them
sudo lsns

Example: namespaces in Docker

# Default: the container has its own PID namespace
docker run --rm alpine ps            # only "ps" is visible, as PID 1

# Share the host PID namespace: the container sees every host process
docker run --rm --pid=host alpine ps | head

# Share the host network namespace: no isolation of interfaces or ports
docker run --rm --network=host alpine ip addr

# Compare namespace IDs of a container with the host
docker run -d --name ns-demo alpine sleep 300
sudo ls -l /proc/$(docker inspect -f '{{.State.Pid}}' ns-demo)/ns
docker rm -f ns-demo

Key takeaways

• Namespaces control what a process can see: processes, network, mounts, hostname, IPC, users, cgroups, time.
• A container is a process running in a new set of namespaces; there is no separate "container object" in the kernel.
• unshare, clone, and setns are the syscalls behind creating and entering containers.
• Docker flags such as --pid=host and --network=host selectively disable isolation by sharing the host's namespace.
• The user namespace is what allows "root in the container" to be an unprivileged user on the host.

🧪 Practice

1. (Beginner) Use sudo unshare --uts /bin/bash to change the hostname inside a new namespace, then verify from another terminal that the host's hostname is unchanged.
2. (Intermediate) Start a container with sleep 600, find its host PID, and use sudo nsenter --target <pid> --net ip addr to view its network configuration from the host. Explain how this relates to docker exec.
3. Interview question: "Inside a container, ps shows the application as PID 1. What special responsibilities does PID 1 have, and why does it matter for containers?" Hint: think about signal handling and orphaned child processes.

Control Groups (cgroups)

Why cgroups exist. Namespaces hide resources but do not limit them. A container that cannot see other processes can still consume all the host's memory or CPU and starve every other workload (the "noisy neighbor" problem). Control groups solve this by grouping processes and enforcing resource accounting and limits on the group.

Analogy. If namespaces are walls between apartments, cgroups are the utility meters and circuit breakers: each apartment gets a measured, capped share of electricity and water.

What cgroups do

• Limit: cap memory, CPU time, I/O bandwidth, number of processes.
• Prioritize: give some groups a larger share of CPU or I/O under contention.
• Account: measure what each group consumes (this powers docker stats).
• Control: freeze and resume all processes in a group (this powers docker pause).

Controller	Controls	Docker flag example
memory	RAM usage, swap; triggers OOM kill when exceeded	--memory=512m
cpu	CPU time quota and relative weight	--cpus=1.5, --cpu-shares=512
cpuset	Which CPU cores and memory nodes may be used	--cpuset-cpus=0,1
io (v2) / blkio (v1)	Block device read/write bandwidth and IOPS	--device-read-bps /dev/sda:10mb
pids	Maximum number of processes (fork bomb protection)	--pids-limit=100
freezer	Suspend and resume processes	docker pause

cgroups v1 vs v2. Modern distributions use cgroups v2, and Docker supports both.

Aspect	cgroups v1	cgroups v2
Hierarchy	One hierarchy per controller	Single unified hierarchy
Mount point	/sys/fs/cgroup/<controller>/...	/sys/fs/cgroup/...
Memory limit file	memory.limit_in_bytes	memory.max
CPU limit file	cpu.cfs_quota_us + cpu.cfs_period_us	cpu.max
Rootless container support	Limited	Full, with delegation
Pressure information (PSI)	No	Yes (memory.pressure, cpu.pressure)

Example: creating a cgroup by hand (v2)

# Check which cgroup version the host uses: "cgroup2fs" means v2
stat -fc %T /sys/fs/cgroup

# A cgroup is just a directory in the cgroup filesystem
sudo mkdir /sys/fs/cgroup/demo

# Set limits by writing to control files
# (the memory and pids controllers must be enabled in the parent's cgroup.subtree_control)
echo 50M | sudo tee /sys/fs/cgroup/demo/memory.max    # hard memory limit
echo 20  | sudo tee /sys/fs/cgroup/demo/pids.max      # at most 20 processes

# Move the current shell into the cgroup; its children inherit membership
echo $$ | sudo tee /sys/fs/cgroup/demo/cgroup.procs
cat /proc/self/cgroup        # 0::/demo

# Any process started from this shell is now limited to 50 MB and 20 PIDs.
# Exit the shell, then remove the (now empty) cgroup:
sudo rmdir /sys/fs/cgroup/demo

Example: cgroups through Docker

# Start a container with memory, CPU, and process limits:
#   --memory=256m    -> memory.max = 268435456
#   --cpus=0.5       -> cpu.max = "50000 100000" (50 ms of CPU per 100 ms period)
#   --pids-limit=100 -> pids.max = 100
docker run -d --name limited \
  --memory=256m \
  --cpus=0.5 \
  --pids-limit=100 \
  nginx:alpine

# The limits are visible from inside the container (cgroup v2 host)
docker exec limited cat /sys/fs/cgroup/memory.max     # 268435456
docker exec limited cat /sys/fs/cgroup/cpu.max        # 50000 100000

# Live usage accounting, provided by cgroups
docker stats --no-stream limited

# If a container exceeds its memory limit, the kernel OOM killer stops it
docker inspect -f '{{.State.OOMKilled}}' limited      # false (for now)

docker rm -f limited

Key takeaways

• cgroups control what a process can use; namespaces control what it can see. Containers need both.
• A cgroup is a directory in /sys/fs/cgroup; limits are set by writing to files in it.
• Exceeding a memory limit triggers the OOM killer; exceeding a CPU limit causes throttling, not termination.
• cgroups v2 uses a single unified hierarchy and is the default on modern Linux distributions.
• docker stats and docker pause are built directly on cgroup accounting and the freezer.

🧪 Practice

1. (Beginner) Run a container with --memory=128m, then read /sys/fs/cgroup/memory.max inside it and convert the value to megabytes.
2. (Intermediate) Run docker run --rm --memory=64m python:3.12-slim python -c "x = bytearray(200*1024*1024)". Observe the exit code (echo $?) and explain what happened and why the code is 137.
3. Interview question: "A containerized service is slow but never crashes, and its CPU usage graph is flat at exactly the same value. What might be happening?" Hint: look at cpu.stat and the difference between a CPU limit and a memory limit.

Union File Systems

Why union file systems exist. If every container needed a full private copy of its image, starting 100 containers from a 500 MB image would consume 50 GB of disk and take a long time to copy. Instead, containers share the read-only image files and record only their own changes. A union file system makes several directories look like a single one, which is the mechanism behind this sharing.

Analogy. Imagine a stack of transparent sheets on an overhead projector. The bottom sheets hold the base drawing (image layers) and cannot be altered. A blank sheet on top (the container layer) is where you draw changes. Looking down, you see one combined picture. To "erase" something on a lower sheet, you paint over it on the top sheet.

How it works

• Lower layers (read-only): image layers, shared between all containers using the image.
• Upper layer (read-write): one per container; holds all of the container's changes.
• Merged view: what the container sees as its root filesystem.
• Copy-on-write (CoW): when a container modifies a file from a lower layer, the file is first copied up into the upper layer and then modified. The original is untouched.
• Whiteouts: deleting a lower-layer file creates a special marker in the upper layer that hides it. The file still exists in the lower layer (which is why deleting files in a later Dockerfile step does not shrink an image).

          Container view (merged)
   /etc/nginx/nginx.conf  (modified)
   /usr/sbin/nginx
   /app/cache.tmp         (new)
   /etc/motd              (deleted -> hidden)
                ^
                |  union mount (OverlayFS)
   +------------+---------------------------------------------+
   | upperdir   | nginx.conf (copied up + edited), cache.tmp, |  read-write,
   |            | whiteout for /etc/motd                      |  per container
   +------------+---------------------------------------------+
   | lowerdir 2 | nginx binary + default config               |  read-only,
   +------------+---------------------------------------------+  shared image
   | lowerdir 1 | base OS files (incl. /etc/motd)             |  layers
   +------------+---------------------------------------------+

OverlayFS. Docker's default storage driver, overlay2, uses the kernel's OverlayFS. Other union implementations (AUFS, and storage drivers like devicemapper or btrfs that achieve similar results with block-level or filesystem snapshots) are legacy or niche today.

Example: building an overlay mount by hand

# Run as root (sudo -i) for simplicity
mkdir -p /tmp/ovl/{lower,upper,work,merged}
echo "original content" > /tmp/ovl/lower/a.txt
echo "to be deleted"   > /tmp/ovl/lower/b.txt

# Combine lower (read-only) and upper (read-write) into merged
# workdir is scratch space OverlayFS needs on the same filesystem as upperdir
mount -t overlay overlay \
  -o lowerdir=/tmp/ovl/lower,upperdir=/tmp/ovl/upper,workdir=/tmp/ovl/work \
  /tmp/ovl/merged

ls /tmp/ovl/merged                      # a.txt  b.txt

echo "changed" > /tmp/ovl/merged/a.txt  # triggers copy-up into upper
rm /tmp/ovl/merged/b.txt                # creates a whiteout in upper
echo "new" > /tmp/ovl/merged/c.txt      # new file goes straight to upper

cat /tmp/ovl/lower/a.txt                # "original content" -> lower untouched
ls -l /tmp/ovl/upper
# -rw-r--r-- ... a.txt                  copied-up and modified
# c--------- ... b.txt                  whiteout: character device 0,0
# -rw-r--r-- ... c.txt                  new file

umount /tmp/ovl/merged

Example: seeing layers in Docker

# Layers that make up an image
docker image history nginx:alpine

# With the overlay2 storage driver, the actual directories are listed here
docker run -d --name ovl-demo nginx:alpine
docker inspect -f '{{json .GraphDriver.Data}}' ovl-demo
# {"LowerDir": "...", "MergedDir": "...", "UpperDir": "...", "WorkDir": "..."}

# Show what the container changed relative to its image (A=added, C=changed, D=deleted)
docker exec ovl-demo sh -c 'echo hi > /tmp/x && rm /etc/motd'
docker diff ovl-demo
docker rm -f ovl-demo

When Docker uses the containerd image store (the default on recent installations), GraphDriver details may differ, but the layering and copy-on-write principles are the same.

Key takeaways

• A union filesystem stacks read-only image layers with one writable container layer into a single view.
• Copy-on-write means a file is copied into the writable layer only when it is first modified.
• Deleting a lower-layer file creates a whiteout; the data still exists in the image layer.
• Layer sharing makes containers fast to start and cheap on disk; overlay2 (OverlayFS) is Docker's default.
• Data written to the container layer disappears with the container; use volumes for persistence.

🧪 Practice

1. (Beginner) Run a container, create and delete files inside it, and use docker diff to list the changes. Map each change type (A, C, D) to an OverlayFS operation.
2. (Intermediate) Write a Dockerfile that downloads a 100 MB file in one RUN step and deletes it in the next. Build it and check its size with docker image ls. Then combine both steps into a single RUN and compare. Explain the difference using whiteouts.
3. Interview question: "Why can write-heavy workloads such as databases perform poorly when writing to the container filesystem?" Hint: consider what happens the first time a large existing file is modified, and what bypasses the union filesystem entirely.

chroot and Process Isolation

Why chroot matters. chroot is the oldest ancestor of containers. It answers a single question: "Which directory should this process treat as /?" Understanding what it does, and especially what it does not do, shows why containers need namespaces and cgroups on top of it.

What chroot does. The chroot() system call changes the root directory for a process and its children. Paths like /bin/sh or /etc/passwd are resolved relative to the new root, so the process cannot name files outside it.

What chroot does not do

• It does not hide other processes: ps (with /proc mounted) shows every process on the host.
• It does not isolate the network, hostname, users, or IPC.
• It does not limit resources.
• It is not a security boundary: a root process can escape a chroot (for example by creating a second chroot and walking up with ..).

pivot_root. Container runtimes use pivot_root() rather than chroot(). Inside a new mount namespace, pivot_root swaps the root mount itself and lets the old root be unmounted completely, so there is no path back to the host filesystem. This closes the classic chroot escape.

Feature	chroot	Container (runc)
Own root filesystem	Yes	Yes (pivot_root in a mount namespace)
Process isolation	No	Yes (PID namespace)
Network isolation	No	Yes (network namespace)
Hostname isolation	No	Yes (UTS namespace)
Resource limits	No	Yes (cgroups)
Privilege reduction	No	Yes (capabilities, seccomp, LSMs)
Escapable by root	Yes	Not by design (relies on kernel isolation)

Example: chroot alone vs chroot plus namespaces

# Extract a real root filesystem from the alpine image
mkdir rootfs
docker export "$(docker create alpine)" | tar -C rootfs -xf -

# 1) Plain chroot: different filesystem, but NO process isolation
sudo chroot rootfs /bin/sh
cat /etc/os-release           # Alpine Linux -> filesystem is isolated
mount -t proc proc /proc
ps | head                     # shows ALL host processes -> no PID isolation
umount /proc
exit

# 2) chroot inside new namespaces: a minimal hand-made "container"
sudo unshare --pid --fork --mount --uts --net --ipc \
  chroot rootfs /bin/sh -c 'mount -t proc proc /proc && hostname mini && exec /bin/sh'
ps                            # only sh and ps; sh is PID 1
hostname                      # mini
ip link                       # only the loopback interface
exit

Key takeaways

• chroot changes only the root directory; it provides filesystem isolation and nothing else.
• chroot is not a security boundary; root can escape it.
• Containers use pivot_root in a private mount namespace so the host filesystem is fully detached.
• A container is roughly "chroot + namespaces + cgroups + privilege restrictions".

🧪 Practice

1. (Beginner) Follow the example above to chroot into an Alpine root filesystem and confirm that cat /etc/os-release shows Alpine while uname -r still shows the host kernel.
2. (Intermediate) Extend the unshare command above to also limit memory by placing the process in a cgroup you create manually (see the cgroups topic). Document each step and which kernel feature it uses.
3. Interview question: "Why do container runtimes use pivot_root instead of chroot?" Hint: think about what remains reachable from the process after each call, and what a privileged process could do with it.

Linux Capabilities

Why capabilities exist. Traditionally, Linux had two kinds of processes: root (UID 0), which could do everything, and everyone else, which was checked against permissions. A program that needed one privileged action, such as binding to port 80, had to run as full root, gaining the power to load kernel modules, change the clock, or read any file. Capabilities break root's power into separate units so a process can receive only the privileges it needs.

Analogy. Instead of handing someone the building's master key, you give them only the key to the one room they need.

Common capabilities

Capability	Allows	In Docker's default set
CAP_CHOWN	Change file ownership	Yes
CAP_NET_BIND_SERVICE	Bind to ports below 1024	Yes
CAP_KILL	Send signals to any process	Yes
CAP_SETUID / CAP_SETGID	Change process user/group IDs	Yes
CAP_NET_RAW	Use raw and packet sockets	Yes
CAP_NET_ADMIN	Configure interfaces, routes, firewall rules	No
CAP_SYS_ADMIN	Mount filesystems and many other admin operations ("the new root")	No
CAP_SYS_MODULE	Load and unload kernel modules	No
CAP_SYS_TIME	Change the system clock	No
CAP_SYS_PTRACE	Trace and inspect arbitrary processes	No

Docker's default set contains 14 capabilities: CHOWN, DAC_OVERRIDE, FOWNER, FSETID, KILL, SETGID, SETUID, SETPCAP, NET_BIND_SERVICE, NET_RAW, SYS_CHROOT, MKNOD, AUDIT_WRITE, and SETFCAP. Everything else (about 25 more) is dropped, which is why "root" inside a container is much weaker than root on the host.

Capability sets. Each process has several sets; the most important are:

• Effective: capabilities currently checked by the kernel.
• Permitted: the upper limit of what the process may make effective.
• Bounding: the ceiling for anything the process or its children can ever acquire.

Example: inspecting capabilities

# Capabilities of a container's root process (hex bitmask)
docker run --rm alpine grep Cap /proc/self/status
# CapPrm: 00000000a80425fb
# CapEff: 00000000a80425fb
# CapBnd: 00000000a80425fb

# Decode the bitmask into names (capsh comes from libcap)
capsh --decode=00000000a80425fb
# 0x00000000a80425fb=cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,...

Example: adding and dropping capabilities

# NET_ADMIN is not in the default set, so changing an interface fails
docker run --rm alpine ip link set lo down
# ip: SIOCSIFFLAGS: Operation not permitted

# Grant only that one capability
docker run --rm --cap-add NET_ADMIN alpine ip link set lo down   # succeeds

# Even root cannot chown without CAP_CHOWN
docker run --rm --cap-drop CHOWN alpine chown nobody /tmp
# chown: /tmp: Operation not permitted

# Least privilege: drop everything, add back only what the app needs
#   NET_BIND_SERVICE          -> listen on port 80
#   CHOWN, SETUID, SETGID     -> nginx prepares its dirs and drops to user "nginx"
# (a starting point: confirm the exact set in Practice exercise 2)
docker run -d --name web \
  --cap-drop ALL \
  --cap-add NET_BIND_SERVICE \
  --cap-add CHOWN --cap-add SETUID --cap-add SETGID \
  -p 8080:80 nginx:alpine
docker rm -f web

Example: file capabilities outside containers

# Allow a non-root binary to bind to port 80 without running as root
sudo setcap cap_net_bind_service=+ep ./my-server
getcap ./my-server     # ./my-server cap_net_bind_service=ep

Key takeaways

• Capabilities split root's privileges into roughly 40 independent units.
• Docker grants a reduced default set of 14; dangerous ones like SYS_ADMIN and SYS_MODULE are dropped.
• Use --cap-drop ALL plus targeted --cap-add to follow least privilege.
• --privileged grants all capabilities (and more) and removes most isolation; avoid it.
• Inspect capabilities with /proc/self/status and decode them with capsh --decode.

🧪 Practice

1. (Beginner) Compare grep CapEff /proc/self/status inside a normal container and inside one started with --privileged. Decode both values with capsh --decode.
2. (Intermediate) Find the minimal set of capabilities needed to run nginx:alpine successfully by starting from --cap-drop ALL and adding capabilities one at a time while reading the container logs.
3. Interview question: "A container runs as root. Is that the same as running as root on the host?" Hint: consider capabilities, the bounding set, seccomp, and what user namespaces change about UID 0.

<a id="13-container-standards-and-ecosystem"></a>

1.3 Container Standards and Ecosystem

This section explains the open standards that make images and runtimes interchangeable, and maps the tools that implement them.

Open Container Initiative (OCI)

Why a standard was needed. By 2014, Docker's image format and runtime were the de facto standard, but they were defined only by Docker's implementation. CoreOS released rkt with a competing format (appc), and the ecosystem risked fragmenting: images built with one tool might not run on another, and orchestrators would have to support every format. Users wanted to build an image once and run it anywhere, with any compliant tool.

What the OCI is. The Open Container Initiative is a Linux Foundation project founded in June 2015 by Docker, CoreOS, Google, Microsoft, Red Hat, and others. It maintains open specifications for container formats and runtimes. Docker donated its image format and its low-level runtime, runc, which became the reference implementation.

Analogy. OCI is to containers what USB is to peripherals: any device that follows the specification works with any port that follows it, regardless of manufacturer.

The three specifications

Specification	Defines	Who uses it
Image spec	How an image is structured: manifest, config, layers, index	Builders (BuildKit, Buildah, Kaniko) and registries
Runtime spec	How to run a container from an unpacked "bundle" (config.json + root filesystem)	Low-level runtimes (runc, crun, youki, gVisor, Kata)
Distribution spec	HTTP API for pushing and pulling content to and from registries	Registries (Docker Hub, GHCR, Harbor, ECR) and clients

How they fit together

   Dockerfile / build tool
            |
            v
   +-----------------+   push / pull    +-------------------+
   |   OCI image     | <--------------> |   OCI registry    |
   | (image spec)    |  (distribution   | (Docker Hub, GHCR)|
   +--------+--------+      spec)       +-------------------+
            |
            | unpack layers + generate config.json
            v
   +-----------------+
   |  OCI bundle     |   rootfs/ + config.json
   +--------+--------+
            |
            v
   +-----------------+
   | OCI runtime     |   runc, crun, gVisor (runsc), Kata ...
   | (runtime spec)  |
   +--------+--------+
            v
     running container

Practical consequence. An image built with docker build can be pushed to any OCI registry and run by Podman, containerd, CRI-O, or Kubernetes. Registries increasingly also store non-image OCI artifacts such as Helm charts, signatures, and SBOMs.

Key takeaways

• OCI is a vendor-neutral standard for container images, runtimes, and distribution.
• It was created in 2015 to prevent fragmentation; Docker donated its format and runc.
• Three specs: image (what an image is), runtime (how to run one), distribution (how to move one).
• OCI compliance is why images built with Docker run on Podman, containerd, CRI-O, and Kubernetes.

🧪 Practice

1. (Beginner) For each OCI specification, name one tool that produces or implements it and one that consumes it.
2. (Intermediate) Build an image with Docker, push it to a local registry (docker run -d -p 5000:5000 registry:2), then pull and run it with Podman or nerdctl. Note which OCI spec each step relies on.
3. Interview question: "Kubernetes no longer uses Docker as its runtime. Why do images built with docker build still work?" Hint: identify which specification governs the image format and which component actually runs the container.

OCI Image Specification

Why it matters. When you docker pull nginx, the client must know which files to download, how they stack, what the default command is, and which variant matches your CPU. The OCI image specification defines this as a small set of JSON documents and compressed tar archives, all addressed by their cryptographic hash.

Building blocks

Component	Purpose
Image index	Optional entry point listing manifests for multiple platforms (e.g. linux/amd64, linux/arm64). Also called a manifest list.
Image manifest	Lists the config blob and the ordered layer blobs for one platform.
Image config	Runtime defaults (Env, Cmd, Entrypoint, WorkingDir, User, exposed ports), platform, layer diff_ids, and history.
Layers	Tar archives (usually gzip- or zstd-compressed) of filesystem changes, including whiteouts.
Descriptors	References of the form {mediaType, digest, size} that point from one object to another.

Content addressing. Every object is identified by a digest such as sha256:4b7ce07..., the SHA-256 hash of its bytes. This gives two guarantees: content can be verified after download, and identical layers are stored and transferred only once. Pulling by digest (nginx@sha256:...) always returns exactly the same image, whereas a tag (nginx:1.27) is a movable pointer.

 nginx:1.27  (tag -> points to)
      |
      v
 +------------------+
 |  Image index     |  sha256:aaa...
 +------------------+
   | linux/amd64            | linux/arm64
   v                        v
 +------------------+   +------------------+
 | Manifest (amd64) |   | Manifest (arm64) |
 +------------------+   +------------------+
   |        |
   |        +-------------------------+
   v                                  v
 +-------------+   +---------+---------+---------+
 | Config JSON |   | Layer 1 | Layer 2 | Layer 3 |   tar.gz blobs
 +-------------+   +---------+---------+---------+

Example: an image manifest

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {                                       // descriptor of the config blob
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:9c7a54a9a43c...",
    "size": 1472
  },
  "layers": [                                       // ordered: base first
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:43c4264eed91...",           // hash of the COMPRESSED tar
      "size": 3623807
    }
  ],
  "annotations": {                                  // optional key/value metadata
    "org.opencontainers.image.source": "https://github.com/example/app"
  }
}

Example: an image config (abridged)

{
  "architecture": "amd64",
  "os": "linux",
  "config": {                                       // defaults used by "run"
    "Env": ["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],
    "Cmd": ["/bin/sh"],
    "WorkingDir": "/"
  },
  "rootfs": {
    "type": "layers",
    "diff_ids": ["sha256:63ca1fbb43ae..."]           // hash of the UNCOMPRESSED tar
  },
  "history": [
    { "created": "2024-09-06T22:20:07Z", "created_by": "ADD alpine-minirootfs.tar.gz / # buildkit" }
  ]
}

Example: exploring a real image

# Raw index/manifest from the registry
docker buildx imagetools inspect --raw alpine:3.20 | jq .

# Copy an image into an OCI layout directory on disk (skopeo)
skopeo copy docker://docker.io/library/alpine:3.20 oci:alpine-oci:3.20

tree alpine-oci
# alpine-oci/
# ├── blobs
# │   └── sha256
# │       ├── 1a2b...   (manifest)
# │       ├── 3c4d...   (config)
# │       └── 5e6f...   (layer tar.gz)
# ├── index.json        (entry point; maps the tag "3.20" to a manifest)
# └── oci-layout        (marker file with the layout version)

# Verify content addressing: the file name IS its sha256
cd alpine-oci/blobs/sha256 && sha256sum * | head -3

Key takeaways

• An OCI image is an index (optional), a manifest, a config, and a set of layer blobs.
• All objects are content-addressed by SHA-256 digest, enabling verification and deduplication.
• Layer digest refers to the compressed blob; diff_id in the config refers to the uncompressed content.
• Tags are mutable pointers; digests are immutable identifiers.
• The image index makes multi-architecture images possible under a single tag.

🧪 Practice

1. (Beginner) Run docker buildx imagetools inspect nginx:alpine and list the platforms available in its index.
2. (Intermediate) Use skopeo copy to export an image into an OCI layout, open index.json, and follow the digests by hand to the manifest, the config, and each layer. Extract one layer with tar -tzf and list its contents.
3. Interview question: "Why does an image config store diff_ids when the manifest already lists layer digests?" Hint: consider what changes if the same layer is recompressed with a different algorithm or compression level.

OCI Runtime Specification

Why it matters. Once an image is pulled and unpacked, something has to turn files and JSON into a running, isolated process. The OCI runtime specification defines exactly what that input looks like and how the runtime must behave, so any compliant runtime can be swapped in.

The bundle. The input to a runtime is a bundle: a directory containing

• config.json: what to run and how to isolate it (process, user, environment, mounts, namespaces, cgroup limits, capabilities, seccomp, hooks).
• rootfs/: the root filesystem the process will see.

Higher-level tools (containerd, CRI-O, Podman) generate the bundle from an image and the user's options; the low-level runtime (runc) only executes it.

Lifecycle and states

   runc create            runc start            process exits
  ------------> created ------------> running ---------------> stopped
  (namespaces,            (user process          |                |
   cgroups, mounts         is exec'd)            | runc kill      | runc delete
   set up; process                               +--------------->|------------> (gone)
   waits)                                                        resources freed

Operation	Effect
create	Sets up namespaces, cgroups, mounts, and the root filesystem; the process is prepared but the user program has not started
start	Executes the user program defined in process.args
state	Reports status (creating, created, running, stopped), PID, and bundle path
kill	Sends a signal to the container process
delete	Releases resources once the container is stopped

Hooks. The spec defines lifecycle hooks (createRuntime, createContainer, startContainer, poststart, poststop) where external programs can run, for example to configure networking before the process starts.

Example: an abridged config.json

{
  "ociVersion": "1.0.2",
  "process": {
    "terminal": true,
    "user": { "uid": 0, "gid": 0 },
    "args": ["sh"],                                   // the program to run
    "env": ["PATH=/usr/sbin:/usr/bin:/sbin:/bin", "TERM=xterm"],
    "cwd": "/",
    "capabilities": {                                 // allowed privileges
      "bounding":  ["CAP_KILL", "CAP_NET_BIND_SERVICE", "CAP_AUDIT_WRITE"],
      "effective": ["CAP_KILL", "CAP_NET_BIND_SERVICE", "CAP_AUDIT_WRITE"]
    },
    "noNewPrivileges": true
  },
  "root": { "path": "rootfs", "readonly": true },     // relative to the bundle
  "hostname": "runc",
  "mounts": [
    { "destination": "/proc", "type": "proc", "source": "proc" }
  ],
  "linux": {
    "namespaces": [                                   // which namespaces to create
      { "type": "pid" }, { "type": "network" }, { "type": "ipc" },
      { "type": "uts" }, { "type": "mount" }, { "type": "cgroup" }
    ],
    "resources": {                                    // cgroup limits
      "memory": { "limit": 268435456 },
      "pids":   { "limit": 100 }
    }
  }
}

Example: running a container with runc directly (no Docker involved at runtime)

# 1. Prepare a bundle
mkdir -p mycontainer/rootfs && cd mycontainer
docker export "$(docker create busybox)" | tar -C rootfs -xf -   # just to get a rootfs

# 2. Generate a default config.json
runc spec

# 3a. Run interactively (create + start in one step)
sudo runc run demo            # drops you into "sh" inside the container
# / # ps    -> sh is PID 1
# / # exit

# 3b. Walk through the lifecycle explicitly
#     (create/start need a non-terminal process)
jq '.process.terminal=false | .process.args=["sleep","300"]' config.json > c.json && mv c.json config.json
sudo runc create demo2
sudo runc list                 # STATUS: created
sudo runc start demo2
sudo runc list                 # STATUS: running
sudo runc kill demo2 KILL
sudo runc list                 # STATUS: stopped
sudo runc delete demo2

Key takeaways

• The runtime spec defines a bundle (config.json + rootfs/) and the operations create, start, state, kill, and delete.
• config.json encodes every isolation setting: namespaces, cgroups, capabilities, mounts, seccomp.
• Separating create from start allows setup (for example networking via hooks) before the program runs.
• runc is the reference implementation; any compliant runtime can execute the same bundle.

🧪 Practice

1. (Beginner) Generate a config.json with runc spec and identify the sections that configure namespaces, mounts, and capabilities.
2. (Intermediate) Modify the generated config.json to set the hostname to oci-lab, add a 64 MB memory limit, and make the root filesystem read-only. Run it with runc run and verify each setting from inside the container.
3. Interview question: "Why does the OCI runtime spec separate create and start instead of providing a single run operation?" Hint: think about what higher-level tools need to configure after the namespaces exist but before the application starts.

Container Runtimes Overview

Why there are so many runtimes. "Container runtime" is used for very different things, from the tiny program that calls clone() to the daemon that pulls images and manages hundreds of containers. Splitting responsibilities into layers lets each part be replaced independently: a more secure low-level runtime, a lighter high-level runtime, or a different orchestrator.

Low-level vs high-level runtimes

Layer	Responsibilities	Examples
Low-level (OCI) runtime	Takes an OCI bundle and creates the isolated process (namespaces, cgroups, capabilities, seccomp)	runc, crun, youki, gVisor (runsc), Kata Containers
High-level runtime / container manager	Pulls and stores images, creates bundles, manages container lifecycle, snapshots, networking hooks, exposes an API	containerd, CRI-O, Podman (libpod)
Engine / user-facing tool	CLI, build, volumes, networks, Compose, developer experience	Docker Engine, Podman CLI, nerdctl
Orchestrator	Schedules containers across many hosts	Kubernetes, Docker Swarm, Nomad

Low-level runtime variants

Runtime	Language	Isolation model	Notes
runc	Go	Namespaces + cgroups	Reference implementation, default for Docker and containerd
crun	C	Namespaces + cgroups	Faster startup, lower memory; default in Podman on many distros
youki	Rust	Namespaces + cgroups	Memory-safe alternative
gVisor (runsc)	Go	User-space kernel intercepts syscalls	Stronger isolation, some performance and compatibility cost
Kata Containers	Go/Rust	Lightweight VM per container or pod	VM-level isolation with a container interface

The shim. Between the high-level runtime and runc sits a small shim process (containerd-shim-runc-v2). runc exits once the container is started; the shim stays as the container's parent, keeps its stdio open, and reports its exit status. This allows containerd or dockerd to restart or upgrade without killing running containers.

The Docker stack

  docker CLI                 (client: parses commands, calls the REST API)
      |  REST API over /var/run/docker.sock
      v
  dockerd                    (Docker Engine: build, volumes, networks, API)
      |  gRPC
      v
  containerd                 (images, snapshots, container lifecycle)
      |
      v
  containerd-shim-runc-v2    (one per container; parent of the container process)
      |
      v
  runc                       (creates namespaces/cgroups, execs process, then exits)
      |
      v
  container process (PID 1 inside the container)

The Kubernetes stack

  kubelet --CRI (gRPC)--> containerd (CRI plugin)  --> shim --> runc / gVisor / Kata
                     \--> CRI-O                    --> conmon --> runc / crun

Example: observing the layers on a Docker host

docker run -d --name stack-demo nginx:alpine

# The process tree shows the shim as the container's parent
ps -ef --forest | grep -A2 containerd-shim
# root  ... /usr/bin/containerd-shim-runc-v2 -namespace moby -id 5f1c... -address ...
# root  ...  \_ nginx: master process nginx -g daemon off;
# 101   ...      \_ nginx: worker process

# Docker's containers are visible to containerd in the "moby" namespace
sudo ctr -n moby containers list

# And to runc, which tracks state under Docker's runtime root
sudo runc --root /run/docker/runtime-runc/moby list

docker rm -f stack-demo

Example: selecting an alternative low-level runtime in Docker

# Assuming gVisor (runsc) is installed and registered in /etc/docker/daemon.json
docker run --rm --runtime=runsc alpine dmesg   # shows gVisor's user-space kernel messages

Key takeaways

• Low-level runtimes (runc, crun, gVisor, Kata) execute OCI bundles; high-level runtimes (containerd, CRI-O) manage images and lifecycle.
• Docker Engine sits on top of containerd, which uses a shim and runc for each container.
• The shim decouples running containers from the daemon so daemons can restart safely.
• Kubernetes talks to high-level runtimes through the CRI; Docker Engine is not required.
• Sandboxed runtimes (gVisor, Kata) trade some performance for stronger isolation.

🧪 Practice

1. (Beginner) Start a container and use ps -ef --forest to identify the shim process and the container's main process on the host.
2. (Intermediate) List the same running container through three interfaces: docker ps, ctr -n moby containers list, and runc list with Docker's runtime root. Explain what each layer knows about the container.
3. Interview question: "What happens to running containers when the Docker daemon is restarted, and which component makes that possible?" Hint: look at which process is the parent of the container process and the live-restore daemon option.

Docker Alternatives (Podman, containerd, CRI-O)

Why alternatives exist. Docker's daemon-based design is convenient for developers but has trade-offs: a long-running root daemon, a single point of failure, and a feature set larger than an orchestrator needs. Thanks to OCI standards, other tools can build, run, and manage the same images while making different design choices.

Podman. Developed by Red Hat, Podman is a daemonless container engine with a Docker-compatible CLI.

• Daemonless: each podman command starts containers as child processes (supervised by conmon), no central daemon.
• Rootless by default: runs containers as an ordinary user using user namespaces.
• Pods: groups of containers sharing namespaces, matching the Kubernetes pod concept.
• Kubernetes YAML: can generate and play Kubernetes manifests.
• systemd integration: Quadlet files run containers as systemd services.
• Companion tools: Buildah (building images) and Skopeo (copying and inspecting images between registries).

containerd. The high-level runtime extracted from Docker and now a CNCF graduated project. It is the runtime inside Docker Engine and the most common Kubernetes runtime.

• Designed to be embedded, with a gRPC API and plugins (snapshotters, CRI).
• ctr: a minimal low-level debugging CLI.
• nerdctl: a Docker-compatible CLI for containerd with Compose support and advanced features such as lazy pulling.

CRI-O. A lightweight runtime built only for Kubernetes.

• Implements the Kubernetes CRI and nothing else; versioned in lockstep with Kubernetes.
• No build or general-purpose developer CLI; debugged via crictl.
• Default runtime in OpenShift.

Feature	Docker Engine	Podman	containerd (+ nerdctl)	CRI-O
Architecture	Client + daemon	Daemonless	Daemon (embeddable)	Daemon (Kubernetes-only)
Rootless	Supported (opt-in)	Default	Supported	Via Kubernetes features
Docker-compatible CLI	Native	Yes	Yes (nerdctl)	No
Builds images	Yes (BuildKit)	Yes (Buildah)	Yes (nerdctl + BuildKit)	No
Compose	Yes	podman compose	nerdctl compose	No
Pods	No	Yes	Via Kubernetes	Via Kubernetes
Kubernetes CRI	No (via cri-dockerd)	No	Yes	Yes
Main audience	Developers, single hosts	Developers, RHEL/Fedora, rootless	Platforms, Kubernetes, embedding	Kubernetes / OpenShift

Example: Podman

# Same syntax as Docker; many users simply alias docker=podman
podman run -d --name web -p 8080:80 docker.io/library/nginx:alpine
podman ps

# Pods: containers that share network namespace (like Kubernetes)
podman pod create --name app -p 8081:80
podman run -d --pod app docker.io/library/nginx:alpine
podman run -d --pod app docker.io/library/redis:7     # reachable via localhost inside the pod

# Export the pod as Kubernetes YAML and recreate it from that file
podman kube generate app > app.yaml
podman pod rm -f app
podman kube play app.yaml

Example: containerd with ctr and nerdctl

# ctr: low-level, explicit image references, no port publishing shortcut
sudo ctr images pull docker.io/library/alpine:latest
sudo ctr run --rm -t docker.io/library/alpine:latest demo sh

# nerdctl: Docker-like experience on containerd
sudo nerdctl run -d --name web -p 8080:80 nginx:alpine
sudo nerdctl ps
sudo nerdctl compose up -d          # reads compose.yaml

Example: CRI-O and crictl on a Kubernetes node

sudo crictl pods            # pod sandboxes managed by the kubelet
sudo crictl ps              # containers
sudo crictl images          # images pulled by the runtime
sudo crictl logs <container-id>

Key takeaways

• OCI standards let different tools build and run the same images interchangeably.
• Podman is daemonless and rootless by default, with a Docker-compatible CLI and pod support.
• containerd is the core runtime inside Docker and most Kubernetes clusters; nerdctl gives it a Docker-like CLI.
• CRI-O is a minimal runtime dedicated to Kubernetes.
• Choose by context: Docker for developer workflow, Podman for rootless and daemonless hosts, containerd or CRI-O for Kubernetes nodes.

🧪 Practice

1. (Beginner) Install Podman and run the same nginx:alpine image you used with Docker. Compare the output of podman ps and docker ps.
2. (Intermediate) Create a Podman pod with an application container and a Redis container, generate its Kubernetes YAML with podman kube generate, and explain how each container reaches the other.
3. Interview question: "When would you choose Podman over Docker on a production Linux server that is not part of a Kubernetes cluster?" Hint: compare the security implications of a root daemon with a daemonless, rootless model, and consider systemd integration.

---

<a id="2-docker-architecture-and-setup"></a>