12. Logging, Monitoring, and Debugging

Containers are short-lived and replaceable. By the time someone notices a problem, the container involved may already have been restarted, rescheduled, or removed. This chapter covers the three ways to see inside a running system: logs (what the application said), metrics (how many resources it used and how the engine behaved over time), and debugging techniques (how to find out why a container crashed, why a build failed, or why two services cannot talk). Together they turn "it doesn't work" into a specific cause you can fix.

<a id="121-logging"></a>

12.1 Logging

This section follows a log line from the application's stdout to its final destination: how Docker captures and stores it, how to keep it from filling the disk, how to ship it off the host, and how to write it so it can be searched later. It builds on Viewing Logs, which covers the docker logs command.

Logging Drivers

Why it matters. Docker captures the stdout and stderr of every container, but what happens to those lines next is a choice. The default, writing JSON files on the host, works well on a laptop. In production you may want the logs in the systemd journal, in a cloud logging service, or in a central log pipeline instead. A logging driver is the component inside the daemon that receives each line from a container and decides where it goes. Changing the driver changes the destination without touching the application.

Analogy. Think of a company mail room. The application only drops letters into its outbox (stdout). The logging driver is the courier contract: file the letters in a cabinet downstairs (json-file, local), hand them to the building's central mail system (journald, syslog), or ship them to a remote warehouse (fluentd, awslogs, splunk). The person writing the letters never needs to know which courier is used.

How a log line travels

 container                 host
 +-----------+  stdout   +-----------------+  FIFO   +--------------------------------+
 | app PID 1 | --------> | containerd-shim | ------> | dockerd: log copier            |
 |           | --------> | (holds the      |         |   splits the stream into lines |
 +-----------+  stderr   |  pipes)         |         |   and hands each one to the    |
                         +-----------------+         |   container's logging driver   |
                                                     +--------------------------------+
                                                         |              |
                       json-file / local / journald      |              |  syslog / fluentd / gelf /
                       (store locally, docker logs       |              |  awslogs / splunk / ...
                        reads them back)                 v              v  (ship elsewhere)
                                                     local storage   remote system
                                                                     + dual-logging cache
                                                                       for docker logs

Built-in drivers

Driver	Destination	docker logs	Typical use
json-file (default)	One JSON-lines file per container under /var/lib/docker/containers/<id>/	Yes	Development; hosts where an agent tails the files
local	Compressed, protobuf-based files with rotation enabled by default	Yes	Recommended for local storage on single hosts
journald	systemd journal, with container metadata as journal fields	Yes	Hosts that already use the journal for all logs
syslog	Local or remote syslog (UDP, TCP, TLS)	Through the cache	Existing syslog infrastructure
fluentd	Fluentd or Fluent Bit (forward protocol)	Through the cache	Log pipelines built on Fluentd or Fluent Bit
gelf	Graylog Extended Log Format endpoint (Graylog, Logstash)	Through the cache	Graylog, ELK stacks
awslogs, gcplogs	Amazon CloudWatch Logs, Google Cloud Logging	Through the cache	Workloads running on that cloud
splunk	Splunk HTTP Event Collector	Through the cache	Splunk
etwlogs	Event Tracing for Windows	No	Windows containers
none	Discarded	No	Containers whose output is useless or must not be stored

Third-party drivers can be installed as plugins (for example grafana/loki-docker-driver for Grafana Loki) and are used the same way.

Dual logging. Drivers that send logs elsewhere cannot read them back. Since Engine 20.10, the daemon therefore also writes every line to a local, rotated cache (5 files of 20 MB by default), so docker logs keeps working whatever driver you use. You can tune the cache with the cache-disabled, cache-max-size, cache-max-file, and cache-compress log options.

Delivery mode: blocking or non-blocking. What happens when the destination is slow?

Mode	Behavior	Risk
blocking (default)	Each write to stdout waits until the driver has accepted the line	A slow or unreachable log endpoint stalls the application
non-blocking	Lines go into an in-memory ring buffer (max-buffer-size, default 1 MB) and are sent in the background	When the buffer is full, lines are dropped

For local drivers (json-file, local) blocking is rarely a problem. For network drivers, choose deliberately: losing some log lines is usually better than freezing the service.

Three places to configure a driver

Scope	How	Applies to
Daemon default	log-driver and log-opts in daemon.json	Containers created after the daemon restarts
One container	docker run --log-driver ... --log-opt key=value	That container
Compose service	logging: with driver: and options:	That service's containers

A container's logging configuration is fixed when the container is created. Changing the daemon default does not affect existing containers; you must recreate them.

Example: inspecting and choosing drivers

# Daemon-wide default driver
docker info --format '{{.LoggingDriver}}'
# json-file

# Driver and options of a specific container
docker run -d --name web nginx:1.27-alpine
docker inspect -f '{{json .HostConfig.LogConfig}}' web
# {"Type":"json-file","Config":{}}

# What json-file actually stores: one JSON object per line
sudo tail -n 1 "$(docker inspect -f '{{.LogPath}}' web)"
# {"log":"2026/10/14 10:02:11 [notice] 1#1: start worker process 31\n","stream":"stderr","time":"2026-10-14T10:02:11.482913Z"}

# journald, tagging every entry with the container name
docker run -d --name web-j --log-driver journald --log-opt tag="{{.Name}}" nginx:1.27-alpine
journalctl CONTAINER_NAME=web-j -n 5 --no-pager    # query the journal by container name
docker logs web-j                                   # still works: journald supports reading

# Ship to Fluent Bit without ever stalling the app:
#   fluentd-async  -> start even if the collector is down, reconnect in the background
#   non-blocking   -> buffer up to 4 MB, drop lines if the buffer fills
docker run -d --name api \
  --log-driver fluentd \
  --log-opt fluentd-address=localhost:24224 \
  --log-opt fluentd-async=true \
  --log-opt mode=non-blocking \
  --log-opt max-buffer-size=4m \
  example/api:1.4.0

# Discard the output entirely
docker run --rm --log-driver none alpine:3.20 echo "nobody will see this"

Example: Compose and the daemon default

# compose.yaml
services:
  api:
    image: example/api:1.4.0
    logging:
      driver: local
      options:
        max-size: "20m"
        max-file: "5"

{
  "log-driver": "local",
  "log-opts": {
    "max-size": "20m",
    "max-file": "5"
  }
}

Tags and metadata. Network drivers and journald accept a tag option built from a template: {{.Name}}, {{.ID}} (12 characters), {{.FullID}}, {{.ImageName}}, {{.DaemonName}}. The labels, labels-regex, env, and env-regex options attach selected container labels or environment variables to each log entry, which lets the destination filter by service or environment without parsing the message.

Key takeaways

• A logging driver decides where a container's stdout/stderr lines go; the application only writes to the standard streams.
• json-file is the default; local is the better local choice because it compresses and rotates out of the box.
• Dual logging keeps docker logs working with remote drivers by writing a local cache.
• With network drivers, mode=non-blocking trades possible line loss for an application that never stalls on logging.
• Logging configuration is fixed at container creation; daemon defaults only apply to new containers.

🧪 Practice

1. (Beginner) Run one container with the default driver and one with --log-driver local. Generate some output in both, then compare docker inspect -f '{{.HostConfig.LogConfig}}' and the files each driver writes under /var/lib/docker/containers/<id>/.
2. (Intermediate) Run a container with --log-driver fluentd --log-opt fluentd-address=localhost:24224 while nothing listens on that port. Observe the error, then repeat with fluentd-async=true and explain the difference.
3. Interview question: "After switching all containers to a remote logging driver, one service's latency spiked whenever the log backend was under maintenance. Why, and how would you prevent it?" Hint: think about what the application's write() to stdout waits for in each delivery mode, and what you give up by not waiting.

Log Rotation

Why it matters. With the default json-file driver, Docker appends every line to the container's log file and never deletes anything until the container is removed. A service that logs 1 MB per second writes about 86 GB per day. When /var/lib/docker fills up, the damage is not limited to that container: images cannot be pulled, databases fail to write, and the daemon itself can misbehave. Rotation caps how much log data each container may keep.

Analogy. Rotation is a notebook with a fixed number of pages. When the current page is full, you start a new one, and when the notebook runs out of pages, you tear out the oldest one. You always have the most recent history, never an unbounded pile of paper.

How it works. When the active file reaches max-size, Docker renames it and starts a new one. When there are more than max-file files, the oldest is deleted.

 --log-opt max-size=10m --log-opt max-file=3

 <id>-json.log       <- active file; grows up to 10 MB
 <id>-json.log.1     <- previous file
 <id>-json.log.2     <- oldest file; deleted at the next rotation

 Worst case per container: max-size x max-file = 30 MB
 (compress=true gzips the rotated files to save more space)

Defaults by driver

Driver	Default	Options
json-file	No rotation (max-size=-1, unlimited)	max-size, max-file (only effective with max-size), compress
local	20 MB x 5 files, compressed (about 100 MB per container)	max-size, max-file, compress
Dual-logging cache	20 MB x 5 files	cache-max-size, cache-max-file, cache-compress
journald	Docker does not rotate; journald does (SystemMaxUse= in journald.conf)	n/a

Capacity planning. The worst-case log footprint of a host is max-size x max-file x number of containers. Fifty containers at 10 MB x 3 files can use 1.5 GB. Size the limits so this total fits comfortably on the disk that holds /var/lib/docker.

Rotation is not retention. Rotated-out lines are gone. If you need logs for audits, investigations, or more than a few hours of history, ship them to a central system (see Centralized Logging) and keep local rotation small.

Common pitfalls

• Numbers must be strings in daemon.json. "max-file": 3 makes the daemon refuse to start; write "max-file": "3".
• Only new containers are affected. After changing the daemon default, recreate the containers (docker compose up -d --force-recreate).
• Do not rotate Docker's files with logrotate. The daemon keeps the files open and tracks their state; external rotation (especially copytruncate) can lose lines or break docker logs. Use Docker's own options.
• docker system df does not count log files. Check them directly.

Example: rotation in action

# 1 MB per file, at most 3 files
docker run -d --name rot \
  --log-opt max-size=1m --log-opt max-file=3 \
  alpine:3.20 sh -c 'yes "filler line to make the log grow quickly" | head -n 100000; sleep 3600'

sudo ls -lh "$(dirname "$(docker inspect -f '{{.LogPath}}' rot)")" | grep json.log
# -rw-r----- 1 root root 512K ... <id>-json.log
# -rw-r----- 1 root root 1.0M ... <id>-json.log.1
# -rw-r----- 1 root root 1.0M ... <id>-json.log.2      <- older lines are already gone

docker logs rot | wc -l     # far fewer than 100000: only what fits in about 3 MB
docker rm -f rot

Example: finding and handling oversized logs

# The five largest container logs on the host
sudo sh -c 'du -h /var/lib/docker/containers/*/*-json.log* | sort -h | tail -n 5'
# 14G  /var/lib/docker/containers/9c1e2f.../9c1e2f...-json.log

# Map the ID in the path to a container name
docker ps -a --no-trunc --format '{{.ID}} {{.Names}}' | grep 9c1e2f
# 9c1e2f...  legacy-worker

# Emergency only: empty the file in place to free space immediately.
# The history is lost; the real fix is to recreate the container with rotation.
sudo truncate -s 0 "$(docker inspect -f '{{.LogPath}}' legacy-worker)"

Example: one rotation policy for every Compose service

x-logging: &default-logging          # shared fragment, defined once
  driver: json-file
  options:
    max-size: "10m"
    max-file: "3"
    compress: "true"

services:
  api:
    image: example/api:1.4.0
    logging: *default-logging
  worker:
    image: example/worker:1.4.0
    logging: *default-logging

Key takeaways

• The default json-file driver never rotates; always set max-size and max-file, or use the local driver.
• Worst-case disk usage is max-size x max-file x containers; size it to fit /var/lib/docker.
• In daemon.json, log option values are strings, and changes apply only to newly created containers.
• Let Docker rotate its own files; do not point logrotate at them.
• Rotation discards history; ship logs to a central system for retention.

🧪 Practice

1. (Beginner) Add log-driver and log-opts with rotation to daemon.json, restart the daemon, and verify with docker inspect that a new container has the settings while an old one does not.
2. (Intermediate) Write a script that lists every container with its log driver, its max-size/max-file options, and the current on-disk size of its log files, and flags containers that have no rotation.
3. Interview question: "Your monitoring shows /var/lib/docker growing by 20 GB a day on one host, but docker system df shows nothing unusual. Where do you look and what do you change?" Hint: consider what docker system df does not account for, and the difference between a daemon-wide default and a setting on existing containers.

Centralized Logging

Why it matters. Local logs have three weaknesses. They disappear with the container (docker rm deletes them). They are spread across many hosts, so following one request through several services means logging in to several machines. And they are only kept as long as rotation allows. Centralized logging copies every line to one system where it is stored, indexed, searchable across all services, protected by access control, and kept for as long as policy requires.

Analogy. Each container keeps a diary on its desk. Centralized logging is a library that photocopies every diary every few seconds, indexes it by author and date, and keeps the copies even after the author leaves the company.

The pipeline

 collect             process / enrich           transport          store + index        use
 +-----------+      +--------------------+      +-----------+      +--------------+     +---------------+
 | container | ---> | parse JSON, add    | ---> | buffer,   | ---> | Loki,        | --> | search,       |
 | stdout    |      | host/service/env,  |      | batch,    |      | Elasticsearch|     | dashboards,   |
 |           |      | drop noise, mask   |      | retry     |      | OpenSearch,  |     | alerts        |
 +-----------+      | secrets            |      +-----------+      | CloudWatch...|     +---------------+
                    +--------------------+                         +--------------+

Two ways to collect from Docker

 Pattern A: the logging driver ships directly        Pattern B: a node agent tails the files

 +-----+ +-----+                                      +-----+ +-----+
 | app | | app |  --log-driver fluentd/gelf/awslogs   | app | | app |  default json-file driver
 +--+--+ +--+--+                                      +--+--+ +--+--+
    |       |                                            |       |
    v       v                                            v       v
 dockerd ----------> collector / cloud service        /var/lib/docker/containers/*/*-json.log
                                                              |
                                                       agent container (Fluent Bit, Vector,
                                                       Grafana Alloy, Filebeat) reads the files
                                                              |
                                                              v
                                                         log backend

	A: logging driver	B: node agent tailing files
Setup	Per container or daemon default	One agent per host
docker logs	Through the dual-logging cache	Native
Collector outage	Blocks the app or drops lines (depends on mode)	Files keep growing locally; agent catches up later
Enrichment	Limited to driver options (tag, labels, env)	Full: the agent can query the Docker API for metadata
Typical use	Cloud services (awslogs, gcplogs), simple setups	Most production setups; mirrors how Kubernetes collects logs

Pattern B needs the json-file (or journald) driver: the local driver's format is internal and not meant to be read by other tools.

Common backends and agents

Component	Examples	Notes
Agents / shippers	Fluent Bit, Fluentd, Vector, Grafana Alloy, Filebeat, OpenTelemetry Collector	Fluent Bit and Vector are lightweight; Alloy replaces the older Promtail
Full-text stores	Elasticsearch / OpenSearch (+ Kibana / Dashboards)	Indexes every word: powerful queries, higher cost
Label-indexed stores	Grafana Loki (+ Grafana)	Indexes only labels, scans content at query time: cheap, needs good labels
Managed services	CloudWatch Logs, Google Cloud Logging, Azure Monitor, Datadog, Splunk	No servers to run; pay per GB ingested and stored

Example: Fluent Bit, Loki, and Grafana with Compose

The application uses the fluentd driver to send its lines to Fluent Bit, which parses the JSON and forwards the records to Loki. Grafana queries Loki.

# compose.yaml
services:
  loki:
    image: grafana/loki:3.2.0
    command: -config.file=/etc/loki/local-config.yaml   # single-node config shipped in the image
    ports:
      - "127.0.0.1:3100:3100"

  fluent-bit:
    image: fluent/fluent-bit:3.1.9
    volumes:
      - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro
    ports:
      - "127.0.0.1:24224:24224"     # dockerd (on the host) connects here, not other containers
    depends_on: [loki]

  grafana:
    image: grafana/grafana:11.2.0
    ports:
      - "127.0.0.1:3000:3000"

  app:
    image: alpine:3.20
    # Prints one JSON line every 2 seconds; $$ is Compose's escape for a literal $
    command:
      - sh
      - -c
      - |
        i=0
        while true; do
          i=$$((i+1))
          echo "{\"level\":\"info\",\"msg\":\"tick\",\"n\":$$i}"
          sleep 2
        done
    logging:
      driver: fluentd
      options:
        fluentd-address: "localhost:24224"   # resolved by dockerd in the HOST network
        fluentd-async: "true"                # start even if Fluent Bit is not ready yet
        tag: "app.{{.Name}}"
    depends_on: [fluent-bit]

# fluent-bit.conf
[SERVICE]
    Flush        1
    Log_Level    info
    Parsers_File parsers.conf

# Receive records sent by Docker's fluentd logging driver
[INPUT]
    Name   forward
    Listen 0.0.0.0
    Port   24224

# The application's line arrives in the "log" field: parse it as JSON into real fields
[FILTER]
    Name         parser
    Match        app.*
    Key_Name     log
    Parser       json
    Reserve_Data On

# Send to Loki; container_name (added by the driver) becomes a Loki label
[OUTPUT]
    Name       loki
    Match      *
    Host       loki
    Port       3100
    Labels     job=docker
    Label_Keys $container_name

docker compose up -d

# Query Loki directly: the last records from the app container
curl -sG localhost:3100/loki/api/v1/query_range \
  --data-urlencode 'query={job="docker"}' --data-urlencode 'limit=3' \
  | jq -r '.data.result[].values[][1]'
# {"container_id":"3c1f...","source":"stdout","level":"info","msg":"tick","n":42,...}

# In Grafana (localhost:3000): add a Loki data source at http://loki:3100, then in Explore:
#   {job="docker"} | json | n > 40

Two details in this example cause most first-time failures:

• The daemon connects to the collector, not the container. fluentd-address is resolved by dockerd in the host's network namespace, so Compose service names do not work there; publish the collector's port on the host and use localhost.
• Without fluentd-async, a container cannot start while the collector is unreachable. The driver connects at container creation and fails if it cannot.

Key takeaways

• Centralize logs so they survive container removal, span all hosts, and can be searched, retained, and alerted on.
• Collect either with a logging driver that ships directly, or with a node agent that tails json-file files; the agent pattern is more resilient and more common.
• Loki indexes labels only and is cheap; Elasticsearch/OpenSearch index full text and cost more; managed services remove operations work.
• Network logging drivers are configured from the daemon's point of view: addresses are resolved on the host.
• Enrich logs with service, environment, and host metadata at collection time so queries do not depend on parsing messages.

🧪 Practice

1. (Beginner) Run the Compose stack above, open Grafana, add the Loki data source, and find all records whose n field is divisible by 10.
2. (Intermediate) Replace Pattern A with Pattern B: remove the logging: section from app, and configure Fluent Bit with a tail input that reads /var/lib/docker/containers/*/*-json.log (mounted read-only) and parses Docker's JSON format. Compare which metadata is available in each approach.
3. Interview question: "Design log collection for 200 Docker hosts running 3,000 containers, with 30 days of searchable retention and no lost logs during a backend outage." Hint: consider where buffering happens (memory, local disk), what happens to the application when the backend is down, label and index cardinality, and the cost difference between full-text and label-based indexing.

Structured Logging Practices

Why it matters. On one container, grep is enough. Across hundreds of containers, the questions become "every error from the payments service for request abc123 in the last hour" or "how many card declines per minute". With free-text logs, each question needs a fragile regular expression that breaks when a developer rewords a message. Structured logs record each event as fields (usually one JSON object per line), so every field can be filtered, counted, and graphed directly.

Analogy. A free-text log line is a handwritten letter; a structured log line is a filled-in form. Both carry the same information, but a machine can sort and count a thousand forms in a second, while letters have to be read one by one.

The same event, two ways

Unstructured:
2026-10-14 10:15:02 ERROR Payment failed for order 4711 (user 42) after 3 retries: card declined

Structured (one JSON object per line):
{"ts":"2026-10-14T10:15:02.114Z","level":"error","service":"payments","msg":"payment failed","order_id":4711,"user_id":42,"retries":3,"reason":"card_declined","trace_id":"4bf92f3577b34da6a3ce929d0e0e4736"}

The structured version lets you count reason values, find everything for order_id=4711, or jump from the log line to a distributed trace via trace_id.

Practices for container logs

Practice	Why
Write to stdout/stderr, one event per line	Docker splits the stream on newlines; a multi-line stack trace becomes many unrelated events
Use JSON with a stable schema	Fields are queryable; consistent names (level, msg, ts) work in shared dashboards
Keep the message constant and put variables in fields	"payment failed" can be counted; "payment failed for order 4711" cannot
Timestamps in UTC, ISO 8601, with milliseconds	Unambiguous ordering across hosts and time zones
Standard levels, chosen by an environment variable (LOG_LEVEL)	Change verbosity per environment without rebuilding the image
Include correlation fields: request_id, trace_id, service, version	Follow one request across services; tie logs to traces and releases
Put exceptions inside a field ("exc": "...")	The stack trace stays part of a single event
Never log secrets, tokens, or unmasked personal data	Logs are copied to many systems, widely readable, and kept for a long time
Disable output buffering	Buffered stdout delays lines and loses the last ones on a crash (Python buffers when stdout is not a terminal)
Let the platform add infrastructure metadata	Container name, image, and host come from the driver or agent, not from application code
Keep events below 16 KB	Docker splits longer lines into partial messages that not every destination reassembles
Log decisions and boundaries, not every loop iteration	Log volume costs storage and money, and noise hides the signal

Example: JSON logging in Python with the standard library

# app.py
import json
import logging
import os
import sys
from datetime import datetime, timezone


class JsonFormatter(logging.Formatter):
    def format(self, record: logging.LogRecord) -> str:
        entry = {
            "ts": datetime.fromtimestamp(record.created, tz=timezone.utc)
                          .isoformat(timespec="milliseconds"),
            "level": record.levelname.lower(),
            "logger": record.name,
            "msg": record.getMessage(),
            "service": os.getenv("SERVICE_NAME", "unknown"),
        }
        # Structured fields passed as logger.info("...", extra={"fields": {...}})
        entry.update(getattr(record, "fields", {}))
        if record.exc_info:
            # The whole stack trace stays inside one JSON event
            entry["exc"] = self.formatException(record.exc_info)
        return json.dumps(entry, ensure_ascii=False)


handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(JsonFormatter())
logging.basicConfig(level=os.getenv("LOG_LEVEL", "INFO").upper(), handlers=[handler])
log = logging.getLogger("payments")

log.info("payment received", extra={"fields": {"order_id": 4711, "amount_cents": 1999}})
log.debug("card network response", extra={"fields": {"raw_status": "05"}})  # hidden at INFO
try:
    raise RuntimeError("card declined")
except RuntimeError:
    log.exception("payment failed", extra={"fields": {"order_id": 4711, "reason": "card_declined"}})

FROM python:3.13-slim
# PYTHONUNBUFFERED: write each line immediately, even though stdout is not a terminal
ENV PYTHONUNBUFFERED=1 \
    LOG_LEVEL=info \
    SERVICE_NAME=payments
COPY app.py /app/app.py
CMD ["python", "/app/app.py"]

docker build -t pay-demo .
docker run --name pay pay-demo

# Query the logs like data
docker logs pay 2>&1 | jq -c 'select(.level == "error") | {ts, msg, order_id, reason}'
# {"ts":"2026-10-14T10:15:02.114+00:00","msg":"payment failed","order_id":4711,"reason":"card_declined"}

# Mixed output (a library printing plain text) breaks plain jq; skip non-JSON lines
docker logs pay 2>&1 | jq -cR 'fromjson? | select(.level == "error")'

# Turn on debug logs for one run, without rebuilding
docker run --rm -e LOG_LEVEL=debug pay-demo | jq -c 'select(.level == "debug")'

Libraries by language

Language	Common choices for JSON logs
Python	structlog, python-json-logger, or a formatter as above
Node.js	pino, winston
Java	Logback with logstash-logback-encoder, Log4j2 JsonTemplateLayout
Go	log/slog with slog.NewJSONHandler(os.Stdout, nil)
.NET	Built-in JSON console formatter, Serilog

Key takeaways

• Structured logs are one JSON object per line with consistent field names, so they can be queried rather than grepped.
• Keep messages constant and put variable data in fields; put stack traces inside a field.
• Use UTC ISO 8601 timestamps, standard levels set by environment variables, and correlation IDs.
• Disable output buffering and never log secrets or unmasked personal data.
• Leave container and host metadata to the logging driver or agent.

🧪 Practice

1. (Beginner) Run the Python example and use jq on docker logs to print only the msg and order_id of every event.
2. (Intermediate) Add a request_id to every log line of a small web service (generate one per request, or take it from an X-Request-ID header) and show that you can extract all lines for one request across two containers that call each other.
3. Interview question: "A team logs pretty-printed, multi-line JSON from their containers and complains that their log platform shows each event as dozens of fragments. What is going on, and what would you change?" Hint: consider how Docker splits the stdout stream into log entries, and where (if anywhere) multi-line reassembly could happen.

<a id="122-monitoring"></a>

12.2 Monitoring

Logs tell you what happened; metrics tell you how much, how often, and how it changes over time. This section explains which numbers a container exposes and where they come from, how cAdvisor collects them for every container, how Prometheus and Grafana store, graph, and alert on them, and how to monitor the Docker daemon itself.

Container Metrics

Why it matters. docker stats shows a snapshot of right now. Real operational questions are about time: "Has memory been growing all week?", "Was the API CPU-throttled at 3 a.m. when latency spiked?", "How often did containers get OOM-killed last month?" Answering them requires collecting the right numbers continuously and knowing exactly what each one means.

Analogy. Container metrics are a car's dashboard. The speedometer is a rate (CPU cores in use), the fuel gauge is a level against a limit (memory against its cap), the odometer only ever counts up (total bytes sent), and the warning lights signal events (OOM kills, throttling). You need all four kinds to understand how the car is being driven.

Logs, metrics, and traces

Signal	Answers	Shape	Example
Logs	What happened, in detail?	Discrete events with text and fields	payment failed, order_id=4711
Metrics	How much, how often, what trend?	Numbers sampled over time	container_memory_working_set_bytes = 210 MB
Traces	Where did the time go in one request?	Tree of timed spans across services	checkout 340 ms -> payments 290 ms -> db 250 ms

Metrics are cheap to store and fast to query, which makes them the basis of dashboards and alerts. Logs and traces are used to explain what a metric revealed.

Where container metrics come from. Docker does not measure anything itself; it reads the kernel's accounting:

 host kernel
 +--------------------------------------------------------------------------+
 | cgroup v2:  /sys/fs/cgroup/system.slice/docker-<id>.scope/               |
 |               cpu.stat        CPU time, throttling                       |
 |               memory.current  memory.stat  memory.events (oom_kill)      |
 |               io.stat         bytes and operations per block device      |
 |               pids.current    number of processes and threads            |
 | network namespace:  /proc/<pid>/net/dev   bytes, packets, drops, errors  |
 +--------------------------------------------------------------------------+
        ^                       ^                          ^
  Engine API stats         cAdvisor                  other agents
  (docker stats)           (Prometheus format)       (OpenTelemetry Collector, Datadog, ...)

Network traffic is not a cgroup resource; it is counted per network namespace. Container state (restart count, health status, exit codes) comes from the Docker Engine, not from the kernel.

The metrics that matter

Metric	Kernel source	What to watch for
CPU usage (cores)	cpu.stat: usage_usec	Sustained usage close to the --cpus limit
CPU throttling	cpu.stat: nr_throttled, nr_periods, throttled_usec	More than a few percent of periods throttled means added latency (see CPU Limits and Shares)
Memory working set	memory.current minus inactive file cache	Steady growth (leak) or values close to the limit
OOM kills	memory.events: oom_kill	Any increase (see OOM Behavior)
Block I/O	io.stat: rbytes, wbytes, rios, wios	One container saturating a shared disk
Network	/proc/net/dev in the container's namespace	Drops, errors, unexpected traffic volumes
Processes	pids.current against pids.max	Thread or process leaks, fork bombs
Restarts, health	Engine: RestartCount, State.Health	Crash loops, flapping health checks

Counters and gauges. Metrics come in two basic kinds, and confusing them produces nonsense graphs:

Kind	Behavior	Examples	How to use
Counter	Only increases (resets to 0 on restart)	CPU seconds used, bytes sent, OOM kills	Take the rate over time: CPU seconds per second = cores used
Gauge	Goes up and down	Memory in use, number of processes	Use the value directly; average or max over time

USE and RED. Two checklists help decide what to monitor. For resources (CPU, memory, disk, network), use USE: Utilization, Saturation (for example throttling, swapping), Errors. For services, use RED: Rate of requests, Errors, Duration. Container metrics cover USE; RED metrics must come from the application itself (an HTTP server exposing request counts and latency histograms), because the kernel cannot know how many requests failed.

Example: reading the raw numbers

docker run -d --name web --memory 256m --cpus 0.5 nginx:1.27-alpine

# cgroup v2 files, read from inside the container
docker exec web cat /sys/fs/cgroup/cpu.stat
# usage_usec 48211
# user_usec 21004
# system_usec 27207
# nr_periods 41
# nr_throttled 0          <- periods in which the CPU quota ran out
# throttled_usec 0        <- total time the container was paused by throttling

docker exec web cat /sys/fs/cgroup/memory.events
# low 0
# high 0
# max 0                   <- times usage hit memory.max
# oom 0
# oom_kill 0              <- processes killed by the OOM killer in this cgroup

# The same files from the host (systemd cgroup driver, cgroup v2)
CID=$(docker inspect -f '{{.Id}}' web)
cat /sys/fs/cgroup/system.slice/docker-$CID.scope/memory.current
# 3375104

# Engine API: one sample, the data docker stats is built on
curl -s --unix-socket /var/run/docker.sock \
  "http://localhost/containers/web/stats?stream=false" \
  | jq '{mem: .memory_stats.usage, limit: .memory_stats.limit,
         cpu_ns: .cpu_stats.cpu_usage.total_usage, pids: .pids_stats.current,
         rx_bytes: .networks.eth0.rx_bytes}'
# { "mem": 3375104, "limit": 268435456, "cpu_ns": 48211000, "pids": 9, "rx_bytes": 1306 }

How CPU percent is computed. The Engine API reports CPU usage as a cumulative counter in nanoseconds. A percentage needs two samples; the API returns the previous one as precpu_stats (with stream=false it waits about a second to take both; one-shot=true skips that and leaves precpu_stats empty):

cpu_delta    = cpu_stats.cpu_usage.total_usage - precpu_stats.cpu_usage.total_usage
system_delta = cpu_stats.system_cpu_usage      - precpu_stats.system_cpu_usage
CPU %        = cpu_delta / system_delta x online_cpus x 100      (100% = one full core)

Key takeaways

• Container metrics come from cgroups (CPU, memory, I/O, PIDs), from the network namespace (traffic), and from the Engine (restarts, health).
• CPU throttling and OOM kills are the two metrics most often missing from dashboards and most often behind unexplained latency or restarts.
• Counters must be turned into rates; gauges are read directly.
• Container metrics cover resource health (USE); service health (RED) needs metrics exposed by the application.

🧪 Practice

1. (Beginner) Start a container with --cpus 0.2 running sh -c 'while :; do :; done'. Read cpu.stat twice, ten seconds apart, and compute the fraction of periods that were throttled.
2. (Intermediate) Write a script that calls the Engine API stats endpoint for every running container and prints CPU %, memory used versus limit, and PID count, using the formula above. Compare its output with docker stats --no-stream.
3. Interview question: "A service's p99 latency doubles every afternoon, but its average CPU usage never exceeds 60% of its limit. What metric would you check first?" Hint: consider how a CPU quota is enforced within each 100 ms period, and what an average over a minute hides.

cAdvisor

Why it matters. The kernel keeps the numbers, but something has to read them for every container, every few seconds, attach the container's name and image, and publish them in a format a monitoring system can collect. cAdvisor (Container Advisor, an open-source project from Google) does exactly that. The same code is built into the Kubernetes kubelet, so the metric names you learn here are the ones used in Kubernetes as well.

Analogy. cAdvisor is a meter reader who walks past every apartment's electricity, water, and gas meters every few seconds and writes the readings on a public board. It does not keep years of readings; someone else (Prometheus) copies the board regularly and keeps the history.

How it works

• Runs as a container with read-only access to the host's /sys (cgroups), /var/lib/docker, and /var/run (Docker socket, used to look up names, images, and labels).
• Discovers every cgroup on the host and samples it on a fixed interval (--housekeeping_interval, 1 s by default).
• Keeps only a short history in memory (--storage_duration, 2 minutes by default): it is a collector, not a database.
• Exposes a web UI (/containers/), a REST API (/api/...), and Prometheus metrics (/metrics).

Example: running cAdvisor

docker run -d --name cadvisor \
  -p 127.0.0.1:8080:8080 \
  --volume /:/rootfs:ro \
  --volume /var/run:/var/run:ro \
  --volume /sys:/sys:ro \
  --volume /var/lib/docker/:/var/lib/docker:ro \
  --volume /dev/disk/:/dev/disk:ro \
  --privileged \
  --device /dev/kmsg \
  gcr.io/cadvisor/cadvisor:v0.49.1 \
  --docker_only=true \
  --housekeeping_interval=10s

# Web UI: http://localhost:8080/containers/
# Prometheus metrics for the "web" container:
curl -s localhost:8080/metrics | grep -E '^container_memory_working_set_bytes\{.*name="web"'
# container_memory_working_set_bytes{id="/system.slice/docker-3f2b....scope",image="nginx:1.27-alpine",name="web",...} 3.375104e+06 1728900000000

Mount or flag	Why cAdvisor needs it
/sys:ro	cgroup files with the actual numbers
/var/run:ro	Docker socket, to map cgroups to container names, images, and labels
/var/lib/docker:ro	Container configuration and filesystem usage per container
/:/rootfs:ro, /dev/disk:ro	Host filesystem and disk statistics
--privileged, /dev/kmsg	Read OOM events from the kernel log, access all cgroups

The metrics you will use most

Metric	Type	Meaning
container_cpu_usage_seconds_total	counter	CPU time used; rate() gives cores in use
container_cpu_cfs_throttled_periods_total	counter	Periods in which the container was throttled
container_cpu_cfs_periods_total	counter	Periods in which the container ran (denominator for the throttling ratio)
container_memory_working_set_bytes	gauge	Memory that cannot easily be reclaimed; what limit and OOM decisions track
container_memory_usage_bytes	gauge	Includes page cache; overstates memory pressure
container_spec_memory_limit_bytes	gauge	The configured limit (0 = unlimited)
container_network_receive_bytes_total, ..._transmit_bytes_total	counter	Network traffic
container_fs_reads_bytes_total, container_fs_writes_bytes_total	counter	Disk traffic
container_oom_events_total	counter	OOM events in the container
container_last_seen	gauge	Unix time the container was last observed

Every series carries labels: name (container name), image, id (cgroup path), and by default every Docker label as container_label_<label>. cAdvisor also reports non-container cgroups (system services), whose name label is empty, so container queries usually filter with name!="".

Running it in production

Concern	Recommendation
CPU overhead	Raise --housekeeping_interval (for example 10s to match a 15 s scrape interval)
Too many series	--docker_only=true; --disable_metrics=percpu,sched,tcp,udp,hugetlb,referenced_memory,cpu_topology,resctrl; --store_container_labels=false if labels are not needed
Security	It runs privileged and can read the Docker socket: never publish its port publicly; keep it on an internal network that only Prometheus can reach
Docker Desktop	It sees the Linux VM, so it shows containers but not your macOS or Windows host

Key takeaways

• cAdvisor reads cgroup statistics for every container, labels them with container metadata, and exposes them for Prometheus at /metrics.
• It keeps only minutes of history; pair it with a time-series database.
• Use container_memory_working_set_bytes (not usage_bytes) for memory alerts, and the throttled-periods ratio for CPU saturation.
• Reduce overhead and cardinality with --docker_only, a longer housekeeping interval, and --disable_metrics.
• It is privileged and sees the whole host; keep its endpoint private.

🧪 Practice

1. (Beginner) Run cAdvisor and an nginx container, open the web UI, and find the nginx container's memory and network graphs. Then find the same values in /metrics.
2. (Intermediate) Count the number of series cAdvisor exposes (curl -s localhost:8080/metrics | grep -vc '^#') with default flags, then with --docker_only=true, --disable_metrics=..., and --store_container_labels=false. Explain why the number matters for Prometheus.
3. Interview question: "Your memory dashboard, based on container_memory_usage_bytes, shows several containers at 98% of their limit, yet none has ever been OOM-killed. Is the dashboard wrong?" Hint: compare what usage_bytes and working_set_bytes include, and which kind of memory the kernel can reclaim before it kills anything.

Prometheus and Grafana Integration

Why it matters. cAdvisor shows the last two minutes. To see trends, compare today with last week, and get woken up only when something is truly wrong, you need a time-series database and an alerting engine. Prometheus collects metrics by pulling (scraping) HTTP endpoints at a fixed interval, stores them, evaluates alert rules, and answers queries in PromQL. Grafana draws dashboards on top of it. Together they are the de facto open-source monitoring stack for containers.

Analogy. cAdvisor is a thermometer in each room. Prometheus is a nurse who reads every thermometer every 15 seconds and writes the values in a chart; its alert rules say "call the doctor if the fever stays above 39 for ten minutes". Grafana is the chart pinned to the wall where everyone can see it.

Architecture

                    scrape: HTTP GET /metrics every 15 s
        +------------------------------------------------------------+
        |                     |                     |                |
        v                     v                     v                v
  +-----------+        +-------------+       +-------------+   +-----------+
  | cAdvisor  |        | dockerd     |       | node-       |   | your app  |
  | :8080     |        | :9323       |       | exporter    |   | /metrics  |
  | containers|        | the engine  |       | the host    |   | RED       |
  +-----------+        +-------------+       +-------------+   +-----------+
        ^
        |         +------------------------+   alerts   +--------------+
        +-------- | Prometheus             | ---------> | Alertmanager | --> email, Slack, pager
                  | TSDB + rule evaluation |            +--------------+
                  +------------------------+
                             ^  PromQL
                             |
                       +-----------+
                       |  Grafana  |  dashboards
                       +-----------+

Example: a monitoring stack with Compose

# compose.yaml
services:
  prometheus:
    image: prom/prometheus:v3.1.0
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.retention.time=15d          # how long samples are kept
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./alerts.yml:/etc/prometheus/alerts.yml:ro
      - prom-data:/prometheus
    extra_hosts:
      - "host.docker.internal:host-gateway"         # reach the daemon's metrics on the host
    ports:
      - "127.0.0.1:9090:9090"

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:v0.49.1
    command: ["--docker_only=true", "--housekeeping_interval=10s"]
    privileged: true
    devices: ["/dev/kmsg"]
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    # no published port: only Prometheus, on the same network, needs it

  grafana:
    image: grafana/grafana:11.2.0
    environment:
      GF_SECURITY_ADMIN_PASSWORD: change-me          # use a secret outside a lab
    volumes:
      - ./grafana/datasources.yml:/etc/grafana/provisioning/datasources/datasources.yml:ro
      - grafana-data:/var/lib/grafana
    ports:
      - "127.0.0.1:3000:3000"

volumes:
  prom-data:
  grafana-data:

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s          # how often alert rules are evaluated

rule_files:
  - /etc/prometheus/alerts.yml

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: cadvisor
    static_configs:
      - targets: ["cadvisor:8080"]   # Compose service name, resolved by Docker's embedded DNS

  - job_name: docker-daemon
    static_configs:
      - targets: ["host.docker.internal:9323"]   # see Daemon Metrics Endpoint

  - job_name: api
    static_configs:
      - targets: ["api:8000"]        # your application's own /metrics (RED); remove if you have none

# grafana/datasources.yml: provisioned, so the data source exists on first start
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true

Useful PromQL queries

Question	PromQL
CPU cores used per container	sum by (name) (rate(container_cpu_usage_seconds_total{name!=""}[5m]))
Memory as a fraction of the limit	container_memory_working_set_bytes{name!=""} / (container_spec_memory_limit_bytes{name!=""} > 0)
Fraction of CPU periods throttled	rate(container_cpu_cfs_throttled_periods_total{name!=""}[5m]) / rate(container_cpu_cfs_periods_total{name!=""}[5m])
Network receive rate (bytes/s)	sum by (name) (rate(container_network_receive_bytes_total{name!=""}[5m]))
Top 5 memory users	topk(5, container_memory_working_set_bytes{name!=""})

The > 0 filter in the memory query removes containers without a limit (limit 0), which would otherwise divide by zero.

Example: alert rules

# alerts.yml
groups:
  - name: containers
    rules:
      - alert: ContainerMemoryNearLimit
        expr: |
          container_memory_working_set_bytes{name!=""}
            / (container_spec_memory_limit_bytes{name!=""} > 0) > 0.9
        for: 5m                       # must be true for 5 minutes before firing
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.name }} uses over 90% of its memory limit"

      - alert: ContainerCpuThrottled
        expr: |
          rate(container_cpu_cfs_throttled_periods_total{name!=""}[5m])
            / rate(container_cpu_cfs_periods_total{name!=""}[5m]) > 0.25
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.name }} is throttled in over 25% of CPU periods"

      - alert: ApiContainerMissing
        expr: absent(container_last_seen{name=~".+-api-[0-9]+"})   # no api container reported at all
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "No api container is running"

Prometheus evaluates the rules and sends firing alerts to Alertmanager, which groups, deduplicates, silences, and routes them to email, chat, or paging systems.

Discovering containers automatically. Static target lists do not scale when containers come and go. Prometheus can ask the Docker API which containers exist and scrape those that opt in with a label:

  - job_name: docker-containers
    docker_sd_configs:
      - host: unix:///var/run/docker.sock        # Prometheus needs the socket: root-equivalent access
    relabel_configs:
      # Only scrape containers labeled prometheus.scrape=true
      - source_labels: [__meta_docker_container_label_prometheus_scrape]
        regex: "true"
        action: keep
      # Use the container name (without the leading "/") as a label
      - source_labels: [__meta_docker_container_name]
        regex: "/(.*)"
        target_label: container

Dashboards. Grafana's dashboard library on grafana.com has ready-made dashboards for cAdvisor, node-exporter, and the Docker daemon that you can import by ID. For anything you maintain, store dashboards as JSON and provision them from files like the data source above, so they are versioned with the stack.

Cardinality. Each unique combination of metric name and label values is a separate time series. Container IDs change on every redeploy, so each deployment creates new series. Avoid labels with unbounded values (user IDs, request IDs, full URLs) in application metrics, and drop unneeded cAdvisor labels; series count is what drives Prometheus memory use.

Key takeaways

• Prometheus scrapes /metrics endpoints (cAdvisor, the daemon, node-exporter, applications), stores time series, and evaluates alert rules; Grafana visualizes them.
• Use rate() on counters, filter with name!="", and guard divisions by zero with > 0.
• Alert on sustained conditions (for:), on memory near the limit, on throttling, and on missing containers.
• docker_sd_configs discovers containers dynamically but requires access to the Docker socket.
• Keep label cardinality bounded; it determines Prometheus resource usage.

🧪 Practice

1. (Beginner) Start the stack, open Prometheus at localhost:9090/targets, and confirm the cadvisor target is up. Run the "CPU cores used per container" query and display it as a graph.
2. (Intermediate) Run a container with --cpus 0.2 that burns CPU, and another with --memory 64m whose memory slowly grows. Build a Grafana dashboard with throttling and memory-versus-limit panels, and verify that both alert rules fire.
3. Interview question: "Why does Prometheus pull metrics instead of having containers push them, and what problems does the pull model create for short-lived containers such as batch jobs?" Hint: think about how Prometheus knows a target is down, what happens to a job that finishes between two scrapes, and what the Pushgateway is for.

Daemon Metrics Endpoint

Why it matters. cAdvisor measures containers; it says nothing about the engine that runs them. When docker run becomes slow under load, health checks fail across many containers at once, or the daemon's own memory keeps growing, you need metrics about dockerd itself. The daemon can expose them in Prometheus format: container counts by state, how long container operations take, failed health checks, failed builds, and Go runtime statistics.

Analogy. cAdvisor reads each apartment's meters. The daemon metrics endpoint is the building management system's own panel: how long the elevator takes, how many doors are open, how often the alarm system reported a fault.

Enabling the endpoint. Add metrics-addr to daemon.json and restart the daemon (older Engine versions also required "experimental": true):

{
  "metrics-addr": "127.0.0.1:9323"
}

sudo systemctl restart docker

curl -s localhost:9323/metrics | grep -E '^engine_daemon_(container_states|health_checks_failed)'
# engine_daemon_container_states_containers{state="paused"} 0
# engine_daemon_container_states_containers{state="running"} 7
# engine_daemon_container_states_containers{state="stopped"} 3
# engine_daemon_health_checks_failed_total 2

On Docker Desktop, add the same key under Settings > Docker Engine; containers reach it at host.docker.internal:9323.

What it exposes

Metric	Type	Use
engine_daemon_container_states_containers{state}	gauge	Running, paused, and stopped container counts
engine_daemon_container_actions_seconds{action}	histogram	Duration of create, start, delete, ... operations
engine_daemon_network_actions_seconds{action}	histogram	Duration of network operations
engine_daemon_health_checks_total, engine_daemon_health_checks_failed_total	counter	Health checks run and failed, across all containers
engine_daemon_events_total	counter	Events emitted by the daemon
engine_daemon_engine_info{version, kernel, os, ...}	gauge (always 1)	Engine version and host details as labels; useful for fleet inventories
builder_builds_failed_total{reason}	counter	Failed builds by reason
go_goroutines, process_resident_memory_bytes, process_open_fds	gauge	Daemon resource usage; steady growth suggests a leak

Exposure and reachability. The endpoint has no authentication or TLS.

Address	Reachable from	Notes
127.0.0.1:9323	The host only	Safest; Prometheus must run on the host or with network_mode: host
172.17.0.1:9323 (the docker0 IP)	The host and containers (via host.docker.internal:host-gateway)	Common for a containerized Prometheus on the same host
0.0.0.0:9323	Every network the host is on	Only with a firewall restricting access to your Prometheus servers

Example: queries on daemon metrics

# p95 duration of container operations, by action
histogram_quantile(0.95,
  sum by (le, action) (rate(engine_daemon_container_actions_seconds_bucket[5m])))

# Failed health checks per second across the host
rate(engine_daemon_health_checks_failed_total[5m])

# Daemon goroutine count growing steadily over a day (possible leak)
deriv(go_goroutines{job="docker-daemon"}[6h]) > 0

Other engine components. containerd can expose its own metrics ([metrics] address = "127.0.0.1:1338" in /etc/containerd/config.toml), and nodes in Swarm mode add swarm_* metrics to the daemon endpoint.

Choosing a metric source

Source	Scope	History	Best for
docker stats	Per container, live	None	Quick interactive checks
Engine API /containers/<id>/stats	Per container, live	None	Scripts and custom tools
cAdvisor	Every container on the host	Minutes (in memory)	Feeding Prometheus with container metrics
Daemon metrics-addr	The engine itself	None (scraped)	Engine health, operation latency
node-exporter	The host (CPU, disks, filesystems, network)	None (scraped)	Host capacity and hardware health
Application /metrics	Requests, errors, latency, business numbers	None (scraped)	RED metrics and SLOs

Key takeaways

• metrics-addr in daemon.json makes dockerd expose Prometheus metrics about itself, not about container resource usage.
• Watch container state counts, operation latency histograms, failed health checks, and the daemon's goroutines and memory.
• The endpoint is unauthenticated: bind it to loopback or the bridge IP, and firewall it if it must be wider.
• A complete picture combines daemon metrics, cAdvisor, node-exporter, and application metrics.

🧪 Practice

1. (Beginner) Enable metrics-addr on 127.0.0.1:9323, start and stop a few containers, and watch engine_daemon_container_states_containers change.
2. (Intermediate) Add the daemon as a Prometheus target in the Compose stack from the previous topic (bind the endpoint to the docker0 IP), then graph the p95 duration of start actions while a loop starts and removes 50 containers.
3. Interview question: "Users report that docker run sometimes takes 20 seconds on a busy CI host, while the containers themselves start quickly once running. Which metrics would you collect to locate the delay?" Hint: consider the daemon's per-action latency histograms, host disk metrics during image extraction and layer creation, and daemon resource usage.

<a id="123-debugging-and-troubleshooting"></a>

12.3 Debugging and Troubleshooting

When something fails, guessing is slow and often makes things worse. This section gives a repeatable method for each kind of failure: containers that crash or never start, images too minimal to contain a shell, cryptic exit codes, failing builds, traffic that never arrives, and problems in the Docker daemon itself.

Debugging Crashed Containers

Why it matters. A crashed container is hard to examine: its process is gone, so docker exec fails, and with a restart policy it may be restarting in a loop that buries the first error under repeated output. The goal is to collect the evidence (state, logs, events, filesystem) before it changes, and then reproduce the failure where you can watch it.

Analogy. Treat it like an accident investigation. Secure the scene (stop the restart loop, do not remove the container), read the flight recorder (docker inspect state and logs), examine the wreckage (the container's filesystem), and finally reconstruct the event step by step (run the same image interactively).

A repeatable workflow

 1. What state is it in?          docker ps -a                       Exited (N)? Restarting? Created?
            |
 2. Why did it stop?              docker inspect -f '{{json .State}}' ExitCode, OOMKilled, Error, Health
            |
 3. What did it say last?         docker logs --tail 50              the lines just before the crash
            |
 4. What happened around it?      docker events --since ...          die, oom, kill, health_status
            |
 5. What is on its filesystem?    docker diff / docker cp /          missing config, unexpected files
                                  docker commit + shell
            |
 6. Reproduce interactively       docker run -it --entrypoint sh     run the startup command by hand

Example: a full investigation

# A service that crashes on startup because its configuration file is missing
docker run -d --name api --restart unless-stopped \
  -e CONFIG_PATH=/etc/api/config.yaml \
  alpine:3.20 sh -c 'test -f "$CONFIG_PATH" || { echo "config not found: $CONFIG_PATH" >&2; exit 3; }; sleep infinity'

# 1. State: a restart loop; the number in parentheses is the last exit code
docker ps -a --filter name=api --format '{{.Names}}\t{{.Status}}'
# api   Restarting (3) 4 seconds ago
docker inspect -f 'restarts={{.RestartCount}}' api
# restarts=6

# Secure the scene: stop the loop but keep the container, its logs, and its filesystem
docker update --restart=no api
docker stop api

# 2. Why it stopped
docker inspect -f 'exit={{.State.ExitCode}} oom={{.State.OOMKilled}} err={{.State.Error}}' api
# exit=3 oom=false err=

# 3. The last output (repeated once per restart)
docker logs --tail 3 api
# config not found: /etc/api/config.yaml
# config not found: /etc/api/config.yaml
# config not found: /etc/api/config.yaml

# 4. The timeline of deaths and their exit codes
docker events --since 10m --until "$(date +%s)" \
  --filter container=api --filter event=die \
  --format '{{.Time}} {{.Action}} exitCode={{.Actor.Attributes.exitCode}}'
# 1728900311 die exitCode=3
# ...

# 5. The filesystem: docker diff and docker cp work on stopped containers
docker diff api                                   # files changed relative to the image
docker cp api:/etc/. ./api-etc && ls ./api-etc    # no "api" directory: the config was never mounted

# Or snapshot it as an image and explore with a shell
docker commit api api-crashed
docker run --rm -it --entrypoint sh api-crashed
# / # ls /etc/api
# ls: /etc/api: No such file or directory

# 6. Reproduce with the same environment, step by step
docker run --rm -it -e CONFIG_PATH=/etc/api/config.yaml --entrypoint sh alpine:3.20

docker commit captures the container's filesystem but not the contents of its volumes; inspect volumes by mounting them into another container.

Keeping a failing container alive. When a container dies too quickly to examine, start it with the same options but a harmless command, then exec into it and run the real startup command by hand:

# Same image, mounts, network, and environment; the entrypoint is replaced with sleep
docker run -d --name api-debug --entrypoint sleep \
  -e CONFIG_PATH=/etc/api/config.yaml \
  example/api:1.4.0 infinity
docker exec -it api-debug sh

# With Compose: run a one-off container from the service's full configuration
docker compose run --rm --entrypoint sh api

Symptoms and first checks

Symptom	First check
Status Created, never started	.State.Error: missing bind-mount source, port already allocated, invalid device; try docker start -a <name>
Exits immediately with code 0	The main process finished: a one-shot command, or a daemon that forked into the background (service nginx start) instead of running in the foreground
Restarting (N)	The restart policy hides the crash; docker update --restart=no, then read logs
Exited (137) with OOMKilled=true	Memory limit reached; see OOM Behavior
Running but (unhealthy)	docker inspect -f '{{json .State.Health}}' <name> | jq '.Log[-1]' shows the last probe's output
Works with docker run -it, exits with -d	The app expects a terminal or open stdin; check whether it needs -t or -i
Crash only in one environment	Compare docker inspect of both (.Config.Env, .Mounts, .HostConfig) with diff

Key takeaways

• Follow a fixed order: state, exit details, logs, events, filesystem, interactive reproduction.
• Stop restart loops with docker update --restart=no instead of removing the container; removal destroys logs and filesystem evidence.
• docker logs, docker cp, docker diff, and docker commit all work on stopped containers; docker exec does not.
• Override the entrypoint (--entrypoint sh or sleep infinity) to get inside an image whose normal command crashes.

🧪 Practice

1. (Beginner) Run docker run -d --name oneshot alpine:3.20 echo done. Explain its status and exit code, and why adding --restart always makes it show as restarting forever.
2. (Intermediate) Build an image whose CMD crashes because a required environment variable is missing. Use only the six-step workflow to identify the cause, then fix it with docker compose run and a corrected environment.
3. Interview question: "A production container restarted 40 times overnight; by morning it is running normally and the logs show nothing unusual. How would you find out what happened?" Hint: think about where restart history, exit codes, and OOM information are recorded (inspect, events, kernel log, centralized logs and metrics), and which of them survive restarts.

docker debug

Why it matters. Well-built production images are minimal: distroless and scratch images (see Distroless and Scratch Images) have no shell, no ls, no curl. That reduces the attack surface, but it also means docker exec -it app sh fails. Adding debugging tools to the image would undo the benefit. The solution is to bring the tools to the container from outside, only while you are debugging.

Analogy. A mechanic does not keep a full toolbox in the trunk of every car. They roll a toolbox trolley up to whichever car needs work and take it away afterwards.

What docker debug does. docker debug (a Docker Desktop CLI command) opens a shell that shares the target's namespaces and filesystem, with its own toolbox of common utilities (vim, nano, htop, curl, and more) on the PATH. Nothing is added to the image or the container's own filesystem.

                        docker debug api
                               |
                               v
 +--------------------------------------------------------------+
 | target container's namespaces (PID, network, mount, ...)     |
 |                                                              |
 |   app process (PID 1)          debug shell                   |
 |   /  (image filesystem)  <---  sees and edits the same files |
 |                                tools from the toolbox:       |
 |                                /nix/...  (not in the image)  |
 +--------------------------------------------------------------+

Target	Behavior
Running container	Shell inside its namespaces; changes to files are real and visible to the app
Stopped container	Shell on a copy of its filesystem; changes are discarded at exit
Image	Shell on the image's filesystem in a sandbox; changes are discarded at exit

Built-in commands inside the debug shell

Command	Purpose
install <package>	Add a tool from the Nix package collection to your toolbox (kept for future sessions, never added to the image)
uninstall <package>	Remove it from the toolbox
entrypoint	Print, lint, and test the image's ENTRYPOINT and CMD (catches shell-form and path mistakes)
builtins	List the built-in commands

docker debug ships with Docker Desktop and is not part of the open-source Docker Engine on Linux servers; depending on the Desktop version, it may require signing in with a paid subscription.

Example: debugging a distroless container

# A distroless Python image: there is no shell in it
docker run -d --name py gcr.io/distroless/python3-debian12 -m http.server 8000

docker exec -it py sh
# OCI runtime exec failed: exec failed: unable to start container process:
# exec: "sh": executable file not found in $PATH: unknown

docker debug py
# Inside the debug shell:
#   ps aux                    the container's processes: python is PID 1
#   ls /usr/lib/python3*      the container's own filesystem
#   curl -sI localhost:8000   same network namespace as the app
#   install jq                add a tool for this and future sessions
#   entrypoint                show and lint the image's ENTRYPOINT/CMD
#   exit

# Debug the image itself, without running its entrypoint
docker debug gcr.io/distroless/python3-debian12

Without Docker Desktop: a sidecar debug container. On a Linux server, attach a tool image to the target's namespaces with standard docker run flags:

# Share the target's network and PID namespaces; SYS_PTRACE allows strace and gdb to attach
docker run --rm -it \
  --network container:py \
  --pid container:py \
  --cap-add SYS_PTRACE \
  nicolaka/netshoot:v0.13

# Inside the sidecar:
ps aux                 # python from the target is visible
ls /proc/1/root/       # the target's root filesystem, through its PID 1
ss -tln                # sockets of the shared network namespace
strace -f -p 1         # system calls made by the application

From the host, nsenter runs host binaries inside a container's namespaces: sudo nsenter -t "$(docker inspect -f '{{.State.Pid}}' py)" -n ss -tln.

Choosing a debugging approach

Approach	Needs a shell in the image	Works on stopped containers and images	Available	Notes
docker exec	Yes	No	Everywhere	Simplest when the image has tools
docker debug	No	Yes (changes discarded)	Docker Desktop	Toolbox, package install, entrypoint linter
Sidecar (--pid/--network container:)	No	No	Everywhere	Any tool image; filesystem via /proc/1/root
nsenter on the host	No	No	Linux hosts, root	Uses the host's binaries
:debug image variants	n/a	n/a	Distroless publishes :debug tags with BusyBox	Swap the image temporarily; changes what you run

Key takeaways

• Minimal images have no shell; bring tools from outside instead of adding them to the image.
• docker debug gives a toolbox shell for running containers, stopped containers, and images, without modifying them; it is a Docker Desktop feature.
• On any Linux host, a sidecar with --pid container: and --network container: gives the same visibility using any tool image.
• The target's filesystem is reachable through /proc/<pid>/root from a container that shares its PID namespace.

🧪 Practice

1. (Beginner) Run a distroless container, confirm that docker exec ... sh fails, then inspect its processes and listening ports with the netshoot sidecar.
2. (Intermediate) Using the sidecar approach, read a configuration file from inside a distroless container through /proc/1/root, and trace the system calls of its main process with strace while sending it a request. If you have Docker Desktop, repeat with docker debug.
3. Interview question: "Your security team bans shells in production images, and an incident needs live investigation inside a running container. What are your options and their risks?" Hint: compare sidecars, nsenter, and docker debug in terms of the privileges they require, what they can modify, and how you would audit their use.

Exit Codes

Why it matters. The exit code is the first and cheapest clue about a container's death: a single number from 0 to 255 that tells you whether the application reported a failure, the kernel killed it with a signal, or Docker could not start it at all. Container States lists the common values; this topic explains how they are produced, how to decode any of them, and how to tell apart causes that share the same code.

Analogy. An exit code is like the reason code on a returned parcel: "refused by recipient" (the application decided), "address not found" (the command does not exist), or "destroyed in transit" (a signal from outside). The code does not tell the whole story, but it tells you where to look next.

How codes are produced

Range	Set by	Meaning
0	Application	Success
1–124	Application	Application-defined failure (1 is generic; 2 often means bad usage or arguments)
125	Docker CLI / daemon	The container could not be run: invalid flag, daemon error, unknown runtime
126	Runtime	The command exists but cannot be executed (no execute permission, a directory, wrong format)
127	Runtime / shell	The command was not found
128 + N	Kernel (signal N)	The process was terminated by signal N
255	Application	Often exit(-1); values wrap modulo 256, so exit 256 reports 0

When a process is killed by a signal, it does not choose an exit code. The kernel reports "terminated by signal N", and Docker (like shells) represents it as 128 + N.

Signal-related codes

Code	Signal	Typical cause in containers
130	SIGINT (2)	Ctrl+C in docker run -it
134	SIGABRT (6)	abort(): failed assertion, heap corruption detected by libc, runtime fatal error
137	SIGKILL (9)	OOM killer, docker kill, or the docker stop timeout expired
139	SIGSEGV (11)	Segmentation fault in native code; binaries built for another libc (glibc vs musl)
141	SIGPIPE (13)	Wrote to a pipe or socket whose reader went away
143	SIGTERM (15)	docker stop and the process used the default action (did not handle the signal)

143 is not necessarily a bug: the JVM, for example, runs its shutdown hooks on SIGTERM and then exits with 143. A 137 after exactly the stop timeout (10 s by default), on the other hand, usually means PID 1 ignored SIGTERM (see Stopping and Killing Containers).

Example: producing and decoding codes

docker run --rm alpine:3.20 sh -c 'exit 3';                     echo $?   # 3   application-defined
docker run --rm alpine:3.20 nosuchcommand;                       echo $?   # 127 command not found
docker run --rm alpine:3.20 /etc/hosts;                          echo $?   # 126 not executable
docker run --rm --no-such-flag alpine:3.20 true;                 echo $?   # 125 Docker itself failed
docker run --rm --memory 32m alpine:3.20 tail /dev/zero;         echo $?   # 137 OOM kill
docker run --rm --init alpine:3.20 sh -c 'kill -SEGV $$';        echo $?   # 139 SIGSEGV
# --init is needed in the last line: as PID 1, sh would ignore the signal it sends itself

# Decode any exit code into a signal name
code=$(docker inspect -f '{{.State.ExitCode}}' api)
if [ "$code" -gt 128 ]; then
  echo "killed by SIG$(kill -l $((code - 128)))"    # 137 -> SIGKILL
else
  echo "exited with code $code"
fi

Telling the causes of 137 apart

Evidence	OOM kill	docker stop timeout	docker kill or external SIGKILL
.State.OOMKilled	true	false	false
docker events	oom, then die	kill (signal 15), kill (signal 9), die, stop	kill (signal 9), die
Last log lines	Stop abruptly	Shutdown messages, then silence for the timeout	Stop abruptly
Kernel log (dmesg, journalctl -k)	Memory cgroup out of memory: Killed process ...	Nothing	Nothing

# Watch the event stream while reproducing a 137
docker events --filter container=api --format '{{.Action}} {{index .Actor.Attributes "signal"}}'
# kill 15
# kill 9
# die
# stop

Exit codes in scripts and CI. docker run (without -d) returns the container's exit code, and docker wait <name> blocks until a container stops and prints its code. With Compose, docker compose run returns the service's code, and docker compose up --abort-on-container-exit --exit-code-from tests returns the code of the tests service. In shell pipelines, the code of docker run ... | tee log.txt is tee's; use set -o pipefail so failures are not hidden.

Key takeaways

• 125, 126, and 127 mean the container never ran properly: a Docker error, a non-executable command, or a missing command.
• Codes above 128 are 128 + signal number; decode them with kill -l $((code - 128)).
• 137 has three causes (OOM, stop timeout, explicit kill); OOMKilled, docker events, and the kernel log tell them apart.
• 143 is a normal result for applications that let SIGTERM terminate them; 137 after exactly the stop timeout points to signal handling problems.
• In CI, propagate exit codes deliberately (--exit-code-from, docker wait, pipefail).

🧪 Practice

1. (Beginner) Reproduce exit codes 0, 1, 126, 127, and 137, and print the name and exit code of all five containers with one docker ps -a --format command (run them without --rm).
2. (Intermediate) Write a script that, for every exited container, prints its name, exit code, the decoded signal name (if any), and the value of OOMKilled, and labels each one as "app error", "never started", "OOM", or "killed by signal".
3. Interview question: "A Java service always exits with code 143 when deployed with a new version, and the on-call engineer considers this a crash. Is it?" Hint: consider which signal produces 143, how the JVM reacts to it, and what code a container would report if the shutdown had actually hung.

Debugging Builds

Why it matters. Build failures differ from runtime failures. The cause is often outside the Dockerfile: a file missing from the build context, a .dockerignore rule, a stale cache, the wrong platform, or a network restriction. BuildKit makes this harder in two ways: its default output collapses each step's log, and, unlike the legacy builder, it does not leave a container behind for the failed step that you could run and inspect.

Analogy. A build is an assembly line. To find a defect, you turn on full recording at every station (--progress=plain), stop the line right after a given station and examine the half-finished product (--target), or force every station to redo its work instead of reusing stock from the warehouse (--no-cache).

The toolkit

Need	Tool
Full output of every step	--progress=plain (or BUILDKIT_PROGRESS=plain)
Force steps to run again	--no-cache, or --no-cache-filter <stage> for one stage
Inspect an intermediate stage	--target <stage>, then docker run -it on the result
See what is in the build context	A throwaway Dockerfile that lists the context
Catch Dockerfile mistakes before building	docker build --check . (build checks)
A shell where a step failed	docker buildx debug (experimental)
Logs and timing of past builds	docker buildx history (recent Buildx), Docker Desktop Builds view
What ended up in each layer	docker history, dive

Example: see everything, then rerun

# Full logs of every step, also saved to a file
docker build --progress=plain -t api:dbg . 2>&1 | tee build.log

# Temporary diagnostics in the Dockerfile only show up if the step actually runs:
#   RUN ls -la /app && cat /app/package.json
# A cached step prints nothing, so combine with --no-cache while debugging
docker build --progress=plain --no-cache -t api:dbg .

Example: stop before the failing step

FROM node:22-alpine AS deps
WORKDIR /app
COPY package*.json ./
RUN npm ci

FROM deps AS build
COPY . .
RUN npm run build            # fails here

FROM nginx:1.27-alpine
COPY --from=build /app/dist /usr/share/nginx/html

# Build only up to the last stage that works, and open a shell in it
docker build --target deps -t api:deps .
docker run --rm -it api:deps sh
# /app # ls node_modules/.bin      <- are the expected tools installed?

# To reproduce the failing RUN itself: comment it out, build --target build,
# then run the command by hand inside that image
docker build --target build -t api:build .
docker run --rm -it api:build sh -c 'npm run build'

Example: what is really in the build context?

# A throwaway Dockerfile from stdin, with the current directory as context
docker build --progress=plain --no-cache -f - . <<'EOF'
FROM busybox:1.36
COPY . /ctx
RUN du -sh /ctx && find /ctx -maxdepth 2 | sort | head -n 50
EOF
# #6 0.312 412.0M  /ctx               <- far too large: node_modules or .git not excluded

Example: build checks and the interactive debugger

# Lint the Dockerfile without building
docker build --check .
# WARNING: JSONArgsRecommended - https://docs.docker.com/go/dockerfile/rule/json-args-recommended/
# JSON arguments recommended for CMD to prevent unintended behavior related to OS signals (line 12)

# Make warnings fail the build: add this directive at the top of the Dockerfile
#   # check=error=true

# Experimental (flags may change between Buildx versions):
# open a shell in the state of the step that failed
BUILDX_EXPERIMENTAL=1 docker buildx debug --on=error build .

# Recent Buildx versions keep a history of builds, including failed ones
docker buildx history ls                 # recent builds with status and duration
docker buildx history logs <build-ref>   # full logs of one build

Common build failures

Message (abridged)	Likely cause	Fix
failed to calculate checksum of ref ...: "/config.yaml": not found	File outside the build context, excluded by .dockerignore, or path not relative to the context	Check the context directory and .dockerignore (see Build Context)
exec /bin/sh: exec format error in a RUN step	Base image for a different CPU architecture, no emulation installed	Pass --platform, or install QEMU binfmt handlers for cross-builds
Temporary failure resolving ... / failed to fetch	DNS, proxy, or firewall in the build network	Pass HTTP_PROXY/HTTPS_PROXY build args, fix the daemon's DNS settings
A change to the source is not picked up	The cache was reused because the step's inputs did not change from BuildKit's point of view (see Cache Invalidation)	--no-cache-filter <stage>, reorder COPY instructions
secret <id> not found or empty secret	--secret not passed to the build	docker build --secret id=<id>,src=<file> .
Build stays on "transferring context"	Huge context (node_modules, .git, data files)	Add them to .dockerignore
npm ci or pip install works locally but fails in the build	Lockfile not copied, or host node_modules copied over the installed ones	Copy lockfiles explicitly; exclude local dependency directories

Key takeaways

• Use --progress=plain to see the full output of each step, and --no-cache so diagnostic RUN lines actually run.
• BuildKit does not keep the failed step's container; use --target to stop at the last working stage, or the experimental buildx debug.
• Most "file not found" errors come from the build context or .dockerignore; inspect the context directly.
• docker build --check catches common Dockerfile mistakes without building.

🧪 Practice

1. (Beginner) Add .git and node_modules to a project's .dockerignore and measure the context size before and after with the throwaway-Dockerfile technique.
2. (Intermediate) Take a multi-stage Dockerfile, introduce a failing RUN in the middle stage, and find the cause in three ways: --progress=plain, --target plus an interactive shell, and (if available) docker buildx debug.
3. Interview question: "A build passes on every developer laptop and fails in CI with exec format error. What is going on?" Hint: compare the CPU architecture of the laptops and the CI runners, the platform of the base images, and what QEMU emulation provides.

Inspecting Network Issues

Why it matters. The layer-by-layer method in Troubleshooting Connectivity finds most problems: a service bound to 127.0.0.1, containers on different networks, a name that does not resolve. Some problems remain after those checks: a published port that works from the host but not from another machine, connections that fail intermittently, DNS that resolves service names but not external ones. These require following the packets themselves through the host: across the network interface, through the NAT and firewall rules Docker creates, over the bridge, and into the container.

Analogy. If a parcel never arrives, you can keep asking the sender and the recipient, or you can check the tracking log at every depot along the route. The first depot that never scanned the parcel is where it got lost.

The path of a packet to a published port

 remote client:  SYN to 203.0.113.10:8080
      |
      v
 [ host eth0 ]                                  (1) tcpdump -ni eth0 tcp port 8080
      |
 [ nat PREROUTING -> DOCKER chain ]             (2) iptables -t nat -L DOCKER -nv     (packet counters)
      |   DNAT 203.0.113.10:8080 -> 172.18.0.5:80
 [ filter FORWARD -> DOCKER-USER, DOCKER ... ]  (3) iptables -L DOCKER-USER -nv       (dropped here?)
      |
 [ bridge br-<network id> ]                     (4) tcpdump -ni br-<id> tcp port 80
      |
 [ veth pair ] --> [ eth0 in the container ]    (5) tcpdump inside the container's namespace
      |
 app listening on 0.0.0.0:80                    (6) does a SYN-ACK come back the same way?

The first checkpoint where the packet stops appearing is where the problem lies: no packets at (1) means an external firewall or cloud security group; packets at (1) but no counter increase at (2) means the port is not published as you think; packets at (2) but none at (4) means a filtering rule; packets at (5) but no reply means the application.

Example: following a packet

docker network create appnet
docker run -d --name web --network appnet -p 8080:80 nginx:1.27-alpine

# (1) Do packets from the remote client arrive at the host?
sudo tcpdump -ni eth0 -c 5 tcp port 8080

# (2) Is the DNAT rule matching? Run before and after a test request: "pkts" should increase
sudo iptables -t nat -L DOCKER -nv | grep 8080
#  pkts bytes target  prot opt in        out  source     destination
#     3   180 DNAT    tcp  --  !br-1a2b  *    0.0.0.0/0  0.0.0.0/0    tcp dpt:8080 to:172.18.0.2:80

# (3) Rules you or other tools added (DOCKER-USER is evaluated before Docker's own rules)
sudo iptables -L DOCKER-USER -nv --line-numbers

# (4) The network's bridge interface is named br-<first 12 characters of the network ID>
BR="br-$(docker network inspect -f '{{.Id}}' appnet | cut -c1-12)"
sudo tcpdump -ni "$BR" -c 5 tcp port 80

# (5) Find the container's veth on the host: eth0's peer index inside the container
docker exec web cat /sys/class/net/eth0/iflink
# 14
ip -o link | grep '^14:'
# 14: veth3c9a1f2@if13: <BROADCAST,MULTICAST,UP,LOWER_UP> ... master br-1a2b3c4d5e6f

# Or capture from inside the container's network namespace, using the host's tcpdump
sudo nsenter -t "$(docker inspect -f '{{.State.Pid}}' web)" -n tcpdump -ni eth0 -c 5 tcp port 80

# Save a capture for Wireshark from a sidecar sharing the namespace
docker run --rm --network container:web -v "$PWD":/cap nicolaka/netshoot:v0.13 \
  tcpdump -ni eth0 -c 200 -w /cap/web.pcap tcp port 80

Chain names differ between Engine versions and between iptables and nftables backends; sudo iptables-save | grep -i docker (or sudo nft list ruleset) shows every rule Docker manages.

Connection tracking. NAT relies on the kernel's connection tracking table (conntrack). When the table is full, new connections are dropped silently, which shows up as random timeouts under load.

# Current NAT translations for the published port (conntrack-tools package)
sudo conntrack -L -p tcp --dport 8080
# tcp 6 117 TIME_WAIT src=198.51.100.7 dst=203.0.113.10 sport=51234 dport=8080
#                     src=172.18.0.2 dst=198.51.100.7 sport=80 dport=51234 [ASSURED] ...

# Table usage versus its limit, and per-CPU statistics (look at drop and insert_failed)
sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max
sudo conntrack -S

Embedded DNS internals. On user-defined networks, containers use 127.0.0.11 as their resolver. Rules inside each container's network namespace redirect that address to a socket served by dockerd, which answers container and service names itself and forwards everything else to the upstream servers (see Embedded DNS and Service Discovery).

# The resolver configuration the container sees
docker exec web cat /etc/resolv.conf
# nameserver 127.0.0.11
# options ndots:0

# The redirect rules inside the container's namespace
sudo nsenter -t "$(docker inspect -f '{{.State.Pid}}' web)" -n iptables -t nat -S DOCKER_OUTPUT

# Compare internal and external resolution from the same namespace
docker run --rm --network container:web nicolaka/netshoot:v0.13 dig +short web
docker run --rm --network container:web nicolaka/netshoot:v0.13 dig +short example.com

# Upstream servers: the "dns" key in daemon.json if set, otherwise the host's resolvers.
# On systemd-resolved hosts Docker skips the 127.0.0.53 stub and reads the real servers here:
grep -s '"dns"' /etc/docker/daemon.json || cat /run/systemd/resolve/resolv.conf

If service names resolve but external names fail, the problem is upstream of the daemon: the host's resolvers, the dns key in daemon.json, or a firewall blocking outbound DNS. With daemon debug logging enabled (see Daemon Logs and Diagnostics), the embedded resolver logs the queries it receives and where it forwards them.

Network events. docker events --filter type=network shows containers being connected to and disconnected from networks, which explains connectivity that "suddenly" changed after a redeploy.

Hard-to-find causes

Symptom	Cause	How to confirm
Published port reachable locally, not remotely, no packets on eth0	Cloud security group or upstream firewall	tcpdump on eth0 shows nothing
Packets reach the host, DNAT counter does not increase	Port published on a specific IP (-p 127.0.0.1:8080:80) or another rule matched first	docker port, iptables -t nat -S
A host firewall seems bypassed by published ports	Docker's NAT rules run before the host firewall's INPUT rules (ufw, firewalld)	Put restrictions in DOCKER-USER
Random timeouts under load	conntrack table full, or SNAT source port exhaustion for many outbound connections	conntrack -S, nf_conntrack_count vs nf_conntrack_max
Large responses or TLS handshakes hang	MTU mismatch on a VPN or overlay	tcpdump shows retransmissions of large segments; ping -M do -s 1472
External names fail, service names work	Upstream DNS not reachable from the daemon	dig against 127.0.0.11 and directly against the upstream server

Key takeaways

• When the basic checks pass, trace the packet: host interface, NAT counters, filter rules, bridge, veth, container interface.
• The first checkpoint where packets disappear identifies the faulty layer.
• nsenter and sidecars with --network container: run capture tools inside a container's namespace without modifying it.
• conntrack exhaustion and MTU mismatches cause intermittent, load-dependent, or size-dependent failures.
• The embedded DNS at 127.0.0.11 answers container names itself and forwards other names upstream; separate the two when debugging DNS.

🧪 Practice

1. (Beginner) Publish a port, send ten requests to it, and confirm that the packet counter of the corresponding DNAT rule increased by the expected amount.
2. (Intermediate) Find the host-side veth interface of a container using iflink, capture traffic on it while the container calls an external HTTPS site, and identify the DNS query, the TCP handshake, and the TLS handshake in the capture.
3. Interview question: "Under peak load, about 1% of requests from containers to an external API time out, while the API provider sees nothing. How would you investigate on the Docker host?" Hint: consider what NAT needs for every outbound connection, which kernel tables and counters could be exhausted, and how to correlate their statistics with the timeouts.

Daemon Logs and Diagnostics

Why it matters. Sometimes the problem is not in any container but in the engine: the daemon does not start after a configuration change, docker ps hangs, containers cannot be removed, or every command fails with "Cannot connect to the Docker daemon". Container logs cannot help here. You need the daemon's own logs, its debug output, and a few diagnostic commands.

Analogy. When one train is late, you ask its driver. When every train is late, you go to the control room and read its log.

Where the daemon logs

Environment	Location
Linux with systemd	journalctl -u docker.service (containerd: journalctl -u containerd.service)
Rootless Docker	journalctl --user -u docker.service
Linux without systemd	Usually /var/log/docker.log, depending on the init system
Docker Desktop on macOS	~/Library/Containers/com.docker.docker/Data/log/ (host/ and vm/ subdirectories)
Docker Desktop on Windows	%LOCALAPPDATA%\Docker\log\ (host\ and vm\ subdirectories)

Debug mode. Setting "debug": true in daemon.json (see daemon.json Configuration) makes the daemon log every API call and much more internal detail. It can be applied with a reload, without stopping containers:

{
  "debug": true
}

sudo systemctl reload docker            # sends SIGHUP: debug is one of the reloadable options
journalctl -u docker.service -f
# level=debug msg="Calling GET /v1.47/containers/json"
# level=debug msg="Calling POST /v1.47/containers/create?name=web"

Debug logs are verbose; switch them off again once you are done. "log-level" (debug, info, warn, error, fatal) sets the verbosity without the other effects of debug mode.

Example: a daemon that will not start

# Status and the last log lines usually contain the reason
systemctl status docker --no-pager
journalctl -u docker.service -b --no-pager | tail -n 20
# dockerd[812]: unable to configure the Docker daemon with file /etc/docker/daemon.json:
#   the following directives are specified both as a flag and in the configuration file: hosts

# Validate the configuration file before restarting
sudo dockerd --validate --config-file /etc/docker/daemon.json

# Run the daemon in the foreground to watch it start (stop the service and socket first)
sudo systemctl stop docker docker.socket
sudo dockerd --debug                    # Ctrl+C when done, then: sudo systemctl start docker

Example: a daemon that hangs

# Dump the stacks of all goroutines; the daemon keeps running
sudo kill -SIGUSR1 "$(pidof dockerd)"
journalctl -u docker.service -n 5 --no-pager | grep -i goroutine
# msg="goroutine stacks written to /var/run/docker/goroutine-stacks-2026-10-14T102233Z.log"

# With debug enabled, Go profiling endpoints are served on the API socket
sudo curl -s --unix-socket /var/run/docker.sock \
  "http://localhost/debug/pprof/goroutine?debug=2" | head -n 40

# Look at containerd, below the daemon: Docker's containers live in the "moby" namespace
sudo ctr -n moby tasks ls
journalctl -u containerd.service --since "30 min ago" --no-pager | tail -n 20

A goroutine dump shows what every part of the daemon is waiting on; many goroutines blocked on the same lock or on a storage operation (for example a hung NFS mount) point directly at the cause.

Quick health checks

docker version                     # client and server versions; fails if the daemon is unreachable
docker context ls                  # which daemon the CLI is talking to (see Docker Contexts)
docker info 2>&1 | grep -i warning # kernel features or configuration problems the daemon detected
# WARNING: No swap limit support
# WARNING: bridge-nf-call-iptables is disabled
docker system df                   # disk used by images, containers, volumes, build cache
df -h /var/lib/docker && df -i /var/lib/docker   # space AND inodes
docker events --since 30m --until "$(date +%s)"  # what the daemon did recently

Docker Desktop diagnostics. Use Troubleshoot > Gather diagnostics in the Desktop UI, or the bundled command-line tool, which checks common problems and produces a bundle you can attach to a bug report:

# macOS
/Applications/Docker.app/Contents/MacOS/com.docker.diagnose check
/Applications/Docker.app/Contents/MacOS/com.docker.diagnose gather -upload

# Windows
& "C:\Program Files\Docker\Docker\resources\com.docker.diagnose.exe" gather -upload

Common daemon problems

Symptom	Likely cause	Check or fix
Cannot connect to the Docker daemon at unix:///var/run/docker.sock	Daemon not running, or the CLI points elsewhere	systemctl status docker; docker context ls; echo $DOCKER_HOST
permission denied while trying to connect to the Docker daemon socket	User not in the docker group	Add the user to the group and log in again (this grants root-equivalent access), or use rootless Docker
Daemon fails to start after a configuration change	Invalid JSON, a number where a string is expected, an option given as both flag and config key	dockerd --validate; journalctl -u docker
no space left on device	Disk or inodes exhausted under /var/lib/docker	docker system df; prune; Log Rotation; df -i
docker ps or docker rm hangs	Stuck shim or container process, deadlock, hung storage	SIGUSR1 goroutine dump; containerd logs; ctr -n moby tasks ls
client version 1.47 is too new. Maximum supported API version is 1.43	Newer CLI than daemon	Upgrade the daemon, or set DOCKER_API_VERSION=1.43 for the client
Containers stop whenever the daemon restarts	live-restore not enabled	Set "live-restore": true in daemon.json

Key takeaways

• Daemon problems are diagnosed from the daemon's logs: journalctl -u docker.service on systemd hosts, the Desktop log folders elsewhere.
• "debug": true plus systemctl reload docker enables detailed logs without restarting containers.
• dockerd --validate and running dockerd --debug in the foreground expose startup errors quickly.
• SIGUSR1 writes a goroutine dump of a hung daemon; containerd's logs and ctr -n moby show the layer below.
• docker info warnings, docker system df, and df -i catch kernel and disk problems early.

🧪 Practice

1. (Beginner) Enable debug mode with a reload, run docker ps, and find the corresponding API call in the daemon's log. Disable debug mode again.
2. (Intermediate) Deliberately break daemon.json in two ways (a JSON syntax error, then "max-file": 3 under log-opts), and for each one find the exact error message with dockerd --validate and with journalctl. Restore the file afterwards.
3. Interview question: "On a production host, docker ps hangs indefinitely but the running containers keep serving traffic. How would you investigate without restarting the daemon?" Hint: consider which tools still work when the API hangs (goroutine dumps, containerd's CLI, kernel logs), and which subsystems commonly block the daemon (storage, a stuck shim, a hung mount).

---

<a id="13-docker-in-development-and-cicd"></a>