7. Docker and Container Workflows

Containers are the unit of execution for most GitLab pipelines. They define the toolchain a job runs in, the databases and helper processes it talks to, and very often the artifact the pipeline ships. This chapter covers how to pick, authenticate and pull job images, how to run sidecar services for integration tests, how to build container images safely inside CI (from Docker-in-Docker to rootless and multi-architecture builds), and how to publish, tag, clean up and proxy images with GitLab's own registries.

<a id="71-using-container-images-in-jobs"></a>

7.1 Using Container Images in Jobs

Section 2.2 introduced the image keyword and how to pin tags. This subchapter covers the decisions behind it: which image to choose, how the runner starts it, how it reaches private registries, and when it pulls a fresh copy.

Choosing Job Images

Why it matters. The job image is the first thing every job pays for. The runner downloads it before a single line of script runs, its contents decide which tools are available, and every package inside it is attack surface. A poorly chosen image shows up as slow pipelines, command not found errors, scripts that spend a minute on apt-get install in every job, and vulnerability reports full of packages nobody uses.

What to choose from. Most job images fall into a few families:

Family	Example	Approximate size (uncompressed)	Strengths	Watch out for
Minimal (Alpine)	alpine:3.20, node:22-alpine	Under 10 MB for the base	Fast pulls, small attack surface	Uses musl instead of glibc: some native binaries and prebuilt packages behave differently or must be compiled
Slim Debian or Ubuntu	python:3.12-slim, debian:12-slim	About 100 to 150 MB	glibc compatibility, apt available, still fairly small	Missing build tools such as gcc and make
Full language image	python:3.12, golang:1.23	Often close to 1 GB	Compilers, headers and common tools included	Slow cold pulls and many unused packages
Vendor tool image	hashicorp/terraform:1.9, amazon/aws-cli:2.17.0	Varies	Exactly the tool you need, maintained upstream	Often has a CLI ENTRYPOINT that must be cleared (next topic)
Custom CI image	$CI_REGISTRY_IMAGE/ci-tools:1.3	Whatever you put in it	All tools preinstalled, versions controlled by you	You must rebuild and patch it yourself
Distroless or scratch	gcr.io/distroless/static	A few MB	Excellent for production images	Cannot be a job image: there is no shell

A practical decision process:

Does an official image of your language/tool exist at the version production uses?
 ├── yes ─> start with its -slim (or -alpine if you have no native dependencies)
 │           └── Do several jobs install the same extra tools in before_script?
 │                 ├── no  ─> done
 │                 └── yes ─> build a custom CI image FROM that base and preinstall them
 └── no  ─> use the vendor's tool image, or build a custom CI image

Building a custom CI image. When jobs repeatedly install the same tools, move the installation into an image built once and reused everywhere. Treat it like any other product: keep its Dockerfile in a repository, version it, and rebuild it on a schedule so security patches land.

# ci/Dockerfile: toolbox image shared by lint, test and deploy jobs
FROM python:3.12-slim

# One RUN layer: install, then remove the apt lists to keep the layer small
RUN apt-get update \
 && apt-get install -y --no-install-recommends git curl jq make \
 && rm -rf /var/lib/apt/lists/*

# Pin tool versions so a rebuild of the same Dockerfile gives the same tools
RUN pip install --no-cache-dir ruff==0.6.9 pytest==8.3.3 poetry==1.8.4

# Run jobs as a non-root user by default
RUN useradd --create-home ci
USER ci

# Build the toolbox only when its Dockerfile changes, or on a weekly schedule for patches
build-ci-image:
  stage: build
  image: docker:27
  services: [docker:27-dind]               # Docker-in-Docker, section 7.3
  variables:
    CI_TOOLS_IMAGE: $CI_REGISTRY_IMAGE/ci-tools:1.3
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"
    - changes: [ci/Dockerfile]
  script:
    - echo "$CI_REGISTRY_PASSWORD" | docker login "$CI_REGISTRY" -u "$CI_REGISTRY_USER" --password-stdin
    - docker build -t "$CI_TOOLS_IMAGE" ci/
    - docker push "$CI_TOOLS_IMAGE"

# Every other job just uses it: no installation step left in before_script
default:
  image: $CI_REGISTRY_IMAGE/ci-tools:1.3

One image or many. A single "kitchen sink" image is convenient but large and changes often. One image per concern keeps each job's pull small.

Approach	Pull cost per job	Maintenance	Best when
One shared toolbox image	High, but cached on warm runners	One Dockerfile	Small teams, few tools, persistent runners
Purpose-built image per job type	Low	Several Dockerfiles	Ephemeral runners (every pull is cold), large pipelines
Official images plus before_script installs	Low pull, high script time	None	Prototypes and rarely run jobs

Variables in image names. The image value is expanded with CI/CD variables defined in .gitlab-ci.yml, in project, group or instance settings, and predefined variables. Variables created inside script do not exist yet when the runner pulls the image, so they cannot select it.

variables:
  PYTHON_VERSION: "3.12"

test:
  image: python:${PYTHON_VERSION}-slim     # expanded before the job container is created
  script:
    - python --version

Key takeaways

• The job image determines available tools, pull time and attack surface. Start from an official slim or Alpine image matching production.
• Alpine images use musl. Prefer slim Debian images when native binaries or prebuilt packages misbehave.
• Move tools that many jobs install into a versioned custom CI image stored in your registry and rebuilt on a schedule.
• Distroless and scratch images have no shell and cannot be job images.
• Image names can use CI/CD variables, but not variables created in the job's own script.

Practice

1. (Beginner) Run cat /etc/os-release; ldd --version 2>&1 | head -1 in alpine:3.20, python:3.12-slim and python:3.12. Note which C library each one uses and how long each job spent in "Preparing the docker executor".
2. (Intermediate) Find a pipeline where at least two jobs install the same packages in before_script. Build a custom CI image containing them, publish it to the project registry, switch the jobs to it, and compare total pipeline duration before and after.
3. Interview question: "Your team wants a single 3 GB CI image that contains every tool any job might need. What are the trade-offs, and what would you propose instead?" Hint: consider cold pulls on ephemeral runners, rebuild frequency, vulnerability counts and who owns the image.

Overriding Entrypoints

Why it matters. Many tool images are built to be run as a command, for example docker run hashicorp/terraform plan. Their ENTRYPOINT is the tool itself. A CI job, however, needs the container to start a shell so the runner can feed it your script. When those two expectations collide, the job fails before your first command with a confusing error from the tool, not from GitLab.

How a container starts. A Docker image has two settings that together form the first process:

ENTRYPOINT   +   CMD (or the arguments passed at run time)   =   process started in the container
["terraform"]    ["--help"]                                       terraform --help
[] (none)        ["sh"]                                           sh
["/entrypoint.sh"]  ["sh"]                                        /entrypoint.sh sh   (works if the script ends with exec "$@")

With the Docker executor, the runner keeps the image's ENTRYPOINT and passes a shell as the command. The result depends on what the entrypoint does with that argument:

Image ENTRYPOINT	What the runner ends up starting	Result
None, or a shell	A shell	Works
A wrapper script ending in exec "$@"	The wrapper runs its setup, then starts the shell	Works, and the setup is useful (the docker image uses this)
A CLI tool (terraform, aws, trivy, helm)	The tool, receiving a shell as its "subcommand"	Fails, for example Terraform has no command named "sh"

The fix. Override the entrypoint in the extended form of image. An empty string clears it, so the runner's shell becomes the first process:

plan:
  image:
    name: hashicorp/terraform:1.9
    entrypoint: [""]                 # clear ENTRYPOINT ["/bin/terraform"]
  script:
    - terraform init                 # terraform is still on PATH; we just call it ourselves
    - terraform plan

scan-image:
  image:
    name: aquasec/trivy:0.56.2
    entrypoint: [""]                 # ENTRYPOINT is ["trivy"]
  script:
    - trivy image --exit-code 1 --severity CRITICAL "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"

Before using a new image, check what it starts with:

# Print ENTRYPOINT and CMD of an image (pull it first if it is not local)
docker image inspect --format 'ENTRYPOINT={{json .Config.Entrypoint}} CMD={{json .Config.Cmd}}' hashicorp/terraform:1.9
# ENTRYPOINT=["/bin/terraform"] CMD=null    -> needs entrypoint: [""]

docker image inspect --format 'ENTRYPOINT={{json .Config.Entrypoint}} CMD={{json .Config.Cmd}}' python:3.12-slim
# ENTRYPOINT=null CMD=["python3"]           -> fine, the runner replaces CMD

Kubernetes executor. By default the Kubernetes executor does not run the image's ENTRYPOINT at all; it starts the shell directly. Wrapper entrypoints that do useful setup are therefore skipped. Administrators can enable the runner feature flag FF_KUBERNETES_HONOR_ENTRYPOINT to make Kubernetes behave like Docker. If the same job behaves differently on the two executors, check the entrypoint first.

Services have entrypoints too. Service containers run their image normally, so you rarely clear their entrypoint. You more often change their arguments with services:command, or replace the entrypoint with services:entrypoint (section 7.2):

integration:
  image: python:3.12-slim
  services:
    - name: postgres:16
      command: ["postgres", "-c", "fsync=off"]   # replaces CMD; the image's entrypoint still initializes the database
  script:
    - pytest tests/integration

Key takeaways

• The runner needs the job container to start a shell. An ENTRYPOINT that is a CLI tool breaks that.
• entrypoint: [""] clears the image's entrypoint. The tool remains available on PATH.
• Wrapper entrypoints that end with exec "$@" work and should usually be kept.
• The Kubernetes executor ignores image entrypoints unless FF_KUBERNETES_HONOR_ENTRYPOINT is enabled.
• Inspect Config.Entrypoint and Config.Cmd before adopting a new tool image.

Practice

1. (Beginner) Run a job with image: amazon/aws-cli:2.17.0 and script: [aws --version]. Read the error, then fix it with entrypoint: [""].
2. (Intermediate) Write a small image whose entrypoint script exports a variable and then runs exec "$@". Use it as a job image and show that the variable is visible in the script. Then remove exec "$@" from the script and explain the new behavior.
3. Interview question: "A job that works on the team's Docker runners fails on the new Kubernetes runners because a tool cannot find its configuration. What do you check?" Hint: compare what each executor does with the image's ENTRYPOINT.

Private Registry Authentication

Why it matters. Real pipelines use images that are not public: the team's custom CI image, a base image with internal certificates, a licensed tool. The runner pulls image and services before your script runs, so a docker login in before_script comes too late. Credentials for pulls must reach the runner in a different way.

Which mechanism applies.

Image location	How the runner authenticates	What you configure
Container registry of the same GitLab project	Automatically, with the job token	Nothing
Container registry of another project on the same instance	Automatically, with the job token	The pipeline's user needs access to that project, and that project's job token allowlist must include yours (section 11.4)
External registry with static credentials (Harbor, private Docker Hub repository, Artifactory)	DOCKER_AUTH_CONFIG	A CI/CD variable or runner setting
Cloud registry with short-lived tokens (Amazon ECR, Google Artifact Registry, Azure ACR)	A Docker credential helper on the runner	Runner host configuration and cloud identity
Any registry, Kubernetes executor	Kubernetes imagePullSecrets or DOCKER_AUTH_CONFIG	image_pull_secrets in config.toml

DOCKER_AUTH_CONFIG. This variable holds the same JSON that docker login writes to ~/.docker/config.json. The auth field is the Base64 encoding of username:password:

# Encode credentials without a trailing newline (-w0 avoids line wrapping in GNU base64)
printf '%s' 'ci-robot:s3cr3t-token' | base64 -w0
# Y2ktcm9ib3Q6czNjcjN0LXRva2Vu

{
  "auths": {
    "harbor.example.com": {
      "auth": "Y2ktcm9ib3Q6czNjcjN0LXRva2Vu"
    }
  }
}

Store the JSON itself as the value of a CI/CD variable named DOCKER_AUTH_CONFIG in the project or group settings. Then any job can use images from that registry:

test:
  image: harbor.example.com/platform/python-ci:3.12   # pulled with the credentials from DOCKER_AUTH_CONFIG
  services:
    - name: harbor.example.com/platform/postgres-test:16   # services use the same credentials
      alias: db
  script:
    - pytest

The runner looks for registry credentials in this order, and uses the first source that has an entry for the image's registry:

1. DOCKER_AUTH_CONFIG CI/CD variable           (project, group or instance settings, or .gitlab-ci.yml)
2. DOCKER_AUTH_CONFIG in the runner's config.toml "environment"
3. ~/.docker/config.json of the user running gitlab-runner

Credential helpers for cloud registries. Cloud registries issue tokens valid for hours, so static JSON would expire. A credential helper is a small binary that fetches a fresh token on demand. For Amazon ECR, install docker-credential-ecr-login on the runner host, give the host an IAM role that can pull, and reference the helper:

# config.toml on the runner host
[[runners]]
  name = "docker-ecr"
  executor = "docker"
  environment = ['DOCKER_AUTH_CONFIG={"credHelpers":{"123456789012.dkr.ecr.eu-west-1.amazonaws.com":"ecr-login"}}']

The job then simply uses image: 123456789012.dkr.ecr.eu-west-1.amazonaws.com/ci/python:3.12. No secret is stored in GitLab.

Kubernetes executor. The job pod is created by Kubernetes, so the cluster must be allowed to pull. Create a docker-registry secret in the runner's namespace and list it in config.toml:

kubectl create secret docker-registry harbor-pull \
  --namespace gitlab-runner \
  --docker-server=harbor.example.com \
  --docker-username=ci-robot \
  --docker-password="$ROBOT_TOKEN"

[runners.kubernetes]
  namespace = "gitlab-runner"
  image_pull_secrets = ["harbor-pull"]

Pulling versus logging in from the script. DOCKER_AUTH_CONFIG is read by the runner to pull image and services. Commands inside the script, such as docker push, buildah push or helm pull oci://..., need their own login. The same credentials can be reused there:

push:
  image: docker:27
  services: [docker:27-dind]
  script:
    - mkdir -p ~/.docker && printf '%s' "$DOCKER_AUTH_CONFIG" > ~/.docker/config.json   # reuse for the CLI
    - docker pull harbor.example.com/platform/base:1.0

Security notes.

• Use a dedicated read-only robot account or token for pulls, not a person's credentials.
• The JSON value contains braces and quotes, which masked variables do not accept, so it cannot be masked. Protect it, limit who has Maintainer access, and never echo it.
• Prefer credential helpers or the job token where possible: nothing long-lived is stored.

Key takeaways

• Images and services are pulled before the script runs. docker login in the script does not help the runner.
• Images in GitLab's own registry are pulled with the job token automatically, subject to project access and job token allowlists.
• DOCKER_AUTH_CONFIG holds Docker config.json content for external registries. It is checked before runner-level configuration.
• Use credential helpers for cloud registries and image_pull_secrets for the Kubernetes executor.
• Script commands that talk to a registry need their own login.

Practice

1. (Beginner) Push a small image to a private repository on an external registry (or a second private GitLab project). Use it as a job image without credentials, read the pull error, then fix it with DOCKER_AUTH_CONFIG.
2. (Intermediate) Use an image from another private project on the same instance. Make the job fail by removing your project from that project's job token allowlist, then restore access, and explain each error you saw.
3. Interview question: "How would you let hundreds of projects pull base images from Amazon ECR without storing AWS keys in GitLab?" Hint: consider where the credential helper runs, which identity the runner host has, and how instance-level configuration reaches every job.

Image Pull Policies

Why it matters. Pulling an image on every job guarantees freshness and enforces credential checks, but costs time, bandwidth and registry rate limits. Reusing a local copy is fast but can run stale or unauthorized images. The pull policy is where you choose that trade-off, and on shared runners the choice has security consequences.

The three policies.

Policy	Behavior	Speed	Freshness	Typical use
always	Contact the registry every time. Download only layers that changed	Slowest (one registry round trip even when nothing changed)	Always current	Shared runners, mutable tags such as node:22, private images
if-not-present	Use the local image if one exists with that name and tag. Pull only if missing	Fast on warm hosts	Stale if the tag was moved upstream	Dedicated runners, images pinned by digest or unique tags
never	Use only local images. Fail if missing	Fastest	Whatever the administrator preloaded	Air-gapped runners with preloaded images

Job needs node:22
      │
      ├── always ─────────> ask registry for the manifest of node:22
      │                        └── download missing layers ─> start container
      │
      ├── if-not-present ─> node:22 on the host?
      │                        ├── yes ─> start container (no registry contact, no credential check)
      │                        └── no  ─> pull ─> start container
      │
      └── never ──────────> node:22 on the host?
                               ├── yes ─> start container
                               └── no  ─> job fails

Who decides. The runner administrator sets the default and the allowed set in config.toml (section 6.2). A job can request a policy with image:pull_policy or services:pull_policy, but only if the runner allows it. If allowed_pull_policies is not set, only the runner's own pull_policy values are allowed.

[runners.docker]
  pull_policy = ["always", "if-not-present"]           # default: try always, fall back to the local copy if the pull fails
  allowed_pull_policies = ["always", "if-not-present"]  # what jobs may request

unit-tests:
  image:
    name: registry.example.com/ci/node@sha256:1f3e...   # digest: content can never change
    pull_policy: if-not-present                         # safe to reuse a local copy
  services:
    - name: redis:7
      pull_policy: [always, if-not-present]             # a list: try in order, fall back on failure
  script:
    - npm test

If a job asks for a policy the runner does not allow, the job fails at startup with an error stating that the requested policy is not one of the allowed pull policies.

Fallback lists. A list such as [always, if-not-present] is tried in order. The runner tries always, and if the registry is unreachable or rate-limited, it falls back to a local copy. This keeps pipelines running through registry outages while staying fresh the rest of the time.

The security issue with if-not-present. With if-not-present, the runner never contacts the registry when the image is already on the host, so no credential check happens. On a runner shared between projects, a job from project B can use a private image that project A pulled earlier, even if B has no access to it. Use always on multi-tenant runners, or make sure hosts are not shared between trust levels (for example, ephemeral VMs per job).

Kubernetes executor. The policies map to Kubernetes imagePullPolicy: always to Always, if-not-present to IfNotPresent and never to Never. If the runner sets no policy, Kubernetes applies its own default (Always for :latest tags, otherwise IfNotPresent).

Key takeaways

• always checks the registry every time, if-not-present reuses local copies, never uses only preloaded images.
• Jobs can request a policy only if the runner lists it in allowed_pull_policies (or as its own pull_policy).
• A policy list falls back in order, which protects pipelines from registry outages.
• if-not-present skips credential checks: avoid it on runners shared by projects of different trust levels.
• Combine if-not-present with digests or unique tags to avoid stale images.

Practice

1. (Beginner) On a runner you control, run the same job twice with pull_policy: always and twice with if-not-present. Compare the "Preparing the docker executor" duration and the log lines about pulling.
2. (Intermediate) Push my-tool:1.0 to your registry and run a job with it on a runner that uses if-not-present. Push different content to the same tag and run the job again. Show which version ran, then fix the problem in two different ways.
3. Interview question: "Why can if-not-present be a security problem on an instance-wide runner, even when every image is in a private registry?" Hint: think about when the runner contacts the registry, and which job's credentials it uses.

<a id="72-services"></a>

7.2 Services

A service is an extra container that runs next to the job for its whole duration: a database for integration tests, a cache, a mock API or a Docker daemon. This subchapter covers how to declare services, how jobs find them by hostname, how to use them as disposable test databases, and how to make sure they are ready before tests start.

services Keyword

Why it matters. Integration tests need real dependencies. Installing PostgreSQL inside the job image makes the image heavy and the setup slow. Pointing tests at a shared staging database makes them flaky and lets parallel jobs interfere with each other. Services give each job its own throwaway dependency, started from an ordinary image and destroyed with the job. It is like renting a fresh, empty workbench next to yours for each task, instead of sharing one cluttered bench with the whole team.

What it is. services is a list of images that the runner starts as sidecar containers before before_script, on the same network as the job container. They keep running until the job ends and are then removed with everything they stored. Services need a container-based executor (Docker, Docker Autoscaler or Kubernetes). The shell executor does not support them.

Job lifecycle with services
  prepare ─> start service containers ─> wait for their ports ─> start job container
          ─> before_script ─> script ─> after_script ─> remove job and service containers (data is gone)

Short and extended forms.

test:
  image: ruby:3.3
  services:
    - redis:7                                   # short form: image name only
    - name: postgres:16                         # extended form
      alias: db                                 # extra hostname (next topic)
      variables:                                # passed only to this service
        POSTGRES_PASSWORD: test
      command: ["postgres", "-c", "max_connections=200"]   # override the image's CMD
      pull_policy: if-not-present               # must be allowed by the runner (section 7.1)
  script:
    - bundle exec rspec

Key	Purpose
name	Image of the service. Same naming rules and registry authentication as image
alias	Additional hostnames for the service
variables	Variables for this service only
command	Replaces the image's CMD
entrypoint	Replaces the image's ENTRYPOINT
pull_policy	always, if-not-present, never, or a list
docker:platform, docker:user	Architecture to pull and user to run as (Docker executor)
kubernetes:user	User to run as (Kubernetes executor)

Variables reach services. Services receive the job's CI/CD variables in addition to their own services:variables. That is convenient but means every service sees every job variable, including secrets. Prefer services:variables for service configuration, and avoid attaching services to jobs that handle deployment credentials.

Testing the image you just built. A service image must exist in a registry before the job starts, so a job cannot use an image it builds itself. It can use an image an earlier job pushed, which is the standard pattern for end-to-end tests:

stages: [build, e2e]

build-app:
  stage: build
  image: docker:27
  services: [docker:27-dind]
  script:
    - echo "$CI_REGISTRY_PASSWORD" | docker login "$CI_REGISTRY" -u "$CI_REGISTRY_USER" --password-stdin
    - docker build -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA" .
    - docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"

e2e:
  stage: e2e
  image: mcr.microsoft.com/playwright:v1.48.0-jammy
  services:
    - name: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA   # the application built in the previous stage
      alias: app
  variables:
    BASE_URL: http://app:8080
  script:
    - npx playwright test

Where to declare them. A service set in default starts for every job (section 2.4). Put services in the jobs that need them, or in a hidden template that those jobs extends.

Key takeaways

• Services are sidecar containers started before before_script and destroyed with the job. Each job gets its own instances.
• They require a container-based executor. The shell executor does not run them.
• The extended form supports alias, variables, command, entrypoint, pull_policy and executor-specific user and platform options.
• Services receive the job's variables. Keep secrets out of jobs with services you do not trust.
• A service can be an image pushed by an earlier job, which enables end-to-end tests of the build.

Practice

1. (Beginner) Add a redis:7 service to a job using redis:7 as job image too, and run redis-cli -h redis ping in the script.
2. (Intermediate) Build an application image in one job, push it with the commit SHA as tag, and run it as a service in a later job that calls its health endpoint with curl.
3. Interview question: "Why do teams prefer per-job service containers over a shared test database, and what do they give up?" Hint: consider isolation, parallel jobs, realism of data and startup time.

Service Aliases

Why it matters. The job reaches a service over the network, so it needs a hostname. GitLab derives one from the image name, but derived names for registry-hosted images are long and awkward, two services from the same image would collide, and configuration files usually expect a fixed host such as db. Aliases give services stable, readable names.

Derived hostnames. Without an alias, the hostname comes from the image name with the tag removed. Slashes are replaced in two ways, and both names work:

Service image	Hostnames
postgres:16	postgres
redis:7-alpine	redis
bitnami/kafka:3.8	bitnami__kafka and bitnami-kafka
registry.example.com/qa/mock-api:2.1	registry.example.com__qa__mock-api and registry.example.com-qa-mock-api

Explicit aliases. alias adds hostnames to the derived ones. Several aliases can be given, separated by commas or spaces:

test:
  image: node:22
  services:
    - name: registry.example.com/qa/mock-api:2.1
      alias: payments-api                    # readable name instead of the derived one
    - name: postgres:16
      alias: db,postgres-primary             # two extra names for the same container
      variables:
        POSTGRES_PASSWORD: test
  variables:
    PAYMENTS_URL: http://payments-api:8080
    DATABASE_URL: postgres://postgres:test@db:5432/postgres
  script:
    - npm run test:integration

Two services from the same image. Without aliases, two postgres:16 services would share the hostname postgres. Give each one its own alias:

replication-tests:
  image: python:3.12-slim
  services:
    - name: postgres:16
      alias: pg-primary
      variables: { POSTGRES_PASSWORD: test }
    - name: postgres:16
      alias: pg-replica
      variables: { POSTGRES_PASSWORD: test }
  script:
    - pytest tests/replication --primary pg-primary --replica pg-replica

Can services reach each other? It depends on how the executor wires the network:

Executor and network mode	Job to service	Service to service	Service to job
Docker, default (legacy container links)	Yes, by hostname or alias	No	No
Docker with FF_NETWORK_PER_BUILD enabled	Yes	Yes, by alias	Yes, the job container has the alias build
Kubernetes (all containers in one pod)	Yes, through localhost or alias	Yes, through localhost	Yes, through localhost

The feature flag can be set as a CI/CD variable for a job or in the runner configuration:

contract-tests:
  variables:
    FF_NETWORK_PER_BUILD: "true"            # one Docker network per job: all containers resolve each other
  image: node:22
  services:
    - name: registry.example.com/qa/orders:1.4
      alias: orders
      variables:
        PAYMENTS_URL: http://payments:8080  # this service calls the next one: needs a per-build network
    - name: registry.example.com/qa/payments:3.0
      alias: payments
  script:
    - npm run test:contracts

On Kubernetes, aliases are added as host entries that point to 127.0.0.1, because every container shares the pod's network namespace. The consequence is that two services cannot listen on the same port there, even with different aliases.

Key takeaways

• Derived hostnames drop the tag and replace / with __ or -. Both forms resolve.
• alias adds stable names. Several aliases are allowed, and they are required for two services from the same image.
• With the Docker executor's default networking, only the job can reach services. Enable FF_NETWORK_PER_BUILD when services must call each other.
• On Kubernetes, all containers share localhost, so service ports must not collide.

Practice

1. (Beginner) Start bitnami/redis:7.4 as a service and connect to it using both derived hostnames, then add an alias and connect with it.
2. (Intermediate) Run two services where one calls the other over HTTP. Show that it fails with the default Docker networking and succeeds with FF_NETWORK_PER_BUILD: "true".
3. Interview question: "A job that connects to db:5432 works on Docker runners but fails on Kubernetes runners because two services both listen on port 8080. Why, and what are your options?" Hint: compare the network namespace of a pod with a per-job Docker network.

Database Services for Testing

Why it matters. Database bugs, such as a missing index, a wrong migration or a constraint the ORM did not expect, only show up against a real database engine. SQLite in place of PostgreSQL or mocked repositories catch fewer of them. A service gives each job a real, empty engine at the exact version production runs, which is the closest a pipeline gets to production without touching it.

Common database services.

Engine	Image	Key variables	Port	Readiness check
PostgreSQL	postgres:16	POSTGRES_PASSWORD (required), POSTGRES_USER, POSTGRES_DB, or POSTGRES_HOST_AUTH_METHOD: trust	5432	pg_isready -h <host>
MySQL	mysql:8.4	MYSQL_ROOT_PASSWORD (required), MYSQL_DATABASE, MYSQL_USER, MYSQL_PASSWORD	3306	mysqladmin ping -h <host>
MariaDB	mariadb:11	MARIADB_ROOT_PASSWORD, MARIADB_DATABASE	3306	mariadb-admin ping -h <host>
Redis	redis:7	None needed	6379	redis-cli -h <host> ping
MongoDB	mongo:7	MONGO_INITDB_ROOT_USERNAME, MONGO_INITDB_ROOT_PASSWORD	27017	mongosh --host <host> --eval 'db.runCommand({ping:1})'
Elasticsearch	elasticsearch:8.15.3	ES_JAVA_OPTS: "-Xms512m -Xmx512m", plus -Ediscovery.type=single-node -Expack.security.enabled=false passed through command	9200	curl -fs http://<host>:9200/_cluster/health

The passwords in this table are throwaway values for a database that lives only for one job, so they can be written directly in .gitlab-ci.yml. Never reuse real credentials here.

A complete example. A Python application tested against PostgreSQL and Redis:

.integration:
  image: python:3.12-slim
  services:
    - name: postgres:16
      alias: db
      variables:
        POSTGRES_DB: app_test
        POSTGRES_USER: app
        POSTGRES_PASSWORD: app                   # disposable credentials
      # Durability is pointless for a database deleted after the job: trade it for speed
      command: ["postgres", "-c", "fsync=off", "-c", "synchronous_commit=off", "-c", "full_page_writes=off"]
    - name: redis:7
      alias: cache
  variables:
    DATABASE_URL: postgresql://app:app@db:5432/app_test
    REDIS_URL: redis://cache:6379/0
  before_script:
    - apt-get update && apt-get install -y --no-install-recommends postgresql-client   # for pg_isready (or bake it into a CI image)
    - pip install -r requirements.txt
    - until pg_isready -h db -U app; do sleep 1; done   # wait until the server accepts connections
    - alembic upgrade head                               # schema from migrations, exactly as in production

integration-tests:
  extends: .integration
  parallel: 4                                   # each of the 4 jobs gets its own db and cache
  script:
    - pytest tests/integration --splits 4 --group "$CI_NODE_INDEX"

Patterns that work well.

• Schema from migrations, not dumps. Running the real migrations tests them on every pipeline. Init scripts such as /docker-entrypoint-initdb.d are hard to use because the service container cannot easily see the repository files, so seed data from the job script instead.
• Isolation for free. Each job, including each parallel instance (section 4.5), has its own database. Tests never collide on shared rows.
• Match production versions. Pin the same major version as production (postgres:16, not postgres:latest), and test upgrades by running a matrix of versions.
• Speed up writes. Turn off durability settings as above, and ask runner administrators about services_tmpfs in [runners.docker], which can place a service's data directory in memory.

# Test against two PostgreSQL versions during an upgrade project
integration-tests-matrix:
  extends: .integration
  parallel:
    matrix:
      - PG_VERSION: ["15", "16"]
  services:
    - name: postgres:${PG_VERSION}
      alias: db
      variables: { POSTGRES_DB: app_test, POSTGRES_USER: app, POSTGRES_PASSWORD: app }
    - name: redis:7
      alias: cache
  script:
    - pytest tests/integration

Services or Testcontainers? Libraries such as Testcontainers start containers from the test code itself. That keeps local runs and CI identical but requires a Docker daemon in the job (Docker-in-Docker or a mounted socket, section 7.3), which brings the security trade-offs of privileged builds. GitLab services need no daemon and no privileges.

Key takeaways

• Database services give each job a real engine at the production version, isolated from every other job.
• Most database images need a required password or auth variable. Use disposable values.
• Build the schema with the application's migrations and seed data from the job script.
• Disable durability settings or use tmpfs to speed up test databases.
• Testcontainers needs a Docker daemon in the job. Services do not.

Practice

1. (Beginner) Start a mysql:8.4 service, wait for it with mysqladmin ping, create a table and insert a row from the job script.
2. (Intermediate) Run the same integration test suite with and without the fsync=off settings on PostgreSQL, using parallel: 3. Compare test durations and confirm that the parallel jobs never share data.
3. Interview question: "Your integration tests pass in CI against PostgreSQL in a service, but a migration fails in production. What could the CI setup be hiding?" Hint: consider data volume, version differences, locks on large tables and the empty database each job starts with.

Service Networking and Health Checks

Why it matters. A container that has started is not necessarily ready. PostgreSQL initializes its data directory before it accepts connections, and Elasticsearch can take half a minute to come up. Tests that run too early fail with "connection refused", and the failure appears random because it depends on how loaded the host is. Knowing what the runner checks, and what it does not, lets you make service-based jobs reliable.

How the network looks. With the Docker executor, services are separate containers. They are reachable by hostname, never through localhost, and their ports are not published to the host:

Docker host
 └── job network
       ├── job container ─── db:5432 ─────> postgres:16 (alias db)
       │                 └── cache:6379 ──> redis:7     (alias cache)
       └── no ports published to the host: nothing outside the job can connect

Kubernetes node
 └── job pod (one network namespace)
       ├── build container ── localhost:5432 ─> postgres container
       └──                ── localhost:6379 ─> redis container

Symptom	Likely cause	Fix
connection refused on localhost:5432 (Docker executor)	Services are not on localhost with Docker	Use the hostname or alias
could not translate host name "db"	Typo in alias, or a service tried to reach another service with legacy links	Check aliases, enable FF_NETWORK_PER_BUILD
Intermittent connection refused at test start	Tests started before the service was ready	Add an explicit readiness loop
Service exits immediately	Missing required variable, such as POSTGRES_PASSWORD	Read the service logs (below)
address already in use (Kubernetes)	Two containers in the pod use the same port	Change one service's port with command

What the runner checks. With the Docker executor, after starting services the runner waits for each service's exposed TCP ports to accept connections, up to wait_for_services_timeout (30 seconds by default, configured in [runners.docker]). Two details matter:

• The check is TCP only. An open port does not mean the application is ready to serve queries.
• If the check times out, the runner prints a warning that the service probably did not start properly, and the job continues anyway.

If an image exposes several ports, or the wrong port is checked, set HEALTHCHECK_TCP_PORT for that service:

search-tests:
  image: python:3.12-slim
  services:
    - name: elasticsearch:8.15.3
      alias: search
      # Settings with dots in their names are passed as -E flags rather than variables
      command: ["bin/elasticsearch", "-Ediscovery.type=single-node", "-Expack.security.enabled=false"]
      variables:
        ES_JAVA_OPTS: "-Xms512m -Xmx512m"   # keep the JVM heap small on CI hosts
        HEALTHCHECK_TCP_PORT: "9200"        # check the HTTP port, not the transport port 9300
  script:
    - pytest tests/search

Always add your own readiness check. Whatever the executor does, wait for the condition your tests need, with a time limit so a broken service fails fast instead of hanging until the job timeout:

.wait-for-services:
  before_script:
    - |
      # Wait up to 60 s for PostgreSQL to accept connections
      for i in $(seq 1 60); do
        pg_isready -h db -U app && break
        echo "waiting for db ($i)"; sleep 1
      done
      pg_isready -h db -U app || { echo "db never became ready"; exit 1; }
    - |
      # Wait up to 60 s for Elasticsearch to report at least yellow health
      timeout 60 sh -c 'until curl -fs "http://search:9200/_cluster/health?wait_for_status=yellow&timeout=5s"; do sleep 2; done'

Reading service logs. By default, service output is not shown. Set CI_DEBUG_SERVICES to stream it into the job log, labeled with the service it came from:

integration-tests:
  variables:
    CI_DEBUG_SERVICES: "true"               # show service container logs in the job log
  services:
    - name: postgres:16
      alias: db                             # forgot POSTGRES_PASSWORD on purpose
  script:
    - pg_isready -h db
# The job log now includes the service's output, labeled with the service name, for example:
#   Error: Database is uninitialized and superuser password is not specified.

Service logs can contain sensitive values, so enable this only while debugging.

Key takeaways

• With Docker, services are reached by hostname or alias, never localhost. On Kubernetes, everything is on localhost.
• The Docker executor waits for exposed TCP ports up to wait_for_services_timeout, then continues with only a warning.
• HEALTHCHECK_TCP_PORT selects which port the runner checks.
• An open port is not readiness. Add a bounded readiness loop using the engine's own check.
• CI_DEBUG_SERVICES: "true" shows service logs, which is the fastest way to diagnose services that crash at startup.

Practice

1. (Beginner) Start a postgres:16 service without POSTGRES_PASSWORD and with CI_DEBUG_SERVICES: "true". Find the error in the job log, then fix it.
2. (Intermediate) Create a job whose tests sometimes fail because Elasticsearch is not ready yet (run it several times). Make it reliable with a bounded readiness loop, and make it fail fast with a clear message if the service never becomes healthy.
3. Interview question: "The runner's log says the service health check failed, yet the job passed. How is that possible, and would you change anything?" Hint: consider what the runner checks, what it does on timeout, and what your script checks on its own.

<a id="73-building-container-images"></a>

7.3 Building Container Images

Building an image inside a CI job means running a container builder inside a container, which raises questions of privilege, isolation, speed and portability. This subchapter covers the classic Docker-based approaches, rootless alternatives, BuildKit's advanced features, images for several CPU architectures, and how to keep builds fast when every job starts with an empty cache.

Docker-in-Docker

Why it matters. The docker build command is a client. It sends the build context to a Docker daemon, which does the actual work. A job container normally has no daemon, so docker build fails with "Cannot connect to the Docker daemon". Docker-in-Docker (DinD) solves this by starting a complete, private Docker daemon as a service for each job. It is the most common way to build images in GitLab and the one most documentation assumes.

How it works. The job uses the docker image, which contains the CLI. The docker:dind service runs the daemon. Its derived hostname is docker, which is why DOCKER_HOST points there.

Runner host (Docker executor, privileged = true)
 └── job network
       ├── job container      image: docker:27        docker CLI ── tcp://docker:2376 (TLS) ──┐
       └── service container  image: docker:27-dind   dockerd  <─────────────────────────────┘
                                                        └── its own image store, layer cache, networks
                                                            (thrown away when the job ends)

The daemon needs kernel features that unprivileged containers do not have (mounting filesystems, managing cgroups and networks), so the runner must run containers in privileged mode.

Runner configuration. The administrator enables privileged mode and adds a volume through which the daemon shares its TLS client certificates with the job container:

[[runners]]
  name = "docker-dind"
  executor = "docker"
  [runners.docker]
    image = "docker:27"
    privileged = true                       # required by dockerd inside the dind service
    volumes = ["/certs/client", "/cache"]   # /certs/client is shared between service and job container

Job configuration with TLS (recommended).

build-image:
  stage: build
  image: docker:27                          # CLI; keep the same major version as the service
  services:
    - docker:27-dind                        # daemon; hostname "docker"
  variables:
    DOCKER_HOST: tcp://docker:2376          # 2376 = TLS port
    DOCKER_TLS_CERTDIR: "/certs"            # dind generates CA, server and client certificates here
    DOCKER_TLS_VERIFY: "1"
    DOCKER_CERT_PATH: "$DOCKER_TLS_CERTDIR/client"   # the CLI reads the client certificates from here
  before_script:
    - until docker info > /dev/null 2>&1; do sleep 1; done   # wait for the daemon (section 7.2)
    - echo "$CI_REGISTRY_PASSWORD" | docker login "$CI_REGISTRY" -u "$CI_REGISTRY_USER" --password-stdin
  script:
    - docker build --tag "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA" .
    - docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"

Without TLS, the daemon listens on port 2375 in plain text. That only makes sense on runners where nothing else can reach the job network. Set DOCKER_HOST: tcp://docker:2375 and DOCKER_TLS_CERTDIR: "" (an empty value disables certificate generation).

Useful daemon options. Arguments after command are passed to dockerd:

services:
  - name: docker:27-dind
    command:
      - "--mtu=1450"                                   # match a smaller network MTU (common on cloud VMs and Kubernetes overlays)
      - "--registry-mirror=https://mirror.gcr.io"      # pull Docker Hub images through a mirror (section 7.4)

Common errors.

Error	Cause	Fix
Cannot connect to the Docker daemon at tcp://docker:2375	Daemon not ready yet, or the daemon is using TLS on 2376 while the client uses 2375	Wait for docker info, and make DOCKER_HOST and DOCKER_TLS_CERTDIR consistent
dind service log shows mount: permission denied (are you root?)	Runner is not privileged	privileged = true on a dedicated runner
x509: certificate signed by unknown authority or missing certificates	/certs/client volume not configured	Add the volume in config.toml
Pulls or pushes hang on large layers	MTU mismatch	Pass --mtu to the daemon
toomanyrequests from Docker Hub	The private daemon pulls anonymously	Registry mirror or Dependency Proxy (section 7.4)

Trade-offs.

Strength	Weakness
Full Docker CLI and Compose, exactly like a laptop	Requires privileged mode, which gives the job root-level control of the host
Each job has its own clean daemon: no interference between jobs	Every job starts with an empty layer cache (fix: registry caching, last topic)
Works on any Docker or Kubernetes runner that allows privileged containers	Daemon startup adds seconds to each job

Key takeaways

• DinD runs a private Docker daemon as a docker:dind service. The job's CLI connects to it at tcp://docker:2376.
• It requires privileged mode on the runner. Keep such runners dedicated to trusted projects.
• Use TLS: set DOCKER_TLS_CERTDIR and share /certs/client through a runner volume.
• Keep the docker and docker:dind versions aligned, and pin them.
• Each job starts with an empty cache and anonymous Docker Hub pulls, so plan for registry caching and a mirror.

Practice

1. (Beginner) Build and push an image of a small web application with DinD and TLS. Then check the image in Deploy > Container registry.
2. (Intermediate) Break the setup in three ways, one at a time: use port 2375 with TLS enabled, remove /certs/client from the runner volumes, and disable privileged mode. Record each error and map it to the table above.
3. Interview question: "Why does Docker-in-Docker need privileged mode, and what could a malicious job do on such a runner?" Hint: think about kernel capabilities, devices, the host filesystem and other jobs on the same host.

Docker Socket Binding

Why it matters. DinD starts a new daemon in every job, with an empty cache. The runner host already has a Docker daemon, which is the one the Docker executor uses to start job containers. Mounting that daemon's socket into the job lets docker build use it directly: no extra daemon, no privileged container, and a layer cache that survives between jobs. The speed is real, and so are the risks.

How it works. The socket /var/run/docker.sock is the daemon's API. Whoever can write to it can control the daemon:

Runner host
 └── dockerd (host daemon)  <── /var/run/docker.sock ──┐
       ├── job container (docker:27) ──────────────────┘  "docker build" talks to the HOST daemon
       ├── containers started by the job                  are siblings of the job, not children
       └── images and layer cache of ALL jobs              shared and persistent

Configuration. The socket is bind-mounted through the runner's volumes:

[runners.docker]
  image = "docker:27"
  privileged = false
  volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]

build-image:
  image: docker:27
  # no dind service and no DOCKER_HOST: the CLI uses the mounted Unix socket
  script:
    - echo "$CI_REGISTRY_PASSWORD" | docker login "$CI_REGISTRY" -u "$CI_REGISTRY_USER" --password-stdin
    - docker build --tag "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA" .   # warm cache from earlier jobs on this host
    - docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"

Pitfalls.

• Host root access. Anyone who can call the socket can run docker run --privileged -v /:/host alpine chroot /host, which is root on the host. The job does not need privileged mode to do this. The socket is the privilege.
• Sibling containers and paths. docker run -v "$CI_PROJECT_DIR:/src" ... asks the host daemon to mount a host path. The checkout lives in a runner-managed volume, so the path usually does not exist on the host and an empty directory gets mounted. docker build is not affected, because it sends the build context over the API.
• Shared names and tags. Two concurrent jobs that run docker build -t myapp:test or start a container named db overwrite or collide with each other. Use unique names, for example with $CI_JOB_ID.
• Visibility. docker ps shows the containers of every job on the host, including their environment variables through docker inspect, which may include secrets.
• Disk growth. Images accumulate on the host and need regular pruning (section 6.4).

When it is acceptable. Socket binding fits a runner whose host is dedicated to one trusted project, or where each job gets a fresh VM (for example the Docker Autoscaler or Instance executor with one job per VM, section 6.2), so the "host" is thrown away with the job.

Aspect	Docker-in-Docker	Socket binding
Runner privilege	privileged = true	Not privileged, but socket access is equivalent to root
Isolation between jobs	Separate daemon per job	One daemon shared by all jobs
Layer cache	Empty each job	Persistent on the host
Startup cost	Daemon start per job	None
Volume mounts in docker run	Paths inside the job work as expected	Paths refer to the host
Suitable for shared runners	Only with trusted projects	No

Key takeaways

• Socket binding lets jobs use the host's Docker daemon: fast, cached and without a dind service.
• Access to the socket is root on the host, so never offer it on shared runners.
• Containers started by the job are siblings on the host. Bind-mount paths refer to the host, and names can collide between jobs.
• It is reasonable on single-tenant hosts or ephemeral one-job VMs.

Practice

1. (Beginner) On a test runner with socket binding, run the same docker build twice in two pipelines and compare durations and CACHED lines.
2. (Intermediate) In the same setup, run docker run --rm -v "$CI_PROJECT_DIR:/src" alpine ls /src and explain the output. Then show how a job could list the containers of another running job.
3. Interview question: "A team wants to switch from DinD to socket binding to save two minutes per pipeline. What questions do you ask before agreeing?" Hint: who else uses the runner, how hosts are recycled, and what the socket allows.

Rootless Builds with Buildah

Why it matters. Both Docker-based approaches give the job root-equivalent power over the host. On shared runners and Kubernetes clusters, security teams rarely allow that. Builders that do not need a daemon or privileged mode remove the problem at the source: the build runs as an ordinary process in an ordinary container.

What it is. Buildah is a daemonless tool from the Podman and CRI-O ecosystem that builds OCI and Docker images from a Dockerfile (or Containerfile). It has no long-running daemon: buildah build does the work in the job process itself and writes layers to local storage. It can then push directly to a registry. Two settings let it run without privileges:

Setting	Value for CI	Meaning
BUILDAH_ISOLATION	chroot	Run RUN instructions in a chroot instead of a full container with its own namespaces, which needs fewer privileges
STORAGE_DRIVER	vfs	Store layers as plain directory copies. Slower and larger than overlay, but needs no special kernel mounts
BUILDAH_FORMAT	docker	Produce Docker-format manifests (instead of OCI) when consumers require them, for example for HEALTHCHECK support

A rootless build job.

build-image:
  stage: build
  image: quay.io/buildah/stable            # pin a version tag in real pipelines
  variables:
    STORAGE_DRIVER: vfs
    BUILDAH_ISOLATION: chroot
    BUILDAH_FORMAT: docker
    IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
  before_script:
    - echo "$CI_REGISTRY_PASSWORD" | buildah login -u "$CI_REGISTRY_USER" --password-stdin "$CI_REGISTRY"
  script:
    - buildah build --tag "$IMAGE" --file Dockerfile .
    - buildah push "$IMAGE"

No privileged, no service and no socket. On many Docker and Kubernetes runners this works as-is. Depending on the host's seccomp, AppArmor or SELinux policy, an administrator may need to relax those profiles for the build container (for example with security_opt in [runners.docker] or a pod security context). That is still far less than privileged mode.

Layer caching. With --layers, Buildah keeps intermediate layers and can store them in a registry for the next job:

buildah build --layers \
  --cache-from "$CI_REGISTRY_IMAGE/cache" \
  --cache-to   "$CI_REGISTRY_IMAGE/cache" \
  --tag "$IMAGE" .

Related tools. Podman uses the same storage and build engine as Buildah, with a Docker-compatible CLI (podman build, podman push), so scripts written for Docker often work with alias docker=podman. Kaniko, long the default rootless choice in GitLab documentation, was archived by Google in 2025 and no longer receives fixes. Prefer Buildah or rootless BuildKit (next topic) for new pipelines, and plan to migrate existing Kaniko jobs.

Choosing a build approach.

Approach	Needs privileged or socket	Daemon	Cache across jobs	Speed	Best for
Docker-in-Docker	Privileged	Per job	Registry cache only	Medium	Trusted runners, full Docker CLI and Compose
Socket binding	Socket (root-equivalent)	Host daemon	Persistent on host	Fast	Single-tenant or one-job VMs
Buildah	No	None	Registry cache	Medium (slower with vfs)	Shared runners, OpenShift, Kubernetes
Rootless BuildKit	No (relaxed seccomp/AppArmor)	Per job, daemonless wrapper	Registry cache	Fast	Shared runners wanting BuildKit features
Kaniko	No	None	Registry cache	Medium	Legacy pipelines only (archived)

Key takeaways

• Buildah builds images without a daemon and without privileged mode, which suits shared runners and Kubernetes.
• In CI, use chroot isolation and the vfs storage driver unless the runner supports more efficient options.
• Log in with buildah login, build with buildah build, publish with buildah push.
• --layers with --cache-from and --cache-to adds registry-backed layer caching.
• Kaniko is archived. Prefer Buildah or rootless BuildKit for new work.

Practice

1. (Beginner) Convert a DinD build job from your project to Buildah and run it on a runner without privileged mode.
2. (Intermediate) Measure the same Buildah build with and without --layers --cache-from --cache-to across two pipelines, and explain where the time goes with the vfs driver.
3. Interview question: "Your security team bans privileged containers on all runners. How do you keep building container images, and what do you lose compared with Docker-in-Docker?" Hint: consider rootless builders, Compose-based workflows, and tools that expect a Docker daemon.

BuildKit

Why it matters. The classic Docker builder runs Dockerfile instructions one after another and can only reuse a cache that lives on the same machine. BuildKit, the builder behind docker build since Docker Engine 23 and behind docker buildx, builds independent stages in parallel, can export and import caches through a registry, mounts secrets without leaking them into layers, and builds for several architectures. Most advanced image-build techniques in CI are BuildKit features.

What it is. BuildKit turns a Dockerfile into a graph of steps, runs independent branches concurrently, and skips steps whose inputs have not changed:

Dockerfile (multi-stage)                        BuildKit execution graph
FROM node:22 AS deps      ─┐                    [deps] ──────┐
FROM node:22 AS build     ─┼─ parsed into ─>                 ├─> [build] ─> [final]
FROM nginx:1.27 AS final  ─┘                    [nginx pull] ┘   (deps and the nginx pull run in parallel)

You can use it in three ways in GitLab:

How	Command	Needs
Through Docker (DinD or socket)	docker buildx build	A Docker daemon. The docker:27 image includes the buildx plugin
Rootless, daemonless	buildctl-daemonless.sh build in moby/buildkit:rootless	No privileged mode, relaxed seccomp and AppArmor profiles
Remote builder	buildctl --addr tcp://buildkitd:1234 build	A long-running BuildKit service (for example in Kubernetes) with mTLS. Keeps a warm cache

docker buildx in a DinD job. The default docker driver of buildx has limited cache export and multi-platform support. Creating a builder with the docker-container driver unlocks all features:

build-image:
  image: docker:27
  services: [docker:27-dind]
  variables:
    DOCKER_HOST: tcp://docker:2376
    DOCKER_TLS_CERTDIR: "/certs"
    DOCKER_TLS_VERIFY: "1"
    DOCKER_CERT_PATH: "$DOCKER_TLS_CERTDIR/client"
  before_script:
    - echo "$CI_REGISTRY_PASSWORD" | docker login "$CI_REGISTRY" -u "$CI_REGISTRY_USER" --password-stdin
    - docker buildx create --use --driver docker-container --name ci-builder   # BuildKit in its own container
  script:
    - docker buildx build --tag "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA" --push .   # --push: export straight to the registry

Rootless BuildKit. The moby/buildkit:rootless image runs BuildKit as an unprivileged user. Its ENTRYPOINT starts the daemon, so clear it (section 7.1) and use the daemonless wrapper, which starts a temporary daemon, runs one build and stops:

build-image-rootless:
  image:
    name: moby/buildkit:v0.18.0-rootless
    entrypoint: [""]
  variables:
    BUILDKITD_FLAGS: --oci-worker-no-process-sandbox   # avoid creating a PID namespace, which needs extra privileges
  before_script:
    # buildctl reads registry credentials from the Docker config file
    - mkdir -p ~/.docker
    - |
      AUTH=$(printf '%s:%s' "$CI_REGISTRY_USER" "$CI_REGISTRY_PASSWORD" | base64 | tr -d '\n')
      echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$AUTH\"}}}" > ~/.docker/config.json
  script:
    - buildctl-daemonless.sh build
        --frontend dockerfile.v0
        --local context=.
        --local dockerfile=.
        --output type=image,name=$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA,push=true

Rootless BuildKit relies on user namespaces. The runner's job containers usually need seccomp and apparmor set to unconfined (or a custom profile that allows the required system calls), which the runner administrator configures.

Dockerfile features that need BuildKit. Start the Dockerfile with a syntax line so the latest stable Dockerfile frontend is used:

# syntax=docker/dockerfile:1
FROM node:22-alpine AS deps
WORKDIR /app
COPY package.json package-lock.json ./

# Secret mount: .npmrc with a registry token is available only during this RUN,
# and never stored in any layer or in the image history
RUN --mount=type=secret,id=npmrc,target=/root/.npmrc \
    # Cache mount: npm's download cache persists between builds on the same builder
    --mount=type=cache,target=/root/.npm \
    npm ci

FROM node:22-alpine
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY . .
CMD ["node", "server.js"]

The secret comes from a file-type CI/CD variable (section 5.2), whose value is a path to a temporary file:

script:
  # NPMRC is a File-type variable: $NPMRC is the path of a file holding the .npmrc content
  - docker buildx build --secret id=npmrc,src="$NPMRC" --tag "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA" --push .

Never pass secrets with --build-arg. Build arguments are recorded in the image metadata and can be read by anyone who can pull the image.

Key takeaways

• BuildKit builds a dependency graph from the Dockerfile, runs independent steps in parallel and powers docker buildx.
• In DinD, create a docker-container builder to get full cache export and multi-platform support.
• Rootless BuildKit with buildctl-daemonless.sh avoids privileged mode but needs relaxed seccomp and AppArmor.
• Use RUN --mount=type=secret for credentials and RUN --mount=type=cache for package caches. Never pass secrets as build arguments.

Practice

1. (Beginner) Add # syntax=docker/dockerfile:1 to a Dockerfile and build it with docker buildx build --progress=plain. Identify the steps that ran in parallel.
2. (Intermediate) Move a private registry token from a --build-arg to a secret mount fed by a file-type variable. Use docker history --no-trunc on both images to prove the token is gone from the second one.
3. Interview question: "Explain the difference between a cache mount and the layer cache, and why a cache mount may not help much on ephemeral CI runners." Hint: where each cache is stored, what gets exported to a registry, and what survives when the builder is deleted.

Multi-Architecture Builds

Why it matters. ARM servers (such as AWS Graviton) are often cheaper per unit of performance, developers work on Apple silicon laptops, and edge devices run ARM. An image built only for linux/amd64 either fails on those machines or runs slowly under emulation. A multi-architecture image lets every consumer pull the same name and automatically get the variant that matches its CPU.

How it works. A multi-architecture image is an image index (also called a manifest list): a small document that points to one image manifest per platform. The client picks the entry that matches its platform.

registry.example.com/shop/api:1.4.2   (image index)
 ├── linux/amd64  ─> manifest sha256:aa11...  ─> layers built for x86-64
 └── linux/arm64  ─> manifest sha256:bb22...  ─> layers built for ARM64

docker pull on a Graviton host   -> gets sha256:bb22...
docker pull on an Intel laptop   -> gets sha256:aa11...

Three ways to produce the variants.

Approach	How	Speed	Needs	Best for
Emulation (QEMU)	One builder builds every platform, running foreign binaries through QEMU	Slow for compile-heavy builds (often several times slower)	binfmt_misc handlers registered on the host kernel	Simple images, occasional builds
Native builders	One job per architecture on matching runners, then combine	Native speed, jobs in parallel	Runners of each architecture (for example saas-linux-*-arm64 on GitLab.com)	Large or compiled builds
Cross-compilation	The build stage runs on the native platform and compiles for the target	Native speed in a single job	A toolchain that cross-compiles well (Go, Rust, Java)	Compiled languages

Emulation with buildx.

build-multiarch-qemu:
  image: docker:27
  services: [docker:27-dind]
  variables:
    DOCKER_HOST: tcp://docker:2376
    DOCKER_TLS_CERTDIR: "/certs"
    DOCKER_TLS_VERIFY: "1"
    DOCKER_CERT_PATH: "$DOCKER_TLS_CERTDIR/client"
  before_script:
    - echo "$CI_REGISTRY_PASSWORD" | docker login "$CI_REGISTRY" -u "$CI_REGISTRY_USER" --password-stdin
    # Register QEMU handlers for foreign architectures. This changes the HOST kernel's binfmt_misc settings
    - docker run --privileged --rm tonistiigi/binfmt --install arm64
    - docker buildx create --use --driver docker-container
  script:
    - docker buildx build
        --platform linux/amd64,linux/arm64
        --tag "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"
        --push .                                    # pushes both manifests and the index

Native builders with parallel:matrix. Each architecture builds on its own runner (section 4.5), then a final job combines the results into one index:

stages: [build, publish]

.docker:
  image: docker:27
  services: [docker:27-dind]
  variables:
    DOCKER_HOST: tcp://docker:2376
    DOCKER_TLS_CERTDIR: "/certs"
    DOCKER_TLS_VERIFY: "1"
    DOCKER_CERT_PATH: "$DOCKER_TLS_CERTDIR/client"
  before_script:
    - echo "$CI_REGISTRY_PASSWORD" | docker login "$CI_REGISTRY" -u "$CI_REGISTRY_USER" --password-stdin

build-arch:
  extends: .docker
  stage: build
  parallel:
    matrix:
      - ARCH: [amd64, arm64]
  tags: ["saas-linux-medium-${ARCH}"]               # matrix variables can select the runner
  script:
    - docker build --tag "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA-$ARCH" .
    - docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA-$ARCH"

publish-index:
  extends: .docker
  stage: publish
  needs: [build-arch]
  script:
    # Create an index that references both per-architecture images (no layers are downloaded)
    - docker buildx imagetools create
        --tag "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"
        "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA-amd64"
        "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA-arm64"
    - docker buildx imagetools inspect "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"   # verify both platforms are listed

Cross-compilation in the Dockerfile. BuildKit provides BUILDPLATFORM (where the build runs) and TARGETOS/TARGETARCH (what it builds for). Running the compile stage on $BUILDPLATFORM avoids emulation entirely:

# syntax=docker/dockerfile:1
FROM --platform=$BUILDPLATFORM golang:1.23 AS build   # always runs natively on the builder
ARG TARGETOS TARGETARCH                                # set by buildx for each requested platform
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=$TARGETOS GOARCH=$TARGETARCH go build -o /out/api ./cmd/api

FROM gcr.io/distroless/static-debian12                 # per-platform base, no RUN: nothing to emulate
COPY --from=build /out/api /api
ENTRYPOINT ["/api"]

Built with docker buildx build --platform linux/amd64,linux/arm64 ..., this produces both variants at native speed in one job.

Things to verify. Base images must exist for every target platform. Native dependencies (C extensions, prebuilt binaries downloaded in RUN curl ...) must match the target architecture. Tests should run on each architecture, not only on the build platform.

Key takeaways

• A multi-architecture image is an index pointing to one manifest per platform. Clients pick their own variant automatically.
• QEMU emulation is the simplest approach but slow, and registering binfmt_misc affects the host kernel.
• Native per-architecture jobs combined with docker buildx imagetools create give native speed and parallelism.
• Cross-compiling with FROM --platform=$BUILDPLATFORM and TARGETARCH avoids emulation in a single job.
• Check that base images and downloaded binaries exist for every target platform.

Practice

1. (Beginner) Run docker buildx imagetools inspect node:22-alpine and list the platforms it supports.
2. (Intermediate) Build a Go or Rust service for linux/amd64 and linux/arm64 twice: once with QEMU emulation and once with cross-compilation. Compare build times and verify both variants with imagetools inspect.
3. Interview question: "You need multi-architecture images for a C++ service with a 25-minute native build. Which approach do you choose, and how do you structure the pipeline?" Hint: emulation overhead, runner availability per architecture, parallel jobs and the final index step.

Layer Caching for Image Builds

Why it matters. An image is a stack of layers, and each Dockerfile instruction produces one. If an instruction and everything before it are unchanged, the builder can reuse the stored layer instead of running the step again. Locally, this makes rebuilds take seconds. In CI, every DinD or rootless job starts with an empty builder, so without extra work every pipeline rebuilds everything, including the npm ci or pip install that takes minutes.

How the layer cache decides. Each step's cache key depends on the instruction, its inputs (for COPY, the content of the copied files) and the key of the previous step. The first miss invalidates every step after it:

Step                               Inputs changed?   Result
FROM node:22-alpine                no                CACHED
COPY package.json package-lock.json ./   no          CACHED
RUN npm ci                         no                CACHED      <- the expensive step is skipped
COPY . .                           yes (src edited)  rebuilt
RUN npm run build                  (after a miss)    rebuilt

Step 1: order the Dockerfile for caching. Copy dependency manifests and install dependencies before copying the source code, which changes far more often:

# Bad: any source change invalidates the dependency install
FROM node:22-alpine
WORKDIR /app
COPY . .
RUN npm ci && npm run build

# Good: dependencies are rebuilt only when the lockfile changes
FROM node:22-alpine
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci
COPY . .
RUN npm run build

A .dockerignore keeps irrelevant files (.git, node_modules, test reports) out of the build context, so they cannot invalidate COPY . .:

# .dockerignore
.git
node_modules
coverage
*.log

Step 2: give the empty CI builder a cache to start from.

Cache backend	How	Stores	Notes
Inline cache	--cache-to type=inline, then --cache-from the previous image	Final-stage layers only, embedded in the pushed image	Simple, but intermediate stages of multi-stage builds are not cached
Registry cache	--cache-to type=registry,ref=...,mode=max	All layers of all stages (with mode=max) in a separate cache image	Best general choice. Needs a docker-container builder or rootless BuildKit
Local cache plus GitLab cache:	--cache-to type=local,dest=.buildx-cache and cache that directory (section 8.2)	Layers as files	Upload and download can be slower than the registry. Useful when there is no registry
Persistent builder	Socket binding, or a remote BuildKit service	Builder's own disk	Fastest, with the isolation trade-offs discussed earlier

Registry cache with a branch fallback. Each branch writes its own cache. A new branch has no cache yet, so it also reads the default branch's cache:

build-image:
  image: docker:27
  services: [docker:27-dind]
  variables:
    DOCKER_HOST: tcp://docker:2376
    DOCKER_TLS_CERTDIR: "/certs"
    DOCKER_TLS_VERIFY: "1"
    DOCKER_CERT_PATH: "$DOCKER_TLS_CERTDIR/client"
    CACHE_REPO: $CI_REGISTRY_IMAGE/buildcache
  before_script:
    - echo "$CI_REGISTRY_PASSWORD" | docker login "$CI_REGISTRY" -u "$CI_REGISTRY_USER" --password-stdin
    - docker buildx create --use --driver docker-container
  script:
    - docker buildx build
        --cache-from type=registry,ref=$CACHE_REPO:$CI_COMMIT_REF_SLUG
        --cache-from type=registry,ref=$CACHE_REPO:$CI_DEFAULT_BRANCH
        --cache-to type=registry,ref=$CACHE_REPO:$CI_COMMIT_REF_SLUG,mode=max,image-manifest=true,oci-mediatypes=true
        --tag "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"
        --push .

• mode=max exports layers of every stage, so the deps stage of a multi-stage build is cached too. mode=min (the default) exports only layers of the final image.
• image-manifest=true,oci-mediatypes=true stores the cache as a regular OCI image manifest, which registries accept more reliably than the older cache format.
• Keep cache images in their own repository (buildcache) so cleanup policies can treat them separately (section 7.4).

Measuring the effect. Build with --progress=plain and count the CACHED steps. A well-cached pipeline that only changed source code should rebuild only the steps after COPY . ..

Cache mounts versus layer cache. RUN --mount=type=cache (previous topic) keeps a package manager's download directory between builds on the same builder. Registry cache exports do not include it, so on ephemeral builders a cache mount starts empty in each job. Layer caching through the registry is what helps most in CI. Cache mounts help most on persistent builders.

Key takeaways

• The layer cache reuses a step only if it and every earlier step are unchanged. Order Dockerfiles from least to most frequently changing.
• Use .dockerignore so unrelated files do not invalidate COPY steps.
• Ephemeral CI builders start empty. Import and export a registry cache with --cache-from and --cache-to.
• Use mode=max for multi-stage builds and fall back to the default branch's cache for new branches.
• Cache mounts are not exported with the registry cache and help most on persistent builders.

Practice

1. (Beginner) Reorder a Dockerfile that runs COPY . . before installing dependencies. Build it twice locally, changing one source file in between, and compare the CACHED steps.
2. (Intermediate) Add registry caching with a default-branch fallback to a DinD build job. Record build times for a cold run, a run with no changes, a source-only change, and a lockfile change on a new branch.
3. Interview question: "Image builds in CI take 9 minutes even when only one source file changed. Walk through how you would find and fix the cause." Hint: --progress=plain output, Dockerfile ordering, the build context, cache import and export settings, and mode=min versus mode=max.

<a id="74-gitlab-container-registry"></a>

7.4 GitLab Container Registry

Every GitLab project can have its own container registry, integrated with the project's permissions and with CI/CD credentials that GitLab injects automatically. This subchapter covers pushing images from pipelines, naming and tagging them so that deployments are traceable, keeping storage under control, and caching Docker Hub images with the Dependency Proxy.

Pushing Images from CI

Why it matters. An image built in a pipeline is only useful once it is stored somewhere that deployments, later jobs and developers can pull from. An external registry needs its own accounts, tokens and permission model. The GitLab registry reuses what the project already has: members, roles and the job token. So a build job can push with no secret configured at all.

What GitLab provides. When the registry is enabled for a project (Settings > General > Visibility, project features, permissions > Container registry), every job receives these predefined variables:

Variable	Example value	Meaning
CI_REGISTRY	registry.gitlab.com	Registry host (and port, if any)
CI_REGISTRY_IMAGE	registry.gitlab.com/acme/shop/api	Base image path of this project, always lowercase
CI_REGISTRY_USER	gitlab-ci-token	Username for the job token
CI_REGISTRY_PASSWORD	(the job token)	Valid only while the job runs
CI_DEPLOY_USER, CI_DEPLOY_PASSWORD	gitlab+deploy-token-7, token	Present only if the project has a deploy token named gitlab-deploy-token

The job token acts with the permissions of the user who triggered the pipeline. Pushing requires at least the Developer role in the project.

Image paths. A project can hold several image repositories under its base path:

registry.gitlab.com/acme/shop/api                 $CI_REGISTRY_IMAGE              the main image
registry.gitlab.com/acme/shop/api/worker          $CI_REGISTRY_IMAGE/worker       a second image
registry.gitlab.com/acme/shop/api/ci/tools        $CI_REGISTRY_IMAGE/ci/tools     nested name
registry.gitlab.com/acme/shop/api/buildcache      $CI_REGISTRY_IMAGE/buildcache   build cache (section 7.3)

A complete build-and-push job.

build-image:
  stage: build
  image: docker:27
  services: [docker:27-dind]
  variables:
    DOCKER_HOST: tcp://docker:2376
    DOCKER_TLS_CERTDIR: "/certs"
    DOCKER_TLS_VERIFY: "1"
    DOCKER_CERT_PATH: "$DOCKER_TLS_CERTDIR/client"
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    - if: $CI_COMMIT_TAG
  before_script:
    # --password-stdin keeps the token out of the process list and the shell history
    - echo "$CI_REGISTRY_PASSWORD" | docker login "$CI_REGISTRY" -u "$CI_REGISTRY_USER" --password-stdin
  script:
    - docker build
        --label org.opencontainers.image.revision=$CI_COMMIT_SHA
        --label org.opencontainers.image.source=$CI_PROJECT_URL
        --tag "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA" .
    - docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"

The OCI labels let anyone who pulls the image find the commit and project that produced it.

Pushing to another project's registry. The job token can pull from other projects you have access to, but pushing to another project's registry needs a credential issued by that project: a deploy token with the write_registry scope, or a project or group access token. Store it in a masked, protected variable:

publish-shared-base:
  image: quay.io/buildah/stable
  variables:
    TARGET: registry.gitlab.com/acme/platform/base-images/python
  script:
    # PLATFORM_REGISTRY_USER / PLATFORM_REGISTRY_TOKEN: deploy token of acme/platform/base-images
    - echo "$PLATFORM_REGISTRY_TOKEN" | buildah login -u "$PLATFORM_REGISTRY_USER" --password-stdin "$CI_REGISTRY"
    - buildah build --tag "$TARGET:3.12-$CI_PIPELINE_IID" .
    - buildah push "$TARGET:3.12-$CI_PIPELINE_IID"

Using images outside CI. Developers log in with a personal access token that has the read_registry scope (or write_registry to push), and Kubernetes clusters use a deploy token with read_registry:

# Local login with a personal access token (the token is the password)
echo "$GITLAB_PAT" | docker login registry.gitlab.com -u my-username --password-stdin
docker pull registry.gitlab.com/acme/shop/api:3f9c2a1e

# List repositories and tags through the API
curl --silent --header "PRIVATE-TOKEN: $GITLAB_PAT" \
  "https://gitlab.com/api/v4/projects/12345/registry/repositories?tags=true" \
  | jq -r '.[] | .path as $p | .tags[]? | "\($p):\(.name)"'

The registry UI (Deploy > Container registry) shows every repository, its tags, sizes and digests.

Key takeaways

• CI_REGISTRY, CI_REGISTRY_IMAGE, CI_REGISTRY_USER and CI_REGISTRY_PASSWORD let jobs push to the project registry without configured secrets.
• The job token carries the permissions of the pipeline's user. Pushing needs at least Developer.
• A project can hold several image repositories under $CI_REGISTRY_IMAGE.
• Pushing to another project's registry needs a deploy token or access token with write_registry.
• Add OCI labels for the revision and source, so that every image can be traced back to its commit.

Practice

1. (Beginner) Build and push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA from a pipeline, then pull it on your machine using a personal access token with read_registry.
2. (Intermediate) Create a second image repository ($CI_REGISTRY_IMAGE/worker) built from a different Dockerfile in the same pipeline, and list both repositories with their tags through the API.
3. Interview question: "Why can a job push to its own project's registry without any configured secret, but not to a sibling project's registry?" Hint: consider what the job token represents, who triggered the pipeline, and what deploy tokens are scoped to.

Image Tagging Strategies

Why it matters. A tag is the name people and systems use to ask for an image. If myapp:latest points at a different image every hour, nobody can say which code is running in production, rollbacks become guesswork, and two environments that "run the same tag" may run different code. A deliberate tagging scheme makes every image traceable and every deployment reproducible.

Mutable and immutable references. A tag is a movable pointer, similar to a Git branch. A digest (@sha256:...) is the content hash of the manifest, similar to a commit SHA, and can never point to anything else.

myapp:latest         ──┐   (moved by every main pipeline)
myapp:main           ──┤
myapp:1.4            ──┼──>  sha256:9b1d...   <── myapp@sha256:9b1d... (can never change)
myapp:1.4.2          ──┤
myapp:3f9c2a1e...    ──┘   (commit SHA: one tag per commit, never reused)

Common tags and what they are for.

Tag	Source variable	Mutable	Use
Commit SHA	$CI_COMMIT_SHA or $CI_COMMIT_SHORT_SHA	No (by convention)	The canonical identity of a build. Deploy and test this
Branch slug	$CI_COMMIT_REF_SLUG	Yes	"Latest build of this branch", for review apps and caches
Semantic version	$CI_COMMIT_TAG (for example v1.4.2 becomes 1.4.2)	Should not change	Releases consumed by others
Major and minor version	Derived from the version	Yes	Consumers that want automatic patch updates (1.4, 1)
latest	Default branch	Yes	Convenience for humans. Never deploy it
Pipeline ID	$CI_PIPELINE_IID	No	Readable increasing build numbers

Tags may contain only letters, digits, _, . and -, may not start with . or -, and are limited to 128 characters. Branch names such as feature/Login-Form are not valid tags, which is why CI_COMMIT_REF_SLUG exists: it lowercases the name, replaces other characters with - and shortens it to 63 characters (feature-login-form).

Build once, promote by retagging. Rebuilding an image for each environment risks shipping something different from what was tested. Build once, tag it with the commit SHA, and add further tags to the same manifest later. Retagging in the registry copies only the small manifest; no layers are downloaded or rebuilt:

stages: [build, test, release, deploy]

variables:
  IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

.registry:
  image: docker:27
  services: [docker:27-dind]
  variables:
    DOCKER_HOST: tcp://docker:2376
    DOCKER_TLS_CERTDIR: "/certs"
    DOCKER_TLS_VERIFY: "1"
    DOCKER_CERT_PATH: "$DOCKER_TLS_CERTDIR/client"
  before_script:
    - echo "$CI_REGISTRY_PASSWORD" | docker login "$CI_REGISTRY" -u "$CI_REGISTRY_USER" --password-stdin

build:
  extends: .registry
  stage: build
  script:
    - apk add --no-cache jq               # the docker image is Alpine-based and has no jq
    - docker buildx create --use --driver docker-container
    - docker buildx build --tag "$IMAGE" --metadata-file build.json --push .
    # Record the immutable digest for later jobs (dotenv reports, section 5.3)
    - echo "IMAGE_DIGEST=$(jq -r '."containerimage.digest"' build.json)" > image.env
  artifacts:
    reports:
      dotenv: image.env

tag-main:
  extends: .registry
  stage: release
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  script:
    - docker buildx imagetools create --tag "$CI_REGISTRY_IMAGE:main" --tag "$CI_REGISTRY_IMAGE:latest" "$IMAGE"

tag-release:
  extends: .registry
  stage: release
  rules:
    - if: $CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/
  script:
    - VERSION=${CI_COMMIT_TAG#v}          # v1.4.2 -> 1.4.2
    - MINOR=${VERSION%.*}                 # 1.4.2  -> 1.4
    - MAJOR=${VERSION%%.*}                # 1.4.2  -> 1
    - docker buildx imagetools create
        --tag "$CI_REGISTRY_IMAGE:$VERSION"
        --tag "$CI_REGISTRY_IMAGE:$MINOR"
        --tag "$CI_REGISTRY_IMAGE:$MAJOR"
        "$IMAGE"

Deploy by digest. Deployment jobs should reference the digest, so that the exact tested content is deployed even if someone moves a tag in between:

deploy-production:
  stage: deploy
  image:
    name: bitnami/kubectl:1.31
    entrypoint: [""]                      # the image's ENTRYPOINT is kubectl (section 7.1)
  needs: [build]                          # receives IMAGE_DIGEST from the dotenv report
  script:
    - kubectl set image deployment/api api="$CI_REGISTRY_IMAGE@$IMAGE_DIGEST"

Protecting important tags. Recent GitLab versions let Maintainers define protected container tag rules (Settings > Packages and registries > Container registry): tags matching a pattern, such as v.* or latest, can be pushed or deleted only by users with a minimum role. Use them to stop a feature branch pipeline from overwriting a release tag.

Key takeaways

• Tags are movable pointers. Digests are immutable content hashes.
• Tag every build with the commit SHA. Add branch, version and latest tags as convenience pointers.
• Use CI_COMMIT_REF_SLUG for branch tags, because branch names are often invalid tags.
• Build once and promote by retagging the same manifest. Never rebuild per environment.
• Deploy by digest, and protect release tags from being overwritten.

Practice

1. (Beginner) Push an image tagged with $CI_COMMIT_SHORT_SHA and $CI_COMMIT_REF_SLUG from a branch named feature/Add-Login, and check both tags in the registry UI.
2. (Intermediate) Implement the build-once pipeline above. Create a Git tag v2.3.1 and verify that 2.3.1, 2.3, 2 and the commit SHA tag all report the same digest with docker buildx imagetools inspect.
3. Interview question: "Production is running myapp:latest, and an incident starts. How do you find out which commit is running, and how would you redesign tagging so the question is trivial next time?" Hint: digests in the running pods, OCI labels, SHA tags and deploying by digest.

Cleanup Policies

Why it matters. A pipeline that pushes one image per commit creates thousands of tags a year. Each takes storage, which counts against namespace storage limits on GitLab.com and costs disk or object storage on self-managed instances, and the registry UI becomes hard to navigate. Deleting tags by hand does not scale. Cleanup policies delete old tags automatically according to rules you define once.

What a policy contains. A cleanup policy is set per project in Settings > Packages and registries > Container registry > Cleanup policies:

Setting	API attribute	Options
Enabled	enabled	On or off
Run cleanup	cadence	Every day (1d), week (7d), two weeks (14d), month (1month), three months (3month)
Keep the most recent	keep_n	1, 5, 10, 25, 50 or 100 tags per image repository
Keep tags matching	name_regex_keep	A regular expression
Remove tags older than	older_than	7, 14, 30, 60 or 90 days (7d to 90d)
Remove tags matching	name_regex_delete	A regular expression, for example .* for all tags

How a tag is evaluated. For each image repository, a tag is deleted only if it passes every filter:

All tags of the repository
  │
  ├─ named "latest"?                          -> keep
  ├─ does NOT match "remove tags matching"?   -> keep
  ├─ matches "keep tags matching"?            -> keep   (keep rules win over remove rules)
  ├─ among the N most recent remaining tags?  -> keep
  ├─ younger than "remove tags older than"?   -> keep
  └─ otherwise                                -> DELETE

Regular expressions use RE2 syntax and must match the whole tag name (they are anchored automatically), so main matches only main, not main-old.

A policy that fits the tagging scheme of the previous topic. Keep release versions and long-lived branch pointers forever, keep the 10 most recent other tags, and remove the rest after 14 days:

curl --request PUT \
  --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  --header "Content-Type: application/json" \
  --data '{
    "container_expiration_policy_attributes": {
      "enabled": true,
      "cadence": "1d",
      "keep_n": 10,
      "older_than": "14d",
      "name_regex_delete": ".*",
      "name_regex_keep": "(main|latest|\\d+(\\.\\d+){0,2})"
    }
  }' \
  "https://gitlab.example.com/api/v4/projects/12345"

The keep expression preserves main, latest, 1, 1.4 and 1.4.2. SHA tags and branch slugs are removed once they are both older than 14 days and outside the 10 most recent.

Cleaning up on demand. Policies run on a schedule. Images that belong to something short-lived, such as a review app (section 12.2), can be deleted as soon as it stops. The registry API needs a token with the api scope, such as a project access token stored in a masked variable:

stop-review:
  stage: deploy
  image: alpine:3.20
  environment:
    name: review/$CI_COMMIT_REF_SLUG
    action: stop
  rules:
    - if: $CI_MERGE_REQUEST_IID
      when: manual
  script:
    - apk add --no-cache curl jq
    # Find the ID of the main image repository of this project
    - |
      REPO_ID=$(curl --silent --header "PRIVATE-TOKEN: $REGISTRY_CLEANUP_TOKEN" \
        "$CI_API_V4_URL/projects/$CI_PROJECT_ID/registry/repositories" \
        | jq -r --arg path "$CI_PROJECT_PATH" '.[] | select(.path == $path) | .id')
    # Delete the tag of this review branch
    - |
      curl --silent --fail --request DELETE --header "PRIVATE-TOKEN: $REGISTRY_CLEANUP_TOKEN" \
        "$CI_API_V4_URL/projects/$CI_PROJECT_ID/registry/repositories/$REPO_ID/tags/$CI_COMMIT_REF_SLUG"

The same API offers bulk deletion with DELETE .../registry/repositories/:id/tags and the parameters name_regex_delete, name_regex_keep, keep_n and older_than, which is useful for a one-time cleanup of a large backlog.

Tags versus storage. Deleting a tag removes a pointer. The layers it referenced are reclaimed only when no other tag uses them and registry garbage collection runs. GitLab.com handles this for you. On self-managed instances, the registry metadata database enables automatic online garbage collection. Without it, administrators must run gitlab-ctl registry-garbage-collect periodically, which needs the registry in read-only mode or offline.

Key takeaways

• Cleanup policies delete tags on a schedule, per project, based on name patterns, count and age.
• A tag is deleted only if it matches the remove pattern, does not match the keep pattern, is outside the N most recent and is older than the age limit. latest is never deleted.
• Patterns are RE2 and match the whole tag name.
• Delete short-lived images, such as review app tags, from the job that stops the environment.
• Deleting tags frees storage only after garbage collection.

Practice

1. (Beginner) Enable a cleanup policy on a test project that keeps 5 tags and removes tags older than 7 days. After the next run, compare the tag list with your expectations.
2. (Intermediate) Write a policy through the API that keeps semantic-version tags and main, keeps the 25 most recent tags, and deletes everything else after 30 days. Before enabling it, write down which of your current tags it will delete, then verify after the run.
3. Interview question: "Storage usage did not go down after a cleanup policy deleted 4,000 tags. Why might that be?" Hint: shared layers between tags, garbage collection, untagged manifests, and other repositories such as build caches.

Dependency Proxy

Why it matters. Most pipelines pull public base images (node, python, postgres) from Docker Hub in every job. Docker Hub rate-limits pulls, especially anonymous ones counted per IP address, and many runners share a single outbound IP behind NAT. Pipelines then fail with toomanyrequests. Every pull also travels across the internet, and a Docker Hub outage stops all builds. The Dependency Proxy is a pull-through cache inside GitLab: the first pull fetches the image from Docker Hub, and later pulls are served from GitLab.

How it works.

Job: image ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/node:22-alpine
          │
          ▼
GitLab Dependency Proxy (group acme)
  ├── manifest cached and still current? ── yes ─> serve from GitLab storage
  └── no ─> fetch from Docker Hub ─> store ─> serve
                     ▲
                     └── later pulls of node:22-alpine by any project in "acme" hit the cache

The proxy works at the group level: one cache is shared by all projects in a top-level group. It proxies Docker Hub only. Enable it in the group under Settings > Packages and registries > Dependency Proxy.

Predefined variables.

Variable	Example value	Meaning
CI_DEPENDENCY_PROXY_SERVER	gitlab.example.com:443	Host to log in to
CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX	gitlab.example.com:443/acme/dependency_proxy/containers	Prefix for the top-level group's proxy
CI_DEPENDENCY_PROXY_DIRECT_GROUP_IMAGE_PREFIX	gitlab.example.com:443/acme/backend/dependency_proxy/containers	Prefix for the project's direct parent group
CI_DEPENDENCY_PROXY_USER	gitlab-ci-token	Username
CI_DEPENDENCY_PROXY_PASSWORD	(the job token)	Password

Using it for job images and services. Put the prefix in front of the Docker Hub image name. The runner authenticates to the proxy automatically when it pulls image and services:

default:
  image: ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/node:22-alpine

integration-tests:
  services:
    - name: ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/postgres:16
      alias: db                           # set an alias: the derived hostname would be very long
      variables:
        POSTGRES_PASSWORD: test
  script:
    - npm run test:integration

Using it inside image builds. A FROM line is resolved by the builder, not by the runner, so the builder must log in and the base image must be configurable. Pass the proxied name as a build argument and keep Docker Hub as the default for local builds:

# Dockerfile
ARG BASE_IMAGE=node:22-alpine             # local builds pull from Docker Hub
FROM ${BASE_IMAGE}
WORKDIR /app
COPY . .
CMD ["node", "server.js"]

build-image:
  image: docker:27
  services:
    - name: ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/docker:27-dind
      alias: docker                       # keep the hostname that DOCKER_HOST expects
  variables:
    DOCKER_HOST: tcp://docker:2376
    DOCKER_TLS_CERTDIR: "/certs"
    DOCKER_TLS_VERIFY: "1"
    DOCKER_CERT_PATH: "$DOCKER_TLS_CERTDIR/client"
  before_script:
    # The dind daemon pulls FROM images itself, so it needs its own login to the proxy
    - echo "$CI_DEPENDENCY_PROXY_PASSWORD" | docker login "$CI_DEPENDENCY_PROXY_SERVER" -u "$CI_DEPENDENCY_PROXY_USER" --password-stdin
    - echo "$CI_REGISTRY_PASSWORD" | docker login "$CI_REGISTRY" -u "$CI_REGISTRY_USER" --password-stdin
  script:
    - docker build
        --build-arg BASE_IMAGE=${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/node:22-alpine
        --tag "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA" .
    - docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"

Managing the cache. Cached images use group storage. Enable the cache cleanup policy in the same settings page to purge images that have not been pulled for a number of days (90 by default once enabled). Recent GitLab versions also let group Owners store Docker Hub credentials in the Dependency Proxy settings, so that the proxy's own upstream pulls are authenticated and get higher rate limits.

Limits and alternatives.

Need	Dependency Proxy	Alternative
Docker Hub images for jobs, services and builds	Yes	Registry mirror on the runner host or in dind (--registry-mirror)
Other upstream registries (Quay, GHCR, ECR Public)	No	A separate pull-through cache, such as Harbor or Artifactory, or mirroring images into the GitLab registry
Projects in personal namespaces	No (group feature)	Move the project into a group
Air-gapped instances	No (needs Docker Hub access)	Preload images into the GitLab registry and use pull_policy: never or internal paths

Key takeaways

• The Dependency Proxy is a group-level pull-through cache for Docker Hub that protects pipelines from rate limits and outages.
• Prefix Docker Hub images with CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX. The runner authenticates automatically for image and services.
• Builders that resolve FROM lines (DinD, Buildah, BuildKit) need an explicit login and a configurable base image.
• Use the direct-group prefix only when the cache should be scoped to a subgroup.
• Enable cache cleanup, and use another pull-through cache for registries other than Docker Hub.

Practice

1. (Beginner) Enable the Dependency Proxy for a group, change a job's image to use CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX, run it twice, and find the cached image in the group's Dependency Proxy page.
2. (Intermediate) Convert a DinD build whose Dockerfile starts with FROM python:3.12-slim so that the base image, the docker:dind service and the job image all come through the proxy. Confirm with the job log and the Dependency Proxy page that no layer was pulled directly from Docker Hub.
3. Interview question: "Pipelines across the company fail every afternoon with toomanyrequests from Docker Hub. Describe the short-term fix and the long-term design." Hint: shared NAT IP addresses, the Dependency Proxy versus runner-level mirrors, authenticated upstream pulls, and images that come from registries other than Docker Hub.

---

<a id="8-artifacts-caching-and-releases"></a>