13. Pipeline Optimization

A pipeline that works is only the beginning: if it takes forty minutes, developers stop waiting for it, batch their changes and merge on hope. If it burns through the compute quota by the middle of the month, it stops running at all. This chapter covers how to find where a pipeline spends its time, how to cut that time and its cost without weakening what the pipeline checks, and how to keep a monorepo pipeline fast when one repository holds many services.

Tiers at a glance. Everything in this chapter works on all tiers: GIT_DEPTH, GIT_STRATEGY, workflow:auto_cancel, rules:changes, parent-child pipelines, include:local with wildcards and inputs, the Usage Quotas page and CI/CD analytics. Compute quotas apply to GitLab-hosted runners on GitLab.com, and their size depends on the tier (section 1.2). Setting compute quotas for instance runners on a self-managed instance requires Premium, as do merge trains and required code owner approval, which this chapter mentions in passing.

<a id="131-speed-and-efficiency"></a>

13.1 Speed and Efficiency

This subchapter is about making the developer's wait shorter: measuring where the time goes, cutting the cost of fetching code and images, and designing the pipeline so that a broken change fails in the first minute, not the last.

A job's duration is more than its script. Every job pays a fixed setup cost before the first command runs, and on ephemeral runners it pays that cost again every time:

 job created
    │
    ▼
 ┌────────┐  ┌──────────────────┐  ┌─────────────┐  ┌───────────────┐  ┌────────────────────┐  ┌─────────────┐  ┌────────────────────┐
 │ queued │─►│ prepare executor │─►│ get sources │─►│ restore cache │─►│ download artifacts │─►│ step_script │─►│ save cache, upload │
 └────────┘  │ (image pull)     │  │ (git)       │  │               │  │                    │  │ after_script│  │ artifacts          │
             └──────────────────┘  └─────────────┘  └───────────────┘  └────────────────────┘  └─────────────┘  └────────────────────┘
  runner      13.1 images          13.1 GIT_*,      8.2 caching        8.1, 3.2 needs           the actual work   8.1, 8.2
  capacity                          13.3 sparse

Analyzing Pipeline Duration

Why it matters. "The pipeline is slow" is a feeling, not a diagnosis. A pipeline can be slow because jobs wait for a free runner, because each job spends two minutes pulling a 3 GB image, because stages force fast jobs to wait for slow ones, or because one test suite really takes twenty minutes. Each cause has a different fix, and fixing the wrong one wastes effort. Worse, speeding up a job that is not on the critical path saves compute minutes but not one second of the developer's wait.

Three numbers, three meanings. GitLab reports duration in several ways, and they answer different questions:

Measure	What it counts	Where to see it	What it tells you
Pipeline duration	Time during which at least one job was running	Pipeline page header, Pipelines API duration	How long developers wait for feedback
Queued duration	Time jobs spent waiting for a runner	Pipeline header, job sidebar, API queued_duration	Whether runner capacity is the bottleneck
Total job time	Sum of all job durations	Sum of job duration values	What you pay in compute minutes and runner capacity

Six jobs of five minutes each, all running in parallel, give a pipeline duration of five minutes but a total job time of thirty. Both numbers matter, but for different audiences: developers care about the first, whoever pays the runner bill cares about the third.

The critical path. The pipeline cannot finish before its longest chain of dependent jobs. That chain is the critical path, and only shortening it shortens the pipeline:

             0   2   4   6   8   10  12 min
             |   |   |   |   |   |   |
lint         ████                          2 min
build        ██████                        3 min
unit               ████████                4 min  (needs: build)
e2e                ████████████████        8 min  (needs: build)   <- critical path: build + e2e = 11 min
docs         ██                            1 min

Pipeline duration: 11 min        Total job time: 2 + 3 + 4 + 8 + 1 = 18 min

Halving unit saves 2 compute minutes and zero minutes of waiting. Splitting e2e into two parallel halves of 4 minutes (section 4.5) cuts the pipeline to 7 minutes, because build + unit and build + e2e are now both 7 minutes long. The critical path has moved, and the next optimization must target whichever chain is now the longest.

Where to look in the UI.

View	What it shows
Pipeline page header	Total duration and queued time of the pipeline
Pipeline graph, Group jobs by: Job dependencies with Show dependencies	The needs graph: follow the arrows to find the critical path (section 3.2)
Job page sidebar	The job's duration, queued time and runner
Job log	Collapsible sections (prepare_executor, get_sources, restore_cache, download_artifacts, step_script, archive_cache, upload_artifacts_on_success). Each section header shows how long it took
Job log with FF_TIMESTAMPS: "true"	A timestamp on every log line, useful for spotting slow commands inside script
Analyze > CI/CD analytics	Duration and success trends across many pipelines (section 14.2)

variables:
  FF_TIMESTAMPS: "true"        # runner feature flag: prefix every log line with a timestamp

Measuring with the API. For a precise picture, pull the job data and draw a timeline. The script below uses only the Python standard library:

#!/usr/bin/env python3
# pipeline_timeline.py: print a text timeline of one pipeline's jobs
# Usage: GITLAB_TOKEN=glpat-... python3 pipeline_timeline.py <project_id> <pipeline_id>
import json
import os
import sys
import urllib.request
from datetime import datetime

API = os.environ.get("GITLAB_API", "https://gitlab.com/api/v4")
project_id, pipeline_id = sys.argv[1], sys.argv[2]
SCALE = 15  # seconds per character in the timeline

def get(path):
    req = urllib.request.Request(API + path, headers={"PRIVATE-TOKEN": os.environ["GITLAB_TOKEN"]})
    with urllib.request.urlopen(req) as resp:
        return json.load(resp)

def ts(value):
    return datetime.fromisoformat(value.replace("Z", "+00:00"))  # API returns ISO 8601 in UTC

pipeline = get(f"/projects/{project_id}/pipelines/{pipeline_id}")
# Trigger (bridge) jobs are listed by /bridges, not /jobs, so they are not shown here
jobs = get(f"/projects/{project_id}/pipelines/{pipeline_id}/jobs?per_page=100")
jobs = [j for j in jobs if j["started_at"] and j["finished_at"]]   # skip jobs that never ran
start = min(ts(j["started_at"]) for j in jobs)

print(f"{'job':<24}{'queued':>8}{'run':>8}  timeline (1 char = {SCALE} s)")
for job in sorted(jobs, key=lambda j: j["started_at"]):
    offset = (ts(job["started_at"]) - start).total_seconds()
    bar = " " * int(offset // SCALE) + "#" * max(1, int(job["duration"] // SCALE))
    queued = job["queued_duration"] or 0                           # can be null for old jobs
    print(f"{job['name']:<24}{queued:>7.0f}s{job['duration']:>7.0f}s  {bar}")

print(f"\nPipeline duration: {pipeline['duration'] / 60:.1f} min, "
      f"queued: {(pipeline['queued_duration'] or 0) / 60:.1f} min, "
      f"total job time: {sum(j['duration'] for j in jobs) / 60:.1f} min")

job                       queued     run  timeline (1 char = 15 s)
lint                          4s    118s  #######
build                         3s    181s  ############
unit                         52s    243s              ################
e2e                          49s    482s              ################################
docs                          5s     61s  ####

Pipeline duration: 11.2 min, queued: 1.1 min, total job time: 18.1 min

The 50 seconds of queueing on unit and e2e are a hint in themselves: when build finished, there was no free runner for the next jobs.

From symptom to fix. Once you know which phase is slow, the fix usually follows:

Symptom	Likely cause	Where to look
High queued time	Too few runners or low concurrent/limit	Sections 6.3, 6.4, 13.2
Long prepare_executor ("Pulling docker image")	Large image, cold pulls, distant registry	Optimizing Image Size and Pull Time (this subchapter)
Long get_sources	Deep clone, large repository, submodules, LFS	GIT_DEPTH, GIT_STRATEGY, section 13.3
Long restore_cache / archive_cache	Oversized caches, slow cache storage	Section 8.2
Long download_artifacts	Jobs download artifacts they do not use	dependencies, needs:artifacts (sections 8.1, 3.2)
Fast jobs wait for slow ones in earlier stages	Stage barriers	needs (section 3.2)
Long step_script	The work itself	Parallel jobs and test splitting (4.5), runner sizing (13.2), tool caches (8.2)

Key takeaways

• Distinguish pipeline duration (developer wait), queued duration (runner capacity) and total job time (cost).
• Only the critical path determines pipeline duration. Find it with the dependency view, then shorten it. It moves after every improvement.
• Job log sections and FF_TIMESTAMPS show which phase of a job is slow. Each phase has its own fix.
• Measure before and after every change, ideally over several pipelines, because single runs vary.

Practice

1. (Beginner) Open your slowest recent pipeline. Write down its duration, its queued time and the sum of its job durations. Which job ends last, and which jobs does it depend on?
2. (Intermediate) Run pipeline_timeline.py on the same pipeline, then extend it to print the time each job spent in get_sources and prepare_executor by downloading the job trace (GET /projects/:id/jobs/:job_id/trace) and parsing the section_start and section_end markers.
3. Interview question: "Your team spent a sprint making the unit tests 60% faster, but developers say the pipeline feels just as slow. What happened?" Hint: critical path, queueing, and what the developer actually waits for.

GIT_DEPTH and Shallow Clones

Why it matters. Before running anything, each job fetches the repository. A ten-year-old repository can have hundreds of thousands of commits, and downloading all of that history in every job costs time and network traffic, while most jobs only need the files of one commit. A shallow clone fetches only the most recent commits, which can turn a two-minute checkout into a few seconds.

What it is. GIT_DEPTH sets how many commits the runner fetches, counting back from the commit being built. It maps to git fetch --depth:

Full history (GIT_DEPTH: 0)         Shallow (GIT_DEPTH: 3)

  A ─ B ─ C ─ ... ─ X ─ Y ─ Z           X ─ Y ─ Z
  └────── 120,000 commits ──┘           └── 3 ──┘   older history is simply absent

You can set the depth in two places:

Where	Effect
Settings > CI/CD > General pipelines > Git shallow clone	Project default. New projects use 20. 0 means a full clone. The maximum is 1000
GIT_DEPTH variable (global or per job)	Overrides the project setting

variables:
  GIT_DEPTH: 10                  # most jobs: just the latest commits

semantic-release:
  variables:
    GIT_DEPTH: 0                 # this job reads the full history and all tags
  script:
    - npx semantic-release

What breaks with shallow clones. Shallow clones are fast because history is missing, and some tools need that history:

Tool or command	Why it fails or misbehaves	Fix
git describe, version tools (setuptools-scm, GitVersion, semantic-release)	The latest tag is older than the fetched commits	GIT_DEPTH: 0 for that job
git diff origin/main...HEAD	The merge base is not in the fetched history, and origin/main may not exist at all (next paragraph)	Fetch the target branch with enough depth
SonarQube blame, git log-based changelogs	History or authorship is incomplete	Full or deeper clone for that job only
Retried jobs and long queues with very low depth	The branch moved more than GIT_DEPTH commits, and the job's commit is no longer reachable from the fetched tip	Keep the depth comfortably above the number of pushes that can happen while a job waits

Only the pipeline's ref is fetched. The runner does not fetch every branch. It fetches the refspecs configured for the project, which by default cover just the ref the pipeline runs for. A job on feature/login therefore has no origin/main. When a job needs to compare against the target branch, fetch it explicitly:

changed-files-check:
  variables:
    GIT_DEPTH: 50
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  script:
    # Fetch the target branch with the same depth so that a merge base is likely to exist
    - git fetch --depth=50 origin "$CI_MERGE_REQUEST_TARGET_BRANCH_NAME"
    # Three dots: diff from the merge base, which is what the merge request changes
    - git diff --name-only "origin/$CI_MERGE_REQUEST_TARGET_BRANCH_NAME...HEAD"

If the merge base is still outside both shallow histories, git diff fails with no merge base. Deepen the fetch (git fetch --deepen=200) or, for small repositories, use git fetch --unshallow.

Related settings. GIT_FETCH_EXTRA_FLAGS adds flags to the runner's git fetch. Its default is --prune --quiet, and setting it replaces that default, so repeat those flags if you want to keep them. Submodules have their own depth setting, GIT_SUBMODULE_DEPTH (see the next topic).

Key takeaways

• GIT_DEPTH controls how many commits each job fetches. New projects default to 20, and 0 means full history.
• Use a small global depth and raise it only for jobs that need history: versioning, changelogs, blame-based analysis.
• The runner fetches only the pipeline's ref. Fetch the target branch yourself before diffing against it.
• A depth that is too low makes queued or retried jobs fail when their commit falls outside the fetched history.

Practice

1. (Beginner) Run the same job with GIT_DEPTH: 0 and GIT_DEPTH: 1 on a repository with long history. Compare the duration of the "Getting source from Git repository" section and the output of git rev-list --count HEAD.
2. (Intermediate) Write an MR-only job that lists the files changed in the merge request using git diff against the target branch, with GIT_DEPTH: 10. Make it work on a branch whose merge base is 40 commits behind the tip of main.
3. Interview question: "A release job using git describe --tags works on a developer's laptop but prints a commit hash instead of a version in CI. Why?" Hint: what a shallow clone contains, and whether the tag's commit is in it.

GIT_STRATEGY Options

Why it matters. Shallow clones reduce how much history is fetched. GIT_STRATEGY decides whether the runner fetches anything at all, and whether it can reuse a checkout left by an earlier job. A deploy job that only uses artifacts does not need the source code, and a persistent runner that already has a checkout should not throw it away and download everything again.

What it is. GIT_STRATEGY is a CI/CD variable with these values:

Value	What the runner does	Speed	Use when
clone	Removes the build directory and fetches the repository from scratch	Slowest, always clean	You suspect a corrupted or polluted working copy
fetch	Reuses an existing checkout if there is one: fetches new commits, then cleans with git clean	Fast on persistent runners	Default for most jobs
none	No Git operations. The build directory may contain leftovers from earlier jobs	Fastest	Jobs that need only artifacts, caches or external tools
empty	Like none, but empties the build directory first (recent runner versions)	Fastest, clean	Like none when leftovers could interfere

The project default is set in Settings > CI/CD > General pipelines > Git strategy (new projects use fetch). The variable overrides it per pipeline or per job.

fetch depends on the executor. It only helps when the build directory survives between jobs:

Executor	Does the checkout persist?	Effect of fetch
Shell, SSH, VirtualBox (persistent hosts)	Yes	Large saving: only new commits are downloaded
Docker	Yes, in a volume per runner, project and concurrency slot (unless the runner disables it)	Large saving on long-lived Docker hosts
Kubernetes, Docker Autoscaler, Instance with fresh VMs, GitLab-hosted runners	No: each job starts on an empty pod or VM	Behaves like clone. Reduce the cost with GIT_DEPTH instead

After a fetch, the runner runs git clean with GIT_CLEAN_FLAGS (default -ffdx: remove untracked and ignored files, including nested repositories). Narrow the flags to keep useful leftovers, for example -ffdx -e node_modules/ to keep installed dependencies on a persistent shell runner.

Skipping Git entirely. Jobs that only deploy or publish what an earlier job built do not need the repository:

build:
  stage: build
  script: make dist
  artifacts:
    paths: [dist/]

deploy:
  stage: deploy
  variables:
    GIT_STRATEGY: none           # no clone: artifacts are still downloaded
  needs: [build]
  script:
    - ./dist/deploy --env production   # comes from the artifact, not from the repository
  environment: production

With none or empty, artifacts and caches are still restored, and predefined variables such as CI_COMMIT_SHA are still set. Only the working copy is missing, so any script that calls git or reads repository files fails.

Related variables.

Variable	Purpose
GIT_CHECKOUT: "false"	Fetch, but do not check out files (useful when a job only runs git commands, or performs its own sparse checkout)
GIT_SUBMODULE_STRATEGY	none (default), normal (top-level submodules) or recursive
GIT_SUBMODULE_DEPTH	Shallow depth for submodules
GIT_SUBMODULE_PATHS	Only these submodules, for example libs/a :(exclude)libs/b
GIT_LFS_SKIP_SMUDGE: 1	Do not download Git LFS file contents during checkout. Pull them later with git lfs pull --include=... only where needed
GIT_CLONE_PATH	Custom build directory (the runner must allow it with custom_build_dir)

lint:
  variables:
    GIT_SUBMODULE_STRATEGY: none       # linting the main repo does not need vendored submodules
    GIT_LFS_SKIP_SMUDGE: "1"           # nor the 2 GB of design assets stored in LFS
  script: make lint

integration-tests:
  variables:
    GIT_SUBMODULE_STRATEGY: recursive
    GIT_SUBMODULE_DEPTH: 1             # submodules at their pinned commit, without their history
  script: make integration-test

Key takeaways

• fetch reuses an existing checkout and is the right default, but only helps where build directories persist. On ephemeral runners it acts like clone.
• none and empty skip Git entirely. Use them for jobs that work only with artifacts.
• GIT_CLEAN_FLAGS controls what fetch removes. Narrow it carefully to keep dependency folders on persistent runners.
• Submodules and LFS can cost more than the main repository. Fetch them only in jobs that need them.

Practice

1. (Beginner) Add GIT_STRATEGY: none to a job that only reads artifacts and compare its log with the default: which section disappears, and how much time is saved?
2. (Intermediate) On a self-managed Docker or shell runner, run the same job three times with clone and three times with fetch. Explain the difference in the "Getting source" section, then repeat on a GitLab-hosted runner and explain why the difference vanishes.
3. Interview question: "After switching a shell runner to GIT_STRATEGY: fetch with GIT_CLEAN_FLAGS: none, tests start passing locally but failing in CI at random. What could be happening?" Hint: files left by previous jobs, possibly from other branches.

Fail-Fast Pipeline Design

Why it matters. A pipeline's most valuable output is a failure, delivered as early as possible. If a missing semicolon is reported after 25 minutes of integration tests, the developer has already switched context, and the runners spent 25 minutes on a change that was never going to pass. Fail-fast design orders and connects jobs so that the cheapest checks most likely to fail run first, and so that a failure stops work that no longer matters.

The principle. Order checks by how much each one costs and how likely it is to fail:

Seconds           Minutes                     Tens of minutes
┌────────────┐    ┌─────────────────────┐     ┌──────────────────────────────┐
│ lint       │    │ unit tests          │     │ integration / e2e tests      │
│ format     │──► │ build               │ ──► │ performance, DAST            │
│ type check │    │ fast security scans │     │ deploy to review/staging     │
└────────────┘    └─────────────────────┘     └──────────────────────────────┘
 cheap, catch      moderate cost              expensive: run only when the
 most mistakes                                cheaper checks have passed

Techniques.

Technique	How	Effect
Cheap checks first	Lint and format jobs with needs: []	Results within a minute, in parallel with everything else
Gate expensive jobs	needs: [lint, unit] on e2e jobs	E2E never starts for a change that fails lint
Stop on first failure	workflow:auto_cancel:on_job_failure: all (section 4.4)	One failure cancels the rest of the pipeline
Fail fast inside a job	pytest -x or --maxfail=N, jest --bail, go test -failfast	The job stops at the first failing test
Tight timeouts	Short timeout on jobs that normally take 2 minutes (section 4.3)	A hung job fails in 10 minutes, not 60
Previously failed tests first	pytest --failed-first with .pytest_cache cached	Re-running a broken MR shows the known failure immediately
Tests for changed code first	GitLab's Fail Fast testing template for RSpec (Verify/FailFast.gitlab-ci.yml, Premium), or tools such as jest --changedSince	Likely failures surface before the full suite

workflow:
  auto_cancel:
    on_job_failure: all            # any failure cancels the remaining jobs (section 4.4)

stages: [check, test, e2e]

lint:
  stage: check
  needs: []                        # starts immediately, does not wait for anything
  timeout: 5m                      # normally takes 40 seconds
  script: npm run lint

typecheck:
  stage: check
  needs: []
  timeout: 5m
  script: npx tsc --noEmit

unit:
  stage: test
  needs: []                        # independent of lint: runs in parallel
  script: npx jest --bail          # stop at the first failing test file
  cache:
    key: jest-$CI_COMMIT_REF_SLUG
    paths: [.jest-cache/]

e2e:
  stage: e2e
  needs: [lint, typecheck, unit]   # expensive: only after all cheap checks pass
  script: npx playwright test --max-failures=5   # stop after five failing tests

Choose what "fail" should cancel. on_job_failure: all is right for merge request pipelines, where any failure blocks the merge anyway. On the default branch, you may prefer to see every failure at once, because several people fix them in parallel. Use workflow:rules:auto_cancel to apply it only to MR pipelines:

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      auto_cancel:
        on_job_failure: all        # MRs: stop at the first failure
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      auto_cancel:
        on_job_failure: none       # main: run everything and report every failure

Trade-offs. Fail-fast can hide information: if lint fails and cancels the unit tests, the developer fixes lint, pushes, and only then discovers a failing test. When the cheap checks are very fast, running them in parallel with the unit tests (instead of gating the unit tests on them) gives fast feedback without hiding failures. Reserve gating for genuinely expensive jobs. Jobs with allow_failure: true do not count as failures and never trigger cancellation.

Key takeaways

• Order work from cheap and failure-prone to expensive and stable, and start cheap checks immediately with needs: [].
• Gate expensive jobs with needs on the cheap checks, so a broken change never reaches them.
• on_job_failure: all cancels the rest of the pipeline on the first failure. Scope it with workflow:rules if some branches need a full failure report.
• Test runners have their own fail-fast flags, and short timeouts stop hung jobs early.

Practice

1. (Beginner) Take a stage-based pipeline and add needs: [] to lint and format jobs. Measure how many seconds after the push the first failure is now reported when you push a formatting error.
2. (Intermediate) Configure on_job_failure: all for MR pipelines only. Push a commit that breaks lint and one that breaks a unit test, and record which jobs run, which are canceled and how many compute minutes each pipeline used.
3. Interview question: "A team gated all tests behind lint and complains that developers now need three pushes to find all problems. How would you redesign the pipeline?" Hint: which checks are cheap enough to run in parallel, and which are expensive enough to justify a gate.

Optimizing Image Size and Pull Time

Why it matters. The prepare_executor phase, where the runner pulls the job image and service images, often takes longer than the script itself. On ephemeral runners, such as GitLab-hosted runners, autoscaled VMs or fresh Kubernetes nodes, every pull is cold: a 2 GB image is downloaded and unpacked in every job. Ten jobs per pipeline, a hundred pipelines a day, and image pulls become a large share of both the wait and the bill.

What determines pull time.

pull time  ≈  bytes not already on the host  /  bandwidth to the registry  +  unpack time
                     │                                  │
     smaller images, shared base layers,      registry close to the runners,
     pull_policy if-not-present               Dependency Proxy, no rate limiting

Lever	How	Section
Smaller base	-slim or -alpine variants instead of full images	7.1
Fewer layers to download	One purpose-built image per job type instead of one huge toolbox	7.1
Shared base layers	Build all CI images FROM the same base: its layers are pulled once per host	This topic
Multi-stage builds	Download and compile tools in a builder stage, copy only the binaries	This topic
Clean in the same layer	apt-get install ... && rm -rf /var/lib/apt/lists/* in a single RUN	7.1
Reuse local copies	pull_policy: if-not-present on persistent runners for images pinned by tag or digest	7.1
Nearby, cached registry	Dependency Proxy for Docker Hub images, or the project's container registry	7.4

Find the fat. Measure the image before optimizing it:

# Size of each local image
docker image ls --format '{{.Repository}}:{{.Tag}}  {{.Size}}'

# Size of each layer and the instruction that created it, newest first
docker history --format '{{.Size}}\t{{.CreatedBy}}' registry.example.com/platform/ci-tools:1.3

# Interactive explorer: wasted space per layer, files added and removed
dive registry.example.com/platform/ci-tools:1.3

In the job log, the "Preparing the docker executor" section shows how long the pull took and whether the image was already present ("Using locally found image" versus "Pulling docker image").

Multi-stage CI images. Tools are often downloaded with curl, unpacked with tar or compiled with a toolchain that the jobs themselves never use. A multi-stage build leaves all of that behind:

# Stage 1: download tools. curl and tar stay in this stage only.
FROM alpine:3.20 AS tools
ARG KUBECTL_VERSION=v1.31.1
ARG HELM_VERSION=v3.16.2
RUN apk add --no-cache curl tar \
 && curl -fsSLo /kubectl "https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl" \
 && chmod +x /kubectl \
 && curl -fsSL "https://get.helm.sh/helm-${HELM_VERSION}-linux-amd64.tar.gz" | tar -xz -C / linux-amd64/helm

# Stage 2: the image jobs pull. Only runtime needs and the two binaries.
FROM alpine:3.20
RUN apk add --no-cache bash git ca-certificates
COPY --from=tools /kubectl /usr/local/bin/kubectl
COPY --from=tools /linux-amd64/helm /usr/local/bin/helm

Pull through the Dependency Proxy. Pulling public images straight from Docker Hub is subject to its rate limits and to the speed of the internet link. The group's Dependency Proxy caches them inside GitLab. The runner authenticates to it automatically:

default:
  # Pulled from the group's Dependency Proxy cache instead of Docker Hub
  image: ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/node:22-alpine
  services:
    - name: ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/postgres:16-alpine
      alias: db

Kubernetes specifics. On the Kubernetes executor, images are pulled by the node's container runtime, so the node's image cache matters. Set the runner's pull_policy to if-not-present for pinned images, keep CI nodes long-lived enough for their cache to pay off, and remember that a node scaled up for a burst of jobs starts with an empty cache. Pulls also count against poll_timeout (section 6.2).

Key takeaways

• Image pull time is part of every job's duration, and it is paid in full on ephemeral runners.
• Measure with docker history or dive and with the prepare_executor section before optimizing.
• Use slim bases, multi-stage builds, shared base layers and one image per job type to keep images small.
• Pull through the Dependency Proxy or a nearby registry, and reuse local copies with if-not-present where the runner persists.

Practice

1. (Beginner) Use docker history on the largest image your pipeline uses and identify the three biggest layers. Which Dockerfile instructions created them?
2. (Intermediate) Rewrite a CI image that downloads tools with curl into a multi-stage build. Compare the compressed size in the container registry and the pull time in the job log before and after, on a GitLab-hosted runner.
3. Interview question: "Pipelines on your Kubernetes runners are fast in the afternoon but slow every morning. Image pulls account for most of the difference. What is going on, and what would you do?" Hint: overnight scale-down, empty node image caches, and pre-pulling or keeping a minimum node pool.

<a id="132-cost-and-resource-management"></a>

13.2 Cost and Resource Management

Speed is one side of optimization and cost is the other. This subchapter covers how GitLab meters CI usage, how to choose machine sizes that are both fast and economical, how to give jobs the CPU and memory they need without wasting a cluster, and how to stop paying for pipelines whose results nobody uses.

Compute Minutes and Quotas

Why it matters. On GitLab.com, jobs on GitLab-hosted runners consume compute minutes from a monthly quota. When the quota runs out, those jobs stop running, and the whole team is blocked until the next month or until someone buys more. Understanding how minutes are counted is the basis for every cost decision in this subchapter.

What it is. Compute usage is metered per job:

compute minutes  =  job duration (minutes)  x  cost factor of the runner

Element	Details
Who consumes	Jobs on GitLab-hosted (instance) runners. Jobs on your own project or group runners consume nothing
Whose quota	The top-level namespace (top-level group or personal namespace). All projects under it share one quota
Monthly quota	Depends on the tier: 400 (Free), 10,000 (Premium), 50,000 (Ultimate) on GitLab.com at the time of writing (section 1.2)
Cost factor	Grows with machine size: 1 for small Linux, up to 12 for 2xlarge (section 6.1). Some programs, such as GitLab for Open Source, get reduced factors
Reset	The monthly quota resets on the first of each month. Unused minutes do not roll over
Additional minutes	Can be purchased. They are used only after the monthly quota is exhausted and are not reset monthly

A worked example. A Premium group with 20 developers:

Item	Value
Pipelines per developer per working day	12
Working days per month	22
Jobs per pipeline	6 on small (factor 1), averaging 4 minutes; 1 on large (factor 3), 6 minutes
Compute minutes per pipeline	6 x 4 x 1 + 1 x 6 x 3 = 24 + 18 = 42
Monthly usage	20 x 12 x 22 x 42 = 221,760
Included quota	10,000

The included quota covers less than 5% of the usage. The options are to buy minutes, to add self-managed runners for the bulk of the work (section 6.4), or to cut the minutes per pipeline with the techniques in this chapter. Usually it is a mix of all three.

What happens when the quota runs out. GitLab notifies namespace owners by email and with in-app banners as the quota runs low. Once it is exhausted, new jobs that need instance runners fail with the reason ci_quota_exceeded, and running jobs may be stopped. Jobs on project and group runners are not affected, which is why many teams keep at least one self-managed runner as a fallback for critical pipelines.

Monitoring usage. Settings > Usage Quotas > Pipelines on the top-level group shows the current month's usage, a monthly history and a per-project breakdown. The same data is available through GraphQL, which is useful for chargeback reports or alerts:

# Monthly compute usage of a group and its projects
curl --silent --request POST "https://gitlab.com/api/graphql" \
  --header "Authorization: Bearer $GITLAB_TOKEN" \
  --header "Content-Type: application/json" \
  --data '{"query": "query { ciMinutesUsage(namespaceId: \"gid://gitlab/Group/1234\") { nodes { month minutes sharedRunnersDuration projects { nodes { name minutes } } } } }"}' \
  | jq '.data.ciMinutesUsage.nodes[] | {month, minutes, projects: [.projects.nodes[] | {name, minutes}]}'

minutes is the compute usage after the cost factor. sharedRunnersDuration is the raw runtime in seconds. A big gap between the two points at expensive machine types.

Self-managed instances. Self-managed GitLab has no included quota, because the runners are your own. Administrators can still set a compute quota for instance runners per namespace (Admin > Settings > CI/CD > Continuous Integration and Deployment > Compute quota, Premium), and set each instance runner's cost factor for public and private projects. This is the usual way to share one central runner fleet fairly between departments.

Levers to reduce consumption.

Lever	Typical saving	Topic
Cancel redundant pipelines and avoid duplicates	10% to 40% in active repositories	Reducing Redundant Pipeline Runs
Run only jobs affected by a change	Large in monorepos	13.3
Right-size machines	Can lower both time and cost	Runner Sizing and Machine Types
Shorter setup (images, clones, caches)	Seconds to minutes per job	13.1, 8.2
Move steady heavy load to own runners	Removes it from the quota entirely	6.4

Key takeaways

• Compute minutes are job duration times the runner's cost factor, counted only on GitLab-hosted runners.
• The quota belongs to the top-level namespace and resets monthly. Purchased minutes are consumed after it runs out.
• Exhausted quotas make instance-runner jobs fail with ci_quota_exceeded. Own runners keep working.
• Track usage per project in Usage Quotas or through GraphQL, and act before the end of the month.

Practice

1. (Beginner) Open Usage Quotas > Pipelines for your top-level group (or personal namespace). Identify the three projects with the highest usage this month and their most frequent pipeline sources.
2. (Intermediate) Write a scheduled job that queries ciMinutesUsage and fails (or sends a message) when the current month's usage exceeds a pro-rated budget, for example 50% of the quota before the 10th of the month.
3. Interview question: "Halfway through the month, your group's compute quota is 90% used. What do you do today, this week and this quarter?" Hint: immediate fallback runners or purchased minutes, then the biggest consumers, then structural changes.

Runner Sizing and Machine Types

Why it matters. Machine size is the one cost decision that can make a pipeline both faster and cheaper. A bigger runner costs more per minute, but if the job uses the extra cores it finishes in proportionally fewer minutes. A wrong choice in either direction wastes money: a large runner for a single-threaded lint job, or a small runner for a parallel test suite that would finish four times faster on a bigger one.

The break-even rule. On GitLab-hosted runners, the cost of a job is duration x cost factor. Moving a job from size A to size B saves money only if it speeds up more than the cost factor grows:

cheaper on B   ⇔   duration_B / duration_A   <   factor_A / factor_B

small (1) → medium (2):   must finish in less than 1/2 of the time
medium (2) → large (3):   must finish in less than 2/3 of the time
large (3) → xlarge (6):   must finish in less than 1/2 of the time

Whether a job speeds up depends on what limits it:

Job type	Bound by	Benefits from more cores?
Lint, format, small scripts	Single thread, startup time	Rarely
Compilation (C/C++, Rust, Go, Java)	CPU, parallel	Yes, if the build tool runs in parallel
Test suites with parallel workers	CPU and memory	Yes, up to the number of workers
Docker image builds	CPU, network, disk	Partly
Deploys, API calls, waiting on external systems	Network or remote services	No
Large in-memory workloads (bundlers, JVM builds, ML)	Memory	Yes, through the extra memory rather than the cores

Measure, do not guess. Run the same job on every size with a matrix:

size-benchmark:
  parallel:
    matrix:
      - SIZE: [small, medium, large, xlarge]
  tags: ["saas-linux-${SIZE}-amd64"]          # matrix variable selects the machine
  image: node:22
  script:
    - echo "CPUs visible to the job: $(nproc)"
    - npm ci
    - npx jest --maxWorkers="$(nproc)"         # make the test runner use every core
  rules:
    - if: $CI_PIPELINE_SOURCE == "web"         # only when started manually from the UI

A typical result for a CPU-heavy suite:

Size	Cost factor	Duration	Compute minutes	Verdict
small	1	20 min	20	Slow
medium	2	11 min	22	Faster, slightly more expensive
large	3	6 min	18	Fastest per minute spent: choose this
xlarge	6	5 min	30	Diminishing returns: the suite cannot use 16 cores

Make the tool use the machine. A bigger runner helps only if the tool runs in parallel. Many tools need to be told:

Tool	Parallelism setting
make	make -j"$(nproc)"
pytest (with pytest-xdist)	pytest -n auto
Jest	--maxWorkers=N
Gradle	--parallel, org.gradle.workers.max
Maven	-T 1C (one thread per core)
Cargo	Parallel by default. -j N to cap

Inside containers, the number of CPUs a tool detects can be wrong. nproc and Node's os.cpus() report the CPUs the process may be scheduled on, not a CPU limit enforced by quota. On a Kubernetes node with 32 cores and a job limit of 2 CPUs, the tool may start 32 workers and spend its time throttled (next topic). Pass explicit worker counts in that case.

Self-managed fleets. The same reasoning applies to your own runners, with more choices:

Option	When it fits
Separate runners per size, selected by tags (ci-small, ci-large)	Mixed workloads: lint on cheap machines, builds on big ones
Spot or preemptible instances	interruptible jobs that can tolerate being stopped. Not for deploys
arm64 instances	Often cheaper per core. Requires arm64-compatible images
Larger instances running several concurrent jobs	Shared caches and images on the host, fewer cold starts

Key takeaways

• A bigger runner is cheaper per job when the job's speed-up exceeds the cost factor increase.
• Benchmark each heavy job on several sizes with parallel:matrix and tags, and choose by compute minutes and duration together.
• More cores help only if the tool runs in parallel. Configure worker counts explicitly, especially inside containers.
• Keep light jobs on small runners: most lint and deploy jobs gain nothing from size.

Practice

1. (Beginner) Run your slowest test job on small and large GitLab-hosted runners. Calculate the compute minutes of each and decide which size it should use.
2. (Intermediate) Use the size-benchmark matrix for your test suite. Plot duration and compute minutes per size, find the size with the lowest compute minutes, and explain where the speed-up stops.
3. Interview question: "Moving the build from medium to xlarge made it only 10% faster. What would you check before concluding the build cannot benefit from more cores?" Hint: the build tool's parallelism setting, detected CPU count, memory limits, and network-bound steps such as dependency downloads.

Job Resource Requests and Limits

Why it matters. On self-managed runners, many jobs share the same hosts. Without limits, one job's memory-hungry build can starve the neighbors or crash the node. With limits set too low, jobs are killed or throttled. With requests set too high, a cluster sized for 50 parallel jobs runs 10. Requests and limits decide how densely jobs are packed and how they behave under pressure.

Requests versus limits. Both concepts come from Kubernetes, and the Docker executor has equivalent settings:

	Request	Limit
Meaning	What the job is guaranteed, and what the scheduler reserves	The most the job may use
CPU	Determines how many jobs fit on a node	Usage above it is throttled: the job runs slower
Memory	Determines how many jobs fit on a node	Usage above it gets the process killed (OOMKilled, exit code 137)
Set too low	Overcommitted nodes, noisy neighbors	Throttled or killed jobs
Set too high	Wasted capacity, pods stuck in Pending	Little protection for neighbors

Node: 8 CPU, 32 GiB         requests: 2 CPU / 4 GiB per job
┌──────────┬──────────┬──────────┬──────────┐
│  job A   │  job B   │  job C   │  job D   │   4 jobs fit by CPU (8 / 2)
│ 2 CPU    │ 2 CPU    │ 2 CPU    │ 2 CPU    │   memory is only half used (16 of 32 GiB)
│ 4 GiB    │ 4 GiB    │ 4 GiB    │ 4 GiB    │   → requests do not match the node shape
└──────────┴──────────┴──────────┴──────────┘

Kubernetes executor. The runner's config.toml sets defaults and the maximum that jobs may request (section 6.2). Jobs override them with variables:

webpack-build:
  image: node:22
  variables:
    KUBERNETES_CPU_REQUEST: "2"                 # reserved: the scheduler places the pod where 2 CPUs are free
    KUBERNETES_CPU_LIMIT: "4"                   # may burst to 4 CPUs, throttled above
    KUBERNETES_MEMORY_REQUEST: "4Gi"
    KUBERNETES_MEMORY_LIMIT: "6Gi"              # above this the build container is OOM-killed
    KUBERNETES_EPHEMERAL_STORAGE_REQUEST: "10Gi"  # node_modules and build output need disk too
    NODE_OPTIONS: "--max-old-space-size=5120"   # keep Node's heap below the container limit
  script:
    - npm ci
    - npx webpack --mode production

Variable family	Applies to	Capped by (config.toml)
KUBERNETES_CPU_*, KUBERNETES_MEMORY_*	Build container	cpu_limit_overwrite_max_allowed, memory_request_overwrite_max_allowed, ...
KUBERNETES_SERVICE_CPU_*, KUBERNETES_SERVICE_MEMORY_*	Service containers	service_*_overwrite_max_allowed
KUBERNETES_HELPER_CPU_*, KUBERNETES_HELPER_MEMORY_*	Helper container (Git, cache, artifacts)	helper_*_overwrite_max_allowed
KUBERNETES_EPHEMERAL_STORAGE_*	Local disk of the build container	ephemeral_storage_*_overwrite_max_allowed

If a job asks for more than the maximum, or the maximum is not configured, the override is refused. Remember that the pod's total is the sum of the build, helper and service containers.

Docker executor. Limits live only in the runner's config.toml. Jobs cannot change them from .gitlab-ci.yml, so offer different sizes as different runners with different tags:

[[runners]]
  name = "docker-large"                  # registered with the tag "ci-large"
  executor = "docker"
  [runners.docker]
    image = "alpine:3.20"
    cpus = "4"                           # build container may use at most 4 CPUs
    memory = "8g"                        # hard limit: exceeding it kills the process (exit 137)
    memory_swap = "8g"                   # same as memory: no swap, so jobs fail instead of crawling
    shm_size = 536870912                 # 512 MiB /dev/shm for headless browsers (Docker default: 64 MiB)
    service_cpus = "1"                   # limits for each service container
    service_memory = "2g"

Reading the symptoms.

Symptom	Likely cause	Fix
exit code 137, OOMKilled in kubectl describe pod	Memory limit too low	Raise the limit, or cap the tool's memory (-Xmx, --max-old-space-size)
Job much slower on the cluster than on a laptop	CPU throttling: more threads than the CPU limit	Raise the limit or set worker counts to match it
Pod Pending until poll_timeout	No node has room for the requests	Lower requests, add larger nodes, or check autoscaling (section 6.4)
Pod evicted for disk pressure	Ephemeral storage not requested	Set KUBERNETES_EPHEMERAL_STORAGE_REQUEST
Chrome or Playwright crashes in Docker	/dev/shm too small	Raise shm_size or use --disable-dev-shm-usage

Size requests from measurements, not guesses: kubectl top pod during a job, or container metrics in Prometheus (section 6.4), over several typical runs. A common starting point is a request near the job's typical usage and a memory limit 25% to 50% above its peak.

Key takeaways

• Requests decide placement and density. Limits decide what happens under pressure: CPU is throttled, memory is killed.
• On Kubernetes, jobs set resources with KUBERNETES_* variables up to the runner's *_overwrite_max_allowed values. Build, helper and service containers are sized separately.
• On Docker, limits are per runner in config.toml. Expose sizes as tagged runners.
• Keep tool settings (heap sizes, worker counts) consistent with container limits.

Practice

1. (Beginner) Run a job that allocates more memory than its limit (for example python -c "x = bytearray(3 * 1024**3)" with a 2Gi limit) and find the evidence of the OOM kill in the job log and, on Kubernetes, in kubectl describe pod.
2. (Intermediate) Measure the CPU and memory usage of your three heaviest jobs with kubectl top pod across five runs each. Propose requests and limits for each, and calculate how many of each job fit on your node type.
3. Interview question: "Your CI cluster has plenty of free CPU according to the dashboards, but jobs sit in Pending. What could cause this?" Hint: requests versus actual usage, memory or ephemeral storage requests, node selectors and taints.

Reducing Redundant Pipeline Runs

Why it matters. The cheapest pipeline is the one that never runs. In active repositories, a large share of pipelines produce results nobody looks at: a branch pipeline and an MR pipeline for the same commit, five pipelines for five quick pushes where only the last one matters, a full test run for a typo in a README, or a nightly schedule that tests the same commit every night for a week. Removing these saves compute minutes and frees runners for the pipelines that matter.

Sources of redundancy and their fixes.

Redundancy	Fix	Section
Branch and MR pipeline for the same push	workflow:rules that pick one pipeline type	4.4
Older pipelines still running after a new push	Auto-cancel plus interruptible: true	4.3, 4.4
Full pipeline for documentation-only changes	rules:changes on code jobs	4.1
Expensive jobs on draft merge requests	Make them manual for drafts	This topic
Pushes that need no CI at all	[skip ci] in the commit message or git push -o ci.skip	This topic
Release tag pipeline rebuilding what main already built	Promote the tested artifact or image instead of rebuilding	This topic
Schedules testing an unchanged commit	Skip early when nothing changed	This topic

A lean MR pipeline. The configuration below combines the main techniques. It creates one pipeline per push, cancels outdated work, skips code jobs for documentation changes and makes the end-to-end tests manual while the MR is a draft:

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"          # MR pipelines for MRs
    - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
      when: never                                               # no duplicate branch pipeline
    - if: $CI_COMMIT_BRANCH                                     # branches without an MR
    - if: $CI_COMMIT_TAG
  auto_cancel:
    on_new_commit: interruptible                                # cancel outdated interruptible jobs

default:
  interruptible: true                                           # safe default; deploy jobs opt out

.code-changes: &code-changes
  - src/**/*
  - package.json
  - package-lock.json
  - .gitlab-ci.yml

unit:
  script: npm test
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      changes: *code-changes                                    # docs-only MRs skip this job
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

e2e:
  script: npx playwright test
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_TITLE =~ /^(\[Draft\]|\(Draft\)|Draft:)/
      when: manual                                              # drafts: run on demand only
      allow_failure: true                                       # do not block the pipeline status
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      changes: *code-changes
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

Skipping CI on purpose. For pushes that need no pipeline, such as a WIP backup or a bulk branch cleanup, there are two ways to skip CI:

git commit -m "Update team contacts [skip ci]"     # [ci skip] also works
git push -o ci.skip                                # push option: no commit message change needed

If the project requires a successful pipeline before merging, enable Settings > Merge requests > Skipped pipelines are considered successful, or skipped MRs cannot be merged.

Build once, promote many times. A common hidden duplicate: main builds and tests an image, then a release tag on the same commit builds it again. Instead, the tag pipeline can retag the image that main already produced:

release-image:
  image:
    name: gcr.io/go-containerregistry/crane:debug      # debug variant includes a shell
    entrypoint: [""]                                   # clear the crane entrypoint (section 7.1)
  variables:
    GIT_STRATEGY: none                                  # nothing from the repository is needed
  rules:
    - if: $CI_COMMIT_TAG
  script:
    - crane auth login "$CI_REGISTRY" -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
    # The image built and tested on main for this exact commit gets the release tag: no rebuild
    - crane tag "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA" "$CI_COMMIT_TAG"

This saves the build minutes, and it also guarantees that the released image is the one that was tested.

Scheduled pipelines. A nightly schedule that runs a two-hour suite against an unchanged commit wastes two hours. A first job can compare the commit with the one tested by the previous scheduled pipeline and stop early. Also review the schedule list regularly: schedules created for one-off experiments tend to stay active indefinitely.

Key takeaways

• One pipeline per push: use workflow:rules to avoid branch and MR duplicates, and auto-cancel with interruptible to stop outdated work.
• Use rules:changes so documentation and unrelated changes skip expensive jobs, and make heavy jobs manual on draft MRs.
• [skip ci] and -o ci.skip skip pipelines intentionally. Allow skipped pipelines to count as successful if merges require a pipeline.
• Build artifacts and images once and promote them, instead of rebuilding the same commit for tags or environments.

Practice

1. (Beginner) Count the pipelines created in your project in the last week by source (Build > Pipelines, filter by source or use the API). How many were canceled, skipped or duplicates of the same commit?
2. (Intermediate) Apply the lean MR pipeline configuration to a project. Push a documentation-only change, a draft MR and three quick consecutive commits, and compare the compute minutes used with the original configuration.
3. Interview question: "Why is rules:changes alone not enough to skip a pipeline for documentation-only changes on a brand-new branch?" Hint: what changes is compared against when no previous push exists, and compare_to.

<a id="133-monorepo-strategies"></a>

13.3 Monorepo Strategies

A monorepo keeps many services and libraries in one repository. That makes atomic cross-service changes easy, but a naive pipeline builds and tests everything on every commit. This subchapter covers how to run only what a change affects, how to give each service its own pipeline, how to share configuration without chaos, and how to fetch only the part of a huge repository that a job needs.

monorepo/                                   change to services/api/handler.py
├── libs/common/      ◄── used by api, worker
├── services/api/     ◄── changed                 ─► run: api pipeline
├── services/web/                                 ─► skip: web
├── services/worker/                              ─► skip: worker (unless libs/common changed)
└── docs/                                         ─► skip: docs

Path-Based Job Triggering

Why it matters. In a monorepo with ten services, a change to one service's handler should not rebuild and retest the other nine. Path-based triggering adds jobs to the pipeline only when files they depend on changed. It is the simplest and most common monorepo optimization, and it often cuts pipeline cost by an order of magnitude.

What it is. rules:changes (section 4.1) matches when at least one changed file matches at least one pattern. In a monorepo, each service's jobs list the paths that affect that service: its own folder, the shared libraries it uses and the CI configuration that defines it.

# Hidden job templates: one rule set per service, reused by all of that service's jobs
.api-rules:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      changes:
        - services/api/**/*
        - libs/common/**/*                  # shared library: its consumers must be retested
        - ci/templates/python.yml           # the template that defines these jobs
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      changes:                              # main: only if this push changed the service
        - services/api/**/*
        - libs/common/**/*

.web-rules:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      changes:
        - services/web/**/*
        - ci/templates/node.yml
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH     # main: always run, as a safety net

api-test:
  extends: .api-rules
  image: python:3.12-slim
  script:
    - cd services/api
    - pip install -r requirements.txt
    - pytest

web-test:
  extends: .web-rules
  image: node:22-alpine
  script:
    - cd services/web
    - npm ci
    - npm test

What "changed" means on each pipeline type. The comparison base is the main source of surprises:

Pipeline	Compared against	Consequence in a monorepo
Merge request	The MR's target branch	Exactly the files the MR changes. The ideal case
Push to an existing branch	The branch's previous tip	Only the latest push, not the whole branch: a job can be skipped although an earlier push in the same branch changed its files
New branch, tag, schedule, API, manual run	No base: changes is always true	Everything runs
With compare_to: refs/heads/main	main	Branch pipelines behave like MR pipelines

Two policies for the default branch. compare_to: refs/heads/main is useless on main itself, because comparing main with main finds no changes. The example above shows the two sensible choices:

1. Run everything on main (as .web-rules does). Simple and safe: the default branch is fully tested after every merge, which catches anything the path rules missed. Costs more.
2. Run only changed services on main (as .api-rules does), using plain changes, which compares with the previous push. Cheaper, but a dependency missing from the path list is never caught by a full run. Pair it with a scheduled full pipeline, for example nightly.

The dependency problem. Path rules are only as correct as the dependencies they encode. If services/worker starts importing libs/common and nobody updates .worker-rules, a breaking change in the library passes CI and fails in production. Mitigations:

Approach	How it keeps rules correct
Include shared paths generously	Over-triggering costs minutes. Under-triggering costs incidents
Full pipeline on the default branch or nightly	Catches missed dependencies within hours
Generate rules from the build graph	Tools such as Nx (nx affected), Turborepo (turbo run test --filter=...[origin/main]), Bazel or Pants know the real dependencies. Use them in a generated child pipeline (section 3.3)
Code owners on ci/ and rule templates	Changes to trigger rules get reviewed by the platform team

Key takeaways

• rules:changes with per-service path lists runs only the affected services' jobs. Put shared libraries and CI files in every dependent list.
• MR pipelines compare with the target branch, push pipelines with the previous push, and new branches, tags and schedules match everything.
• Decide deliberately what the default branch runs: everything, or changed services plus a regular full run.
• Hand-written path lists drift from real dependencies. Keep a safety net, or derive the lists from a build tool's dependency graph.

Practice

1. (Beginner) In a repository with two folders, frontend/ and backend/, create one test job per folder with rules:changes. Open MRs that change only one folder, both folders, and only README.md, and record which jobs run each time.
2. (Intermediate) Add a libs/shared/ folder used by both services. Update the rules so that a library change runs both test jobs, and add a nightly schedule that runs every job regardless of changes.
3. Interview question: "Your monorepo uses path-based rules, and a library change broke a service in production although CI was green. How did that happen, and how would you prevent it structurally?" Hint: the rules encode a dependency graph by hand. Where can the real graph come from?

Per-Service Child Pipelines

Why it matters. Path rules on individual jobs work until the root .gitlab-ci.yml holds 150 jobs for 15 services. Then every job name must be unique across the repository, every team edits the same file, all services share one stage list, and the pipeline graph is unreadable. Giving each service its own child pipeline (section 3.3) isolates its jobs, its stages and its status, while the parent decides which children to start.

What it is. The parent pipeline contains one trigger job per service. Each trigger job uses rules:changes to decide whether the service is affected, and starts a child pipeline from a configuration file:

parent pipeline
├── api      (trigger, changes: services/api/**, libs/common/**)  ─►  child: build ─► test ─► package
├── web      (trigger, changes: services/web/**)                  ─►  not created
└── worker   (trigger, changes: services/worker/**, libs/common/**) ─► child: build ─► test ─► package

When the services are built the same way, one child template with inputs (section 9.3) serves them all. Inputs are interpolated when the child configuration is loaded, so they can be used anywhere in the file, including job names and changes paths:

# ci/service-pipeline.yml: child pipeline template shared by all Python services
spec:
  inputs:
    service:
      description: Folder name under services/
    python_version:
      default: "3.12"
---
stages: [build, test, package]

default:
  image: python:$[[ inputs.python_version ]]-slim
  interruptible: true

build:
  stage: build
  script:
    - cd services/$[[ inputs.service ]]
    - pip wheel --wheel-dir dist .
  artifacts:
    paths:
      - services/$[[ inputs.service ]]/dist/   # block style: brackets in a [flow] list would break YAML

test:
  stage: test
  needs: [build]
  script:
    - cd services/$[[ inputs.service ]]
    - pip install dist/*.whl pytest
    - pytest

package:
  stage: package
  needs: [test]
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH           # images only from main
  image: docker:27
  services: [docker:27-dind]
  script:
    - docker build -t "$CI_REGISTRY_IMAGE/$[[ inputs.service ]]:$CI_COMMIT_SHA" services/$[[ inputs.service ]]

# .gitlab-ci.yml (parent)
stages: [services]

.service-trigger:
  stage: services
  trigger:
    strategy: depend                    # the parent's status reflects the child's (section 3.3)

api:
  extends: .service-trigger
  trigger:
    include:
      - local: ci/service-pipeline.yml
        inputs:
          service: api
  rules:
    - changes: [services/api/**/*, libs/common/**/*, ci/service-pipeline.yml]

worker:
  extends: .service-trigger
  trigger:
    include:
      - local: ci/service-pipeline.yml
        inputs:
          service: worker
          python_version: "3.11"        # this service has not migrated yet
  rules:
    - changes: [services/worker/**/*, libs/common/**/*, ci/service-pipeline.yml]

web:
  extends: .service-trigger
  trigger:
    include: services/web/.gitlab-ci.yml   # a service with its own, hand-written child pipeline
  rules:
    - changes: [services/web/**/*]

extends deep-merges the trigger hash, so each job ends up with both strategy: depend from the template and its own include. The examples assume workflow:rules already prevent duplicate pipelines (section 4.4), which is why the trigger rules can use bare changes.

Comparing the approaches.

	Path rules in one pipeline	Per-service child pipelines	Dynamically generated children
Job naming	Must be unique across the repository	Unique per child only	Generated
Stages	One shared list	Each child has its own	Generated
Pipeline view	One large graph	One collapsible box per service	One collapsible box per service
Cross-service needs	Possible	Not between sibling children	Not between sibling children
Overhead	Lowest	One extra pipeline and trigger job per service	Plus a generator job
Best for	A few services	Many services with separate owners	Many services where the set of affected ones comes from a build tool (section 3.3)

Things to watch.

• Variables. By default, a child receives the trigger job's YAML variables but not pipeline variables (for example from a manual run). Use trigger:forward:pipeline_variables: true if children need them.
• Artifacts across services. A child job can download artifacts from the parent with needs:pipeline:job (section 3.3), but not from a sibling child. Build shared outputs in the parent.
• Depth limits. Children can trigger their own children, but nesting is limited (section 3.3). One level of per-service children is usually enough.
• Merge request status. With strategy: depend, the MR shows a failure when any affected service fails, without anyone opening the child pipelines.

Key takeaways

• One trigger job per service, each with rules:changes, starts only the child pipelines that a change affects.
• A child template with spec:inputs lets many similar services share one definition. Inputs are interpolated anywhere in the file.
• Children isolate job names, stages and status. Use strategy: depend so failures reach the parent and the MR.
• Siblings cannot needs each other. Produce shared artifacts in the parent.

Practice

1. (Beginner) Convert a two-service repository from path rules on jobs to two trigger jobs, each starting a child pipeline from the service's own .gitlab-ci.yml. Verify that changing one service starts only one child.
2. (Intermediate) Replace the per-service files with a single ci/service-pipeline.yml template using spec:inputs. Add an input that controls whether the package job runs, and use it to disable packaging for one service.
3. Interview question: "When would you choose dynamically generated child pipelines over static per-service trigger jobs in a monorepo?" Hint: number of services, how often services are added, and where the knowledge of what is affected actually lives.

Shared Configuration in Monorepos

Why it matters. In a monorepo, one CI configuration serves many teams. If every service copies the same twenty lines of setup, a fix to that setup means twenty edits and twenty chances to miss one. If everything sits in one shared file, every team blocks on every other team's review. A good structure keeps common behavior in one place and lets each team own its service's specifics.

A layout that scales.

.
├── .gitlab-ci.yml                  # entry point: workflow, stages, includes only
├── .gitlab/CODEOWNERS
├── ci/
│   ├── defaults.yml                # default:, global variables, workflow:rules
│   ├── templates/
│   │   ├── python.yml              # .python-test, .python-build (hidden jobs)
│   │   └── node.yml                # .node-test, .node-build
│   └── service-pipeline.yml        # child template (previous topic)
└── services/
    ├── api/.gitlab-ci.yml          # owned by the api team: extends the templates
    ├── web/.gitlab-ci.yml
    └── worker/.gitlab-ci.yml

The root file only wires things together. include:local accepts wildcards, so adding a service does not require editing it:

# .gitlab-ci.yml
include:
  - local: ci/defaults.yml
  - local: ci/templates/*.yml           # every template file
  - local: services/*/.gitlab-ci.yml    # every service contributes its own jobs

stages: [build, test, package, deploy]

# ci/templates/python.yml
.python-test:
  image: python:3.12-slim
  variables:
    PIP_CACHE_DIR: $CI_PROJECT_DIR/.cache/pip
  cache:
    key: pip-$SERVICE                   # one cache per service: no cross-service thrashing
    paths: [.cache/pip]
  script:
    - cd "services/$SERVICE"
    - pip install -r requirements.txt
    - pytest --junitxml=report.xml
  artifacts:
    reports:
      junit: services/$SERVICE/report.xml

# services/api/.gitlab-ci.yml (owned by the api team)
.api-rules:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      changes: [services/api/**/*, libs/common/**/*, ci/templates/python.yml]
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

api-test:                               # prefixed with the service: names are global
  extends: [.python-test, .api-rules]
  variables:
    SERVICE: api

Rules that keep it sane. Everything included into one pipeline shares one namespace, so conventions matter:

Rule	Reason
Prefix job and hidden-job names with the service (api-test, .api-rules)	Two includes defining the same key are merged, and the later one silently wins (section 9.1)
Define stages, workflow and default only in root-level files	They are global. A service file that redefines them affects every service
Use variables or inputs for service differences, not copies	One template, many services
Give each service its own cache key	Shared keys make services overwrite each other's caches
Protect ci/ with code owners	Template changes affect every team

# .gitlab/CODEOWNERS (the last matching pattern wins)
/ci/                              @platform-team
/services/api/                    @api-team
/services/api/.gitlab-ci.yml      @api-team @platform-team

With required code owner approval (Premium), the platform team must approve every template change, and both teams review changes to a service's CI file.

Where shared configuration should live.

Location	Versioning	Best for
ci/ in the monorepo, included with include:local	Same commit as the code	Monorepo-specific conventions that change with the code
include:local with inputs	Same commit	Parameterized templates without copying
CI/CD components from a separate project (section 9.3)	Released versions	Organization-wide building blocks shared with other repositories

Templates in the same repository change atomically with the code that uses them, which is one of the main advantages of a monorepo. Use components for what genuinely needs to be shared beyond it.

Key takeaways

• Keep the root .gitlab-ci.yml small: global settings plus wildcard includes of templates and per-service files.
• Templates hold common behavior, and service files extend them with variables or inputs. No copying.
• All included files share one namespace: prefix names, keep global keywords in root files, and use per-service cache keys.
• Protect shared CI files with code owners, and use versioned components only for configuration shared outside the monorepo.

Practice

1. (Beginner) Move the jobs of two services from the root .gitlab-ci.yml into services/<name>/.gitlab-ci.yml files and include them with a wildcard. Confirm in the pipeline editor's Full configuration tab that the merged result is unchanged.
2. (Intermediate) Create a .node-test template used by two services, with per-service cache keys. Then define a job with the same name in both service files and observe what the merged configuration contains.
3. Interview question: "Ten teams share one monorepo pipeline. How do you let each team change its own CI configuration freely while preventing one team from breaking everyone else's pipeline?" Hint: file ownership, namespacing, global keywords, and where validation happens before merge.

Sparse and Partial Clones

Why it matters. Large monorepos can reach tens of gigabytes, with millions of files and years of history. Even a shallow clone downloads every file of the current commit, so a job that tests one small service still transfers and writes the entire tree, and fifty parallel jobs do it fifty times. Git can fetch less: partial clones skip file contents until they are needed, and sparse checkouts write only selected folders to disk.

Three ways to fetch less.

Technique	Git option	What is skipped	Cost
Shallow clone	--depth N	Old history (commits, trees and blobs)	Tools that need history break (13.1)
Partial clone	--filter=blob:none	File contents (blobs) not needed for the checkout. Fetched on demand later	Later operations that need old blobs contact the server
Sparse checkout	git sparse-checkout set <dirs>	Files outside the chosen folders in the working tree	Jobs cannot see files outside those folders

                       history          file contents         working tree
full clone             all commits      all blobs             all files
shallow (depth 1)      1 commit         blobs of 1 commit     all files
partial (blob:none)    all commits      only checked-out      all files
partial + sparse       all commits      only sparse folders   only sparse folders   ◄ smallest for monorepos

The combination of partial clone and sparse checkout fits monorepo jobs best: the job gets full history (so git log and git diff work) but downloads only the contents of the folders it uses.

Doing it in a job. The runner's standard Git step does not set up sparse checkouts, so the usual approach is to skip it with GIT_STRATEGY: none and clone manually. CI_REPOSITORY_URL already contains the job token, so no extra credentials are needed:

api-test:
  image: python:3.12                      # full image: includes git (slim does not)
  variables:
    GIT_STRATEGY: none                    # skip the runner's full checkout
  script:
    # Partial clone: commits and trees, but no file contents yet; no checkout yet either
    - git clone --filter=blob:none --no-checkout "$CI_REPOSITORY_URL" repo
    - cd repo
    # Cone mode: these folders plus all files at the repository root
    - git sparse-checkout set --cone services/api libs/common
    # MR and merged results commits may not be on a branch, so fetch the exact commit
    - git fetch --filter=blob:none origin "$CI_COMMIT_SHA"
    - git checkout --detach "$CI_COMMIT_SHA"     # downloads blobs for the sparse folders only
    - cd services/api
    - pip install -r requirements.txt
    - pytest

Make sure the shared paths in sparse-checkout set match the paths in the job's rules:changes. They describe the same dependency, and when they drift apart, a job either misses files it needs or runs for changes it cannot see.

Native clone on recent runners. Recent GitLab Runner versions can run a real git clone instead of git init plus git fetch when the FF_USE_GIT_NATIVE_CLONE feature flag is enabled with GIT_STRATEGY: clone, and then pass extra flags with GIT_CLONE_EXTRA_FLAGS (for example --filter=blob:none). This gives a partial clone without a manual script, but no sparse checkout. Check your runner version's documentation before relying on it.

Interaction with other features.

Feature	Effect of sparse or partial clones
rules:changes	None: GitLab evaluates changes on the server, before any clone
Artifacts and caches	Paths are relative to the project directory. With a manual clone into repo/, artifact paths must include repo/
Generators that run git diff	Work with partial clones, because commits and trees are present. Only file contents are fetched lazily
Git LFS	Combine with GIT_LFS_SKIP_SMUDGE=1 and git lfs pull --include="services/api/**"
Persistent runners with fetch	A manual clone into a fresh folder discards the reuse benefit. Weigh the two on persistent hosts

Server side. Every clone also costs the GitLab server (Gitaly) CPU to compute and compress the data. Many parallel full clones of a large monorepo can slow Git operations for everyone. Partial and shallow clones reduce that load too, and administrators of self-managed instances can enable Gitaly's pack-objects cache, which lets identical concurrent clones share one computation.

Key takeaways

• Shallow clones cut history, partial clones (--filter=blob:none) cut file contents, and sparse checkouts cut the working tree.
• Partial plus sparse is the best fit for monorepo jobs: full history, but only the folders the job needs.
• Use GIT_STRATEGY: none and a manual clone from CI_REPOSITORY_URL, and fetch CI_COMMIT_SHA explicitly.
• Keep sparse paths aligned with rules:changes paths, and adjust artifact paths to the manual clone folder.

Practice

1. (Beginner) On a large public repository, compare git clone, git clone --depth 1 and git clone --filter=blob:none --sparse followed by git sparse-checkout set <one folder>. Record time, transferred size (from Git's output) and du -sh of the result.
2. (Intermediate) Convert one monorepo test job to a partial, sparse clone using the pattern above. Make it publish a JUnit report as an artifact, and compare its get_sources plus clone time with the original job.
3. Interview question: "Fifty parallel jobs clone the same 8 GB monorepo, and developers report that git push becomes slow during pipeline peaks. What is happening, and what would you change on the CI side and on the server side?" Hint: Gitaly load from pack generation, partial and sparse clones, fewer jobs that need sources, and the pack-objects cache.

---

<a id="14-monitoring-debugging-and-troubleshooting"></a>