14. Monitoring, Debugging and Troubleshooting

Every pipeline eventually fails in a way its author did not foresee. How quickly you recover depends on how well you can see inside it. This chapter covers three levels of visibility. The first is a single failing job: reading its log, tracing it, opening a terminal into it and reproducing it locally. The second is the pipeline's behavior over weeks and months: success rates, durations, DORA metrics and value streams. The third is getting the right signal to the right people at the right moment, through notifications, webhooks and commit statuses.

Tiers at a glance. Job logs, CI_DEBUG_TRACE, collapsible sections, interactive web terminals (on your own runners), CI Lint, project-level CI/CD analytics, badges, pipeline emails, Slack and Microsoft Teams integrations, project webhooks and the Commit Status API work on all tiers. Group webhooks, group-level value stream analytics with custom value streams, and CI/CD for external repositories (which reports status back to GitHub) require Premium. DORA metrics, the Value Streams Dashboard and external status checks require Ultimate. Root Cause Analysis requires a GitLab Duo add-on. Its packaging has changed several times, so check the current documentation before planning around it.

<a id="141-debugging-pipelines"></a>

14.1 Debugging Pipelines

This subchapter is a toolkit for answering one question as quickly as possible: why did this job fail? The tools are ordered from cheapest to most invasive. Read the log first, then add tracing, then structure the output, then open a shell inside the job, then reproduce it on your machine. The subchapter ends with a catalog of configuration errors that stop a pipeline before any job runs.

            cheap, always available                                   expensive, invasive
  ┌────────────┐  ┌───────────────┐  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐
  │ read the   │─►│ CI_DEBUG_TRACE│─►│ log sections │─►│ web terminal │─►│ run locally  │
  │ job log    │  │ / set -x      │  │ + context    │  │ into the job │  │ (same image) │
  └────────────┘  └───────────────┘  └──────────────┘  └──────────────┘  └──────────────┘
   failure_reason    every command     find the slow     inspect files,    iterate in
   exit code         and variable      or broken step    env, network      seconds

Reading Job Logs

Why it matters. The job log is the only first-hand record of what happened inside a job. Most failures can be diagnosed from it alone, provided you know where to look. Section 1.3 introduced the structure of a log. This topic is about reading a log systematically when it is long, noisy or misleading.

Start with the failure reason, not the log. Before scrolling, check how the job failed. GitLab classifies every failure, and the class tells you which half of the problem to investigate: your code, or the infrastructure around it. The reason appears in the job header (for example "There has been a timeout failure or the job got stuck") and as failure_reason in the Jobs API:

failure_reason	What happened	Where to look
script_failure	A command in before_script or script exited non-zero	The step_script section: your code, tests or scripts
runner_system_failure	The runner could not prepare or run the job (image pull, disk, Docker daemon, network)	The prepare_executor and get_sources sections, and the runner host
stuck_or_timeout_failure	No runner picked the job up in time, or the job exceeded its timeout	Runner tags and availability, timeout (sections 4.3, 6.3)
job_execution_timeout	The job ran longer than its timeout	Slow step or hanging process: look at the last lines before the cut
missing_dependency_failure	Artifacts the job depends on expired or were deleted	needs/dependencies, artifacts:expire_in (section 8.1)
unmet_prerequisites	Something required before the job could start was missing, such as a Kubernetes environment the job deploys to	Environment and agent configuration (section 12.4)
scheduler_failure	GitLab could not assign the job to a runner	Instance health, runner registration
api_failure	The runner and GitLab lost contact while the job ran	Network between runner and GitLab, GitLab availability

# List the failed jobs of pipeline 98765, with their failure reason and the stage they belong to
curl --silent --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.example.com/api/v4/projects/12345/pipelines/98765/jobs?scope[]=failed" |
  jq -r '.[] | "\(.stage)\t\(.name)\t\(.failure_reason)\t\(.web_url)"'

Read the exit code. The last line of a failed script job is ERROR: Job failed: exit code N. The number is the exit status of the command that failed, and some values have a fixed meaning:

Exit code	Usual meaning	Typical cause in CI
1	Generic failure	Failing tests, linter findings, a script's exit 1
2	Misuse of a shell builtin, or a tool's "usage error"	Wrong flags, syntax error in a shell script
126	Command found but not executable	Missing chmod +x on a script committed from Windows
127	Command not found	Tool not installed in the image, typo, wrong PATH
130	Interrupted (SIGINT)	Rare in CI: an interactive tool waiting for input
137	Killed (SIGKILL, 128 + 9)	Out of memory (OOM killer or Kubernetes OOMKilled), or a forced kill
139	Segmentation fault (128 + 11)	Native crash, often an image or architecture mismatch
143	Terminated (SIGTERM, 128 + 15)	Job canceled or timed out while the process was running

How the runner decides that a job failed. With Bash-family shells, the runner executes before_script and script as one script with set -eo pipefail. Two consequences follow:

• The first command that exits non-zero stops the job. Lines below it never run, so the error is almost always just above the ERROR: Job failed line.
• A failure anywhere in a pipe fails the line. pytest | tee report.txt fails when pytest fails, even though tee succeeded.

The reverse is also true: a command that fails without a non-zero exit code cannot fail the job. Common examples are a script that ends with echo "done" after an ignored error, || true added to "fix" a flaky step, and tools that print errors but always exit 0. after_script runs in a separate shell, and by default its failures do not change the job status at all.

Reading a long log. For logs of thousands of lines, a few techniques save most of the time:

Job log (simplified, 4,812 lines)

 1  Running with gitlab-runner 19.4.0 (abcdef12)
 2    on k8s-runner-03 Qx81aB2c, system ID: r_9f8e7d6c5b4a
 ...
41  $ npm ci                                                <- step_script begins
 ...
4790  FAIL src/cart/total.test.ts
4791    ● total › applies the discount once
4792      Expected: 90
4793      Received: 81                                      <- first real error, read upward from the end
4805  npm ERR! Lifecycle script `test` failed with error
4806  npm ERR! code 1                                        <- noise caused by the error above
4811  Cleaning up project directory and file based variables
4812  ERROR: Job failed: exit code 1

1. Read from the bottom up. Find the last $ command line, then the first error message below it. Later messages are usually consequences ("npm ERR! code 1") rather than causes.
2. Use the log search. The job log page has a search box that highlights matches, and every line number is a link (#L4793) you can paste into an issue or chat.
3. Open the raw log. Very long logs are truncated in the browser. The Show complete raw link (/-/jobs/<id>/raw) returns the full plain-text log, which you can pipe through grep.
4. Compare with the last green run. Open the same job in the most recent successful pipeline and compare the image digest, tool versions and the last few commands. "What changed?" is often faster than "what is wrong?".
5. Enable timestamps with FF_TIMESTAMPS: "true" (section 13.1) when you suspect a hang or a slow step.

# Fetch the full log of job 554433 and show every line that looks like an error, with its line number
curl --silent --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.example.com/api/v4/projects/12345/jobs/554433/trace" |
  grep -n -i -E "error|fail|fatal|exception|denied|not found" | head -40

# Same thing with the glab CLI, which also streams the log of a running job
glab ci trace 554433

Messages that are not your code. Some log lines point straight at infrastructure or at a limit:

Log message	Meaning
This job is stuck because you don't have any active runners online or available with any of these tags assigned to them: gpu	No online runner matches the job's tags (section 6.3)
ERROR: Job failed: execution took longer than 1h0m0s seconds	Job timeout reached
Job's log exceeded limit of 4194304 bytes.	The runner's output_limit (default 4096 KB) was reached. The job keeps running, but further output is lost
ERROR: Job failed (system failure): ... no space left on device	Runner host or volume is full
ERROR: Preparation failed: ... pull access denied / manifest unknown	Image name or tag is wrong, or registry credentials are missing (section 7.1)
WARNING: Uploading artifacts ... 413 Request Entity Too Large	Artifact exceeds the maximum artifact size
[MASKED] where you expected a value	A masked variable's value was redacted in the log, which is expected (section 5.2)

Key takeaways

• Check failure_reason first. It tells you whether to debug your code (script_failure) or the infrastructure (runner_system_failure, stuck_or_timeout_failure).
• The exit code carries meaning: 127 is a missing command, 137 usually means out of memory, 143 a cancellation or timeout.
• Scripts run with set -eo pipefail. The first failing command ends the job, and the real error sits just above the final ERROR line.
• Commands that swallow errors (|| true, tools that always exit 0) produce green jobs that hide real failures.
• For long logs, use the raw log, grep -n and a comparison with the last successful run.

Practice

1. (Beginner) Write a job whose script runs false | tee out.txt followed by echo "still here". Predict the result, run it, and explain what you see using pipefail.
2. (Intermediate) Write a script that lists every failed job across the last 50 pipelines of your default branch and counts them by failure_reason. Use the result to decide whether your instability comes mostly from code or from infrastructure.
3. Interview question: "A job fails with exit code 137 roughly once in ten runs, always in the same test step. What do you suspect, and how do you confirm it?" Hint: 128 + 9, memory limits of the executor, parallel test workers, and the runner's or the Kubernetes pod's events.

CI_DEBUG_TRACE

Why it exists. Sometimes the log does not contain enough information. A variable may have an unexpected value, a rules-dependent variable may not be set, a script may branch the wrong way, or the failure may happen inside the runner's own preparation steps, which are normally hidden. Debug tracing makes the runner print every command it executes and every variable it sets, including the runner's internal setup script.

What it does. Setting the predefined variable CI_DEBUG_TRACE to "true" enables shell tracing (the equivalent of set -x) for the whole job script that the runner generates. The log then shows:

• the export of every CI/CD variable available to the job,
• each command, after variable expansion, prefixed by +,
• the runner's internal steps, such as the Git fetch commands and cache and artifact handling.

unit-tests:
  variables:
    CI_DEBUG_TRACE: "true"           # trace this job only; remove after debugging
  script:
    - ./scripts/run-tests.sh

Job log with CI_DEBUG_TRACE (simplified)

++ export CI_COMMIT_REF_NAME=feature/cart
++ export CI_PIPELINE_SOURCE=merge_request_event
++ export DATABASE_URL=postgres://ci:[MASKED]@postgres:5432/test   <- masked part stays hidden
++ export TEST_SHARD=                                              <- empty: the culprit
+ ./scripts/run-tests.sh
+ '[' -z '' ']'
+ echo 'No shard given, running the full suite'

Where to set it.

Scope	How	When to use
One job	variables: in that job	You know which job misbehaves
One run	Add CI_DEBUG_TRACE = true in the Run pipeline form, or in the variables of a manual job before you start it	Reproduce a failure once without committing anything
Whole pipeline	Top-level variables:	Rarely: the logs of all jobs become huge
Services	CI_DEBUG_SERVICES: "true"	Print the logs of services: containers (databases, Docker-in-Docker) in the job log

Security: treat a debug log as sensitive. Debug tracing prints the values of all variables. The runner still replaces masked values with [MASKED]. Unmasked variables are printed in clear text, and so are values derived from secrets, such as a token that was decoded, concatenated or URL-encoded. To limit the damage:

• Enable it only for one job or one run, never as a permanent setting on a shared branch.
• GitLab restricts access to logs produced with debug logging enabled: only users with at least the Developer role can view them. In public projects, check this before you trace.
• When you are done, erase the log (Erase job log on the job page, or POST /projects/:id/jobs/:job_id/erase) if it may contain anything sensitive.
• Rotate any secret that appeared unmasked.

Narrower alternatives. Full tracing is often more than you need. Tracing only your own script keeps the runner's internals and the variable dump out of the log:

unit-tests:
  script:
    - set -x                         # trace from here on, in this job only
    - ./scripts/run-tests.sh
    - set +x                         # stop tracing before anything sensitive
    - ./scripts/upload-results.sh

For a single value, echo "TEST_SHARD=${TEST_SHARD:-<unset>}" is even cheaper. Problems on the runner side (registration, executor, autoscaling) are not visible in job logs at all. For those, raise the runner's own log level (log_level = "debug" in config.toml, or gitlab-runner --debug run) and read the runner's service logs (section 6.3).

Key takeaways

• CI_DEBUG_TRACE: "true" prints every command and every variable, including the runner's internal setup steps.
• Set it for one job or one run, through job variables or the Run pipeline form, and remove it afterwards.
• Masked values stay masked, but unmasked and derived secrets appear in clear text. Erase the log and rotate secrets if needed.
• set -x in your own script is a narrower, safer form of tracing. CI_DEBUG_SERVICES shows service container logs.

Practice

1. (Beginner) Run a job once with CI_DEBUG_TRACE: "true" set from the Run pipeline form. Find in its log the Git commands the runner used to fetch the source, and the value of CI_PIPELINE_SOURCE.
2. (Intermediate) A job uses a PostgreSQL service and fails with "connection refused". Use CI_DEBUG_SERVICES to find out whether the database started, how long it took, and whether the job tried to connect too early. Fix it with a readiness loop.
3. Interview question: "A colleague proposes enabling CI_DEBUG_TRACE permanently on all pipelines 'to make debugging easier'. What is your answer?" Hint: secrets in logs, who can read logs, log size and storage, and narrower alternatives.

Collapsible Log Sections

Why they exist. A long log is hard to read because important lines are buried among routine ones. Package installs, downloads and verbose test output push the one interesting line out of sight. Sections let you fold the routine parts away and name the steps, and each section header shows how long that step took. The runner already does this for its own phases (prepare_executor, get_sources, step_script and so on, see section 13.1). Custom sections apply the same idea to your script.

How they work. A section is delimited by two special lines that the job log viewer recognizes. The ANSI escape sequences (\e[0K, an "erase line" code, and \r) hide the markers themselves, so only the header text remains visible:

\e[0Ksection_start:<unix timestamp>:<section name>[collapsed=true]\r\e[0K<header text>
... lines inside the section ...
\e[0Ksection_end:<unix timestamp>:<section name>\r\e[0K

• The section name may contain only letters, numbers, _, . and -, and must be the same in the start and end markers.
• The timestamps are Unix seconds. GitLab subtracts them to display the duration next to the header.
• [collapsed=true] is optional. Without it, the section is expanded by default.

build:
  script:
    - echo -e "\e[0Ksection_start:$(date +%s):deps[collapsed=true]\r\e[0KInstall dependencies"
    - npm ci
    - echo -e "\e[0Ksection_end:$(date +%s):deps\r\e[0K"
    - npm run build                  # stays visible: this is the part people care about

Reusable helpers. Writing escape sequences by hand is error-prone. Put two small functions in a script that every job sources. This version uses printf with octal escapes, so it works in sh (for example on Alpine images) as well as in bash, where echo -e is not portable:

# ci/log.sh: source it in before_script, then call section_start / section_end
section_start() {   # usage: section_start <name> "<header>" [collapsed]
  _collapsed=""
  [ "${3:-}" = "collapsed" ] && _collapsed="[collapsed=true]"
  printf '\033[0Ksection_start:%s:%s%s\r\033[0K\033[1;36m%s\033[0m\n' \
    "$(date +%s)" "$1" "$_collapsed" "$2"          # \033[1;36m: bold cyan header
}

section_end() {     # usage: section_end <name>
  printf '\033[0Ksection_end:%s:%s\r\033[0K\n' "$(date +%s)" "$1"
}

default:
  before_script:
    - . ci/log.sh                    # "." (source) keeps the functions in the job's shell

integration-tests:
  script:
    - section_start db "Start database and apply migrations" collapsed
    - ./scripts/db-up.sh
    - section_end db
    - section_start tests "Integration tests"
    - pytest tests/integration
    - section_end tests

The rendered log then reads like a table of contents:

> Start database and apply migrations                                     00:42
v Integration tests                                                       03:17
    ============================= test session starts =============================
    ...

Automatic per-command sections. The runner feature flag FF_SCRIPT_SECTIONS: "true" wraps every line of script in its own section, with its duration. It requires no script changes, and is useful when you want to see which command is slow without editing anything.

variables:
  FF_SCRIPT_SECTIONS: "true"         # one collapsible, timed section per script line

PowerShell. On Windows runners, use the PowerShell escape character:

Write-Host "`e[0Ksection_start:$([DateTimeOffset]::UtcNow.ToUnixTimeSeconds()):restore[collapsed=true]`r`e[0KRestore NuGet packages"
dotnet restore
Write-Host "`e[0Ksection_end:$([DateTimeOffset]::UtcNow.ToUnixTimeSeconds()):restore`r`e[0K"

Sections are machine-readable too. The markers remain in the raw log, so you can parse them to measure step durations across many jobs. This is how the timeline script in section 13.1 could be extended to break each job down by phase.

Key takeaways

• Sections are delimited by section_start and section_end markers with Unix timestamps, hidden by ANSI escape codes.
• Use [collapsed=true] for routine steps, so that the important output is visible when the log opens.
• Keep the escape sequences in a sourced helper script. Prefer printf over echo -e for portability.
• FF_SCRIPT_SECTIONS gives a timed section for every script line, with no script changes.
• Section markers in the raw log allow automated per-step timing.

Practice

1. (Beginner) Add a collapsed "Install dependencies" section and an expanded "Run tests" section to one of your jobs. Check that the durations shown in the UI match your expectations.
2. (Intermediate) Create ci/log.sh with the helpers above, include it in a shared template (section 9.1), and use it in three jobs. Then write a script that downloads one job's raw log and prints each section's name and duration from the markers.
3. Interview question: "Your test job's log is 30,000 lines and developers say they 'never find the failure'. How would you restructure the output without losing information?" Hint: sections, collapsing, quieter tool flags, test reports (section 10.1) and artifacts for the full output.

Interactive Web Terminals

Why they exist. Some failures cannot be understood from the outside. A file may be missing from a path you thought it was in, DNS may resolve differently inside the job, or a tool may behave differently in the runner's environment than on your machine. In these cases you want what you would do on any broken server: open a shell and look around. An interactive web terminal opens a shell inside the running job's environment, directly from the job page.

How it works.

 Browser (job page, "Debug")
     │  WebSocket over HTTPS
     ▼
 GitLab (Workhorse proxies the terminal connection)
     │  WebSocket to advertise_address
     ▼
 Runner session server (:8093) ───► job container / pod  (same image, files, variables)

The runner opens a small session server that GitLab connects to. GitLab proxies your browser's connection to it, and the runner attaches a shell to the job's container.

Requirements.

Requirement	Details
Runner you control	GitLab-hosted runners on GitLab.com do not offer web terminals. You need your own runner
Supported executor	Docker and Kubernetes (and the Shell executor on Unix-like systems). Not SSH, VirtualBox or Parallels
[session_server] in config.toml	listen_address, plus an advertise_address that the GitLab server can reach (section 6.3)
Network path	GitLab must reach the runner on the session port, and any reverse proxy in front of GitLab must allow WebSocket upgrades
Permissions	Only users allowed to run and update the job can open its terminal

# config.toml (runner)
[session_server]
  listen_address = "[::]:8093"                        # where the runner listens
  advertise_address = "runner-01.example.com:8093"    # what GitLab connects to
  session_timeout = 1800                              # seconds a session may stay open

Using it. While a job is running, the job page shows a Debug button that opens the terminal. Jobs often finish (or fail) too quickly to catch, so give yourself time. Add an opt-in pause that only applies when you ask for it:

integration-tests:
  script:
    - |
      if ! ./scripts/run-integration-tests.sh; then
        if [ "${DEBUG_PAUSE:-}" = "true" ]; then           # set in the Run pipeline form
          echo "Tests failed, waiting 15 minutes for a debug session"
          sleep 900
        fi
        exit 1                                            # still fail the job afterwards
      fi

Start a pipeline with DEBUG_PAUSE=true, wait for the failure message, click Debug, and inspect the environment: env | sort, ls -la, cat the config files, curl the service, rerun the failing command with extra flags. When a terminal session is attached, the runner keeps the job's environment available until you close the session or session_timeout expires.

Security. A web terminal is a shell with all of the job's variables, including protected and masked secrets, and with the job's network access. Treat access to it like SSH access to a deployment host. Restrict who can run jobs on runners that expose a session server, and do not expose the session port to the internet.

When you cannot use it. On GitLab-hosted runners, the closest alternatives are richer diagnostics in after_script, artifacts with the full output (artifacts:when: always), or reproducing the job locally with the same image (next topic).

Key takeaways

• A web terminal is a shell inside the running job, opened from the job page with Debug.
• It needs your own runner, a supported executor (Docker, Kubernetes, Shell) and a reachable [session_server].
• Keep a failing job alive on purpose with an opt-in sleep gated by a variable such as DEBUG_PAUSE.
• The terminal has the job's secrets and network access. Control who can use it, and keep the session port private.

Practice

1. (Beginner) On a self-managed runner, configure [session_server], run a job that sleeps for five minutes, open its terminal and print the job's working directory and the image's OS release (cat /etc/os-release).
2. (Intermediate) Add the DEBUG_PAUSE pattern to a template so that every job can be paused on failure. Make sure the pause never applies to pipelines on protected branches.
3. Interview question: "The Debug button appears, but the terminal never connects and eventually shows a connection error. Where do you look?" Hint: advertise_address, firewalls between GitLab and the runner, the WebSocket configuration of the reverse proxy, and TLS.

Running Pipelines Locally

Why it matters. The push-wait-read cycle is slow: every attempt costs a commit, a pipeline and a few minutes. When you are fixing a script or a build step, you want to iterate in seconds. Running the job, or at least its commands, on your own machine shortens that loop dramatically.

What "locally" can mean. A GitLab pipeline has two layers that can be reproduced separately:

Layer	What it covers	How to reproduce it locally
Configuration	Which jobs exist, includes, extends, rules, needs	CI Lint, pipeline editor, Lint API (section 1.3), gitlab-ci-local --preview
Execution	What a job's script does in its image	docker run with the job's image, gitlab-ci-local, or plain scripts

What does not work: gitlab-runner exec. Older guides suggest gitlab-runner exec docker <job>. This command is deprecated. It never supported includes, artifacts passed between jobs, needs or most of modern GitLab CI, so do not build a workflow around it.

Option 1: run the job's commands in the job's image. This is the most faithful option and needs nothing beyond Docker. Use the same image (ideally the same digest) and the same working directory, and set the variables the script relies on:

# Reproduce the "unit-tests" job by hand in its image
docker run --rm -it \
  --volume "$PWD":/builds/shop/api \
  --workdir /builds/shop/api \
  --env CI=true \
  --env CI_PROJECT_DIR=/builds/shop/api \
  --env CI_COMMIT_REF_NAME="$(git branch --show-current)" \
  --env CI_COMMIT_SHA="$(git rev-parse HEAD)" \
  python:3.12-slim bash
# Inside the container: paste the job's before_script and script lines one by one

The differences that remain are the user ID, mounted volumes, services and network, and secrets. Each of them can be the real cause of a CI-only failure, so check them deliberately when a job passes locally but fails in CI.

Option 2: gitlab-ci-local. This open-source tool (not maintained by GitLab) reads your .gitlab-ci.yml, resolves includes, extends, needs and artifacts, and runs jobs in Docker containers on your machine:

npm install --global gitlab-ci-local      # also available through Homebrew

gitlab-ci-local --list                    # jobs, stages, when and needs, as the tool understands them
gitlab-ci-local --preview                 # fully merged configuration (includes and extends resolved)
gitlab-ci-local unit-tests                # run one job in its image
gitlab-ci-local --needs e2e               # run "e2e" and every job it needs, in order
gitlab-ci-local --variable DEPLOY_ENV=staging deploy-staging   # override a variable for this run

Things to keep in mind:

• Jobs without an image run directly in your shell, not in a container. Set a default:image to avoid surprises.
• Secrets are not available. Provide test values through the tool's variables file (see its documentation), and never commit real secrets to it.
• Remote includes from private projects need a token, and rules that depend on merge request or pipeline-source variables may evaluate differently than on the server.

Option 3: design jobs so that they run anywhere. The most durable fix is structural. Keep the YAML thin and put the logic into scripts or make targets that run unchanged on a laptop and in CI:

# Thin job: the YAML only chooses the environment; the logic lives in the repository
unit-tests:
  image: python:3.12-slim
  script:
    - make test                      # the same command developers run locally

# Makefile
test:
	pip install -r requirements-dev.txt
	pytest --junitxml=report.xml

Comparison.

Approach	Fidelity	Effort	Best for
CI Lint / pipeline editor	Configuration only	None	"Why does this job not appear?"
docker run with the job's image	High for one job	Low	Script and tooling failures
gitlab-ci-local	Medium to high, multi-job	Low to medium	Iterating on multi-job flows, needs and artifacts
Thin YAML + make targets	High, permanent	One-time refactoring	Preventing CI-only bugs in the first place
Web terminal in the real job	Exact	Needs a self-managed runner	Environment-specific failures

Key takeaways

• Separate configuration problems (use Lint and the pipeline editor) from execution problems (run the job in its image).
• docker run with the job's image, working directory and key variables reproduces most script failures in seconds.
• gitlab-ci-local simulates multi-job pipelines locally. It is third-party software, and secrets and some rules behave differently.
• gitlab-runner exec is deprecated and incomplete. Do not rely on it.
• Thin YAML that calls scripts or make targets makes every job reproducible by design.

Practice

1. (Beginner) Pick a job from one of your pipelines and reproduce it with docker run using the same image. List every difference you had to work around (variables, services, files).
2. (Intermediate) Install gitlab-ci-local, run --preview on a project that uses includes and extends, and compare the output with the pipeline editor's Full configuration tab. Then run a two-job chain with --needs and check that artifacts are passed.
3. Interview question: "A job passes on every developer's laptop but fails in CI with a permissions error when writing to a directory. What differences between the two environments would you check first?" Hint: the user ID inside the container, file ownership of the cloned sources, read-only paths, and the image tag versus the local image.

AI-Assisted Root Cause Analysis

Why it exists. Many CI failures are repetitive: a dependency conflict, a missing environment variable, an out-of-memory kill, a breaking change in a tool's new release. Recognizing them is easy for someone who has seen them before and slow for someone who has not. Large language models have seen a great many such logs, so they can often explain an unfamiliar error quickly and point to a likely fix.

GitLab Duo Root Cause Analysis. On a failed job's page, GitLab offers a Troubleshoot action (also available as /troubleshoot in GitLab Duo Chat when you are viewing the job). It sends the relevant part of the job log to GitLab Duo, which returns:

• a plain-language explanation of what failed,
• the likely root cause,
• one or more suggested fixes, often with a code or configuration change.

 Failed job page ──"Troubleshoot"──► GitLab Duo
                                        │ input: truncated job log (mostly the end),
                                        │        plus job context
                                        ▼
                               Explanation + likely cause + suggested fix
                                        │
                                        ▼
                     You verify, apply in a branch, rerun the pipeline

Availability. Root Cause Analysis requires a GitLab Duo add-on, and on self-managed instances an administrator must turn GitLab Duo on. Groups and projects can also turn it off. Packaging has changed between releases, and newer releases add agent-based flows that can propose a fix as a merge request. Check the current documentation for your version and tier.

Using it well. The model sees mostly the log. It does not see your infrastructure, your runner configuration or code that is not in the log. Its answer is a hypothesis, not a diagnosis:

Do	Avoid
Check the cited log lines yourself before acting	Applying a suggested fix you do not understand
Ask follow-up questions in Duo Chat ("why would this pass locally?")	Treating "add `
Verify the fix in a branch pipeline	Pasting logs into public AI tools: logs can contain internal hostnames and unmasked values
Use it to learn unfamiliar tools' error formats	Relying on it for infrastructure failures that are invisible in the log

Make failures easier to analyze, for people and for AI. Both work better when the log ends with clear context. A small after_script that prints diagnostics only on failure helps every reader:

default:
  after_script:
    - |
      if [ "$CI_JOB_STATUS" = "failed" ]; then             # set by the runner in after_script
        echo "---------- diagnostic context ----------"
        echo "image:  $CI_JOB_IMAGE"
        echo "runner: $CI_RUNNER_DESCRIPTION"
        df -h . | tail -1                                  # disk space left in the build dir
        (free -m 2>/dev/null || cat /proc/meminfo | head -3) # memory, whichever tool exists
        git log -1 --format='commit: %h %s'
      fi

A tool-independent triage script. Whether you then read the result yourself or hand it to an assistant, extracting the essentials of every failed job in a pipeline saves time:

#!/usr/bin/env python3
# failed_excerpts.py: print failure reason and the error excerpt of every failed job in a pipeline
# Usage: GITLAB_TOKEN=glpat-... python3 failed_excerpts.py <project_id> <pipeline_id>
import json
import os
import re
import sys
import urllib.request

API = os.environ.get("GITLAB_API", "https://gitlab.com/api/v4")
project_id, pipeline_id = sys.argv[1], sys.argv[2]
ERROR = re.compile(r"error|fail|fatal|exception|denied|not found|killed", re.IGNORECASE)
ANSI = re.compile(r"\x1b\[[0-9;]*[A-Za-z]")              # strip color codes and section markers

def get(path, raw=False):
    req = urllib.request.Request(API + path, headers={"PRIVATE-TOKEN": os.environ["GITLAB_TOKEN"]})
    with urllib.request.urlopen(req) as resp:
        body = resp.read().decode("utf-8", errors="replace")
        return body if raw else json.loads(body)

jobs = get(f"/projects/{project_id}/pipelines/{pipeline_id}/jobs?scope[]=failed&per_page=100")
for job in jobs:
    lines = ANSI.sub("", get(f"/projects/{project_id}/jobs/{job['id']}/trace", raw=True)).splitlines()
    tail = lines[-200:]                                   # the cause is almost always near the end
    hits = [l for l in tail if ERROR.search(l)][:15]
    print(f"=== {job['stage']} / {job['name']}  ({job['failure_reason']})  {job['web_url']}")
    print("\n".join(hits or tail[-15:]))                  # fall back to the last lines
    print()

Key takeaways

• GitLab Duo Root Cause Analysis (Troubleshoot on a failed job) explains failures and suggests fixes from the job log. It requires a Duo add-on, and availability varies by version.
• The model sees mostly the log, so its answer is a hypothesis: verify the cited lines and test the fix in a branch.
• Reject "fixes" that hide failures (|| true, longer timeouts) unless you understand why they are correct.
• Failure-only diagnostics in after_script and short, clear error output make logs easier to analyze for both people and AI.
• Logs can contain sensitive data. Use only AI tools approved for your organization's code and logs.

Practice

1. (Beginner) Break a job on purpose (a missing dependency, a wrong command name). If you have GitLab Duo, use Troubleshoot and judge whether the explanation and fix are correct. If not, use failed_excerpts.py and diagnose it yourself.
2. (Intermediate) Add the diagnostic after_script to a shared template. Then cause an out-of-memory failure (for example, a test that allocates a large array) and check whether the diagnostics make the cause obvious.
3. Interview question: "An AI assistant suggests fixing a flaky integration test by adding retry: 2 to the job. When is that acceptable, and when is it harmful?" Hint: infrastructure flakiness versus real race conditions, retry:when, and how retries hide failure trends.

Common Configuration Errors

Why it matters. Configuration errors are a different kind of failure: no job runs at all, or the wrong jobs run. GitLab reports them as a pipeline with the yaml invalid label, an error banner in the pipeline editor, or sometimes as a pipeline that silently does not appear. Knowing the usual messages turns a confusing banner into a one-line fix.

A method for any configuration error.

 Error banner / no pipeline
        │
        ├─ 1. Read the message: it names the job and the key ("jobs:build:script ...")
        ├─ 2. Pipeline editor ► Validate (or CI Lint with "simulate") for the right branch
        ├─ 3. Full configuration tab: inspect the merged YAML after includes, extends, !reference
        └─ 4. Bisect: comment out includes or jobs until the error disappears

Step 3 matters most when includes are involved, because the error refers to the merged configuration, which may not look like any single file you wrote. The Lint API returns the merged result too:

# Validate the configuration stored on a branch and print the merged YAML GitLab actually sees
curl --silent --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.example.com/api/v4/projects/12345/ci/lint?ref=feature/cart&dry_run=true" |
  jq -r '.valid, .errors[], .merged_yaml'

Errors that block the pipeline. The messages below are typical. Exact wording varies slightly between versions.

Error message (abridged)	Cause	Fix
(<unknown>): did not find expected key while parsing a block mapping at line 7 column 3	YAML syntax: wrong indentation, tabs, an unclosed quote	Fix indentation at or above the reported line, and use spaces only
jobs:build config contains unknown keys: scirpt	Typo in a keyword	Correct the keyword
jobs:build:script config should be a string or a nested array of strings up to 10 levels deep	A script line was parsed as a map, usually because of : inside an unquoted line	Quote the whole line: - 'echo "status: ok"'
deploy job: chosen stage deploy does not exist; available stages are .pre, build, test, .post	stage: names a stage missing from stages:	Add the stage, or fix its name
'unit' job needs 'build' job, but 'build' does not exist in the pipeline...	The needed job is absent: a typo, or excluded by its rules	Fix the name, align the rules, or use needs: [{ job: build, optional: true }]
jobs:test config key may not be used with rules: only	rules mixed with only/except (or job-level when) in one job	Use rules alone (section 4.2)
test: unknown key in extends (.base-job)	extends points to a job that does not exist after merging	Fix the name, or check that the include defining it is loaded
test: circular dependency detected in extends	Two templates extend each other	Break the cycle
Local file ci/build.yml does not exist!	Wrong path in include:local, or the file is missing on this branch	Fix the path. Local includes are resolved on the pipeline's commit
Project platform/ci-templates not found or access denied!	Wrong path, or the user running the pipeline cannot read that project	Fix the path, or grant access (also check the scheduled pipeline owner's access)
jobs config should contain at least one visible job	Every job is hidden (starts with .), or only templates were included	Add a visible job, or fix the include
Maximum of 150 nested includes are allowed!	Include chain or fan-out too large	Flatten the includes, or use components (section 9.3)
variables config should be a hash of key value pairs...	A variable value has the wrong type, for example an unquoted true, yes or a list	Quote the value: ENABLE_X: "true"

Errors that are not reported. The most confusing problems produce a valid configuration that does the wrong thing:

Symptom	Common cause	Where it is covered
No pipeline at all after a push	workflow:rules excluded it. Run pipeline shows "Pipeline filtered out by workflow rules."	Section 4.4
Pipeline not created, "The rules configuration prevented any jobs from being added to the pipeline."	Every job's rules evaluated to "do not add"	Section 4.1
Job missing from the pipeline	First-match semantics in rules, or a comparison against an unset variable	Section 4.1
Two pipelines per push (branch + MR)	Missing workflow:rules for MR pipelines	Section 4.4
Job runs on every push despite rules:changes	New branches and some pipeline sources have no "before" commit to compare with	Section 4.1, compare_to
Value has unexpected content	YAML typing: 0755 is read as the octal number 493, 1.10 as 1.1	Section 2.1
A password containing $ is cut off	Variable expansion inside the value	$$ or expand: false (section 5.1)

Prevention. Lint configuration changes in merge requests, and run glab ci lint or the Lint API in a pre-commit hook or in the template project's own pipeline (section 1.3). Keep shared templates versioned (include:ref or component versions), so that a broken template does not break every consumer at once.

Key takeaways

• Error messages name the job and the key at fault. Read them literally before guessing.
• Errors refer to the merged configuration: inspect it in the pipeline editor's Full configuration tab or in the Lint API's merged_yaml.
• Most blocking errors are typos, YAML structure problems, missing stages, unresolvable needs and extends, and include paths or access.
• The hardest problems are valid configurations that behave wrongly: workflow:rules, rules order, rules:changes and YAML typing.
• Lint in merge requests and version shared templates, so that errors are caught before they reach a pipeline.

Practice

1. (Beginner) Introduce each of these errors in a test project one at a time and record the exact message: a keyword typo, a missing stage, a needs on a nonexistent job, an unquoted echo "a: b" script line.
2. (Intermediate) Create a project that includes a template from another project, then remove your access to that project. Run a scheduled pipeline and a manual pipeline as different users, and explain the results.
3. Interview question: "Developers say 'the pipeline doesn't start any more' after a template change, but CI Lint on their .gitlab-ci.yml says it is valid. How do you investigate?" Hint: which ref Lint validated against, workflow:rules in the template, simulated pipeline creation, and the merged configuration.

<a id="142-observability-and-metrics"></a>

14.2 Observability and Metrics

Debugging explains one failure. Observability shows trends: whether the pipeline is getting slower, whether the default branch is red more often than last quarter, whether changes take days or hours to reach production. This subchapter covers GitLab's built-in analytics for pipelines, delivery performance (DORA) and the flow of work (value streams), and the badges that put a project's health on its front page. Runner-level metrics with Prometheus are covered in section 6.4.

 Level                 Question                                   GitLab feature
 ───────────────────── ────────────────────────────────────────── ──────────────────────────
 Runner fleet          Do we have enough capacity?                 Runner Prometheus metrics (6.4)
 Pipeline              Is CI fast and reliable?                    CI/CD analytics, API
 Delivery              How fast and safely do we ship?             DORA metrics
 Whole process         Where does work wait, idea to production?   Value stream analytics
 At a glance           Is this project healthy right now?          Badges

CI/CD Analytics

Why it matters. A pipeline that slowly degrades never produces an incident. Five extra seconds per week, one more flaky test per month, and after a year the pipeline takes twice as long and fails one run in five. Developers adapt by retrying, waiting and batching changes, and the cost is invisible until someone measures it.

What GitLab provides. In a project, Analyze > CI/CD analytics shows pipeline statistics over a chosen period: the number of pipelines by status, success and failure rates, and duration trends, with filters such as branch and pipeline source. The page has been redesigned in recent releases, so the exact charts and labels depend on your version. At group level, the same menu shows release statistics (Premium) and, on Ultimate, DORA metrics (next topic). Related views elsewhere:

View	Shows	Section
Analyze > CI/CD analytics	Pipeline counts, success rate, durations over time	This topic
Analyze > Repository analytics	Code coverage history per job	10.2
Build > Pipelines, Tests tab	Test failures per pipeline, recently failed tests	10.1
Usage quotas	Compute minutes per project	13.2
Runner metrics in Prometheus	Queue, concurrency, failures per runner	6.4

Metrics worth tracking. The built-in charts cover the basics. These are the metrics that most often reveal real problems:

Metric	Why it matters	Healthy direction
Default-branch success rate	A red default branch blocks everyone who branches from it	High and stable
Time the default branch stays red	How quickly breakage is noticed and fixed	Short (minutes to hours)
Median and 95th-percentile duration	The median is the usual wait. The p95 is the wait that drives people to context-switch	Flat or falling
Queued duration	Runner capacity (section 13.1)	Near zero
Retry rate per job	Flakiness: jobs that pass only on retry	Low, and trending down

Measuring it yourself. For metrics the UI does not show, such as the time the default branch stays red, query the data directly. The GraphQL API returns pipeline durations in bulk, which the REST list endpoint does not:

#!/usr/bin/env python3
# ci_health.py: success rate, duration percentiles and "time red" for one branch
# Usage: GITLAB_TOKEN=glpat-... python3 ci_health.py shop/api main 2026-09-01T00:00:00Z
import json
import os
import statistics
import sys
import urllib.request
from datetime import datetime

URL = os.environ.get("GITLAB_GRAPHQL", "https://gitlab.com/api/graphql")
project, ref, since = sys.argv[1], sys.argv[2], sys.argv[3]
QUERY = """
query($project: ID!, $ref: String!, $since: Time!, $after: String) {
  project(fullPath: $project) {
    pipelines(ref: $ref, updatedAfter: $since, first: 100, after: $after) {
      nodes { status duration createdAt finishedAt }
      pageInfo { hasNextPage endCursor }
    }
  }
}"""

def fetch_all():
    after, nodes = None, []
    while True:
        payload = json.dumps({"query": QUERY, "variables":
                              {"project": project, "ref": ref, "since": since, "after": after}}).encode()
        req = urllib.request.Request(URL, data=payload, headers={
            "Authorization": f"Bearer {os.environ['GITLAB_TOKEN']}", "Content-Type": "application/json"})
        with urllib.request.urlopen(req) as resp:
            page = json.load(resp)["data"]["project"]["pipelines"]
        nodes += page["nodes"]
        if not page["pageInfo"]["hasNextPage"]:
            return nodes
        after = page["pageInfo"]["endCursor"]

def ts(value):
    return datetime.fromisoformat(value.replace("Z", "+00:00"))

done = [p for p in fetch_all() if p["status"] in ("SUCCESS", "FAILED") and p["finishedAt"]]
done.sort(key=lambda p: p["finishedAt"])                       # chronological order
durations = sorted(p["duration"] for p in done if p["duration"])

# "Time red": from the first failed pipeline to the next successful one
red_seconds, red_since = 0.0, None
for p in done:
    if p["status"] == "FAILED" and red_since is None:
        red_since = ts(p["finishedAt"])
    elif p["status"] == "SUCCESS" and red_since is not None:
        red_seconds += (ts(p["finishedAt"]) - red_since).total_seconds()
        red_since = None

success = sum(p["status"] == "SUCCESS" for p in done)
print(f"pipelines: {len(done)}, success rate: {success / len(done):.0%}")
print(f"duration p50: {statistics.median(durations) / 60:.1f} min, "
      f"p95: {durations[int(len(durations) * 0.95) - 1] / 60:.1f} min")
print(f"{ref} was red for {red_seconds / 3600:.1f} hours in total")

pipelines: 412, success rate: 87%
duration p50: 11.4 min, p95: 23.9 min
main was red for 31.5 hours in total

Run a script like this as a scheduled pipeline (section 3.5) and store the output as an artifact, or push the numbers to your monitoring system to get a trend line.

Key takeaways

• Analyze > CI/CD analytics shows pipeline counts, success rates and durations. Labels vary by version, and group-level views add release statistics and DORA.
• Track default-branch success rate, time red, median and p95 duration, queued time and retry rate.
• p95 duration is the wait that makes developers context-switch, so watch it alongside the median.
• Use the GraphQL API for bulk pipeline data (durations included), and schedule the analysis to get trends.

Practice

1. (Beginner) Open Analyze > CI/CD analytics for a project you work on and write down its success rate and typical duration for the last month. Compare the default branch with all branches.
2. (Intermediate) Run ci_health.py for your default branch, then extend it to count, per job name, how often a job failed and then passed on retry, using job data from the REST API. Name the three flakiest jobs.
3. Interview question: "Your manager wants one number for 'CI health' on a dashboard. Which would you choose, and what would it hide?" Hint: success rate versus time red versus p95 duration, and why any single metric can be gamed.

DORA Metrics

Why they exist. "Are we getting better at delivering software?" needs an answer that does not depend on opinion. The DevOps Research and Assessment (DORA) program found four metrics that, measured together, distinguish high-performing teams. Two measure throughput and two measure stability, so a team cannot improve one side by sacrificing the other without it showing (section 1.1).

How GitLab calculates them. GitLab computes all four from data it already has, mainly deployments to production-tier environments and incidents:

Metric	GitLab's definition	Data it needs
Deployment frequency	Number of successful deployments to production-tier environments per day	Jobs with environment: and deployment_tier: production (section 12.1)
Lead time for changes	Median time from a merge request being merged to its code running in production	MR merges, followed by production deployments
Time to restore service	Median time incidents stayed open in the period	Incidents (issues of type incident)
Change failure rate	Incidents divided by production deployments in the period	Incidents and deployments

Recent DORA reports rename "time to restore service" to "failed deployment recovery time", and GitLab's labels may follow. The meaning is the same.

 merge ──────────── lead time for changes ────────────► deployed to production ──┐
                                                                                 │ deployment frequency
                                                                                 ▼ counts these events
                        incident opened ── time to restore ──► incident closed
                        └── change failure rate = incidents / deployments ──┘

Feeding the metrics correctly. DORA dashboards are only as good as the events behind them:

deploy-production:
  stage: deploy
  script:
    - ./scripts/deploy.sh production
  environment:
    name: prod-eu                    # any name works...
    deployment_tier: production      # ...but only the tier makes it count for DORA
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

# Deployments done outside GitLab CI (for example by a legacy tool) can be recorded through the API
curl --request POST --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.example.com/api/v4/projects/12345/deployments" \
  --data "environment=production" --data "sha=$DEPLOYED_SHA" \
  --data "ref=main" --data "tag=false" --data "status=success"

# Open an incident when production breaks: time to restore runs until it is closed
curl --request POST --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.example.com/api/v4/projects/12345/issues" \
  --data "title=Checkout returns 500 after deploy" --data "issue_type=incident"

Incidents can also be created automatically from alerts (GitLab's alert integrations or an external monitoring tool calling the API), which removes the human delay from time to restore.

Reading the metrics. On Ultimate, DORA charts are available under Analyze > CI/CD analytics in projects and groups, and in the Value Streams Dashboard, which compares periods and groups and can classify projects into DORA performance levels. The DORA API returns the raw numbers:

# Monthly median lead time for changes (seconds) for one project
curl --silent --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.example.com/api/v4/projects/12345/dora/metrics?metric=lead_time_for_changes&interval=monthly&start_date=2026-04-01&end_date=2026-09-30" |
  jq -r '.[] | "\(.date)  \((.value // 0) / 3600 | floor) h"'

# Group-wide deployment frequency; other metrics: deployment_frequency, time_to_restore_service, change_failure_rate
curl --silent --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.example.com/api/v4/groups/42/dora/metrics?metric=deployment_frequency&interval=monthly&environment_tiers[]=production"

Common pitfalls.

Problem	Effect	Fix
Production environment without deployment_tier: production (for example live, eu-central)	Deployments not counted	Set the tier explicitly
Review apps or staging accidentally tiered as production	Deployment frequency inflated, lead time distorted	Audit tiers of all environments
Incidents tracked outside GitLab	Time to restore and change failure rate stay empty or near zero	Create incidents in GitLab, via alerts or the API
Direct pushes to the default branch, without MRs	Lead time misses those changes	Protect the branch, require MRs
Targets attached to individual metrics	People optimize the number, not the outcome (smaller, meaningless deployments; incidents not filed)	Look at all four together, as trends, at team level

Key takeaways

• DORA measures throughput (deployment frequency, lead time) and stability (time to restore, change failure rate). Read all four together.
• GitLab counts deployments to environments with deployment_tier: production, and uses incidents for the stability metrics.
• Record deployments made outside GitLab CI with the Deployments API, and create incidents through alerts or the Issues API.
• DORA dashboards and the DORA API require Ultimate. Use them for team-level trends, not individual targets.

Practice

1. (Beginner) List every environment in one of your projects with its deployment tier. Identify any that are misclassified for DORA purposes.
2. (Intermediate) Simulate a month of activity in a test project: ten production deployments through CI, two incidents opened and closed through the API. Query all four metrics with the DORA API and check that they match your manual calculation.
3. Interview question: "After introducing feature flags, your deployment frequency doubled but change failure rate also rose. How would you interpret this, and what would you look at next?" Hint: smaller batches, what counts as an incident, flag-related failures, and time to restore.

Value Stream Analytics

Why it exists. DORA's lead time starts at merge. For most teams, the larger delays come before that point: an issue waits weeks to be planned, a merge request waits days for review, a fix waits for the next release window. Value stream analytics (VSA) measures the time work spends in each step from idea to production, so you can see where it waits.

The model. A value stream is a sequence of stages. Each stage is defined by a start event and an end event, and GitLab measures, for every issue or merge request, the time between the two. The default value stream has six stages:

 Issue ───► Plan ───► Code ───► Test ───► Review ───► Staging
   │          │         │         │          │           │
 issue     added to   first     MR's last   MR created  MR merged
 created   board or   commit    pipeline    → MR        → first deployed
 → added   milestone  mention   started →   merged      to production
 to board/ → first    → MR      finished
 milestone commit     created
           mention

Stage	Start event	End event	What a long stage means
Issue	Issue created	Issue added to a board or milestone	Triage backlog
Plan	Issue added to a board or milestone	Issue first mentioned in a commit	Work waits before anyone starts it
Code	Issue first mentioned in a commit	Merge request created	Long-running branches
Test	MR's last pipeline started	MR's last pipeline finished	Slow CI (chapter 13)
Review	Merge request created	Merge request merged	Review bottleneck, slow approvals
Staging	Merge request merged	MR first deployed to production	Infrequent deployments, manual gates

Above the stages, VSA shows summary metrics such as lead time (issue created to closed), cycle time (first commit to issue closed), new issues and deploys. Ultimate adds deployment frequency here as well.

Linking matters. Stages are calculated from links between objects, so the data is only complete when work is connected. Use closing patterns (Closes #42) in merge request descriptions or commit messages, mention issues in commits, and deploy through environments. An MR that is not linked to an issue contributes to Review and Staging, but not to Plan or Code.

Custom value streams. On Premium and above, you can create value streams at group level with your own stages. Start and end events include issue and MR creation, closing and merging, label added or removed, first commit time, last pipeline start and finish, and first deployment to production. For example:

Custom stage	Start event	End event	Purpose
Waiting for review	Label workflow::ready-for-review added	Label workflow::ready-for-review removed	Measure review queue time precisely
CI feedback	Merge request created	MR's last pipeline finished	First feedback a developer gets
Release wait	Merge request merged	MR first deployed to production	Time code sits on main before shipping

Label-based stages are the most flexible: if your workflow uses scoped labels for states, VSA can measure the time spent in each state.

Reading VSA responsibly. Each stage shows the median time and the list of items, sortable by duration. Look at the slowest items, not only at the median: a handful of items that waited three weeks usually tell you more than the median does. As with DORA, use the data to find bottlenecks in the process, not to rank people.

Key takeaways

• VSA measures time from idea to production as a chain of stages, each defined by a start and an end event.
• The default stages (Issue, Plan, Code, Test, Review, Staging) need linked work: closing patterns, commit mentions and environment deployments.
• The Test stage reflects CI duration, and the Staging stage reflects deployment frequency and release gates.
• Custom value streams (Premium) with label-based stages model your team's actual workflow.

Practice

1. (Beginner) Open Analyze > Value stream analytics for a project and find its slowest stage. Open the three slowest items in that stage and describe why each one waited.
2. (Intermediate) Design a custom value stream for a team that uses the scoped labels workflow::in-progress, workflow::review and workflow::done. Define the start and end events of each stage.
3. Interview question: "DORA lead time for changes is under a day, yet customers wait months for requested features. How can both be true, and which tool shows the gap?" Hint: lead time starts at merge, and the Issue and Plan stages of VSA.

Pipeline and Coverage Badges

Why they exist. A badge is a tiny image, usually at the top of a README, that shows a project's status at a glance: whether the default branch is passing, how much code is covered by tests, what the latest release is. It costs nothing to maintain, and it makes a broken default branch visible to everyone who opens the project.

Built-in badges. GitLab generates three badges as SVG images:

Badge	Image URL	Options
Pipeline status	https://<host>/<project>/badges/<ref>/pipeline.svg	ignore_skipped=true shows the last non-skipped status, key_text and key_width change the label
Coverage	https://<host>/<project>/badges/<ref>/coverage.svg	job=<name> picks one job's coverage, min_good, min_acceptable and min_medium set the color thresholds (defaults 95, 90, 75)
Latest release	https://<host>/<project>/-/badges/release.svg	order_by=release_at instead of creation date

The coverage badge shows the value extracted by the coverage keyword (section 10.2). Without job=, it shows the average across all jobs that report coverage.

<!-- README.md -->
[![pipeline status](https://gitlab.example.com/shop/api/badges/main/pipeline.svg)](https://gitlab.example.com/shop/api/-/commits/main)
[![coverage report](https://gitlab.example.com/shop/api/badges/main/coverage.svg?job=unit-tests&min_good=85&min_acceptable=75)](https://gitlab.example.com/shop/api/-/commits/main)
[![latest release](https://gitlab.example.com/shop/api/-/badges/release.svg)](https://gitlab.example.com/shop/api/-/releases)

Project and group badges. Instead of editing every README, add badges under Settings > General > Badges. They appear in the project header. Group badges appear on every project in the group, and placeholders make one definition work everywhere:

Field	Value
Name	pipeline
Link	https://gitlab.example.com/%{project_path}/-/commits/%{default_branch}
Badge image URL	https://gitlab.example.com/%{project_path}/badges/%{default_branch}/pipeline.svg

Other placeholders include %{project_id}, %{project_name}, %{commit_sha} and %{latest_tag}. Badges can also be managed through the API (POST /projects/:id/badges with link_url and image_url, or /groups/:id/badges).

Custom badges from a job. For any other metric, such as lint score, bundle size or dependency freshness, generate an SVG in a job, store it as an artifact, and reference the latest artifact by URL:

lint-badge:
  stage: test
  image: python:3.12-slim
  script:
    - pip install anybadge pylint
    # Extract the score from "Your code has been rated at 8.73/10"
    - score=$(pylint src --exit-zero | sed -n 's/^Your code has been rated at \([-0-9.]*\)\/.*/\1/p')
    - anybadge --label=pylint --value="$score" --file=pylint.svg 4=red 6=orange 8=yellow 10=green
  artifacts:
    paths: [pylint.svg]              # the latest artifacts of a ref are kept by default
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

![pylint](https://gitlab.example.com/shop/api/-/jobs/artifacts/main/raw/pylint.svg?job=lint-badge)

Visibility. Badge URLs follow the project's visibility. For a private or internal project, the image loads only for signed-in users with access. A badge copied into an external site, such as a public mirror's README, shows a broken image.

Key takeaways

• GitLab serves pipeline, coverage and latest-release badges as SVGs under /badges/<ref>/... and /-/badges/release.svg.
• Use job= and the min_* thresholds to make the coverage badge show the right job with meaningful colors.
• Group badges with placeholders such as %{project_path} and %{default_branch} give every project the same badges with one definition.
• Custom metrics become badges through an SVG artifact and the /-/jobs/artifacts/<ref>/raw/<path>?job=<name> URL.
• Badges of private projects do not render for anonymous viewers.

Practice

1. (Beginner) Add pipeline and coverage badges to a project's README, with the coverage badge pointing at one specific job and custom color thresholds.
2. (Intermediate) Define a group-level pipeline badge with placeholders, and check that it renders correctly in three projects with different default branch names.
3. Interview question: "Your README's pipeline badge says 'passed', but the last commit on main has a failed pipeline. What could explain it?" Hint: which ref the badge points to, skipped and canceled pipelines, ignore_skipped, and image caching by the browser or a proxy.

<a id="143-notifications-and-integrations"></a>

14.3 Notifications and Integrations

A failure that nobody notices is worse than one that is noticed late. This subchapter covers how GitLab tells people and systems what happened: email, chat integrations, webhooks for your own automation, and commit statuses that connect GitLab with external CI systems and other Git hosts. The common goal is the right event, to the right audience, with as little noise as possible.

Mechanism	Audience	Direction	Typical use
Email notifications	People	GitLab → inbox	"Your pipeline failed"
Slack / Microsoft Teams	Teams	GitLab → channel	"main is broken", deployment announcements
Webhooks	Systems	GitLab → your HTTP service	Custom automation, dashboards, ticketing
Commit Status API	Systems	External system → GitLab	External CI or checks shown in the pipeline
External status checks	Systems	GitLab ↔ external service	Merge request approval by an external tool

Pipeline Email Notifications

Why it matters. Email is the default channel: it reaches everyone without setup. It is also the easiest one to ruin with noise. A developer who receives fifty pipeline emails a day filters them all away, including the one that matters.

Two independent mechanisms.

1. Personal notifications. Each user receives pipeline emails based on their own notification settings (User settings > Notifications, per group or project):

Event	Who receives it	Default
Failed pipeline	The user who triggered the pipeline	On
Fixed pipeline	The user who triggered the pipeline, when a pipeline succeeds after a failure	On
Successful pipeline	The user who triggered the pipeline	Off: enable it with the Custom notification level

"The user who triggered the pipeline" is the pusher, the person who clicked Run pipeline, or the owner of a schedule for scheduled pipelines. Nightly failures therefore go to whoever owns the schedule, which is a common reason why nobody notices them. Assign schedule ownership to someone who acts on the results (section 3.5).

2. The Pipeline status emails integration. Under Settings > Integrations > Pipeline status emails, a project sends pipeline results to a fixed list of addresses, such as a team mailing list, regardless of who triggered the pipeline:

Setting	Purpose
Recipients	Comma-separated addresses (for example a team distribution list)
Notify only broken pipelines	Skip successful pipelines
Branches for which notifications are sent	All, default, protected, or default and protected branches

# Configure the integration through the API: only broken pipelines on default and protected branches
curl --request PUT --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.example.com/api/v4/projects/12345/integrations/pipelines-email" \
  --data "recipients=team-checkout@example.com" \
  --data "notify_only_broken_pipelines=true" \
  --data "branches_to_be_notified=default_and_protected"

Filtering on the receiving side. GitLab emails carry headers you can filter on, such as X-GitLab-Project, X-GitLab-Project-Path, X-GitLab-Pipeline-Id and X-GitLab-NotificationReason. A Sieve rule (supported by many mail servers) can sort CI mail into its own folder:

require ["fileinto"];
# Pipeline emails from the checkout service go to their own folder
if allof (exists "X-GitLab-Pipeline-Id",
          header :is "X-GitLab-Project-Path" "shop/checkout") {
  fileinto "CI/checkout";
}

Reducing noise.

• Keep "Failed pipeline" and "Fixed pipeline" on, and "Successful pipeline" off.
• Send team-wide email only for broken pipelines on default and protected branches. Feature-branch failures belong to their author.
• Avoid noise sources: duplicate pipelines (section 4.4) and flaky jobs both produce emails nobody needs.

Key takeaways

• Personal notifications go to the user who triggered the pipeline: the pusher, the person who ran it, or the schedule owner.
• Failed and fixed pipeline emails are on by default. Successful pipeline emails need the Custom notification level.
• The Pipeline status emails integration sends results to fixed recipients, filtered to broken pipelines and selected branches.
• Use X-GitLab-* headers to filter mail, and remove noise at the source (duplicates, flakiness).

Practice

1. (Beginner) Set your notification level for one project to Custom with only Failed and Fixed pipeline enabled. Break and fix a pipeline, and check which emails arrive.
2. (Intermediate) Configure Pipeline status emails for a team address, limited to broken pipelines on the default branch. Then create a scheduled pipeline owned by a bot or service account and work out who is notified when it fails.
3. Interview question: "A nightly security scan has been failing for three weeks and nobody noticed. What went wrong, and how do you prevent it from happening again?" Hint: schedule ownership, who receives failure emails, team-level notifications and chat alerts.

Slack and Microsoft Teams Integrations

Why they exist. Teams coordinate in chat. A failed deployment posted in the team channel is seen within minutes, discussed in a thread and fixed, while an email to one person may sit unread. Chat integrations also make events visible to people who did not trigger them, which is what you want for the default branch and production.

Slack. The recommended integration is the GitLab for Slack app. It sends notifications to channels and adds slash commands. On GitLab.com you install it from the project's Settings > Integrations. On self-managed instances, an administrator must set it up first. The older Slack notifications integration, based on Slack incoming webhooks, is deprecated in favor of the app.

Microsoft Teams. The Microsoft Teams notifications integration posts to a channel through an incoming webhook URL that you create in Teams. Microsoft is retiring the older Office 365 connectors, so create the webhook with the Teams Workflows app and follow GitLab's documentation for the current steps.

What to configure. Both integrations offer similar options:

Option	Recommended setting
Trigger events	Pipeline and deployment events, and whatever else the channel's audience needs. Avoid push events in busy projects
Notify only broken pipelines	On, for team channels
Branches for which notifications are sent	Default branch, or default and protected branches
Channel per event (Slack)	Separate channels for deployments and failures, so one does not drown out the other

# Microsoft Teams: notify a channel about broken pipelines on default and protected branches only
curl --request PUT --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.example.com/api/v4/projects/12345/integrations/microsoft-teams" \
  --data-urlencode "webhook=$TEAMS_WEBHOOK_URL" \
  --data "pipeline_events=true" \
  --data "push_events=false" \
  --data "notify_only_broken_pipelines=true" \
  --data "branches_to_be_notified=default_and_protected"

ChatOps with Slack slash commands. The GitLab for Slack app adds /gitlab commands, so that people can act without leaving the chat:

/gitlab shop/api issue show 42                  # show an issue in the channel
/gitlab shop/api deploy staging to production   # run the manual production job of the latest staging deployment
/gitlab shop/api run smoke-test                 # ChatOps: run a CI job whose output is posted back

ChatOps commands run CI jobs with the permissions of the GitLab user linked to the Slack account, so protected branch and environment rules still apply.

Custom messages from a job. Integrations send generic event messages. For a tailored message, such as a deployment announcement with the version and changelog link, post to a webhook from a job. This works with any chat tool that accepts webhooks:

announce-deploy:
  stage: .post
  image:
    name: curlimages/curl:8.10.1
    entrypoint: [""]                 # the image's entrypoint is curl itself; reset it for the runner
  rules:
    - if: $CI_COMMIT_TAG             # only for release pipelines
  script:
    - |
      curl --fail --silent --show-error -X POST -H 'Content-Type: application/json' \
        --data "{\"text\": \"Deployed ${CI_PROJECT_PATH} ${CI_COMMIT_TAG} to production: ${CI_PIPELINE_URL}\"}" \
        "$SLACK_WEBHOOK_URL"         # masked, protected variable (section 5.2)

Build the JSON with jq if the message contains free text, such as a commit title, so that quotes and newlines are escaped correctly.

Key takeaways

• Use the GitLab for Slack app (the webhook-based Slack integration is deprecated) and the Microsoft Teams notifications integration (with a Workflows webhook).
• Limit team channels to broken pipelines on default and protected branches, and to deployments.
• Slack slash commands enable ChatOps, which respects the linked user's permissions.
• For tailored messages, post to a chat webhook from a job, with the webhook URL in a masked, protected variable.

Practice

1. (Beginner) Connect a test project to a Slack or Teams channel and enable pipeline events for broken pipelines on the default branch only. Break the default branch and check the message.
2. (Intermediate) Write a job that runs only when a production deployment fails (when: on_failure in its rules) and posts a message to a chat webhook containing the environment, the failed job name and a link to the pipeline. Use jq to build the payload.
3. Interview question: "Your deployment channel receives so many notifications that people muted it. How would you redesign the notification setup?" Hint: events per channel, broken-only, branch filters, deployments versus pipelines, and actionable messages.

Pipeline Webhooks

Why they exist. Built-in integrations cover common tools. Everything else needs a hook: opening a ticket in your service desk when production deployment fails, updating an internal dashboard, triggering a job in another CI system, collecting pipeline data for analytics. Webhooks let GitLab call your HTTP endpoint whenever an event happens, so you do not have to poll the API.

What is sent. Under Settings > Webhooks you choose a URL, a secret token and the trigger events. For CI/CD, the relevant triggers are:

Trigger	X-Gitlab-Event header	object_kind	Fires when
Pipeline events	Pipeline Hook	pipeline	A pipeline's status changes (created, running, success, failed, canceled...)
Job events	Job Hook	build	A job's status changes
Deployment events	Deployment Hook	deployment	A deployment starts, succeeds, fails or is canceled
Release events	Release Hook	release	A release is created, updated or deleted

A pipeline event payload (abridged) looks like this:

{
  "object_kind": "pipeline",
  "object_attributes": {
    "id": 98765,
    "ref": "main",
    "tag": false,
    "sha": "1a2b3c4d5e6f...",
    "source": "push",
    "status": "failed",
    "duration": 412,
    "queued_duration": 9,
    "url": "https://gitlab.example.com/shop/api/-/pipelines/98765"
  },
  "user": { "username": "alice", "name": "Alice Example" },
  "project": { "path_with_namespace": "shop/api", "default_branch": "main" },
  "commit": { "title": "Apply discount once", "url": "https://gitlab.example.com/shop/api/-/commit/1a2b3c4d" },
  "builds": [
    { "name": "unit-tests", "stage": "test", "status": "failed", "failure_reason": "script_failure" },
    { "name": "build", "stage": "build", "status": "success" }
  ]
}

Receiving webhooks safely. A webhook receiver is a public endpoint that triggers actions, so treat it like any other API:

1. Verify the secret token. GitLab sends it in the X-Gitlab-Token header. Compare it in constant time and reject mismatches.
2. Respond quickly. GitLab waits only a few seconds (10 by default) for a response. Acknowledge immediately and do the work asynchronously.
3. Expect duplicates and many events. One pipeline produces several events (pending, running, then a final status), and deliveries can be retried. Filter on the status you care about, and use the X-Gitlab-Event-UUID header (and Idempotency-Key in recent versions) to ignore repeats.
4. Filter in the receiver. Branch filters in the webhook settings apply to push events only. For pipelines, check ref, source (child pipelines have source: parent_pipeline) and status yourself.

#!/usr/bin/env python3
# hook_receiver.py: react to failed pipelines on the default branch
# Usage: GITLAB_WEBHOOK_SECRET=... python3 hook_receiver.py
import hmac
import json
import os
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer

SECRET = os.environ["GITLAB_WEBHOOK_SECRET"]
seen = set()                                              # event UUIDs already handled (in memory)

def handle(event):
    attrs, project = event["object_attributes"], event["project"]
    if attrs["status"] != "failed" or attrs["ref"] != project["default_branch"]:
        return                                            # only a red default branch matters here
    if attrs.get("source") == "parent_pipeline":
        return                                            # the parent pipeline reports it already
    failed = [b["name"] for b in event.get("builds", []) if b["status"] == "failed"]
    # Replace print with a call to your ticketing or paging system
    print(f"{project['path_with_namespace']}: {attrs['ref']} is red, "
          f"failed jobs: {', '.join(failed)} ({attrs['url']})")

class Hook(BaseHTTPRequestHandler):
    def do_POST(self):
        token = self.headers.get("X-Gitlab-Token", "")
        if not hmac.compare_digest(token, SECRET):        # constant-time comparison
            self.send_response(401)
            self.end_headers()
            return
        body = self.rfile.read(int(self.headers["Content-Length"]))
        self.send_response(200)                           # acknowledge first...
        self.end_headers()
        uuid = self.headers.get("X-Gitlab-Event-UUID")
        if self.headers.get("X-Gitlab-Event") == "Pipeline Hook" and uuid not in seen:
            seen.add(uuid)
            threading.Thread(target=handle, args=(json.loads(body),)).start()   # ...then work

HTTPServer(("0.0.0.0", 8000), Hook).serve_forever()

# Register the webhook for pipeline events only
curl --request POST --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.example.com/api/v4/projects/12345/hooks" \
  --data "url=https://hooks.example.com/gitlab" \
  --data "token=$GITLAB_WEBHOOK_SECRET" \
  --data "pipeline_events=true" \
  --data "push_events=false"

Operating webhooks.

• Recent events. Each webhook's edit page lists recent deliveries, with request, response and status code, and can resend them. It is the first place to look when "the hook didn't fire".
• Test. The Test button sends a sample event of the chosen type, which is useful while building a receiver.
• Automatic disabling. Webhooks that keep failing are disabled temporarily, and permanently after many consecutive failures. Monitor your receiver and re-enable the hook after fixing it.
• Scope. Project webhooks are available on all tiers. Group webhooks (Premium) receive events from every project in the group, which suits central tooling. Instance-wide system hooks are an administrator feature and do not include pipeline events.

Key takeaways

• Pipeline, job, deployment and release events can be delivered to any HTTP endpoint as JSON, identified by the X-Gitlab-Event header.
• Verify X-Gitlab-Token in constant time, respond within seconds, and process events asynchronously.
• Expect several events per pipeline and possible repeats. Filter by status, ref and source, and deduplicate with the event UUID.
• Use Recent events to debug deliveries. Repeatedly failing webhooks are disabled automatically.
• Group webhooks (Premium) centralize automation across many projects.

Practice

1. (Beginner) Point a project webhook with pipeline events at a request inspection service you control (or the receiver above, exposed through a tunnel). Run a pipeline and count how many events it produces.
2. (Intermediate) Extend hook_receiver.py to compute, per project, the time from a default-branch pipeline failing to the next successful one, using only webhook events. Persist state so that a restart does not lose it.
3. Interview question: "You built a webhook receiver that creates a ticket for every failed production deployment, but sometimes two tickets are created for the same failure. Why, and how do you fix it?" Hint: retries, multiple event types, idempotency keys, and acknowledging before processing.

External Commit Status Reporting

Why it exists. Not every check runs in GitLab CI. A legacy Jenkins job, a performance lab, a third-party scanner or a hardware test rig may validate the same commit. Their results belong next to GitLab's own jobs, in the pipeline and the merge request, and must be able to block a merge. In the other direction, when GitLab CI builds code hosted elsewhere (for example on GitHub), its results must show up on that platform. Commit statuses connect the two.

Options.

Mechanism	Direction	Attaches to	Can block merging	Tier
Commit Status API	External system → GitLab	A commit (appears as a job in the pipeline)	Yes, through "Pipelines must succeed"	All
External status checks	GitLab → external service → GitLab	A merge request	Yes, with "Status checks must succeed"	Ultimate
CI/CD for external repositories + GitHub integration	GitLab CI → GitHub	A GitHub commit or pull request	Yes, through GitHub branch protection	Premium
Calling another platform's status API from a job	GitLab CI → anywhere	Whatever that platform supports	Depends on that platform	All

The Commit Status API. An external system posts a state for a commit SHA. GitLab shows each status as a job in an external stage of that commit's pipeline. If no pipeline exists for the SHA, GitLab creates one:

Pipeline #98765  (push, main)
     build              test                 external
 ┌────────────┐   ┌──────────────┐   ┌─────────────────────────┐
 │ (v) build  │──►│ (v) unit     │   │ (>) perf/load-test      │  <- posted by the perf lab,
 └────────────┘   └──────────────┘   │ (v) jenkins/integration │     links to its own UI
                                     └─────────────────────────┘

# Called by the external system: report that the load test started, then its result
API="https://gitlab.example.com/api/v4/projects/12345/statuses/$SHA"

curl --fail --request POST --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "$API" \
  --data-urlencode "state=running" \
  --data-urlencode "name=perf/load-test" \
  --data-urlencode "ref=feature/cart" \
  --data-urlencode "target_url=https://perf.example.com/runs/881" \
  --data-urlencode "description=Load test started"

# ... later ...
curl --fail --request POST --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "$API" \
  --data-urlencode "state=failed" \
  --data-urlencode "name=perf/load-test" \
  --data-urlencode "ref=feature/cart" \
  --data-urlencode "target_url=https://perf.example.com/runs/881" \
  --data-urlencode "description=p95 latency 640 ms exceeds budget of 400 ms"

The key fields are:

Field	Meaning
state	pending, running, success, failed, canceled or skipped
name (or context)	Identifies the check. Posting again with the same name updates that status
ref	The branch or tag. It determines which pipeline the status joins
pipeline_id	Attach to a specific pipeline when one SHA has several
target_url, description	Link to and summary of the external result, shown in GitLab
coverage	Optional coverage value, as for GitLab jobs

Two details often cause errors. First, states must follow a valid sequence: a status that is already success cannot go back to running, and the API rejects such a transition. Use a new name for a rerun, or post the final state directly. Second, the token's user must have at least the Developer role in the project. Use a project access token with the api scope for each external system, so that every status can be traced to its source.

Jenkins is the most common example: the GitLab plugin for Jenkins uses this API to report stage results back to GitLab, so the merge request shows Jenkins results alongside GitLab CI jobs.

External status checks (Ultimate). These work at the merge request level instead of the commit level. GitLab sends merge request data to a configured external service whenever the MR changes. The service evaluates it (for example a compliance or change-management system) and responds through the API:

# Called by the external service after evaluating merge request !128
curl --request POST --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.example.com/api/v4/projects/12345/merge_requests/128/status_check_responses" \
  --data "sha=$MR_HEAD_SHA" \
  --data "external_status_check_id=7" \
  --data "status=passed"         # or failed

Reporting GitLab results to other platforms. When the code lives on GitHub, CI/CD for external repositories (Premium) mirrors the repository into GitLab and runs pipelines there, and the GitHub integration posts pipeline statuses back to GitHub commits and pull requests. For other platforms, or for finer control, a job can call the platform's status API directly:

.report-github:
  stage: .post
  image:
    name: curlimages/curl:8.10.1
    entrypoint: [""]
  script:
    - |
      curl --fail --silent -X POST \
        -H "Authorization: Bearer $GITHUB_TOKEN" \
        -H "Accept: application/vnd.github+json" \
        "https://api.github.com/repos/acme/shop-api/statuses/$CI_COMMIT_SHA" \
        -d "{\"state\": \"$GH_STATE\", \"context\": \"gitlab-ci\", \"target_url\": \"$CI_PIPELINE_URL\"}"

report-success:
  extends: .report-github
  variables:
    GH_STATE: success
  rules:
    - when: on_success               # every job in earlier stages passed

report-failure:
  extends: .report-github
  variables:
    GH_STATE: failure
  rules:
    - when: on_failure               # at least one job in an earlier stage failed

GitLab has no predefined variable that holds the overall pipeline status during the run, so the pattern uses two jobs in .post and lets when: on_success and when: on_failure choose which one runs. The built-in GitHub integration does this for you.

Key takeaways

• The Commit Status API lets any external system add a check to a commit. It appears as a job in the external stage and counts for "Pipelines must succeed".
• Identify each check with a stable name, attach it to the right ref (or pipeline_id), and respect valid state transitions.
• Give each external system its own project access token with the api scope.
• External status checks (Ultimate) gate merge requests through an external service's pass or fail response.
• For code hosted on GitHub, CI/CD for external repositories with the GitHub integration (Premium) reports GitLab pipeline results back automatically.

Practice

1. (Beginner) Use the Commit Status API to post running and then success for a check named manual/review on the latest commit of a branch. Find it in the pipeline graph and in the merge request.
2. (Intermediate) Enable "Pipelines must succeed" on a project, then post a failed external status to an MR's head commit. Confirm that merging is blocked, and work out how the external system can unblock it.
3. Interview question: "Your company runs GitLab CI and a separate hardware-in-the-loop test lab. How would you make the lab's results block merge requests, and what would happen when the lab is offline?" Hint: commit status versus external status checks, pending states, timeouts, and who can override a blocked merge.

---

<a id="15-administration-automation-and-migration"></a>