10. Testing and Code Quality

A pipeline that only prints "Job succeeded" or "Job failed" makes reviewers dig through thousands of log lines to find out what broke. GitLab can read the machine-readable output of test runners, coverage tools, linters and performance tools, and show the results in the merge request where decisions are made: which tests failed for the first time, which changed lines are not covered, which new lint issues a change introduces, and whether a page got slower. This chapter covers the report types behind those views, how to produce them from common tools, and how to turn them into gates such as approval rules.

<a id="101-test-reporting"></a>

10.1 Test Reporting

Test reporting turns raw test runner output into structured results that GitLab can compare between branches, show in merge requests and track over time. This subchapter covers the JUnit report format, the merge request test summary, failure history on the default branch, and how to find and manage flaky tests.

JUnit Test Reports

Why it matters. When a test job fails, the log tells you that something failed, but finding the failing assertion in 5,000 lines of output is slow, and comparing the result with the target branch is impossible by eye. Test runners in almost every language can write their results as a JUnit XML file. GitLab reads that file and turns it into a list of test cases with status, duration and failure message, which is what makes every other feature in this subchapter possible.

What it is. JUnit XML is a de facto standard that started in the Java JUnit project and is now produced by pytest, Jest, Go tools, RSpec, PHPUnit, .NET loggers and many others. You declare the file with artifacts:reports:junit. The runner uploads it as a special report artifact and GitLab parses it.

A JUnit file is a tree of suites and cases:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <!-- One suite per test file, class or package, depending on the tool -->
  <testsuite name="tests.test_cart" tests="3" failures="1" errors="0" skipped="1" time="0.042">
    <!-- No child element: the test passed -->
    <testcase classname="tests.test_cart" name="test_add_item" file="tests/test_cart.py" time="0.010"/>

    <!-- <failure>: an assertion did not hold -->
    <testcase classname="tests.test_cart" name="test_total_with_discount" file="tests/test_cart.py" time="0.021">
      <failure message="AssertionError: assert 90 == 85">tests/test_cart.py:27: AssertionError</failure>
      <!-- Attachment syntax: GitLab links this file in the test details -->
      <system-out>[[ATTACHMENT|screenshots/test_total_with_discount.png]]</system-out>
    </testcase>

    <!-- <skipped>: not executed -->
    <testcase classname="tests.test_cart" name="test_currency" time="0.000">
      <skipped message="multi-currency not implemented"/>
    </testcase>
  </testsuite>
</testsuites>

XML child of <testcase>	GitLab status	Typical meaning
none	success	The test passed
<failure>	failed	An assertion failed
<error>	error	The test crashed before an assertion (exception, setup failure)
<skipped>	skipped	The test was not executed

Configuring the job.

unit-tests:
  stage: test
  image: python:3.12-slim
  script:
    - pip install -r requirements.txt
    - pytest --junitxml=reports/junit.xml    # pytest writes JUnit XML natively
  artifacts:
    when: always                              # keep the browsable files (paths) when tests fail too
    paths:
      - reports/                              # optional: lets you download or browse the raw XML
      - screenshots/                          # files referenced with [[ATTACHMENT|...]]
    reports:
      junit: reports/junit.xml                # parsed by GitLab
    expire_in: 2 weeks                        # reports vanish from the UI when the artifact expires

Report artifacts declared under artifacts:reports are always uploaded, whether the job succeeds or fails. Files under artifacts:paths follow artifacts:when, which defaults to on_success. Set when: always so that the raw XML and screenshots are also kept after a failure, when you need them most.

Producing JUnit XML with common tools.

Language / tool	Command or setting	Report path
Python, pytest	pytest --junitxml=report.xml	report.xml
JavaScript, Jest	jest --ci --reporters=default --reporters=jest-junit (package jest-junit)	junit.xml (change with JEST_JUNIT_OUTPUT_DIR)
Java, Maven Surefire	Writes reports by default	target/surefire-reports/TEST-*.xml
Java, Gradle	Writes reports by default	build/test-results/test/**/TEST-*.xml
Go	gotestsum --junitfile report.xml	report.xml
Ruby, RSpec	rspec --format RspecJunitFormatter --out rspec.xml (gem rspec_junit_formatter)	rspec.xml
.NET	dotnet test --logger "junit;LogFilePath=report.xml" (package JunitXml.TestLogger)	report.xml

junit accepts a single path, a wildcard, or a list. Several files are combined into one report:

java-tests:
  image: maven:3.9-eclipse-temurin-21
  script:
    - mvn --batch-mode verify
  artifacts:
    when: always
    reports:
      junit:
        - "**/target/surefire-reports/TEST-*.xml"    # every module of a multi-module build
        - "**/target/failsafe-reports/TEST-*.xml"    # integration tests

Rules and limits.

• JUnit reports do not change the job status. The job fails only if the script exits with a non-zero code. A test runner that exits 0 despite failures produces a green job with failed tests in the report.
• Each JUnit file must be smaller than 30 MB, and the total per job smaller than 100 MB. Larger reports are not shown.
• If two test cases have the same name and class, only the first is used. Make names unique, especially in parameterized tests.
• Expired artifacts mean an empty report in the UI. Choose expire_in to cover your review cycle.
• Reports from all jobs of a pipeline, including child pipelines, are combined in the pipeline's Tests tab. Each suite is listed with its counts and duration, and each failed case can be opened for its message, stack trace, file and attachment.

Pipeline #98765  >  Tests tab
+--------------------+-------+--------+---------+---------+----------+
| Job / suite        | Tests | Failed | Errors  | Skipped | Duration |
+--------------------+-------+--------+---------+---------+----------+
| unit-tests         |   412 |      1 |       0 |       3 |   38.2 s |
| java-tests         |   980 |      0 |       0 |      12 |  2m 41 s |
| e2e 1/2            |    64 |      2 |       1 |       0 |  4m 05 s |
+--------------------+-------+--------+---------+---------+----------+

You can also read the parsed report through the API, which is useful for scripts and dashboards:

# Summary per suite: totals, failed, skipped, errors
curl --silent --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.example.com/api/v4/projects/12345/pipelines/98765/test_report_summary" | jq '.total'

# Full report: print every failed test case as "classname.name"
curl --silent --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.example.com/api/v4/projects/12345/pipelines/98765/test_report" \
  | jq -r '.test_suites[].test_cases[] | select(.status == "failed") | "\(.classname).\(.name)"'

Key takeaways

• Declare JUnit XML files with artifacts:reports:junit. GitLab parses them into test cases with status, duration and failure details.
• Report artifacts are uploaded even when the job fails. Use when: always so that raw files and screenshots under paths are uploaded too.
• The report never changes the job result. The test command's exit code does.
• Watch the limits: 30 MB per file, 100 MB per job, unique test names, and artifact expiry.

Practice

1. (Beginner) Add artifacts:reports:junit to an existing test job in your language of choice, make one test fail on purpose, and find the failure message in the pipeline's Tests tab.
2. (Intermediate) For an end-to-end test job, write a screenshot on failure and reference it with [[ATTACHMENT|...]] in <system-out>. Configure artifacts so that both the report and the screenshot are available after a failed run.
3. Interview question: "A pipeline is green, but the Tests tab shows 4 failed tests. How is that possible, and how would you fix it?" Hint: what decides the job status, how the test command's exit code is handled (|| true, wrappers, runner flags), and allow_failure.

Test Summary in Merge Requests

Why it matters. A reviewer does not need to know that 3 of 2,000 tests fail. They need to know whether this merge request broke them. A test that already failed on main yesterday is a different problem from one that this branch broke. The merge request test summary answers that question by comparing the branch's test report with the target branch's.

What it is. When the latest pipeline of a merge request contains at least one job with a JUnit report, the merge request shows a Test summary panel. GitLab compares the results with the report of a baseline pipeline on the target branch and sorts the differences:

Category	Target branch	Source branch	What it tells the reviewer
Newly failed	passed	failed	Most likely caused by this merge request
Newly encountered errors	passed	error	Most likely caused by this merge request (crash, setup failure)
Existing failures	failed	failed	Already broken before this change
Resolved failures	failed	passed	This change fixes them

+------------------------------------------------------------------+
| Test summary: 2 failed, 1 fixed test result out of 1456 total    |
|                                                                  |
|  Newly failed                                                    |
|   x  tests.test_cart.test_total_with_discount       [View details]|
|  Existing failures                                               |
|   x  tests.test_tax.test_rounding_eu                [View details]|
|  Resolved failures                                               |
|   v  tests.test_cart.test_empty_cart                [View details]|
|                                                                  |
|  [Copy failed tests]                                [Full report]|
+------------------------------------------------------------------+

• View details opens the failure message, stack trace, file, attachments and the failure history (next topic).
• Copy failed tests copies the failed test names, ready to paste into a local command to re-run them.
• Full report opens the pipeline's Tests tab.

Conditions for the comparison.

• The target branch must have a pipeline with test reports to compare against. If it does not, the panel shows only the failed tests of the branch, without categories.
• The report must come from the latest pipeline of the merge request. A merge request whose newest pipeline skipped the test jobs (for example because of rules:changes) or was not created (a [skip ci] commit, or [skip ci] in the title) shows no panel.
• Report artifacts must not have expired, on either side.

The most common cause of a misleading summary is a target branch that does not run the same test jobs. If main never uploads JUnit reports, every merge request sees only "failed" without context. Make sure the test jobs run with reports on the default branch too:

unit-tests:
  script:
    - pytest --junitxml=report.xml
  artifacts:
    when: always
    reports:
      junit: report.xml
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"   # compared side
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH         # baseline side: required for the categories

Key takeaways

• The test summary compares the merge request's report with a baseline from the target branch and splits results into newly failed, newly erroring, existing and resolved.
• "Newly failed" is the category that matters most in review. "Existing failures" points to problems that were already on the target branch.
• The target branch must run the same reporting jobs, and the latest MR pipeline must contain them.
• Use Copy failed tests to reproduce failures locally, and Full report to see the complete results.

Practice

1. (Beginner) Open a merge request that breaks one test and fixes another that already fails on main. Identify each test's category in the Test summary.
2. (Intermediate) Configure a pipeline in which the test job runs only in merge request pipelines and observe the panel. Then add the default branch to the job's rules, run a pipeline on main, re-run the MR pipeline, and compare the panel.
3. Interview question: "Developers complain that the merge request test summary always shows failures they did not cause. What would you check?" Hint: the baseline pipeline on the target branch, whether its tests already fail there, and expired artifacts.

Unit Test Failure History

Why it matters. A test fails in your merge request. Did you break it, or has it been failing on and off for a week? Without history you can spend an hour debugging code that has nothing to do with the failure. GitLab answers this directly in the test details.

What it is. For a failed test, GitLab counts how often the same test (matched by name and class) failed on the default branch in the last 14 days, and shows a message such as:

tests.test_tax.test_rounding_eu
Failed 9 times in main in the last 14 days

The history appears in the details of a failed test, which you open from the merge request test summary or from the pipeline's Tests tab. It counts completed pipelines on the default branch and excludes blocked pipelines (for example pipelines waiting at a manual job). It is available in all tiers.

Reading the signal.

What you see	Likely interpretation	Next step
No history (0 failures on the default branch)	The failure is new, so most likely caused by this branch	Debug your change
A few failures, all recent	A regression reached the default branch recently	Find the commit that broke it, check open issues
Failures spread across many days, with passes in between	Probably a flaky test	Follow the flaky test process (next topic)
Failing in every pipeline	The default branch is broken	Fix or revert on the default branch first

Prerequisite. History only exists if the default branch runs the test jobs with artifacts:reports:junit. Projects that run tests only in merge request pipelines never build a history. Scheduled pipelines on the default branch (for example nightly) add more data points and make flaky tests visible sooner.

For longer windows or other branches, build your own history from the test report API, for example in a scheduled job:

#!/usr/bin/env bash
# Count failures per test over the last 50 finished pipelines on main
set -euo pipefail
API="https://gitlab.example.com/api/v4/projects/12345"
AUTH="PRIVATE-TOKEN: $GITLAB_TOKEN"

# IDs of the most recent finished pipelines on main
ids=$(curl --silent --header "$AUTH" \
  "$API/pipelines?ref=main&scope=finished&per_page=50" | jq -r '.[].id')

for id in $ids; do
  curl --silent --header "$AUTH" "$API/pipelines/$id/test_report" \
    | jq -r '.test_suites[].test_cases[] | select(.status == "failed") | "\(.classname).\(.name)"'
done | sort | uniq --count | sort --reverse --numeric-sort | head -20   # top 20 failing tests

Key takeaways

• Failure history shows how often a failed test failed on the default branch in the last 14 days.
• No history suggests that your change caused the failure. Scattered history suggests a flaky test or a pre-existing problem.
• It only works if the default branch runs the test jobs with JUnit reports. Scheduled pipelines add data.
• For other windows or branches, aggregate the pipeline test report API yourself.

Practice

1. (Beginner) Make a test fail on the default branch in a few pipelines, then fail it in a merge request. Open View details and read the failure history message.
2. (Intermediate) Turn the script above into a scheduled CI job that writes the top failing tests to a Markdown artifact, and open an issue automatically when a test failed more than 5 times.
3. Interview question: "Your test fails in a merge request, and the details say it failed 11 times on main in the last 14 days. How do you decide whether to block the merge?" Hint: compare the failure message with the ones on main, check whether your change touches that area, and what the team's flaky test policy says.

Flaky Test Detection

Why it matters. A flaky test passes and fails on the same code. Each flaky failure costs a re-run and some trust. When "just retry it" becomes the team's reflex, real failures get retried away too. Flaky tests are a technical debt that grows quietly until nobody trusts a red pipeline.

What it is. Flakiness usually comes from things the test does not control:

Cause	Example	Typical fix
Timing and async	sleep(1) before checking a UI element	Wait for a condition, not for a fixed time
Order dependence	Test B passes only if test A ran first and left data behind	Isolate state, run tests in random order
Shared resources	Two parallel jobs use the same database or port	One database per job (services, section 7.2), random ports
External systems	Calls to a real third-party API	Mocks, contract tests, local test doubles
Time and randomness	Tests that fail around midnight or with certain random seeds	Freeze time, fix and log the seed
Resource limits	Timeouts on a busy shared runner	Realistic timeouts, right-sized runners

GitLab has no single "flaky" switch. You combine several signals.

1. Detect.

• Failure history (previous topic): failures spread across many days on the default branch are the strongest signal.
• Pass on retry: a job that fails, then passes on retry with the same commit SHA, is flaky by definition. The pipeline UI shows retried jobs, and the pipeline jobs API lists them with include_retried=true.
• Stress runs: run suspect tests many times in a scheduled pipeline to reproduce flakiness on purpose.

# Scheduled "flaky hunt": run the suite repeatedly, in random order, without failing the pipeline
flaky-hunt:
  stage: test
  image: python:3.12-slim
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule" && $SCHEDULE_TYPE == "flaky-hunt"
  parallel: 5                                        # 5 independent runs at the same time
  script:
    - pip install -r requirements.txt pytest-repeat pytest-randomly
    # --count runs each test 10 times; pytest-randomly shuffles order and prints the seed
    - pytest --count=10 --junitxml=reports/flaky-$CI_NODE_INDEX.xml || true
  artifacts:
    when: always
    reports:
      junit: reports/flaky-*.xml

2. Contain. Retrying hides symptoms, so limit it. GitLab's retry keyword (section 4.3) re-runs the whole job. Test-level retries in the runner (pytest-rerunfailures, Jest jest.retryTimes, Playwright retries) re-run only the failing test, which is faster and keeps the job log readable.

e2e-tests:
  script:
    - npx playwright test           # playwright.config.ts: retries: process.env.CI ? 1 : 0
  retry:
    max: 1                          # job-level retry only for infrastructure problems
    when:
      - runner_system_failure
      - stuck_or_timeout_failure    # never "script_failure": that would retry real test failures

3. Quarantine. Move known flaky tests out of the blocking path, but keep running them so you know when they are fixed:

unit-tests:
  script:
    - pytest -m "not quarantine" --junitxml=report.xml     # blocking: stable tests only
  artifacts:
    reports:
      junit: report.xml

quarantined-tests:
  allow_failure: true                                      # visible as a warning, never blocks
  script:
    - pytest -m quarantine --junitxml=quarantine.xml
  artifacts:
    when: always
    reports:
      junit: quarantine.xml

import pytest

@pytest.mark.quarantine            # registered in pytest.ini: markers = quarantine: known flaky
# Tracking issue: https://gitlab.example.com/shop/api/-/issues/812
def test_checkout_sends_confirmation_email():
    ...

4. Fix and track. Open an issue for each quarantined test with a label such as flaky-test, an owner, and a deadline. A quarantine without an exit plan is just deletion with extra steps.

detect                contain               quarantine              fix
(history, retries) -> (test-level retry, -> (marker + issue,    -> (root cause,
                       job retry only for     non-blocking job)      remove marker,
                       infra failures)                               close issue)

Key takeaways

• A flaky test changes result on the same code. Detect it through failure history, retries that pass, and stress runs.
• Prefer test-level retries to job-level retry. Never retry on script_failure as a fix for flakiness.
• Quarantine flaky tests into a non-blocking job, and track each one in an issue with an owner.
• Most flakiness comes from timing, shared state, order, external systems and resource limits.

Practice

1. (Beginner) Write a test that fails randomly about 30% of the time. Run it in a job with retry: 2 and observe how often the problem is hidden.
2. (Intermediate) Implement the quarantine pattern above for your stack (pytest markers, Jest test name patterns or Playwright tags), and add a scheduled job that reports when a quarantined test has passed 20 times in a row.
3. Interview question: "Your team's merge pipelines are red about 30% of the time, and everyone retries until green. How would you get out of this situation?" Hint: measure first (history, retry rates), separate infrastructure failures from test failures, quarantine with ownership, and set a policy for new flaky tests.

<a id="102-code-coverage"></a>

10.2 Code Coverage

Code coverage measures which parts of the code ran during the tests. GitLab handles it in two separate ways: a single percentage per job, extracted from the log, which drives the merge request widget, history graphs, badges and approval rules; and a line-by-line report (Cobertura or JaCoCo), which annotates the merge request diff. This subchapter covers both and how to make them agree.

Coverage Regex Parsing

Why it matters. "Coverage went from 82% to 79%" is a useful sentence in a code review. To say it, GitLab needs a number for every pipeline, and it gets that number in the simplest way possible: by reading it from the job log.

What it is. The coverage job keyword is a regular expression. After the job finishes, GitLab searches the log with it and stores the result as the job's coverage percentage.

The matching rules:

• The pattern uses RE2 syntax and must start and end with /. RE2 does not support lookaheads, lookbehinds or backreferences.
• If several lines match, the last matching line is used.
• In the matched fragment, the first number is the coverage. Leading zeros are removed.
• If nothing matches, the job has no coverage value.

What counts as "the matched fragment" explains why most published patterns contain a capturing group: when the regex has groups, GitLab uses the last captured group as the fragment, and otherwise the whole match. Without the group in the pytest pattern below, the fragment would start at TOTAL and its first number would be the statement count, 79.

Job log of "unit-tests" (pytest-cov)

Name                Stmts   Miss  Cover
---------------------------------------
shop/cart.py           48      4    92%
shop/tax.py            31     11    65%
---------------------------------------
TOTAL                  79     15    81%      <- last matching line; captured group "81%", value 81

unit-tests:
  image: python:3.12-slim
  script:
    - pip install -r requirements.txt pytest-cov
    - pytest --cov=shop --cov-report=term       # prints the table above
  coverage: '/TOTAL.*? (100(?:\.0+)?\%|[1-9]?\d(?:\.\d+)?\%)$/'   # single quotes: no YAML escaping needed

Common patterns.

# pytest-cov (Python): pytest --cov --cov-report=term
coverage: '/TOTAL.*? (100(?:\.0+)?\%|[1-9]?\d(?:\.\d+)?\%)$/'

# Jest (JavaScript): jest --ci --coverage   (captures the "% Stmts" column of "All files")
coverage: '/All files[^|]*\|[^|]*\s+([\d\.]+)/'

# Go, single package: go test -cover
coverage: '/coverage: \d+.\d+% of statements/'

# SimpleCov (Ruby): rspec
coverage: '/\(\d+.\d+\%\) covered/'

Some tools do not print a total that is easy to match. Print one yourself:

java-tests:
  image: maven:3.9-eclipse-temurin-21
  script:
    - mvn --batch-mode verify                      # jacoco-maven-plugin writes target/site/jacoco/jacoco.csv
    # Sum covered and missed instructions over all classes and print a single matchable line
    - awk -F"," 'NR>1 { missed += $4; covered += $5 } END { printf "Total %.2f%% covered\n", 100*covered/(covered+missed) }' target/site/jacoco/jacoco.csv
  coverage: '/Total ([0-9]{1,3}\.[0-9]+)% covered/'

Where the value appears. The merge request widget (with the change compared to the target branch), the pipeline and job lists, the coverage history chart under Analyze > Repository analytics (per job, with a CSV download), the coverage badge (section 14.2), and the Coverage-Check approval rule (later in this subchapter).

Pipeline coverage is an average. If several jobs in a pipeline report coverage, the pipeline value is the average of the job values. That is often wrong:

parallel: 3 test shards, each runs one third of the tests

  shard 1/3: 38%    shard 2/3: 41%    shard 3/3: 35%
  pipeline coverage = (38 + 41 + 35) / 3 = 38%      <- but the whole suite covers 84%

When tests are split (section 4.5), collect the raw coverage data from all shards and compute the total in one job, which is the only one with a coverage regex:

test:
  parallel: 3
  script:
    - pytest --cov=shop --cov-report= --splits 3 --group $CI_NODE_INDEX   # pytest-split; no report yet
    - mv .coverage .coverage.$CI_NODE_INDEX                                 # unique name per shard
  artifacts:
    paths:
      - .coverage.*

coverage-total:
  stage: report                                    # a stage after "test", declared in "stages"
  image: python:3.12-slim
  needs: [test]                                    # downloads the artifacts of all 3 shards
  script:
    - pip install coverage
    - coverage combine .coverage.*                  # merge the shard data
    - coverage report                               # prints TOTAL ... 84%
  coverage: '/TOTAL.*? (100(?:\.0+)?\%|[1-9]?\d(?:\.\d+)?\%)$/'

Testing the regex. A wrong regex fails silently: no coverage, or a wrong number. Python's re module behaves like RE2 for typical coverage patterns, so you can test locally against a saved log:

import re

pattern = re.compile(r"TOTAL.*? (100(?:\.0+)?\%|[1-9]?\d(?:\.\d+)?\%)$")   # without the slashes

for line in reversed(open("job.log").read().splitlines()):   # the last matching line wins
    found = pattern.findall(line)            # groups if the regex has any, else whole matches
    if not found:
        continue
    last = found[-1]
    fragment = last[-1] if isinstance(last, tuple) else last  # last captured group, or the match
    number = re.search(r"\d+(\.\d+)?", fragment)              # first number in the fragment
    if number:
        print(line.strip(), "->", number.group())
        break

Key takeaways

• coverage is an RE2 regex applied to the job log. The last matching line wins, and the first number in it is the value.
• The value drives the MR widget, history charts, badges and the Coverage-Check rule.
• Pipeline coverage is the average of job values, which is misleading with parallel shards. Combine the data in one job.
• When a tool does not print a usable total, print one yourself.

Practice

1. (Beginner) Add a coverage regex to a test job, run it, and find the value in the job list and in Analyze > Repository analytics.
2. (Intermediate) Split a test suite into 3 parallel jobs, each with a coverage regex, and note the pipeline value. Then refactor to a single coverage-total job that combines shard data, and compare.
3. Interview question: "Coverage in the merge request widget dropped from 85% to 42% after a CI change, but no tests were removed. What happened?" Hint: which jobs now report coverage, how the pipeline value is aggregated, and what a regex matches when the log format changes.

Cobertura and JaCoCo Reports

Why it matters. A single percentage tells you how much is covered, not what. To show reviewers which changed lines are not tested, GitLab needs per-line data. Coverage tools produce it in standard XML formats, and GitLab reads two of them.

What it is. artifacts:reports:coverage_report uploads a coverage report with a declared format:

Format	coverage_format	Produced by	Notes
Cobertura	cobertura	coverage.py, Jest/Istanbul, nyc, gcovr, PHPUnit, simplecov-cobertura, converters for Go and others	The most widely supported format. Maximum file size 10 MiB
JaCoCo	jacoco	JaCoCo (Java, Kotlin)	Supported natively since GitLab 17.6. Earlier versions need a conversion to Cobertura

This keyword produces only the diff annotations described in the next topic. It does not produce the percentage in the MR widget or the history graphs. Configure the coverage regex for those. The older artifacts:reports:cobertura keyword was replaced by coverage_report and removed in GitLab 15.0.

Configurations by language.

# Python: coverage.py writes Cobertura XML
python-tests:
  image: python:3.12-slim
  script:
    - pip install -r requirements.txt pytest-cov
    - pytest --cov=shop --cov-report=term --cov-report=xml:coverage.xml   # term for the regex, xml for the report
  coverage: '/TOTAL.*? (100(?:\.0+)?\%|[1-9]?\d(?:\.\d+)?\%)$/'
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage.xml

# JavaScript: Jest with the Cobertura reporter
js-tests:
  image: node:22
  script:
    - npm ci
    - npx jest --ci --coverage --coverageReporters=text-summary --coverageReporters=cobertura
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml

# Go: convert the native profile to Cobertura
go-tests:
  image: golang:1.23
  script:
    - go install github.com/boumenot/gocover-cobertura@latest
    - go test ./... -coverprofile=coverage.txt -covermode=atomic   # "count" does not work together with -race
    - gocover-cobertura < coverage.txt > coverage.xml
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage.xml

# Java: native JaCoCo XML (GitLab 17.6 and later)
java-tests:
  image: maven:3.9-eclipse-temurin-21
  script:
    - mvn --batch-mode verify jacoco:report       # writes target/site/jacoco/jacoco.xml
  artifacts:
    reports:
      coverage_report:
        coverage_format: jacoco
        path: target/site/jacoco/jacoco.xml

How GitLab finds the files. A report is useless if GitLab cannot map its entries to files in the repository, and when it cannot, the annotation is skipped silently. The two formats store paths differently:

<!-- Cobertura: <source> roots plus a filename relative to one of them -->
<coverage line-rate="0.81" version="7.6">
  <sources>
    <source>/builds/shop/api/shop</source>      <!-- absolute path on the runner -->
  </sources>
  <packages>
    <package name="shop">
      <classes>
        <class name="cart.py" filename="cart.py" line-rate="0.92">
          <lines>
            <line number="12" hits="3"/>        <!-- covered -->
            <line number="13" hits="0"/>        <!-- not covered -->
          </lines>
        </class>
      </classes>
    </package>
  </packages>
</coverage>

Format	How paths are resolved
Cobertura	GitLab combines each <source> with each filename and uses the first combination that exists in the repository. This only works when <source> paths look like <CI_BUILDS_DIR>/<project full path>/.... With more than 100 <source> nodes, annotations may be missing or wrong. If filename is already relative to the repository root, no <source> is needed
JaCoCo	The report has paths relative to the source roots (package plus file name). GitLab matches them against the files changed in the merge request. Relative paths must be unique, otherwise GitLab cannot decide which file is meant

Multiple reports. Several jobs, including parallel shards, can each upload a coverage report. GitLab merges them into one for the pipeline. Reports from child pipelines are not shared with the parent pipeline.

Key takeaways

• artifacts:reports:coverage_report takes coverage_format (cobertura or jacoco) and path.
• It produces diff annotations only. The percentage comes from the coverage regex, so configure both from the same test run.
• JaCoCo is native from GitLab 17.6. Older versions convert JaCoCo to Cobertura.
• Path mismatches are skipped silently. Check <source> and filename (Cobertura) or uniqueness of relative paths (JaCoCo). Cobertura files must stay under 10 MiB.

Practice

1. (Beginner) Add a Cobertura report to a test job in your language and download the artifact. Find one covered and one uncovered line in the XML.
2. (Intermediate) Move your tests into a subdirectory build (for example cd backend && pytest ...) so that the report's paths no longer start at the repository root. Observe that annotations disappear, then fix the paths (tool configuration or <source>) until they come back.
3. Interview question: "The merge request widget shows 87% coverage, but no lines in the diff are annotated. List possible causes." Hint: two separate mechanisms, path resolution, file size, pipeline state, and whether the changed files are in the report at all.

Coverage Visualization in Diffs

Why it matters. Total coverage can stay at 85% while a merge request adds 200 untested lines, because the rest of the code base hides them. Reviewers care about the coverage of the lines being changed. Coverage visualization shows it where they already look: in the diff.

What it is. After the pipeline completes, GitLab processes the merged coverage report in the background and marks each line of the merge request diff in the gutter:

Changes tab: shop/cart.py

     | 10  def total(items, discount=None):
  G  | 11      subtotal = sum(i.price for i in items)
  G  | 12      if discount is None:
  G  | 13          return subtotal
  R  | 14      if discount.kind == "percent":
  R  | 15          return subtotal * (1 - discount.value / 100)
  O  | 16      raise ValueError("unknown discount")
     | 17

  G  green   covered by tests
  R  red     not covered by tests
  O  orange  loaded but never executed (Cobertura only)
     blank   no coverage data for this line (comments, blank lines, files not in the report)

Lines 14 to 16 tell the reviewer exactly which branch of the new logic has no test, which a total percentage never could.

Behavior to know.

• Annotations appear only on files changed in the merge request. Coverage data for other files is ignored.
• They appear only after the pipeline completes. While a blocking manual job is waiting, there is no visualization.
• If a merge request changes its target branch, the pipeline may still use the previous diff. Re-run the pipeline after the change.
• A path that cannot be resolved is skipped without an error (previous topic).

Troubleshooting checklist.

No annotations in the diff?
 1. Did the pipeline finish (no pending manual job that blocks)?          -> wait or run it
 2. Does a job declare artifacts:reports:coverage_report?                 -> check the job's artifacts
 3. Is the file in the diff also in the report?                           -> grep the XML for the filename
 4. Do the report's paths resolve to repository paths?                    -> <source> + filename, or relative path uniqueness
 5. Is the Cobertura file under 10 MiB?                                   -> exclude tests and vendored code from coverage

Reducing report size also speeds up processing. Measure only your own source code:

# .coveragerc (coverage.py)
# Measure only the application package, not tests or migrations
[run]
source = shop
omit =
    */tests/*
    */migrations/*

Key takeaways

• Coverage visualization marks changed lines in the MR diff as covered (green), not covered (red) or loaded but not executed (orange, Cobertura only).
• It only annotates files in the diff, and only after the pipeline completes.
• Failures are silent. Work through the checklist: pipeline state, report presence, file presence, paths, size.
• Focus reports on application code to keep them small and relevant.

Practice

1. (Beginner) Open a merge request that adds a function with two branches, test only one of them, and find the red lines in the Changes tab.
2. (Intermediate) Add a blocking manual job to the pipeline and confirm that the annotations do not appear until it runs. Then change it to allow_failure: true and compare.
3. Interview question: "Why can a merge request with 100% coverage of its changed lines still lower the project's total coverage, and the reverse?" Hint: what each number is computed over, deleted tested code, and files outside the diff.

Coverage Check Approval Rules

Why it matters. Showing coverage is information. Many teams also want a guardrail: a merge request that lowers coverage should not get in unnoticed. The Coverage-Check approval rule makes a drop in coverage require an explicit approval from someone responsible for quality, without blocking every merge request outright.

What it is. Coverage-Check (Premium and Ultimate) is a special merge request approval rule. When the merge request's pipeline reports a lower coverage than the target branch, the rule requires the configured approvals. It compares the pipeline coverage values from the coverage regex, not the Cobertura or JaCoCo reports. The current documentation also states that the rule requires approval when the merge base pipeline has no coverage data, even if the merge request itself improves coverage.

Configuring it.

1. Make sure test jobs report coverage with coverage in both merge request pipelines and default branch pipelines.
2. Go to Settings > Merge requests > Merge request approvals.
3. Enable the Coverage-Check rule, or add an approval rule named Coverage-Check.
4. Choose the target branches, the number of approvals and the eligible users or groups (for example a qa-leads group).

The rule can also be created through the API as a report approver rule:

# Create a Coverage-Check rule that needs 1 approval from group 42
curl --request POST --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  --data "name=Coverage-Check" \
  --data "rule_type=report_approver" \
  --data "report_type=code_coverage" \
  --data "approvals_required=1" \
  --data "group_ids[]=42" \
  "https://gitlab.example.com/api/v4/projects/12345/approval_rules"

Approval rule versus a failing threshold. Coverage-Check is a soft gate: a human can accept a drop. A hard gate fails the job below a fixed value, and works in every tier:

	Coverage-Check rule	Threshold in the test tool
Tier	Premium, Ultimate	Free
Triggers on	Any decrease compared to the target branch	Value below a fixed number
Outcome	Extra approval required	Job fails
Flexibility	A reviewer can accept the drop with a reason	No exceptions unless the threshold is changed
Typical use	Prevent gradual erosion	Enforce a hard minimum

unit-tests:
  script:
    - pytest --cov=shop --cov-report=term --cov-fail-under=80   # hard floor: exit code 2 below 80%
  coverage: '/TOTAL.*? (100(?:\.0+)?\%|[1-9]?\d(?:\.\d+)?\%)$/'  # still feeds Coverage-Check

// jest.config.js: hard floor in Jest
module.exports = {
  coverageThreshold: {
    global: { lines: 80, branches: 70 },   // the test run fails below these values
  },
};

Pitfalls.

• Averaging across parallel jobs (section 10.2, Coverage Regex Parsing) makes comparisons noisy. Report coverage from a single combined job.
• If the default branch pipeline does not report coverage, there is nothing to compare against, and merge requests need approval anyway.
• Small fluctuations (for example 84.31% to 84.29% after deleting tested code) trigger the rule too. Teams usually accept this and approve quickly, or combine the rule with a fixed floor and rely on review for the rest.

Key takeaways

• Coverage-Check (Premium and Ultimate) requires approval when coverage decreases compared to the target branch.
• It uses the coverage regex values, so the default branch must report coverage too.
• It is a soft gate. Combine it with a hard threshold in the test tool if you need a minimum.
• Report coverage from one combined job to keep comparisons meaningful.

Practice

1. (Beginner) Add --cov-fail-under (or your tool's equivalent) to a test job, then delete a test and observe the failing job.
2. (Intermediate) On a Premium project, enable Coverage-Check for main with a dedicated group. Open one merge request that lowers coverage and one that raises it, and compare the approval requirements.
3. Interview question: "Management asks for a policy that coverage must never drop. What are the trade-offs between a hard threshold, the Coverage-Check rule and changed-line coverage in review?" Hint: false positives from refactoring, gaming the metric with useless tests, and where human judgment belongs.

<a id="103-quality-analysis"></a>

10.3 Quality Analysis

Tests prove that the code does what it should. Quality analysis looks at everything else: maintainability and style issues, accessibility problems in the rendered pages, and how fast the application is in a browser and under load. This subchapter covers the Code Quality report format, how to design linting jobs, and GitLab's accessibility, browser performance and load performance testing.

Code Quality Reports

Why it matters. Linters find hundreds of issues in an existing code base. Nobody reads a log with 400 warnings, and failing the pipeline on all of them blocks every merge request. What reviewers need is the difference: "this merge request introduces 3 new issues and fixes 1". GitLab's Code Quality report provides exactly that comparison.

What it is. A Code Quality report is a JSON file in a format based on Code Climate's. You upload it with artifacts:reports:codequality. GitLab compares the issues of the merge request pipeline with those of the target branch, using each issue's fingerprint to recognize the same issue on both sides.

[
  {
    "description": "`os` imported but unused",
    "check_name": "F401",
    "fingerprint": "7815696ecbf1c96e6894b779456d330e",
    "severity": "minor",
    "location": {
      "path": "shop/cart.py",
      "lines": { "begin": 3 }
    }
  }
]

Field	Required content
description	Human-readable message
check_name	The rule identifier, such as F401 or no-unused-vars
fingerprint	A stable, unique ID for this issue. It must not change when unrelated lines move, or every issue shows as "new"
severity	info, minor, major, critical or blocker
location.path	Path relative to the repository root, without a ./ prefix
location.lines.begin or location.positions.begin.line	The line number

The file must be a single JSON array and must not start with a byte order mark.

Where results appear.

View	Tier	Shows
Merge request widget	Free, Premium, Ultimate	New and resolved issues compared to the target branch
Pipeline details, Code Quality tab	Premium, Ultimate	All issues found in the pipeline
Merge request Changes view	Ultimate	Inline markers on the affected lines in the diff
Project quality summary	Ultimate (beta, behind a feature flag)	Overview under Analyze > CI/CD analytics

Producing the report. GitLab's built-in Code Climate-based template (Jobs/Code-Quality.gitlab-ci.yml) was deprecated in GitLab 17.3, with removal planned for 19.0. It needed Docker-in-Docker and was slow. The recommended approach is to run your own linters and have them write the format directly:

Tool	How to produce the report
Ruff (Python)	Native: --output-format=gitlab
golangci-lint (Go)	Native Code Climate output (--out-format code-climate in v1, --output.code-climate.path in v2)
ESLint, Stylelint, Flake8, Pylint, markdownlint-cli2	A formatter or plugin package, such as eslint-formatter-gitlab
Hadolint (Dockerfiles)	Native: --format gitlab_codeclimate
MyPy and others	A small conversion script

ruff:
  stage: lint
  image: python:3.12-slim
  needs: []                                       # start immediately, no dependencies
  script:
    - pip install ruff
    # 1. Machine-readable report for GitLab; --exit-zero so this step never fails the job
    - ruff check --exit-zero --output-format=gitlab --output-file=gl-code-quality-report.json .
    # 2. Human-readable output and the real exit code
    - ruff check .
  artifacts:
    when: always                                  # keep the report when the second step fails
    reports:
      codequality: gl-code-quality-report.json

Several jobs can each upload a codequality report. GitLab combines them, so you can have one report per linter.

Writing a converter. For a tool without a native format, convert its JSON output. The fingerprint must be stable, so hash fields that identify the issue, not the line number alone:

#!/usr/bin/env python3
"""Convert mypy's JSON output to a GitLab Code Quality report.

Usage: mypy --output json shop/ | python mypy_to_codequality.py > gl-mypy.json
"""
import hashlib
import json
import sys

SEVERITY = {"error": "major", "note": "info"}

issues = []
for raw in sys.stdin:
    item = json.loads(raw)                                   # one JSON object per line
    # Stable fingerprint: file + rule + message, so moving code does not create a "new" issue
    key = f'{item["file"]}:{item["code"]}:{item["message"]}'
    issues.append({
        "description": item["message"],
        "check_name": item["code"] or "mypy",
        "fingerprint": hashlib.md5(key.encode()).hexdigest(),
        "severity": SEVERITY.get(item["severity"], "minor"),
        "location": {"path": item["file"], "lines": {"begin": item["line"]}},
    })

json.dump(issues, sys.stdout, indent=2)

Key takeaways

• Code Quality reports are a JSON array of issues with description, check name, fingerprint, severity and location. Upload them with artifacts:reports:codequality.
• GitLab compares fingerprints between the MR and the target branch to show new and resolved issues. Unstable fingerprints make every issue look new.
• The MR widget is available in all tiers. Pipeline view is Premium, inline diff markers are Ultimate.
• The Code Climate template is deprecated. Run linters directly and use their native GitLab output, a formatter, or a converter.

Practice

1. (Beginner) Add the Ruff job above (or the ESLint equivalent) to a project, introduce two lint issues in a merge request, and find them in the merge request widget.
2. (Intermediate) Add a second linter job with its own Code Quality report, and confirm that both appear together. Then write a converter for a tool that has no native format and validate its output against the required fields.
3. Interview question: "After a refactoring that only moved code, the Code Quality widget reports 150 new issues and 150 resolved ones. What is wrong?" Hint: how GitLab matches issues between branches, and what the fingerprint was computed from.

Linting Jobs

Why it matters. Linting is the cheapest quality check you have: it takes seconds and catches errors before any test runs. Badly designed lint jobs are still a common source of frustration, though: slow because they install the world, blocking because of 400 legacy warnings, or inconsistent with what developers run locally. A good design makes lint fast, local-first and predictable.

What it is. A lint job runs static checks on source files: style (formatting), correctness (unused variables, undefined names), and file-specific rules for Dockerfiles, shell scripts, YAML or Markdown. Design decisions:

Decision	Recommendation
When it runs	First, in parallel with builds, with needs: [] (fail fast, section 13.1)
One job or many	One job per tool or language, so failures are easy to read and jobs can be skipped by path
Blocking or not	Block on formatting and errors. Report the rest through Code Quality
What it checks	Only files that can be affected, using rules:changes
Speed	Small images with the tool preinstalled, cached dependencies
Consistency	The same versions and configuration locally (pre-commit, editor) and in CI

stages: [lint, build, test]

.lint:
  stage: lint
  needs: []                                         # do not wait for anything
  interruptible: true                               # cancel when a newer commit arrives

lint:python:
  extends: .lint
  image: python:3.12-slim
  rules:
    - changes: ["**/*.py", "pyproject.toml"]        # skip when no Python changed
  script:
    - pip install ruff==0.6.9                       # pinned: same version as pre-commit
    - ruff format --check .                         # formatting: blocking
    - ruff check --output-format=gitlab --output-file=gl-code-quality-report.json --exit-zero .
    - ruff check --select E,F .                     # errors: blocking; style issues go to the report only
  artifacts:
    when: always
    reports:
      codequality: gl-code-quality-report.json

lint:dockerfile:
  extends: .lint
  image: hadolint/hadolint:v2.12.0-debian
  rules:
    - changes: ["**/Dockerfile*"]
  script:
    - hadolint --format gitlab_codeclimate Dockerfile > gl-hadolint.json || true
    - hadolint --failure-threshold error Dockerfile # block only on errors
  artifacts:
    when: always
    reports:
      codequality: gl-hadolint.json

lint:shell:
  extends: .lint
  image: koalaman/shellcheck-alpine:v0.10.0
  rules:
    - changes: ["**/*.sh"]
  script:
    - find . -name "*.sh" -not -path "./node_modules/*" -print0 | xargs -0 shellcheck

One source of truth with pre-commit. If your team uses the pre-commit framework, run the same hooks in CI. Developers then get the same result locally before pushing:

lint:pre-commit:
  stage: lint
  image: python:3.12-slim
  needs: []
  variables:
    PRE_COMMIT_HOME: $CI_PROJECT_DIR/.cache/pre-commit   # inside the project so it can be cached
  cache:
    key:
      files: [.pre-commit-config.yaml]                    # new cache when hook versions change
    paths: [.cache/pre-commit]
  script:
    - apt-get update && apt-get install -y --no-install-recommends git
    - pip install pre-commit
    - pre-commit run --all-files --show-diff-on-failure

Legacy code bases. For an existing project with many issues, do not turn everything on at once. Choose one of these strategies:

• Report, do not block: publish all issues to Code Quality. The MR widget shows only new ones, which reviewers can address.
• Lint only changed files: pass the files changed against the target branch to the linter.
• Ratchet: block on a rule set that is clean today, and add rules as the code base improves.

# Lint only Python files changed in this merge request (merge request pipelines only)
git fetch --depth=50 origin "$CI_MERGE_REQUEST_TARGET_BRANCH_NAME"
git diff --name-only --diff-filter=d "origin/$CI_MERGE_REQUEST_TARGET_BRANCH_NAME...HEAD" -- '*.py' \
  | xargs --no-run-if-empty ruff check

Do not forget the CI configuration itself: validating .gitlab-ci.yml with the CI Lint API or glab ci lint (section 1.3) catches syntax errors before they reach the default branch.

Key takeaways

• Run lint first and in parallel (needs: []), with one job per tool and rules:changes to skip unaffected files.
• Block on formatting and real errors. Send everything else to Code Quality reports, where reviewers see only new issues.
• Pin tool versions and share configuration between local tools (pre-commit, editors) and CI.
• For legacy code, report without blocking, lint changed files, or ratchet rule sets up over time.

Practice

1. (Beginner) Add a lint job for one language with needs: [] and rules:changes. Push a commit that touches only documentation and confirm that the job is skipped.
2. (Intermediate) Introduce pre-commit with two hooks, run the same configuration in CI with caching, and measure the job duration with a cold and a warm cache.
3. Interview question: "You join a team whose lint job fails with 2,000 warnings and has been allow_failure: true for a year. How would you make linting useful again?" Hint: separate blocking and reporting, a baseline, only new issues, and a plan to ratchet up.

Accessibility Testing

Why it matters. Accessibility problems (missing image alternative text, low color contrast, form fields without labels) exclude users of screen readers, keyboard navigation or magnification, and in many countries they are also a legal requirement. They are easy to introduce and hard to notice for developers who do not use assistive technology. An automated check on every merge request catches the mechanical part of the problem.

What it is. GitLab's accessibility testing runs Pa11y, which loads pages in a headless browser and checks them against the WCAG 2 guidelines (level AA by default). The template Verify/Accessibility.gitlab-ci.yml adds a job named a11y in a stage named accessibility. The job tests the URLs in the a11y_urls variable and uploads gl-accessibility.json as an accessibility report. The merge request shows an Accessibility Report comparing the source and target branches, with issues classified as errors, warnings and notices. The feature is available in all tiers.

stages:
  - build
  - test
  - deploy
  - accessibility                         # the template's job uses this stage, so it must exist

variables:
  a11y_urls: "https://staging.example.com https://staging.example.com/login"   # space-separated

include:
  - template: Verify/Accessibility.gitlab-ci.yml

Testing review apps. Testing a fixed staging URL tells you little about a merge request. Point the job at the merge request's review app (section 12.2) instead:

review:
  stage: deploy
  script:
    - ./deploy-review.sh "$CI_COMMIT_REF_SLUG"
    - echo "https://$CI_COMMIT_REF_SLUG.review.example.com" > environment_url.txt
  environment:
    name: review/$CI_COMMIT_REF_SLUG
    url: https://$CI_COMMIT_REF_SLUG.review.example.com
  artifacts:
    paths: [environment_url.txt]

a11y:                                     # extends the template's job (deep merge, section 9.4)
  needs: [review]
  variables:
    a11y_urls: "https://$CI_COMMIT_REF_SLUG.review.example.com https://$CI_COMMIT_REF_SLUG.review.example.com/login"

What automation can and cannot do.

Detected automatically	Needs a human
Missing alt attributes, empty links and buttons	Whether the alt text is meaningful
Insufficient color contrast	Whether the reading order makes sense
Form fields without labels	Usability with a real screen reader
Missing document language, invalid ARIA attributes	Keyboard traps in complex widgets, focus management in dialogs

Automated tools find only part of all accessibility issues. Use the report as a regression guard, not as proof that the site is accessible.

If you need more control (authenticated pages, user flows, custom rules), run Pa11y CI or axe-core in your own job, and publish the results as a Code Quality report or as JUnit test cases:

// e2e/a11y.spec.js: Playwright with axe-core, reported as normal JUnit test cases
import { test, expect } from "@playwright/test";
import AxeBuilder from "@axe-core/playwright";

test("checkout page has no WCAG 2 AA violations", async ({ page }) => {
  await page.goto(process.env.REVIEW_URL + "/checkout");   // authenticated flows are possible here
  const results = await new AxeBuilder({ page })
    .withTags(["wcag2a", "wcag2aa"])                       // rule sets to check
    .analyze();
  expect(results.violations).toEqual([]);                  // a violation fails the test
});

Key takeaways

• Verify/Accessibility.gitlab-ci.yml runs Pa11y against the URLs in a11y_urls and shows an Accessibility Report in the merge request. It needs an accessibility stage.
• Point it at review apps, so each merge request is tested on its own deployment.
• Automated checks find mechanical issues only. Manual testing with assistive technology is still needed.
• For authenticated pages or flows, run axe-core or Pa11y CI in your own tests.

Practice

1. (Beginner) Include the accessibility template with one public URL, run it, and download gl-accessibility.json. Find one error and look up the WCAG criterion it refers to.
2. (Intermediate) Connect the a11y job to a review app, then open a merge request that removes an image's alt attribute and confirm that the report flags it as a new issue.
3. Interview question: "Your accessibility job reports zero errors. Can you tell the product owner that the site is accessible?" Hint: the share of WCAG criteria automated tools can check, and which tests need humans.

Browser Performance Testing

Why it matters. Performance often degrades by small amounts: a larger JavaScript bundle here, an unoptimized image there. Each change looks harmless, and after six months the page takes twice as long to load. Measuring the rendered page on every merge request shows each change's impact while it is still easy to fix.

What it is. Browser performance testing (Premium and Ultimate) runs sitespeed.io in a container against one or more URLs and uploads a browser_performance report (browser-performance.json). The merge request widget compares key metrics for each page with the target branch and labels them degraded, unchanged or improved. The main metric is the sitespeed.io Total Score (0 to 100, combining performance, accessibility and best practice checks), alongside timings such as first visual change and largest contentful paint, transfer size and number of requests.

Template	Use
Verify/Browser-Performance.gitlab-ci.yml	Standard pipelines. Runs sitespeed.io through Docker-in-Docker, so it needs a runner that allows privileged containers
Jobs/Browser-Performance-Testing.gitlab-ci.yml	Auto DevOps pipelines and Kubernetes environments

stages: [build, deploy, performance]       # the template's job runs in "performance"

include:
  - template: Verify/Browser-Performance.gitlab-ci.yml

browser_performance:
  variables:
    URL: https://staging.example.com       # the page to measure
    SITESPEED_OPTIONS: "-n 5"              # 5 iterations: the median is less noisy than one run
    DEGRADATION_THRESHOLD: 5               # only flag a drop of 5 or more Total Score points

Variable	Purpose
URL	Page to test (or a file with several URLs)
SITESPEED_IMAGE, SITESPEED_VERSION	Image and version of sitespeed.io. The template sets a tested default version. If you change it, verify that the template's GitLab plugin still works, and expect metrics to shift
SITESPEED_OPTIONS	Extra sitespeed.io options, such as iterations or a connectivity profile
SITESPEED_DOCKER_OPTIONS	Extra docker run options, such as --network
DEGRADATION_THRESHOLD	Minimum Total Score drop before the widget reports a degradation

Review apps. As with accessibility testing, measure each merge request's own deployment. The documented pattern passes the URL as a file from the deploy job:

browser_performance:
  needs: [review]                         # the job that deploys and writes environment_url.txt
  dependencies: [review]                  # download its artifacts
  variables:
    URL: environment_url.txt              # the template reads the URL from this file

Conditions and caveats.

• The widget appears only after the job has run at least once on the target branch (to have a baseline) and only if it ran in the latest pipeline of the merge request.
• GitLab cannot combine several browser_performance reports in one pipeline. Use one job.
• Measurements are noisy on shared runners. Keep the tool version fixed, use several iterations, use the same runner type for both branches, and set DEGRADATION_THRESHOLD so that noise does not raise alarms.

Key takeaways

• Browser performance testing (Premium) runs sitespeed.io and compares page metrics, mainly the Total Score, between the MR and the target branch.
• Include Verify/Browser-Performance.gitlab-ci.yml, add a performance stage, and set URL. The job needs Docker-in-Docker.
• Test review apps, and run the job on the target branch to create a baseline.
• Control noise with iterations, a fixed tool version, consistent runners and DEGRADATION_THRESHOLD.

Practice

1. (Beginner) Run the template against a public page of your own site and open the sitespeed.io HTML report from the job artifacts.
2. (Intermediate) Connect the job to a review app, add a large unoptimized image in a merge request, and observe the degradation in the widget. Then tune SITESPEED_OPTIONS and DEGRADATION_THRESHOLD so that a merge request with no front-end change reports "unchanged".
3. Interview question: "Your browser performance results flip between degraded and improved for merge requests that do not touch the front end. How do you make the signal trustworthy?" Hint: sources of variance (runners, network, caches, single runs) and how to reduce each one.

Load Performance Testing

Why it matters. Browser performance measures one user. Production has thousands. An endpoint that responds in 50 ms for one request can take seconds under concurrent load because of database locks, connection pool limits or an N+1 query. Load testing in the pipeline finds these regressions before users do.

What it is. Load performance testing (Premium and Ultimate) runs k6, an open-source load testing tool whose tests are written in JavaScript. The template Verify/Load-Performance-Testing.gitlab-ci.yml adds a load_performance job in a performance stage. It runs the test file in K6_TEST_FILE and uploads a load_performance report (load-performance.json). The merge request widget compares these metrics with the target branch:

Metric	Meaning
Checks	Pass rate (%) of the k6 check() assertions
TTFB P90	90th percentile time to first byte
TTFB P95	95th percentile time to first byte
RPS	Average requests per second achieved

A k6 test describes virtual users (VUs) and what each one does:

// tests/load/checkout.js
import http from "k6/http";
import { check, sleep } from "k6";

export const options = {
  vus: 20,                                   // 20 concurrent virtual users
  duration: "1m",
  thresholds: {
    http_req_duration: ["p(95)<500"],        // fail the run if p95 latency is 500 ms or more
    checks: ["rate>0.99"],                   // fail if more than 1% of checks fail
  },
};

const BASE_URL = __ENV.ENVIRONMENT_URL || "https://staging.example.com";

export default function () {
  const res = http.get(`${BASE_URL}/api/products?page=1`);
  check(res, {
    "status is 200": (r) => r.status === 200,
    "has products": (r) => r.json("items").length > 0,
  });
  sleep(1);                                  // think time between iterations of one user
}

stages: [build, deploy, performance]

include:
  - template: Verify/Load-Performance-Testing.gitlab-ci.yml

load_performance:
  variables:
    K6_TEST_FILE: tests/load/checkout.js       # path in the repository
    K6_VERSION: "0.54.0"                       # pin the k6 image version
    K6_OPTIONS: "--duration 30s"               # override the script's options from CI

Variable	Purpose
K6_TEST_FILE	Path of the k6 test in the repository (required)
K6_IMAGE, K6_VERSION	Image and version of k6
K6_OPTIONS	Extra k6 command line options
K6_DOCKER_OPTIONS	Extra docker run options, for example to pass environment variables

As with browser testing, there is a Jobs/Load-Performance-Testing.gitlab-ci.yml variant for Kubernetes and Auto DevOps.

Testing a review app. Pass the review app URL into the k6 container through an environment file:

load_performance:
  needs: [review]
  dependencies: [review]
  before_script:
    - echo "ENVIRONMENT_URL=$(cat environment_url.txt)" > k6.env   # read by __ENV.ENVIRONMENT_URL
  variables:
    K6_TEST_FILE: tests/load/checkout.js
    K6_DOCKER_OPTIONS: "--env-file k6.env"                          # passed to docker run

Designing load tests for CI.

Test type	Load	Duration	Where it belongs
Smoke	1 to 5 VUs	Seconds	Every merge request: the endpoints work under minimal load
Average load	Expected normal traffic	A few minutes	Merge requests to performance-sensitive services, or nightly
Stress / spike	Above expected peak	Longer	Scheduled pipelines against a dedicated environment
Soak	Normal load	Hours	Scheduled, before major releases

Caveats.

• The widget appears only after the job has run at least once on the target branch. Several load_performance reports in one pipeline are not combined.
• Results depend on the environment: a review app on a shared cluster does not behave like production. Compare trends between branches on the same environment type, and use k6 thresholds for absolute limits.
• Never point a load test at production or at third-party services without explicit agreement. A load test is indistinguishable from an attack for whoever runs the target system.

Key takeaways

• Load performance testing (Premium) runs a k6 script and compares checks, TTFB P90/P95 and RPS with the target branch in the merge request.
• Include Verify/Load-Performance-Testing.gitlab-ci.yml, add a performance stage, and set K6_TEST_FILE.
• Use k6 thresholds for absolute limits and the widget for relative regressions.
• Keep merge request load tests short (smoke or average load) and run heavy tests on a schedule against dedicated environments.

Practice

1. (Beginner) Write a k6 script with one check() against a test endpoint, run it through the template with K6_OPTIONS: "--vus 5 --duration 20s", and read the metrics in the job log.
2. (Intermediate) Point the job at a review app through K6_DOCKER_OPTIONS, add a thresholds block, and introduce an artificial 300 ms delay in the endpoint in a merge request. Observe both the widget and the threshold result.
3. Interview question: "Where in your delivery process would you run load tests, and how would you keep them from slowing down every merge request?" Hint: the test types in the table above, scheduled pipelines, rules:changes for performance-sensitive paths, and dedicated environments.

---

<a id="11-security-and-compliance-devsecops"></a>