3. Pipeline Architecture

Chapter 2 treated a pipeline as one file that produces one ordered list of stages. Real delivery systems need more: different pipelines for branches, tags and merge requests, job graphs that skip unnecessary waiting, pipelines that start other pipelines, and pipelines started by schedules or external systems. This chapter covers those architectural building blocks, which decide how fast feedback arrives, how safely changes reach the default branch, and how CI scales across many components and projects.

<a id="31-pipeline-types"></a>

3.1 Pipeline Types

The same .gitlab-ci.yml can produce several kinds of pipeline, depending on what event created it and which commit it tests. This subchapter covers each type, what it tests, and when to use it.

Branch Pipelines

Why it matters. The most basic question CI answers is "is the code I just pushed still working?". Branch pipelines answer it for every push, on every branch, with no extra setup. They are also the pipelines that run after a merge to the default branch, which usually means they are the ones that deploy.

What it is. A branch pipeline runs when commits are pushed to a branch. It tests the latest commit of the push exactly as it exists on the branch, without considering any merge request or target branch.

Property	Value in a branch pipeline
CI_PIPELINE_SOURCE	push (or web, schedule, api when started another way on a branch)
CI_COMMIT_BRANCH	The branch name, such as main or feature/login
CI_COMMIT_TAG	Not set
CI_MERGE_REQUEST_* variables	Not set
Protected variables and runners	Available only if the branch is protected

A few behaviors are worth knowing:

• If a push contains several commits, GitLab creates one pipeline for the newest commit, not one per commit.
• Pushing a new branch creates a pipeline even if it contains no new commits.
• A pipeline can be skipped by adding [skip ci] or [ci skip] to the commit message, or with the push option git push -o ci.skip.
• Jobs with no rules (or only/except) run in branch and tag pipelines by default, but not in merge request pipelines.

stages: [test, deploy]

unit-tests:
  stage: test
  script:
    - make test                                      # no rules: runs on every branch push

deploy-production:
  stage: deploy
  script:
    - ./deploy.sh production
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH    # only branch pipelines on the default branch

git push origin feature/login     ──> branch pipeline: unit-tests
git push origin main              ──> branch pipeline: unit-tests ──> deploy-production
git push -o ci.skip origin main   ──> no pipeline

Key takeaways

• A branch pipeline tests the branch's latest commit as it is, in isolation from any target branch.
• It is identified by CI_COMMIT_BRANCH being set and no CI_MERGE_REQUEST_* variables.
• Protected variables reach it only when the branch is protected, which is why deployments usually run from protected branch pipelines.
• [skip ci] and git push -o ci.skip prevent the pipeline from being created.

Practice

1. (Beginner) Push three commits at once to a feature branch and confirm that only one pipeline is created. Note which commit SHA it tested.
2. (Beginner) Add a job that prints CI_PIPELINE_SOURCE, CI_COMMIT_BRANCH and CI_COMMIT_TAG. Run it on a feature branch and on main, and compare the output.
3. Interview question: "A branch pipeline on a feature branch passed, but the same code broke main after merging. How is that possible?" Hint: think about what a branch pipeline does not test, and which pipeline types in this subchapter close that gap.

Tag Pipelines

Why it matters. Tags mark releases. A tag such as v2.3.0 says "this exact commit is version 2.3.0", so tag pipelines are the natural place to build release artifacts, publish packages and create release entries. Keeping release work in tag pipelines also keeps it out of everyday branch pipelines.

What it is. Pushing a Git tag creates a tag pipeline for the tagged commit. Its source is still push, but the variables differ from a branch pipeline:

Variable	Branch pipeline	Tag pipeline
CI_COMMIT_BRANCH	Branch name	Not set
CI_COMMIT_TAG	Not set	Tag name, such as v2.3.0
CI_COMMIT_REF_NAME	Branch name	Tag name
CI_COMMIT_REF_PROTECTED	true if the branch is protected	true if the tag is protected

The missing CI_COMMIT_BRANCH is a common surprise: a rule such as if: $CI_COMMIT_BRANCH == "main" never matches in a tag pipeline, even if the tagged commit is on main.

Protected tags. Under Settings > Repository > Protected tags, a pattern such as v* can be protected so that only maintainers can create matching tags. Protected tag pipelines receive protected variables, such as registry or signing credentials, so protecting release tags is what makes it safe to publish from them.

stages: [build, publish]

default:
  image: python:3.12-slim
  before_script:
    - pip install build twine

build-package:
  stage: build
  script:
    - python -m build                           # writes dist/*.whl and dist/*.tar.gz
  artifacts:
    paths: [dist/]

publish-package:
  stage: publish
  script:
    - python -m twine upload --repository-url "$CI_API_V4_URL/projects/$CI_PROJECT_ID/packages/pypi" dist/*
  variables:
    TWINE_USERNAME: gitlab-ci-token
    TWINE_PASSWORD: $CI_JOB_TOKEN
  rules:
    - if: $CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/  # only semantic-version tags, such as v2.3.0

git tag v2.3.0 && git push origin v2.3.0
                        │
                        v
tag pipeline: build-package ──> publish-package   (CI_COMMIT_TAG=v2.3.0)

Creating releases with the release keyword is covered in chapter 8.

Key takeaways

• Tag pipelines run when a tag is pushed and set CI_COMMIT_TAG instead of CI_COMMIT_BRANCH.
• Rules that test CI_COMMIT_BRANCH do not match in tag pipelines.
• Protect release tag patterns so that only trusted users can trigger publishing jobs that use protected variables.
• Match tag formats with a regular expression in rules:if to avoid publishing from arbitrary tags.

Practice

1. (Beginner) Add a job that runs only when CI_COMMIT_TAG is set and prints the tag name. Push a lightweight tag and an annotated tag, and confirm both create pipelines.
2. (Intermediate) Protect the tag pattern v*, create a protected variable RELEASE_TOKEN, and show that a job in a v1.0.0 tag pipeline can read it while a job in a test-1 tag pipeline cannot.
3. Interview question: "Your deployment job uses if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH, and the team now wants to deploy only from release tags. What changes, and what security setting should accompany the change?" Hint: compare the variables available in tag pipelines and consider who is allowed to create tags.

Merge Request Pipelines

Why it matters. Branch pipelines know nothing about merge requests: they cannot see the target branch, the MR number or which files the MR changes. Merge request pipelines run in the context of an MR, so they can report results directly in the MR widget, run checks only for MRs, and limit expensive jobs to changes that are actually proposed for merging.

What it is. A merge request pipeline runs when an MR is created or updated with new commits. It tests the head of the source branch (ref refs/merge-requests/<iid>/head), but with MR context available. In the UI it carries a merge request label, and in the API it appears as a "detached" pipeline, because it is not attached to a branch ref.

Variable	Example value
CI_PIPELINE_SOURCE	merge_request_event
CI_MERGE_REQUEST_IID	42
CI_MERGE_REQUEST_SOURCE_BRANCH_NAME	feature/login
CI_MERGE_REQUEST_TARGET_BRANCH_NAME	main
CI_MERGE_REQUEST_DIFF_BASE_SHA	The merge base commit, used by rules:changes
CI_MERGE_REQUEST_EVENT_TYPE	detached (or merged_result, merge_train)
CI_COMMIT_BRANCH	Not set

Jobs must opt in. Jobs without rules do not run in MR pipelines. A job needs a rule that matches merge_request_event:

lint:
  script:
    - npm run lint
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"   # only in MR pipelines

unit-tests:
  script:
    - npm test
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"   # in MR pipelines...
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH        # ...and on the default branch

api-contract-tests:
  script:
    - npm run test:contract
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      changes:
        - api/**/*                                       # compared against the MR's merge base

Duplicate pipelines. Once some jobs use MR rules and others have none, a push to a branch with an open MR can create two pipelines: a branch pipeline and an MR pipeline. The usual fix is workflow:rules that pick one type per situation, covered in chapter 4.

push to feature/login (no MR)        ──> branch pipeline
push to feature/login (MR !42 open)  ──> branch pipeline  +  MR pipeline   <- duplicate unless workflow:rules prevents it

Forks and security. MR pipelines from a fork run in the fork's project by default, using the fork's runners and variables. Protected variables are not exposed to MR pipelines by default, because the source branch can contain arbitrary changes. Members of the parent project can choose to run a fork MR's pipeline in the parent project, after reviewing the code.

Key takeaways

• MR pipelines run for MR events, test the source branch's head and expose CI_MERGE_REQUEST_* variables.
• Jobs must have a rule matching merge_request_event to run in them.
• rules:changes in MR pipelines compares against the MR's merge base, which makes it reliable for "only if these files changed" logic.
• Mixing MR rules with rule-less jobs creates duplicate pipelines. Use workflow:rules to prevent this.

Practice

1. (Beginner) Add a job that runs only in MR pipelines and prints the source and target branch names. Open an MR and find the job.
2. (Intermediate) Create a configuration that produces duplicate pipelines for a branch with an open MR. Observe both pipelines, then explain which job rules caused each one.
3. Interview question: "Why might a team want MR pipelines instead of branch pipelines for feature branches, and what do they give up?" Hint: think about rules:changes, the MR widget, pipelines for branches without an MR, and protected variables.

Merged Results Pipelines

Why it matters. An MR pipeline still tests the source branch alone. If main has moved on since the branch was created, the merged code has never been tested: two MRs that pass individually can break main together. A merged results pipeline tests what main will look like after the merge, before it happens.

What it is. When merged results pipelines are enabled (Premium and Ultimate), GitLab creates a temporary merge commit that merges the source branch into the current target branch and runs the MR pipeline on that commit (ref refs/merge-requests/<iid>/merge). The pipeline is labeled merged results and sets CI_MERGE_REQUEST_EVENT_TYPE=merged_result.

main:            A ── B ── C                    (C was merged after the branch was created)
                  \
feature/login:     D ── E

MR pipeline tests:              E                 (feature alone, without C)
Merged results pipeline tests:  M = merge(C, E)   (what main becomes after merging)

Setup.

1. Configure jobs to run in MR pipelines (previous topic). Merged results pipelines are a variant of MR pipelines and use the same rules.
2. Go to Settings > Merge requests and enable merged results pipelines.

No YAML change is needed beyond the MR rules. A job can still tell the variants apart:

integration-tests:
  script:
    - make integration-test
  rules:
    - if: $CI_MERGE_REQUEST_EVENT_TYPE == "merged_result"   # only when testing the merge commit
    - if: $CI_MERGE_REQUEST_EVENT_TYPE == "merge_train"     # merge trains also test merge commits

Limitations.

Situation	Behavior
The MR has merge conflicts	GitLab cannot build the merge commit and runs a normal (detached) MR pipeline instead
The target branch moves after the pipeline ran	The result becomes stale, because it tested an older main. Rerun the pipeline, or use merge trains
The MR comes from a fork and runs in the fork	A normal MR pipeline runs. Merged results need the pipeline to run in the parent project

Key takeaways

• Merged results pipelines test the merge of source and target branch, not the source branch alone.
• They are an MR pipeline variant (Premium and Ultimate) enabled in the project's merge request settings.
• CI_MERGE_REQUEST_EVENT_TYPE distinguishes detached, merged_result and merge_train.
• Results go stale when the target branch moves. Merge trains solve this.

Practice

1. (Beginner) Enable merged results pipelines, open an MR, and print CI_COMMIT_SHA and CI_MERGE_REQUEST_SOURCE_BRANCH_SHA in a job. Explain why they differ.
2. (Intermediate) Create two branches that each pass on their own but conflict semantically (for example, one renames a function and the other adds a call to the old name). Merge one, then show that the merged results pipeline of the other fails while a plain MR pipeline would pass.
3. Interview question: "Merged results pipelines are enabled, but main still broke after two MRs were merged within minutes of each other. Why?" Hint: consider what version of main each pipeline tested and when.

Merge Trains

Why it matters. On a busy repository, merged results go stale constantly: by the time an MR's pipeline finishes, someone else has merged. Rerunning pipelines before every merge is slow and still racy. A merge train queues MRs and tests each one together with all the MRs ahead of it, so the default branch stays green even when many people merge at once. Think of a train where each carriage is checked with all the carriages in front of it attached.

What it is. Merge trains (Premium and Ultimate) build on merged results pipelines. When an MR is added to the train, GitLab creates a merge train pipeline that tests a merge commit containing the target branch plus every MR ahead of it in the queue, plus this MR. When a pipeline passes and all MRs ahead of it have merged, its MR is merged.

main: ── X

Train:   [MR !1]          [MR !2]               [MR !3]
Tests:   X + !1           X + !1 + !2           X + !1 + !2 + !3
         (running)        (running in parallel) (running in parallel)

If !2's pipeline fails:
  - !2 is removed from the train, and its author is notified
  - !3's pipeline is canceled and restarted as  X + !1 + !3
  - !1 is unaffected

Pipelines for different train positions run in parallel (up to 20 at a time), so the train does not serialize all testing. If every pipeline passes, MRs merge in order as fast as their pipelines finish.

Setup.

1. Configure MR pipelines and enable merged results pipelines.
2. In Settings > Merge requests, enable merge trains.
3. Usually also enable Pipelines must succeed, so MRs cannot bypass the train.

Users then add an MR to the train with Set to auto-merge (or Merge when the pipeline has already passed). Merge immediately skips the train and restarts every pipeline behind it, and can be disallowed in the settings.

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

default:
  interruptible: true                  # restarted train pipelines can cancel their old jobs quickly

test:
  script: [make test]

e2e:
  script: [make e2e]
  rules:
    - if: $CI_MERGE_REQUEST_EVENT_TYPE == "merge_train"   # run the slowest suite only on the train

	MR pipeline	Merged results pipeline	Merge train pipeline
Commit tested	Source branch head	Source merged into current target	Target + MRs ahead in queue + this MR
Protects main from semantic conflicts	No	Partly (results can go stale)	Yes
Tier	Free	Premium, Ultimate	Premium, Ultimate
CI_MERGE_REQUEST_EVENT_TYPE	detached	merged_result	merge_train

Key takeaways

• A merge train tests each MR combined with every MR queued ahead of it, then merges in order.
• A failure removes only the failing MR. Pipelines behind it restart without it.
• Train pipelines run in parallel, so throughput stays high when pipelines mostly pass.
• Make jobs interruptible and keep train pipelines reliable: flaky tests eject good MRs and restart pipelines behind them.

Practice

1. (Beginner) Enable merge trains, add two MRs to the train and watch the second pipeline's merge commit. Identify which commits it contains.
2. (Intermediate) Configure an expensive job that runs only when CI_MERGE_REQUEST_EVENT_TYPE == "merge_train", and a cheap job that runs in every MR pipeline. Explain the trade-off in feedback time for developers.
3. Interview question: "Your merge train keeps restarting and MRs take hours to merge. What would you investigate?" Hint: look at flaky jobs, pipeline duration, Merge immediately usage and how many MRs fail on the train.

<a id="32-directed-acyclic-graph-pipelines"></a>

3.2 Directed Acyclic Graph Pipelines

Stages are a coarse ordering: every job waits for every job in the stage before it. A directed acyclic graph (DAG) pipeline instead lets each job declare exactly which jobs it depends on. This subchapter covers the needs keyword, how it controls artifacts, optional dependencies, pipelines without stages, and how to read the resulting graph.

needs Keyword

Why it matters. In a stage-based pipeline, the slowest job in a stage delays everything after it (chapter 2). If a monorepo builds a backend and a frontend, the frontend's deploy should not wait for the backend's integration tests. needs replaces stage barriers with explicit dependencies, often cutting pipeline duration substantially.

What it is. needs lists the jobs that must finish successfully before a job can start. As soon as those jobs succeed, the job starts, whatever its stage and whatever other jobs are still running. Because dependencies cannot form a cycle, the result is a directed acyclic graph.

Rules of needs:

• Needed jobs must be in an earlier stage or the same stage.
• needs: [] means "depend on nothing": the job starts as soon as the pipeline is created, even in a late stage.
• A job without needs still follows stage ordering.
• Every needed job must exist in the pipeline. If rules remove it, the configuration is invalid (see optional needs).
• If a needed job fails, the dependent job is skipped, unless the needed job has allow_failure: true.
• A job can list up to 50 needs by default (self-managed administrators can change the limit).

stages: [build, test, deploy]

build-backend:
  stage: build
  script: [make -C backend build]

build-frontend:
  stage: build
  script: [make -C frontend build]

test-backend:
  stage: test
  needs: [build-backend]                 # starts when build-backend is done, ignores build-frontend
  script: [make -C backend test]

test-frontend:
  stage: test
  needs: [build-frontend]
  script: [make -C frontend test]

deploy-frontend:
  stage: deploy
  needs: [test-frontend]                 # does not wait for test-backend
  script: [./deploy.sh frontend]

deploy-backend:
  stage: deploy
  needs: [test-backend]
  script: [./deploy.sh backend]

check-links:
  stage: deploy
  needs: []                              # no dependencies: starts immediately
  script: [./scripts/check-links.sh]

Stage-based (no needs):            DAG (with needs):

build-backend  ─┐                  build-backend  ──> test-backend  ──> deploy-backend
build-frontend ─┴─> test-* ─> ...  build-frontend ──> test-frontend ──> deploy-frontend
                                   check-links  (runs at once)

The frontend chain no longer waits for slow backend jobs.

Key takeaways

• needs lets a job start as soon as its listed jobs succeed, independent of stage barriers.
• needs: [] starts a job immediately. Jobs without needs still wait for earlier stages.
• Needed jobs must exist in the pipeline and be in the same or an earlier stage.
• A failed dependency skips its dependents, so failures stay contained to their own branch of the graph.

Practice

1. (Beginner) Convert a three-stage pipeline with two independent build/test chains to use needs. Compare the total pipeline duration before and after.
2. (Beginner) Add a job in the last stage with needs: [] and confirm in the pipeline graph that it starts with the first jobs.
3. Interview question: "When would you not use needs for a job?" Hint: think about jobs that genuinely depend on everything before them, readability, and jobs that should run only after all tests pass.

needs:artifacts

Why it matters. Without needs, a job downloads artifacts from all jobs in earlier stages. With a DAG, that would be unpredictable, because earlier stages may not have finished. needs therefore also defines where artifacts come from, and controlling that avoids slow downloads of artifacts a job never uses.

What it is. A job with needs downloads artifacts only from the jobs it lists, and by default from all of them. The long form of needs controls this per dependency:

Form	Artifacts downloaded
No needs	All artifacts from jobs in earlier stages (or those listed in dependencies)
needs: [build]	Artifacts from build
needs: [{ job: build, artifacts: false }]	None from build, but the job still waits for it
needs: []	None

build:
  stage: build
  script:
    - make build                       # produces bin/app
    - make docs                        # produces site/ (large)
  artifacts:
    paths: [bin/, site/]

unit-tests:
  stage: test
  needs:
    - job: build
      artifacts: true                  # default; needs bin/app to run tests
  script:
    - ./bin/app --self-test

license-check:
  stage: test
  needs:
    - job: build
      artifacts: false                 # waits for build to pass, but skips the download
  script:
    - ./scripts/check-licenses.sh

Do not combine dependencies and needs in the same job. Use needs with artifacts: false to express "wait for this job, but do not download its artifacts".

needs can also fetch artifacts from outside the current pipeline: needs:pipeline:job reads from a parent pipeline (section 3.3), and needs:project reads from another project (section 3.4).

Key takeaways

• With needs, a job downloads artifacts only from the jobs it needs.
• artifacts: false keeps the ordering dependency but skips the download.
• needs: [] downloads nothing.
• Prefer needs:artifacts over dependencies in DAG pipelines.

Practice

1. (Beginner) Create a build job with a 200 MB artifact and two dependent jobs, one with artifacts: false. Compare the "Downloading artifacts" section of both job logs.
2. (Intermediate) A job needs compile for ordering but needs artifacts only from generate-schema. Write its needs definition and verify which artifacts are present in the job's working directory.
3. Interview question: "A test job suddenly fails with 'file not found' after a teammate added needs: [lint] to it. The missing file is produced by the build job. What happened?" Hint: compare how artifacts are selected with and without needs.

Optional Needs

Why it matters. rules often remove jobs from a pipeline: a database migration job may run only when migrations/ changes. A job that needs it would then make the configuration invalid in every pipeline where the migration job is absent. Optional needs express "wait for this job if it exists".

What it is. In the long form, optional: true makes a dependency conditional on the job being present in the pipeline:

• If the needed job is in the pipeline, the dependent job waits for it as usual (and downloads its artifacts unless artifacts: false).
• If the needed job is not in the pipeline, the dependency is ignored.

stages: [build, migrate, deploy]

build:
  stage: build
  script: [make build]

migrate-db:
  stage: migrate
  script: [./scripts/migrate.sh]
  rules:
    - changes: [migrations/**/*]         # only present when migrations change

deploy:
  stage: deploy
  needs:
    - build
    - job: migrate-db
      optional: true                     # wait for it only if it was added to the pipeline
  script: [./deploy.sh]

Pipeline where migrations/ changed:     build ──> migrate-db ──> deploy
Pipeline where migrations/ unchanged:   build ──────────────────> deploy
Same pipeline without "optional: true": error: 'deploy' job needs 'migrate-db' job, but 'migrate-db' does not exist in the pipeline

Optional needs check existence only. If migrate-db exists and fails, deploy is skipped as with any other dependency. A related keyword, rules:needs, changes a job's needs list based on conditions and is covered in chapter 4.

Key takeaways

• optional: true makes a need apply only when the needed job is in the pipeline.
• It is the standard fix for "needs a job that does not exist" errors caused by rules.
• It does not change failure behavior: an existing needed job that fails still skips dependents.

Practice

1. (Beginner) Reproduce the "does not exist in the pipeline" error with a rule-limited job and a dependent job, then fix it with optional: true.
2. (Intermediate) Design a pipeline where deploy needs build always, migrate-db only when migrations change, and rebuild-search-index only on the default branch. Test both a feature branch and main.
3. Interview question: "What is the risk of marking a need as optional when the job is actually essential?" Hint: think about what happens if someone changes that job's rules later.

Stageless Pipelines

Why it matters. Once every job declares needs, stages mostly become labels. Defining and maintaining a stage list adds little when the graph already expresses the order. A stageless pipeline drops stages entirely and lets dependencies alone define the flow, which suits pipelines that are naturally graph-shaped, such as monorepos with many independent components.

What it is. If no job sets stage, all jobs go into the default test stage. Because needs can reference jobs in the same stage, dependencies alone order the pipeline:

• Jobs without needs start immediately.
• Jobs with needs start when their dependencies succeed.

# No "stages" and no "stage" keywords: every job is in "test"
lint:
  script: [npm run lint]                 # no needs: starts immediately

build:
  script: [npm run build]                # no needs: starts immediately
  artifacts:
    paths: [dist/]

unit-tests:
  needs: [build]
  script: [npm test]

e2e-tests:
  needs: [build]
  script: [npm run e2e]

deploy-review:
  needs: [lint, unit-tests, e2e-tests]   # the only "gate" before deploying
  script: [./deploy.sh review]

lint ─────────────────────────────┐
build ──┬──> unit-tests ──────────┼──> deploy-review
        └──> e2e-tests  ──────────┘

Stage-based	Stageless
Order comes from the stage list	Order comes only from needs
Easy to read for linear pipelines	Easy to extend for graph-shaped pipelines
Forgotten dependencies are hidden by stage barriers	A forgotten needs lets a job start too early
The pipeline graph shows columns per stage	Use the "Job dependencies" view for a meaningful graph

A common middle ground keeps a few stages for readability, such as build, test and deploy, and uses needs inside them.

Key takeaways

• A stageless pipeline puts all jobs in one stage and orders them only with needs.
• Jobs without needs start immediately, so every real dependency must be declared.
• It suits graph-shaped pipelines. For linear pipelines, stages remain easier to read.

Practice

1. (Beginner) Rewrite a four-stage pipeline without any stages or stage keywords, using only needs. Confirm that the run order is unchanged.
2. (Intermediate) In your stageless pipeline, remove one needs entry and describe the failure that would follow. Then add a validation step in review (or a lint script) that would catch it.
3. Interview question: "Would you make your team's main pipeline stageless? Defend your choice." Hint: consider who reads the pipeline, how often it changes and how its graph is shaped.

Visualizing Job Dependencies

Why it matters. A DAG is only useful if people can see it. When a job starts too early, waits too long or is skipped, the dependency graph explains why. Visual tools also reveal the critical path, the chain of jobs that determines total pipeline duration.

What it is. GitLab shows dependencies in two places:

Tool	Where	What it shows
Pipeline graph, grouped by Job dependencies	A pipeline's page, using the "Group jobs by" control	Jobs arranged by dependency instead of stage. Show dependencies draws lines between jobs and their needs
Visualize tab	Pipeline editor	The graph of the configuration before it runs, useful for checking needs while editing

Reading a graph to find the critical path:

Job durations:  build 2m, unit 3m, e2e 12m, lint 1m, package 1m, deploy 2m

build (2m) ──┬──> unit (3m) ─────┐
             └──> e2e (12m) ─────┼──> package (1m) ──> deploy (2m)
lint (1m) ───────────────────────┘

Critical path: build -> e2e -> package -> deploy = 2 + 12 + 1 + 2 = 17 minutes
Speeding up unit or lint does not shorten the pipeline. Speeding up e2e does.

The same graph can be produced from the API, which helps when auditing many pipelines:

# List each job of pipeline 98765 with its stage, status and duration (seconds)
curl --silent --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.example.com/api/v4/projects/12345/pipelines/98765/jobs?per_page=100" \
  | jq -r '.[] | [.name, .stage, .status, (.duration // 0 | floor)] | @tsv'

Key takeaways

• Group the pipeline graph by "Job dependencies" and enable "Show dependencies" to see needs relationships.
• The pipeline editor's Visualize tab shows the graph before anything runs.
• The critical path is the longest chain of dependent jobs. Only shortening it shortens the pipeline.

Practice

1. (Beginner) Open a DAG pipeline, switch between "Stage" and "Job dependencies" grouping, and describe what each view makes easier to see.
2. (Intermediate) Use the jobs API to compute the critical path of a recent pipeline by hand, given its needs. Identify the single job whose speed-up would help most.
3. Interview question: "Management asks you to halve pipeline duration. How do you decide where to spend effort?" Hint: critical path, job durations versus queue time, and parallelism.

<a id="33-parent-child-pipelines"></a>

3.3 Parent-Child Pipelines

A single .gitlab-ci.yml becomes hard to read and slow to evaluate as a project grows, especially in monorepos. Parent-child pipelines split one pipeline into a parent that triggers separate child pipelines in the same project. This subchapter covers how to trigger them, generate them dynamically, wait for them, pass data to them and stay within nesting limits.

trigger:include

Why it matters. In a monorepo with a frontend, a backend and docs, one configuration file holds every job, every rule and every variable. A change to the docs still loads the whole file and creates a pipeline view with dozens of jobs. Child pipelines let each component own its CI configuration, run only when needed and appear as a separate, readable pipeline.

What it is. A trigger job (also called a bridge job) uses trigger:include to start a child pipeline from another configuration file in the same project. The child pipeline runs for the same commit and ref as the parent, and its source is parent_pipeline.

Trigger jobs do not run scripts. They accept a limited set of keywords, including stage, trigger, variables, rules, needs, when, allow_failure, extends and inherit.

repository/
├── .gitlab-ci.yml           # parent: decides which children to start
├── frontend/
│   └── .gitlab-ci.yml       # child configuration for the frontend
└── backend/
    └── .gitlab-ci.yml       # child configuration for the backend

# .gitlab-ci.yml (parent)
stages: [triggers]

frontend:
  stage: triggers
  trigger:
    include: frontend/.gitlab-ci.yml        # path relative to the repository root
  rules:
    - changes: [frontend/**/*]              # start the child only if frontend files changed

backend:
  stage: triggers
  trigger:
    include:
      - local: backend/.gitlab-ci.yml       # long form, which allows several files
      - local: ci/shared-jobs.yml
  rules:
    - changes: [backend/**/*, ci/**/*]

# frontend/.gitlab-ci.yml (child)
workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "parent_pipeline"   # the child's own workflow rules

stages: [build, test]

build:
  stage: build
  script:
    - cd frontend && npm ci && npm run build

test:
  stage: test
  script:
    - cd frontend && npm test

trigger:include accepts local, template, project with file, and artifact with job for generated files (next topic).

Things that surprise people.

• A child pipeline is a separate pipeline with its own configuration. It does not inherit the parent's stages, default or workflow.
• In the child, CI_PIPELINE_SOURCE is parent_pipeline, so rules testing merge_request_event or push never match there. Use workflow:rules in the child or rules based on other variables.

Key takeaways

• trigger:include starts a child pipeline in the same project, for the same commit, from another configuration file.
• Trigger jobs run no script. Their rules decide whether the child starts.
• Child pipelines have their own configuration and see CI_PIPELINE_SOURCE == "parent_pipeline".
• Use child pipelines to split monorepo CI by component and to keep each pipeline view small.

Practice

1. (Beginner) Move two jobs from your .gitlab-ci.yml into child.yml and trigger it from the parent. Find the child pipeline in the parent's pipeline graph.
2. (Intermediate) Set up a monorepo with frontend/ and backend/ children that start only when their directory changes. Push a change to one directory and confirm only one child runs.
3. Interview question: "A job in a child pipeline uses rules: - if: $CI_PIPELINE_SOURCE == 'merge_request_event' and never runs. Why, and how would you fix it?" Hint: consider which source the child pipeline has, and how the child could learn about the parent's context.

Dynamically Generated Child Pipelines

Why it matters. Sometimes the jobs a pipeline needs are not known until it runs: one job per changed microservice, one test job per detected database version, one deploy per region listed in a file. Writing every possible job by hand leads to huge configurations full of near-duplicates. A dynamic child pipeline generates its configuration in a job and then runs it.

What it is. It is a two-step process:

1. A generator job writes a YAML file and saves it as an artifact.
2. A trigger job starts a child pipeline from that artifact with trigger:include:artifact and job.

# .gitlab-ci.yml (parent)
stages: [generate, run]

generate-pipeline:
  stage: generate
  image: python:3.12                             # full image: includes git, which the generator calls
  script:
    - python ci/generate.py > generated.yml      # writes one job per service
  artifacts:
    paths: [generated.yml]

run-services:
  stage: run
  needs: [generate-pipeline]
  trigger:
    include:
      - artifact: generated.yml                  # the file inside the artifact
        job: generate-pipeline                   # the job that produced it
    strategy: depend                             # parent waits for the child (see next topic)

# ci/generate.py
import pathlib
import subprocess

# Services are the top-level folders under services/
services = sorted(p.name for p in pathlib.Path("services").iterdir() if p.is_dir())

# Compare against the previous commit to find services that changed
changed = subprocess.run(
    ["git", "diff", "--name-only", "HEAD~1", "HEAD"],
    capture_output=True, text=True, check=True,
).stdout.splitlines()
changed_services = {path.split("/")[1] for path in changed if path.startswith("services/")}

print("stages: [test]")
if not changed_services:
    # A child pipeline must contain at least one job, so emit a placeholder
    print("nothing-to-do:\n  stage: test\n  script: [echo 'No service changed']")

for name in services:
    if name in changed_services:
        print(f"""
test-{name}:
  stage: test
  image: python:3.12-slim
  script:
    - cd services/{name}
    - pip install -r requirements.txt
    - pytest
""")

Jobs clone the repository shallowly (controlled by GIT_DEPTH), so make sure the clone depth covers the commits the generator compares.

Using parent artifacts in the child. A child job can download artifacts from a job in the parent pipeline with needs:pipeline:job. The parent passes its own pipeline ID, because the child's CI_PIPELINE_ID is different:

# parent: trigger job
run-services:
  trigger:
    include:
      - artifact: generated.yml
        job: generate-pipeline
  variables:
    PARENT_PIPELINE_ID: $CI_PIPELINE_ID          # pass the parent's ID to the child

# child (generated): a job that needs the parent's build output
integration-test:
  needs:
    - pipeline: $PARENT_PIPELINE_ID              # which pipeline to read from
      job: build                                 # which parent job's artifacts to download
  script:
    - ./dist/app --self-test

Practical tips. Validate the generated file in the generator job (for example, with the CI Lint API) so syntax errors appear in a readable log. Keep generators small and deterministic, and use extends or !reference in generated YAML to reuse templates from an included file rather than generating everything from scratch.

Key takeaways

• A generator job writes YAML as an artifact, and a trigger job runs it with trigger:include:artifact plus job.
• Generated configurations must be valid and contain at least one job.
• Pass CI_PIPELINE_ID from the parent so that child jobs can use needs:pipeline:job to fetch parent artifacts.
• Use dynamic children when the set of jobs depends on data available only at runtime.

Practice

1. (Beginner) Write a shell generator that emits one job per line of a regions.txt file, each printing its region name. Trigger the generated file as a child pipeline.
2. (Intermediate) Extend the Python example so that a parent build job produces an artifact that each generated child job downloads through needs:pipeline:job.
3. Interview question: "What are the downsides of dynamically generated pipelines compared with static configuration?" Hint: think about reviewability, debugging, the pipeline editor's validation and failure modes of the generator.

trigger:strategy depend

Why it matters. By default, a trigger job succeeds as soon as GitLab creates the downstream pipeline, not when that pipeline finishes. The parent can then report "passed" while the child is still running or has failed, and later parent jobs run without waiting. For a child that runs the actual tests, that is almost never what you want.

What it is. trigger:strategy: depend makes the trigger job wait for the downstream pipeline to finish and take on its status.

	Default (no strategy)	strategy: depend
Trigger job finishes	When the downstream pipeline is created	When the downstream pipeline finishes
Trigger job status	success if creation succeeded	Mirrors downstream: success, failed, canceled, and so on
Later jobs in the parent	Start immediately	Wait for the downstream pipeline
Parent pipeline status	Ignores downstream results	Reflects downstream results

stages: [test, deploy]

backend-tests:
  stage: test
  trigger:
    include: backend/.gitlab-ci.yml
    strategy: depend               # wait for the child and take its status

deploy:
  stage: deploy
  script: [./deploy.sh]            # runs only if the child pipeline passed

Without depend:   backend-tests (passed, in 2s) ──> deploy runs  ...child fails 10 minutes later
With depend:      backend-tests (running ... failed after 10m)   ──> deploy skipped

strategy: depend combines with other job keywords as expected. With allow_failure: true, a failed child shows as a warning on the trigger job but does not block the parent. Retrying a trigger job creates a new downstream pipeline. Newer GitLab releases also add a strategy: mirror option that copies the downstream status even more strictly (including states such as manual or blocked), so check the keyword reference for your version.

Key takeaways

• By default, trigger jobs are fire-and-forget: they succeed once the downstream pipeline is created.
• strategy: depend makes the trigger job wait for and mirror the downstream status.
• Use it whenever later parent jobs, or the parent's final status, depend on the child's result.

Practice

1. (Beginner) Trigger a child pipeline containing a failing job, once without and once with strategy: depend. Compare the parent's final status.
2. (Intermediate) Configure two child pipelines where one is critical (blocks deploy) and one is advisory (shows a warning only). Use strategy: depend and allow_failure to express this.
3. Interview question: "Your main pipeline is green, but a release shipped with failing tests that ran in a child pipeline. What configuration mistake likely caused this?" Hint: consider when a trigger job is considered finished by default.

Passing Variables to Child Pipelines

Why it matters. A child pipeline often needs context from the parent: the version being built, the environment to deploy to or the ID of the parent pipeline. Knowing which variables pass automatically, which do not and how to send data computed at runtime prevents both missing values and accidental leaks.

What it is. Several mechanisms control what a child pipeline sees:

Source of the variable	Passed to the child?
variables defined in the trigger job	Yes
Top-level (global) variables in the parent YAML	Yes, by default, through trigger:forward:yaml_variables: true
Pipeline variables (from a manual run, schedule, API or trigger token)	No, unless trigger:forward:pipeline_variables: true
Project, group and instance CI/CD variables	Not passed, but the child sees them anyway because it runs in the same project
Variables created at runtime (dotenv reports)	Not passed. Child jobs fetch them with needs:pipeline:job
Predefined variables such as CI_PIPELINE_ID	Not passed. Child gets its own values. Map them into trigger variables explicitly

Variables passed to the child become pipeline variables there, which have high precedence and override the same names in the child's YAML.

# .gitlab-ci.yml (parent)
variables:
  APP_NAME: shop                         # global: forwarded by default

compute-version:
  stage: .pre
  script:
    - echo "APP_VERSION=2.4.$CI_PIPELINE_IID" >> build.env
  artifacts:
    reports:
      dotenv: build.env                  # runtime variable

deploy-child:
  trigger:
    include: deploy/.gitlab-ci.yml
    strategy: depend
    forward:
      yaml_variables: true               # default: forward APP_NAME and this job's variables
      pipeline_variables: true           # also forward variables set when the pipeline was run manually
  variables:
    TARGET_ENV: staging                  # explicit value for the child
    PARENT_PIPELINE_ID: $CI_PIPELINE_ID  # predefined values must be mapped explicitly

# deploy/.gitlab-ci.yml (child)
deploy:
  needs:
    - pipeline: $PARENT_PIPELINE_ID
      job: compute-version               # downloads the dotenv report, which sets APP_VERSION
  script:
    - echo "Deploying $APP_NAME $APP_VERSION to $TARGET_ENV"

To stop global variables reaching a child, use inherit:variables: false (or a list of names) on the trigger job, or forward:yaml_variables: false.

Key takeaways

• Trigger job variables and global YAML variables reach the child. Pipeline variables need forward:pipeline_variables: true.
• Predefined variables are not passed: map them explicitly, for example PARENT_PIPELINE_ID: $CI_PIPELINE_ID.
• Runtime dotenv variables reach child jobs through needs:pipeline:job.
• Passed variables override the child's own YAML variables of the same name.

Practice

1. (Beginner) Pass TARGET_ENV from a trigger job and print it in the child. Then define TARGET_ENV: local in the child's YAML and observe which value wins.
2. (Intermediate) Run the parent pipeline manually with a variable DRY_RUN=true. Make the child see it, and then make a second child that explicitly does not.
3. Interview question: "A child pipeline prints CI_COMMIT_MESSAGE correctly but CI_PIPELINE_ID shows an unexpected value. Explain both." Hint: the child is a separate pipeline for the same commit.

Nested Pipeline Limits

Why it matters. Child pipelines can trigger their own children, and every pipeline can trigger several downstream pipelines. Without limits, a configuration bug could create thousands of pipelines and exhaust runners. Knowing the limits avoids designs that fail only at scale.

What it is. GitLab enforces limits on pipeline hierarchies:

Limit	Default value
Depth of parent-child nesting	2 levels of children below the top-level parent (parent, child, grandchild)
Downstream pipelines in one pipeline hierarchy	1000 (configurable by self-managed administrators)
Jobs per pipeline, pipeline creation rate and similar	Instance and plan limits, configured by administrators

Parent pipeline                    level 0
├── child: frontend                level 1
│   └── grandchild: e2e-shards     level 2   <- deepest allowed level
│       └── (trigger:include)      level 3   <- fails: maximum depth exceeded
└── child: backend                 level 1

A trigger job that would exceed a limit fails with a message describing the limit.

Design guidance. Most monorepos need one level of children. If you find yourself at level 3, flatten the design: generate one child pipeline that contains the jobs of several would-be grandchildren, or use parallel:matrix (chapter 4) inside a child. Keep fan-out bounded, for example by generating one child per changed component rather than per file.

Key takeaways

• Parent-child nesting is limited to two levels below the parent.
• A whole hierarchy is capped at 1000 downstream pipelines by default.
• Exceeding a limit fails the trigger job. Flatten designs instead of nesting deeper.

Practice

1. (Beginner) Build a parent, child and grandchild pipeline, then add a fourth level and read the error message.
2. (Intermediate) Redesign a hypothetical "one child per package, one grandchild per test suite, one great-grandchild per shard" layout to fit within the limits.
3. Interview question: "Why would GitLab cap nesting depth and hierarchy size instead of letting runners' capacity limit things naturally?" Hint: think about runaway generators, fairness on shared runners and the cost of evaluating configurations.

<a id="34-multi-project-pipelines"></a>

3.4 Multi-Project Pipelines

Systems often span several repositories: a library and the services that use it, an application and a separate deployment repository. Multi-project pipelines let a pipeline in one project start a pipeline in another, pass data to it, track its result and share artifacts. This subchapter covers each of those mechanisms.

trigger:project

Why it matters. When a shared library changes, its consumers should be tested against it before anyone finds out in production. When an application passes its tests, a separate infrastructure repository may need to deploy it. Doing this by hand, or with ad hoc API scripts, is slow and opaque. trigger:project makes cross-project orchestration part of the pipeline and shows the downstream pipeline in the upstream graph.

What it is. A trigger job with trigger:project starts a pipeline in another project. The downstream pipeline:

• Runs on the downstream project's configuration and on the branch given by trigger:branch (the downstream default branch if omitted).
• Runs as the user who started the upstream pipeline, who must have permission to run pipelines on that branch in the downstream project (for a protected branch, permission to push or merge to it).
• Has CI_PIPELINE_SOURCE set to pipeline.

# In project "platform/payments-lib"
stages: [test, downstream]

unit-tests:
  stage: test
  script: [make test]

test-checkout-service:
  stage: downstream
  trigger:
    project: shop/checkout-service      # full path of the downstream project
    branch: main                        # optional; defaults to the downstream default branch
    strategy: depend                    # wait for its result (see "status mirroring")
  variables:
    PAYMENTS_LIB_REF: $CI_COMMIT_SHA    # tell the downstream which library commit to test
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

platform/payments-lib pipeline                shop/checkout-service pipeline
┌────────────┐   ┌───────────────────────┐    ┌────────┐   ┌────────┐
│ unit-tests │──>│ test-checkout-service │───>│ build  │──>│ test   │
└────────────┘   └───────────────────────┘    └────────┘   └────────┘
                          (trigger job)        source: pipeline

	Parent-child (trigger:include)	Multi-project (trigger:project)
Downstream project	Same project	Different project
Configuration used	File chosen by the parent	Downstream project's own .gitlab-ci.yml
Commit	Same commit as the parent	Latest commit of the chosen downstream branch
CI_PIPELINE_SOURCE downstream	parent_pipeline	pipeline
Typical use	Split one project's CI	Coordinate separate repositories

A pipeline can also start a downstream pipeline from a script, using the trigger API with CI_JOB_TOKEN (section 3.5). The trigger keyword is preferred because it links the pipelines in the UI and supports strategy: depend.

Key takeaways

• trigger:project starts a pipeline in another project, on a chosen branch, using that project's configuration.
• The downstream pipeline runs with the upstream user's permissions, so that user needs access to the downstream branch.
• Downstream pipelines see CI_PIPELINE_SOURCE == "pipeline".
• Use multi-project pipelines to coordinate repositories, and parent-child pipelines to split one repository.

Practice

1. (Beginner) Create two projects and make the first one's pipeline trigger the second's default branch. Find the downstream pipeline in the upstream graph.
2. (Intermediate) Trigger a protected branch in the downstream project as a user who cannot push to it. Read the error, then fix the permissions.
3. Interview question: "When would you choose a multi-project pipeline over moving everything into one monorepo with child pipelines?" Hint: consider ownership, access control, release cadence and build isolation.

Cross-Project Variable Passing

Why it matters. A downstream pipeline almost always needs context: which upstream commit to test, which image tag to deploy, which environment to target. Passing that context through variables keeps the downstream project generic. It also creates a trust question, because variables from another project can change how the downstream pipeline behaves.

What it is. Variable passing works as in parent-child pipelines:

• variables in the trigger job are passed.
• Global YAML variables are passed by default (trigger:forward:yaml_variables: true).
• Upstream pipeline variables are passed only with trigger:forward:pipeline_variables: true.
• Upstream project CI/CD settings variables are not passed, and the downstream sees its own project, group and instance variables instead.

Passed variables become pipeline variables in the downstream pipeline, so they override variables defined in the downstream YAML.

# Upstream project: shop/checkout-service
variables:
  INTERNAL_DEBUG: "false"                   # global; would be forwarded by default

build-image:
  stage: build
  script:
    - docker build -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA" .
    - docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA"

deploy-via-infra:
  stage: deploy
  inherit:
    variables: false                         # do not leak global variables downstream
  trigger:
    project: ops/infrastructure
    strategy: depend
  variables:
    SERVICE: checkout                        # only these two variables are sent
    IMAGE_TAG: $CI_COMMIT_SHORT_SHA

# Downstream project: ops/infrastructure
deploy:
  script:
    - ./deploy.sh "$SERVICE" "$IMAGE_TAG"
  rules:
    - if: $CI_PIPELINE_SOURCE == "pipeline" && $SERVICE && $IMAGE_TAG   # only when called with both inputs

Restricting incoming variables. Because passed variables override the downstream's own values, a downstream project can restrict who may set pipeline variables (Settings > CI/CD > Variables, minimum role for pipeline variables). Recent GitLab versions restrict this by default for new projects. When the triggering user is not allowed, the trigger fails, so check this setting first if a trigger job fails with a variables-related error.

Runtime values across projects. dotenv variables created in the upstream pipeline are not forwarded. The downstream job can fetch them with needs:project (last topic in this subchapter), because a dotenv report is an artifact.

Key takeaways

• Trigger job variables and global YAML variables are passed. Pipeline variables need forward:pipeline_variables.
• Upstream project settings variables are never passed, and secrets stay in their own project.
• Passed variables override downstream YAML variables, so downstream projects may restrict who can pass them.
• Use inherit:variables: false to send only what the downstream needs.

Practice

1. (Beginner) Pass IMAGE_TAG from an upstream trigger job and print it downstream.
2. (Intermediate) Restrict pipeline variables in the downstream project to the Maintainer role, trigger it as a Developer, and record what happens. Decide whether this restriction is right for your deployment repository.
3. Interview question: "A downstream deployment pipeline accepts a TARGET_ENV variable from any upstream project. What could go wrong, and how would you harden it?" Hint: think about overriding values, validating inputs in rules, protected environments and restricting pipeline variables.

Downstream Pipeline Status Mirroring

Why it matters. An upstream pipeline that triggers a downstream test suite and immediately reports "passed" gives false confidence. The upstream status should reflect whether the change actually works for consumers. Mirroring connects the two results, and understanding how statuses propagate helps when debugging failures that span projects.

What it is. trigger:strategy: depend (section 3.3) applies to multi-project triggers in the same way. The trigger job waits for the downstream pipeline and takes its final status, which then affects the upstream pipeline like any other job.

stages: [test, integration, release]

unit-tests:
  stage: test
  script: [make test]

consumer-checkout:
  stage: integration
  trigger:
    project: shop/checkout-service
    strategy: depend                    # failure downstream fails this job

consumer-reporting:
  stage: integration
  trigger:
    project: analytics/reporting
    strategy: depend
  allow_failure: true                   # advisory: a failure shows as a warning only

publish-library:
  stage: release
  script: [make publish]                # runs only if consumer-checkout passed

Upstream: payments-lib
  unit-tests (passed)
  consumer-checkout (failed)  <── mirrors shop/checkout-service pipeline (failed)
  consumer-reporting (failed, allowed to fail) <── analytics/reporting (failed)
  publish-library (skipped)
Upstream status: failed

Working with linked pipelines.

• The upstream graph shows downstream pipelines as cards that can be expanded, and each downstream pipeline links back to its upstream.
• Retrying a trigger job creates a new downstream pipeline. Retrying jobs inside the downstream pipeline updates its status, and with strategy: depend, the trigger job follows it.
• In the API, trigger jobs are listed as bridges: GET /projects/:id/pipelines/:pipeline_id/bridges returns each bridge with its downstream_pipeline.

# Show each trigger job in pipeline 98765 with its downstream pipeline's project, ID and status
curl --silent --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.example.com/api/v4/projects/12345/pipelines/98765/bridges" \
  | jq -r '.[] | [.name, .downstream_pipeline.project_id, .downstream_pipeline.id, .downstream_pipeline.status] | @tsv'

Key takeaways

• strategy: depend makes a multi-project trigger job wait for and mirror the downstream status.
• Combine it with allow_failure: true for advisory downstream checks.
• Retrying a trigger job creates a new downstream pipeline.
• The API exposes trigger jobs as bridges, each linking to its downstream pipeline.

Practice

1. (Beginner) Trigger a downstream project whose pipeline fails, with and without strategy: depend. Record both upstream statuses.
2. (Intermediate) Use the bridges API to write a script that prints the status of every downstream pipeline triggered by the latest pipeline on main.
3. Interview question: "A library triggers 15 consumer projects with strategy: depend. Releases are now slow and often blocked by unrelated consumer failures. How would you redesign this?" Hint: consider advisory versus blocking consumers, allow_failure, scheduled compatibility runs and contract tests.

Fetching Artifacts Across Projects

Why it matters. Multi-project setups often hand over files: a library build produces a package that a service tests against, or an application build produces a manifest that the infrastructure project deploys. Rebuilding those files downstream wastes time and risks testing something different from what was built.

What it is. There are two main ways to fetch artifacts from another project:

Method	Tier	How it works
needs:project	Premium, Ultimate	Downloads artifacts from the latest successful pipeline for a given ref in another project
Job artifacts API with CI_JOB_TOKEN	All	A script downloads artifacts over the API, authenticated by the job token

needs:project.

# In project "ops/infrastructure"
deploy:
  needs:
    - project: shop/checkout-service         # source project
      job: build-manifest                    # job that produced the artifacts
      ref: main                              # branch or tag whose latest successful pipeline is used
      artifacts: true
  script:
    - kubectl apply -f manifest.yaml         # file from build-manifest's artifacts

needs:project does not wait for anything: it fetches whatever the latest successful pipeline for that ref produced. If no such artifact exists, the job fails. When the downstream was triggered by the upstream, use a variable for the ref (such as a tag or branch passed by the trigger job) to fetch the matching build. Within one pipeline hierarchy, use needs:pipeline:job instead (section 3.3).

Job artifacts API.

# Download the artifacts of the latest successful "build-manifest" job on main of project 4321
curl --fail --location --output artifacts.zip \
  --header "JOB-TOKEN: $CI_JOB_TOKEN" \
  "$CI_API_V4_URL/projects/4321/jobs/artifacts/main/download?job=build-manifest"
unzip artifacts.zip

Job token access. Both methods authenticate with the job token of the consuming job. The source project must allow this: in the source project's Settings > CI/CD > Job token permissions, add the consuming project (or its group) to the allowlist. Without that entry, the request is rejected even if the triggering user can access both projects.

ops/infrastructure job ──(CI_JOB_TOKEN)──> shop/checkout-service
                                            │
                                            ├─ is ops/infrastructure on the job token allowlist?   no  -> 403
                                            └─ yes -> latest successful build-manifest on main -> artifacts.zip

Key takeaways

• needs:project (Premium) downloads artifacts from another project's latest successful pipeline for a ref.
• It does not wait for or trigger anything, so the artifact must already exist.
• The job artifacts API with CI_JOB_TOKEN works on all tiers.
• The source project's job token allowlist must include the consuming project.

Practice

1. (Beginner) Build an artifact in project A, then download it in project B with the job artifacts API and CI_JOB_TOKEN. Fix the 403 error by updating A's job token allowlist.
2. (Intermediate) Make project A trigger project B with a variable SOURCE_REF: $CI_COMMIT_REF_NAME, and make B's job use needs:project with ref: $SOURCE_REF. Explain what happens if A's pipeline has not finished when B's job starts.
3. Interview question: "Why does GitLab require an explicit job token allowlist when the user who started the pipeline already has access to both projects?" Hint: think about least privilege and what a compromised job in any project could otherwise read.

<a id="35-triggering-pipelines"></a>

3.5 Triggering Pipelines

Pushes are not the only way to start pipelines. People run them by hand, schedules run them at fixed times, and external systems start them through tokens, the API or webhooks. This subchapter covers each method and the variable that tells jobs how their pipeline was started.

Manual Pipeline Runs

Why it matters. Some pipelines are needed on demand: rerunning a deployment with different parameters, running a heavy test suite before a demo, or regenerating documentation. A manual run lets a user start a pipeline for any branch or tag, optionally with variables or inputs, without pushing a commit.

What it is. In Build > Pipelines, select Run pipeline, choose a branch or tag, and optionally add variables or inputs. The pipeline's source is web.

• The user needs at least the Developer role, and for a protected branch, permission to push or merge to it.
• Variables entered in the form are pipeline variables with high precedence. The project's pipeline variable restriction (section 3.4) also applies here.
• variables with description, value and options in the YAML prefill the form with documented defaults and drop-down choices (chapter 5).

variables:
  DEPLOY_ENV:
    value: staging
    options: [staging, production]            # shown as a drop-down in the Run pipeline form
    description: Target environment for a manual run.

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "web"        # manual runs
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

deploy:
  script:
    - ./deploy.sh "$DEPLOY_ENV"
  rules:
    - if: $CI_PIPELINE_SOURCE == "web"        # only when someone starts the pipeline by hand

Build > Pipelines > Run pipeline
┌───────────────────────────────────────┐
│ Run for branch name or tag: [ main  v]│
│ Variables                              │
│   DEPLOY_ENV   [ staging        v ]    │
│   Target environment for a manual run. │
│ [ Run pipeline ]                       │
└───────────────────────────────────────┘

Do not confuse a manual pipeline with a manual job (when: manual, chapter 4): a manual job is a step inside a pipeline that waits for a click.

Key takeaways

• Run pipeline starts a pipeline for any branch or tag, with optional variables, and sets CI_PIPELINE_SOURCE to web.
• Protected branches require the user to have push or merge rights on them.
• YAML variables with options and description turn the form into a documented menu.
• Use rules on web to create jobs that exist only in manual runs.

Practice

1. (Beginner) Add a job that runs only when CI_PIPELINE_SOURCE == "web" and start it from the Run pipeline form.
2. (Intermediate) Add a prefilled LOG_LEVEL variable with options info, debug and trace. Confirm that branch pushes use the default while manual runs can choose another value.
3. Interview question: "Why might an organization restrict who can set variables when running pipelines manually?" Hint: think about variables that change scripts, images or deployment targets.

Pipeline Schedules

Why it matters. Some work belongs to the calendar rather than to commits: nightly full test runs, weekly dependency scans, hourly data imports, daily cleanup of review environments. Pipeline schedules run pipelines at fixed times, like a cron job that lives in GitLab next to the code.

What it is. In Build > Pipeline schedules, a schedule defines:

Field	Meaning
Description	Name shown in the list
Interval pattern	Cron syntax, such as 0 2 * * * for 02:00 every day
Cron timezone	Timezone used to interpret the pattern
Target branch or tag	The ref whose configuration and commit are used
Variables	Pipeline variables for this schedule, such as SCHEDULE_TYPE=nightly
Activated	Whether the schedule runs

Scheduled pipelines have CI_PIPELINE_SOURCE == "schedule". Each schedule has an owner, and its pipelines run with the owner's permissions. If the owner loses access (for example, leaves the project), the schedule stops working until another user takes ownership. Schedules are evaluated by a background worker, so a pipeline may start a few minutes after the configured time.

Cron pattern fields:
┌───────────── minute (0-59)
│ ┌─────────── hour (0-23)
│ │ ┌───────── day of month (1-31)
│ │ │ ┌─────── month (1-12)
│ │ │ │ ┌───── day of week (0-6, Sunday = 0)
│ │ │ │ │
0 2 * * *      every day at 02:00
0 6 * * 1      every Monday at 06:00
30 */4 * * *   every 4 hours, at minute 30

A common pattern uses a schedule variable to select jobs:

nightly-full-tests:
  script: [make test-all]
  timeout: 3h
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule" && $SCHEDULE_TYPE == "nightly"

weekly-dependency-audit:
  script: [make audit]
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule" && $SCHEDULE_TYPE == "weekly"

unit-tests:
  script: [make test]
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"
      when: never                                 # skip regular jobs in scheduled pipelines
    - when: on_success

Schedules can also be managed through the API:

# Create a schedule that runs nightly at 02:00 Europe/Rome on main
curl --fail --request POST --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  --form description="Nightly full tests" \
  --form ref="main" \
  --form cron="0 2 * * *" \
  --form cron_timezone="Europe/Rome" \
  "https://gitlab.example.com/api/v4/projects/12345/pipeline_schedules"
# Then add the SCHEDULE_TYPE variable to it with POST .../pipeline_schedules/:schedule_id/variables

Key takeaways

• Schedules run pipelines on a cron pattern for a chosen ref, with CI_PIPELINE_SOURCE == "schedule".
• Pipelines run as the schedule owner. An owner losing access breaks the schedule.
• Use schedule variables to distinguish different schedules in rules.
• Exclude regular jobs from scheduled pipelines when they add no value.

Practice

1. (Beginner) Create a schedule that runs every day at 03:00 in your timezone with a variable SCHEDULE_TYPE=nightly, and a job that runs only for it. Use Run in the schedule list to test it immediately.
2. (Intermediate) Configure two schedules (nightly and weekly) that run different job sets from the same .gitlab-ci.yml, with no regular jobs in either.
3. Interview question: "A nightly pipeline silently stopped running three weeks ago. What are the likely causes?" Hint: look at the schedule owner, its activation state, the target branch and the workflow rules.

Pipeline Trigger Tokens

Why it matters. External systems sometimes need to start a pipeline: a CMS publishing new content, a tool in another CI system finishing a build, or an operator script. Giving them a personal access token would grant far more than they need. A trigger token can only start pipelines in one project.

What it is. A pipeline trigger token is created in Settings > CI/CD > Pipeline trigger tokens and has the prefix glptt-. It is used with the trigger API endpoint:

# Start a pipeline on main of project 12345, with two variables
curl --fail --request POST \
  --form token="$TRIGGER_TOKEN" \
  --form ref=main \
  --form "variables[DEPLOY_ENV]=staging" \
  --form "variables[REASON]=cms-publish" \
  "https://gitlab.example.com/api/v4/projects/12345/trigger/pipeline"

Properties of trigger tokens:

• The pipeline's source is trigger, and CI_PIPELINE_TRIGGERED is true.
• The pipeline runs as the user who created the token, with that user's permissions. If the user loses access, the token stops working.
• Variables sent with the request are pipeline variables and are subject to the project's pipeline variable restrictions.
• Anyone who has the token can start pipelines as its owner, so store it as a secret and revoke it when it is no longer needed.

CI_JOB_TOKEN instead of a trigger token. Inside a GitLab job, the same endpoint accepts token=$CI_JOB_TOKEN. That creates a linked downstream pipeline with source pipeline, without managing a long-lived token. The target project's job token allowlist must allow the calling project. Prefer trigger:project when the call can be written in YAML.

notify-docs-site:
  script:
    - >
      curl --fail --request POST
      --form token="$CI_JOB_TOKEN"
      --form ref=main
      "$CI_API_V4_URL/projects/6789/trigger/pipeline"

Credential	Created by	Pipeline runs as	Source	Best for
Trigger token	A project member	Token creator	trigger	External systems
CI_JOB_TOKEN	GitLab, per job	User of the calling pipeline	pipeline	Jobs in other GitLab projects
Personal or project access token (Pipelines API)	A user or project	Token's user or bot	api	Automation needing more API features

Key takeaways

• Trigger tokens start pipelines in one project through /trigger/pipeline, and the source is trigger.
• Pipelines run as the token creator, so ownership and revocation matter.
• Inside GitLab jobs, use trigger:project or CI_JOB_TOKEN instead of long-lived tokens.
• Treat trigger tokens as secrets: mask them in the calling system and rotate them.

Practice

1. (Beginner) Create a trigger token, start a pipeline with curl, and confirm CI_PIPELINE_SOURCE and CI_PIPELINE_TRIGGERED in a job.
2. (Intermediate) Pass a variable REASON through the trigger request and make a job run only when REASON == "cms-publish". Then restrict pipeline variables in the project and observe the result.
3. Interview question: "A trigger token was accidentally committed to a public repository. What do you do, and what did the attacker gain?" Hint: consider whose permissions the token carries, what pipelines can access and how to revoke it.

Triggering via API and Webhooks

Why it matters. Platform teams automate pipelines: starting them from chatbots, release tooling or internal portals, passing inputs and reading results. The Pipelines API is the fully featured route. Webhooks let other services start pipelines when something happens on their side, without custom code in between.

What it is.

Pipelines API. POST /projects/:id/pipeline creates a pipeline for a ref, authenticated with a personal, project or group access token with api scope. The source is api. Variables are passed as a JSON list, and pipelines that declare spec:inputs can receive inputs (chapter 5).

# Create a pipeline on main with a variable, then print its ID, status and URL
curl --silent --fail --request POST \
  --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  --header "Content-Type: application/json" \
  --data '{
    "ref": "main",
    "variables": [
      { "key": "DEPLOY_ENV", "value": "staging", "variable_type": "env_var" }
    ]
  }' \
  "https://gitlab.example.com/api/v4/projects/12345/pipeline" \
  | jq -r '[.id, .status, .web_url] | @tsv'

The glab CLI wraps the same API, which is convenient for people and scripts:

glab ci run --branch main      # create a pipeline for main in the current repository's project
glab ci status                 # follow the status of the latest pipeline on the current branch

Webhooks. Many services can send an HTTP request when an event occurs. Point their webhook at the trigger URL, with the token and ref in the query string:

https://gitlab.example.com/api/v4/projects/12345/ref/main/trigger/pipeline?token=glptt-xxxxxxxx

When a pipeline is started this way, GitLab stores the request body in the file variable TRIGGER_PAYLOAD, so jobs can read details of the event:

handle-webhook:
  image: alpine:3.20
  script:
    - apk add --no-cache jq
    - jq -r '.event_type // "unknown"' "$TRIGGER_PAYLOAD"   # TRIGGER_PAYLOAD is a path to the JSON body
  rules:
    - if: $CI_PIPELINE_SOURCE == "trigger" && $TRIGGER_PAYLOAD

Method	Authentication	Source	Strength
Pipelines API	Access token (api scope)	api	Full control: variables, inputs, reading results
Trigger API	Trigger token or CI_JOB_TOKEN	trigger or pipeline	Narrow credential, simple form request
Webhook to trigger URL	Trigger token in the URL	trigger	No code needed, payload available in TRIGGER_PAYLOAD

A token in a URL can end up in logs of the sending service, so use a dedicated trigger token for each webhook and rotate it if it is exposed.

Key takeaways

• POST /projects/:id/pipeline with an access token creates pipelines with source api, variables and inputs.
• glab ci run is a convenient front end to the same API.
• Webhooks can call the trigger URL directly, and the request body arrives in TRIGGER_PAYLOAD.
• Use one narrowly scoped token per integration so each can be revoked independently.

Practice

1. (Beginner) Create a pipeline with the Pipelines API using a project access token, and print its web_url.
2. (Intermediate) Use a request-inspection tool or a script to send a JSON webhook to the trigger URL, then write a job that prints one field from TRIGGER_PAYLOAD.
3. Interview question: "Your internal portal starts deployments through the Pipelines API with one admin's personal access token. What is wrong with this design, and what would you replace it with?" Hint: consider bus factor, least privilege, auditability and project access tokens or trigger tokens.

CI_PIPELINE_SOURCE

Why it matters. The same .gitlab-ci.yml serves every way a pipeline can start. A nightly schedule should not deploy, an MR pipeline should not publish packages, and a child pipeline should not re-run the parent's checks. CI_PIPELINE_SOURCE tells each job how its pipeline started, and it is the main input to rules and workflow:rules that shape pipelines per situation.

What it is. A predefined variable set on every pipeline. Its main values:

Value	Pipeline started by
push	A push of commits or tags to a branch or tag
merge_request_event	Creating or updating an MR (including merged results and merge train pipelines)
web	Run pipeline in the UI
schedule	A pipeline schedule
api	The Pipelines API
trigger	A trigger token (API or webhook)
pipeline	A multi-project trigger, or the trigger API called with CI_JOB_TOKEN
parent_pipeline	A parent pipeline's trigger:include
chat	A ChatOps command
webide	The Web IDE
external	An external CI service reporting to GitLab
external_pull_request_event	A pull request on GitHub, for CI/CD for external repositories
security_orchestration_policy	A scan execution or pipeline execution policy
ondemand_dast_scan, ondemand_dast_validation	On-demand DAST features

Using it. A typical configuration routes behavior by source:

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"    # MR pipelines for feature work
    - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS && $CI_PIPELINE_SOURCE == "push"
      when: never                                         # avoid duplicate branch pipelines when an MR is open
    - if: $CI_COMMIT_BRANCH                               # branch pipelines otherwise
    - if: $CI_COMMIT_TAG                                  # tag pipelines
    - if: $CI_PIPELINE_SOURCE =~ /^(web|schedule|api|trigger|pipeline)$/

test:
  script: [make test]
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"
      when: never                                         # schedules run their own job set
    - when: on_success

nightly:
  script: [make test-all]
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"

deploy-production:
  script: [./deploy.sh production]
  rules:
    - if: $CI_COMMIT_TAG && $CI_PIPELINE_SOURCE == "push" # releases only, never from schedules or API runs

CI_PIPELINE_SOURCE describes how a pipeline started, not which ref it runs on, so combine it with CI_COMMIT_BRANCH, CI_COMMIT_TAG and CI_MERGE_REQUEST_EVENT_TYPE for precise rules. A quick debugging job helps when a rule does not behave as expected:

debug-source:
  script:
    - echo "source=$CI_PIPELINE_SOURCE branch=${CI_COMMIT_BRANCH:-<none>} tag=${CI_COMMIT_TAG:-<none>} mr=${CI_MERGE_REQUEST_IID:-<none>}"

Key takeaways

• CI_PIPELINE_SOURCE tells jobs how their pipeline started: push, MR, UI, schedule, API, trigger, upstream pipeline and more.
• It is the main building block for rules and workflow:rules that shape pipelines per trigger.
• Tag and branch pushes both have source push. Use CI_COMMIT_TAG and CI_COMMIT_BRANCH to tell them apart.
• Child pipelines see parent_pipeline and multi-project downstreams see pipeline, never the original trigger's source.

Practice

1. (Beginner) Add the debug-source job and start pipelines in four different ways (push, MR, Run pipeline, schedule). Record each CI_PIPELINE_SOURCE value.
2. (Intermediate) Write workflow:rules that create MR pipelines for feature branches, branch pipelines only for the default branch, tag pipelines for v* tags and nothing for other sources. Test each case.
3. Interview question: "A deployment job ran unexpectedly from a scheduled pipeline on main. Its rule was if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH. Explain why and fix the rule." Hint: the branch is the same for pushes and schedules, and only one variable tells them apart.

---

<a id="4-job-control-and-execution-flow"></a>