12. Deployment and Environments

A pipeline that builds and tests code but cannot get that code to users safely only does half the job. This chapter covers the delivery half: how GitLab models the places you deploy to as environments, how it gives every merge request a temporary deployment, which strategies limit the damage of a bad release, how to deploy to common targets such as Kubernetes, the major clouds and plain servers, how to manage infrastructure itself through Git, and what Auto DevOps automates for you. Knowing these mechanisms is what lets a team deploy many times a day without making each deployment an event.

Tiers at a glance. Environments, deployments, review apps, auto_stop_in, rollbacks, feature flags, GitLab Pages, the GitLab agent for Kubernetes, GitLab-managed Terraform state and Auto DevOps are available on all tiers. Protected environments, deployment approvals, the cross-project environments dashboard, Pages parallel deployments and agent impersonation for CI jobs require Premium. Each topic notes its tier.

<a id="121-environments"></a>

12.1 Environments

An environment is a named place your code runs, such as staging or production, and every deployment to it is recorded. This subchapter covers how jobs declare environments, the difference between fixed and generated environments, how tiers and URLs describe them, and how to control who may deploy to them and who must approve first.

Without environments, a pipeline is only a list of scripts. GitLab cannot tell you what is running in production, which commit it came from or who deployed it. With environments, every deployment job leaves a record:

  pipeline #812 (commit a1b2c3)                       GitLab records
  ┌───────┐   ┌──────┐   ┌──────────────────┐          ┌───────────────────────────────────────────┐
  │ build │──►│ test │──►│ deploy-prod      │────────► │ Environment "production"                  │
  └───────┘   └──────┘   │ environment:     │          │   Deployment #57  a1b2c3  success  alice  │
                         │   production     │          │   Deployment #56  9f8e7d  success  bob    │
                         └──────────────────┘          │   Deployment #55  4c5d6e  failed   alice  │
                                                       └───────────────────────────────────────────┘
                                                          ▲ history, rollback, protection, approvals

environment Keyword

Why it matters. "What version is running in staging right now?" should never need a Slack thread. The environment keyword tells GitLab that a job deploys somewhere. GitLab then tracks each run of that job as a deployment: which commit, which pipeline, who started it and whether it succeeded. Every other feature in this chapter (rollbacks, protection, approvals, review apps, dashboards) builds on that record.

What it is. environment is a job keyword. Its short form is just a name:

deploy-staging:
  stage: deploy
  script: ./deploy.sh staging
  environment: staging                  # short form: only the name

The long form accepts more settings:

deploy-production:
  stage: deploy
  script: ./deploy.sh production
  environment:
    name: production                    # required; creates the environment on first use
    url: https://shop.example.com       # "Open" button in the UI and the MR widget
    deployment_tier: production         # explicit tier (next topics)
    action: start                       # default: this job creates a deployment
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

Sub-keyword	Purpose
name	Environment name. Letters, digits, spaces and - _ / $ { }. Cannot start or end with /
url	Public address of the deployed app
deployment_tier	production, staging, testing, development or other
action	What the job does to the environment (table below)
on_stop	Job that tears the environment down (section 12.2)
auto_stop_in	Stop the environment automatically after a period (section 12.2)
kubernetes	Link the environment to a GitLab agent and namespace (section 12.4)

environment:action. Not every job that touches an environment deploys to it. A database backup, a smoke test or a terraform plan needs the environment's protection and scoped variables, but should not appear as a new deployment. action expresses this:

Action	Creates a deployment	Typical use
start (default)	Yes	The actual deploy
prepare	No	Work before deploying: build with environment-scoped secrets, run terraform plan
verify	No	Smoke tests or checks after deploying
access	No	Jobs that only need the environment's variables or protection, such as a backup
stop	No (stops the environment)	Tear down the environment (section 12.2)

smoke-test-production:
  stage: verify
  script: ./smoke-test.sh "$CI_ENVIRONMENT_URL"   # URL comes from the environment
  environment:
    name: production
    url: https://shop.example.com
    action: verify                                # gets production variables, no new deployment record

Predefined variables. Inside a job with an environment, GitLab sets:

Variable	Example	Notes
CI_ENVIRONMENT_NAME	review/feature-login	The name after variable expansion
CI_ENVIRONMENT_SLUG	review-feature-lo-x7k2pa	Lowercase, DNS-safe, at most 24 characters. May get a short random suffix to stay unique. Use it in hostnames and Kubernetes names
CI_ENVIRONMENT_URL	https://...	Only if url is set
CI_ENVIRONMENT_TIER	development	The deployment tier
CI_ENVIRONMENT_ACTION	start	The job's action

Environment-scoped variables. A variable can be limited to one environment or a wildcard such as review/* (section 5.2). Only jobs whose environment:name matches the scope receive it. This is the main reason to give even non-deploying jobs an environment with action: access or prepare.

The deployment lifecycle. The environment record is created the first time a pipeline contains a job that uses it. Each start job creates a deployment that follows the job's state:

created ──► running ──► success
   │            └─────► failed / canceled
   └──► blocked (waiting for approval, 12.1) ──► running ...
   └──► skipped (job did not run)

The environment's "last deployment" is the most recent successful one. A failed deployment does not change what GitLab reports as running.

Key takeaways

• environment turns a job into a tracked deployment. That record is the basis for history, rollback, protection and approvals.
• Use the long form for url, deployment_tier, action, on_stop and auto_stop_in.
• action: prepare, verify and access give a job the environment's variables and protection without creating a deployment.
• CI_ENVIRONMENT_SLUG is the DNS-safe name to use in hostnames and resource names.

Practice

1. (Beginner) Add environment: staging to a deploy job that only runs echo. Run the pipeline and open Operate > Environments. Find the deployment, its commit and the user who triggered it.
2. (Intermediate) Add a smoke-test job with action: verify and a backup job with action: access to the same environment. Confirm that neither creates a new entry in the deployment history, and that both can read a variable scoped to that environment.
3. Interview question: "Why would you attach an environment to a job that does not deploy anything?" Hint: environment-scoped variables, protected environment access rules, and keeping the deployment history accurate.

Static vs Dynamic Environments

Why it matters. A team has a few long-lived environments that always exist (staging, production) and many short-lived ones that come and go with branches (one per merge request). Hardcoding a job per branch is impossible. Dynamic environments let one job definition create a new environment for every branch.

What it is.

	Static environment	Dynamic environment
Name	Fixed string: staging, production	Contains variables: review/$CI_COMMIT_REF_SLUG
Lifetime	Long-lived, rarely stopped	Created per branch or MR, stopped when no longer needed
Count	A handful per project	Potentially hundreds
URL	Fixed	Usually derived from the slug, or reported by the deploy job
Typical use	Shared staging, production	Review apps (section 12.2), per-developer sandboxes

deploy-review:
  stage: deploy
  script: ./deploy.sh "$CI_ENVIRONMENT_SLUG"
  environment:
    name: review/$CI_COMMIT_REF_SLUG                       # one environment per branch
    url: https://$CI_ENVIRONMENT_SLUG.review.example.com   # hostname built from the DNS-safe slug
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

Which variables can appear in the name. The name is computed when the pipeline is created, so it can only use predefined variables, project/group variables and variables defined in the YAML or passed to the pipeline. It cannot use variables created in the job's script or by a dotenv report, because those do not exist yet.

Environment folders. A / in the name groups environments into folders in the UI. All review/* environments collapse into one "review" row, which keeps the environments list readable even with 200 open merge requests:

Operate > Environments
  ▸ review (37)                 ← folder: review/feature-login, review/fix-cart, ...
    staging       a1b2c3  2 h ago
    production    9f8e7d  1 d ago

The folder prefix also works as a wildcard scope for variables (review/*) and protected environments.

Dynamic URLs. Some platforms assign a random URL only after deploying (serverless functions, preview hosts). The deploy job writes the URL to a dotenv report, and environment:url references that variable (section 5.3):

deploy-preview:
  script:
    - URL=$(./deploy-preview.sh)                   # the platform prints the generated URL
    - echo "DYNAMIC_ENVIRONMENT_URL=$URL" >> deploy.env
  artifacts:
    reports:
      dotenv: deploy.env                           # GitLab reads it when the job finishes
  environment:
    name: preview/$CI_COMMIT_REF_SLUG
    url: $DYNAMIC_ENVIRONMENT_URL                  # filled in from the dotenv report

Key takeaways

• Static environments have fixed names and live long. Dynamic environments put variables in the name and are created per branch or MR.
• Environment names are resolved at pipeline creation. Variables from script or dotenv cannot be used in the name, but dotenv can provide the URL.
• A / in the name creates a UI folder and a natural wildcard for variables and protection.
• Build hostnames from CI_ENVIRONMENT_SLUG, not from the raw branch name.

Practice

1. (Beginner) Create a dynamic environment review/$CI_COMMIT_REF_SLUG with an echo deploy job. Push three branches and look at the folder in Operate > Environments.
2. (Intermediate) Write a deploy job that generates a random URL in its script, passes it through a dotenv report, and sets it as environment:url. Check that the "Open" button in the environment page uses that URL.
3. Interview question: "A developer wants the environment name to include the version number computed in the build job's script. Why does that not work, and what would you suggest?" Hint: when the name is evaluated, and whether the version could come from a tag or pipeline variable instead.

Environment Tiers and URLs

Why it matters. Names are free text. One team calls production prod, another live, a third eu-production. GitLab needs a fixed vocabulary to know which environments matter most: for group-level protection rules, for DORA metrics such as deployment frequency, and for showing the right environment in merge requests. Deployment tiers provide it. URLs connect the environment record to the running application.

Tiers. Every environment has exactly one of five tiers. If you do not set deployment_tier, GitLab guesses from the name:

Tier	Names GitLab maps to it automatically (examples)	Meaning
production	production, prod, live	Serves real users
staging	staging, stage, pre, demo, model	Last check before production
testing	test, qa, qc	Automated or manual testing
development	dev, review, trunk	Developer and review environments
other	Anything else	Unclassified

Set the tier explicitly whenever the name is ambiguous:

deploy-eu:
  script: ./deploy.sh eu-central
  environment:
    name: eu-central               # GitLab would guess "other"
    deployment_tier: production    # it really serves customers

What tiers drive.

Feature	How the tier is used
Group-level protected environments (Premium)	Protect every production-tier environment in all projects of a group (next topic)
DORA metrics and CI/CD analytics	Deployment frequency, lead time and change failure rate count deployments to production tier
CI_ENVIRONMENT_TIER	Scripts can branch on the tier instead of parsing names

URLs. environment:url adds an Open button on the environment page and a View app link in the merge request widget. It can use variables, so dynamic environments get per-branch links.

Route maps. For static sites and documentation, a route map links changed source files to their public pages. Reviewers then see "View on review/..." next to each changed file in the merge request. The map is a YAML file at .gitlab/route-map.yml:

# .gitlab/route-map.yml
- source: /content\/(.+?)\.md/      # regex on the source path (delimited by /)
  public: '\1.html'                  # public path, using capture groups
- source: 'index.md'                 # exact path (quoted string)
  public: 'index.html'

Key takeaways

• The five tiers are production, staging, testing, development and other. GitLab infers the tier from common names.
• Set deployment_tier explicitly for names like eu-central or customer-a, or group-level protection and DORA metrics will miss them.
• environment:url powers the Open button and the MR's "View app" link. Route maps link individual changed files to their deployed pages.

Practice

1. (Beginner) Create environments named prod, qa and blue with echo jobs. Check which tier each received, then fix blue with deployment_tier.
2. (Intermediate) Add a route map to a small static site with Markdown pages deployed as a review app. Open a merge request that changes one page and use the per-file link to view it.
3. Interview question: "Your DORA dashboard shows zero production deployments although the team deploys daily. What is the likely cause?" Hint: how GitLab decides which environments are production.

Protected Environments

Why it matters. Protected branches stop just anyone from changing main, but a deploy job on an unprotected branch can still deploy to production unless something stops it. A Developer could add a job with environment: production to a feature branch, or retry an old deploy job. Protected environments (Premium) move the decision of who may deploy from the pipeline YAML, which anyone with push access can edit, into project or group settings that only Maintainers control.

What it is. A protected environment has a list of who is allowed to deploy: roles, specific users or groups. When a job targets that environment, GitLab checks the person who triggered the job. If that person is not allowed, the job fails with an access error and the deployment never runs. The check applies to the first run and to every retry or manual play.

deploy-prod job created by Dana (Developer)
        │
        ▼
 Is "production" protected? ──no──► run
        │ yes
        ▼
 Is Dana in "allowed to deploy"? ──no──► job fails: "You do not have permission to deploy"
        │ yes
        ▼
 Approval rules? (next topic) ──none──► run

Configuring. In Settings > CI/CD > Protected environments, choose the environment (names and wildcards such as review/* work) and the deployers. The same through the API:

# Protect "production": Maintainers (access level 40) and the "sre" group (ID 77) may deploy
curl --request POST \
  --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  --header "Content-Type: application/json" \
  --data '{
        "name": "production",
        "deploy_access_levels": [
          { "access_level": 40 },
          { "group_id": 77 }
        ]
      }' \
  "https://gitlab.example.com/api/v4/projects/12345/protected_environments"

Group-level protection. Large organizations protect environments by tier across every project in a group (Group > Settings > CI/CD > Protected environments). One rule then covers every production-tier environment in hundreds of projects, and project Maintainers cannot weaken it. This is why setting deployment_tier correctly matters.

Combining it with the rest. A protected environment is one layer:

Layer	What it stops	Where
Protected branch or tag	Unreviewed code reaching the deploy ref	Repository settings
Protected variables (section 5.2)	Secrets reaching pipelines on unprotected refs	CI/CD variables
Environment-scoped variables	Production credentials reaching non-production jobs	CI/CD variables
Protected environment	Unauthorized people starting a production deployment	CI/CD settings
Deployment approvals	A single person deploying without a second look	CI/CD settings

A common setup: production credentials are both protected and scoped to production, the production environment is protected for the release team, and the deploy job runs only on main.

Things to know.

• Protection checks the person who triggered the job. A scheduled pipeline runs as the schedule owner, so that person must be allowed to deploy.
• Users and groups listed as deployers must be members of the project, or the group must be invited to it.
• Jobs that use the environment with action: prepare, verify or access are subject to the same protection.

Key takeaways

• Protected environments (Premium) restrict who can run jobs that target an environment, independently of who can edit .gitlab-ci.yml.
• Allow deployers by role, user or group. Wildcards protect whole folders.
• Group-level rules protect a tier across all projects. They rely on correct deployment_tier values.
• Combine with protected branches, protected and environment-scoped variables, and approvals for defense in depth.

Practice

1. (Beginner) Protect a production environment so that only Maintainers can deploy. Trigger the deploy job as a Developer and read the failure message.
2. (Intermediate) Protect production with the API, then use GET /projects/:id/protected_environments to list the rule. Add a group as an additional deployer and verify that a member of that group with the Developer role can now deploy.
3. Interview question: "A Developer cannot edit protected branches but can push a feature branch. List every way they could still reach production credentials or deploy, and how you would close each gap." Hint: YAML changes on a feature branch, unprotected variables, environment scopes, and who triggers retries.

Deployment Approvals

Why it matters. Some changes need a second person: a regulated release, a database migration, a Friday-evening hotfix. Asking for approval in chat leaves no record and is easy to skip. Deployment approvals (Premium) make GitLab hold the deployment until the right people approve it in GitLab, and store who approved, when and why.

What it is. Approval rules belong to a protected environment. Each rule names an approver (a role, a user or a group) and how many approvals are required. When a job targets the environment, its deployment goes into the blocked state and waits until all rules are satisfied.

pipeline ──► deploy-prod  [waiting for approval]
                 │
   Rule "QA"    needs 1 of group qa         [approved] by Ana (10:02)
   Rule "SRE"   needs 1 of group sre        [approved] by Bo  (10:15)  "Load test OK"
                 │
                 ▼
             deployment proceeds    (a rejection by any approver blocks it for good)

Configuring. When protecting the environment, add approval rules in the UI or the API:

curl --request POST \
  --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  --header "Content-Type: application/json" \
  --data '{
        "name": "production",
        "deploy_access_levels": [ { "group_id": 77 } ],
        "approval_rules": [
          { "group_id": 81, "required_approvals": 1 },
          { "access_level": 40, "required_approvals": 1 }
        ]
      }' \
  "https://gitlab.example.com/api/v4/projects/12345/protected_environments"

The first rule requires one approval from the QA group (ID 81), the second one approval from any Maintainer. Both must be satisfied.

Approving. Approvers open the environment, find the blocked deployment and choose Approve or Reject, optionally with a comment. Automation can do the same through the Deployments API:

# Approve deployment 4711 with a comment
curl --request POST \
  --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  --data "status=approved" \
  --data "comment=Load test passed, change ticket CHG-2231" \
  "https://gitlab.example.com/api/v4/projects/12345/deployments/4711/approval"

Rules of thumb.

Question	Answer
Can the person who triggered the pipeline approve?	No, by default. The project setting Allow pipeline triggerer to approve deployment changes this.
Does a rejection count?	One rejection blocks the deployment. Run a new pipeline after fixing the problem.
Do approvals carry over to a retried job?	No. A new deployment needs new approvals.
Is the job still manual after approval?	If the job is also when: manual, someone allowed to deploy still has to start it. Use when: manual only if you want an extra "go" moment after approval.
Where is the audit trail?	On the deployment's approval details and in audit events.

Approvals versus manual jobs. when: manual asks anyone who can run the job to click play. Approvals ask specific people to sign off, require a count, record a decision and comment, and cannot be bypassed by editing the YAML. Use manual jobs for convenience gates and approvals for control.

Key takeaways

• Deployment approvals (Premium) attach approval rules to a protected environment. The deployment stays blocked until every rule is satisfied.
• Rules can require approvals from roles, users or groups, with a count per rule.
• Approve or reject in the UI or with POST /deployments/:id/approval. Each decision is recorded with its comment.
• By default, the person who triggered the pipeline cannot approve their own deployment.

Practice

1. (Beginner) Add an approval rule requiring one Maintainer approval to a protected production environment. Run a deploy and approve it from the environment page with a comment.
2. (Intermediate) Configure two rules (QA group and SRE group). Write a small script that lists blocked deployments with GET /projects/:id/deployments?status=blocked and approves one through the API. Then reject a different deployment and observe the job.
3. Interview question: "Compare when: manual on a production job with deployment approvals. When is a manual job enough?" Hint: who can click it, whether the YAML can be changed to remove it, and what an auditor wants to see.

Environment Dashboard

Why it matters. An on-call engineer at 3 a.m. needs to answer "what changed, where and when" quickly. The environment pages answer it for one project; the environments dashboard answers it across many.

What it is. GitLab has three levels of environment views:

View	Where	Tier	Shows
Environments list	Operate > Environments in a project	All	Every environment with its last deployment, commit, deployer, URL, and buttons to open, stop, roll back or re-deploy
Environment details	Click an environment	All	Full deployment history, approvals, Kubernetes resources when linked to an agent (section 12.4), stop and rollback controls
Environments dashboard	Your work > Environments	Premium	Your chosen projects side by side, each with its environments' latest deployment and pipeline status

Environments dashboard
┌────────────────────────┬────────────────────────┬────────────────────────┐
│ shop/web               │ shop/api               │ shop/payments          │
│  production  ● success │  production  ● success │  production  ● failed  │
│   a1b2c3  alice  12m   │   77aa01  bob   3h     │   c0ffee  dana  5m     │
│  staging     ● running │  staging     ● success │  staging     ● success │
└────────────────────────┴────────────────────────┴────────────────────────┘

Reading deployments with the API. The data behind these views is available for scripts, status pages and chat bots:

# Last successful production deployment: commit, user and finish time
curl --silent --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.example.com/api/v4/projects/12345/deployments?environment=production&status=success&order_by=finished_at&sort=desc&per_page=1" \
  | jq '.[0] | {sha, ref, user: .user.username, finished_at}'

# All environments that are still running (not stopped)
curl --silent --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.example.com/api/v4/projects/12345/environments?states=available" \
  | jq -r '.[] | "\(.name)\t\(.tier)\t\(.external_url)"'

Deployments in merge requests and commits. A merge request shows which environments its changes were deployed to, with links. A commit can be checked in the environment's history to see whether, when and where it was deployed. This is often the fastest way to answer "is my fix live yet?".

Key takeaways

• The project environments list and details pages are available on all tiers and offer history, stop, rollback and re-deploy.
• The cross-project environments dashboard (Premium) shows many projects' environments at once.
• The Environments and Deployments APIs expose the same data for scripts and status tooling.

Practice

1. (Beginner) After several deployments, use the environment details page to find which commit is in production and who deployed it. Then find the same data with the Deployments API.
2. (Intermediate) Write a script that prints, for a list of project IDs, the commit SHA and age of the last successful production deployment, and flags any project whose last production deployment failed.
3. Interview question: "During an incident you suspect a deployment caused it, but several services deployed in the last hour. How would you use GitLab to narrow it down quickly?" Hint: cross-project views, deployment timestamps, and the commits each deployment contained.

<a id="122-review-apps"></a>

12.2 Review Apps

A review app is a temporary, per-merge-request deployment of the application, so reviewers can click through a change instead of imagining it from a diff. This subchapter shows how to build review apps from dynamic environments, how to tear them down reliably, and how to keep forgotten ones from piling up and costing money.

 MR !41 "New checkout"  ──► review/new-checkout   https://review-new-checkout.review.example.com
 MR !42 "Fix cart"      ──► review/fix-cart       https://review-fix-cart.review.example.com
 main                   ──► staging ──► production

 MR merged or branch deleted  ──► on_stop job tears down review/<branch>
 nobody pushes for 3 days     ──► auto_stop_in stops it anyway

Configuring Review Apps

Why it matters. Screenshots in a merge request show what the author chose to show. A product manager, designer or QA engineer who can open the actual running change finds problems a code reviewer will not: a broken layout on mobile, a confusing flow, a slow page. Catching them before merge is much cheaper than after release. Review apps also give automated tests such as DAST, accessibility and performance testing (chapters 10 and 11) a realistic target per merge request.

What it is. A review app is not a separate feature. It is the combination of:

1. A dynamic environment named after the branch (section 12.1).
2. A deploy job that runs in merge request pipelines and deploys to a unique hostname or namespace.
3. A stop job that removes the deployment (next topic).
4. Optionally, auto_stop_in for automatic expiry.

GitLab then shows a View app button in the merge request widget.

A complete example for a container image deployed to Kubernetes with Helm:

stages: [build, deploy]

variables:
  IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

build:
  stage: build
  image: docker:27.3
  services: [docker:27.3-dind]
  script:
    - echo "$CI_REGISTRY_PASSWORD" | docker login -u "$CI_REGISTRY_USER" --password-stdin "$CI_REGISTRY"
    - docker build -t "$IMAGE" .
    - docker push "$IMAGE"

deploy-review:
  stage: deploy
  image: alpine/k8s:1.31.2                 # contains kubectl and helm
  script:
    - kubectl config use-context acme/infra:review-agent   # GitLab agent context (section 12.4)
    # One release and one namespace per review app; the slug already starts with "review-"
    - >
      helm upgrade --install "$CI_ENVIRONMENT_SLUG" ./chart
      --namespace "$CI_ENVIRONMENT_SLUG" --create-namespace
      --set image.tag="$CI_COMMIT_SHA"
      --set ingress.host="$CI_ENVIRONMENT_SLUG.review.example.com"
      --wait --timeout 5m
  environment:
    name: review/$CI_COMMIT_REF_SLUG
    url: https://$CI_ENVIRONMENT_SLUG.review.example.com
    on_stop: stop-review                   # next topic
    auto_stop_in: 3 days                   # topic after next
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

stop-review:
  stage: deploy
  image: alpine/k8s:1.31.2
  variables:
    GIT_STRATEGY: none                     # the branch may already be deleted
  script:
    - kubectl config use-context acme/infra:review-agent
    - helm uninstall "$CI_ENVIRONMENT_SLUG" --namespace "$CI_ENVIRONMENT_SLUG" --ignore-not-found
    - kubectl delete namespace "$CI_ENVIRONMENT_SLUG" --ignore-not-found
  environment:
    name: review/$CI_COMMIT_REF_SLUG
    action: stop
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      when: manual                         # runs on demand or when GitLab stops the environment

Design decisions for review apps.

Decision	Options	Advice
Isolation	Namespace per app, shared namespace with prefixed names, separate VM	A namespace per review app makes cleanup a single delete namespace
Routing	Wildcard DNS *.review.example.com plus an ingress controller	One wildcard DNS record and certificate, then hostnames are free
Data	Shared review database, a database per app, seeded fixtures	Seeded per-app data avoids reviewers stepping on each other
Secrets	Variables scoped to review/*	Never give review apps production credentials
Who triggers	Every MR, or only labeled MRs	Gate on a label for expensive apps: if: $CI_MERGE_REQUEST_LABELS =~ /review-app/
Hostname length	DNS labels are limited to 63 characters	Use CI_ENVIRONMENT_SLUG (24 characters), not the branch name

Review apps without Kubernetes. The same pattern works for any target that can host many instances: a static site uploaded to s3://bucket/$CI_ENVIRONMENT_SLUG/, a Docker Compose project per slug on a shared VM, a preview deployment on a hosting platform with a dotenv URL, or GitLab Pages parallel deployments (section 12.4).

Key takeaways

• A review app is a dynamic environment plus a deploy job in MR pipelines plus a stop job. GitLab adds a View app link to the merge request.
• Name environments review/$CI_COMMIT_REF_SLUG and build hostnames and namespaces from CI_ENVIRONMENT_SLUG.
• Isolate each review app (namespace, prefix, path) so it can be removed in one step, and scope only non-production secrets to review/*.
• Gate expensive review apps on a label or manual trigger.

Practice

1. (Beginner) Deploy a static site as a review app to a path or bucket prefix named after CI_ENVIRONMENT_SLUG. Open a merge request and use the View app button.
2. (Intermediate) Make review apps opt-in: deploy only when the merge request has the review-app label. Add the label to an existing MR and confirm that a new pipeline creates the app.
3. Interview question: "Your review apps share one staging database, and QA keeps reporting flaky results. How would you redesign the data layer?" Hint: per-app databases or schemas, seeded fixtures, and the cost and time of each approach.

environment:on_stop

Why it matters. Starting a review app is easy. Remembering to delete it is not. Every forgotten review app keeps pods, databases and load balancers running. on_stop links a deploy job to the job that undoes it, so GitLab can run the teardown for you at the right moment.

What it is. environment:on_stop: <job name> on the deploy job names a stop job. The stop job targets the same environment with environment:action: stop. GitLab runs the stop job when:

• Someone clicks Stop on the environment (in the environments list or the MR widget).
• The branch is deleted, for example when a merge request is merged with "Delete source branch".
• The environment reaches its auto_stop_in time (next topic).
• The stop environment API is called.

When the stop job succeeds, the environment moves to stopped.

deploy-review  ──(environment: review/x, on_stop: stop-review)──► environment "review/x" available
                                                                         │
       branch deleted / Stop clicked / auto_stop_in / API ───────────────┤
                                                                         ▼
stop-review    ──(environment: review/x, action: stop)───────────────► environment "review/x" stopped

Requirements that trip people up.

Requirement	Why
Stop job uses the same environment name	GitLab matches the two jobs by environment
Stop job has action: stop	Otherwise it is just another deployment
Stop job is in the same pipeline, in the same stage as the deploy job or a later one, with compatible rules	GitLab looks for the stop job in the pipeline of the last deployment. If rules excluded it, there is nothing to run
Stop job is when: manual	It must not run immediately after the deploy
GIT_STRATEGY: none (or GIT_DEPTH with a checkout of a still-existing ref)	When the branch has been deleted, cloning it would fail
Teardown does not need repository files	With GIT_STRATEGY: none there is no checkout; use an image that has the tools, or fetch scripts from elsewhere

Using needs with stop jobs. If the stop job is in the same stage, it does not need needs. If you use needs elsewhere, make sure the stop job does not depend on jobs that might not exist in that pipeline; a stop job blocked by missing dependencies cannot run.

Handling stop failures. If the stop job fails, the environment stays available and the resources keep running. Make teardown idempotent (--ignore-not-found, || true only for "already gone" cases) so retries succeed. An environment can also be force-stopped in the UI or API without running a job, which marks it stopped but leaves any resources behind.

# Stop an environment through the API (runs its on_stop job)
curl --request POST --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.example.com/api/v4/projects/12345/environments/88/stop"

# Force: mark it stopped without running the stop job (resources are NOT removed)
curl --request POST --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  --data "force=true" \
  "https://gitlab.example.com/api/v4/projects/12345/environments/88/stop"

Beyond review apps. Any temporary environment benefits from on_stop: a load-test cluster, a per-branch database, a demo for a customer.

Key takeaways

• on_stop names the stop job. The stop job uses the same environment name, action: stop and usually when: manual.
• GitLab runs it on Stop, branch deletion, auto_stop_in expiry and the stop API.
• Keep the stop job in the same pipeline with compatible rules, and use GIT_STRATEGY: none because the branch may be gone.
• Make teardown idempotent. A failed stop job leaves the environment and its resources running.

Practice

1. (Beginner) Add an on_stop job to a review app that deletes its files or namespace. Merge the MR with "Delete source branch" enabled and confirm that the stop job ran and the environment is stopped.
2. (Intermediate) Break the setup on purpose: give the stop job different rules from the deploy job. Observe what happens when you click Stop, then fix it. Also make the teardown idempotent and run it twice.
3. Interview question: "Why does the stop job usually set GIT_STRATEGY: none, and what does that imply for how the teardown script is written?" Hint: the state of the branch when the stop job runs, and where the teardown logic must live.

auto_stop_in

Why it matters. Merge requests get abandoned. Branches stay open for weeks while nobody looks at the review app. on_stop only helps if something triggers it. auto_stop_in provides that trigger based on time, so an idle review app expires on its own.

What it is. environment:auto_stop_in sets how long after the last deployment the environment should be stopped. GitLab periodically checks for expired environments and stops them, running the on_stop job if one is defined.

deploy-review:
  script: ./deploy-review.sh
  environment:
    name: review/$CI_COMMIT_REF_SLUG
    on_stop: stop-review
    auto_stop_in: 2 days            # natural language durations: "1 hour", "1 week", "30 minutes"
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

How the timer behaves.

Event	Effect on the timer
A new deployment to the environment	The timer restarts from that deployment
A user clicks Prevent auto stopping (pin) on the environment	The environment is kept until unpinned or stopped manually
auto_stop_in: never	Disables automatic stopping for that environment
The timer expires	GitLab stops the environment and runs on_stop if configured

The check runs as a background task, so an environment may be stopped somewhat after its exact expiry time.

Choosing a duration.

Environment type	Typical auto_stop_in
Review app with significant cost (cluster resources, databases)	1 day to 3 days
Cheap static review app	1 week
Load test or ephemeral demo	2 hours
Staging, production	Do not set it (they should never expire)

Restarting a stopped review app. Re-run the deploy job of the latest pipeline, or push a new commit. The environment becomes available again and the timer starts over.

Key takeaways

• auto_stop_in stops an environment a set time after its last deployment, and runs its on_stop job.
• Each new deployment resets the timer. Pin an environment in the UI to keep it, or use never.
• Use short durations for costly review apps and never set it on long-lived environments.

Practice

1. (Beginner) Set auto_stop_in: 30 minutes on a review app and observe the "Auto stop in" time on the environment page. Wait for it to expire and check that the stop job ran.
2. (Intermediate) Pin one review app with Prevent auto stopping, push a new commit to another, and compare the stop times of both. Then read the auto_stop_at field of each environment through the Environments API.
3. Interview question: "Your cloud bill doubled because of review apps even though on_stop is configured everywhere. What would you check?" Hint: abandoned but undeleted branches, failed stop jobs, and resources created outside the namespace the stop job deletes.

Cleaning Up Stale Environments

Why it matters. Even with on_stop and auto_stop_in, environments accumulate. Projects that existed before you added auto-stop, stop jobs that failed, branches renamed by force pushes: after a year a project can have thousands of stopped and forgotten environments. They slow down the UI, confuse dashboards and, when a stop job failed, still hold real resources.

What it is. Cleanup has two levels: stopping environments (and their resources) and deleting stopped environment records. GitLab provides API endpoints for both.

Goal	API	Notes
Stop environments with no activity since a date	POST /projects/:id/environments/stop_stale with before	Runs their on_stop jobs. Protected environments are skipped
Delete stopped review apps older than a date	DELETE /projects/:id/environments/review_apps with before, limit, dry_run	Only affects environments in the review/ folder. dry_run is true by default
Delete one environment	DELETE /projects/:id/environments/:environment_id	The environment must be stopped first

A scheduled cleanup pipeline. Run it from a pipeline schedule (section 3.5) with a project access token that has the api scope, stored as a masked variable:

cleanup-environments:
  image: alpine:3.20
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule" && $SCHEDULE_TYPE == "env-cleanup"
  before_script:
    - apk add --no-cache curl jq coreutils
  script:
    - API="$CI_API_V4_URL/projects/$CI_PROJECT_ID"
    - CUTOFF=$(date -u -d '14 days ago' +%Y-%m-%dT%H:%M:%SZ)   # GNU date from coreutils
    # 1. Stop environments that have not been updated for 14 days (runs their on_stop jobs)
    - >
      curl --fail --request POST --header "PRIVATE-TOKEN: $CLEANUP_TOKEN"
      "$API/environments/stop_stale?before=$CUTOFF"
    # 2. Preview which stopped review apps would be deleted
    - >
      curl --fail --silent --request DELETE --header "PRIVATE-TOKEN: $CLEANUP_TOKEN"
      "$API/environments/review_apps?before=$CUTOFF&dry_run=true"
      | jq -r '.scheduled_entries[].name'
    # 3. Delete them for real
    - >
      curl --fail --request DELETE --header "PRIVATE-TOKEN: $CLEANUP_TOKEN"
      "$API/environments/review_apps?before=$CUTOFF&dry_run=false&limit=100"

Do not forget the resources outside GitLab. Deleting an environment record removes nothing in your cloud. A complete cleanup also covers:

• Kubernetes namespaces with a review- prefix whose environment no longer exists. A periodic job can list namespaces with a label such as app.gitlab.com/env and delete orphans.
• Container images tagged per branch in the registry (section 7.4).
• DNS records, buckets, databases or certificates created per review app.
• stop-review jobs that failed. Search for failed jobs named stop-review and retry them after fixing the cause.

Key takeaways

• Stopping and deleting are different steps. Only stopped environments can be deleted.
• stop_stale stops inactive environments and runs their on_stop jobs. DELETE environments/review_apps bulk-deletes old stopped review apps, with dry_run on by default.
• Schedule a cleanup job with a scoped access token.
• Clean up the resources outside GitLab too: namespaces, images, DNS, databases.

Practice

1. (Beginner) Call DELETE /projects/:id/environments/review_apps with dry_run=true on a project with several review apps and read the response.
2. (Intermediate) Build the scheduled cleanup job above, then add a step that deletes Kubernetes namespaces labeled as review apps whose environment is no longer available.
3. Interview question: "Design a lifecycle policy for review apps in a monorepo with 300 open merge requests and a fixed cloud budget." Hint: opt-in creation, short auto_stop_in, resource quotas per namespace, scheduled cleanup, and reporting on cost per environment.

<a id="123-deployment-strategies"></a>

12.3 Deployment Strategies

Every deployment is a bet that the new version works. Deployment strategies limit how much you lose when the bet goes wrong: they expose a new version to a small share of traffic first, keep the old version ready to take over, or separate shipping code from turning it on. This subchapter covers the common strategies, how to express them in GitLab pipelines, and how to roll back and guard against unsafe deployments.

Strategy	Exposure of the new version	Rollback speed	Extra cost	Best for
Recreate (stop old, start new)	100% at once, with downtime	Slow (redeploy)	None	Internal tools, batch jobs
Rolling / incremental	Grows in steps (10% → 100%)	Medium	Little	Most stateless services
Canary	Small share, then a decision	Fast (remove canary)	Small	Risky changes on high-traffic services
Blue-green	0% then 100% by switching traffic	Instant (switch back)	Double capacity during switch	Releases that need instant rollback
Feature flags	Per feature, per user group	Instant (toggle)	Code complexity	Separating deploy from release

Incremental Rollouts

Why it matters. Deploying a new version to all servers at once means every user meets a bug at the same moment. Rolling the version out in steps (10%, 25%, 50%, 100%) means that if errors spike at 10%, 90% of users never notice, and you stop the rollout.

What it is. An incremental rollout replaces old instances with new ones in several steps, with a check between steps. In GitLab you model each step as a job: either manual (a person decides to continue) or timed (when: delayed continues automatically after a pause, unless someone cancels). The jobs all deploy to the same environment and share a resource_group (section 4.5) so two steps never run at once.

          ┌── manual ──┐   ┌── manual ──┐   ┌── manual ──┐   ┌── manual ──┐
deploy ──►│ rollout 10%│──►│ rollout 25%│──►│ rollout 50%│──►│ rollout100%│
          └────────────┘   └────────────┘   └────────────┘   └────────────┘
              ▲ check dashboards between steps; stop clicking if errors rise

          timed variant: when: delayed, start_in: 5 minutes between steps

A timed rollout. A hidden job holds the logic, and each step sets a percentage:

stages: [build, rollout]

.rollout:
  stage: rollout
  image: alpine/k8s:1.31.2
  resource_group: production                       # one step at a time
  environment:
    name: production
    url: https://shop.example.com
  script:
    - kubectl config use-context acme/infra:prod-agent
    # Scale the "new" deployment up and the "old" one down to match the percentage
    - ./scripts/rollout.sh --image "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA" --percent "$ROLLOUT_PERCENT"
    - ./scripts/check-error-rate.sh --max 1%      # fail the step if errors are too high
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

rollout-10:
  extends: .rollout
  variables: { ROLLOUT_PERCENT: "10" }            # first step runs immediately

rollout-25:
  extends: .rollout
  needs: [rollout-10]
  variables: { ROLLOUT_PERCENT: "25" }
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: delayed
      start_in: 10 minutes                        # observation window before continuing

rollout-50:
  extends: .rollout
  needs: [rollout-25]
  variables: { ROLLOUT_PERCENT: "50" }
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: delayed
      start_in: 10 minutes

rollout-100:
  extends: .rollout
  needs: [rollout-50]
  variables: { ROLLOUT_PERCENT: "100" }
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual                                # a human confirms the last step

A delayed job can be canceled or started immediately from the pipeline view. A failed check job stops the chain because the later jobs need it.

Kubernetes's built-in rolling update. A single kubectl apply with a new image already replaces pods gradually, controlled by maxSurge and maxUnavailable in the Deployment. That protects availability, but it rolls through all pods in minutes without waiting for your metrics. Explicit steps add the observation window. Progressive delivery controllers (Argo Rollouts, Flagger) automate the "check metrics, then continue" loop if you need it at scale.

Auto DevOps offers this pattern built in: INCREMENTAL_ROLLOUT_MODE set to manual or timed creates rollout 10% through rollout 100% jobs (section 12.6).

Key takeaways

• Incremental rollouts expose a new version in steps, with a check between steps, so problems affect a fraction of users.
• Model steps as jobs with needs, using when: manual for human gates or when: delayed with start_in for timed ones.
• Use one resource_group so steps never overlap, and make each step fail if health checks fail.
• Kubernetes rolling updates handle availability. Explicit steps or a progressive delivery controller handle risk.

Practice

1. (Beginner) Build a four-step manual rollout pipeline whose jobs only echo the percentage. Run it and play each step from the pipeline graph.
2. (Intermediate) Convert the middle steps to when: delayed with start_in: 2 minutes, and add a check script that fails randomly. Observe how a failed check stops the remaining steps, and how to cancel a delayed job.
3. Interview question: "Kubernetes already performs rolling updates. Why would you build an incremental rollout in the pipeline on top of it?" Hint: observation time between steps versus speed of replacement, and who decides to continue.

Canary Deployments

Why it matters. Some bugs only appear under real traffic: a slow query on production-sized data, a memory leak after an hour, an incompatibility with one client version. A canary runs the new version alongside the stable one and sends it a small share of real requests. If the canary's error rate or latency is worse than the stable version's, you remove it before most users are affected. The name comes from the canaries miners carried to detect gas early.

What it is. A canary deployment has two tracks of the same application:

                      ┌──────────────────────────────┐
                 95%  │ stable  v1.8  (10 replicas)  │
 users ──► router ───►│                              │
                  5%  │ canary  v1.9  (1 replica)    │──► compare error rate, latency
                      └──────────────────────────────┘
                                     │
                 healthy ────────────┼──────────── unhealthy
                    ▼                                  ▼
          promote: stable = v1.9              abort: delete canary

The difference from an incremental rollout is the explicit decision point: the canary stays at a fixed small share until someone (or an automated analysis) promotes or aborts it.

A canary with NGINX Ingress. The NGINX Ingress controller can split traffic by weight between two ingresses for the same host. The pipeline has three jobs: deploy the canary, promote, abort.

deploy-canary:
  stage: canary
  image: alpine/k8s:1.31.2
  resource_group: production
  environment:
    name: production
    deployment_tier: production
  script:
    - kubectl config use-context acme/infra:prod-agent
    - >
      helm upgrade --install shop-canary ./chart --namespace shop
      --set image.tag="$CI_COMMIT_SHA"
      --set replicaCount=1
      --set track=canary
      --set ingress.annotations."nginx\.ingress\.kubernetes\.io/canary"=true
      --set ingress.annotations."nginx\.ingress\.kubernetes\.io/canary-weight"=5
      --wait
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

promote-canary:
  stage: promote
  image: alpine/k8s:1.31.2
  resource_group: production
  environment:
    name: production
  script:
    - kubectl config use-context acme/infra:prod-agent
    # Stable release takes the canary's version, then the canary is removed
    - helm upgrade --install shop ./chart --namespace shop --set image.tag="$CI_COMMIT_SHA" --wait
    - helm uninstall shop-canary --namespace shop
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual

abort-canary:
  stage: promote
  image: alpine/k8s:1.31.2
  resource_group: production
  environment:
    name: production
    action: access                       # does not record a deployment of this commit
  script:
    - kubectl config use-context acme/infra:prod-agent
    - helm uninstall shop-canary --namespace shop
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual

Making the decision. A canary is only useful if you compare it against the stable track:

Signal	How to compare
Error rate	HTTP 5xx ratio per track label in Prometheus
Latency	p95 and p99 per track
Business metric	Checkout conversion, sign-ups
Logs	New exception types only in canary pods

Label pods with the track (track: canary versus track: stable) so every metric can be split. Keep the canary running long enough to see slow failures, and make sure it gets enough traffic for the numbers to mean something.

Where GitLab helps. The environment's Kubernetes overview (section 12.4) shows the pods of both tracks. Auto DevOps creates a canary job when CANARY_ENABLED is set (section 12.6). For automated analysis and traffic shifting, tools such as Flagger or Argo Rollouts run in the cluster and are deployed by the pipeline like any other manifest.

Key takeaways

• A canary runs the new version next to the stable one, sends it a small, fixed share of traffic and waits for a promote or abort decision.
• Traffic splitting needs router support: ingress weights, a service mesh, or a load balancer with weighted targets.
• Model the canary as deploy, promote (manual) and abort (manual) jobs that share a resource_group.
• Label everything by track and compare canary against stable, not against a fixed threshold alone.

Practice

1. (Beginner) Explain in your own words the difference between a canary and the first step of an incremental rollout. Then sketch the three pipeline jobs you need.
2. (Intermediate) In a local cluster with the NGINX Ingress controller, deploy a stable and a canary version that return different responses. Send 1000 requests with curl in a loop, count the responses per version, and change the canary weight from the pipeline.
3. Interview question: "Your canary receives 1% of traffic for 10 minutes and shows no errors, but production breaks after promotion. What went wrong with the experiment?" Hint: sample size, sticky sessions, which users and endpoints the canary actually received, and time-dependent failures.

Blue-Green Deployments

Why it matters. Rolling and canary deployments mix versions for a while, and rolling back means rolling forward again with the old version. Some systems need a cleaner switch: all users on one version or the other, and a rollback that takes seconds. Blue-green keeps two complete production environments and moves all traffic from one to the other at once.

What it is. "Blue" serves traffic. "Green" is idle. You deploy the new version to green, test it there with production configuration, then switch the router. Blue stays untouched, so switching back is instant.

Before:   router ──► BLUE  (v1.8)  live           GREEN (idle)
Deploy:   router ──► BLUE  (v1.8)  live           GREEN (v1.9) ◄── smoke tests on internal URL
Switch:   router ──► GREEN (v1.9)  live           BLUE  (v1.8) kept warm for rollback
Rollback: router ──► BLUE  (v1.8)  live           (one switch, no redeploy)

In GitLab. Model each color as its own environment, and the switch as a job on the production environment that records which color went live:

variables:
  KUBE_CONTEXT: acme/infra:prod-agent

.k8s:
  image: alpine/k8s:1.31.2
  before_script:
    - kubectl config use-context "$KUBE_CONTEXT"
    # The idle color is whichever one the production Service does not select
    - LIVE=$(kubectl -n shop get service shop -o jsonpath='{.spec.selector.color}')
    - if [ "$LIVE" = "blue" ]; then IDLE=green; else IDLE=blue; fi
    - echo "live=$LIVE idle=$IDLE"

deploy-idle:
  extends: .k8s
  stage: deploy
  script:
    - helm upgrade --install "shop-$IDLE" ./chart -n shop --set color="$IDLE" --set image.tag="$CI_COMMIT_SHA" --wait
    - ./scripts/smoke-test.sh "http://shop-$IDLE.shop.svc.cluster.local"   # internal URL, no real users
  environment:
    name: production-idle
    deployment_tier: staging
  resource_group: production
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

switch-traffic:
  extends: .k8s
  stage: release
  script:
    # Point the public Service at the idle color: all new requests go there at once
    - kubectl -n shop patch service shop -p "{\"spec\":{\"selector\":{\"app\":\"shop\",\"color\":\"$IDLE\"}}}"
  environment:
    name: production
    url: https://shop.example.com
  resource_group: production
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual

Rolling back is running switch-traffic again (it always flips to the other color), or re-deploying the previous production deployment from the environment page (later topic).

On cloud platforms the switch is usually a load balancer or DNS change: swapping target groups on an AWS Application Load Balancer, swapping deployment slots in Azure App Service (az webapp deployment slot swap), or moving a Cloud Run traffic tag.

The hard part: state. Two application versions can share a database. A schema change that blue cannot read breaks the rollback. Use the expand and contract pattern:

Step	Release	Schema	Both versions work?
1. Expand	v1.9	Add the new column, keep the old one; write both	Yes
2. Migrate	v1.9	Backfill the new column	Yes
3. Switch reads	v2.0	Read the new column	Yes, old version still has its column
4. Contract	v2.1	Drop the old column once no version uses it	Only after rollback window has passed

Key takeaways

• Blue-green keeps two full environments and switches all traffic at once. Rollback is one switch back.
• Test the idle color with production configuration before switching. Model the switch as its own job on the production environment.
• It needs double capacity during the switch, and database changes must be compatible with both versions (expand and contract).

Practice

1. (Beginner) Draw the state of blue, green and the router before deployment, after deployment, after switching, and after a rollback.
2. (Intermediate) Implement deploy-idle and switch-traffic in a local cluster with two simple web apps. Verify with curl that the switch is immediate, then roll back by running the switch job again.
3. Interview question: "You need to rename a database column in a service deployed blue-green. Describe the sequence of releases." Hint: both versions must work against the schema at every moment, including after a rollback.

Feature Flags

Why it matters. Deploying and releasing are different decisions. A deployment puts code on servers; a release lets users see a feature. When they are the same event, every incomplete feature blocks deployment, and every rollback of a feature means a rollback of everything deployed with it. Feature flags separate the two: code ships to production switched off, and the feature is turned on for chosen users when ready, and off again in seconds if it misbehaves.

What it is. GitLab has a built-in feature flag service on all tiers, compatible with the open-source Unleash client libraries. You define flags in Deploy > Feature flags. Your application asks the flag service (through an Unleash SDK) whether a flag is on for the current user. Each flag has one or more strategies, and each strategy applies to one or more environments:

Strategy	Turns the flag on for	Example
All users	Everyone	Feature complete, flag kept as a kill switch
Percent of users	A percentage of logged-in users, consistently by user ID	20% of users in production
Percent rollout	A percentage, consistently by user ID, session ID or randomly	Anonymous traffic
User IDs	A list of user IDs	Internal testers
User list	A reusable named list of user IDs	"beta-customers" shared by several flags

Flag "new_checkout"
  strategy 1: User list "qa-team"       environments: staging, production
  strategy 2: Percent of users 10%       environments: production
  strategy 3: All users                  environments: review/*

Connecting an application. Click Configure on the feature flags page to see the API URL and instance ID. The Unleash appName is the environment name, which is how GitLab knows which strategies apply:

// Node.js with the Unleash client (npm install unleash-client)
const { initialize } = require('unleash-client');

const unleash = initialize({
  url: 'https://gitlab.example.com/api/v4/feature_flags/unleash/12345', // project feature flag API
  appName: process.env.CI_ENVIRONMENT_NAME || 'production',             // must match a strategy's environment
  instanceId: process.env.GITLAB_FF_INSTANCE_ID,                        // from "Configure", not secret but project-specific
  refreshInterval: 15000,                                               // poll for changes every 15 s
});

app.get('/checkout', (req, res) => {
  const ctx = { userId: String(req.user.id) };        // used by percentage and user-ID strategies
  if (unleash.isEnabled('new_checkout', ctx)) {
    return renderNewCheckout(req, res);
  }
  return renderOldCheckout(req, res);                 // keep the old path until the flag is removed
});

Pass the environment name and instance ID to the application at deploy time, for example as Helm values set from CI_ENVIRONMENT_NAME and a CI/CD variable. GitLab's flag service follows an older Unleash API, so check the documentation for supported client versions before upgrading an SDK.

Managing flags from a pipeline. The Feature Flags API lets a pipeline create flags or change rollout percentages, for example widening a rollout after a successful deployment:

# Create "new_checkout": 10% of users in production, everyone in review apps
curl --request POST \
  --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  --header "Content-Type: application/json" \
  --data '{
        "name": "new_checkout",
        "version": "new_version_flag",
        "description": "Redesigned checkout, issue #812",
        "strategies": [
          { "name": "gradualRolloutUserId",
            "parameters": { "groupId": "default", "percentage": "10" },
            "scopes": [ { "environment_scope": "production" } ] },
          { "name": "default", "parameters": {},
            "scopes": [ { "environment_scope": "review/*" } ] }
        ]
      }' \
  "https://gitlab.example.com/api/v4/projects/12345/feature_flags"

Flag hygiene. Every flag is a branch in your code. Flags that are never removed turn into a maze of combinations nobody has tested.

• Link each flag to an issue, and create a cleanup issue when the flag reaches 100%.
• Distinguish release flags (temporary, remove after rollout) from operational flags (kill switches, kept on purpose).
• Test both paths while the flag exists.
• Search for flag names in the code before deleting a flag in GitLab. A missing flag evaluates as off.

Key takeaways

• Feature flags separate deploying code from releasing a feature. Turning a feature off takes seconds and needs no deployment.
• GitLab's flags (all tiers) use Unleash clients. The client's appName is the environment name, which selects the applicable strategies.
• Strategies cover all users, percentages (sticky by user, session or random), user IDs and reusable user lists, each scoped to environments.
• Flags are technical debt. Link them to issues and remove release flags after rollout.

Practice

1. (Beginner) Create a flag with a "User IDs" strategy for production. Write a ten-line script using an Unleash SDK that prints whether the flag is on for user 1 and user 2.
2. (Intermediate) Add a pipeline job that runs after a successful production deployment and raises a flag's percentage from 10 to 50 through the API, but only if a health check passes.
3. Interview question: "If you have feature flags, do you still need canaries and rollbacks?" Hint: which failures a flag can switch off (feature logic) and which it cannot (dependency upgrades, configuration, memory leaks in shared code).

Rollbacks and Re-deployments

Why it matters. When a deployment breaks production, the fastest safe action is usually to put back what worked. Rolling forward with a fix is often right too, but only when the fix is small and certain. GitLab keeps every deployment of an environment, so going back is a button, provided your pipeline was built to support it.

What it is. Each deployment record points to the job that produced it. A rollback in GitLab re-runs the deployment job of an earlier, successful deployment. The job uses that pipeline's commit, variables and artifacts, so it deploys the older version again.

Environment "production"
  #57  c0ffee  success   ← current, broken
  #56  a1b2c3  success   [Rollback]  ──► re-runs job 9921 from pipeline #801 ──► #58 a1b2c3
  #55  9f8e7d  success   [Rollback]

From the environment page, choose Rollback environment (or the re-deploy icon) on the deployment you want. The new run appears as a new deployment, so the history stays complete. Re-deploying the current deployment uses the same mechanism, which is useful when configuration drifted or a pod was deleted by hand.

What makes rollbacks work.

Requirement	Why	How
Deploy jobs are reproducible	The re-run must deploy the same thing	Deploy immutable artifacts: an image tagged by commit SHA or digest, never latest
Artifacts still exist	A re-run that needs a build artifact fails if it expired	Deploy from the registry, or keep deploy artifacts long enough (section 8.1)
The older job can still run	Outdated job protection may block it	Enable "Allow job retries for rollback deployments" (next topic)
Schema and data compatibility	Old code may not understand the new schema	Expand and contract migrations (blue-green topic)
Same credentials still valid	Secrets may have rotated	Read secrets at run time from a secret store or current variables, not from old artifacts

deploy-production:
  stage: deploy
  image: alpine/k8s:1.31.2
  script:
    - kubectl config use-context acme/infra:prod-agent
    # Immutable reference: the commit SHA of *this* pipeline, so re-running an old job deploys the old image
    - helm upgrade --install shop ./chart -n shop --set image.tag="$CI_COMMIT_SHA" --wait --atomic
  environment:
    name: production
    url: https://shop.example.com
  resource_group: production
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

Rollback through the API. The same action from a script or chat bot: find the job of the deployment, then retry it.

API=https://gitlab.example.com/api/v4/projects/12345
# Job ID of the second most recent successful production deployment (the one before current)
JOB_ID=$(curl --silent --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "$API/deployments?environment=production&status=success&order_by=finished_at&sort=desc&per_page=2" \
  | jq '.[1].deployable.id')
# Retry it: GitLab records a new deployment of the older commit
curl --request POST --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "$API/jobs/$JOB_ID/retry"

Rollback alternatives.

Approach	When
Re-run an older deploy job	Default: fast, auditable, uses the known-good pipeline
helm rollback / kubectl rollout undo	Emergency when GitLab itself is unavailable; then reconcile so GitLab's record matches reality
Switch blue-green traffic back	Blue-green setups
Turn off a feature flag	The problem is in a flagged feature
Revert commit and deploy	Fix needs to stay in history; slower, goes through the full pipeline

Key takeaways

• GitLab rollback re-runs the deploy job of an earlier successful deployment, creating a new deployment record.
• Rollbacks only work if deploy jobs are reproducible: immutable image references, available artifacts, compatible schemas.
• The same mechanism re-deploys the current version to fix drift.
• Know your fallback when GitLab is down, and reconcile GitLab's records afterwards.

Practice

1. (Beginner) Deploy three versions of an app that prints its commit SHA, then roll back to the first one from the environment page. Check the new entry in the deployment history.
2. (Intermediate) Change your deploy job so it uses latest instead of the commit SHA, roll back, and explain the result. Then fix it and write the API-based rollback script above for your project.
3. Interview question: "A rollback in GitLab succeeded, but the application is still broken. List possible reasons." Hint: mutable tags, expired artifacts, database migrations that ran forward, configuration and secrets outside the job, caches.

Deployment Safety Settings

Why it matters. Pipelines are concurrent and people make mistakes. Two pipelines finish in the wrong order and an older commit overwrites a newer one in production. Someone retries a deploy job from last month. A release goes out during the holiday freeze. GitLab has settings that prevent each of these without relying on everyone remembering the rules.

What it is. The main safety controls, in one place:

Risk	Control	Where	Tier
Two deployments to the same environment at once	resource_group (section 4.5)	Job keyword	All
An older pipeline deploying after a newer one	Prevent outdated deployment jobs	Settings > CI/CD > General pipelines	All
That protection blocking intentional rollbacks	Allow job retries for rollback deployments	Same section	All
Deploying during a freeze period	Deploy freezes + CI_DEPLOY_FREEZE	Settings > CI/CD > Deploy freezes	All
Unauthorized deployers	Protected environments	Settings > CI/CD > Protected environments	Premium
Deployment without sign-off	Deployment approvals	Same	Premium
Old pipelines wasting runners on obsolete deploys	interruptible and auto-cancel (section 4.3)	Job keyword and settings	All

Prevent outdated deployment jobs. Pipelines can finish out of order: pipeline #102 (newer commit) deploys first, then the slower #101 deploys and overwrites it. With this setting enabled, GitLab fails a deploy job whose pipeline is older than the environment's latest successful deployment, and the job's failure reason says so.

         #101 (older commit)  ─────────────────── deploy? ──► blocked: outdated
         #102 (newer commit)  ───────── deploy ──► production = #102

Rollbacks are deliberately old jobs, so they need the companion setting Allow job retries for rollback deployments, which lets someone explicitly retry an older successful deployment job.

Deploy freezes. A deploy freeze declares windows when deployments should not happen, as cron expressions with a time zone. During a freeze, GitLab sets CI_DEPLOY_FREEZE in every pipeline; your jobs decide what to do with it:

deploy-production:
  stage: deploy
  script: ./deploy.sh production
  environment: production
  rules:
    - if: $CI_DEPLOY_FREEZE                       # inside a freeze window
      when: manual                                # still possible, but only on purpose
      allow_failure: true
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH # normal days: deploy automatically

Or block deployments completely during freezes:

workflow:
  rules:
    - if: $CI_DEPLOY_FREEZE && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: never                                 # no main pipelines at all during the freeze
    - when: always

# Freeze from Friday 18:00 to Monday 08:00, Europe/Rome
curl --request POST --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  --header "Content-Type: application/json" \
  --data '{"freeze_start": "0 18 * * 5", "freeze_end": "0 8 * * 1", "cron_timezone": "Europe/Rome"}' \
  "https://gitlab.example.com/api/v4/projects/12345/freeze_periods"

Health checks as part of the job. A deploy job that exits 0 while the application is crashing gives false confidence. Make the job fail when the release is unhealthy:

• helm upgrade --atomic --wait rolls back automatically if pods do not become ready.
• kubectl rollout status deployment/shop --timeout=5m waits for the rollout.
• A smoke test against CI_ENVIRONMENT_URL as a final step or a verify job.

Key takeaways

• Use resource_group for one deployment at a time, and Prevent outdated deployment jobs so older pipelines cannot overwrite newer deployments.
• Enable Allow job retries for rollback deployments so deliberate rollbacks still work.
• Deploy freezes set CI_DEPLOY_FREEZE. Your rules or workflow decide whether to make deploys manual or skip them.
• Make deploy jobs fail on unhealthy releases, so the deployment record tells the truth.

Practice

1. (Beginner) Create a deploy freeze that covers the current hour. Run a pipeline and confirm that CI_DEPLOY_FREEZE is set and your deploy job became manual.
2. (Intermediate) Enable Prevent outdated deployment jobs. Start two pipelines where the older one has a slower build (add sleep), and observe the older deploy job being blocked. Then roll back to the older deployment with the rollback setting enabled.
3. Interview question: "Two engineers merged within a minute of each other, and production ended up running the older commit. Explain how that happened and which settings prevent it." Hint: pipeline ordering, resource group process modes, and outdated deployment protection.

<a id="124-deployment-targets"></a>

12.4 Deployment Targets

The environment model is the same everywhere, but every target has its own way to receive code and its own way to authenticate. This subchapter covers GitLab Pages for static sites, Kubernetes through the GitLab agent, the three major clouds with short-lived OIDC credentials, and plain servers over SSH.

The common principle is the same for every target: the pipeline should hold no long-lived credentials. Wherever possible, the job proves its identity with an ID token (section 5.4), or an agent inside the target pulls instructions from GitLab, so a leaked CI variable cannot be used to reach production.

Target	Authentication	Connection direction
GitLab Pages	None needed: GitLab hosts it	Artifacts uploaded to GitLab
Kubernetes	GitLab agent (agentk)	Agent connects out to GitLab, jobs tunnel through it
AWS	OIDC to IAM role (AssumeRoleWithWebIdentity)	Job calls AWS APIs
Google Cloud	OIDC to Workload Identity Federation	Job calls Google APIs
Azure	OIDC to Entra ID federated credential	Job calls Azure APIs
Servers	SSH key in a file variable	Job connects to the server

GitLab Pages

Why it matters. Documentation, a project website, a test coverage report or a single-page application often needs nothing more than static hosting. Setting up a bucket, a CDN, certificates and DNS for each one is overhead. GitLab Pages hosts static files straight from a pipeline, with HTTPS, custom domains and optional access control.

What it is. A Pages job builds static files into a folder. GitLab takes the folder from the job's artifacts and publishes it. The site appears at a URL like https://<namespace>.gitlab.io/<project> on GitLab.com (or a unique domain such as https://<project>-<hash>.gitlab.io), or at the Pages domain your administrator configured on a self-managed instance. Pages is available on all tiers.

create-pages:
  stage: deploy
  image: node:22-alpine
  script:
    - npm ci
    - npm run build                       # writes the site to dist/
  pages:
    publish: dist                         # folder to publish (default: public)
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

In GitLab 17.9 and later, any job with the pages keyword deploys Pages, and pages:publish names the folder. On older versions, the job must be named pages, the files must be in public/, and that folder must be listed under artifacts:paths:

pages:                                    # job name is what makes it a Pages job (older versions)
  script:
    - npm ci && npm run build
    - mv dist public
  artifacts:
    paths: [public]

After the job succeeds, GitLab runs an internal pages:deploy job. The site's URL is shown under Deploy > Pages, and CI_PAGES_URL contains it in jobs.

Parallel deployments (Premium). One project can serve several versions of its site side by side, each under a path prefix. That makes Pages a simple review app host for documentation and front ends:

pages-preview:
  stage: deploy
  script: npm ci && npm run build
  pages:
    publish: dist
    path_prefix: "mr-$CI_MERGE_REQUEST_IID"    # served at <site URL>/mr-42/
    expire_in: 1 week                          # removed automatically afterwards
  environment:
    name: pages/mr-$CI_MERGE_REQUEST_IID
    url: $CI_PAGES_URL                         # includes the path prefix
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

Single-page applications using client-side routing need the build tool to set the base path to the prefix, or links will point at the root.

Settings worth knowing.

Setting	Where	Purpose
Custom domains	Deploy > Pages > New domain	Serve from docs.example.com. Verified with a DNS TXT record
HTTPS certificates	Same page	Automatic Let's Encrypt certificates, or upload your own
Access control	Settings > General > Visibility	Restrict the site to project members
Redirects	_redirects file in the published folder	Path-based redirects and rewrites
Unique domain	Deploy > Pages	Use a per-project subdomain instead of a path on the namespace domain

Limits. Pages serves static files only: no server-side code, no environment variables at request time. Anything dynamic must run in the browser or call an API hosted elsewhere.

Key takeaways

• GitLab Pages publishes a folder of static files from a pipeline, with HTTPS and custom domains, on all tiers.
• Current versions use the pages keyword with pages:publish. Older versions need a job named pages and a public artifact.
• Parallel deployments (Premium) with path_prefix and expire_in give each merge request its own preview.
• Pages cannot run server-side code.

Practice

1. (Beginner) Publish a static site generator's output (MkDocs, Hugo, Vite) with Pages, and open the URL from Deploy > Pages.
2. (Intermediate) Add merge request previews with path_prefix and expire_in, tied to an environment so the merge request shows a View app link. Fix asset paths so the preview works under the prefix.
3. Interview question: "When would you choose GitLab Pages over S3 plus CloudFront for a front end, and when not?" Hint: operational effort, access control, edge caching and regions, server-side needs, and where the API lives.

Deploying to Kubernetes with the GitLab Agent

Why it matters. The old way to deploy to Kubernetes from CI was to store a cluster-admin kubeconfig as a CI variable and open the cluster API to the internet or to the runners. Anyone who could read the variable, or push a job that printed it, owned the cluster. The GitLab agent for Kubernetes reverses the direction: a small component inside the cluster connects out to GitLab, and jobs reach the cluster through that tunnel with permissions you define per project.

What it is.

   GitLab                                              Your cluster (private network)
  ┌──────────────────────────────┐                    ┌──────────────────────────────┐
  │ CI job in acme/shop          │                    │                              │
  │  kubectl --context           │                    │  agentk                      │
  │   acme/infra:prod-agent      │                    │    │  1. opens an outbound   │
  │        │                     │                    │    │     connection to KAS   │
  │        ▼                     │   2. tunnel        │    │                         │
  │  KAS (agent server)  ◄───────┼────────────────────┼────┘                         │
  │        │                     │                    │                              │
  │        └── 3. job's requests ┼──── through ──────►│  agentk ──► Kubernetes API   │
  │                              │      tunnel        │                              │
  │ Agent config (acme/infra):   │                    │                              │
  │  .gitlab/agents/prod-agent/  │                    │                              │
  │    config.yaml               │                    │                              │
  └──────────────────────────────┘                    └──────────────────────────────┘

• agentk runs in the cluster and keeps a connection open to the GitLab agent server (KAS). No inbound firewall rule is needed.
• The agent is registered in one agent configuration project (for example acme/infra), with an optional config file at .gitlab/agents/<agent-name>/config.yaml.
• The config file's ci_access section lists which projects and groups may use the agent from CI jobs.
• In an authorized project's jobs, GitLab injects a kubeconfig with one context per agent, named <agent-project-path>:<agent-name>.

Step 1: register and install the agent. In the agent configuration project, go to Operate > Kubernetes clusters > Connect a cluster, enter a name and copy the token and command. The install uses the official Helm chart:

helm repo add gitlab https://charts.gitlab.io
helm repo update
# AGENT_TOKEN is the glagent-... token shown once in the UI.
# kasAddress is wss://kas.gitlab.com on GitLab.com, or wss://gitlab.example.com/-/kubernetes-agent/ on self-managed.
helm upgrade --install prod-agent gitlab/gitlab-agent \
  --namespace gitlab-agent-prod-agent --create-namespace \
  --set config.token="$AGENT_TOKEN" \
  --set config.kasAddress=wss://kas.gitlab.com

Step 2: authorize projects. In the agent configuration project:

# .gitlab/agents/prod-agent/config.yaml
ci_access:
  projects:
    - id: acme/shop                          # this project's jobs get the prod-agent context
  groups:
    - id: acme/platform-services             # every project in this group too

On Premium, ci_access can also make the agent impersonate the CI job, so Kubernetes RBAC can tell projects, branches and environments apart instead of granting every job the agent's own service account permissions:

ci_access:
  projects:
    - id: acme/shop
      access_as:
        ci_job: {}                           # Kubernetes sees user "gitlab:ci_job:<job id>" with groups
                                             # such as gitlab:project:<id> and gitlab:project_env:<id>:<env>

Step 3: use it in a job.

deploy-production:
  stage: deploy
  image: alpine/k8s:1.31.2
  script:
    - kubectl config get-contexts                           # lists the agents this project may use
    - kubectl config use-context acme/infra:prod-agent
    - kubectl apply -n shop -f k8s/                          # or helm upgrade --install ...
    - kubectl rollout status -n shop deployment/shop --timeout=5m
  environment:
    name: production
    url: https://shop.example.com
    kubernetes:
      agent: acme/infra:prod-agent                         # link the environment to the agent
      dashboard:
        namespace: shop                                     # show this namespace on the environment page
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

Linking the environment to the agent turns on the Kubernetes overview on the environment page: pods, services, deployments and their status, and Flux sync state for GitOps setups (section 12.5). The exact sub-keys under environment:kubernetes have changed across GitLab versions (older versions put namespace directly under kubernetes), so check the keyword reference for your version.

Security model.

Concern	Recommendation
Blast radius	One agent per cluster or per trust zone. Do not share a production agent with every project
Agent permissions	Without impersonation, jobs act as the agent's service account. Give it only the namespaces and verbs needed, not cluster-admin
Who can deploy	Combine with protected environments and branches. With impersonation, write RBAC bindings for gitlab:project_env:<id>:production
Configuration changes	The agent config file controls access. Protect the agent configuration project and require review for changes

CI push versus GitOps pull. The pattern above is push-based: the pipeline runs kubectl or helm against the cluster. The agent also supports pull-based GitOps with Flux, where the cluster pulls desired state from Git (section 12.5). Many teams use push for application deployments and pull for cluster-wide configuration, or move entirely to pull.

Key takeaways

• The GitLab agent runs inside the cluster and connects out to GitLab. Jobs reach the cluster through it without stored kubeconfigs or an exposed API server.
• ci_access in .gitlab/agents/<name>/config.yaml authorizes projects and groups. Jobs select <agent-project>:<agent-name> as the kubectl context.
• Impersonation (Premium) lets Kubernetes RBAC distinguish projects and environments. Otherwise the job has the agent's permissions.
• Link environments to the agent with environment:kubernetes to get the Kubernetes overview on the environment page.

Practice

1. (Beginner) Install an agent in a local cluster (kind, k3d or minikube), authorize a second project with ci_access, and run a job that executes kubectl get pods -A through the agent context.
2. (Intermediate) Restrict the agent's service account to one namespace with a Role and RoleBinding. Show that a deploy to that namespace works and a deploy to another namespace fails. Then link the environment to the agent and inspect the Kubernetes overview.
3. Interview question: "Compare deploying through the GitLab agent with storing a kubeconfig as a masked CI variable." Hint: direction of the connection, credential lifetime, what a leaked value allows, and who controls which projects may deploy.

Deploying to AWS

Why it matters. AWS is the most common deployment target, and the most common mistake is a long-lived IAM user access key stored in a CI variable, with far more permissions than the deployment needs. Keys like that leak through logs, forks and job artifacts, and they never expire on their own. Deploying with OIDC and narrowly scoped roles removes the long-lived key entirely.

What it is. The job requests an ID token, and AWS STS exchanges it for temporary credentials of an IAM role whose trust policy accepts tokens only from your project and branch. Section 5.4 covers the trust setup in detail. The AWS CLI and SDKs pick up the role automatically when two environment variables are set:

.aws:
  image:
    name: amazon/aws-cli:2.17.0
    entrypoint: [""]                                    # the image's ENTRYPOINT is "aws" (section 7.1)
  id_tokens:
    AWS_ID_TOKEN:
      aud: sts.amazonaws.com                            # audience the IAM OIDC provider expects
  variables:
    AWS_REGION: eu-central-1
    AWS_WEB_IDENTITY_TOKEN_FILE: /tmp/web-identity-token
  before_script:
    - printf '%s' "$AWS_ID_TOKEN" > "$AWS_WEB_IDENTITY_TOKEN_FILE"
    - export AWS_ROLE_SESSION_NAME="gitlab-$CI_PROJECT_ID-$CI_JOB_ID"   # visible in CloudTrail
    - aws sts get-caller-identity                       # fail early if the role cannot be assumed

AWS_ROLE_ARN is set per environment as an environment-scoped variable, so staging jobs assume the staging role and production jobs the production role.

Example 1: static site to S3 and CloudFront.

deploy-site:
  extends: .aws
  stage: deploy
  needs: [build-site]                                   # artifacts: dist/
  script:
    - aws s3 sync dist/ "s3://$SITE_BUCKET/" --delete --cache-control "public,max-age=300"
    - aws cloudfront create-invalidation --distribution-id "$CF_DISTRIBUTION_ID" --paths "/*"
  environment:
    name: production
    url: https://www.example.com
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

Example 2: a new image on Amazon ECS. ECS services run a task definition revision. Deploying means registering a new revision with the new image and pointing the service at it:

deploy-ecs:
  extends: .aws
  stage: deploy
  script:
    - yum install -y jq > /dev/null                     # the aws-cli image is Amazon Linux based
    - IMAGE="$ECR_REPOSITORY:$CI_COMMIT_SHA"
    # Copy the current task definition, swap the image, drop read-only fields
    - >
      aws ecs describe-task-definition --task-definition shop --query taskDefinition
      | jq --arg IMG "$IMAGE" '.containerDefinitions[0].image = $IMG
      | del(.taskDefinitionArn, .revision, .status, .requiresAttributes, .compatibilities, .registeredAt, .registeredBy)'
      > taskdef.json
    - NEW_ARN=$(aws ecs register-task-definition --cli-input-json file://taskdef.json --query taskDefinition.taskDefinitionArn --output text)
    - aws ecs update-service --cluster prod --service shop --task-definition "$NEW_ARN"
    - aws ecs wait services-stable --cluster prod --services shop    # fail if the rollout does not settle
  environment:
    name: production
  resource_group: production
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

The image itself is pushed to Amazon ECR in the build job after aws ecr get-login-password | docker login --username AWS --password-stdin <account>.dkr.ecr.<region>.amazonaws.com, using the same OIDC role setup.

Other AWS options.

Target	Typical deploy command
Lambda	aws lambda update-function-code --function-name f --image-uri ..., or SAM/CDK deploy
EKS	The GitLab agent (previous topic), which avoids managing aws-auth mappings for CI
Elastic Beanstalk	aws elasticbeanstalk create-application-version then update-environment
CloudFormation / CDK	aws cloudformation deploy or cdk deploy in a job with plan review (section 12.5)
GitLab templates	AWS/Deploy-ECS.gitlab-ci.yml and the Auto DevOps ECS/EC2 targets (section 12.6)

Least privilege. The production deploy role should be able to do exactly the deployment: ecs:RegisterTaskDefinition, ecs:UpdateService, ecs:Describe* and iam:PassRole for the task role, not AdministratorAccess. Restrict its trust policy to the protected branch, for example "gitlab.example.com:sub": "project_path:acme/shop:ref_type:branch:ref:main".

Key takeaways

• Use id_tokens and an IAM role with a narrow trust policy. Set AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE and the CLI assumes the role itself.
• Scope AWS_ROLE_ARN per environment so each environment has its own role.
• Make deploys verify themselves: aws ecs wait services-stable, health checks, CloudFront invalidations.
• Give deploy roles only the permissions the deployment needs, and restrict trust to protected refs.

Practice

1. (Beginner) Configure an OIDC role and a job that runs aws sts get-caller-identity and aws s3 ls on one bucket. Confirm that a job on a feature branch cannot assume the production role.
2. (Intermediate) Implement the ECS deployment for a sample service, including services-stable. Break the image tag on purpose and observe how long it takes the job to fail, then add a timeout.
3. Interview question: "Your team uses one IAM user with AdministratorAccess for all pipelines. Propose a migration plan." Hint: OIDC provider, one role per environment and purpose, trust conditions on sub, CloudTrail session names, and removing the old key last.

Deploying to Google Cloud

Why it matters. As on AWS, a service account key file stored in a CI variable is a permanent credential that works from anywhere. Google Cloud's Workload Identity Federation (WIF) lets jobs exchange their GitLab ID token for short-lived Google credentials, so no key file exists at all.

What it is. A workload identity pool and provider trust your GitLab instance as an OIDC issuer and map token claims to attributes. IAM bindings then grant roles to principals defined by those attributes, either directly or by allowing them to impersonate a service account. Section 5.4 shows how to create the pool and provider. In the job, gcloud uses a credential configuration file that points at the token:

.gcp:
  image: google/cloud-sdk:497.0.0-slim
  id_tokens:
    GCP_ID_TOKEN:
      aud: https://iam.googleapis.com/projects/${GCP_PROJECT_NUMBER}/locations/global/workloadIdentityPools/${GCP_POOL_ID}/providers/${GCP_PROVIDER_ID}
  before_script:
    - printf '%s' "$GCP_ID_TOKEN" > .ci_id_token
    # Build an external-account credential file: "read the token from this file, exchange it, impersonate this SA"
    - >
      gcloud iam workload-identity-pools create-cred-config
      "projects/$GCP_PROJECT_NUMBER/locations/global/workloadIdentityPools/$GCP_POOL_ID/providers/$GCP_PROVIDER_ID"
      --service-account="$GCP_SERVICE_ACCOUNT"
      --credential-source-file=.ci_id_token
      --output-file=.gcp_cred.json
    - gcloud auth login --cred-file=.gcp_cred.json
    - gcloud config set project "$GCP_PROJECT_ID"

GCP_SERVICE_ACCOUNT is environment-scoped, so production jobs impersonate the production deployer account. Granting roles directly to the federated principal (without --service-account) also works for many services and removes the service account step.

Example: deploy a container to Cloud Run.

deploy-cloud-run:
  extends: .gcp
  stage: deploy
  variables:
    REGION: europe-west1
    IMAGE: europe-west1-docker.pkg.dev/acme-prod/apps/shop:$CI_COMMIT_SHA   # Artifact Registry
  script:
    - >
      gcloud run deploy shop
      --image "$IMAGE"
      --region "$REGION"
      --no-allow-unauthenticated
      --quiet
    - URL=$(gcloud run services describe shop --region "$REGION" --format='value(status.url)')
    - echo "DEPLOY_URL=$URL" >> deploy.env
  artifacts:
    reports:
      dotenv: deploy.env
  environment:
    name: production
    url: $DEPLOY_URL                                     # from the dotenv report
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

Cloud Run keeps every revision, so a gradual rollout is a traffic split: deploy with --no-traffic --tag canary, then gcloud run services update-traffic shop --to-tags canary=10, and finally --to-latest.

GitLab and Google Cloud integration. GitLab.com offers a Google Cloud IAM integration that creates the pool and provider for you, and Google publishes CI/CD components for Artifact Registry, Cloud Run and Cloud Deploy in the CI/CD Catalog. They save setup effort, but they rely on the same WIF mechanism shown above, so understanding the manual setup helps when something fails.

Other targets. GKE clusters are best reached through the GitLab agent. App Engine uses gcloud app deploy, Cloud Functions gcloud functions deploy.

Key takeaways

• Workload Identity Federation exchanges the job's ID token for short-lived Google credentials. No service account key is stored.
• create-cred-config builds a credential file from the token, and gcloud auth login --cred-file uses it.
• Scope service accounts or principals per environment, and restrict the provider's attribute conditions to your project and protected refs.
• Cloud Run revisions and tagged traffic splits give canaries and instant rollbacks without extra infrastructure.

Practice

1. (Beginner) Configure WIF for a test project and run a job that prints gcloud auth list and gcloud projects describe. Inspect the token's aud and sub claims.
2. (Intermediate) Deploy a container to Cloud Run with --no-traffic --tag canary, send 10% of traffic to it in a manual job, and promote it in a second manual job.
3. Interview question: "What attribute condition would you put on the workload identity provider, and why is mapping sub alone not enough?" Hint: forks and other projects on the same GitLab instance, project_path, ref_protected, and namespace_id.

Deploying to Azure

Why it matters. On Azure, the old approach is a service principal client secret in a CI variable, which expires at awkward moments and works from anywhere until it does. Federated identity credentials in Microsoft Entra ID let jobs log in with their GitLab ID token instead, with no secret to rotate.

What it is. An app registration (or a user-assigned managed identity) gets a federated credential that trusts tokens from your GitLab issuer with an exact subject. The job passes its ID token to az login, and Azure RBAC on the identity decides what it can change. Section 5.4 covers Key Vault access with the same identity.

Azure side. Create one federated credential per branch or tag pattern you deploy from:

# Trust tokens from main of acme/shop on GitLab.com
az ad app federated-credential create --id "$APP_OBJECT_ID" --parameters '{
  "name": "gitlab-acme-shop-main",
  "issuer": "https://gitlab.com",
  "subject": "project_path:acme/shop:ref_type:branch:ref:main",
  "audiences": ["api://AzureADTokenExchange"]
}'

# Allow the identity to update one resource group only
az role assignment create --assignee "$APP_CLIENT_ID" --role Contributor \
  --scope "/subscriptions/$SUB_ID/resourceGroups/rg-shop-prod"

The subject must match the token's sub claim exactly (classic federated credentials do not accept wildcards). Tokens from other branches, from tags, or from merge request pipelines have a different sub, so each needs its own credential, or a flexible credential with claim-matching expressions where available.

GitLab side.

.azure:
  image: mcr.microsoft.com/azure-cli:2.64.0
  id_tokens:
    AZURE_ID_TOKEN:
      aud: api://AzureADTokenExchange                    # must match the federated credential's audience
  before_script:
    - >
      az login --service-principal
      --username "$AZURE_CLIENT_ID"
      --tenant "$AZURE_TENANT_ID"
      --federated-token "$AZURE_ID_TOKEN"
      --output none
    - az account set --subscription "$AZURE_SUBSCRIPTION_ID"

deploy-container-app:
  extends: .azure
  stage: deploy
  script:
    - >
      az containerapp update
      --name shop --resource-group rg-shop-prod
      --image "acmeregistry.azurecr.io/shop:$CI_COMMIT_SHA"
  environment:
    name: production
    url: https://shop.example.com
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

Common Azure targets.

Target	Deploy command	Safer rollout
App Service	az webapp deploy --src-path app.zip --type zip	Deploy to a staging slot, test, then az webapp deployment slot swap (blue-green)
Container Apps	az containerapp update --image ...	Multiple revision mode with traffic weights
AKS	The GitLab agent	Kubernetes strategies from section 12.3
Functions	az functionapp deployment source config-zip	Slots, as for App Service
Bicep / ARM	az deployment group create --template-file main.bicep	what-if in merge requests (section 12.5)

Key takeaways

• Use federated identity credentials: az login --federated-token "$AZURE_ID_TOKEN" with an ID token whose audience is api://AzureADTokenExchange.
• The credential's subject must match the token's sub exactly, so plan one credential per deploying ref (or use flexible credentials).
• Scope role assignments to resource groups per environment, and set client and subscription IDs as environment-scoped variables.
• Deployment slots and Container Apps revisions provide blue-green and canary patterns natively.

Practice

1. (Beginner) Create an app registration with a federated credential for main, and a job that runs az login with the ID token and lists the resource groups it can see.
2. (Intermediate) Deploy a web app to a staging slot, run a smoke test against the slot URL, and swap slots in a manual job. Roll back by swapping again.
3. Interview question: "Your Azure login works on main but fails in tag pipelines with AADSTS70021: No matching federated identity record found. Why?" Hint: compare the sub claim of both pipelines with the credential's subject.

SSH-Based Deployments

Why it matters. Not everything runs on Kubernetes or a managed cloud service. Virtual machines, on-premises servers and small single-host setups are deployed over SSH, and the details (host key checking, key permissions, atomic switch-over) decide whether the deployment is secure and safe to repeat.

What it is. The job loads a private key into ssh-agent, trusts the server's host key from a variable, copies the build output and runs commands remotely. Store the key as a file-type, protected, environment-scoped variable, and the known hosts line as another file variable.

# One-time setup on your machine
ssh-keygen -t ed25519 -C "gitlab-deploy acme/shop" -f deploy_key -N ""   # key without passphrase for CI
ssh-keyscan -t ed25519 app1.example.com > known_hosts                     # verify the fingerprint out of band!
# deploy_key  -> CI/CD variable SSH_PRIVATE_KEY (type File, protected, scope production)
# known_hosts -> CI/CD variable SSH_KNOWN_HOSTS (type File)
# deploy_key.pub -> ~deploy/.ssh/authorized_keys on the server

Atomic releases. Copying files over a running application leaves it half old, half new for a while. Upload each release to its own directory and switch a symlink, which is a single atomic rename:

/srv/shop/
  releases/
    20261015-0912-a1b2c3/
    20261015-1430-c0ffee/      ← new release uploaded here first
  current -> releases/20261015-1430-c0ffee     (ln -sfn switches atomically)
  shared/  (uploads, .env: survive releases)

deploy-vm:
  stage: deploy
  image: alpine:3.20
  variables:
    DEPLOY_HOST: app1.example.com
    DEPLOY_USER: deploy
  before_script:
    - apk add --no-cache openssh-client rsync
    - eval "$(ssh-agent -s)"
    - chmod 400 "$SSH_PRIVATE_KEY"                     # file variable: the variable holds the file's path
    - ssh-add "$SSH_PRIVATE_KEY"
    - mkdir -p ~/.ssh && chmod 700 ~/.ssh
    - cp "$SSH_KNOWN_HOSTS" ~/.ssh/known_hosts         # strict host key checking stays on
  script:
    - RELEASE="/srv/shop/releases/$(date +%Y%m%d-%H%M)-$CI_COMMIT_SHORT_SHA"
    - rsync -az --delete dist/ "$DEPLOY_USER@$DEPLOY_HOST:$RELEASE/"
    # Switch the symlink, restart, then keep only the last 5 releases
    - >
      ssh "$DEPLOY_USER@$DEPLOY_HOST"
      "ln -sfn $RELEASE /srv/shop/current
      && sudo systemctl restart shop
      && ls -1dt /srv/shop/releases/* | tail -n +6 | xargs -r rm -rf"
    - curl --fail --retry 5 --retry-delay 3 "https://$DEPLOY_HOST/healthz"   # verify after restart
  environment:
    name: production
    url: https://shop.example.com
  resource_group: production
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

Rolling back means pointing current at the previous release directory, or re-running an earlier deploy job (section 12.3).

Hardening.

Measure	How
Never disable host key checking	No StrictHostKeyChecking=no. It lets anyone in the middle receive your deployment and key usage
Dedicated deploy user	No shell login for people, no general sudo. Allow only systemctl restart shop in sudoers
Restrict the key on the server	authorized_keys options such as restrict,from="203.0.113.0/24", or a forced command= that runs only the deploy script
One key per project and environment	Revoke one without affecting others
Network	Allow SSH only from runner IPs, or use a bastion host with ProxyJump
Alternative	A runner with the shell executor installed on the server pulls jobs, so no inbound SSH is needed

Several servers. Loop over hosts and remove each from the load balancer while it is updated, or use a configuration management tool such as Ansible from the job (ansible-playbook -i inventory/production deploy.yml), which handles parallelism, rolling batches (serial: 1) and idempotency.

Key takeaways

• Store the private key and known hosts as file variables. Keep the key protected and environment-scoped.
• Keep strict host key checking on. Pin the server's host key from a variable.
• Deploy into a new release directory and switch a symlink for an atomic cutover and easy rollback.
• Restrict the deploy user and key on the server. For many servers, use Ansible or a similar tool.

Practice

1. (Beginner) Deploy a static site to a VM over SSH with rsync, using file variables for the key and known hosts. Then change the server's host key and observe the job fail.
2. (Intermediate) Implement atomic releases with a current symlink, keep the last five releases, and add a manual rollback-vm job that points current at the previous release.
3. Interview question: "Why is StrictHostKeyChecking=no dangerous in a deployment job, even inside a private network?" Hint: what the job sends to the host, and what an attacker who controls DNS or the network path can do with it.

<a id="125-gitops-and-infrastructure-as-code"></a>

12.5 GitOps and Infrastructure as Code

Applications are not the only thing that gets deployed. Clusters, networks, databases and DNS records are infrastructure, and when they are changed by hand they drift, cannot be reviewed and cannot be rebuilt after a disaster. This subchapter covers managing infrastructure and cluster state as code in Git: GitOps with Flux and the GitLab agent, Terraform and OpenTofu pipelines, GitLab's built-in Terraform state backend, Helm chart deployments, and reviewing infrastructure plans in merge requests.

Push versus pull. Two models deliver changes from Git to a target:

 PUSH (CI-driven)                                  PULL (GitOps)
 ────────────────                                  ─────────────
 git push ──► pipeline ──► kubectl / tofu apply    git push ──► Git repository (desired state)
                │                                                      ▲
                └─ the pipeline holds credentials                      │ watches and reconciles
                   for the target                        controller in the target (Flux)
                                                         holds the credentials, corrects drift

	Push	Pull (GitOps)
Who applies changes	The pipeline	A controller inside the target
Credentials	Pipeline needs access to the target	Target needs read access to Git or a registry
Drift (manual changes)	Remains until the next deploy	Reverted automatically on the next reconcile
Fits	Cloud resources (Terraform/OpenTofu), any target	Kubernetes clusters

GitOps with Flux and the GitLab Agent

Why it matters. In a cluster changed by kubectl apply from many pipelines and laptops, nobody can say with certainty what should be running. GitOps makes a Git repository the single source of truth: the cluster continuously converges to what the repository says, every change is a reviewed commit, and rolling back is reverting a commit. Rebuilding a lost cluster means pointing a new one at the same repository.

What it is. Flux is a CNCF project that runs controllers inside the cluster. A source (GitRepository or OCIRepository) tells Flux where to fetch manifests. A Kustomization (or a HelmRelease) tells it what to apply and keeps applying it at an interval. GitLab recommends Flux for GitOps with the agent, and the agent adds integration on top:

• When a project that Flux watches receives a push, the agent triggers an immediate Flux reconciliation instead of waiting for the next poll.
• The environment's Kubernetes overview shows Flux's sync status when the environment is linked to the agent and a Flux resource.

Bootstrapping Flux from a GitLab repository.

export GITLAB_TOKEN=glpat-...          # used once by bootstrap to create a deploy key/token and commit manifests
flux bootstrap gitlab \
  --hostname=gitlab.example.com \
  --owner=acme/infra \
  --repository=fleet \
  --branch=main \
  --path=clusters/prod \
  --deploy-token-auth                  # Flux reads the repository with a GitLab deploy token

Bootstrap installs Flux, commits its own manifests to clusters/prod/flux-system/ and configures Flux to watch that path. From then on, the repository describes the cluster:

fleet/                                   (acme/infra/fleet)
├── clusters/
│   ├── prod/
│   │   ├── flux-system/                 created by bootstrap
│   │   ├── gitlab-agent.yaml            HelmRelease for the GitLab agent itself
│   │   └── apps.yaml                    Kustomization -> apps/overlays/prod
│   └── staging/ ...
└── apps/
    ├── base/shop/                       Deployment, Service, Ingress
    └── overlays/
        ├── staging/                     image tag, replicas for staging
        └── prod/                        image tag, replicas for prod

# clusters/prod/apps.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  interval: 10m                  # re-apply every 10 minutes, reverting manual drift
  sourceRef:
    kind: GitRepository
    name: flux-system            # the repository bootstrap configured
  path: ./apps/overlays/prod
  prune: true                    # delete objects that were removed from Git
  wait: true                     # report Ready only when resources are healthy

How application pipelines fit in. Application code lives in its own repository. Its pipeline builds the image and then changes the desired state instead of touching the cluster:

Approach	The app pipeline...	Notes
Commit to the fleet repo	Updates the image tag in apps/overlays/prod and opens or merges an MR	Simple, fully reviewable. Needs a token that can write to the fleet repo
Flux image automation	Only pushes the image. Flux's image controllers detect new tags and commit the update	No write token in CI. Tag policies decide what deploys
OCI artifacts	Renders manifests, pushes them as an OCI artifact to the container registry and moves a tag such as production	No Git writes. Flux uses an OCIRepository source

The OCI artifact flow, which keeps manifests versioned next to the image:

publish-manifests:
  stage: deploy
  image:
    name: fluxcd/flux-cli:v2.4.0
    entrypoint: [""]
  script:
    - >
      flux push artifact "oci://$CI_REGISTRY_IMAGE/manifests:$CI_COMMIT_SHORT_SHA"
      --path=./deploy/manifests
      --source="$CI_PROJECT_URL"
      --revision="$CI_COMMIT_REF_NAME@sha1:$CI_COMMIT_SHA"
      --creds "$CI_REGISTRY_USER:$CI_REGISTRY_PASSWORD"
    # Moving the "production" tag is the deployment: Flux follows that tag
    - >
      flux tag artifact "oci://$CI_REGISTRY_IMAGE/manifests:$CI_COMMIT_SHORT_SHA"
      --tag production
      --creds "$CI_REGISTRY_USER:$CI_REGISTRY_PASSWORD"
  environment:
    name: production
    kubernetes:
      agent: acme/infra:prod-agent
      dashboard:
        namespace: shop
        flux_resource_path: kustomize.toolkit.fluxcd.io/v1/namespaces/flux-system/kustomizations/shop
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

The pipeline's deploy job succeeds when the artifact is published, not when the cluster has converged. Flux applies the change shortly after, and the environment's Kubernetes overview shows the reconciliation status. Add a verify job that waits for the Kustomization to be ready (flux get kustomizations through the agent) if the pipeline must report the real outcome.

Key takeaways

• GitOps makes Git the source of truth. A controller in the cluster (Flux) pulls and reconciles, reverting drift and pruning removed objects.
• flux bootstrap gitlab installs Flux and commits its configuration to a GitLab repository.
• The GitLab agent triggers immediate reconciliation on push and shows Flux status on the environment page.
• Application pipelines change desired state (commit, image automation or OCI artifact) instead of calling kubectl.

Practice

1. (Beginner) Bootstrap Flux in a local cluster against a GitLab repository. Add a Deployment through a commit, then delete it with kubectl and watch Flux recreate it.
2. (Intermediate) Build the OCI artifact flow: an app pipeline pushes manifests to the container registry and moves the production tag, and an OCIRepository plus Kustomization deploy it. Roll back by moving the tag to an older artifact.
3. Interview question: "With GitOps, how do you roll back a bad release, and how do you handle an emergency change when Git is unavailable?" Hint: revert commits or move tags, Flux suspend, and reconciling the emergency change back into Git afterwards.

Terraform and OpenTofu Pipelines

Why it matters. Infrastructure changes are high-impact and often irreversible: deleting a database, replacing a load balancer, changing a security group. Running apply from a laptop means no review, no record and a state file nobody else can see. A pipeline enforces a fixed sequence: format check, validation, a reviewed plan, and an apply that executes exactly that plan.

Terraform and OpenTofu. In 2023 HashiCorp moved Terraform to the Business Source License. OpenTofu is the open-source fork maintained by the Linux Foundation and is largely compatible (tofu instead of terraform). GitLab responded by deprecating its Terraform CI/CD templates (removed in GitLab 18.0) and maintaining an OpenTofu CI/CD component. GitLab's Terraform state backend and the terraform report type work with both tools.

The pipeline shape.

 MR pipeline:       fmt ──► validate ──► plan ──(report in MR, section below)
 main pipeline:     fmt ──► validate ──► plan ──► apply (manual, protected environment)
                                            │         │
                                       plan.cache ────┘  apply uses exactly this plan

Using the OpenTofu component.

include:
  - component: $CI_SERVER_FQDN/components/opentofu/validate-plan-apply@X.Y.Z   # pick the current release in the CI/CD Catalog
    inputs:
      version: X.Y.Z                    # component version: selects the matching gitlab-opentofu image
      opentofu_version: 1.8.5           # OpenTofu CLI version
      root_dir: infra/                  # where the .tf files live
      state_name: production            # GitLab-managed state name (next topic)

stages: [validate, build, deploy]

The component creates fmt, validate, plan and apply jobs, wires the GitLab-managed state backend and the merge request report, and uses the gitlab-tofu wrapper script. Other components in the same project cover just validation, destroying environments, or building a full pipeline with several states.

A hand-written pipeline shows what the component does under the hood:

variables:
  TF_ROOT: infra
  TF_STATE_NAME: production
  TF_PLUGIN_CACHE_DIR: $CI_PROJECT_DIR/.tofu-plugin-cache

.tofu:
  image:
    name: ghcr.io/opentofu/opentofu:1.8.5
    entrypoint: [""]                         # the image's ENTRYPOINT is "tofu"
  cache:
    key: tofu-providers-$TF_ROOT
    paths: [.tofu-plugin-cache/]             # reuse downloaded providers across pipelines
  before_script:
    - mkdir -p "$TF_PLUGIN_CACHE_DIR"
    - cd "$TF_ROOT"
    - tofu init -input=false                 # backend settings come from TF_HTTP_* variables (next topic)

fmt:
  extends: .tofu
  stage: validate
  script: [tofu fmt -check -recursive -diff]

validate:
  extends: .tofu
  stage: validate
  script: [tofu validate]

plan:
  extends: .tofu
  stage: build
  script:
    - apk add --no-cache jq > /dev/null
    - tofu plan -input=false -lock=false -out=plan.cache    # -lock=false: see state permissions, next topic
    # Summary for the MR widget: counts of create/update/delete
    - >
      tofu show -json plan.cache
      | jq -r '([.resource_changes[]?.change.actions?] | flatten)
      | {"create": (map(select(. == "create")) | length),
         "update": (map(select(. == "update")) | length),
         "delete": (map(select(. == "delete")) | length)}'
      > plan.json
    - tofu show -no-color plan.cache > plan.txt   # human-readable plan for the MR comment (last topic)
  artifacts:
    access: developer                         # plans can contain sensitive values
    paths: [$TF_ROOT/plan.cache, $TF_ROOT/plan.txt]
    reports:
      terraform: $TF_ROOT/plan.json

apply:
  extends: .tofu
  stage: deploy
  needs: [plan]
  script:
    - tofu apply -input=false plan.cache     # applies exactly the reviewed plan; fails if state changed since
  environment:
    name: $TF_STATE_NAME
    deployment_tier: production
  resource_group: tofu-$TF_STATE_NAME        # never two applies on one state
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual

Rules for safe infrastructure pipelines.

Rule	Reason
Apply the saved plan file, never a fresh apply -auto-approve	What runs is what was reviewed. A stale saved plan fails instead of applying something new
One resource_group per state	Two concurrent applies on one state corrupt it or fight over the lock
Do not mark apply jobs interruptible	Canceling mid-apply leaves partly created infrastructure and a held lock (section 4.3)
Protect the apply environment	Only the platform team can apply to production (section 12.1)
Restrict plan artifacts	Plan files can contain secrets in plain text
Credentials through OIDC	Cloud providers via id_tokens (sections 5.4 and 12.4), not stored keys

Several environments. Use one state per environment and a parallel:matrix or separate jobs with TF_STATE_NAME and TF_ROOT per environment, each with its own protected environment and resource group.

Key takeaways

• Infrastructure pipelines run fmt, validate, plan and a manual apply of the saved plan file.
• GitLab's Terraform templates were removed in 18.0. Use the OpenTofu component, or write the jobs yourself.
• Serialize applies per state with resource_group, never interrupt them, and protect the apply environment.
• The terraform report type shows a create/update/delete summary in merge requests.

Practice

1. (Beginner) Build a pipeline with fmt, validate and plan jobs for a small configuration that creates a local file or a null resource. Break the formatting and watch fmt fail.
2. (Intermediate) Add a manual apply job that consumes the plan artifact. Change the state between plan and apply (apply a different change locally) and observe the stale plan error. Explain why this is the desired behavior.
3. Interview question: "Why should the apply job use the plan file from the plan job rather than running apply -auto-approve?" Hint: the time between review and apply, concurrent changes, and what the reviewer actually approved.

GitLab-Managed Terraform State

Why it matters. Terraform and OpenTofu record what they created in a state file. Kept locally, it is lost with the laptop, and two people applying at once corrupt it. Teams usually set up a remote backend (an S3 bucket with a DynamoDB lock table, a GCS bucket) before writing any real infrastructure, which is itself infrastructure to manage. GitLab provides a remote backend in every project, with locking, versioning, encryption and access control based on project roles.

What it is. GitLab implements the HTTP backend API. Each project can hold many named states. Jobs authenticate with the job token; people authenticate with a personal access token.

# infra/backend.tf
terraform {
  backend "http" {}        # all settings come from TF_HTTP_* variables or -backend-config
}

In CI, set the backend settings as environment variables:

variables:
  TF_STATE_NAME: production
  TF_HTTP_ADDRESS: ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/${TF_STATE_NAME}
  TF_HTTP_LOCK_ADDRESS: ${TF_HTTP_ADDRESS}/lock
  TF_HTTP_UNLOCK_ADDRESS: ${TF_HTTP_ADDRESS}/lock
  TF_HTTP_LOCK_METHOD: POST
  TF_HTTP_UNLOCK_METHOD: DELETE
  TF_HTTP_USERNAME: gitlab-ci-token
  TF_HTTP_PASSWORD: ${CI_JOB_TOKEN}              # job token: permissions of the user who ran the pipeline
  TF_HTTP_RETRY_WAIT_MIN: "5"

The OpenTofu component's gitlab-tofu wrapper sets all of these automatically from state_name.

Working with the state locally. Operate > Terraform states lists each state with its last update and lock status, and the Copy Terraform init command action prints the matching init command:

export GITLAB_ACCESS_TOKEN=glpat-...                 # personal access token with the api scope
STATE=https://gitlab.example.com/api/v4/projects/12345/terraform/state/production
tofu init \
  -backend-config="address=$STATE" \
  -backend-config="lock_address=$STATE/lock" \
  -backend-config="unlock_address=$STATE/lock" \
  -backend-config="username=alice" \
  -backend-config="password=$GITLAB_ACCESS_TOKEN" \
  -backend-config="lock_method=POST" \
  -backend-config="unlock_method=DELETE" \
  -backend-config="retry_wait_min=5"

Permissions.

Action	Minimum role
Read the state (plan -lock=false, output)	Developer
Lock, write (apply), unlock	Maintainer
Remove a state, manually unlock in the UI	Maintainer

Because the job token carries the permissions of the user who started the pipeline, a plan job in a merge request pipeline started by a Developer must use -lock=false (locking is a write). Apply jobs must be started by Maintainers, which fits well with protected environments.

Features.

• Versions: every write keeps the previous version, which can be downloaded from the UI or API to recover from a bad apply.
• Locking: GitLab holds the lock during plan (unless -lock=false) and apply. A stuck lock from a killed job can be released in the UI.
• Encryption: states are encrypted at rest.
• Remote data: other projects can read outputs with the terraform_remote_state data source using the same HTTP backend address and a token with read access.

When to use another backend. GitLab-managed state is convenient for teams already on GitLab. Organizations with existing S3 or GCS backends, cross-tool state access needs or very large states may keep their cloud backend. Both work in GitLab pipelines.

Key takeaways

• GitLab provides an HTTP state backend per project, with named states, locking, versioning and encryption.
• In CI, configure it with TF_HTTP_* variables and the job token. Locally, use a personal access token and the copied init command.
• Reading needs Developer; locking, writing and unlocking need Maintainer. Use -lock=false for plans started by Developers.
• Old state versions are kept and can be restored after a bad apply.

Practice

1. (Beginner) Configure GitLab-managed state for a small configuration, run apply from CI, then open Operate > Terraform states and download the state.
2. (Intermediate) Initialize the same state locally with a personal access token, run plan, and kill the process while it holds the lock. Release the lock from the UI. Then restore the previous version of the state through the API.
3. Interview question: "A Developer's merge request pipeline fails at plan with a 403 from the state backend, but the Maintainer's pipeline works. Why, and what is the correct fix?" Hint: what plan does to the state by default, and which role the job token carries.

Helm Chart Deployments

Why it matters. A Kubernetes application is a dozen manifests that differ slightly per environment: image tag, replica count, hostname, resources. Copying them per environment breaks the moment someone forgets to change one copy. Helm packages manifests as a templated chart with per-environment values, installs it as a named release, and tracks release history so a failed upgrade can be undone.

What it is.

chart/                      values-staging.yaml        values-production.yaml
├── Chart.yaml              replicaCount: 1            replicaCount: 6
├── values.yaml (defaults)  ingress.host: staging...   ingress.host: shop.example.com
└── templates/                       │                          │
    ├── deployment.yaml              └──── helm upgrade --install ────┘
    └── ingress.yaml                         + --set image.tag=$CI_COMMIT_SHA

Deploying a chart from a pipeline.

.helm-deploy:
  image: alpine/k8s:1.31.2
  script:
    - kubectl config use-context "$KUBE_CONTEXT"          # agent context, environment-scoped variable
    - >
      helm upgrade --install shop ./chart
      --namespace "$K8S_NAMESPACE" --create-namespace
      --values "chart/values-$CI_ENVIRONMENT_NAME.yaml"
      --set image.repository="$CI_REGISTRY_IMAGE"
      --set image.tag="$CI_COMMIT_SHA"
      --atomic --wait --timeout 10m
      --history-max 10
  resource_group: $CI_ENVIRONMENT_NAME

helm-lint:
  stage: test
  image: alpine/k8s:1.31.2
  script:
    - helm lint ./chart --values chart/values-production.yaml
    - helm template shop ./chart --values chart/values-production.yaml > rendered.yaml
    - kubeconform -strict -summary rendered.yaml            # validate against Kubernetes schemas

deploy-staging:
  extends: .helm-deploy
  stage: deploy
  environment: { name: staging, url: "https://staging.shop.example.com" }
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

deploy-production:
  extends: .helm-deploy
  stage: production
  environment: { name: production, url: "https://shop.example.com" }
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual

Important flags.

Flag	Effect
upgrade --install	Install if the release does not exist, upgrade if it does. Idempotent
--atomic	Wait for readiness and roll back automatically if the upgrade fails (implies --wait). Renamed --rollback-on-failure in Helm 4
--wait --timeout	Job fails if resources are not ready in time, so the deployment record is honest
--history-max	Limit stored release revisions (each is a Secret in the namespace)
--values / --set	Environment values file plus per-pipeline overrides such as the image tag

Publishing charts. When several services or teams share a chart, publish it as a versioned package instead of copying it. GitLab's package registry supports Helm repositories, and the container registry accepts charts as OCI artifacts:

publish-chart:
  stage: release
  image: alpine/k8s:1.31.2
  rules:
    - if: $CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/
  script:
    - VERSION="${CI_COMMIT_TAG#v}"
    - helm package ./chart --version "$VERSION" --app-version "$VERSION"
    # Option A: GitLab Helm repository (package registry), channel "stable"
    - >
      curl --fail --request POST
      --user "gitlab-ci-token:$CI_JOB_TOKEN"
      --form "chart=@shop-$VERSION.tgz"
      "$CI_API_V4_URL/projects/$CI_PROJECT_ID/packages/helm/api/stable/charts"
    # Option B: OCI artifact in the container registry
    - echo "$CI_REGISTRY_PASSWORD" | helm registry login "$CI_REGISTRY" -u "$CI_REGISTRY_USER" --password-stdin
    - helm push "shop-$VERSION.tgz" "oci://$CI_REGISTRY_IMAGE/charts"

Consumers add the repository with helm repo add --username <user> --password <token> acme "https://gitlab.example.com/api/v4/projects/<id>/packages/helm/stable", or install directly from oci://registry.example.com/acme/shop/charts/shop --version 1.4.0.

Helm and GitOps. Instead of running helm upgrade from CI, a Flux HelmRelease can install the chart from the GitLab registry and reconcile it continuously. The pipeline then only publishes the chart and updates the HelmRelease version in the fleet repository (previous topics).

Key takeaways

• Helm templates manifests into a chart, uses values files per environment and tracks release history.
• Deploy with helm upgrade --install --atomic --wait --timeout, setting the image tag to the commit SHA.
• Lint and render charts in CI (helm lint, helm template, schema validation) before deploying.
• Publish shared charts to the GitLab Helm repository or as OCI artifacts in the container registry.

Practice

1. (Beginner) Create a chart with helm create, add values-staging.yaml and values-production.yaml, and deploy the staging release from a pipeline through the GitLab agent.
2. (Intermediate) Deploy a version whose readiness probe always fails with --atomic. Observe the automatic rollback in helm history, and check what GitLab records for the deployment.
3. Interview question: "Your team has a chart copied into 40 service repositories, and each copy has drifted. How would you consolidate?" Hint: a versioned shared chart in the registry, per-service values, a migration plan, and how to roll out chart upgrades safely.

Infrastructure Plan Review in Merge Requests

Why it matters. Reviewing a .tf diff tells you what the author changed in the code. It does not tell you what will happen to real infrastructure. A one-line change to a resource's name can force a replacement that destroys and recreates a production database. The plan shows the actual effect, so it must be in front of the reviewer before the merge.

What it is. Three layers of plan visibility in a merge request:

Layer	What the reviewer sees	How
Terraform report widget	"Reported Resource Changes: 2 to add, 1 to change, 1 to delete", with a link to the full job log	artifacts:reports:terraform with the JSON summary (previous topic)
Plan as an MR comment	The full human-readable plan in a collapsible block	A job that posts tofu show output with the Notes API
Policy checks	Pass or fail on organization rules (no public buckets, no deletions of protected resources)	IaC scanning (chapter 11), OPA/Conftest on tofu show -json

Posting the plan as a comment. Update a single comment instead of adding a new one on every push. A hidden HTML marker identifies it:

plan-comment:
  stage: build
  image: alpine:3.20
  needs: [plan]
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  before_script:
    - apk add --no-cache curl jq
  script:
    - cd "$TF_ROOT"                           # plan.txt comes from the plan job's artifacts
    - MARKER="<!-- tofu-plan:$TF_STATE_NAME -->"
    # Build the comment body: marker, summary, full plan in a collapsible block
    - |
      {
        echo "$MARKER"
        echo "### OpenTofu plan for \`$TF_STATE_NAME\`"
        echo
        grep -E '^Plan:|^No changes' plan.txt || true
        echo
        echo '<details><summary>Full plan</summary>'
        echo
        echo '```diff'
        head -c 60000 plan.txt              # MR notes have a size limit
        echo '```'
        echo '</details>'
      } > body.md
    - API="$CI_API_V4_URL/projects/$CI_PROJECT_ID/merge_requests/$CI_MERGE_REQUEST_IID/notes"
    # Find our previous comment, if any
    - >
      NOTE_ID=$(curl --silent --header "PRIVATE-TOKEN: $PLAN_BOT_TOKEN" "$API?per_page=100"
      | jq --arg m "$MARKER" '[.[] | select(.body | startswith($m))][0].id // empty')
    - |
      if [ -n "$NOTE_ID" ]; then
        curl --fail --silent --request PUT --header "PRIVATE-TOKEN: $PLAN_BOT_TOKEN" \
          --data-urlencode "body@body.md" "$API/$NOTE_ID" > /dev/null
      else
        curl --fail --silent --request POST --header "PRIVATE-TOKEN: $PLAN_BOT_TOKEN" \
          --data-urlencode "body@body.md" "$API" > /dev/null
      fi

PLAN_BOT_TOKEN is a project access token with the api scope and the Reporter role, stored as a masked variable. The job only reads plan.txt from the plan job's artifacts, so it needs no OpenTofu binary or cloud credentials, and the comment shows exactly the plan that the widget summarizes.

Apply after merge versus apply before merge.

	Apply after merge (common)	Apply before merge
Flow	MR shows the plan; after merge, main plans again and a person runs the manual apply	The MR's own plan is applied from the MR, then the MR is merged
main reflects reality	Yes, once applied	Only if the merge follows a successful apply
Risk	The plan on main can differ from the MR plan if other MRs merged in between, so review it again before applying	Several open MRs can apply conflicting changes out of order
GitLab support	Native: protected environment on the apply job	Needs merge checks, locking and discipline

For apply-after-merge, keep the window small: merged results pipelines or merge trains (section 3.1) make the MR plan run against the latest main, so it matches what main will plan.

Approvals for infrastructure. Combine plan visibility with ownership rules:

# CODEOWNERS
/infra/production/   @acme/platform-team      # platform team must approve production changes
/infra/network/      @acme/network-team

With CODEOWNERS approval required on the protected branch and the apply job behind a protected environment (with deployment approvals for production), an infrastructure change needs a reviewed plan, an owner's approval and an authorized person to apply it.

Key takeaways

• Reviewers need the plan, not just the code diff: replacements and deletions are only visible in the plan.
• The terraform report gives a summary widget. Posting the full plan as one updated MR comment gives the details.
• Add policy checks (IaC scanning, OPA) on the plan for rules that should never depend on a reviewer noticing.
• With apply after merge, re-check the plan on main before applying, and use merged results pipelines to keep the MR plan current.

Practice

1. (Beginner) Add the terraform report to your plan job and open a merge request that adds one resource and changes another. Read the widget and follow its link to the full plan.
2. (Intermediate) Implement the plan comment job with a project access token. Push twice to the same merge request and confirm that the comment is updated, not duplicated. Then add a Conftest check that fails the pipeline if the plan deletes any resource of type aws_db_instance.
3. Interview question: "A reviewer approved an MR whose plan showed one change, yet the apply on main destroyed a resource. How could that happen, and how do you prevent it?" Hint: other merges between plan and apply, stale plans, and what the apply job should show or check before running.

<a id="126-auto-devops"></a>

12.6 Auto DevOps

Auto DevOps is GitLab's ready-made pipeline: it detects the project's language, builds a container image, runs security scanners, creates review apps and deploys to Kubernetes, all without a .gitlab-ci.yml. This subchapter explains what it does, which stages it runs, how to customize it without giving up its defaults, how Auto Deploy works, and when it is the right choice.

Auto DevOps Overview

Why it matters. A team with a dozen small services may not want to write and maintain a dozen pipelines. Auto DevOps gives every project a complete, opinionated CI/CD pipeline from build to production, based on GitLab's best-practice templates. It is also a useful reference: reading its templates shows how GitLab itself recommends wiring builds, scans and deployments.

What it is. Auto DevOps is a large CI/CD template, Auto-DevOps.gitlab-ci.yml, that GitLab uses as the pipeline of a project when:

1. Auto DevOps is enabled for the project, its group or the instance, and
2. The project has no .gitlab-ci.yml (or a custom CI configuration path).

The template includes job templates for each stage (Jobs/Build.gitlab-ci.yml, Jobs/Deploy.gitlab-ci.yml, the security scanner templates and others). Each job has rules that check variables and repository contents, so only the relevant jobs run.

            ┌───────── always (if detected) ─────────┐  ┌─────── needs Kubernetes or AWS ───────┐
 push ──►   build ──► test ──► scanners ──────────────►  review app (branches) ──► DAST
                                (SAST, secrets,          staging / canary / production (default branch)
                                 dependencies,           incremental rollout, browser performance
                                 containers)

Enabling it.

Level	Where	Notes
Project	Settings > CI/CD > Auto DevOps > Default to Auto DevOps pipeline	Also choose the deployment strategy here (Auto Deploy topic)
Group	Group > Settings > CI/CD > Auto DevOps	Applies to all projects in the group that do not override it
Instance	Admin > Settings > CI/CD	When enabled instance-wide, GitLab disables it in a project after the first failed Auto DevOps pipeline, so it does not keep failing

What it needs.

Feature	Requirement
Build	Runners that can build images: Docker-in-Docker with privileged mode, or a supported alternative
Build detection	A Dockerfile, or a language Cloud Native Buildpacks can detect (Node.js, Python, Go, Java, Ruby, PHP, .NET...)
Review apps and deploys to Kubernetes	A cluster connected with the GitLab agent, KUBE_CONTEXT set to the agent context, KUBE_INGRESS_BASE_DOMAIN set to a wildcard domain, and an ingress controller
Deploys to AWS	AUTO_DEVOPS_PLATFORM_TARGET set to ECS, FARGATE or EC2, plus AWS settings (Auto Deploy topic)

# Minimal project or group CI/CD variables for a Kubernetes target (set in Settings > CI/CD > Variables)
KUBE_CONTEXT: acme/infra:auto-devops-agent        # agent context, as in section 12.4
KUBE_INGRESS_BASE_DOMAIN: apps.example.com        # *.apps.example.com points at the ingress controller

Without a deploy target, Auto DevOps still builds, tests and scans, which is useful on its own.

Key takeaways

• Auto DevOps is a GitLab-maintained template pipeline that runs when enabled and the project has no .gitlab-ci.yml.
• It detects the build method (Dockerfile or Cloud Native Buildpacks) and enables jobs based on repository contents and variables.
• Deployment features need a Kubernetes cluster through the agent (KUBE_CONTEXT, KUBE_INGRESS_BASE_DOMAIN) or an AWS target.
• Enabled instance-wide, it disables itself in a project after the first failure.

Practice

1. (Beginner) Enable Auto DevOps on a small project with a Dockerfile and no .gitlab-ci.yml. List the jobs it created and find the image it pushed to the container registry.
2. (Intermediate) Open the Auto-DevOps.gitlab-ci.yml template in the GitLab source, and trace which rules decide whether the review and production jobs are added. Note which variables control them.
3. Interview question: "Why does GitLab ignore Auto DevOps as soon as a project adds a .gitlab-ci.yml, and what does that imply for teams that want to extend it?" Hint: precedence of explicit configuration, and including the template from your own file.

Auto DevOps Stages

Why it matters. Using Auto DevOps well means knowing what each stage does and what it expects, so you know which parts to keep, tune or replace. Some stages have been deprecated over the years as GitLab moved functionality elsewhere, and older tutorials still mention them.

What it is. The main stages and the jobs they create:

Stage (feature)	Job(s)	What it does	Runs when
Auto Build	build	Builds an image with the Dockerfile or Cloud Native Buildpacks, pushes it to the container registry tagged with the commit SHA	Always
Auto Test	test	Runs tests detected by Herokuish buildpacks	Only with Herokuish-based builds; not supported with Cloud Native Buildpacks (the default). Most projects add their own test jobs
Auto Code Quality	code_quality	Code Climate based analysis	Deprecated since GitLab 17.3 (section 10.3); run your own linters with a Code Quality report instead
Auto SAST	semgrep-sast and others	Static analysis (section 11.1)	Matching source files exist
Auto Secret Detection	secret_detection	Credentials in code (section 11.1)	Always
Auto Dependency Scanning	dependency scanning jobs	Vulnerable dependencies	Supported lockfiles exist
Auto Container Scanning	container_scanning	Vulnerabilities in the built image	After build
Auto Review Apps	review, stop_review	Deploys every branch except the default to review/<branch>	Kubernetes configured
Auto DAST	dast, dast_environment_deploy	Dynamic scan against the review app or a dedicated DAST environment	Ultimate, review apps or DAST environment available
Auto Deploy	staging, canary, production, production_manual, rollout <n>%	Deploys the default branch (next topics)	Kubernetes or AWS configured
Auto Browser Performance	browser_performance	sitespeed.io against the deployed URL (chapter 10)	After deploy, Premium for the MR widget

Each stage can be switched off with a *_DISABLED variable:

# Project CI/CD variables (or variables: in a .gitlab-ci.yml that includes the template)
TEST_DISABLED: "true"                   # we run our own tests
CODE_QUALITY_DISABLED: "true"
BROWSER_PERFORMANCE_DISABLED: "true"
REVIEW_DISABLED: "true"                 # no review apps for this project
DAST_DISABLED: "true"

Build details. Auto Build prefers a Dockerfile when one exists. Otherwise it uses Cloud Native Buildpacks, with the builder image set by AUTO_DEVOPS_BUILD_IMAGE_CNB_BUILDER. Useful variables:

Variable	Purpose
DOCKERFILE_PATH	Use a Dockerfile in another location
AUTO_DEVOPS_BUILD_IMAGE_EXTRA_ARGS	Extra docker build arguments, such as --build-arg
AUTO_DEVOPS_BUILD_IMAGE_FORWARDED_CI_VARIABLES	CI variables to pass into the build as build args
AUTO_DEVOPS_BUILD_IMAGE_CNB_BUILDER	Buildpacks builder image

Key takeaways

• Auto DevOps chains build, (optional) test, security scanners, review apps, DAST, deploy and performance testing.
• Every stage has rules. Most can be disabled with a *_DISABLED variable.
• Auto Build uses the Dockerfile if present, otherwise Cloud Native Buildpacks.
• Auto Test and Auto Code Quality are legacy. Plan to provide your own test and lint jobs.

Practice

1. (Beginner) Run Auto DevOps on a project without a Dockerfile and identify which buildpack was used from the build job log. Then add a Dockerfile and compare.
2. (Intermediate) Disable the stages you do not need with *_DISABLED variables, and confirm in the next pipeline which jobs remain. Explain why the security jobs still appear.
3. Interview question: "Auto DevOps reports no test results for your Node.js service, yet the pipeline is green. Why, and what would you change?" Hint: which build method is in use and how Auto Test depends on it.

Customizing Auto DevOps

Why it matters. Few projects fit any default exactly. You may need a different test command, an extra job, a custom Helm chart or a secret passed to the app. Rewriting the whole pipeline throws away everything Auto DevOps does well. Customization lets you keep the template and change only what differs.

What it is. There are four levels of customization, from lightest to heaviest:

Level	How	Keeps upstream updates
1. Variables	Project or group CI/CD variables (*_DISABLED, STAGING_ENABLED, HELM_UPGRADE_EXTRA_ARGS...)	Yes
2. Include and override	.gitlab-ci.yml that includes Auto-DevOps.gitlab-ci.yml and redefines or adds jobs	Mostly (overridden jobs are yours)
3. Pick job templates	Include only Jobs/Build.gitlab-ci.yml, Jobs/Deploy.gitlab-ci.yml and others you want	Yes, per template
4. Custom chart or values	chart/ directory in the repository, AUTO_DEVOPS_CHART, or .gitlab/auto-deploy-values.yaml	Deploy logic yes, chart is yours

Level 2: include and override. Once you add a .gitlab-ci.yml, Auto DevOps no longer runs by itself, so include the template explicitly. Jobs with the same name are merged with your definition:

# .gitlab-ci.yml
include:
  - template: Auto-DevOps.gitlab-ci.yml

variables:
  STAGING_ENABLED: "1"                    # staging, then manual production
  TEST_DISABLED: "true"                   # replaced by our own job below

unit-tests:                               # new job in the template's "test" stage
  stage: test
  image: node:22-alpine
  script:
    - npm ci
    - npm test -- --reporter=junit --reporter-options output=junit.xml
  artifacts:
    reports:
      junit: junit.xml

production_manual:                        # override part of a template job
  rules:                                  # simplified: copy the template's other conditions too
    - if: $CI_DEPLOY_FREEZE
      when: never                         # no production deploys during freezes
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual

Overriding rules replaces the template's rules for that job entirely, so copy any conditions you still need (look them up in the template).

Level 3: pick individual templates.

include:
  - template: Jobs/Build.gitlab-ci.yml           # Auto Build only
  - template: Jobs/Deploy.gitlab-ci.yml          # review, staging, production jobs
  - template: Security/SAST.gitlab-ci.yml

# Every stage a template job uses must exist, even if that job's rules exclude it
stages:
  - build
  - test
  - deploy
  - review
  - dast
  - staging
  - canary
  - production
  - incremental rollout 10%
  - incremental rollout 25%
  - incremental rollout 50%
  - incremental rollout 100%
  - performance
  - cleanup

Application configuration and secrets. Auto Deploy passes every CI/CD variable prefixed with K8S_SECRET_ into a Kubernetes Secret that the application receives as environment variables, with the prefix removed. Scope them per environment:

CI/CD variable                         scope         →  in the pod
K8S_SECRET_DATABASE_URL = postgres://… production    →  DATABASE_URL=postgres://…
K8S_SECRET_FEATURE_X     = "on"        review/*      →  FEATURE_X=on

Level 4: deployment chart and values. Auto Deploy uses the auto-deploy-app Helm chart. Adjust it with a values file, or replace it:

# .gitlab/auto-deploy-values.yaml (picked up automatically, or set HELM_UPGRADE_VALUES_FILE)
replicaCount: 3
livenessProbe:
  path: /healthz
readinessProbe:
  path: /ready
resources:
  requests: { cpu: 200m, memory: 256Mi }
  limits:   { memory: 512Mi }

Put a complete chart in a chart/ directory to use your own chart instead, or point AUTO_DEVOPS_CHART and AUTO_DEVOPS_CHART_REPOSITORY at a published chart. HELM_UPGRADE_EXTRA_ARGS adds flags to the helm upgrade call.

Key takeaways

• Prefer variables, then include-and-override, then individual job templates, then a custom chart. Each step takes on more maintenance.
• A .gitlab-ci.yml turns off implicit Auto DevOps, so include Auto-DevOps.gitlab-ci.yml explicitly to extend it.
• Overriding a template job's rules replaces all of them.
• K8S_SECRET_* variables become environment variables in the deployed pods. .gitlab/auto-deploy-values.yaml or a chart/ directory customize the deployment.

Practice

1. (Beginner) Create a .gitlab-ci.yml that includes Auto-DevOps.gitlab-ci.yml, disables Auto Test and adds your own test job with a JUnit report.
2. (Intermediate) Add a K8S_SECRET_ variable scoped to production, deploy, and confirm with kubectl exec ... env through the agent that the pod received it without the prefix. Then add .gitlab/auto-deploy-values.yaml with custom probes and check the rendered Deployment.
3. Interview question: "You override the production job's rules to add a condition, and now production deploys run on feature branches. What happened?" Hint: how job merging treats arrays like rules.

Auto Deploy

Why it matters. Auto Deploy is the part of Auto DevOps that delivers the built image to environments. It packs many of this chapter's techniques (review apps, staging, canaries, incremental rollouts, stop jobs) into a few variables. Understanding it lets you use it confidently or copy its patterns into your own pipelines.

What it is. Auto Deploy jobs run the auto-deploy-image tool, which runs helm upgrade with the auto-deploy-app chart (or your chart) using the agent context in KUBE_CONTEXT. What it deploys depends on the branch and the chosen strategy:

feature branch ──► review  (review/<branch>, auto-stops, stop_review on branch deletion)

default branch, strategy:
  "Continuous deployment to production"           ──► production
  "...using timed incremental rollout"            ──► timed rollout 10% ─► 25% ─► 50% ─► 100% (5 min apart)
  "Automatic deployment to staging, manual to production"
                                                  ──► staging ──► production_manual (or rollout n% manual)
  + CANARY_ENABLED                                ──► canary (manual) before production

Variable	Effect
STAGING_ENABLED	Deploy the default branch to staging automatically, production becomes manual
CANARY_ENABLED	Add a manual canary job that deploys to the production environment with a canary track
INCREMENTAL_ROLLOUT_MODE	manual or timed: production in steps of 10/25/50/100%
REVIEW_DISABLED, PRODUCTION_REPLICAS, REPLICAS	Turn off review apps; replica counts per environment
KUBE_NAMESPACE	Override the default namespace (by default derived from project name, project ID and environment)
ROLLOUT_RESOURCE_TYPE	deployment (default) or statefulset
AUTO_DEPLOY_IMAGE_VERSION	Pin the deploy tool version

Databases and migrations. For non-production convenience, Auto Deploy can provision a PostgreSQL database per environment with POSTGRES_ENABLED: "true". For production, use a managed database and pass its URL in a K8S_SECRET_DATABASE_URL variable. Migrations run as part of the deployment when DB_MIGRATE is set to a command:

variables:
  DB_MIGRATE: "bin/rails db:migrate"      # runs in the new image before the app is upgraded
  DB_INITIALIZE: "bin/rails db:setup"     # runs only on the first deployment of an environment

Migrations still need to be backward compatible with the running version (expand and contract, section 12.3), especially with canaries and incremental rollouts, where old and new versions run side by side.

Deploying to AWS instead of Kubernetes. AUTO_DEVOPS_PLATFORM_TARGET switches Auto Deploy to AWS:

Target	Required settings (examples)	Behavior
ECS / FARGATE	CI_AWS_ECS_CLUSTER, CI_AWS_ECS_SERVICE, CI_AWS_ECS_TASK_DEFINITION	Registers a task definition with the new image and updates the service
EC2	CI_AWS_CF_CREATE_STACK_FILE, CI_AWS_EC2_DEPLOYMENT_FILE, S3 push settings	CloudFormation stack plus CodeDeploy

These targets authenticate with AWS variables in the project. Check whether your GitLab version supports OIDC for them; otherwise keep the credentials protected, environment-scoped and narrowly permissioned.

What you get in GitLab. Auto Deploy jobs declare environments with URLs, on_stop and auto_stop_in for review apps, so everything from sections 12.1 and 12.2 works: environment history, rollbacks, protected environments and approvals on production, and deploy freezes through your own rule overrides.

Key takeaways

• Auto Deploy deploys branches to review apps and the default branch to production, optionally through staging, canary and incremental rollout steps.
• Choose the strategy in the Auto DevOps settings or with STAGING_ENABLED, CANARY_ENABLED and INCREMENTAL_ROLLOUT_MODE.
• DB_MIGRATE and DB_INITIALIZE run database commands during deployment. Use the bundled PostgreSQL only outside production.
• AUTO_DEVOPS_PLATFORM_TARGET deploys to ECS, Fargate or EC2 instead of Kubernetes.

Practice

1. (Beginner) With a cluster connected through the agent, enable Auto DevOps on a sample app, open a merge request and use the review app. Merge it and watch the review app stop and production deploy.
2. (Intermediate) Switch to "Automatic deployment to staging, manual deployment to production" with INCREMENTAL_ROLLOUT_MODE: manual and CANARY_ENABLED. Run the canary, then the rollout steps, and inspect the pods of each track in the environment's Kubernetes overview.
3. Interview question: "Auto Deploy runs DB_MIGRATE before upgrading the app. During a 10% rollout, what can go wrong with a migration that renames a column?" Hint: which application versions run against the migrated schema, and for how long.

When to Use Auto DevOps

Why it matters. Auto DevOps saves a lot of work in some situations and adds friction in others. Choosing it, or choosing to move away from it, is an architectural decision about how much control a team needs over its delivery process.

Comparison.

Criterion	Auto DevOps	Custom pipeline
Time to first deployment	Minutes, if a cluster is ready	Hours to days
Fit for standard web services in containers	Good	Good
Unusual builds (monorepos, mobile apps, native binaries, multi-image)	Poor	Full control
Security scanning	Included and maintained by GitLab	You include templates or components yourself
Deployment target	Kubernetes (auto-deploy chart) or AWS ECS/EC2	Anything
Understanding and debugging	Large template, many variables, behavior defined upstream	Your own, readable YAML
Upgrades	Template changes with GitLab versions, sometimes unexpectedly	Changes only when you change it
Consistency across many projects	Very high by default	Needs shared components or templates (chapter 9)

Good fits.

• Many small, similar containerized services, where consistency matters more than per-project tuning.
• Prototypes, internal tools and teams new to CI/CD that want a working pipeline with security scanning from day one.
• Learning: reading the generated jobs is a practical guide to GitLab CI/CD patterns.

Poor fits.

• Projects with non-standard builds, monorepos with selective builds, or targets other than Kubernetes and AWS.
• Regulated environments that require every pipeline step to be explicitly reviewed and pinned.
• Teams that have already overridden most jobs: at that point the template adds indirection without value.

A common path. Many organizations use Auto DevOps to start, then move to shared CI/CD components maintained by a platform team (chapter 9): the same consistency, but with explicit inputs, versioning, and only the pieces each project needs. The Auto DevOps job templates (Jobs/Build.gitlab-ci.yml, Jobs/Deploy.gitlab-ci.yml) are a reasonable intermediate step.

Auto DevOps (implicit) ──► include Auto-DevOps + overrides ──► individual Jobs/* templates ──► own components
   zero config               small deviations                  pick what you need             full control, versioned

Key takeaways

• Auto DevOps fits fleets of standard containerized services and teams that want a full pipeline immediately.
• It fits poorly with unusual builds, non-Kubernetes targets, strict audit requirements and heavily customized projects.
• Migrate gradually: include and override, then individual job templates, then your own versioned CI/CD components.

Practice

1. (Beginner) List three projects you know (or invent) and decide for each whether Auto DevOps fits, with one reason per project.
2. (Intermediate) Take a project running Auto DevOps and replace it with an explicit .gitlab-ci.yml that uses Jobs/Build.gitlab-ci.yml, a scanner component and your own deploy job. Compare the job lists before and after to make sure nothing important was dropped.
3. Interview question: "Your company has 150 microservices on Auto DevOps and a GitLab upgrade changed a template default that broke deployments. How do you prevent that in the future without losing consistency?" Hint: pinning (AUTO_DEPLOY_IMAGE_VERSION), versioned CI/CD components, and testing template changes on a canary group of projects.

---

<a id="13-pipeline-optimization"></a>