6. GitLab Runners

Every job in the previous chapters ran "somewhere": that somewhere is a GitLab Runner, a separate agent that asks GitLab for work, prepares an environment, runs the script and reports the result back. This chapter covers how runners connect to and authenticate with GitLab, how executors decide where and how a job runs, how config.toml shapes a runner's behavior, and how fleets of runners are autoscaled, monitored and upgraded. Understanding runners explains why jobs sit in pending, why the same job behaves differently on two machines, and where most of the CI/CD infrastructure bill comes from.

<a id="61-runner-architecture"></a>

6.1 Runner Architecture

A runner is the bridge between GitLab, which decides what must run, and the machines that actually run it. This subchapter covers the polling protocol, the three runner scopes, the runners GitLab operates for you, how runners authenticate, and how one runner can be served by many processes.

How Runners Pick Up Jobs

Why it matters. GitLab itself never executes a job. Keeping execution out of the GitLab server protects it from untrusted build code, lets builds run on any operating system or network, and lets build capacity scale independently of GitLab. The price is a protocol between the two, and most "my job is stuck" problems are really questions about that protocol.

What it is. GitLab Runner is a single Go binary, gitlab-runner, that runs on Linux, Windows, macOS and FreeBSD, in a container, or inside Kubernetes. It works on a pull model: the runner always starts the conversation. Think of delivery couriers who periodically ask the dispatch office "anything for me?", rather than a dispatcher who has to know where every courier is and phone them.

GitLab (Rails + Workhorse)                                   Runner (gitlab-runner)
───────────────────────────                                  ──────────────────────
                     <── POST /api/v4/jobs/request ───────── "Any job for token glrt-...?
                                                              I use the docker executor, v19.4"
Workhorse holds the request open (long polling, ~50 s)
── 204 No Content ─────────────────────────────────────────> nothing yet: wait check_interval, ask again
...
Job 554433 becomes pending and matches this runner
── 201 Created + job payload ──────────────────────────────> script, image, services, variables,
                                                              CI_JOB_TOKEN, git ref and SHA
                     <── PATCH /api/v4/jobs/554433/trace ─── stream log chunks while running
                     <── PUT   /api/v4/jobs/554433 ───────── update state: running, success, failed

Because the runner opens every connection, it only needs outbound HTTPS to GitLab. A runner can sit behind NAT, in a private subnet or on a laptop, and GitLab never needs credentials for the build machine. The only inbound connection is optional: the session server used by interactive web terminals (section 6.3).

How GitLab chooses a job for a runner. When a runner asks for work, GitLab looks for a pending job that this runner is allowed to take:

Check	The runner can take the job only if...
Scope	The runner is available to the job's project (instance, group or project runner, see the next topic)
Tags	The runner has every tag the job lists (section 2.2)
Untagged jobs	The job has no tags and the runner has "Run untagged jobs" enabled
Protection	The runner is not protected, or the pipeline runs for a protected branch or tag
State	The runner is not paused

If several jobs qualify, project and group runners take them roughly in queue order. Instance runners use a fair usage queue: jobs from projects that currently have the fewest jobs running on instance runners go first, so one project with 200 parallel jobs cannot starve everyone else.

What happens after pickup. Once the runner accepts a job, it hands it to an executor (section 6.2), which runs a fixed sequence of stages. These stages are exactly the collapsible sections in the job log (section 1.3):

prepare executor  ─> get sources ─> restore cache ─> download artifacts
   (pull image,       (git fetch,     (from local      (from earlier jobs
    start services)    checkout)       or S3 cache)     via needs/dependencies)
        ─> step_script (before_script + script) ─> after_script
        ─> archive cache ─> upload artifacts ─> cleanup ─> report final status

The runner exposes information about itself to the job through predefined CI_RUNNER_* variables, which are job-only (section 5.1):

whoami:
  script:
    - echo "Runner ID:        $CI_RUNNER_ID"
    - echo "Description:      $CI_RUNNER_DESCRIPTION"
    - echo "Tags:             $CI_RUNNER_TAGS"            # JSON-like list, for example ["docker", "linux"]
    - echo "Version:          $CI_RUNNER_VERSION"
    - echo "Architecture:     $CI_RUNNER_EXECUTABLE_ARCH" # for example linux/amd64
    - echo "Short token:      $CI_RUNNER_SHORT_TOKEN"     # identifies the runner in logs, safe to print

The job page shows two durations worth separating: queued time (pending until a runner took the job) and duration (running). Long queue times are a capacity or matching problem, never a code problem.

Key takeaways

• Runners pull jobs from GitLab over outbound HTTPS. GitLab never connects to a runner to start a job.
• Long polling makes pickup almost immediate without hammering the API.
• A job matches a runner only if scope, tags, untagged setting, protection and paused state all allow it.
• Instance runners use a fair usage queue across projects.
• A job's lifecycle on the runner (prepare, sources, cache, script, artifacts) maps one-to-one to the sections of the job log.

Practice

1. (Beginner) Add the whoami job above to a project and run it twice. Compare the runner ID, tags and version, and find the same runner in Settings > CI/CD > Runners.
2. (Intermediate) Run a self-managed runner in the foreground with gitlab-runner --debug run and trigger a pipeline. In the output, identify the job request that returned no job, the request that returned a job, and the trace updates. Measure the delay between the pipeline starting and the job being picked up.
3. Interview question: "Why does GitLab Runner pull jobs instead of GitLab pushing jobs to runners?" Hint: think about firewalls and NAT, which side needs credentials for the other, and how capacity is added or removed.

Instance, Group and Project Runners

Why it matters. Not every runner should serve every project. A company-wide pool of Linux runners is useful to everyone, a GPU machine bought by the data team is not, and a runner holding production credentials must serve only one deployment project. Runner scope decides who can use a runner and who can manage it.

What it is. Every runner belongs to exactly one of three types, which mirror the hierarchy from section 1.2:

Type	Available to	Created and managed by	Where in the UI	Typical use
Instance runner (formerly "shared runner")	Every project on the instance (unless disabled)	Administrators	Admin > CI/CD > Runners	General-purpose pool. GitLab-hosted runners on GitLab.com are instance runners
Group runner	All projects in the group and its subgroups	Group Owners	Group Build > Runners	A team's or business unit's own capacity
Project runner	The projects it is assigned to	Project Maintainers	Project Settings > CI/CD > Runners	Special hardware, deployment runners, isolated workloads

A project runner can be assigned to several projects, unless it is locked to its current projects. A Maintainer of another project can then enable the unlocked runner for their project too.

Instance runners:  [linux-pool-1] [linux-pool-2]          available to every project (if allowed)
  └── Group acme:  [acme-docker]                          available to all acme projects
        └── Subgroup acme/payments: [payments-arm64]      available to payments projects only
              └── Project shop: [shop-deploy] (locked)    only shop

Runners the "shop" project can use: linux-pool-1, linux-pool-2, acme-docker, payments-arm64, shop-deploy

Each level can switch off what it inherits:

Setting	Where	Effect
Enable instance runners for this project	Project Settings > CI/CD > Runners	Turn off to keep jobs off the shared pool (or off paid GitLab-hosted minutes)
Enable instance runners for this group	Group Settings > CI/CD > Runners	Disables them for every project in the group. Projects can be allowed to re-enable them
Enable group runners	Project Settings > CI/CD > Runners	Ignore runners inherited from groups

The same information is available through the API, which helps when auditing many projects:

# List every runner the project can use (project, inherited group, and allowed instance runners)
curl --silent --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.example.com/api/v4/projects/12345/runners?per_page=100" \
  | jq -r '.[] | [.id, .runner_type, .status, .description] | @tsv'

# Assign an existing, unlocked project runner (ID 77) to another project
curl --silent --request POST --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.example.com/api/v4/projects/67890/runners" --data "runner_id=77"

# Turn off instance runners for one project
curl --silent --request PUT --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.example.com/api/v4/projects/12345" --data "shared_runners_enabled=false"

Choosing a scope is a trade-off between sharing (fewer idle machines) and isolation (fewer people and projects that can run code on a machine):

Concern	Prefer broader scope (instance, group)	Prefer narrower scope (project)
Utilization	Many projects share idle capacity	Capacity reserved even when idle
Maintenance	One fleet to upgrade and monitor	Many small fleets
Trust	Workloads of similar trust level	Runners with credentials, network access to production, or special hardware
Cost attribution	Harder to charge back per team	Clear ownership

Key takeaways

• Instance runners serve the whole instance, group runners serve a group subtree, project runners serve the projects they are assigned to.
• Runner types follow the group hierarchy: a project sees its own runners plus those of every ancestor group plus allowed instance runners.
• Lock project runners that must never be reused by other projects.
• Projects and groups can disable inherited instance and group runners.

Practice

1. (Beginner) For one of your projects, list all available runners in the UI and classify each as instance, group or project runner.
2. (Intermediate) Design runner scopes for a company with a shared Linux pool, a data team with two GPU machines and a payments team whose deploy runner can reach production. State which runners are locked and which inherited runners each team's projects disable.
3. Interview question: "An auditor asks which projects can run code on the runner that holds production database credentials. How do you answer, and how would you make the answer short?" Hint: consider runner type, locking, assignment lists, and the protected flag from section 6.3.

GitLab-Hosted Runners

Why it matters. Running your own build machines means patching, scaling and paying for them even when idle. On GitLab.com, GitLab operates a fleet of runners for you, so a new project can run pipelines with zero infrastructure. Knowing what they offer, and where they stop, tells you when to bring your own.

What it is. GitLab-hosted runners are instance runners operated by GitLab. They are available on GitLab.com and, as an add-on, on GitLab Dedicated. Self-managed instances have none. Key properties:

• One fresh VM per job. Each job gets a new, ephemeral virtual machine that is deleted after the job, so nothing leaks between jobs or projects.
• Linux runners use the Docker executor in privileged mode, so Docker-in-Docker builds (chapter 7) work out of the box.
• Billing in compute minutes. Job duration is multiplied by a cost factor that grows with machine size, and the result counts against the namespace's quota (section 1.2).
• Selection by tags. Each machine type has a predefined tag. Untagged jobs run on the small Linux amd64 machine.

Platform	Tags	Notes
Linux amd64	saas-linux-small-amd64, -medium-, -large-, -xlarge-, -2xlarge-amd64	Small is the default for untagged jobs
Linux arm64	saas-linux-small-arm64, saas-linux-medium-arm64, saas-linux-large-arm64	Native ARM builds without emulation
Linux GPU	saas-linux-medium-amd64-gpu-standard	For ML training and GPU tests
macOS	saas-macos-medium-m1, saas-macos-large-m2pro	Apple silicon, Xcode images. Premium and Ultimate only
Windows	saas-windows-medium-amd64	Windows Server with common build tools

The Linux amd64 sizes scale linearly in resources and cost. The values below are a guide; check the current documentation, because machine specs and cost factors change over time.

Size	vCPUs	Memory	Cost factor
small	2	8 GB	1
medium	4	16 GB	2
large	8	32 GB	3
xlarge	16	64 GB	6
2xlarge	32	128 GB	12

lint:
  image: node:20-alpine
  script: [npm ci, npm run lint]                 # untagged: small Linux amd64, cost factor 1

build-arm-image:
  tags: [saas-linux-medium-arm64]                # native arm64: no QEMU emulation needed
  image: docker:27
  services: [docker:27-dind]                     # works because hosted Linux runners are privileged
  script:
    - docker build -t "$CI_REGISTRY_IMAGE:arm64-$CI_COMMIT_SHORT_SHA" .

ios-build:
  tags: [saas-macos-medium-m1]
  image: macos-15-xcode-16                       # on macOS runners, image selects the VM image
  script:
    - xcodebuild -scheme App -destination 'generic/platform=iOS' build

A bigger machine is not automatically more expensive per pipeline: a test suite that takes 20 minutes on small (20 x 1 = 20 compute minutes) and 6 minutes on large (6 x 3 = 18) is both faster and cheaper on large. Measure before choosing (section 13.2 covers sizing).

When to bring your own runners

Need	Why hosted runners do not fit
Access to private networks (internal registries, databases, on-premises clusters)	Hosted runners run on the public internet
Fixed egress IPs for allowlisting	Hosted runner IP ranges are not guaranteed to stay fixed
Custom hardware, OS versions, or kernel settings	Machine types and images are fixed by GitLab
Very high, steady load	Your own autoscaled fleet can be cheaper than compute minutes
Self-managed GitLab	No hosted runners exist

Both can coexist: a project can use hosted runners for ordinary jobs and route deployment jobs to its own runner with a tag.

Key takeaways

• GitLab-hosted runners are instance runners on GitLab.com (and Dedicated) that give each job a fresh, ephemeral VM.
• Jobs pick a machine type with predefined saas-* tags. Untagged jobs use small Linux amd64.
• Usage is billed as job duration times the machine's cost factor.
• Bring your own runners for private network access, fixed IPs, special hardware, steady heavy load or self-managed GitLab.

Practice

1. (Beginner) Run the same CPU-heavy job (for example a large test suite or a compression benchmark) on small, medium and large Linux runners. Record duration and compute minutes consumed for each.
2. (Intermediate) Your pipeline builds a multi-architecture image using QEMU emulation on amd64 and takes 25 minutes. Redesign it to build each architecture natively on hosted runners and combine the results into one manifest. Estimate the change in compute minutes.
3. Interview question: "When would you recommend that a team on GitLab.com stops using hosted runners for some of its jobs?" Hint: network reachability, IP allowlists, hardware, and the cost curve at high volume.

Runner Authentication Tokens and Registration

Why it matters. A runner receives every job it picks up, including the job's CI/CD variables, secrets and CI_JOB_TOKEN. Whoever holds a runner's token can therefore receive jobs and their secrets. How runners are created and how their tokens are protected is a core security question, not an installation detail.

What it is. Registration links a gitlab-runner process to a runner record in GitLab. The current workflow separates the two steps cleanly:

1. Create the runner in GitLab (UI or API). Here you set the scope, description, tags, "Run untagged jobs", protected, paused, locked and maximum job timeout. GitLab returns a runner authentication token with the prefix glrt-, shown only once.
2. Register on the machine with gitlab-runner register --token glrt-.... The command verifies the token, asks for the executor, and writes a [[runners]] entry to config.toml.
3. The runner starts polling. In the runner list, its status changes from "Never contacted" to "Online".

Maintainer / Owner / Admin                     Build machine
──────────────────────────                     ─────────────
New runner: tags, scope, protected...
GitLab creates runner #42, token glrt-AbC...
            ── token handed over securely ──>  gitlab-runner register --url ... --token glrt-AbC...
                                               config.toml: [[runners]] token = "glrt-AbC..."
                                               gitlab-runner run  ─── polls with the token ──> GitLab

Creating a runner through the API needs a personal, group or project access token with the create_runner scope:

# Create a group runner; the response contains the authentication token exactly once
RUNNER_TOKEN=$(curl --silent --request POST \
  --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.example.com/api/v4/user/runners" \
  --data "runner_type=group_type" \
  --data "group_id=42" \
  --data "description=docker-linux-amd64" \
  --data "tag_list=docker,linux,amd64" \
  --data "run_untagged=false" \
  | jq -r '.token')                       # glrt-...: store it in a secret manager, never in git

Then register it on the target machine. Non-interactive registration is what provisioning scripts and images use:

sudo gitlab-runner register \
  --non-interactive \
  --url "https://gitlab.example.com" \
  --token "$RUNNER_TOKEN" \
  --executor "docker" \
  --docker-image "alpine:3.20" \
  --name "docker-linux-01"                # local name of this entry in config.toml

With an authentication token, settings that belong to the runner record (tags, untagged, locked, protected, paused, maximum timeout) cannot be passed to register. They live in GitLab and are changed in the UI or with PUT /runners/:id. Only local settings, such as the executor and its options, are set on the machine.

Tokens around runners. Several tokens are easy to confuse:

Token	Prefix	Held by	Purpose	Lifetime
Runner authentication token	glrt-	config.toml of each runner manager	Authenticates the runner when requesting jobs	Until reset, or until the configured expiration (rotated automatically)
Runner registration token (legacy)	GR1348941	Admins, group Owners, project Maintainers	Created runner records implicitly at registration time	Until reset
Job token (CI_JOB_TOKEN)	glcbt-	One running job	Lets the job call selected GitLab APIs (section 5.1)	Expires when the job finishes

The legacy registration token workflow (gitlab-runner register --registration-token ...) is deprecated. Anyone with the shared registration token could create runners and set their own tags, which made it impossible to track who created which runner. It has been disabled by default since GitLab 17.0 and is scheduled for removal. New setups should use authentication tokens only.

Protecting and rotating tokens.

• config.toml contains the token in plain text. Keep it readable only by root (or the service account) and never bake it into a shared VM or container image.
• Administrators can enforce a maximum lifetime for authentication tokens (Admin > Settings > CI/CD > Runners). The runner rotates its token automatically before it expires and records token_obtained_at and token_expires_at in config.toml.
• If a token leaks, reset it immediately. The old token stops working at once, and every runner manager using it must be updated:

# Reset the authentication token of runner 42 and print the new one
curl --silent --request POST --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.example.com/api/v4/runners/42/reset_authentication_token" \
  | jq -r '.token'

To remove a runner from a machine cleanly, use gitlab-runner unregister --name docker-linux-01. To prune config.toml entries whose runner no longer exists in GitLab, run gitlab-runner verify --delete.

Key takeaways

• Create the runner in GitLab first (scope, tags, protection), then register the machine with the glrt- token it returns.
• Tags, protection, locking and timeouts are server-side settings and cannot be set by register when using an authentication token.
• A runner token grants access to every job the runner picks up, including secrets: protect config.toml and reset leaked tokens.
• Legacy registration tokens are deprecated and disabled by default. Do not build new automation on them.

Practice

1. (Beginner) Create a project runner in the UI with the tag local-docker, register it on your machine with the Docker executor, and run a job that targets it with that tag.
2. (Intermediate) Write a provisioning script that creates a group runner through the API, stores the token in your secret manager (or a file with mode 0600) and registers the runner non-interactively. Then reset the token through the API and update the runner without re-creating it.
3. Interview question: "Why did GitLab replace registration tokens with runner authentication tokens created in the UI?" Hint: think about who could create runners, who controlled their tags and protection, and traceability.

Runner Managers

Why it matters. One build machine is a single point of failure. Teams run the same runner configuration on several hosts for redundancy, in autoscaling groups, or as several Kubernetes replicas. Without a way to group them, each host would appear as a separate runner with duplicated settings, and runner lists would fill up with entries for machines that no longer exist.

What it is. GitLab separates two concepts:

• A runner is the configuration record in GitLab: its token, scope, tags, protection and description.
• A runner manager is a gitlab-runner process that uses that runner's token to request jobs.

Several managers can share one runner. Each manager is identified by a system ID, which it creates on first start and stores in a .runner_system_id file next to config.toml. IDs starting with s_ are stable. IDs starting with r_ were generated randomly because the file could not be persisted, which is typical for containers without a persistent volume, and they create a new manager on every restart.

Runner #42 "docker-linux-amd64"   tags: docker, linux, amd64   token: glrt-AbC...
  ├── manager s_1a2b3c4d5e6f   host build-01   v19.4.0   last contact: 4 s ago
  ├── manager s_7a8b9c0d1e2f   host build-02   v19.4.0   last contact: 2 s ago
  └── manager s_3f4e5d6c7b8a   host build-03   v19.3.1   last contact: 9 days ago (stale)

The runner's detail page lists every manager with its system ID, version, platform, IP address and last contact. The job log names both the runner and the manager that ran the job:

Running with gitlab-runner 19.4.0 (abcdef12)
  on docker-linux-amd64 AbC1234x, system ID: s_1a2b3c4d5e6f

Adding a manager is simply registering (or copying the same [[runners]] entry) with the same token on another host:

# On build-02: same token as build-01, so it becomes another manager of runner #42
sudo gitlab-runner register --non-interactive \
  --url "https://gitlab.example.com" \
  --token "$RUNNER_TOKEN" \
  --executor "docker" --docker-image "alpine:3.20" \
  --name "docker-linux-01"

cat /etc/gitlab-runner/.runner_system_id    # s_7a8b9c0d1e2f: keep this file on a persistent volume

Aspect	Shared by all managers (set in GitLab)	Per manager (set in each config.toml)
Tags, scope, protected, locked, paused	Yes
Executor, images, volumes, resources		Yes
concurrent, limit		Yes
Version		Yes (can differ during upgrades)

Because local settings are per manager, two managers of the same runner can behave differently if their config.toml files drift. Generate the file from one template (configuration management, Helm values or a VM image) so all managers stay identical. Pausing a runner in GitLab pauses all of its managers. To drain a single manager, stop that process gracefully instead (section 6.4).

Key takeaways

• A runner is the GitLab-side record. A runner manager is a process using its token. One runner can have many managers.
• Managers are identified by a system ID stored in .runner_system_id. Persist that file to avoid creating a new manager on every restart.
• Tags and protection are shared by all managers. Executor settings, concurrency and version are per manager.
• Keep managers' config.toml identical through automation, and drain individual managers by stopping them gracefully.

Practice

1. (Beginner) Register the same runner token on two machines (or two containers with separate config volumes). Run ten short parallel jobs and check in the job logs which system ID ran each job.
2. (Intermediate) Start a runner in a container without a persistent volume for /etc/gitlab-runner, restart it three times, and look at the runner's manager list. Fix the setup so the manager list stays stable.
3. Interview question: "Your runner list shows hundreds of managers with r_ system IDs that were last seen weeks ago. What is the likely cause, and how do you fix it?" Hint: where the system ID is stored, and what happens when that location is ephemeral.

<a id="62-executors"></a>

6.2 Executors

The runner process does not run job scripts by itself. It delegates each job to an executor, which decides where the script runs (host, container, pod, virtual machine) and how isolated it is. Each [[runners]] entry in config.toml uses exactly one executor. The table below gives an overview, and the topics that follow explain each executor in detail.

Executor	Where the job runs	Isolation between jobs	image: honored	Elastic capacity	Typical use
Shell	Directly on the runner host	None	No	No	Native toolchains, special hardware
Docker	A container on one Docker host	Container	Yes	No (one host)	Default choice for Linux builds
Kubernetes	A pod in a cluster	Pod	Yes	Through cluster autoscaling	Organizations that already run Kubernetes
Docker Autoscaler	A container on autoscaled cloud VMs	Container, optionally one VM per job	Yes	Yes	Elastic Docker capacity on AWS, Google Cloud or Azure
Instance	Directly on autoscaled cloud VMs	One VM per job (if configured)	Only with nested VM isolation	Yes	Jobs that need a whole machine
VirtualBox, Parallels	A clone of a VM on one host	VM	No	No	Windows or macOS builds on your own hardware
SSH	A remote host over SSH	None	No	No	Legacy targets
Custom	Whatever your scripts provision	Your choice	Your choice	Your choice	Platforms with no built-in executor

Shell Executor

Why it matters. Some jobs cannot run in a container: an iOS build needs Xcode on macOS, a firmware job must flash a board attached by USB, a Windows build needs a licensed toolchain installed on the machine. The shell executor is the simplest way to run such jobs, and also the easiest way to create an unsafe runner.

What it is. The shell executor runs the job script directly on the runner host, as the user the runner service runs as (on Linux packages, the gitlab-runner user). It uses whatever tools are installed on the machine. The image and services keywords are ignored. Supported shells are bash and sh on Unix, and pwsh and powershell on Windows.

Runner host (macOS, user gitlab-runner)
 ├── gitlab-runner service
 └── ~/builds/<short-token>/<concurrency-id>/acme/ios-app   ← working directory, persists between jobs
       job 1: git fetch, xcodebuild ...                       same user, same file system, same keychain
       job 2: git fetch, xcodebuild ...                       sees everything job 1 left behind

[[runners]]
  name = "macos-xcode-01"
  url = "https://gitlab.example.com"
  token = "glrt-..."                      # written by gitlab-runner register
  executor = "shell"
  shell = "bash"
  builds_dir = "/Users/gitlab-runner/builds"
  environment = ["LANG=en_US.UTF-8", "DEVELOPER_DIR=/Applications/Xcode-16.app/Contents/Developer"]

ios-build:
  tags: [macos, xcode16]                  # route to the shell runner with the toolchain
  variables:
    GIT_STRATEGY: fetch                   # reuse the existing checkout: fast on persistent hosts
    GIT_CLEAN_FLAGS: -ffdx                # but remove untracked files from earlier jobs (the default)
  script:
    - xcodebuild -scheme App -configuration Release build

Advantages	Disadvantages
No container runtime needed. Works on any OS the runner supports	No isolation: jobs share the user, files, installed tools and credentials
Direct access to hardware, GUI sessions, keychains and licenses	Tool versions drift with manual updates. "Works on runner A only" bugs
Fast: no image pull, persistent working directories	State left by one job affects the next

Security. Any job on a shell runner can read the working directories of other projects that run there, install software for later jobs, read files the gitlab-runner user can read, and in some setups escalate privileges. Use shell runners only as project runners (or narrowly scoped group runners) for trusted code, never as instance runners. If a job needs Docker, do not add the gitlab-runner user to the docker group on a shared host: that is equivalent to giving every job root on the machine.

Key takeaways

• The shell executor runs scripts directly on the host as the runner's user and ignores image and services.
• It is the right choice for native toolchains and attached hardware, typically on macOS, Windows or embedded test benches.
• It provides no isolation between jobs: restrict it to trusted projects and keep it off shared pools.

Practice

1. (Beginner) Register a shell runner on your machine and run a job that prints whoami, pwd and the installed versions of git and your language runtime.
2. (Intermediate) Run two jobs on the same shell runner: the first writes a file outside the project directory (for example in $HOME), the second reads it. Then describe two configuration changes that limit this kind of leakage.
3. Interview question: "A team wants a shell runner as an instance runner because 'Docker is too slow'. What do you advise?" Hint: compare isolation between projects with the actual sources of slowness, and consider caching and image choice.

Docker Executor

Why it matters. Most build problems come from environment differences. The Docker executor gives every job a fresh container from the image it asks for, so the toolchain is declared in .gitlab-ci.yml, identical on every runner, and thrown away afterwards. It is the default choice for Linux runners.

What it is. The runner talks to a Docker Engine (local socket or remote TCP) and, for each job, creates several containers on a per-job network:

Docker host
 ├── gitlab-runner                     (service or container; talks to the Docker socket)
 └── per-job network
       ├── helper container      image: gitlab-runner-helper:<arch>-<version>
       │     git fetch, restore cache, download artifacts, upload artifacts
       ├── build container       image: from the job's "image:" (or the runner default)
       │     before_script + script + after_script
       └── service containers    images: from "services:" (postgres, docker:dind, ...)
       shared volumes: /builds (the checkout), /cache (local cache)

The helper image contains git, the artifact and cache tools, and is pulled automatically in the version matching the runner. This is why job images do not need git (section 2.2). The build container runs the script, and service containers are reachable by their hostname or alias (chapter 7).

[[runners]]
  name = "docker-linux-01"
  url = "https://gitlab.example.com"
  token = "glrt-..."
  executor = "docker"
  [runners.docker]
    image = "alpine:3.20"                                    # default when a job has no image:
    pull_policy = ["always", "if-not-present"]               # try always, fall back to local copy if the registry is down
    allowed_pull_policies = ["always", "if-not-present"]     # what jobs may request with image:pull_policy
    allowed_images = ["registry.example.com/*", "docker.io/library/*"]   # restrict job images
    privileged = false                                       # true only for Docker-in-Docker, and only on trusted runners
    volumes = ["/cache"]                                     # persistent local cache volume
    memory = "4g"                                            # per-container limits avoid noisy neighbors
    cpus = "2"
    shm_size = 268435456                                     # 256 MiB /dev/shm (browsers in e2e tests need more than the default)

Option	Effect	Default
image	Image for jobs without image:	Required at registration
pull_policy	always, if-not-present, never, or a list tried in order	always
allowed_images, allowed_services	Patterns of images that jobs may use	Any
privileged	Runs containers with full host capabilities	false
volumes	Extra volumes or bind mounts for every job container	["/cache"] (written at registration)
memory, cpus	Resource limits per job container	Unlimited
disable_cache	Disables the local /cache volume (distributed cache still works)	false
network_mode	Network for job containers	Per-job network

Privileged mode and the Docker socket. Building images with Docker inside a job needs either privileged = true (Docker-in-Docker) or mounting /var/run/docker.sock into the job. Both effectively give the job root on the host. Prefer rootless builders such as BuildKit in rootless mode or Buildah, covered in section 7.3, and keep privileged runners for trusted projects only.

Windows. The docker-windows executor runs Windows containers on a Windows host. The job image's Windows version must match the host's (for example servercore:ltsc2022 on Windows Server 2022), and the Linux-only options of [runners.docker] do not apply.

Key takeaways

• The Docker executor runs each job in a fresh container from image:, with a helper container for git, cache and artifacts.
• Services run as sibling containers on a per-job network.
• allowed_images, allowed_pull_policies, memory and cpus let administrators constrain what jobs can do.
• privileged = true or a mounted Docker socket gives jobs root on the host: avoid both on shared runners.

Practice

1. (Beginner) Register a Docker executor runner with alpine:3.20 as the default image. Run one job without image: and one with image: python:3.12-slim, and compare cat /etc/os-release in both.
2. (Intermediate) While a job with a postgres service runs, run docker ps and docker network ls on the host. Identify the helper, build and service containers and the per-job network. Then restrict the runner with allowed_images and show that a job using a disallowed image fails.
3. Interview question: "Why is enabling privileged = true on an instance-wide Docker runner dangerous, even if all users are employees?" Hint: consider what a privileged container can do to the host and to other jobs' containers and secrets.

Kubernetes Executor

Why it matters. Organizations that already run Kubernetes have a scheduler, autoscaling, monitoring and access control in place. The Kubernetes executor turns that cluster into CI capacity: every job becomes a pod, scheduled and scaled like any other workload, with no separate VM fleet to operate.

What it is. The runner manager usually runs inside the cluster, installed with the official Helm chart (or the GitLab Runner Operator on OpenShift). For each job, it creates a pod through the Kubernetes API:

Namespace gitlab-runner
 ├── Deployment gitlab-runner (runner manager pod)   ── polls GitLab, creates and deletes job pods
 └── Pod runner-abc123-project-12345-concurrent-0     (one per job)
       ├── init container  init-permissions          prepares shared volumes
       ├── container       build                     job image: before_script, script, after_script
       ├── container       helper                    git fetch, cache, artifacts
       └── containers      svc-0, svc-1, ...         services: share the pod network, reachable on localhost

Install the runner with Helm and keep its token in a Kubernetes Secret:

helm repo add gitlab https://charts.gitlab.io
kubectl create namespace gitlab-runner
kubectl --namespace gitlab-runner create secret generic gitlab-runner-token \
  --from-literal=runner-registration-token="" \
  --from-literal=runner-token="$RUNNER_TOKEN"        # the glrt- token of a runner created in GitLab
helm install gitlab-runner gitlab/gitlab-runner \
  --namespace gitlab-runner --values values.yaml

# values.yaml
gitlabUrl: https://gitlab.example.com/
concurrent: 20                       # max job pods this runner manager runs at once
rbac:
  create: true                       # lets the manager create pods, secrets and services in its namespace
runners:
  secret: gitlab-runner-token        # Secret created above
  config: |
    [[runners]]
      [runners.kubernetes]
        namespace = "{{.Release.Namespace}}"
        image = "alpine:3.20"                       # default job image
        cpu_request = "500m"                        # requests drive scheduling and node autoscaling
        memory_request = "1Gi"
        memory_limit = "2Gi"
        helper_cpu_request = "100m"
        helper_memory_request = "128Mi"
        cpu_limit_overwrite_max_allowed = "4"       # jobs may raise their CPU limit up to 4 cores
        memory_limit_overwrite_max_allowed = "8Gi"  # and their memory limit up to 8 GiB
        memory_request_overwrite_max_allowed = "8Gi"
        poll_timeout = 600                          # seconds to wait for a pod to start (node scale-up, image pull)
        service_account = "ci-jobs"                 # identity of job pods inside the cluster

Jobs request resources through variables, capped by the *_overwrite_max_allowed settings:

integration-tests:
  image: maven:3.9-eclipse-temurin-21
  variables:
    KUBERNETES_CPU_REQUEST: "2"
    KUBERNETES_MEMORY_REQUEST: "6Gi"
    KUBERNETES_MEMORY_LIMIT: "8Gi"       # above 8Gi would be rejected by the runner
  services:
    - name: postgres:16
      alias: db                          # services are also reachable on localhost in the pod
  script:
    - mvn verify

Option	Purpose
namespace	Namespace for job pods
cpu_request, cpu_limit, memory_request, memory_limit	Defaults for the build container (helper_* and service_* variants exist)
*_overwrite_max_allowed	Upper bound for job-level KUBERNETES_* overrides. Overrides are refused if unset
node_selector, node_tolerations, affinity	Place job pods on dedicated CI nodes
service_account	Kubernetes identity of job pods (for example for cloud workload identity)
privileged	Privileged build containers. Avoid on shared clusters
poll_timeout	How long to wait for the pod to become ready before failing the job

Differences from Docker. Services share the pod's network namespace, so two services cannot listen on the same port. There is no Docker daemon, so building images needs a rootless builder or a privileged Docker-in-Docker service. Pods that stay Pending longer than poll_timeout (no node with enough resources, image pull problems) fail the job with a system failure.

Key takeaways

• The Kubernetes executor creates one pod per job, with build, helper and service containers.
• Install the runner with the Helm chart and keep the glrt- token in a Secret.
• Default requests and limits come from config.toml. Jobs can override them with KUBERNETES_* variables up to the configured maximums.
• Services share the pod network and are reachable on localhost. Pod scheduling delays count against poll_timeout.

Practice

1. (Beginner) Install the runner in a local cluster (kind, minikube or k3d) with the Helm chart and run a job. While it runs, inspect the job pod with kubectl describe pod and identify its containers.
2. (Intermediate) Configure memory_limit_overwrite_max_allowed and run three jobs requesting 1Gi, 4Gi and a value above the maximum. Explain each outcome, then trigger an out-of-memory kill and find the evidence in the job log and in kubectl get events.
3. Interview question: "Your Kubernetes jobs often fail with a pod timeout during business hours only. What would you investigate?" Hint: node autoscaling speed, resource requests versus node sizes, image pull time, and poll_timeout.

Docker Autoscaler Executor

Why it matters. CI load is spiky: quiet at night, overloaded before a release. A fixed Docker host is either idle (wasting money) or saturated (making developers wait). The Docker Autoscaler executor adds and removes cloud VMs to match the job queue while keeping the familiar Docker executor behavior.

What it is. The runner manager drives a cloud instance group (an AWS Auto Scaling group, a Google Cloud managed instance group or an Azure virtual machine scale set) through a fleeting plugin. It connects to each instance over SSH (or WinRM) and runs jobs as Docker containers on that instance, just like the Docker executor does on a single host.

Runner manager (small, always on)
  ├── taskscaler: how many job slots are needed now?
  └── fleeting plugin (aws | googlecloud | azure) ── resize ──> Instance group
                                                                 ├── VM 1: Docker ── job A, job B
                                                                 ├── VM 2: Docker ── job C
                                                                 └── VM 3: idle, ready for the next job

It replaces the Docker Machine executor (docker+machine), which depended on the abandoned Docker Machine project and is deprecated. New autoscaling setups should use Docker Autoscaler.

[[runners]]
  name = "docker-autoscaler-aws"
  url = "https://gitlab.example.com"
  token = "glrt-..."
  executor = "docker-autoscaler"
  [runners.docker]
    image = "alpine:3.20"                 # same [runners.docker] options as the Docker executor
  [runners.autoscaler]
    plugin = "aws"                        # installed with: gitlab-runner fleeting install
    capacity_per_instance = 2             # jobs that may run at once on one VM
    max_use_count = 20                    # replace a VM after 20 jobs (limits state build-up)
    max_instances = 30                    # hard cap on the group size
    [runners.autoscaler.plugin_config]
      name = "ci-runner-asg"              # name of the Auto Scaling group to control
    [runners.autoscaler.connector_config]
      username = "ubuntu"                 # SSH user on the instances
    [[runners.autoscaler.policy]]
      idle_count = 4                      # keep 4 idle job slots ready
      idle_time = "20m0s"                 # remove idle capacity after 20 minutes

Setting	Effect	Trade-off
capacity_per_instance	Jobs per VM	Higher is cheaper, but jobs share CPU, disk and the Docker daemon
max_use_count	Jobs per VM lifetime	1 gives one fresh VM per job (strong isolation, slower). Higher reuses warm caches and images
max_instances	Upper bound on VMs	Caps cost and protects quotas
idle_count, idle_time	Warm capacity	More idle slots mean faster pickup and higher cost

Setting up the instance group, the plugin and scaling policies is covered in section 6.4.

Key takeaways

• Docker Autoscaler runs jobs in Docker containers on cloud VMs that are created and removed on demand.
• It uses fleeting plugins for AWS, Google Cloud and Azure, and replaces the deprecated Docker Machine executor.
• capacity_per_instance and max_use_count set the balance between cost, speed and isolation.

Practice

1. (Beginner) Explain in your own words what happens, step by step, when 10 jobs arrive at a Docker Autoscaler runner with idle_count = 2, capacity_per_instance = 2 and no running VMs.
2. (Intermediate) Compare two configurations for a fleet running 500 short jobs a day: (a) capacity_per_instance = 4, max_use_count = 50; (b) capacity_per_instance = 1, max_use_count = 1. Discuss isolation, pickup latency, cache warmth and cost.
3. Interview question: "Why did GitLab replace the Docker Machine executor rather than continue maintaining it?" Hint: upstream project status, cloud-provider coupling, and how the new design uses native instance groups.

Instance Executor

Why it matters. Some jobs need a whole machine rather than a container: system tests that load kernel modules, builds that run Docker without a privileged container, tools that expect a full OS, or workloads that must not share a kernel with anyone. The Instance executor gives such jobs autoscaled VMs with the same elasticity as Docker Autoscaler.

What it is. The Instance executor uses the same fleeting and taskscaler machinery, but runs the job script directly on the VM through a shell, like the shell executor, instead of inside a container. With max_use_count = 1, every job gets a fresh VM that is destroyed afterwards. This gives shell-like freedom without shell-executor leakage.

[[runners]]
  name = "instance-aws"
  url = "https://gitlab.example.com"
  token = "glrt-..."
  executor = "instance"
  shell = "bash"                          # the VM image must have bash, git and the tools jobs need
  [runners.autoscaler]
    plugin = "aws"
    capacity_per_instance = 1             # one job per VM
    max_use_count = 1                     # fresh VM for every job, deleted afterwards
    max_instances = 20
    [runners.autoscaler.plugin_config]
      name = "ci-instance-asg"
    [runners.autoscaler.connector_config]
      username = "ec2-user"
    [[runners.autoscaler.policy]]
      idle_count = 2
      idle_time = "15m0s"

kernel-module-test:
  tags: [full-vm]                         # tag of the instance runner
  script:
    - sudo modprobe ./build/mydriver.ko   # possible because the job owns the whole VM
    - ./tests/run-driver-tests.sh

The Instance executor can also run nested virtualization ([runners.autoscaler.vm_isolation]), where each job runs in its own VM inside the instance. Here image: selects the nested VM image. This is useful, for example, for macOS jobs on Apple hardware, where each job gets its own macOS VM.

	Docker Autoscaler	Instance
Job runs in	Container on the VM	Shell directly on the VM
image:	Container image	Ignored (or nested VM image with VM isolation)
Tools come from	Job image	VM image (AMI, machine image)
Typical isolation	Container, optional VM per job	VM per job
Best for	Ordinary builds and tests	Jobs needing root, kernel access, or a full OS

Key takeaways

• The Instance executor runs jobs directly on autoscaled VMs, without a container.
• With max_use_count = 1, each job gets a clean, disposable VM.
• Tools must be baked into the VM image, because image: does not apply (except with nested VM isolation).

Practice

1. (Beginner) List three kinds of jobs from your own projects that would benefit from the Instance executor rather than Docker, and explain why.
2. (Intermediate) Build a VM image (for example with Packer) containing git, bash and Docker Engine, configure an Instance executor that uses it with max_use_count = 1, and run a job that builds a Docker image without a privileged container.
3. Interview question: "Shell executor and Instance executor both run scripts directly on a machine. Why is one considered safe for untrusted code and the other not?" Hint: lifetime of the machine and what the next job inherits.

VirtualBox and Parallels Executors

Why it matters. Teams that build for Windows or macOS on their own hardware need full OS isolation, but may not have a cloud or a Kubernetes cluster. The VirtualBox and Parallels executors give each job a clean VM on a single physical host.

What it is. You prepare a base VM with the OS, build tools, an SSH server, git and gitlab-runner (the runner binary inside the VM handles artifacts and caches). For each job, the executor creates a linked clone of the base VM from a snapshot, starts it, connects over SSH, runs the job, and deletes the clone.

Physical host (Linux or Windows with VirtualBox, or macOS with Parallels)
 ├── gitlab-runner
 ├── base VM "win2022-build" + snapshot "tools-2026-09"     never modified by jobs
 └── per job: linked clone  ── SSH ──> run script  ──> delete clone

[[runners]]
  name = "windows-vbox-01"
  url = "https://gitlab.example.com"
  token = "glrt-..."
  executor = "virtualbox"
  shell = "pwsh"                                      # shell inside the guest
  [runners.ssh]
    user = "builder"
    identity_file = "/home/gitlab-runner/.ssh/id_ed25519"
  [runners.virtualbox]
    base_name = "win2022-build"                       # VM to clone
    base_snapshot = "tools-2026-09"                   # clone from this snapshot (linked clone)
    disable_snapshots = false                         # false: fast linked clones; true: full copy per job

Parallels is configured the same way with a [runners.parallels] section (base_name, template_name, disable_snapshots) and is mainly used for macOS guests on Mac hardware.

Strength	Limitation
Full OS isolation on your own hardware	Capacity bounded by one host's CPU and memory
Clean VM per job from a known snapshot	Slower start than containers (clone and boot)
Works for Windows and macOS builds	No autoscaling. Base images must be maintained by hand
	Licensing: macOS guests require Apple hardware

To update tools, update the base VM, take a new snapshot and change base_snapshot. Jobs never see half-updated images.

Key takeaways

• VirtualBox and Parallels executors clone a base VM per job and run the script over SSH.
• The base VM needs an SSH server, a shell, git and gitlab-runner.
• They provide strong isolation on a single host but no elastic capacity. For cloud capacity, prefer the Instance executor.

Practice

1. (Beginner) Describe the steps to prepare a base VM for the VirtualBox executor and explain why each component (SSH server, git, gitlab-runner) is needed.
2. (Intermediate) Design a process for updating the toolchain in a Parallels base VM used by 15 iOS projects without breaking running jobs, including how to roll back.
3. Interview question: "Your company has three Mac minis for iOS builds. Would you use the shell executor or the Parallels executor, and why?" Hint: isolation between projects, signing keys in keychains, boot time and capacity.

SSH and Custom Executors

Why it matters. Not every environment fits a built-in executor. Some legacy systems can only be reached over SSH, and some organizations run jobs on platforms GitLab has no native support for, such as LXD, Podman on a hardened host, a mainframe partition or an in-house VM orchestrator. These two executors cover the edge cases.

SSH executor. The runner connects to a fixed remote host over SSH and runs the job script there. It is the least featured executor: it supports only Bash scripts, does not support caching, needs gitlab-runner on the remote host to upload artifacts, and provides no isolation, because every job runs on the same remote host and user. Use it only when nothing else can reach a target, and prefer installing a runner on that host instead.

[[runners]]
  name = "legacy-aix-ssh"
  executor = "ssh"
  [runners.ssh]
    host = "aix-build.internal.example.com"
    port = "22"
    user = "builder"
    identity_file = "/home/gitlab-runner/.ssh/id_ed25519"

Custom executor. The Custom executor lets you write the executor yourself as four executables that the runner calls at fixed points:

config_exec   (optional) print JSON to adjust builds_dir, cache_dir, driver name
prepare_exec  create the environment (VM, container, chroot...)
run_exec      called once per stage: run_exec <script-path> <stage-name>
                 stages: prepare_script, get_sources, restore_cache, download_artifacts,
                         step_script, after_script, archive_cache, upload_artifacts_on_success,
                         upload_artifacts_on_failure, cleanup_file_variables
cleanup_exec  destroy the environment, always called, even after failures

Your scripts receive the job's variables with a CUSTOM_ENV_ prefix (for example CUSTOM_ENV_CI_JOB_ID, CUSTOM_ENV_CI_JOB_IMAGE). They report failures with two exit codes provided by the runner: $BUILD_FAILURE_EXIT_CODE means the job's script failed, $SYSTEM_FAILURE_EXIT_CODE means the infrastructure failed (which retry:when: runner_system_failure can target, section 4.3).

A minimal driver that runs each job in its own Podman container:

[[runners]]
  name = "podman-custom"
  url = "https://gitlab.example.com"
  token = "glrt-..."
  executor = "custom"
  builds_dir = "/builds"                          # path inside the job environment
  cache_dir = "/cache"
  [runners.custom]
    prepare_exec = "/opt/ci-driver/prepare.sh"
    run_exec = "/opt/ci-driver/run.sh"
    cleanup_exec = "/opt/ci-driver/cleanup.sh"

#!/usr/bin/env bash
# /opt/ci-driver/prepare.sh: create one container per job
set -eo pipefail
trap 'exit "$SYSTEM_FAILURE_EXIT_CODE"' ERR             # any failure here is an infrastructure failure

CONTAINER="ci-job-$CUSTOM_ENV_CI_JOB_ID"
IMAGE="${CUSTOM_ENV_CI_JOB_IMAGE:-debian:bookworm}"     # honor the job's image: keyword
podman run --detach --name "$CONTAINER" "$IMAGE" sleep infinity
# get_sources needs git; artifacts and cache need the gitlab-runner binary in the environment
podman exec "$CONTAINER" sh -c 'command -v git >/dev/null || (apt-get update && apt-get install -y git curl)'
podman cp /usr/bin/gitlab-runner "$CONTAINER":/usr/bin/gitlab-runner

#!/usr/bin/env bash
# /opt/ci-driver/run.sh: $1 = generated script for this stage, $2 = stage name
CONTAINER="ci-job-$CUSTOM_ENV_CI_JOB_ID"
if ! podman exec --interactive "$CONTAINER" /bin/bash < "$1"; then
  exit "$BUILD_FAILURE_EXIT_CODE"                        # the job failed, not the infrastructure
fi

#!/usr/bin/env bash
# /opt/ci-driver/cleanup.sh: always runs; never leave containers behind
podman rm --force "ci-job-$CUSTOM_ENV_CI_JOB_ID" >/dev/null 2>&1 || true

	SSH	Custom
Effort	Configuration only	You write and maintain the driver
Isolation	None	Whatever your driver provides
Cache and artifacts	No cache. Artifacts only if gitlab-runner is installed on the remote host	Supported if gitlab-runner is available in the environment
Use when	A remote host cannot run a runner itself	No built-in executor fits your platform

Key takeaways

• The SSH executor runs Bash scripts on one remote host. It has no isolation or caching and is a last resort.
• The Custom executor delegates prepare, run (per stage) and cleanup to your own executables.
• Custom drivers read job variables with the CUSTOM_ENV_ prefix and signal build versus system failures with the provided exit codes.
• Cleanup must be idempotent and must never fail the job.

Practice

1. (Beginner) Implement the Podman driver above (or the same idea with Docker) and run a job with image: python:3.12-slim. Print $2 in run.sh to see each stage name in the job log.
2. (Intermediate) Extend the driver with config_exec that reports a driver name and version, and make prepare.sh fail on purpose. Check that the job fails as a system failure and is retried by retry: { max: 1, when: [runner_system_failure] }.
3. Interview question: "When would you write a Custom executor instead of using the Kubernetes or Instance executor?" Hint: think about platforms without built-in support, compliance-mandated sandboxes, and the maintenance cost you take on.

<a id="63-runner-configuration"></a>

6.3 Runner Configuration

A runner's behavior is split between settings stored in GitLab and the local config.toml file on each runner manager. This subchapter covers the structure of config.toml, how concurrency is controlled, how tags and protection decide which jobs a runner accepts, and how administrators inject commands into every job.

config.toml Structure

Why it matters. Almost every runner problem, from "only one job runs at a time" to "the job cannot reach the cache", is fixed in config.toml. Knowing its layout, and which settings live there rather than in GitLab, saves hours of guessing.

What it is. config.toml is a TOML file read by the runner process. Its location depends on how the runner runs:

Installation	Location
Linux package, running as root (the usual service)	/etc/gitlab-runner/config.toml
Running as a non-root user	~/.gitlab-runner/config.toml
Windows	Next to gitlab-runner.exe, for example C:\GitLab-Runner\config.toml
Docker container	/etc/gitlab-runner/config.toml inside the container (mount a volume)
Helm chart	Generated from values.yaml (runners.config)

The file has three levels: global settings for the whole process, an optional [session_server], and one or more [[runners]] entries (double brackets mean "array of tables": each entry is one runner the process serves). Each entry has executor- and feature-specific subsections.

# ---------- Global: the whole gitlab-runner process ----------
concurrent = 8                     # max jobs running at once across all [[runners]] entries
check_interval = 3                 # seconds between job requests per entry (long polling makes this rarely visible)
log_level = "info"                 # debug | info | warn | error
log_format = "json"                # runner (default) | text | json; json suits log shipping
shutdown_timeout = 7200            # seconds to wait for running jobs on graceful shutdown
listen_address = ":9252"           # Prometheus metrics endpoint (section 6.4)

[session_server]                   # needed only for interactive web terminals
  listen_address = "[::]:8093"
  advertise_address = "runner-01.example.com:8093"
  session_timeout = 1800

# ---------- One runner entry ----------
[[runners]]
  name = "docker-linux-01"         # local label
  url = "https://gitlab.example.com"
  id = 42                          # runner ID in GitLab
  token = "glrt-..."               # authentication token (keep this file 0600)
  token_obtained_at = 2026-09-01T08:00:00Z
  token_expires_at = 0001-01-01T00:00:00Z   # zero value: no expiration configured
  executor = "docker"
  limit = 6                        # max jobs for this entry (0 = no per-entry limit)
  request_concurrency = 2          # parallel job requests this entry may have open
  output_limit = 16384             # max job log size in KB (default 4096)
  environment = ["HTTPS_PROXY=http://proxy.internal:3128", "NO_PROXY=.internal,localhost"]
  pre_get_sources_script = "git config --global http.sslCAInfo /etc/ssl/certs/internal-ca.pem"

  [runners.feature_flags]
    FF_USE_FASTZIP = true          # faster artifact and cache compression

  [runners.cache]                  # distributed cache shared by all managers (chapter 8)
    Type = "s3"
    Shared = true                  # share cache between runners, not only between jobs of one runner
    [runners.cache.s3]
      ServerAddress = "s3.amazonaws.com"
      BucketName = "acme-runner-cache"
      BucketLocation = "eu-central-1"

  [runners.docker]                 # executor-specific options (section 6.2)
    image = "alpine:3.20"
    privileged = false
    volumes = ["/cache"]

Server-side versus local settings. Some settings live in GitLab and are shared by all runner managers, others live only in config.toml:

Set in GitLab (UI or PUT /runners/:id)	Set in config.toml
Description, tags, run untagged jobs	Executor and its options ([runners.docker], [runners.kubernetes]...)
Protected, locked, paused	concurrent, limit, request_concurrency
Maximum job timeout	Cache backend, proxy and environment variables
Scope (instance, group, project) and project assignments	Hooks, feature flags, log size, metrics endpoint

Changing the file. The runner watches config.toml and reloads most changes automatically within a few seconds, without interrupting running jobs. A few global settings, such as listen_address, need a restart. Useful commands:

sudo gitlab-runner list                  # entries in config.toml with their URL and executor
sudo gitlab-runner verify                # check each entry's token against GitLab
sudo gitlab-runner verify --delete       # remove entries whose runner no longer exists
sudo gitlab-runner restart               # restart the service (running jobs are interrupted unless drained)

Keep config.toml under configuration management (Ansible, Chef, a Helm chart, a VM image build) with the token injected from a secret store. Hand-edited files on many machines drift apart, and managers of the same runner start behaving differently (section 6.1).

Key takeaways

• config.toml has global settings, an optional [session_server] and one [[runners]] entry per runner served by the process.
• Executor, cache, concurrency, hooks and environment live in config.toml. Tags, protection, locking and timeouts live in GitLab.
• Most edits are reloaded automatically. Manage the file as code and keep it readable only by the runner's service account.

Practice

1. (Beginner) Open the config.toml of a runner you registered, label each line as global, entry-level or executor-level, and look up two settings you have not seen before in the runner documentation.
2. (Intermediate) Configure an S3-compatible distributed cache (for example MinIO) for two runner managers of the same runner. Prove that a cache written by a job on one manager is restored by a job on the other.
3. Interview question: "A developer asks you to add the tag gpu to a runner by editing its config.toml. Why does that not work with the current registration workflow, and where do you change it?" Hint: which side owns tags when using runner authentication tokens.

Concurrency Settings

Why it matters. A newly registered runner has concurrent = 1, so it runs one job at a time no matter how powerful the machine is. Conversely, a runner allowed to run too many jobs overloads its host and makes every job slow. Three settings control parallelism, and they interact in ways that surprise many teams.

What it is.

Setting	Scope	Meaning	Default
concurrent	Global	Maximum jobs running at once across all [[runners]] entries of the process	1
limit	Per [[runners]] entry	Maximum jobs for this entry. For autoscaling executors, also see max_instances	0 (no per-entry limit)
request_concurrency	Per [[runners]] entry	How many job requests this entry may have open at once	1

The effective limit is the smaller of the two caps. Think of concurrent as the number of seats in a restaurant, and limit as the number of seats reserved for each party.

concurrent = 10                  # the whole process: at most 10 jobs

[[runners]]
  name = "docker-general"
  executor = "docker"
  limit = 8                      # at most 8 of the 10 slots
  request_concurrency = 2        # ask for up to 2 jobs at once: fills free slots faster under load

[[runners]]
  name = "docker-heavy"          # second runner record, e.g. with the tag "heavy"
  executor = "docker"
  limit = 2                      # heavy jobs never take more than 2 slots

Queue: 12 general jobs, 3 heavy jobs
  docker-general  runs 8  (its limit)
  docker-heavy    runs 2  (its limit)
  total           10      (concurrent): the remaining 5 jobs stay pending

If concurrent were 6:  general and heavy compete for 6 slots, limits never reached

Sizing concurrency. Choose concurrent from the host's resources and the typical job, not from the queue length:

Host: 16 vCPU, 64 GB RAM, fast SSD
Typical job: ~2 vCPU, ~6 GB RAM peak
  CPU bound:    16 / 2  = 8
  Memory bound: 64 / 6 ≈ 10  (leave headroom for Docker, page cache, the runner)
  => concurrent = 8, and set [runners.docker] cpus = "2", memory = "6g" so one job cannot starve the others

On autoscaling executors, capacity is mostly controlled by the autoscaler (max_instances times capacity_per_instance, section 6.4), while concurrent and limit must be at least that large or they become the real bottleneck. On Kubernetes, concurrent caps the number of job pods per runner manager, and cluster resources cap the rest.

request_concurrency matters when many jobs are queued: with the default of 1, an entry fills free slots one request at a time. Raising it to a small number (2 to 4) fills slots faster. Very high values only add load on GitLab. Increase it if the metric gitlab_runner_request_concurrency_exceeded_total (section 6.4) keeps growing.

Key takeaways

• concurrent (global) and limit (per entry) both cap running jobs. The smaller one wins.
• The default concurrent = 1 is the most common reason a powerful runner runs jobs one by one.
• Size concurrency from CPU and memory per job, and set per-job resource limits to avoid noisy neighbors.
• On autoscaling executors, make sure concurrent and limit do not cap below the autoscaler's maximum capacity.

Practice

1. (Beginner) On a runner with concurrent = 1, start a pipeline with five parallel jobs and observe them run one by one. Raise concurrent to 5 without restarting the runner and observe the next pipeline.
2. (Intermediate) Configure two [[runners]] entries on one host (one for general jobs, one for jobs tagged heavy) so heavy jobs never use more than 2 slots and the host never runs more than 6 jobs in total. Prove it with a pipeline of 10 general and 5 heavy jobs.
3. Interview question: "Jobs wait in pending for minutes although your runner host's CPU is at 20%. Walk through what you check." Hint: concurrent versus limit, request_concurrency, tag matching, and whether the runner is actually online.

Runner Tags and Untagged Jobs

Why it matters. Section 2.2 showed tags from the job's side: a job lists the capabilities it needs. This topic covers the runner's side: which tags to give runners, whether they should accept untagged jobs, and how bad choices send jobs to the wrong machine, or nowhere.

What it is. A runner's tags and its "Run untagged jobs" flag are set on the runner record in GitLab (at creation, in the runner's Edit page, or with the API). The matching rule from section 2.2 still applies: a runner can take a job only if it has all of the job's tags.

Runner configuration	Takes tagged jobs	Takes untagged jobs
Tags [docker, linux], run untagged: on	Jobs whose tags are a subset of [docker, linux]	Yes
Tags [docker, linux], run untagged: off	Jobs whose tags are a subset of [docker, linux]	No
No tags, run untagged: on	None (any job with a tag needs a runner having it)	Yes

A runner created without tags runs untagged jobs. A runner created with tags must have "Run untagged jobs" turned on explicitly if it should also take them.

# Change tags and untagged behavior of runner 42 (server-side settings)
curl --silent --request PUT --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.example.com/api/v4/runners/42" \
  --data "tag_list=docker,linux,amd64" \
  --data "run_untagged=true"

Designing tags. Tags work best when they describe capabilities, not machines. A job asking for build-server-07 breaks when that server is replaced. A job asking for [linux, arm64, docker] keeps working on any runner that offers those capabilities.

Tag style	Examples	Use for
Platform and architecture	linux, windows, macos, amd64, arm64	Choosing an OS and CPU
Executor or tooling	docker, kubernetes, xcode16, gpu	Choosing an environment
Size class	small, large, high-mem	Cost and performance (section 13.2)
Purpose or trust zone	prod-deploy, pci	Restricted runners, combined with protection (next topic)
Machine identity	build-server-07	Avoid: couples jobs to one machine

Untagged jobs as a policy decision. Untagged jobs go to any runner that accepts them. Decide deliberately which runners those are:

• A general-purpose pool should usually accept untagged jobs, so simple projects work without knowing any tags.
• Specialized runners (GPU, macOS, deployment, large machines) should not accept untagged jobs, or every ordinary lint job may land on your most expensive machine.
• On GitLab.com, if your project has its own runners and hosted instance runners enabled, untagged jobs may run on either. Disable instance runners for the project, or tag your jobs, if consumption of compute minutes matters.

default:
  tags: [docker, linux]           # every job targets the general pool unless it says otherwise

unit-tests:
  script: [make test]

train-model:
  tags: [linux, gpu]              # replaces the default list: only GPU runners
  script: [python train.py]

deploy-prod:
  tags: [prod-deploy]             # restricted runner, see "Protected Runners"
  script: [./deploy.sh production]

Key takeaways

• Runner tags and "Run untagged jobs" are server-side settings on the runner record.
• Name tags after capabilities (OS, architecture, tooling, size, trust zone), not after machines.
• Only general-purpose runners should accept untagged jobs. Specialized and expensive runners should not.
• Mixed fleets of own and hosted runners can route untagged jobs to either. Tag jobs or disable instance runners when that matters.

Practice

1. (Beginner) Create two runners: one with tags [docker, linux] that accepts untagged jobs, one with [docker, linux, large] that does not. Predict and then verify where an untagged job, a [docker] job and a [large] job run.
2. (Intermediate) Audit all runners available to a group using the API (GET /groups/:id/runners, then GET /runners/:id for tag_list and run_untagged). Produce a table of which runners accept untagged jobs and flag any specialized runner that does.
3. Interview question: "Your cloud bill doubled after a team added a 32-core runner. Pipelines did not change. What is a likely cause?" Hint: which jobs the new runner is allowed to take, and what it means for an untagged job.

Protected Runners

Why it matters. Deployment runners often hold the most sensitive things in CI: cloud credentials in their IAM role, network access to production, signing keys. If any branch could send a job to them, any developer (or any merge request from a fork) could run arbitrary code with those privileges. Protected runners close that gap.

What it is. A runner marked Protected (API: access_level=ref_protected) only takes jobs from pipelines that run for protected branches or protected tags. Jobs from other refs never reach it: if no other runner matches, they stay pending.

Pipeline for                       Job tags         Protected runner "prod-deploy"
───────────────────────────────    ─────────────    ───────────────────────────────
main (protected)                   [prod-deploy]    takes the job
v2.4.0 (protected tag)             [prod-deploy]    takes the job
feature/login (unprotected)        [prod-deploy]    refuses: job stays pending (stuck)
MR pipeline from a fork            [prod-deploy]    never sees it: runs in the fork's project

# Make runner 77 protected, untagged-proof and locked to its project
curl --silent --request PUT --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.example.com/api/v4/runners/77" \
  --data "access_level=ref_protected" \
  --data "run_untagged=false" \
  --data "locked=true" \
  --data "tag_list=prod-deploy"

Protection is one layer. Combine it with the others from earlier chapters so that each covers a different gap:

Control	What it prevents
Protected branches and tags (section 11.4)	Unreviewed code becoming a protected ref
Protected runner	Code from unprotected refs running on the privileged machine
Dedicated tag, untagged jobs off	Ordinary jobs from protected branches landing on the deploy runner by accident
Locked project runner	Other projects enabling the runner for themselves
Protected variables and environments (chapter 5, chapter 12)	Secrets and deployment approvals leaking to unprotected pipelines

Protection controls which refs can use the runner, not what the code does. Anyone who can push or merge to a protected branch can still run arbitrary code on the runner, so the protected branch rules (who can push, who can merge, required approvals) are part of the runner's security. Section 11.4 builds a complete least-privilege runner design on these pieces.

Key takeaways

• Protected runners take jobs only from pipelines for protected branches and tags.
• Use them for every runner with deployment credentials or privileged network access.
• Combine protection with a dedicated tag, untagged jobs off and locking.
• Protection is only as strong as the rules on who can push and merge to protected refs.

Practice

1. (Beginner) Mark a project runner as protected and give it a unique tag. Run a job with that tag from main (protected) and from a feature branch, and compare the outcomes.
2. (Intermediate) Design the full set of controls for a runner that deploys to production through an IAM role, including branch protection rules, runner settings, tags, and environment protection. For each control, describe the attack it prevents.
3. Interview question: "A runner is protected, so only main can use it. A developer opens a merge request that changes .gitlab-ci.yml to print the runner's cloud credentials. Under what conditions does this succeed?" Hint: consider who can merge to main, approval rules, and code owners for CI configuration.

Pre- and Post-Build Hooks

Why it matters. Platform teams often need something to happen in every job on their runners: trust the company's internal certificate authority, configure a proxy or a package mirror, rewrite git URLs, record audit information. Asking every project to add these lines to before_script does not scale and is easy to forget. Runner hooks inject commands centrally.

What it is. Four config.toml settings in a [[runners]] entry add commands at fixed points of every job. They run in the job's own environment with the job's privileges, and a failing hook fails the job.

Setting	Runs	Typical use
pre_get_sources_script	Before the repository is fetched	Git configuration, CA certificates for git, cleaning the workspace
post_get_sources_script	After fetch and checkout	Checking out large files, verifying the commit
pre_build_script	Before before_script	Installing CA certificates, configuring mirrors and proxies, logging
post_build_script	After script, before after_script	Collecting metrics, cleanup

Together with the job-level hooks:pre_get_sources_script keyword (section 2.4), the order inside a job is:

get sources step     (helper container for Docker and Kubernetes executors)
  1. pre_get_sources_script               config.toml (administrator)
  2. hooks:pre_get_sources_script         .gitlab-ci.yml (developer)
  3. git fetch + checkout
  4. post_get_sources_script              config.toml
restore cache, download artifacts
step script          (build container)
  5. pre_build_script                     config.toml
  6. before_script + script               .gitlab-ci.yml
  7. post_build_script                    config.toml
after_script                              .gitlab-ci.yml

For the Docker and Kubernetes executors, note where each hook runs: the get-sources hooks run in the helper container, the build hooks in the job's image. A hook that installs a certificate with update-ca-certificates works only if the job image has that tool, so write build hooks defensively.

[[runners]]
  name = "docker-internal"
  executor = "docker"
  # Trust the internal CA for git operations (runs in the helper image, which has git)
  pre_get_sources_script = '''
    git config --global http.sslCAInfo /certs/internal-ca.pem
    git config --global url."https://mirror.internal/github/".insteadOf "https://github.com/"
  '''
  # Add the internal CA to the job image if it is Debian- or Alpine-based; never fail other images
  pre_build_script = '''
    if command -v update-ca-certificates >/dev/null 2>&1; then
      cp /certs/internal-ca.pem /usr/local/share/ca-certificates/internal-ca.crt
      update-ca-certificates >/dev/null 2>&1 || true
    fi
    echo "Runner $CI_RUNNER_SHORT_TOKEN, job $CI_JOB_ID, project $CI_PROJECT_PATH, ref $CI_COMMIT_REF_NAME"
  '''
  environment = ["PIP_INDEX_URL=https://pypi.mirror.internal/simple", "npm_config_registry=https://npm.mirror.internal/"]
  [runners.docker]
    image = "alpine:3.20"
    volumes = ["/cache", "/etc/gitlab-runner/certs:/certs:ro"]   # mount the CA into every container

Hooks are not a security control. They run with the job's privileges, so job code can undo whatever a hook did, and a hook cannot stop a malicious script. Use them for convenience and consistency. Enforce policy with runner settings, protection, network controls and pipeline execution policies (chapter 11).

	Runner hooks (config.toml)	default: before_script / hooks (YAML)
Who controls it	Runner administrators	Project developers
Applies to	Every job on that runner, all projects	Jobs of that project (or that include the template)
Visible to developers	Only in job logs	In .gitlab-ci.yml
Best for	Environment-specific setup (CA, proxy, mirrors)	Project-specific setup

Key takeaways

• pre_get_sources_script, post_get_sources_script, pre_build_script and post_build_script inject commands into every job on a runner.
• Get-sources hooks run in the helper container, build hooks in the job image. Write them so they work, or skip safely, in any image.
• Use environment in config.toml for simple variables such as proxies and mirrors.
• Hooks run with the job's privileges and are not a security boundary.

Practice

1. (Beginner) Add a pre_build_script that prints the runner, job and project in a single line, and verify it appears in the logs of jobs from two different projects.
2. (Intermediate) Your runners sit behind a TLS-intercepting proxy with an internal CA. Configure hooks, environment and volumes so that git fetch, pip install and npm ci work in jobs using python:3.12-slim and node:20-alpine images, without changing any project's .gitlab-ci.yml.
3. Interview question: "Security asks you to block curl uploads to the internet from CI jobs by adding a check to pre_build_script. Is that a good control?" Hint: who controls the rest of the job's commands, and which layer can enforce egress rules.

<a id="64-autoscaling-and-fleet-management"></a>

6.4 Autoscaling and Fleet Management

One runner on one machine is easy to run. Dozens of runner managers serving hundreds of projects need elastic capacity, visibility and a maintenance routine. This subchapter covers cloud autoscaling with fleeting, scaling on Kubernetes, the fleet dashboard, Prometheus monitoring and keeping runners up to date.

Fleeting Plugins and Cloud Autoscaling

Why it matters. Paying for peak capacity around the clock is expensive, and sizing for the average makes developers wait at peak times. Autoscaling adds VMs when jobs queue up and removes them when they are idle. Done well, it gives short queue times at a fraction of the cost of a static fleet.

What it is. Autoscaling for the Docker Autoscaler and Instance executors (section 6.2) is built from three parts:

GitLab ── jobs ──> runner manager
                    ├── taskscaler      decides how many job slots are needed:
                    │                   running jobs + idle target from the active policy
                    ├── fleeting plugin  translates "N instances" into cloud API calls
                    │     aws ─────────> EC2 Auto Scaling group      (SetDesiredCapacity, terminate)
                    │     googlecloud ─> Compute Engine instance group
                    │     azure ───────> Virtual machine scale set
                    └── connector        SSH or WinRM into each instance to run Docker or the job shell

• The cloud provider only hosts an instance group. Its own scaling rules must be off: the runner sets the desired size itself.
• The fleeting plugin is a small separate binary, distributed as an OCI image and installed with gitlab-runner fleeting install, which downloads every plugin referenced in config.toml.
• Taskscaler keeps track of slots: each instance offers capacity_per_instance job slots and is retired after max_use_count jobs.

Preparing AWS (example).

1. Build an AMI with Docker Engine (for Docker Autoscaler) or your toolchain (for Instance), for example with Packer.
2. Create a launch template from it, and an Auto Scaling group with minimum and desired capacity 0, a maximum such as 30, and instance scale-in protection enabled so AWS never terminates instances that are running jobs.
3. Give the runner manager an IAM role that can describe and resize the group, terminate its instances, and connect to them (for example autoscaling:SetDesiredCapacity, autoscaling:TerminateInstanceInAutoScalingGroup, autoscaling:DescribeAutoScalingGroups, ec2:DescribeInstances, ec2-instance-connect:SendSSHPublicKey).
4. Configure the runner and install the plugin.

concurrent = 60                            # must not cap below max_instances x capacity_per_instance

[[runners]]
  name = "docker-autoscaler-aws"
  url = "https://gitlab.example.com"
  token = "glrt-..."
  executor = "docker-autoscaler"
  limit = 60
  [runners.docker]
    image = "alpine:3.20"
  [runners.cache]                          # instances are short-lived: use a distributed cache
    Type = "s3"
    Shared = true
    [runners.cache.s3]
      BucketName = "acme-runner-cache"
      BucketLocation = "eu-central-1"
  [runners.autoscaler]
    plugin = "aws"                         # short for the official AWS fleeting plugin image
    capacity_per_instance = 2
    max_use_count = 40
    max_instances = 30
    [runners.autoscaler.plugin_config]
      name = "ci-docker-asg"               # Auto Scaling group name
    [runners.autoscaler.connector_config]
      username = "ubuntu"
      use_external_addr = false            # connect over private IPs inside the VPC

    # Policies: when several match, the last matching one applies
    [[runners.autoscaler.policy]]          # baseline: nights and weekends
      idle_count = 0
      idle_time = "10m0s"

    [[runners.autoscaler.policy]]          # business hours: keep warm capacity
      periods = ["* 8-18 * * mon-fri"]     # cron syntax
      timezone = "Europe/Rome"
      idle_count = 6                       # idle job slots, not instances: 6 slots = 3 VMs here
      idle_time = "30m0s"
      scale_factor = 0.5                   # also keep idle slots equal to 50% of slots in use...
      scale_factor_limit = 10              # ...but at most 10 extra

sudo gitlab-runner fleeting install        # downloads the plugins referenced in config.toml
sudo gitlab-runner restart

Cost and speed levers.

Lever	Lower cost	Faster pickup
idle_count	0 outside working hours	Higher during peaks
idle_time	Short	Long (fewer cold starts)
capacity_per_instance	Higher (better packing)	Not directly
max_use_count	Higher (fewer boots, warm image cache)	Higher (no boot between jobs)
Instance type and purchasing	Spot or preemptible capacity through the instance group's mixed instances policy	Faster instance families, pre-pulled images in the AMI
max_instances	Lower (hard spend cap)	Higher (no queueing at peak)

Spot instances can be reclaimed mid-job. Combine them with retry: { when: [runner_system_failure] } (section 4.3) for jobs that tolerate restarts, and keep deployment jobs on on-demand capacity.

Key takeaways

• Autoscaling uses taskscaler (how many slots) plus a fleeting plugin (how to resize the cloud instance group) plus a connector (how to reach instances).
• The instance group is created by you with desired capacity 0 and scale-in protection. The runner, not the cloud, decides its size.
• Scaling policies set idle capacity per time window. idle_count counts job slots, and the last matching policy wins.
• Use a distributed cache, because autoscaled instances are short-lived.

Practice

1. (Beginner) Given capacity_per_instance = 4, idle_count = 8, max_instances = 10 and 30 running jobs, compute how many instances the runner wants and how many it actually gets.
2. (Intermediate) Set up a Docker Autoscaler runner on a cloud account with an instance group of maximum size 3. Run a pipeline of 10 parallel jobs and record when instances start and stop. Then add an office-hours policy and verify that idle instances disappear outside it.
3. Interview question: "Finance wants the CI bill cut by 40% without slowing developers down. Which autoscaling settings and infrastructure changes would you consider first?" Hint: idle capacity by time of day, packing, spot capacity, image pull time and max_use_count.

Kubernetes Runner Scaling

Why it matters. With the Kubernetes executor, "add capacity" means something different: job pods are scheduled by Kubernetes, and the cluster itself must grow when pods do not fit. Three independent layers have to scale together, and a bottleneck in any one of them shows up to developers as jobs stuck in pending or failing with pod timeouts.

What it is.

Layer 1: runner managers    Deployment replicas x concurrent  = max job pods
            │                (2 replicas x concurrent 40 = up to 80 job pods)
            ▼
Layer 2: job pods           created per job with resource requests (cpu_request, memory_request)
            │                Pending if no node has room
            ▼
Layer 3: nodes              Cluster Autoscaler or Karpenter adds nodes for pending pods,
                             removes empty or underused nodes afterwards

Layer	Controlled by	Typical bottleneck
Runner managers	Helm replicas, concurrent	concurrent too low: jobs stay pending in GitLab although the cluster is idle
Job pods	Resource requests and limits, ResourceQuota on the namespace	Requests larger than any node: pods never schedule
Nodes	Cluster Autoscaler or Karpenter, node group limits, cloud quotas	Slow node start or image pulls: pod timeouts

Dedicated CI nodes. Keep job pods off the nodes that run production or platform services. Taint a node pool for CI and let only job pods tolerate it:

[[runners]]
  [runners.kubernetes]
    namespace = "gitlab-runner"
    cpu_request = "1"
    memory_request = "2Gi"
    poll_timeout = 900                     # allow time for a new node to join and pull the image
    [runners.kubernetes.node_selector]
      "workload" = "ci"                    # schedule only on CI nodes
    [runners.kubernetes.node_tolerations]
      "ci=true" = "NoSchedule"             # tolerate the taint that keeps other pods away
    [runners.kubernetes.pod_annotations]
      "karpenter.sh/do-not-disrupt" = "true"   # do not consolidate a node while a job runs on it

A matching Karpenter NodePool provisions CI nodes on demand, prefers spot capacity, and removes nodes when they are empty or underused:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: ci-jobs
spec:
  template:
    metadata:
      labels:
        workload: ci                       # matches node_selector above
    spec:
      taints:
        - key: ci
          value: "true"
          effect: NoSchedule               # only pods tolerating ci=true land here
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
  limits:
    cpu: "256"                             # spend cap for the whole pool
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 5m

With Cluster Autoscaler, the equivalent of the do-not-disrupt annotation is cluster-autoscaler.kubernetes.io/safe-to-evict: "false".

Practical guidance.

• Set requests realistically. Node autoscalers scale on requests, not on actual usage. Requests that are too small overpack nodes and cause out-of-memory kills. Requests that are too large waste nodes.
• Scale managers for availability, not throughput. Two replicas give redundancy. Each replica is a separate runner manager with its own concurrent, so total capacity is replicas times concurrent.
• Reduce startup time. Image pulls dominate job start on fresh nodes. Use a registry mirror or pull-through cache, small images, and node images with common images pre-pulled.
• Bound the namespace. A ResourceQuota on the runner namespace prevents a burst of CI jobs from consuming the whole cluster.

Key takeaways

• Kubernetes CI capacity has three layers: runner managers (replicas x concurrent), job pods (requests and limits) and nodes (cluster autoscaling).
• Put job pods on a dedicated, tainted node pool with node_selector and node_tolerations.
• Protect running jobs from node consolidation with do-not-disrupt (Karpenter) or safe-to-evict: "false" (Cluster Autoscaler).
• Raise poll_timeout to cover node scale-up and image pull time.

Practice

1. (Beginner) For a runner Deployment with 3 replicas and concurrent = 25, and job pods requesting 1 CPU and 2 GiB on nodes with 16 CPUs and 64 GiB, compute the maximum number of job pods and the number of nodes needed at full load.
2. (Intermediate) In a test cluster with a node autoscaler, configure a tainted CI node pool and the matching runner settings. Start 50 parallel jobs and measure the time from job creation to pod start for the first and the last job. Identify where the time went.
3. Interview question: "Jobs on your Kubernetes runner are randomly killed after a few minutes, with no error in the job script. What could be the cause?" Hint: memory limits, node consolidation or spot reclamation, and how to tell them apart with kubectl get events.

Runner Fleet Dashboard

Why it matters. Once an organization has dozens of runners, simple questions become hard: how many are actually online, which are overloaded, which have not been seen for months, and how long developers wait for a job to start. The fleet dashboard answers these questions in one place.

What it is. The Runner fleet dashboard (Ultimate) is available to administrators at Admin > CI/CD > Runners > Fleet dashboard, and to group Owners for a group's runners at group Build > Runners > Fleet dashboard. Its panels include:

Panel	Shows	Useful for
Active runners	Counts of online, offline and stale runners	Fleet health at a glance
Runner usage	Compute minutes used by the top projects and runners, exportable as CSV	Cost attribution and chargeback
Wait time to pick a job	Median and percentile queue times of jobs on instance runners	Spotting capacity shortages before developers complain
Most busy runners	Runners with the most running jobs compared to their capacity	Finding hot spots

On self-managed instances, the usage and wait time panels need the ClickHouse integration for analytics data.

Runner statuses. The dashboard and runner lists classify runners by their last contact with GitLab:

Status	Meaning
Online	Contacted GitLab within the last 2 hours
Offline	No contact for more than 2 hours
Stale	No contact for more than 7 days (or never contacted and created more than 7 days ago)
Never contacted	Created, but no runner manager has used its token yet

Runner lists can also show an upgrade status badge (Ultimate) that marks runners whose version has an available or recommended upgrade, for example because of a security fix.

Cleaning up. Stale runners clutter lists and may still have valid tokens. Review and remove them regularly. Groups can also enable automatic cleanup of stale group runners in their CI/CD settings.

# List stale runners on the instance (administrator token)
curl --silent --header "PRIVATE-TOKEN: $GITLAB_ADMIN_TOKEN" \
  "https://gitlab.example.com/api/v4/runners/all?status=stale&per_page=100" \
  | jq -r '.[] | [.id, .runner_type, .status, .description] | @tsv'

# Delete one after review (its token stops working)
curl --silent --request DELETE --header "PRIVATE-TOKEN: $GITLAB_ADMIN_TOKEN" \
  "https://gitlab.example.com/api/v4/runners/1234"

Key takeaways

• The fleet dashboard (Ultimate) shows runner health, usage, queue wait times and the busiest runners for the instance or a group.
• Online means contact within 2 hours, offline more than 2 hours, stale more than 7 days.
• Usage and wait time panels need ClickHouse on self-managed instances.
• Remove stale runners regularly through the UI, the API or automatic group cleanup.

Practice

1. (Beginner) Using the API, count your instance's or group's runners by status (online, offline, stale, never contacted).
2. (Intermediate) Write a script that lists stale runners, prints their type, owner (project or group) and description, and deletes them only after an explicit confirmation. Schedule it as a monthly pipeline.
3. Interview question: "Developers say pipelines 'feel slow', but job durations have not changed. How would you use fleet data to confirm or reject that?" Hint: queue time versus run time, percentiles rather than averages, and which runners are saturated.

Monitoring Runners with Prometheus

Why it matters. The fleet dashboard shows the view from GitLab. Operating runners also needs the view from each runner manager: is the process up, how full are its slots, are API requests failing, how long do jobs take? GitLab Runner exposes these as Prometheus metrics, so they fit into the same monitoring and alerting as the rest of your infrastructure.

What it is. Setting listen_address in the global section of config.toml starts an HTTP server that serves metrics at /metrics:

listen_address = ":9252"           # restart needed after changing it

curl --silent http://runner-01.internal:9252/metrics | grep '^gitlab_runner_' | head

Metric	Type	Meaning
gitlab_runner_jobs	Gauge	Jobs currently running, with labels such as runner, state and executor_stage
gitlab_runner_concurrent	Gauge	The configured concurrent value
gitlab_runner_limit	Gauge	The configured limit per runner entry
gitlab_runner_job_duration_seconds	Histogram	Job durations
gitlab_runner_failed_jobs_total	Counter	Failed jobs by failure_reason
gitlab_runner_errors_total	Counter	Errors and warnings logged by the runner, by level
gitlab_runner_api_request_statuses_total	Counter	API calls to GitLab by endpoint and HTTP status
gitlab_runner_request_concurrency_exceeded_total	Counter	Job requests skipped because request_concurrency was reached
gitlab_runner_version_info	Gauge	Version, revision and architecture as labels

Standard Go process metrics (process_*, go_*) are exposed as well. Autoscaling executors add fleeting and taskscaler metrics about instances and slots.

Scraping. For VMs, add a static or service-discovery scrape job. With the Helm chart, enable the metrics endpoint and a ServiceMonitor for the Prometheus Operator:

# prometheus.yml (VM-based runners)
scrape_configs:
  - job_name: gitlab-runner
    static_configs:
      - targets: ["runner-01.internal:9252", "runner-02.internal:9252"]

# Helm values.yaml (Kubernetes runners)
metrics:
  enabled: true
  serviceMonitor:
    enabled: true                  # requires the Prometheus Operator CRDs
service:
  enabled: true

Alerting. A few rules catch most runner incidents:

groups:
  - name: gitlab-runner
    rules:
      - alert: GitLabRunnerDown
        expr: up{job="gitlab-runner"} == 0                     # process or host gone
        for: 5m
      - alert: GitLabRunnerSaturated
        expr: |
          sum by (instance) (gitlab_runner_jobs)
            / sum by (instance) (gitlab_runner_concurrent) > 0.9
        for: 15m                                               # sustained, not a short burst
        annotations:
          summary: "{{ $labels.instance }} is using over 90% of its job slots"
      - alert: GitLabRunnerAPIErrors
        expr: |
          sum by (instance) (rate(gitlab_runner_api_request_statuses_total{status=~"5.."}[10m])) > 0.1
        for: 10m                                               # GitLab or network trouble
      - alert: GitLabRunnerSystemFailures
        expr: |
          sum by (instance) (increase(gitlab_runner_failed_jobs_total{failure_reason="runner_system_failure"}[30m])) > 5

Combine runner metrics with host or node metrics (disk space for Docker images, CPU steal, memory pressure). Many "flaky" jobs turn out to be a full Docker disk on one runner.

Key takeaways

• listen_address exposes Prometheus metrics at /metrics on each runner manager.
• Running jobs versus concurrent measures saturation. API status codes, errors and failure reasons reveal infrastructure problems.
• Alert on runners down, sustained saturation, API errors and system failures.
• Correlate runner metrics with host metrics such as disk space and memory.

Practice

1. (Beginner) Enable listen_address on a runner, fetch /metrics with curl, and find the current value of gitlab_runner_concurrent and the number of running jobs while a pipeline runs.
2. (Intermediate) Build a Grafana dashboard with four panels: slot utilization per runner, jobs per minute, failed jobs by reason, and the 95th percentile job duration from gitlab_runner_job_duration_seconds_bucket.
3. Interview question: "Which single runner metric would you look at first when developers complain that jobs start slowly, and why might it not be enough?" Hint: saturation versus queue time in GitLab, and capacity from runners that do not match the jobs' tags.

Upgrading and Maintaining Runners

Why it matters. Runners execute untrusted code with access to secrets, so security fixes matter. New GitLab features, such as GitLab Secrets Manager requiring Runner 19.0 or later (section 5.4), also need a matching runner version. Yet upgrading carelessly kills running jobs. A maintenance routine keeps runners current without disrupting pipelines.

What it is.

Version compatibility. GitLab Runner is released monthly together with GitLab and uses the same version numbers. Keep the runner's major and minor version in line with GitLab's. Runners a few minor versions behind usually keep working, because compatibility is maintained within a major version, but new features may be unavailable. Major versions can remove deprecated features, so read the release's removals before a major upgrade. Upgrade GitLab first, then the runners.

Graceful shutdown. The runner handles signals differently:

Signal	Behavior
SIGQUIT	Stop requesting new jobs, wait for running jobs to finish (up to shutdown_timeout), then exit
SIGTERM, SIGINT	Stop quickly: running jobs are aborted

Drain-and-upgrade procedure for one runner manager. On systemd hosts, a process that exits after SIGQUIT may be restarted by the unit right away and pick up new jobs. Make systemctl stop itself graceful with a drop-in, once per host:

# /etc/systemd/system/gitlab-runner.service.d/graceful.conf
[Service]
KillSignal=SIGQUIT          # "systemctl stop" drains instead of aborting jobs
TimeoutStopSec=7200         # at least as long as shutdown_timeout in config.toml

sudo systemctl daemon-reload                               # once, after adding the drop-in

# 1. Drain: stop taking new jobs, wait for running ones (this manager only)
sudo systemctl stop gitlab-runner
#    Docker installation: docker kill --signal=SIGQUIT gitlab-runner && docker wait gitlab-runner

# 2. Upgrade the package (Debian/Ubuntu); pin the version to match GitLab
sudo apt-get update
apt-cache madison gitlab-runner | head -3                  # see available versions
sudo apt-get install gitlab-runner=19.4.0-1 gitlab-runner-helper-images=19.4.0-1

# 3. Start and verify
sudo systemctl start gitlab-runner
gitlab-runner --version
sudo gitlab-runner verify

To stop all managers of a runner from taking new jobs (for example before maintenance of shared storage), pause the runner in GitLab instead and wait for its running jobs to finish:

# Pause runner 42 (all managers), then wait until it has no running jobs
curl --silent --request PUT --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.example.com/api/v4/runners/42" --data "paused=true"

until [ "$(curl --silent --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.example.com/api/v4/runners/42/jobs?status=running" | jq length)" -eq 0 ]; do
  sleep 30
done
# ... maintenance ...  then unpause with paused=false

Upgrading by installation type.

Installation	How to upgrade	Notes
Linux package	apt-get install / dnf install the new version	Pin versions. The helper images package must match
Docker container	Pull gitlab/gitlab-runner:v19.4.0, recreate the container with the same config volume	Use explicit version tags, not latest
Helm chart	helm upgrade with a new chart version	Chart versions differ from runner versions. Check appVersion with helm search repo gitlab/gitlab-runner --versions
Autoscaled VMs	Update the runner manager as above. Rebuild instance images (AMI) on a schedule	max_use_count and idle_time naturally replace old instances
Several managers per runner	Upgrade one manager at a time	The manager list shows mixed versions during the rollout

# Helm: upgrade to a specific chart version, keeping your values file
helm repo update
helm upgrade gitlab-runner gitlab/gitlab-runner \
  --namespace gitlab-runner --version "$CHART_VERSION" --values values.yaml

A Helm upgrade replaces the runner manager pods. Set the chart's terminationGracePeriodSeconds (and shutdown_timeout) long enough for your longest jobs, and verify in a test namespace that a running job survives an upgrade before relying on it.

Routine maintenance checklist.

Task	Frequency	How
Upgrade runners	Monthly, or on security releases	Procedure above, one manager at a time
Clean Docker images and volumes on static Docker hosts	Daily or weekly	/usr/share/gitlab-runner/clear-docker-cache (shipped with the package) or docker system prune with filters
Rebuild autoscaling and base VM images	Monthly	Packer pipeline with OS patches and pre-pulled common images
Remove stale runners and managers	Monthly	Fleet dashboard or API (previous topics)
Rotate runner authentication tokens	Per policy	Token expiration setting or reset_authentication_token
Review config.toml drift	Continuously	Configuration management, compare against the template

Key takeaways

• Keep runner major and minor versions aligned with GitLab. Upgrade GitLab first.
• SIGQUIT drains a manager gracefully. SIGTERM aborts running jobs. Pausing a runner in GitLab drains all its managers.
• Upgrade one manager at a time, pin versions, and use explicit image and chart versions.
• Routine maintenance includes Docker disk cleanup, VM image rebuilds, stale runner removal and token rotation.

Practice

1. (Beginner) Find the version of each runner available to your project and compare it to your GitLab version. Identify any runner more than two minor versions behind.
2. (Intermediate) Start a 10-minute job on a runner, then upgrade that runner using the graceful drain procedure above. Confirm that the running job finishes, that no new job starts on that manager during the upgrade, and that the next job runs on the new version.
3. Interview question: "Describe how you would upgrade a fleet of 40 runner managers across three runner records without failing any jobs and without reducing capacity by more than 10% at any time." Hint: batches, graceful shutdown, shutdown_timeout, and verification before moving to the next batch.

---

<a id="7-docker-and-container-workflows"></a>