How to Configure GitLab Runner to Run Multiple Pipelines in Parallel for Multiple Projects: Avoid Queueing Issues

Table of Contents#

1. Understanding GitLab Runner Basics
2. Root Causes of Pipeline Queueing
3. Prerequisites
4. Configuring GitLab Runner: Core Settings
5. Setting Up Concurrency: The Key to Parallelism
6. Handling Multiple Projects: Shared Runners & Access Control
7. Resource Allocation: Preventing Job Starvation
8. Advanced Tips: Autoscaling and Caching
9. Monitoring and Troubleshooting Queueing Issues
10. Conclusion
11. References

1. Understanding GitLab Runner Basics#

Before diving into configuration, let’s clarify how GitLab Runner works:

• GitLab Runner is an agent that runs on your infrastructure (VM, container, or bare metal) and executes jobs defined in .gitlab-ci.yml files.
• Executors: Runners use executors to spawn environments for jobs (e.g., docker, shell, kubernetes, virtualbox). The docker executor is most common for isolation.
• Job Lifecycle: Runners poll GitLab for pending jobs, fetch the job, execute it in an isolated environment, and report results back.

2. Root Causes of Pipeline Queueing#

Queueing occurs when the number of pending jobs exceeds the runner’s capacity to process them in parallel. Common causes include:

• Insufficient Concurrency: The runner is not configured to run enough jobs in parallel.
• Resource Starvation: Jobs consume excessive CPU/memory, slowing down others or crashing the runner.
• Runner Scope: Runners are tied to a single project, leaving other projects without resources.
• Poor Job Scheduling: Lack of tagging or prioritization leads to inefficient job distribution.

3. Prerequisites#

To follow this guide, ensure you have:

• A GitLab instance (self-managed or GitLab.com) with admin access (to register shared runners).
• A server/VM to host the runner (recommended: 4+ CPU cores, 8GB+ RAM for moderate workloads).
• GitLab Runner installed (see GitLab’s installation docs).
• Basic familiarity with .gitlab-ci.yml and GitLab CI/CD concepts.

4. Configuring GitLab Runner: Core Settings#

GitLab Runner’s behavior is controlled by the config.toml file (location varies by OS: e.g., /etc/gitlab-runner/config.toml on Linux). Let’s break down key settings for parallelism.

Step 1: Locate and Edit config.toml#

First, find your config.toml. For Linux systems using the official package, it’s typically at /etc/gitlab-runner/config.toml. Edit it with:

sudo nano /etc/gitlab-runner/config.toml  

Step 2: Key Settings Overview#

Here’s a simplified config.toml template (we’ll customize it later):

concurrent = 4  # Total parallel jobs across all runners  
check_interval = 0  # How often runner polls GitLab (in seconds; 0 = default)  
 
[[runners]]  
  name = "Shared Docker Runner"  
  url = "https://gitlab.example.com/"  
  token = "RUNNER_REGISTRATION_TOKEN"  
  executor = "docker"  
  limit = 4  # Max parallel jobs for THIS runner  
  [runners.custom_build_dir]  
  [runners.cache]  
    [runners.cache.s3]  
    [runners.cache.gcs]  
  [[runners.docker]]  
    tls_verify = false  
    image = "alpine:latest"  # Default image for jobs  
    privileged = false  
    disable_entrypoint_overwrite = false  
    oom_kill_disable = false  
    disable_cache = false  
    volumes = ["/cache"]  
    shm_size = 0  
    # Resource limits (critical for avoiding starvation)  
    cpu_shares = 1024  # Relative CPU weight (default 1024)  
    memory = "2g"  # Max memory per job  
    memory_swap = "4g"  # Max swap + memory  

5. Setting Up Concurrency: The Key to Parallelism#

The concurrent setting in config.toml defines the total number of parallel jobs all runners on the host can execute. This is the most critical setting for avoiding queueing.

How to Calculate concurrent#

Set concurrent based on your server’s resources:

• CPU: Each job typically uses 1+ CPU core. A server with 8 cores can handle ~8 concurrent CPU-bound jobs.
• Memory: If jobs use 2GB RAM each, an 8GB server can run ~4 concurrent jobs.

Example: For a server with 4 CPU cores and 16GB RAM, set concurrent = 8 (assuming jobs are not extremely resource-heavy).

Per-Runner Limits with limit#

The limit setting under [[runners]] restricts the number of parallel jobs for that specific runner (useful if you have multiple runners on one host).

Example: If concurrent = 8 and you have two runners, set limit = 4 for each to split capacity evenly.

6. Handling Multiple Projects: Shared Runners & Access Control#

To serve multiple projects, use shared runners (vs. project-specific runners). Shared runners are available to all projects in your GitLab instance (or group) and prevent siloing resources.

Step 1: Register a Shared Runner#

Register the runner as a shared runner during setup:

sudo gitlab-runner register  

• URL: Your GitLab instance URL (e.g., https://gitlab.example.com).
• Registration Token: For shared runners, use the instance-wide token (found in GitLab Admin > Runners).
• Executor: Choose docker (recommended for isolation).
• Default image: e.g., alpine:latest.

Step 2: Enable Shared Runners for Projects#

By default, shared runners are disabled for projects. Enable them in:
Project > Settings > CI/CD > Runners > Enable shared runners.

Step 3: Control Job Assignment with Tags#

Use tags to ensure jobs are routed to the right runners. For example:

• Tag the runner with docker, linux during registration.
• In .gitlab-ci.yml, specify tags for jobs:

job1:  
  script: echo "Hello"  
  tags: [docker, linux]  # Only runs on runners with these tags  

7. Resource Allocation: Preventing Job Starvation#

Even with concurrency, jobs may hog resources, slowing others. Use these settings to enforce limits.

Docker Executor Resource Limits#

Resource limits for jobs are configured in config.toml under [[runners.docker]], not in .gitlab-ci.yml. The Docker executor provides the following settings:

• memory: Maximum RAM per job (e.g., memory = "2g").
• memory_swap: Total memory + swap (e.g., memory_swap = "4g").
• cpu_shares: Relative CPU priority (higher = more CPU when contention occurs).

Example:

[[runners.docker]]  
  memory = "2g"  
  memory_swap = "4g"  
  cpu_shares = 512  # Lower than default (1024) to prioritize other jobs  

For more granular control, consider building custom Docker images with built-in resource limits or using wrapper scripts to enforce resource constraints at the application level.

8. Advanced Tips: Autoscaling and Caching#

Autoscaling for Variable Workloads#

For teams with variable job volumes (e.g., peak hours), use autoscaling to dynamically provision runners:

• Docker Machine Executor: Spins up VMs on cloud providers (AWS, GCP) when jobs queue.
• Kubernetes Executor: Uses Kubernetes pods to run jobs, scaling with cluster resources.

Example: Enable autoscaling in config.toml with [runners.autoscaler] (see GitLab’s autoscaling docs).

Caching to Reduce Job Runtime#

Caching dependencies (e.g., node_modules, venv) speeds up jobs, reducing queue time. Define caches in .gitlab-ci.yml:

cache:  
  paths:  
    - node_modules/  
 
build:  
  script: npm install && npm run build  

9. Monitoring and Troubleshooting Queueing Issues#

Monitor Runner Performance#

• GitLab UI: Go to Admin > Runners to see active jobs, pending jobs, and runner status.
• Logs: Check runner logs with journalctl -u gitlab-runner (Linux systemd) to identify crashes or resource errors.
• Metrics: Use Prometheus with the GitLab Runner exporter to track gitlab_runner_jobs_running and gitlab_runner_jobs_pending.

Common Issues & Fixes#

• Jobs Stuck in Pending:

  • Check if concurrent or limit is too low.
  • Verify the runner is online (Admin > Runners).
  • Ensure jobs have matching tags with the runner.

• Resource Exhaustion:

  • Reduce concurrent or set stricter memory/cpu limits in config.toml.
  • Use tools like htop to identify resource-heavy jobs.

10. Conclusion#

By configuring concurrency, using shared runners, and managing resources, you can eliminate pipeline queueing and ensure fast, parallel CI/CD for multiple projects. Key takeaways:

• Set concurrent based on CPU/memory to maximize parallelism.
• Use shared runners with tags for multi-project support.
• Enforce resource limits to prevent job starvation.
• Monitor and autoscale to handle variable workloads.

11. References#

• GitLab Runner Configuration
• Shared Runners
• Docker Executor Settings
• Autoscaling Runners
• Docker Executor