Skip to content

Faster GitLab CI pipelines with needs and parallel

Your jobs are efficient but your pipeline is slow, because stages make every job wait for jobs it does not depend on. Replacing stage barriers with a needs graph, and splitting the slowest job across runners with parallel.

3 min read
  • gitlab
  • ci-cd
  • devops

After parts 2 and 3, no job in the pipeline repeats another job’s work. The pipeline is still slow, and the reason is no longer inside the jobs — it’s between them. Stages are barriers: every job in a stage waits for the entire previous stage, whether it depends on those jobs or not. On a five-job pipeline nobody notices. As the file grows, the waiting quietly becomes the biggest line item in your wall-clock time.

Stages make everything wait for everything

Take a pipeline that builds and deploys an app and its documentation site:

stages: [setup, build, deploy]
 
install-deps: # setup    ~60s
build-app: #    build    ~180s
build-docs: #   build    ~30s
deploy-app: #   deploy   ~60s
deploy-docs: #  deploy   ~20s

Follow the docs path. build-docs finishes 30 seconds into the build stage, and then deploy-docs (which needs nothing but build-docs) sits idle for another 150 seconds waiting for build-app to release the stage barrier. The docs are deployable at second 110 (60 + 30 + 20) and actually deploy around second 260. More than half the docs path is spent waiting on a job it doesn’t care about.

needs: declare the real graph

needs lets a job list its actual dependencies and start the moment they finish, ignoring the stage barrier:

build-docs:
  stage: build
  needs: [install-deps]
  script: pnpm --filter docs build
 
deploy-docs:
  stage: deploy
  needs: [build-docs]
  script: ./scripts/deploy-docs.sh

Same stages, same jobs, but now deploy-docs runs at second 110 instead of 260. The app path doesn’t change (install-deps → build-app → deploy-app was the critical path all along), but every path that isn’t the critical path stops paying rent on it. The more independent chains your pipeline has, the more this buys you; GitLab’s docs call this a directed acyclic graph pipeline, and the pipeline view even renders the graph.

Two side effects worth knowing:

  • A job with needs downloads artifacts only from the jobs it lists. That narrows the download-everything-from-earlier-stages default from part 3, which is usually what you wanted anyway.
  • needs: [] makes a job start immediately when the pipeline is created, ignoring stages altogether. Good for jobs with no dependencies at all: a lint that only reads source, a notification job.

One warning from production: needs moves the dependency information from one stages: list into every job, and it’s on you to keep it honest. Forget to add a needed job and you’ve built a race condition: the job starts early, sometimes before its input exists, and fails intermittently. When a needs pipeline flakes, check the graph before blaming the runner.

parallel: split one slow job across runners

The graph is only as fast as its slowest node, and in most frontend pipelines that node is the test suite. parallel: N clones a job N times and hands each clone two variables, CI_NODE_INDEX and CI_NODE_TOTAL, which map directly onto test-runner sharding flags:

e2e:
  stage: quality
  parallel: 4
  script: pnpm playwright test --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL

Four runners, a quarter of the suite each. A 20-minute suite becomes a 5-something-minute suite for the price of three extra runners being busy. The economics justify themselves: runner minutes are cheap, engineers waiting on a merge are not.

The related parallel:matrix runs a job once per combination of variables, which is how one build job covers several apps without three copies of the YAML:

build-app:
  stage: build
  parallel:
    matrix:
      - APP: [storefront, admin, docs]
  script: pnpm --filter $APP build

Each matrix entry is a separate job in the pipeline view (build-app: [storefront] and so on), and each can be needs-ed individually: a deploy job can depend on exactly the one build it ships.

Where this leaves you

Four parts in, the pipeline is honest work only: installs once, builds once, tests in shards, and every job starts the moment its real dependencies are met.

What nothing so far controls is when any of this runs at all. Push a feature branch and the deploy jobs tag along. Open a merge request and you may get two pipelines for the same commit. Tag a release and the whole quality stage re-runs on code that already passed. Deciding which jobs exist in which situations starts with job-level rules in part 5; the pipeline-level half (workflow and the choice of pipeline type, the most consequential decision in the whole file) is part 6.

References