A shared CI library: one pipeline config for ten repos

Part 7 ended with pipelines triggering each other across repositories. This part is about the configuration itself. We had ten sibling microfrontend repos, each carrying its own copy of the same ~300-line .gitlab-ci.yml: copied from the first repo once, then patched independently for a year. The jobs had the same names everywhere and behaved the same nowhere: one repo had the artifact-expiry fix, another still targeted a runner tag we’d retired, a third carried a rules tweak that quietly skipped its deploy job on release branches. Every pipeline improvement now cost ten merge requests, so improvements had mostly stopped.

The fix is the one you’d apply to any duplicated code: extract a library. GitLab CI has the mechanics for it: include: to pull config from another repo, hidden jobs as the API surface, !reference for the parts extends can’t reach. What the docs don’t cover is the operational side: what you promise your consumers, how you roll out a change that lands in ten repos at once, and what versioning looks like when nobody wants to maintain tags. That part we had to work out in production.

Four flavors of include

include: merges other YAML files into your configuration before the pipeline is created. There are four flavors:

include:
  # a file in the same repository
  - local: ci/deploy.yml
 
  # a file in another project on the same instance, at a specific ref
  - project: group/shared-ci
    ref: main
    file: frontend.yml
 
  # a URL fetched with a plain HTTP(S) GET
  - remote: https://example.com/ci/base.yml
 
  # templates that ship with GitLab itself
  - template: Jobs/SAST.gitlab-ci.yml

Each has a job. local splits one repo’s config into readable pieces. template pulls in GitLab’s stock jobs (the managed security scanners are the usual reason). remote fetches over plain HTTP(S), which works for genuinely public config and gets awkward for anything that needs credentials. project is the shared-library flavor: the file lives on the same GitLab instance, so normal project permissions apply, and ref pins a branch, tag, or commit SHA. Everything below uses it.

The merge is predictable: included files are combined with your own, and when the same key is defined in both, your file wins. Hash maps merge deeply; anything else, including arrays, is replaced outright, so redefining a job’s script in the consumer replaces the library’s version rather than appending to it. Includes also nest: an included file can include further files, up to a default limit of 150 per pipeline.

Hidden jobs as a public API

Part 1 introduced hidden jobs (names starting with .) as a way to deduplicate within one file. Across repos they become something more deliberate: the library’s public API. The shared repo defines no runnable jobs at all (only hidden ones) and consumers instantiate the ones they want with extends:

# group/shared-ci — frontend.yml
.frontend_base:
  image: node:22-slim
  tags: [frontend-k8s]
  before_script:
    - corepack enable
    - pnpm install --frozen-lockfile
 
.frontend_build:
  extends: .frontend_base
  stage: build
  script: pnpm build
  artifacts:
    paths: [dist/]
    expire_in: 1 day

A consumer’s whole file collapses into includes plus one-liners:

# my-app — .gitlab-ci.yml
include:
  - project: group/shared-ci
    ref: main
    file: frontend.yml
 
build:
  extends: .frontend_build
 
deploy-preview:
  extends: .frontend_deploy_preview
  variables:
    APP_NAME: my-app

Our consumer files went from ~300 lines to ~130, and most of those 130 are exactly this shape: a job name, an extends, occasionally a variable. The docs are explicit that combining extends with include is the supported way to reuse configuration across files, and the merge is a reverse deep merge: keys defined on the extending job override the base, and arrays like script are swapped wholesale, never concatenated. That replacement behavior is the consumer’s escape hatch and the library’s biggest hazard at the same time: a consumer that sets script: on an extended job has silently opted out of everything the library’s script did.

One convention that paid off: every hidden job in the library carries a .frontend_ prefix, so it can never collide with a hidden job a consumer defines locally for its own purposes.

A rules vocabulary with !reference

Part 5 was about rules, and rules were the worst-drifted part of the ten copies: long if: strings like $CI_MERGE_REQUEST_EVENT_TYPE == "merged_result" pasted into every file, each copy one typo away from a job that silently never runs. Inside a single file you’d deduplicate those with YAML anchors, but anchors are only valid in the file where they’re defined: they cannot cross an include boundary. The !reference tag exists for exactly this: it selects a section from any job, hidden or not, including jobs that arrived via include.

So the library defines a vocabulary. One hidden job, .shared_vars, holds named rule fragments with descriptive names:

# group/shared-ci — frontend.yml
.shared_vars:
  we_are_in_merge_result_pipeline:
    - if: '$CI_MERGE_REQUEST_EVENT_TYPE == "merged_result"'
  we_are_pushing_to_default_branch:
    - if: '$CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
  this_is_a_scheduled_run:
    - if: '$CI_PIPELINE_SOURCE == "schedule"'

Hidden jobs are never validated as jobs, so those custom keys are legal — a free namespace. Library jobs and consumers alike consume the vocabulary by reference:

.frontend_deploy_preview:
  rules:
    - !reference [.shared_vars, we_are_in_merge_result_pipeline]

The rule reads like a sentence, and the condition string exists in exactly one place across ten repositories. When we later moved the microfrontends to merged-results pipelines (part 6), the predicate changed once, in the library, and every consumer followed. A consumer can also mix vocabulary entries with its own local rule entries in the same rules: list; GitLab flattens the referenced list into place.

The consumer contract

An include line creates a dependency, and dependencies need contracts. Ours lives in the shared-ci README and has four clauses:

• Required CI/CD variables. Each consumer sets APP_NAME and its deploy target variables in project settings. The library’s jobs fail fast with a clear message if they’re missing.
• Required repo files. A package.json exposing build, lint, and test scripts, plus the lockfile. The library calls scripts by name; it never guesses at tooling.
• Documented overrides. Which keys a consumer may safely override (variables, extra rules entries) and which void the warranty (script on deploy jobs).
• Secure by default. Security scan jobs run in every consumer without being asked for. Opting out requires an explicit variable (SKIP_DEPENDENCY_SCAN: "true") in the consumer’s own file.

The opt-out design matters more than it looks. Because the escape hatch is a literal string in the consumer’s YAML, one grep across the group lists exactly which repos are skipping the scanner — no dashboard required, and every opt-out went through a reviewed merge request.

Shipping changes when everyone tracks main

We never versioned the library with tags. Every consumer pins ref: main, which means a merge into shared-ci reaches ten repos immediately — cheap to operate, and exactly as dangerous as it sounds. The discipline that made it safe is a pilot-then-flip rollout:

1. Open a branch on shared-ci with the change.
2. Pick one consumer as the pilot and point its ref: at the feature branch, in a merge request.
3. Let real pipelines run in the pilot repo. Iterate on the library branch until they’re green: the change gets validated by real jobs against a real project before anyone else sees it.
4. Flip the pilot’s ref: back to main, merge the library branch, and the other nine repos pick it up on their next pipeline.

The rollout that proved the model was a runner migration. We moved the fleet from the tag frontend-ci to frontend-k8s, which in the copy-paste era would have been ten merge requests and a week of stragglers. In the library era it was one edit to tags: in .frontend_base, one pilot validation, one merge. No consumer touched its own YAML; the change traveled through extends inheritance.

Tracking main has a defensive corollary. A consumer’s default: block applies to every job in its pipeline, including jobs that came from the library, wherever a job doesn’t set the key itself. So the library over-specifies on purpose: tags, cache, and before_script are re-set explicitly on the jobs that matter, even where a shared base would have covered them, because a consumer adding an innocent default: must not silently reroute the library’s deploy job to a different runner.

The library repo also runs a pipeline of its own: yamllint over its templates on every merge request. The config is modest: the relaxed preset, with line-length raised, because !reference [.shared_vars, ...] lines are long and breaking them helps nobody:

# .yamllint
extends: relaxed
rules:
  line-length:
    max: 160

The cache block we shipped commented out

The most instructive block in the library is one that does nothing:

.frontend_base:
  # Cache disabled on purpose. Our runners have no object-storage
  # backend configured ([runners.cache] is unset), so archiving the
  # pnpm store into pod-local cache measured as pure overhead —
  # ~90s per job, zero hits after the pod recycles. See part 1.
  # When the backend lands, restore:
  # cache:
  #   key:
  #     files: [pnpm-lock.yaml]
  #   paths: [.pnpm-store]
  cache: []

This is part 1’s self-hosted cache trap, four years of series-time later, still unfixed at the infrastructure level, and the shared config is honest about it. cache: [] forces the cache off even if a consumer sets a default: cache:, the comment explains why to anyone reading any of the ten repos, and the real configuration sits right there waiting. The day the object-storage backend is configured in the runner, uncommenting that block is a single merge request, and it warms ten repositories at once.

Where this leaves you

Ten repos now consume one pipeline definition: hidden jobs as the API, one rules vocabulary shared through !reference, and a pilot-then-flip ritual instead of version tags. Consumer files shrank from ~300 lines to ~130, and a runner migration that used to be ten merge requests became one.

The library works because the ten repos are shaped alike: same stack, same stages, same deploy story. Part 9 takes on the repo where that assumption breaks: a monorepo with ~50 packages and 8 apps, where you can’t rebuild the world on every commit and the pipeline has to figure out what actually changed.

References

• Use CI/CD configuration from other files (include)
• YAML optimization: anchors, extends, and !reference