Scheduled pipelines and E2E: signal without blocking

Ten parts in, every pipeline in this series has started the same way: someone pushed. Lint, build, deploy, even the cross-repo trigger from part 7 — all of it reacts to a commit landing somewhere. The E2E suite is the first thing that doesn’t fit that shape. It drives a real browser against a real, access-gated staging environment over a real network, it takes long enough that nobody would sit and wait for it on a merge request, and on any given day a handful of its failures are the network’s fault rather than the code’s.

Dropping the suite wasn’t an option either: it catches the category of bug nothing else in the pipeline can see. What it needed was a clock, not a trigger: run every night, run on demand when a human asks, and never stand between anyone and a merge.

A job that runs on a clock

Schedules aren’t defined in .gitlab-ci.yml. They live in the UI, under Build → Pipeline schedules: a cron expression, a target branch, and optionally some variables specific to that schedule. When the cron fires, GitLab starts an ordinary pipeline on that branch with CI_PIPELINE_SOURCE set to schedule. From there, your rules (part 5) decide what actually runs:

e2e:
  stage: e2e
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"
    - if: $CI_PIPELINE_SOURCE == "web"
      when: manual
      allow_failure: true
  script:
    - pnpm exec playwright test

The first clause is the nightly run. The second is the escape hatch: when someone starts a pipeline by hand from the UI (source web), the job shows up as a manual play button, so anyone can fire the suite right after fixing something instead of waiting for 2 a.m. Push pipelines match neither clause, so day-to-day merges never see this job at all.

There’s a second escape hatch built into schedules themselves: each one has a run-now button on the schedules page, limited to once per minute, and it runs with the permissions of whoever pressed it rather than the schedule’s owner. And because variables can be attached per schedule, one YAML file can serve several cadences — we keep an hourly schedule that sets SUITE=smoke and a nightly one that runs everything.

Getting past the access gate

Our staging environment sits behind an identity-aware proxy, Cloudflare Access in our case. Humans get through it with an SSO redirect. A headless browser in a CI job can’t do that dance, and scripting a real login flow against an identity provider is the kind of test setup that breaks monthly.

The non-interactive answer is a service token. You create one in the Cloudflare dashboard and get a client ID and secret; any request that presents them as two headers, CF-Access-Client-Id and CF-Access-Client-Secret, is let through without a login flow.

The obvious move is to make Playwright attach those headers to every request with extraHTTPHeaders. It works, but now the credentials live inside the browser context — they end up in traces, HAR files, anything that records what the browser sent. We went a different way: the browser never handles credentials at all. The job builds the app, serves the build locally, and puts a tiny reverse proxy in front. Page loads are served from the local build (the exact code this pipeline produced) and API calls are forwarded to the gated origin with the headers attached server-side:

// proxy.mjs — forwards /api/* to the gated origin, injecting the token
import http from 'node:http';
import https from 'node:https';
 
http
  .createServer((req, res) => {
    const upstream = new URL(req.url, process.env.STAGING_ORIGIN);
    const proxied = https.request(
      upstream,
      {
        method: req.method,
        headers: {
          ...req.headers,
          host: upstream.host,
          'CF-Access-Client-Id': process.env.CF_ACCESS_CLIENT_ID,
          'CF-Access-Client-Secret': process.env.CF_ACCESS_CLIENT_SECRET,
        },
      },
      r => {
        res.writeHead(r.statusCode, r.headers);
        r.pipe(res);
      },
    );
    req.pipe(proxied);
  })
  .listen(8080);

That’s the core of it. The real script also routes non-API paths to the local static server. The job side is three lines:

npx serve dist --listen 4173 &   # the app built earlier in this pipeline
node scripts/proxy.mjs &         # :8080 → /api to staging, the rest to :4173
BASE_URL=http://localhost:8080 pnpm exec playwright test

As a bonus, this tests the branch’s own frontend build against the real staging backend, instead of whatever staging last deployed. The ID and secret live as masked CI/CD variables. One operational note: service tokens expire after the duration you picked at creation, and Cloudflare can alert you a week before that happens — configure the alert, because the alternative is the nightly run failing on a random Tuesday with a 403 and no obvious cause.

Sharding the suite across runners

A suite this size on one runner is most of the pipeline’s wall-clock time. Part 4’s parallel: keyword lines up directly with Playwright’s --shard flag:

e2e:
  parallel: 4
  script:
    - pnpm exec playwright test
      --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
      --reporter blob
  artifacts:
    when: always
    paths:
      - blob-report/

GitLab spawns four copies of the job with CI_NODE_INDEX set to 1 through 4, and Playwright’s shard syntax happens to want exactly that: --shard=1/4. There are commercial orchestration services that split tests across machines by historical timing instead of by file count; they’re worth a look at a much larger scale, but plain sharding cut our wall-clock time to roughly a quarter for free.

Each shard writes a blob report (Playwright’s mergeable intermediate format) as a regular artifact. The when: always matters: a shard with failing tests is exactly the shard whose report you need, so upload has to happen on failure too.

Run stats as pipeline data

Four shards means four partial reports and no single place that knows how the run went. A merge job stitches them together and, while it’s there, distills the run into numbers, the pattern from part 3, with artifacts:reports:dotenv carrying structured state between jobs:

merge-report:
  stage: report
  needs: ['e2e']
  script:
    - pnpm exec playwright merge-reports --reporter html ./blob-report
    - node scripts/stats.mjs # reads merged results, writes run.env
  artifacts:
    when: always
    paths:
      - playwright-report/
    reports:
      dotenv: run.env

run.env is three lines of plain assignments:

E2E_PASSED=182
E2E_FAILED=3
E2E_FLAKY=7

Every later job in the pipeline sees those as ordinary variables — no parsing, no artifact digging. The mechanism is sized for exactly this kind of payload: the dotenv file is capped at 5 KB, and by default only around 20 variables are inherited, so it’s a channel for a summary, with the full report traveling as a normal artifact next to it.

Reports that outlive the pipeline

GitLab artifacts are the wrong home for test history. They’re scoped to one pipeline, they expire (part 3 set short expiries on purpose), and there’s no view that lines up last night’s run next to the previous thirty. For trends, and especially for watching the flaky count move over weeks, we wanted append-only storage that nothing in GitLab ever cleans up.

The shape: an object-storage bucket, one prefix per run, and a manifest that accumulates:

publish-report:
  stage: publish
  needs: ['merge-report']
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"
    - if: $CI_PIPELINE_SOURCE == "web"
      when: manual
  script:
    - RUN_ID=$(date -u +%Y-%m-%d-%H%M)
    - aws s3 cp playwright-report/ "s3://e2e-reports/runs/$RUN_ID/" --recursive
    - node scripts/update-manifest.mjs "$RUN_ID"

RUN_ID is just the UTC date and time, which sorts chronologically for free. The manifest script downloads manifest.json from the bucket root, appends an entry (run ID plus the E2E_* counts, which arrived through the dotenv report without any extra plumbing), and uploads it back. A static index.html at the bucket root fetches the manifest and renders the list: every run, its pass/fail/flaky numbers, a link to its full HTML report. A serverless test-history dashboard whose entire hosting bill is object storage and a few kilobytes of traffic — pennies.

The read-modify-write on manifest.json is a race condition in general. With one nightly writer and the occasional manual run it never bites; if you ever add overlapping schedules, wrap the publish job in a resource_group (part 5) so runs serialize.

Why this doesn’t gate merges

The uncomfortable part. These tests cross a real network to a gated environment, ride on data that other teams mutate, and pay Access-gate latency on every request. The flaky count in run.env is not zero on a good day. If this suite were a required MR check, the sequence is predictable: MRs go red for infrastructure reasons, people retry until green, and within a month nobody reads the failures at all. A gate nobody trusts is worse than no gate.

We’d already tried the softer version of gating (part 7’s downstream trigger runs a related E2E suite from another repo on merges, manual and allowed to fail), so this decision was made with eyes open: scheduled plus manual, no gate. Which means, stated plainly, that bugs land on main first and get caught that night. That’s the accepted tradeoff. The suite’s job is to find regressions within a day of landing, with a browsable record of when each one appeared, and the E2E_FLAKY number in the manifest tells us whether the suite itself is trending toward or away from being trustworthy enough to gate anything someday.

Where this leaves you

The E2E suite now runs every night and on demand, walks through the access gate without the browser ever touching a credential, spreads across four runners, and leaves a permanent, browsable history of every run. None of it blocks a merge, and that’s deliberate.

The other thing running on autopilot is doing much worse. The security scan takes 21 minutes, flags things nobody reads, and gets ignored by everyone — which is the worst possible state for the one job whose findings might actually matter. Making it fast and specific enough to be taken seriously is part 12.

References

• Scheduled pipelines
• Specify when jobs run with rules
• artifacts:reports:dotenv
• Pass CI/CD variables with dotenv artifacts
• Cloudflare Access service tokens