Security scanning nobody ignores

Part 11 moved the slow, flaky signal (E2E) onto a schedule where it could fail loudly without blocking anyone. The security scan on the same project had the opposite problem: it ran on every merge request, took 21 minutes, and reported findings that mostly lived in code the author had never touched. The team adapted the way teams adapt to any alarm that fires constantly: they learned where the widget was and stopped reading it.

A scan nobody reads is worse than no scan at all, because it burns 21 minutes per pipeline while producing the comfortable feeling that something is being checked. This part is about earning that attention back: GitLab’s managed templates where they’re enough, a diff-aware Semgrep job for the case they don’t cover well, and a dependency scanner with a severity gate so it only speaks up when it matters.

Start with the managed templates

GitLab ships maintained scanner jobs as templates. Two lines pull in SAST and secret detection:

include:
  - template: Jobs/SAST.gitlab-ci.yml
  - template: Jobs/Secret-Detection.gitlab-ci.yml

The SAST template inspects the repository, decides which analyzers apply, and for most languages runs Semgrep with GitLab-managed rules. The secret detection template runs Gitleaks over recent commits. Both upload their results as security reports, which is what makes findings appear in the pipeline’s Security tab and in the merge request widget, where GitLab shows what the MR newly introduced or resolved rather than the whole backlog.

For a small or mid-sized repo, stop here. Two lines, zero maintenance, rules updated by GitLab, reports wired correctly. Replacing a scanner is justified only when you can point at the concrete problem the template can’t solve; ours was scan time and noise on a large repo, and it only affected SAST.

The secret detection template we kept exactly as-is. It’s fast, and on feature branches it scans the commits between the merge base and the head, so it checks what you’re about to merge. One variable worth knowing: SECRET_DETECTION_HISTORIC_SCAN: "true" makes a run sweep the entire git history. Do that once, right after enabling the template; after that, scanning new commits is enough.

Diff-aware SAST with Semgrep

On a repo with a few hundred thousand lines, the managed SAST job was the 21-minute offender, and almost every finding pointed at code that predated the merge request. Both symptoms share a root: the job scans the whole repository on every pipeline.

Since the managed template runs Semgrep anyway, we replaced the job with our own Semgrep invocation and narrowed it to the files the MR actually changed: the diff against git merge-base with the default branch.

The catch is the clone. Runners clone shallow (new projects default to a depth of 20 commits), and git merge-base needs enough shared history to find the point where the branch forked. GIT_DEPTH can be raised per job, and 200 turned out to be deep enough for nearly every branch we saw. Nearly — so the script falls back to a full scan whenever the merge base can’t be resolved, instead of failing or, worse, silently scanning nothing:

#!/bin/sh
set -e
 
report() {
  semgrep scan --config p/default --gitlab-sast \
    --output gl-sast-report.json "$@"
}
 
git fetch --depth "${GIT_DEPTH:-200}" origin "$CI_DEFAULT_BRANCH"
 
if base=$(git merge-base "origin/$CI_DEFAULT_BRANCH" HEAD); then
  changed=$(git diff --name-only --diff-filter=d "$base" HEAD)
  if [ -n "$changed" ]; then
    report $changed
  else
    mkdir -p .ci-empty && report .ci-empty
  fi
else
  echo "merge base not found; falling back to a full scan"
  report .
fi

Three details in there earn their keep. --diff-filter=d drops deleted files, which Semgrep would otherwise try and fail to open. The empty-diff branch scans an empty directory, which is the laziest way to produce a valid report with zero vulnerabilities, so the artifact upload never warns about a missing file. And the fallback turns the worst case (a branch older than GIT_DEPTH) into a slow-but-correct scan.

The full scan didn’t disappear either. It moved to the scheduled pipeline from part 11, where 21 minutes against the whole repo bothers nobody.

This change, more than any rule tuning, is what fixed the wolf-crying. Before it, findings were mostly pre-existing issues in files the MR never opened, and ignoring them was the rational response. The problem was never the scanner’s precision — it was the scope. Once findings pointed at lines the author had just written, engineers read them, and most got fixed before a reviewer ever opened the MR.

Putting findings back in the MR widget

Scanning the right files is half of it; the other half is where the findings land, because a finding that only exists in a job log gets read once, maybe. Semgrep emits GitLab’s SAST report format natively (the --gitlab-sast flag already in the script) and declaring that file under artifacts:reports:sast makes GitLab treat the custom job like one of its own scanners: findings show up in the merge request security report and the pipeline’s Security tab.

sast-diff:
  stage: quality
  image: semgrep/semgrep:1.78.0
  variables:
    GIT_DEPTH: '200'
  script:
    - sh ci/semgrep-diff.sh
  artifacts:
    when: always
    reports:
      sast: gl-sast-report.json
  rules:
    - if: $CI_COMMIT_BRANCH && $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH

This is the same artifacts:reports family from part 3: there it was junit annotating failed tests on the MR, here it’s sast feeding the security widget. Two details in the job: when: always uploads the report even if you later decide the job should fail on findings, and the rules clause (part 5) skips it on the default branch, which the scheduled full scan already covers.

Dependency scanning with OSV-Scanner

SAST reads your code; the third front is the dependency tree. For that we use OSV-Scanner, which checks lockfiles against the OSV.dev vulnerability database and runs in seconds. Getting it into a job that fails when it should and survives when it shouldn’t took three rounds of fixes, all worth writing down.

First, the binary’s path. Between one image version and the next, the executable moved: one release kept it at /osv-scanner, a later one under /usr/local/bin. A small discovery loop is cheaper than chasing it on every upgrade:

for path in /osv-scanner /usr/local/bin/osv-scanner; do
  [ -x "$path" ] && scanner="$path" && break
done
[ -n "$scanner" ] || { echo "osv-scanner binary not found" >&2; exit 1; }

That relocation is also the case for pinning: run scanners on :latest and the pipeline changes behavior overnight, with no commit anywhere to explain it. Part 1 made this argument for build images; for scanners it applies double, since their whole job is to change your pipeline’s verdict. Pin the tag (ghcr.io/google/osv-scanner:v1.8.5) and upgrade on purpose.

Second, exit codes. Like most scanners, OSV-Scanner exits non-zero when it finds anything, and under set -e that kills the job before your gating logic runs. Turning set -e off for the whole script hides real breakage; instead, allowlist the exit codes you understand and treat everything else as an error:

set +e
"$scanner" --format json --lockfile pnpm-lock.yaml > osv-report.json
status=$?
set -e
 
case "$status" in
  0) echo "no known vulnerabilities" ;;
  1) echo "vulnerabilities found; gating on severity" ;;
  *) echo "osv-scanner failed (exit $status)" >&2; exit "$status" ;;
esac

Third, the severity gate. The JSON report groups vulnerabilities, and each group carries max_severity, a CVSS score encoded as a string ("7.5"). jq comparisons between a string and a number never error; they fall back to type ordering, where any string sorts above any number, so a naive >= 7 quietly classifies every finding as blocking. tonumber? // 0 handles both that and the groups where the field is missing entirely:

high=$(jq '[.results[].packages[].groups[]?.max_severity
  | tonumber? // 0] | map(select(. >= 7)) | length' osv-report.json)
 
if [ "$high" -gt 0 ]; then
  echo "blocking: $high finding(s) with CVSS >= 7.0" >&2
  exit 1
fi
echo "findings below threshold; not blocking"

Where to set the threshold is a team decision. We block merges at 7.0 and up (high and critical) and leave the rest visible in the log; we tried a lower bar once, and all it did was rebuild the wall of findings this whole part exists to avoid.

Where this leaves you

SAST now reads only the diff and reports into the same widget as the managed scanners, secrets are covered by the template, and dependencies only block above a severity you chose on purpose. All of this tells you what’s vulnerable today; it does nothing about the growing pile of outdated dependencies that will produce tomorrow’s findings. Keeping those current automatically (bots that open the upgrade MRs for you) is the other half of the job, and it’s part 13, the last one in the series.

References

• SAST
• Pipeline secret detection
• artifacts:reports:sast
• Pipeline settings: limit the number of changes fetched (GIT_DEPTH)
• Semgrep CLI reference