Supply Chain Security¶

Two complementary controls sit on the supply chain into the cluster: preventing secrets from leaking into git in the first place, and scanning every image for known vulnerabilities before and after it reaches a node. The first runs at commit time and in CI; the second runs at build time and again four times a day against the running pods.

Secret Management is the sibling page — that one is about how secrets are handled at runtime (Akeyless customer fragment, External Secrets Operator). This page is about how secrets are kept out of source control, and how images are kept honest about what they contain.

Secret Leak Prevention¶

Four layers, each picking up what the layer above missed.

flowchart LR
    dev["Developer edit"] --> hook["L1: pre-commit hook<br/>gitleaks<br/>~/.git-hooks/pre-commit"]
    hook -->|leak found| reject["commit rejected<br/>locally"]
    hook -->|clean| push["git push"]
    push --> ci["L2: CI .pre stage<br/>secret-scan job<br/>blocks pipeline"]
    ci -->|leak found| fail["pipeline fails<br/>before build"]
    ci -->|clean| build["build / scan / deploy"]
    gi["L3: .gitignore baseline<br/>.env* / kubeconfig* / keys"] -.->|never staged| hook
    audit["L4: reactive audit<br/>gitleaks historical scan<br/>+ rotation runbook"] -.->|periodic sweep| ci

L1 — Pre-commit hook (local)¶

A single global hook covers every repo on the workstation. git config --global core.hooksPath ~/.git-hooks redirects the hook search path; ~/.git-hooks/pre-commit runs zricethezav/gitleaks against the staged diff and exits non-zero on any finding.

git config --global core.hooksPath ~/.git-hooks
# ~/.git-hooks/pre-commit runs gitleaks on the staged diff

This is the highest-impact layer: one install, every repo covered, zero per-repo configuration. A blocked commit never reaches the remote, so the secret never enters anyone's git history and there is no rotation to do.

L2 — CI `.pre` stage¶

Every repo pulls a shared .secret-scan base from a central ci-templates project (extends: .secret-scan) so the job stays identical everywhere and fixes ship once. It runs in the .pre stage so it gates every downstream job in the pipeline, and scans the same zricethezav/gitleaks tool against the full checkout, not just the diff, so it catches secrets that bypassed L1 (e.g. commits from another workstation, or from before the global hook was installed).

include:
  - project: mdapi/ci-templates
    file: /secret-scan.yml
    ref: main

secret-scan:
  extends: .secret-scan

The template pulls its image through the GitLab dependency proxy (${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/zricethezav/gitleaks:latest) rather than docker.io directly, so a transient docker.io rate-limit can't fail this gating .pre job — the failure mode that skipped a scheduled Renovate run in June. GIT_STRATEGY: clone + GIT_DEPTH: 1 sidestep a .pre-stage race where the default fetch strategy looks for a pipeline ref that hasn't propagated to Gitaly yet; runner-side transients get two retries, but a real gitleaks finding (allow_failure: false) always fails the pipeline. The tags: [mdapi] on the base job is load-bearing — the instance runner has accepted untagged jobs as a fallback since 2026-06-12, but that isn't a durable guarantee, and the separate gitlab-runner-builder project runner still hard-enforces run_untagged=false.

L3 — `.gitignore` baseline¶

A blanket .gitignore block applied to every repo covers the common shapes of credential files that operators do touch but should never commit: .env*, kubeconfig*, SSH private keys, PKCS12 bundles, .netrc. This catches the most common mode of accidental leak (operator drops a kubeconfig into a working dir for debugging, forgets, runs git add .) before L1 even sees the staged diff.

L4 — Reactive audit¶

When a secret does slip through all three preventive layers — typically a credential that pre-dates the hook deployment, or one that was committed before being rotated to Akeyless — the recovery pattern is:

Rotate at source (the upstream service: Pushover, GitLab, etc.) so the leaked value loses value the moment it's noticed.
Store the new value in Akeyless under /mdapi/<namespace>/<name>/<key>.
Refactor consumers to fetch via External Secrets Operator instead of hardcoding.
Patch live workloads in place (kubectl patch secret + restart) so the rotation propagates before the next reconcile.

A periodic gitleaks sweep across all local repos surfaces anything still hiding in history; combined with the three preventive layers above it makes net-new leaks rare and gives a documented response when one occurs.

CVE Scanning¶

Container image CVE scanning is integrated into every custom image CI pipeline using Syft (SBOM generation) and Grype (vulnerability matching).

Pipeline integration¶

flowchart LR
    code["Source / upstream bump"] --> build["build<br/>buildkit rootless<br/>multi-arch"] --> scan["scan<br/>Syft → SBOM<br/>Grype → CVE match"] --> digest["daily digest<br/>(grype curator)"]
    build --> push["push :latest<br/>registry.mdapi.ch/mdapi/"]
    scan --> artifact["SBOM artifact<br/>SPDX JSON<br/>7-day retention"]

The scan stage runs after the image is built but the result never blocks the pipeline (allow_failure: true) — production systems cannot be held hostage by upstream vulnerabilities that may have no fix yet. Findings are not notified per pipeline either: they are collected by the grype curator and delivered as a single daily digest, one message per day regardless of how many pipelines ran.

Every pipeline also carries a daily cache-buster — CI injects a CACHEBUST_DAY build-arg that invalidates a dedicated apk/apt upgrade layer once a day — plus a 4-hour schedule trigger, so base-image security patches reach the registry within hours of being published upstream without any manual rebuild.

New builder repo? The schedule is not optional

A builder repo without its 4-hourly pipeline schedule silently stops getting base-image patches — and this has now been forgotten five times (GitlabImageBuilderNoSchedule alert). Creating the repo is only done when the schedule exists:

Pick a collision-free minute (list every active schedule's cron across the mdapi group first; the gitlab_image_builder_health probe alerts on cron collisions).
Create the schedule on the default branch with cron <minute> */4 * * * and the canonical description Every 4h at minute <minute> hour-base */4 (the probe flags description/cron drift).
Verify it shows active: true and a sane next_run_at.

Covered images¶

Custom images live under registry.mdapi.ch/mdapi/; most have their own CI pipeline running the scan stage (nameserver builds via a simpler pipeline and relies on the runtime scan below instead). Representative subset:

Image	Base	Purpose
`keycloak`	`registry.access.redhat.com/ubi9-micro` (binary download, not image-based)	OIDC IdP
`chrony`	`alpine`	Stratum-1 NTP with optional GPS
`nameserver`	`debian:bookworm-slim`	BIND9 + Webmin
`unbound`	`alpine:latest` + `bind-tools`	Split-horizon resolver (shell + `dig` for the exec readiness probe)
`certspotter`	`debian:latest` (`golang` is only the build stage)	Certificate Transparency monitor
`autoconfig`	`python:alpine`	Mail-client auto-configuration
`opennic-tier2`	`debian:stable`	BIND9 OpenNIC Tier-2
`threadfin`	`ubuntu:latest` (`golang` build stage)	IPTV proxy for Plex
`joplin-mcp`	`python:slim`	HTTP/SSE MCP wrapper for Joplin
`znc`	`debian:bookworm-slim`	IRC bouncer

(Plus the static-site builders for the WordPress / Joomla properties, which inherit the same scan stage from the shared CI template.)

SBOM artifacts¶

Syft generates an SPDX JSON SBOM for each image build. This provides:

A point-in-time snapshot of every package installed in the image.
A queryable artifact for retroactive CVE analysis when new vulnerabilities are published against packages that scanned clean at build time.
Compliance evidence for software supply-chain auditing (an SBOM with a known build provenance is the artefact regulators and customers ask for).

SBOMs are stored in GitLab CI artifacts for 7 days per build; the registry mirror itself keeps the images far longer.

Registry mirror (zot)¶

Some custom images pull through the zot.mdapi.ch mirror rather than the GitLab registry directly — this keeps those images pullable while GitLab itself is down or mid-upgrade, and gives the scanners a single, stable registry to reason about. Others (e.g. nameserver, unbound, chrony, certspotter, autoconfig, opennic-tier2, threadfin, znc) still pull straight from registry.mdapi.ch.

Runtime CVE scanning¶

A Windmill script (f/security/pod_image_cve_scan) runs four times a day (02:25 / 08:25 / 14:25 / 20:25), each run scanning a rotating batch of 12 images (of roughly 150 tracked — every image currently running) against the live Grype database — a full sweep of every running pod's image takes about three days, with results cached and republished to Pushgateway on every run. This catches two classes of finding the CI scan can't:

Third-party images (from docker.io, ghcr.io, quay.io) that were never built in-cluster and so never went through the CI scan.
Newly published CVEs affecting images that passed scan at build time but now match a freshly disclosed vulnerability.

Findings feed the same daily digest, scoped by namespace and image, so runtime and build-time results arrive through one channel.

Ignore-list curators¶

Raw scanner output is too noisy to page on — most findings are known, accepted, or unfixable upstream. Two curator scripts turn the raw stream into a reviewable signal:

f/security/grype_ignore_curator — reconciles new CVE findings against per-repo grype ignore lists, opening auto-merge MRs that record each accepted finding in the repo where the image is built. The delta — genuinely new findings — is what lands in the daily digest.
f/security/gitleaks_ignore_curator — the sibling for secret scanning: maintains gitleaks ignore lists the same way, so recurring false positives are acknowledged once, in git, instead of re-triaged on every pipeline.

The ignore lists live in the repos themselves, so every acceptance decision is a reviewable commit with history — not a scanner-side setting that silently drifts.

Why two layers (build-time + runtime)¶

Build-time scanning gives a clean signal on the image you're about to ship; runtime scanning gives a clean signal on the image you're actually running, including everything the cluster pulled from upstream registries that you didn't build yourself. Neither alone is sufficient — a third-party image can introduce a CVE the build scan never saw, and a self-built image can pick up a new CVE long after the pipeline last ran. The combination converges on "everything running, every few days".

A clean result is a question, not an answer¶

Every scanner is only as good as the advisory data behind it, and that data does not cover every distribution release equally or at the same time.

This was measured directly: a second, independent scanner was run against images the platform already scans, at the same moment. On a mainstream image the two agreed on roughly four findings in five — and disagreed on how many were critical by close to a factor of two. On an image built on a very recent distribution release, one scanner reported several high-severity findings while the other identified the operating system correctly, examined every package, and reported clean. Its advisory database simply did not cover that release yet. It did not say "no data"; it said "clean".

Two practices follow from that:

A zero is read as a question. A clean verdict on a recent base image, or on an unusual one, is checked for coverage before it is trusted.
What matters is what can be fixed. Findings are ranked by whether an upstream fix exists and by image, not by raw count. A count of image × vulnerability pairs is inflated by every shared base layer; the unit of action is the image to rebuild. Findings with no fix available are a different conversation entirely — they are accepted, recorded and revisited, not queued as work.