Skip to content

GitLab

GitLab EE runs in the bootstrap namespace, deployed via the official Helm chart (vendored into the fleet repo and reconciled by Fleet) on stock upstream images.

Container Images

Every component runs the upstream CNG image that the chart itself selects (registry.gitlab.com/gitlab-org/build/cng/…). The Helm values carry no version pin for the webservice, sidekiq or toolbox components, so the chart version alone determines which GitLab version runs — there is no second place to remember on an upgrade, and no way to end up with a mismatched pair of components.

The one deliberate exception is gitlab-exporter, pinned to its own v16.8.0 line (the chart omits the v prefix that image's tags use). It is not version-coupled to GitLab itself.

Two components — webservice and sidekiq — previously ran custom images built from internal Dockerfiles and mirrored to zot.mdapi.ch. They were retired once their only remaining content turned out to be inert: the sidekiq image set concurrency via ENV, which the chart already overrides in the pod spec, and the webservice image's CI-cache patches had stopped matching anything upstream. The CI cache limit they used to carry now lives where it belongs, as an instance setting (ci_max_caches_per_job). Retiring them also broke a circular dependency, since the builds lived in GitLab's own registry — which is precisely what the mirror existed to work around.

Upgrade Procedure

Upgrade along GitLab's required upgrade stops, not one minor at a time. Since 17.5 the stops fall on x.2, x.5, x.8 and x.11: hop to the first required stop after your current version, let the background migrations drain, then repeat. Minors in between are skipped outright — from 19.0 the path is 19.2 → 19.5, never touching 19.1, 19.3 or 19.4. Always take the latest patch of whichever version you land on. The upstream upgrade-path documentation is the source of truth.

An upgrade is a merge request. The chart pin in the fleet repo's gitlab/ bundle moves together with the vendored chart, Renovate proposes the bump as an MR a human merges, and Fleet rolls it out. Patch bumps arrive ready to merge; a minor bump needs the chart re-vendored and a check against the required-stop path. A hand-run helm upgrade is not a path at all — Fleet reverts it.

A Windmill flow, f/gitlab/upgrade_auto_flow, plans and verifies that MR. It never upgrades anything itself. It reads the running chart, the pin on main and the upstream chart index. It names the newest release up to the next required stop, and fails if the pin skips one. After a merge it waits until Fleet has rolled the pin and /-/readiness answers 200. The older flow that ran helm upgrade directly is archived.

GitLab Runners

Type API visibility Notes
Group Visible to non-admin /api/v4/runners Used for group-scoped pipelines
Instance (Helm-managed) Invisible to non-admin API Handles the majority of CI jobs

Instance runners do not appear in non-admin runner list API responses. To verify health, use the Rancher API to check pod status directly, or mint a temporary admin PAT.

The instance runner accepts untagged jobs as a fallback, but that isn't a durable guarantee — shared CI templates still carry tags: [mdapi] via extends: so a job is never left depending on runner-registration state. (The separate gitlab-runner-builder project runner does hard-enforce run_untagged=false, so it never steals ordinary mdapi-tagged jobs.)

Authentication

GitLab supports SSO login via the MDAPI Keycloak realm (OIDC, idp.mdapi.ch), alongside standard Google and GitHub OAuth2 — all three are configured as omniauth providers in the Helm values.

Object Storage & Backups

All ten GitLab object stores (artifacts, registry, LFS, uploads, packages, and friends) plus the CI runner cache live on the in-cluster Ceph RGW gateway (ceph-objectstore in the rook-ceph namespace) — low-latency S3 right next to the workload.

Backups are the deliberate exception: the nightly toolbox backup ships to the gitlab-backups bucket on the in-cluster Ceph RGW, which is replicated off-site daily — so a restore never depends on GitLab itself being up.

Separately from backups, a set of public repositories push-mirror to github.com/tillo/* via per-repository SSH deploy keys.

CI Security Scanning

All custom Docker image pipelines run a Grype scan:

  • Stage order: build → scan → notify
  • Grype matches the built image against its vulnerability database and writes grype-report.json, retained as a 30-day CI artifact. The shared template reads the registry reference directly; a few older per-project jobs still build an SBOM with Syft first
  • Grype flags HIGH/CRITICAL CVEs, and the job sends a per-pipeline notification for fixable HIGH/CRITICAL findings — wont-fix CVEs are deliberately excluded and handled by the daily curator. Findings also roll into the daily CVE digest
  • allow_failure: true — CVEs never block a deployment

Applies to every custom-image pipeline across the mdapi GitLab group — see Supply Chain Security → Covered images for a representative subset.

API Access

Standing tokens are scoped to read-only operations. For full API access, mint a temporary PAT via the toolbox pod:

kubectl --context mdapi-prod -n bootstrap exec <toolbox-pod> -- gitlab-rails runner "
token = PersonalAccessToken.create!(user: User.find_by_username('<your-username>'), name: 'tmp', scopes: ['api'], expires_at: 1.day.from_now)
puts token.token
"

Zot Registry

zot.mdapi.ch is the estate's OCI registry: CI publishes images there (mdapi/<project> from the default branch, mdapi-dev/<project> from other refs) and workloads pull from it. GitLab's own container registry is read-only — writes are rejected, but existing images remain pullable — so it no longer takes new images.

  • Endpoint: zot.mdapi.ch
  • Auth: zot / password in Akeyless at /mdapi/zot/zot-auth/password
  • Pull secret: zot-registry in bootstrap ns (created imperatively)
  • skopeo quirk: list-tags always needs --creds zot:PASSWORD even after skopeo login

GitLab Pages

GitLab Pages runs as a separate deployment (gitlab-gitlab-pages) sharing the cluster's public ingress controller (RKE2-builtin ingress-nginx, class nginx, behind the VIP 192.168.1.191). The chart-managed wildcard ingress for *.pages.mdapi.ch is on the same class. Custom domains (e.g. docs.mdapi.ch) require:

  1. An explicit Ingress with ingressClassName: nginx pointing to gitlab-gitlab-pages:8090
  2. A _gitlab-pages-verification-code.<domain> TXT record in DNS, verified by GitLab
  3. A cert-manager Certificate for the custom domain

The public documentation site is at docs.mdapi.ch. Its manifests are at https://gitlab.mdapi.ch/mdapi/fleet/-/tree/main/docs (public mirror), reconciled by Fleet.