Skip to content

GitLab

GitLab EE runs in the bootstrap namespace, deployed via the official Helm chart on stock upstream images.

Container Images

Every component runs the upstream CNG image that the chart itself selects (registry.gitlab.com/gitlab-org/build/cng/…). The Helm values carry no version pin for the webservice, sidekiq or toolbox components, so the chart version alone determines which GitLab version runs — there is no second place to remember on an upgrade, and no way to end up with a mismatched pair of components.

The one deliberate exception is gitlab-exporter, pinned to its own v16.8.0 line (the chart omits the v prefix that image's tags use). It is not version-coupled to GitLab itself.

Two components — webservice and sidekiq — previously ran custom images built from internal Dockerfiles and mirrored to zot.mdapi.ch. They were retired once their only remaining content turned out to be inert: the sidekiq image set concurrency via ENV, which the chart already overrides in the pod spec, and the webservice image's CI-cache patches had stopped matching anything upstream. The CI cache limit they used to carry now lives where it belongs, as an instance setting (ci_max_caches_per_job). Retiring them also broke a circular dependency, since the builds lived in GitLab's own registry — which is precisely what the mirror existed to work around.

Upgrade Procedure

Upgrade along GitLab's required upgrade stops, not one minor at a time. Since 17.5 the stops fall on x.2, x.5, x.8 and x.11: hop to the first required stop after your current version, let the background migrations drain, then repeat. Minors in between are skipped outright — from 19.0 the path is 19.2 → 19.3, never touching 19.1. Always take the latest patch of whichever version you land on. The upstream upgrade-path documentation is the source of truth.

helm repo update
helm upgrade gitlab gitlab/gitlab -f bootstrap/gitlab-values.yaml --version <version>

Automated upgrades

Windmill flow f/gitlab/upgrade_auto_flow runs the full upgrade chain end-to-end, currently triggered on demand only (no recurring Windmill schedule — see Windmill → Representative schedules). The flow short-circuits when there is nothing safe to do, otherwise:

flowchart LR
    detect["upgrade_detect\nchart + app version\nsafe-hop check"]
    branch{{"safe_hop\n&& needs_upgrade?"}}
    skip["skip\n(reason logged)"]
    bump["upgrade_bump_dockerfile\nFROM …:vX.Y.Z"]
    ci["upgrade_wait_ci\nbuildkit pipelines"]
    mirror["upgrade_mirror_zot\nregistry → zot\n:vX.Y.Z + :latest"]
    helm["upgrade_helm\nhelm upgrade --version"]
    rr["upgrade_rollout_restart\nwebservice + sidekiq"]
    health["upgrade_health\nGitLab /-/health"]

    detect --> branch
    branch -->|no| skip
    branch -->|yes| bump --> ci --> mirror --> helm --> rr --> health

Safe-hop policy: upgrade_detect only allows automation when the target is either a patch (same major.minor) or exactly one minor ahead; anything further needs a manual run.

Vestigial steps: the upgrade_bump_dockerfile, upgrade_wait_ci and upgrade_mirror_zot stages exist because the webservice and sidekiq components once ran custom images. Those images are retired, so the components the chart now installs are unaffected by that build-and-mirror chain. The flow is on-demand only and has not been re-cut around the required-stop model described above — a manual helm upgrade along the stops is the current path.

The semi-automated flow f/gitlab/gitlab_upgrade_flow (with manual approvals between stages) still exists for the multi-stop case.

GitLab Runners

Type API visibility Notes
Group Visible to non-admin /api/v4/runners Used for group-scoped pipelines
Instance (Helm-managed) Invisible to non-admin API Handles the majority of CI jobs

Instance runners do not appear in non-admin runner list API responses. To verify health, use the Rancher API to check pod status directly, or mint a temporary admin PAT.

The instance runner has accepted untagged jobs as a fallback since 2026-06-12, but that isn't a durable guarantee — shared CI templates still carry tags: [mdapi] via extends: so a job is never left depending on runner-registration state. (The separate gitlab-runner-builder project runner does hard-enforce run_untagged=false, so it never steals ordinary mdapi-tagged jobs.)

Authentication

GitLab supports SSO login via the MDAPI Keycloak realm (OIDC, idp.mdapi.ch), alongside standard Google and GitHub OAuth2 — all three are configured as omniauth providers in the Helm values.

Object Storage & Backups

All ten GitLab object stores (artifacts, registry, LFS, uploads, packages, and friends) plus the CI runner cache live on the in-cluster Ceph RGW gateway (ceph-objectstore in the rook-ceph namespace) — low-latency S3 right next to the workload.

Backups are the deliberate exception: the nightly toolbox backup ships to the gitlab-backups bucket on the in-cluster Ceph RGW, which is replicated off-site daily — so a restore never depends on GitLab itself being up.

Separately from backups, 12 public repositories push-mirror to github.com/tillo/* via per-repository SSH deploy keys.

CI Security Scanning

All custom Docker image pipelines include Grype + Syft:

  • Stage order: build → scan → notify
  • Syft generates an SPDX JSON SBOM, stored as a 7-day CI artifact
  • Grype checks for HIGH/CRITICAL CVEs; findings roll into the daily CVE digest
  • allow_failure: true — CVEs never block a deployment

Applies to most custom-image pipelines across the mdapi GitLab group — see Supply Chain Security → Covered images for a representative subset. (GitLab webservice/sidekiq build via a simpler build → mirror pipeline and are covered only by the runtime scan.)

API Access

Standing tokens are scoped to read-only operations. For full API access, mint a temporary PAT via the toolbox pod:

kubectl --context mdapi-prod -n bootstrap exec <toolbox-pod> -- gitlab-rails runner "
token = PersonalAccessToken.create!(user: User.find_by_username('<your-username>'), name: 'tmp', scopes: ['api'], expires_at: 1.day.from_now)
puts token.token
"

Zot Registry

  • Endpoint: zot.mdapi.ch
  • Auth: zot / password in Akeyless at /mdapi/zot/zot-auth/password
  • Pull secret: zot-registry in bootstrap ns (created imperatively)
  • skopeo quirk: list-tags always needs --creds zot:PASSWORD even after skopeo login

GitLab Pages

GitLab Pages runs as a separate deployment (gitlab-gitlab-pages) sharing the cluster's only ingress controller (RKE2-builtin nginx at 192.168.1.191). The chart-managed wildcard ingress for *.pages.mdapi.ch is on the same class. Custom domains (e.g. docs.mdapi.ch) require:

  1. An explicit Ingress with ingressClassName: nginx pointing to gitlab-gitlab-pages:8090
  2. A _gitlab-pages-verification-code.<domain> TXT record in DNS, verified by GitLab
  3. A cert-manager Certificate for the custom domain

The public documentation site is at docs.mdapi.ch. Its manifests are at https://gitlab.mdapi.ch/mdapi/fleet/-/tree/main/docs (public mirror), reconciled by Fleet.