GitLab¶

GitLab EE runs in the bootstrap namespace, deployed via the official Helm chart with custom images for the webservice and sidekiq components.

Custom Images¶

Two components run custom EE images built from internal Dockerfiles (each FROM the upstream gitlab-<svc>-ee image). CI builds them and pushes to the GitLab project registry (registry.mdapi.ch/mdapi/); a mirror job then copies them to zot.mdapi.ch, which is what the cluster actually pulls from:

zot.mdapi.ch/mdapi/gitlab-webservice-ee
zot.mdapi.ch/mdapi/gitlab-sidekiq-ee

The Helm values pin both to a version tag on zot (:vX.Y.Z) with pullPolicy: IfNotPresent and the zot-registry pull secret. Rollout is therefore not automatic: a freshly built image sits in zot until the tag: is bumped in the values and helm upgrade is re-run. Pinning to zot rather than the GitLab-served registry keeps the images pullable even while GitLab itself is mid-upgrade.

Upgrade Procedure¶

Upgrade one minor version at a time — GitLab requires sequential minor upgrades, and the version compatibility matrix on the upstream release notes is the source of truth for safe hops.

helm repo update
helm upgrade gitlab gitlab/gitlab -f bootstrap/gitlab-values.yaml --version <version>

Automated upgrades¶

Windmill flow f/gitlab/upgrade_auto_flow runs the full upgrade chain end-to-end, currently triggered on demand only (no recurring Windmill schedule — see Windmill → Representative schedules). The flow short-circuits when there is nothing safe to do, otherwise:

flowchart LR
    detect["upgrade_detect\nchart + app version\nsafe-hop check"]
    branch{{"safe_hop\n&& needs_upgrade?"}}
    skip["skip\n(reason logged)"]
    bump["upgrade_bump_dockerfile\nFROM …:vX.Y.Z"]
    ci["upgrade_wait_ci\nbuildkit pipelines"]
    mirror["upgrade_mirror_zot\nregistry → zot\n:vX.Y.Z + :latest"]
    helm["upgrade_helm\nhelm upgrade --version"]
    rr["upgrade_rollout_restart\nwebservice + sidekiq"]
    health["upgrade_health\nGitLab /-/health"]

    detect --> branch
    branch -->|no| skip
    branch -->|yes| bump --> ci --> mirror --> helm --> rr --> health

Safe-hop policy: upgrade_detect only allows automation when the target is either a patch (same major.minor) or exactly one minor ahead. Multi-minor jumps need a manual run because GitLab requires sequential minor upgrades.

Why a separate rollout-restart step: after the new image is mirrored to zot and the chart is upgraded, the rollout-restart step explicitly rolls the webservice and sidekiq Deployments so the freshly mirrored image is the one actually running — rather than relying on the controller to notice on its own.

The semi-automated flow f/gitlab/gitlab_upgrade_flow (with manual approvals between stages) still exists for the multi-minor case.

GitLab Runners¶

Type	API visibility	Notes
Group	Visible to non-admin `/api/v4/runners`	Used for group-scoped pipelines
Instance (Helm-managed)	Invisible to non-admin API	Handles the majority of CI jobs

Instance runners do not appear in non-admin runner list API responses. To verify health, use the Rancher API to check pod status directly, or mint a temporary admin PAT.

All runners register with run_untagged=false — every job must declare tags: [mdapi] or it queues forever with no error. Shared CI templates carry the tag via extends: so individual jobs don't have to remember it.

Object Storage & Backups¶

All ten GitLab object stores (artifacts, registry, LFS, uploads, packages, and friends) plus the CI runner cache live on the in-cluster Ceph RGW gateway (ceph-objectstore in the rook-ceph namespace) — low-latency S3 right next to the workload.

Backups are the deliberate exception: the nightly toolbox backup ships to the gitlab-backups bucket on the external Garage cluster, which is replicated off-site daily — so a restore never depends on the cluster that just died.

Separately from backups, 12 public repositories push-mirror to github.com/tillo/* via per-repository SSH deploy keys.

CI Security Scanning¶

All custom Docker image pipelines include Grype + Syft:

Stage order: build → scan → notify
Syft generates an SPDX JSON SBOM, stored as a 7-day CI artifact
Grype checks for HIGH/CRITICAL CVEs; findings roll into the daily CVE digest
allow_failure: true — CVEs never block a deployment

Applies to most custom-image pipelines across the mdapi GitLab group — see Supply Chain Security → Covered images for a representative subset. (GitLab webservice/sidekiq build via a simpler build → mirror pipeline and are covered only by the runtime scan.)

API Access¶

Standing tokens are scoped to read-only operations. For full API access, mint a temporary PAT via the toolbox pod:

kubectl --context mdapi-prod -n bootstrap exec <toolbox-pod> -- gitlab-rails runner "
token = PersonalAccessToken.create!(user: User.find_by_username('<your-username>'), name: 'tmp', scopes: ['api'], expires_at: 1.day.from_now)
puts token.token
"

Zot Registry¶

Endpoint: zot.mdapi.ch
Auth: zot / password in Akeyless at /mdapi/zot/zot-auth/password
Pull secret: zot-registry in bootstrap ns (created imperatively)
skopeo quirk: list-tags always needs --creds zot:PASSWORD even after skopeo login

GitLab Pages¶

GitLab Pages runs as a separate deployment (gitlab-gitlab-pages) sharing the cluster's only ingress controller (RKE2-builtin nginx at 192.168.1.191). The chart-managed wildcard ingress for *.pages.mdapi.ch is on the same class. Custom domains (e.g. docs.mdapi.ch) require:

An explicit Ingress with ingressClassName: nginx pointing to gitlab-gitlab-pages:8090
A _gitlab-pages-verification-code.<domain> TXT record in DNS, verified by GitLab
A cert-manager Certificate for the custom domain

The public documentation site is at docs.mdapi.ch. Its manifests are at https://gitlab.mdapi.ch/mdapi/fleet/-/tree/main/docs (public mirror), reconciled by Fleet.