Skip to content

GitLab

GitLab EE runs in the bootstrap namespace, deployed via the official Helm chart with custom images for the webservice and sidekiq components.

Custom Images

Two components run custom EE images built from internal Dockerfiles (each FROM the upstream gitlab-<svc>-ee image). CI builds them and pushes to the GitLab project registry (registry.mdapi.ch/mdapi/); a mirror job then copies them to zot.mdapi.ch, which is what the cluster actually pulls from:

  • zot.mdapi.ch/mdapi/gitlab-webservice-ee
  • zot.mdapi.ch/mdapi/gitlab-sidekiq-ee

The Helm values pin both to a version tag on zot (:vX.Y.Z) with pullPolicy: IfNotPresent and the zot-registry pull secret. Rollout is therefore not automatic: a freshly built image sits in zot until the tag: is bumped in the values and helm upgrade is re-run. Pinning to zot rather than the GitLab-served registry keeps the images pullable even while GitLab itself is mid-upgrade.

Upgrade Procedure

Upgrade one minor version at a time — GitLab requires sequential minor upgrades, and the version compatibility matrix on the upstream release notes is the source of truth for safe hops.

helm repo update
helm upgrade gitlab gitlab/gitlab -f bootstrap-gitlab-values.yaml --version <version>

Automated upgrades — every 4h

Windmill flow f/gitlab/upgrade_auto_flow runs unattended every 4h (cron 0 0 */4 * * *, no_flow_overlap=true). The flow short-circuits when there is nothing safe to do, otherwise runs the full upgrade chain end-to-end:

flowchart LR
    detect["upgrade_detect\nchart + app version\nsafe-hop check"]
    branch{{"safe_hop\n&& needs_upgrade?"}}
    skip["skip\n(reason logged)"]
    bump["upgrade_bump_dockerfile\nFROM …:vX.Y.Z"]
    ci["upgrade_wait_ci\nbuildkit pipelines"]
    mirror["upgrade_mirror_zot\nregistry → zot\n:vX.Y.Z + :latest"]
    helm["upgrade_helm\nhelm upgrade --version"]
    rr["upgrade_rollout_restart\nwebservice + sidekiq"]
    health["upgrade_health\nGitLab /-/health"]

    detect --> branch
    branch -->|no| skip
    branch -->|yes| bump --> ci --> mirror --> helm --> rr --> health

Safe-hop policy: upgrade_detect only allows automation when the target is either a patch (same major.minor) or exactly one minor ahead. Multi-minor jumps need a manual run because GitLab requires sequential minor upgrades.

Why a separate rollout-restart step: after the new image is mirrored to zot and the chart is upgraded, the rollout-restart step explicitly rolls the webservice and sidekiq Deployments so the freshly mirrored image is the one actually running — rather than relying on the controller to notice on its own.

The semi-automated flow f/gitlab/gitlab_upgrade_flow (with manual approvals between stages) still exists for the multi-minor case.

GitLab Runners

Type API visibility Notes
Group Visible to non-admin /api/v4/runners Used for group-scoped pipelines
Instance (Helm-managed) Invisible to non-admin API Handles the majority of CI jobs

Instance runners do not appear in non-admin runner list API responses. To verify health, use the Rancher API to check pod status directly, or mint a temporary admin PAT.

CI Security Scanning

All custom Docker image pipelines include Grype + Syft:

  • Stage order: build → scan → notify
  • Syft generates an SPDX JSON SBOM, stored as a 7-day CI artifact
  • Grype checks for HIGH/CRITICAL CVEs; Pushover notification on findings
  • allow_failure: true — CVEs never block a deployment

Applies to: GitLab webservice/sidekiq, Keycloak, Chrony, Certspotter, Autoconfig, OpenNIC, Mirror, Threadfin, Joplin MCP, ZNC.

API Access

The locally-cached ~/.git-credentials token is scoped to read-only operations only. For full API access, mint a temporary PAT via the toolbox pod:

kubectl --context mdapi-prod -n bootstrap exec <toolbox-pod> -- gitlab-rails runner "
token = PersonalAccessToken.create!(user: User.find_by_username('<your-username>'), name: 'tmp', scopes: ['api'], expires_at: 1.day.from_now)
puts token.token
"

openwrt project ID: 33 (mdapi/openwrt)

Zot Registry

  • Endpoint: zot.mdapi.ch
  • Auth: zot / password in Akeyless at /mdapi/zot/zot-auth/password
  • Pull secret: zot-registry in bootstrap ns (created imperatively)
  • skopeo quirk: list-tags always needs --creds zot:PASSWORD even after skopeo login

GitLab Pages

GitLab Pages runs as a separate deployment (gitlab-gitlab-pages) sharing the cluster's only ingress controller (RKE2-builtin nginx at 192.168.1.191). The chart-managed wildcard ingress for *.pages.mdapi.ch is on the same class. Custom domains (e.g. docs.mdapi.ch) require:

  1. An explicit Ingress with ingressClassName: nginx pointing to gitlab-gitlab-pages:8090
  2. A _gitlab-pages-verification-code.<domain> TXT record in DNS, verified by GitLab
  3. A cert-manager Certificate for the custom domain

The public documentation site is at docs.mdapi.ch. Its manifests are at https://gitlab.mdapi.ch/mdapi/fleet/-/tree/main/docs (public mirror), reconciled by Fleet.