GitOps with Fleet¶

Design: Git as the Source of Truth¶

Every resource in the cluster — Helm releases, raw manifests, RBAC, ingress rules, ExternalSecrets — is declared in the mdapi/fleet Git repository at https://gitlab.mdapi.ch/mdapi/fleet (public mirror). No kubectl apply is ever run manually. This makes the cluster state fully auditable, reproducible, and diff-able.

Rancher Fleet watches the repo and continuously reconciles the cluster state to match. A single GitRepo named mdapi-prod points at the repo's main branch; the Fleet controller itself runs on the Rancher local (management) cluster and pushes BundleDeployments down to mdapi-prod.

Pipeline¶

flowchart TD
    push["git push\n(main)"]

    subgraph gitlab["GitLab — mdapi/fleet"]
        main_br["main branch"]
    end

    subgraph rancher_mgmt["mdapi-mgmt (management cluster)"]
        fleet_ctrl["Fleet Controller\n(GitRepo: mdapi-prod)"]
        bd_prod["BundleDeployments"]
    end

    prod["mdapi-prod\nHelm releases + raw manifests"]

    push --> gitlab
    main_br --> fleet_ctrl
    fleet_ctrl --> bd_prod --> prod

Repository Structure¶

fleet/
├── gitlab/           # GitLab (vendored Helm chart)
├── bootstrap/        # GitLab's adjacent resources (CNPG, Valkey, ExternalSecrets)
├── windmill/         # Windmill (Helm)
├── keycloak/         # Keycloak
├── joplin/           # Joplin + MCP server
├── nameserver/       # BIND9 + Webmin
├── tv/               # Media stack
├── longhorn/         # Longhorn config
├── ...               # 100+ more bundles
└── README.md

Helm History and ErrApplied¶

Fleet does not need a deep Helm release history. The agent reads only the deployed revision (the one in the BundleDeployment's status.release) and the one before it, and a missing older revision is not an error. The real failure mode is a prune race: the agent's Helm storage reads release secrets from an informer cache but deletes them through the API. When a release at its history cap prunes, a revision that a concurrent upgrade already deleted can still be in the cache, the delete returns secrets "sh.helm.release.v1.<release>.vN" not found, and the BundleDeployment enters ErrApplied. The race can fire on any prune, whatever the cap.

History depth: every Helm-type fleet.yaml sets helm.maxHistory explicitly, because an unset key means Fleet's default of 2. The key is maxHistory, not historyMax — the wrong key is silently ignored. The value is currently 25, which is a convenience for helm history and helm get manifest --revision, not a requirement; a lower value is on trial.

Recovery: clear status.release on the BundleDeployment to force a fresh helm install:

kubectl --context mdapi-mgmt -n <cluster-ns> patch bundledeployment <name> \
  --type=merge --subresource=status \
  -p '{"status":{"release":""}}'

Renovate — Automated Image Updates¶

Renovate owns every container image in the fleet repo — manifest image: fields, Helm values.yml pins, and the fleet.yml chart versions — and raises a merge request when upstream ships a new build.

Floating tags (latest, stable, 3-alpine) are pinned to the @sha256 digest they currently resolve to: the tag keeps tracking upstream while git records the exact build, so a moving tag becomes a reviewable commit instead of a live patch. Pinned version tags get ordinary version updates plus the same digest pinning, so an in-place rebuild upstream still arrives as a commit. Minor and patch updates auto-merge; a major is always a merge request a human approves. A short MR-only list overrides this. The GitLab chart is on it, because GitLab must follow its required upgrade stops.

Reloader — Config-Change Rollouts¶

Stakater Reloader is the companion to Renovate: where Renovate rolls a workload when its image changes, Reloader rolls it when a ConfigMap or Secret it mounts changes. It watches every workload by default — the reloader.stakater.com/auto: "true" annotation is legacy and no longer required — and covers the hand-managed config maps Fleet delivers that would otherwise need a manual kubectl rollout restart to take effect.

Reloader rollouts show as Modified in Fleet

When Reloader triggers a rollout it patches the live workload, so Fleet briefly reports that bundle as Modified. Fleet correctDrift is off, so it only reports the drift — it does not fight Reloader. Renovate image bumps and VPA request mutations are the other two usual sources of benign Modified drift — when a bundle shows drift, one of those three is almost always the author.

cert-manager and DNS-01¶

cert-manager uses RFC 2136 dynamic updates to add _acme-challenge TXT records to BIND9. The target nameserver is 31.3.128.59:53 (external IP of ns.mdapi.ch, configured on the single cluster-wide letsencrypt-prod ClusterIssuer that every public-facing Certificate references).

The mdapi.ch zone requires allow-update { key "mdapi"; } in BIND9's named.conf. This is configured at https://gitlab.mdapi.ch/mdapi/fleet/-/tree/main/nameserver (public mirror) and applied via Fleet.

mdapi.ch uses offline DNSSEC signing managed by Webmin. Dynamic updates land in a zone journal without DNSSEC signatures. This is acceptable — Let's Encrypt verifies with a non-validating resolver.

external-dns and internal service discovery¶

The same RFC 2136 protocol is reused to keep the internal home.tillo.ch zone in sync with cluster state — but against a different nameserver, with a different TSIG key, and a much narrower update policy.

external-dns watches Service and Ingress objects and dynamically publishes A / AAAA / CNAME records into Technitium (the authoritative server for home.tillo.ch, see Hardware → DNS & DHCP Architecture). A dedicated TSIG key is registered in Technitium with an update policy scoped to the home.tillo.ch zone apex plus *.home.tillo.ch — the key cannot touch any other zone the server hosts.

The controller is opt-in by annotation: only resources explicitly tagged are published, so existing manual / DHCP records keep working while the system is burned in.

metadata:
  annotations:
    external-dns.alpha.kubernetes.io/manage: mdapi
    external-dns.alpha.kubernetes.io/target: 192.168.1.191    # optional VIP pin
    external-dns.alpha.kubernetes.io/ttl: "300"

The controller runs with policy=sync (full reconcile, including delete) and a txt-prefix=extdns- registry so ownership is recorded on a sibling TXT record — the earlier upsert-only bring-up period ended once the manual records were retired.

RFC 2136 mode needs AXFR enabled

external-dns must read the current zone to know which records already exist. With --rfc2136-tsig-axfr it pulls the zone over a TSIG-signed AXFR; without it, it cannot see existing records and re-issues an ADD for every managed record on each reconcile, never converging.

Non-Fleet Resources¶

A small number of resources apply directly with kubectl apply (against mdapi-mgmt) and are not managed by Fleet:

The mdapi/rancher-local repo — the mdapi-mgmt cluster's own cluster-issuer. Committed for traceability but not auto-reconciled.

GitLab itself (namespace bootstrap on mdapi-prod) is the gitlab/ bundle: the upstream chart is vendored into the repo so the Bundle stays small, and Fleet reconciles it like everything else. The docs.mdapi.ch ingress and its Certificate (referencing the cluster-wide letsencrypt-prod ClusterIssuer) are in the fleet/docs/ bundle at https://gitlab.mdapi.ch/mdapi/fleet/-/tree/main/docs (public mirror), reconciled by the mdapi-prod GitRepo.

Fleet's own settings (Rancher-side, not in git)¶

Rancher installs and upgrades Fleet itself, so two Fleet settings cannot live in the fleet repo. Both were set on 2026-11-06 after a burst of eight Renovate merges made the mdapi-prod fleet-agent render about 25 Helm bundles at once: it grew from 0.8 to 5.9 GiB with no memory limit and tipped node quo into a memory wedge.

Setting	Where it lives	Value
Agent reconcile workers	`rancher-config` ConfigMap in `cattle-system` on `mdapi-mgmt`, key `fleet` (Rancher passes it to the Fleet chart as values)	`agent.reconciler.workers.bundledeployment: "10"`, `drift: "10"` (chart default 50/50)
fleet-agent memory	management cluster `c-m-zk4x2xrw` (`mdapi`), `spec.fleetAgentDeploymentCustomization.overrideResourceRequirements` (mirrored on the provisioning cluster `fleet-default/mdapi`)	request `1Gi`, limit `4Gi`

Fewer workers is the cause fix: 100 concurrent workers rendered many charts at once and also exhausted the agent's client-side request budget (GET /version waited ~20 s per call; ~4 s after the change). The limit comes second on purpose: at 50/50 workers an OOM restart would re-sync every bundle at once and could crash-loop.

Two traps when changing them

A new agentWorkers value reaches the agent only after fleet-controller reloads its config (delete its pod; a Helm revision that only changes the ConfigMap does not restart it) and the agent bundle is regenerated (bump spec.redeployAgentGeneration on the Fleet Cluster, the UI's Force Update). Before the reload the controller rewrites fleet-agent-mdapi with the old value and the agent reverts within minutes.
For this Harvester-provided cluster, Rancher reads fleetAgentDeploymentCustomization from the management cluster; setting it only on the provisioning cluster changes nothing.

Rollback: remove the fleet key from rancher-config (Rancher redeploys Fleet with the chart defaults), delete the fleet-controller pod and bump redeployAgentGeneration; remove fleetAgentDeploymentCustomization from both cluster objects. The Rancher chart renders only priorityClassName into that ConfigMap, so a Rancher upgrade should keep the fleet key, but that is not yet proven: check kubectl -n cattle-fleet-system get cm fleet-controller -o jsonpath='{.data.config}' | grep -A3 agentWorkers after each one.