Longhorn Backup Policy¶
Longhorn provides replicated block storage across the three bare-metal nodes. Snapshot scheduling and off-site backup to Garage (S3) are enforced cluster-wide by a labelling script.
Label Groups¶
Label enforcement runs weekly via the Windmill f/heal/backup_label_enforcer job — the same policy as the longhorn-backup-labels.sh script — which ensures every PVC carries the correct labels. Longhorn maps labels to recurring job groups.
| Label | Applied to | Effect |
|---|---|---|
recurring-job.longhorn.io/source=enabled |
All PVCs | Opts the PVC into the recurring job system |
recurring-job-group.longhorn.io/default=enabled |
All PVCs | Joins the default jobs: monthly backup to Garage (0 5 1 * *, retain 2) + daily filesystem-trim (27 15 * * *) |
recurring-job-group.longhorn.io/weekly=enabled |
Selected namespaces | Weekly off-site backup to Garage (Fri 01:00, retain 2) |
recurring-job-group.longhorn.io/nosnapshots=enabled |
Cache / metrics / re-acquirable-media PVCs | No snapshots or backups; only snapshot-cleanup (30 2/4 * * *) |
Namespaces in the weekly group: appdaemon, bootstrap, envuassu, esphome, frigate, home-assistant, openldap, and others where data loss would be significant.
Source of truth: kubectl --context mdapi-prod -n longhorn-system get recurringjob.
Backup Target¶
Backups ship to the external Garage cluster. The backup target is configured as a Longhorn BackupTarget resource with the URL s3://harvester-backup@garage/.
Replica Policy¶
The cluster's default StorageClass, harvester-longhorn-2replicas, provisions new volumes at 2 replicas spread across nodes — but most live volumes now run 3 replicas in practice. A parallel 3-replica class — the plain longhorn StorageClass — is also in active use for core infra state (Grafana/VictoriaMetrics data, the jump seat's Postgres, the BIND zone store, ZNC, and a few web-hosting sites). 3-replica is the common case today; 2-replica is the class a new PVC lands on unless it opts into another one. The tv namespace's bulk media library used to run a permanent replica-1 exception, but that library has since moved entirely off Longhorn onto the Ceph tier (see Storage Architecture) — the last namespace-wide exception. A handful of individual PVCs still pin to the explicit harvester-longhorn-1replica class where a single replica is an accepted trade-off (Technitium's primary/secondary pair, OpenLDAP's LAM UI, Tabby's model cache, the GitLab runner's build scratch space). Rebuild concurrency is hard-capped at 2 per node, also deliberately: a wider cap would let a node failure trigger enough parallel rebuilds to saturate disk I/O on the surviving nodes.
Longhorn now runs exclusively on the three nodes' FusionIO ioMemory disks — the SATA/SSD tier was handed fully to Ceph in the August storage rework, so there's no longer a slower fallback disk class in the scheduling mix.
VM Backups¶
Longhorn also backs up KubeVirt VMs (e.g. the CipherTrust Manager appliance) as standard Longhorn volumes. This is what makes the CipherTrust Manager recoverable — a Longhorn snapshot restore brings the entire VM disk back, returning the appliance to service without rebuilding its configuration or re-enrolling its clients.
Monitoring Thresholds¶
The f/probe/daily_infra_health Windmill probe measures every backed-up volume's lastBackupAt age and exports it as longhorn_backup_age_days. A single LonghornBackupStale alert fires when a volume's backup is more than 40 days old — one flat threshold across every recurring-job group, since the metric carries no per-group label to split on. The 40-day window is sized to catch a missed monthly cycle while leaving margin for a run that starts a few days late.
PV Reclaim Policy¶
Production PVs default to reclaimPolicy: Delete — disk space is finite and recovery from a deleted PVC almost always means restoring from backup anyway. Retain is reserved for the small set of PVCs where no backup path exists and accidental deletion would be unrecoverable. See Storage → PV Reclaim Policy for the cross-check alert that guards this.
Mount-stall auto-recovery¶
A persistent Longhorn behavior on the current hardware: under disk / network pressure, individual pods occasionally end up with a stale mount — Longhorn reports the volume attached + healthy, dmesg on the node is clean, but the pod's writes return EIO or its ext4 silently flips read-only. The fix is always the same — delete the pod, the new one gets a fresh mount and writes resume.
Two automated paths cover this without a human in the loop:
| Pod state | What catches it | What it does |
|---|---|---|
CrashLoopBackOff with EIO / os error 5 / input/output error in recent logs |
longhorn_eio_autoheal Windmill flow (per-pod 30 min cooldown, global cap 6/hour) |
Deletes the pod; new pod attaches and recovers in ~20 s |
| Running but ext4 went read-only mid-flight (writes failing silently, no log signal) | LonghornVolumeReadOnlyRemount alert (longhorn_volume_file_system_read_only > 0 for 2 min) |
Pages with a runbook link; same pod-delete recovery |
The auto-healer's regex also has a sinceSeconds=300 Running-pod scan for the case where logs do carry the EIO signature even though the pod hasn't crashed. The Pushgateway exposes _last_run_* plus a rate_limit_hits counter so a runaway healer would itself trip BatchJobStale.
The complementary PodIOError log-pattern detector (see Monitoring → log-based alerts) fires on the same EIO signature from any namespace and tags the alert with namespace / pod / container, so an operator running the manual kubectl delete pod knows exactly which one before opening Lens.