Skip to content

Backups & Disaster Recovery

Backup is a cross-cutting concern — every data class in the stack (block volumes, relational databases, GitLab object stores, cluster configuration, network device snapshots) has its own native backup mechanism, but they all converge on the same destinations and the same off-site path. Reading any one page in this site only shows you that page's slice; this one is the cross-section.

The design goal is two independent off-site copies for every byte that matters, with a recovery path that doesn't depend on any single piece of on-prem hardware staying up. The fast-restore tier (local snapshots) is a convenience on top of that, not a substitute for it.

Architecture at a Glance

flowchart LR
    subgraph apps["Workloads"]
        pvc["Block PVCs<br/>(Longhorn)"]
        ceph_pvc["Ceph PVCs<br/>(RBD · CephFS via Rook)"]
        cnpg["PostgreSQL<br/>(CloudNativePG x5)"]
        gl["GitLab object stores<br/>(uploads · artifacts · LFS · registry · …)"]
        net["Network devices + cluster config<br/>(BPI-R4 · TrueNAS · Cisco · Technitium · OpenObserve · Scrypted)"]
    end

    subgraph snap["L0 — Local snapshots"]
        lh_snap["Longhorn snapshots<br/>(per-volume, on-cluster)"]
    end

    subgraph onsite["L1 — On-site backup (external Garage S3)"]
        g_hb["harvester-backup"]
        g_ceph["ceph-backup"]
        g_pg["*-pg-backup<br/>(barman stores)"]
        g_gl["gitlab-backups"]
        g_cb["config-backups"]
    end

    subgraph off["L2 + L3 — Off-site (daily rclone, 04:00 Europe/Zurich)"]
        b2["Backblaze B2"]
        sftp["o2switch SFTP"]
    end

    pvc -->|"recurring snapshot"| lh_snap
    pvc -->|"recurring backup"| g_hb
    ceph_pvc -->|"nightly CSI snapshot +<br/>restic (VolSync)"| g_ceph
    cnpg -->|"continuous WAL +<br/>nightly base"| g_pg
    gl -->|"daily backup-utility"| g_gl
    net -->|"daily Windmill snapshot"| g_cb
    onsite --> b2
    onsite --> sftp

Solid arrows are scheduled data flow. The dashed-line "L0 → L1" relationship (snapshot then backup) is collapsed in the diagram because Longhorn handles both as recurring jobs on the same volume — they're configured side-by-side, not in series.

The Four Layers

L0 — Local Longhorn snapshots are copy-on-write snapshots of the underlying volume image, kept on the same nodes as the live data. They cost almost nothing in time and storage, and a restore is a volume.spec.fromBackup flip plus a pod restart. They're the right tool for "I just broke a PVC, get the previous state back" but they're worthless against losing the cluster — they don't leave the cluster.

L1 — On-site backup to the external Garage cluster moves the data off the production nodes onto separate hardware (the salt + pepper TrueNAS hosts, with a quorum-only in-cluster witness pod completing the 3-node layout — the witness holds no data). This is the layer that survives a Harvester-node-class failure. The Garage cluster itself is 3-node, so it tolerates losing one node without losing the data. Each consumer has its own bucket scoped by its own access key — see Storage → Garage S3 for the full bucket list.

L2 — Off-site to Backblaze B2 is a daily rclone sync of every L1 bucket to a Backblaze B2 bucket. B2 is in a different jurisdiction and a different failure domain from the home lab. This is the layer that survives losing the entire on-prem site.

L3 — Off-site to o2switch SFTP is the same daily sync, fanned out to an SFTP account at a second provider. Two independent off-site destinations means losing either one (provider outage, credential revocation, account-level mishap) still leaves a recoverable copy.

The L2/L3 fan-out is a single Windmill flow (f/config_backup/garage_offsite_backup) that runs at 04:00 UTC daily. It is idempotent — re-runs only transfer changed objects — and exception-handlers wrap each bucket-pair so a failure on one bucket doesn't abort the others.

By Data Class

Block PVCs — Longhorn

PVCs are labelled at provision time, mapping each one to Longhorn recurring-job groups. Every PVC gets at least the default group (monthly backup to Garage, retain 2, plus a daily filesystem trim); namespaces holding active state (Home Assistant, GitLab, Frigate, OpenLDAP, ...) additionally get weekly (weekly backup, Fridays 01:00, retain 2). Cache, metrics and re-acquirable-media PVCs (Valkey, metrics stores, the bulk media library) get nosnapshots, whose only recurring job is snapshot cleanup — rebuilding them from scratch is faster than restoring.

Backups ship to the harvester-backup bucket on the external Garage cluster via Longhorn's BackupTarget resource (s3://harvester-backup@garage/). See Longhorn Backup Policy for the per-label schedule.

Ceph PVCs — VolSync

Volumes on the in-cluster Ceph tier (RBD block and CephFS via Rook) don't participate in Longhorn's recurring jobs, so they get their own equivalent path: VolSync drives a per-PVC ReplicationSource that takes a nightly CSI snapshot for consistency and ships the contents with restic to the ceph-backup bucket on the external Garage cluster — an on-site (L1) copy only, since ceph-backup isn't currently in the off-site fan-out. Primary-data volumes (the OwnCloud file store) are enrolled; the bulk media library is deliberately not — that data is re-acquirable and doesn't justify off-site cost. The newer paperless-data-ceph PVC (Paperless DMS documents, moved to Ceph RBD) is on this tier too but has no ReplicationSource yet — it is not currently backed up at all.

PostgreSQL — CloudNativePG with Barman

The stack runs five CloudNativePG clusters: bootstrap/gitlab-pg (the GitLab Rails database), home-assistant/ha-recorder-pg (the Home Assistant recorder), paperless/paperless-pg (the Paperless DMS metadata store), jump/guac-pg (the Guacamole session store behind the jump seat) and monitoring/grafana-pg (Grafana's unified-storage backend). The first two (gitlab-pg, ha-recorder-pg) use continuous WAL streaming via Barman Cloud plus a nightly ScheduledBackup; paperless-pg currently has WAL archiving configured but no ScheduledBackup CR, so it has no base backup yet. This gives gitlab-pg and ha-recorder-pg a point-in-time restore path independent of any Longhorn snapshot of the underlying PVC — the PVC backup is a consistency hedge, the Barman store is the actual recovery substrate. guac-pg and grafana-pg carry no Barman configuration; their persistence relies on the Longhorn backups of their Longhorn-replicated volumes (grafana-pg at 2 replicas, guac-pg at 3).

GitLab object stores — backup-utility

GitLab Rails has its own backup-utility that snapshots every internal object store (uploads, artifacts, LFS, packages, dependency proxy, container registry, terraform state, pages) into a single tarball and ships it to the gitlab-backups bucket. The job runs on schedule from the gitlab-toolbox pod. Combined with the CNPG backup of gitlab-pg and the Longhorn backup of the GitLab PVCs, a GitLab restore can pick whichever consistency boundary is cheapest for the recovery scenario.

Rancher cluster config

Rancher's management-plane state — CRDs, RBAC, secrets, project mappings, Fleet bindings, downstream cluster registrations, Keycloak integrations — has no dedicated backup path today. The Rancher Backup Operator that would dump it to a rancher-backup bucket on a daily schedule isn't installed on the cluster; this data class is a known gap against the design goal of two independent off-site copies, not yet closed.

Network devices and cluster configuration — Windmill config_backup

Things that don't live in a PVC also need backing up, and they're split across four separate Windmill flows that each own one target. f/config_backup/config_backup snapshots the BPI-R4 OpenWrt config tarball, the salt/pepper TrueNAS system config, and the Cisco C9300 stack's running-config into the config-backups Garage bucket daily with 30-day retention. Three sibling flows — technitium_zone_backup, openobserve_meta_backup, and scrypted_data_backup — cover the Technitium DNS zone exports, OpenObserve internal metadata, and Scrypted's encrypted device store respectively, each on a daily schedule with 14-day retention — long enough to roll back a bad change, short enough that storage cost stays trivial.

Off-site Sync — garage_offsite_backup

The single daily Windmill flow that owns the L2/L3 fan-out reads every L1 bucket and rclone-syncs each one to both Backblaze B2 and an o2switch SFTP account. It runs at 04:00 Europe/Zurich, after the per-class backup jobs above have finished. Each bucket-pair sync is wrapped in its own exception handler, so a transient B2 outage on one bucket doesn't strand the rest.

Object-shape matters for tuning here: buckets holding a few large objects (SQL dumps, archives) sync fastest with --fast-list; buckets holding millions of small objects (Longhorn block storage in harvester-backup) need streaming listing instead, or they OOM the worker by materializing the full listing in memory. This is why the flow carries a per-bucket policy rather than one set of flags for all of them.

Encryption at the Off-site Boundary

The on-site copies (L1) sit on hardware in the same trust domain as the cluster; the off-site copies don't. Every routed bucket passes through an rclone crypt remote on its way to Backblaze B2; all but harvester-backup (excluded from SFTP — its hash-sharded Longhorn object layout is too slow to list there) also reach the o2switch SFTP account. ceph-backup does not currently participate in the off-site fan-out. Object contents are encrypted with authenticated encryption, while filenames and paths are deliberately left clear so listings, partial restores and object-level dedup stay cheap.

Two design points matter more than the cipher:

  • Key custody is out-of-band. The crypt keys live in the secret plane and in offline custody — never only inside the backups themselves. A provider-side copy of the backups without the keys is noise; losing the keys is the one failure the backup system cannot self-recover from, which is exactly why they're held in more than one place.
  • ceph-backup needs no wrapper. Its contents are restic repositories, and restic brings its own encryption — every object is already ciphertext before it reaches Garage. (It ships on-site only for now; see the note above.)

Monitoring

Backups that fail silently are worse than no backups at all, so every layer has an age-based liveness check.

Probe Cadence What fires
f/probe/daily_infra_health Hourly Per-PVC Longhorn backup age (warn / stale tiers), per-CNPG cluster last-base-backup age
f/probe/rancher_backup_check Daily 01:30 Rancher Backup CR age + error state
f/probe/gitlab_backup_health Daily 01:30 GitLab backup-utility tarball age, gitlab-toolbox pod state, group-runner status
garage_offsite_backup self-metrics Per-run garage_offsite_backup_success, pairs_ok, pairs_failed pushed to Pushgateway
PVMissingBackupStrategy Continuous Any Delete-policy PV without a backup label group or an explicit Windmill snapshot allowlist

The last guard is the most important: it makes "this PVC has no backup path" a Prometheus alert rather than a discovery made at restore time. See Storage → PV Reclaim Policy for the cross-check.

Recovery Targets

The four layers translate into different recovery characteristics per data class. These are design targets, not hard guarantees, and they assume the destination hardware is itself healthy (a corrupt Garage cluster doesn't get you to L1).

Data class Fast restore (RPO / RTO) Off-site restore (RPO / RTO)
Block PVCs (Longhorn) Snapshot interval (≤ 1 week) / minutes Daily L2/L3 / hours (rclone pull + Longhorn restore)
Ceph PVCs (VolSync) Nightly restic on Garage / hours Not yet — ceph-backup isn't in the off-site fan-out
PostgreSQL (CNPG) WAL gap (seconds) / minutes (PITR from local Barman cache) WAL gap / hours (Barman bootstrap from B2)
GitLab object stores Daily backup-utility / 1–2 h Daily L2/L3 / a working day end-to-end
Network / config snapshots Daily / minutes (paste a config back) Daily L2/L3 / minutes

The deliberately worst-case end-to-end story — "lose the entire on-prem site and recover from B2 + SFTP alone" — is the one this architecture is engineered to make merely tedious, rather than impossible.