Skip to content

Backups & Disaster Recovery

Backup is a cross-cutting concern — every data class in the stack (block volumes, relational databases, GitLab object stores, cluster configuration, network device snapshots) has its own native backup mechanism, but they all converge on the same destinations and the same off-site path. Reading any one page in this site only shows you that page's slice; this one is the cross-section.

The design goal is two independent off-site copies for every byte that matters, with a recovery path that doesn't depend on any single piece of on-prem hardware staying up. The fast-restore tier (local snapshots) is a convenience on top of that, not a substitute for it.

Architecture at a Glance

flowchart LR
    subgraph apps["Workloads"]
        pvc["Block PVCs<br/>(Longhorn)"]
        ceph_pvc["Ceph PVCs<br/>(RBD · CephFS via Rook)"]
        cnpg["PostgreSQL<br/>(CloudNativePG ×6 with barman)"]
        gl["GitLab object stores<br/>(uploads · artifacts · LFS · registry · …)"]
        net["Network devices + cluster config<br/>(BPI-R4 · Cisco · XikeStor · Technitium · OpenObserve · Scrypted)"]
    end

    subgraph snap["L0 — Local snapshots"]
        lh_snap["Longhorn snapshots<br/>(per-volume, on-cluster)"]
    end

    subgraph onsite["L1 — Local tier (in-cluster Ceph RGW)"]
        g_hb["longhorn-backup"]
        g_pg["cnpg-*<br/>(barman stores)"]
        g_gl["gitlab-backups"]
        g_cb["config-backups"]
    end

    subgraph off["L2 + L3 — Off-site (daily rclone, 04:00 Europe/Zurich)"]
        b2["Backblaze B2"]
        sftp["o2switch SFTP"]
    end

    pvc -->|"recurring snapshot"| lh_snap
    pvc -->|"recurring backup"| g_hb
    ceph_pvc -->|"nightly CSI snapshot +<br/>restic (VolSync)<br/>client-side encrypted"| b2
    cnpg -->|"continuous WAL +<br/>nightly base"| g_pg
    gl -->|"daily backup-utility"| g_gl
    net -->|"daily Windmill snapshot"| g_cb
    onsite --> b2
    onsite --> sftp

Solid arrows are scheduled data flow. The dashed-line "L0 → L1" relationship (snapshot then backup) is collapsed in the diagram because Longhorn handles both as recurring jobs on the same volume — they're configured side-by-side, not in series.

The Four Layers

L0 — Local Longhorn snapshots are copy-on-write snapshots of the underlying volume image, kept on the same nodes as the live data. They cost almost nothing in time and storage, and a restore is a volume.spec.fromBackup flip plus a pod restart. They're the right tool for "I just broke a PVC, get the previous state back" but they're worthless against losing the cluster — they don't leave the cluster.

L1 — Local tier on the in-cluster Ceph RGW keeps a short-window, granular restore path close to the data: point-in-time recovery for the databases, per-volume rollback for Longhorn. Each consumer has its own bucket scoped by its own access key — see Storage → Object Storage for the bucket list.

Two things follow from this tier living on the same Ceph cluster as the primary data, and both are deliberate. It is not a disaster-recovery layer — that job belongs entirely to the off-site channel below, which is why the off-site copies are client-side encrypted rather than a mirror of this one. And because the contents are not encrypted at rest, the hop to RGW runs over TLS with a house-PKI leaf: pod-to-pod traffic crosses the physical LAN between the three nodes, so "in-cluster" is not the same as "private".

This tier used to live on separate hardware

Until September 2026 the local tier was an external 3-node Garage S3 cluster on two TrueNAS hosts plus a quorum-only witness pod. Those hosts were retired and the tier moved in-cluster onto Ceph RGW. The trade is explicit: the old layout survived a Harvester-node-class failure, the new one does not — so disaster coverage was moved to the client-side-encrypted off-site channel, where it is now the only thing that carries it.

L2 — Off-site to Backblaze B2 is a daily rclone sync of every L1 bucket to a Backblaze B2 bucket. B2 is in a different jurisdiction and a different failure domain from the home lab. This is the layer that survives losing the entire on-prem site.

L3 — Off-site to o2switch SFTP is the same daily sync, fanned out to an SFTP account at a second provider. Two independent off-site destinations means losing either one (provider outage, credential revocation, account-level mishap) still leaves a recoverable copy.

The L2/L3 fan-out is a single Windmill flow (f/config_backup/rgw_offsite_backup) that runs at 04:00 Europe/Zurich daily. It is idempotent — re-runs only transfer changed objects — and exception-handlers wrap each bucket-pair so a failure on one bucket doesn't abort the others.

The direct off-site channel sits beside all of this rather than under it. Anything that can encrypt client-side skips the local tier entirely and writes straight to Backblaze B2 as ciphertext: the eleven VolSync ReplicationSource repositories (restic) and the management-cluster rancher-backup (self-encrypting). That one question — can this producer encrypt before the bytes leave? — is what decides which channel a given data class uses, because an unencrypted store can be replicated off-site but should not be the off-site copy of record.

By Data Class

Block PVCs — Longhorn

PVCs are labelled at provision time, mapping each one to Longhorn recurring-job groups. Every PVC gets at least the default group (monthly backup, retain 2, plus a daily filesystem trim); namespaces holding active state (Home Assistant, GitLab, Frigate, OpenLDAP, ...) additionally get weekly (weekly backup, Fridays 01:00, retain 2). Cache, metrics and re-acquirable-media PVCs (Valkey, metrics stores, the bulk media library) get nosnapshots, whose only recurring job is snapshot cleanup — rebuilding them from scratch is faster than restoring.

Backups ship to the longhorn-backup bucket on the in-cluster Ceph RGW via Longhorn's BackupTarget resource (s3://longhorn-backup@us-east-1/). See Longhorn Backup Policy for the per-label schedule.

Ceph PVCs — VolSync

Volumes on the in-cluster Ceph tier (RBD block and CephFS via Rook) don't participate in Longhorn's recurring jobs, so they get their own equivalent path: VolSync drives a per-PVC ReplicationSource that takes a nightly CSI snapshot for consistency and ships the contents with restic directly to Backblaze B2. Because restic encrypts before anything leaves the cluster, these repositories are the archetype of the direct off-site channel — they skip the local tier rather than being replicated out of it. The enrolled sources are the primary-data volumes — the Nextcloud file stores, Paperless documents, ESPHome config, spider3 and Open WebUI — together with the bastion home and its PKI, and a handful of smaller state volumes. The bulk media library is deliberately not — that data is re-acquirable and doesn't justify off-site cost.

The honest consequence of encrypting client-side is that these volumes have one tier, not two: their only copy is the restic repository in B2. There is no local granular-restore path for them, which is the cost of the encryption guarantee rather than an oversight.

PostgreSQL — CloudNativePG with Barman

The stack runs seven CloudNativePG clusters: bootstrap/gitlab-pg (the GitLab Rails database), home-assistant/ha-recorder-pg (the Home Assistant recorder), paperless/paperless-pg (the Paperless DMS metadata store), jump/guac-pg (the Guacamole session store behind the jump seat), monitoring/grafana-pg (Grafana's unified-storage backend), bussola/bussola-pg and litellm/litellm-pg. Six of the seven use continuous WAL streaming via Barman Cloud into their own cnpg-<cluster> bucket, plus a nightly ScheduledBackup. This gives them a point-in-time restore path independent of any snapshot of the underlying PVC — the PVC backup is a consistency hedge, the Barman store is the actual recovery substrate.

litellm-pg is the deliberate exception: it carries no Barman store, so its only coverage is volume-level snapshots — crash-consistent, with no PITR. That is an accepted trade for a cache-shaped workload rather than a gap, and it is tracked as such.

GitLab object stores — backup-utility

GitLab Rails has its own backup-utility that snapshots every internal object store (uploads, artifacts, LFS, packages, dependency proxy, container registry, terraform state, pages) into a single tarball and ships it to the gitlab-backups bucket. The job runs on schedule from the gitlab-toolbox pod. Combined with the CNPG backup of gitlab-pg and the Longhorn backup of the GitLab PVCs, a GitLab restore can pick whichever consistency boundary is cheapest for the recovery scenario.

Rancher cluster config

Rancher's management-plane state — CRDs, RBAC, secrets, project mappings, Fleet bindings, downstream cluster registrations, Keycloak integrations — has no dedicated backup path today. The Rancher Backup Operator that would dump it to a rancher-backup bucket on a daily schedule isn't installed on the cluster; this data class is a known gap against the design goal of two independent off-site copies, not yet closed.

Network devices and cluster configuration — Windmill config_backup

Things that don't live in a PVC also need backing up, and they're split across four separate Windmill flows that each own one target. f/config_backup/config_backup snapshots the BPI-R4 OpenWrt config tarball, the Cisco C9300 stack's running-config, and the XikeStor AP-leaf switch's config blob into the config-backups bucket daily with 30-day retention. Three sibling flows — technitium_zone_backup, openobserve_meta_backup, and scrypted_data_backup — cover the Technitium DNS zone exports, OpenObserve internal metadata, and Scrypted's encrypted device store respectively, each on a daily schedule with 14-day retention — long enough to roll back a bad change, short enough that storage cost stays trivial.

Off-site Sync — rgw_offsite_backup

The single daily Windmill flow that owns the L2/L3 fan-out reads every L1 bucket on the Ceph RGW and rclone-syncs each one to both Backblaze B2 and an o2switch SFTP account. It runs at 04:00 Europe/Zurich, after the per-class backup jobs above have finished. Each bucket-pair sync is wrapped in its own exception handler, so a transient B2 outage on one bucket doesn't strand the rest.

Object-shape matters for tuning here: buckets holding a few large objects (SQL dumps, archives) sync fastest with --fast-list; buckets holding millions of small objects (Longhorn block storage in longhorn-backup) need streaming listing instead, or they OOM the worker by materializing the full listing in memory. This is why the flow carries a per-bucket policy rather than one set of flags for all of them.

Encryption at the Off-site Boundary

The local tier (L1) sits in the same trust domain as the cluster; the off-site copies don't. Every routed bucket passes through an rclone crypt remote on its way to Backblaze B2; all but longhorn-backup (excluded from SFTP — its hash-sharded Longhorn object layout is too slow to list there) also reach the o2switch SFTP account. Object contents are encrypted with authenticated encryption, while filenames and paths are deliberately left clear so listings, partial restores and object-level dedup stay cheap.

Two design points matter more than the cipher:

  • Key custody is out-of-band. The crypt keys live in the secret plane and in offline custody — never only inside the backups themselves. A provider-side copy of the backups without the keys is noise; losing the keys is the one failure the backup system cannot self-recover from, which is exactly why they're held in more than one place.
  • The VolSync repositories need no wrapper. Their contents are restic, which brings its own encryption — every object is ciphertext before it leaves the cluster. That is precisely why they bypass this fan-out and write straight to B2.

Monitoring

Backups that fail silently are worse than no backups at all, so every layer has an age-based liveness check.

Probe Cadence What fires
f/probe/daily_infra_health Hourly Per-PVC Longhorn backup age (warn / stale tiers), per-CNPG cluster last-base-backup age
f/probe/rancher_backup_check Daily 01:30 Rancher Backup CR age + error state
f/probe/gitlab_backup_health Daily 01:30 GitLab backup-utility tarball age, gitlab-toolbox pod state, group-runner status
rgw_offsite_backup self-metrics Per-run rgw_offsite_backup_success, pairs_ok, pairs_failed pushed to Pushgateway
f/config_backup/rgw_offsite_restore_drill Daily 06:50 Restores from the off-site tier and verifies the result decrypts and reads back — see below
f/probe/restic_snapshot_freshness Daily Per-repository VolSync snapshot age, and snapshots that succeeded but are empty
PVMissingBackupStrategy Continuous Any Delete-policy PV without a backup label group or an explicit Windmill snapshot allowlist

PVMissingBackupStrategy makes "this PVC has no backup path" a Prometheus alert rather than a discovery made at restore time. See Storage → PV Reclaim Policy for the cross-check.

A restore is proven every day

Every check above measures that backups are taken. None of them proves a backup can be used — and a backup that has never been restored is a hypothesis, not a control.

So once a day a drill pulls real data back from the off-site tier and verifies it: that it decrypts with the keys held in custody, and that what comes back is intact. It exercises exactly the path a real disaster would take, including the part most estates only discover under pressure — whether the keys and the ciphertext still belong together.

Two details make it worth trusting. Snapshot ages are reported per repository, never averaged, because an average hides precisely the one repository that quietly stopped. And a snapshot that completed successfully but contains nothing is counted as a failure, because it is the failure mode that looks exactly like success.

Recovery Targets

The four layers translate into different recovery characteristics per data class. These are design targets, not hard guarantees, and they assume the destination is itself healthy (a corrupt object store doesn't get you to L1).

Data class Fast restore (RPO / RTO) Off-site restore (RPO / RTO)
Block PVCs (Longhorn) Snapshot interval (≤ 1 week) / minutes Daily L2/L3 / hours (rclone pull + Longhorn restore)
Ceph PVCs (VolSync) No local tier — see above Nightly restic in B2 / hours (client-side encrypted)
PostgreSQL (CNPG) WAL gap (seconds) / minutes (PITR from local Barman cache) WAL gap / hours (Barman bootstrap from B2)
GitLab object stores Daily backup-utility / 1–2 h Daily L2/L3 / a working day end-to-end
Network / config snapshots Daily / minutes (paste a config back) Daily L2/L3 / minutes

The deliberately worst-case end-to-end story — "lose the entire on-prem site and recover from B2 + SFTP alone" — is the one this architecture is engineered to make merely tedious, rather than impossible.