Storage Architecture

Overview

Storage is split across complementary tiers: Longhorn for replicated block storage on the cluster nodes (landing on the FusionIO flash tier), and an in-cluster Rook Ceph cluster that serves both bulk RWX / block capacity on the repurposed SSDs and every S3 bucket in the stack through its RGW gateway.

graph LR
    subgraph k8s["Kubernetes workloads"]
        pvc_lh["Longhorn PVCs\nharvester-longhorn-2replicas"]
        pvc_ceph["Ceph PVCs\nceph-block · CephFS"]
        s3["S3 clients\n(live stores: GitLab, OpenObserve, Frigate, zot\nbackups: Longhorn, CNPG, GitLab, config)"]
    end

    subgraph longhorn["Longhorn"]
        lh_ctrl["Longhorn controller\nmostly 3-replica"]
        qui_d["qui — FusionIO"]
        quo_d["quo — FusionIO"]
        qua_d["qua — FusionIO"]
    end

    subgraph ceph["Rook Ceph (in-cluster)"]
        rgw["Ceph RGW\nS3 gateway"]
        osds["OSDs across qui · quo · qua"]
    end

    pvc_lh --> lh_ctrl
    lh_ctrl --> qui_d & quo_d & qua_d
    lh_ctrl -->|"backup snapshots"| rgw
    pvc_ceph --> osds
    s3 --> rgw
    rgw --- osds

One storage substrate, two access shapes

Every tier now lives on the three storage nodes (qui/quo/qua); a fourth node, quarto, adds compute only. Longhorn serves replicated block volumes off the FusionIO cards; Rook Ceph serves block (ceph-block), shared filesystems (CephFS) and — through its RGW gateway — every S3 bucket, live stores and backup stores alike.

Until September 2026 the S3 tier was an external 3-node Garage cluster on two IBM System x3550 M2 TrueNAS hosts (salt + pepper) plus a quorum-only in-cluster witness. Both hosts were retired and their buckets moved to RGW. salt had also once backed NFS/iSCSI PVCs via democratic-csi; that driver had already been undeployed when its ZFS pool filled.

Storage Classes

Class	Type	Replication	Use case
harvester-longhorn-2replicas	Longhorn block (RWO)	2× across nodes	Default class — most Longhorn PVCs
harvester-longhorn	Longhorn block (RWO)	3× across nodes	Harvester-provisioned default class — backs GitLab's Gitaly repo-data PVC
longhorn	Longhorn block (RWO)	3× across nodes	Legacy default class — still backs several live PVCs (the VictoriaMetrics/Grafana/Pushgateway stack, the jump seat's Guacamole Postgres, nameserver zones, spider3, znc, and a hosted WordPress site)
harvester-longhorn-2replicas-notmigratable	Longhorn block (RWX)	2× fixed nodes	Vestigial — no PVCs (the media library has since moved to CephFS)
harvester-longhorn-1replica	Longhorn block (RWO)	1×	Re-derivable / low-value state where a rebuild is cheaper than the extra replica (VolSync mover caches, OpenWrt image-build workspaces, LDAP admin UI, Technitium primary + secondary)
longhorn-volsync-cache-uid33 / -uid1000	Longhorn block (RWO)	1×	VolSync restic mover caches for movers that run as a fixed UID
longhorn-iomemory-large	Longhorn block (RWO) on the large FusionIO disks	2× across nodes	Large-capacity volumes; Retain
leap-micro-62-1replica	Longhorn block (RWX)	1×	VM root disks for the in-cluster Rancher management trio (rancher-mgmt-1/2/3) — migratable, best-effort data locality
longhorn-iomemory	Longhorn block (RWO) on FusionIO	2× across nodes	Vestigial since the storage rework — zero PVCs currently opt in, because every Longhorn volume (2-replica or 3-replica alike) already lands on FusionIO now that the SSD tier moved to Ceph
ceph-block	Ceph RBD (RWO) via Rook	3× / min 1	In-cluster block for general / app volumes
ceph-filesystem	CephFS (RWX) via Rook	data 3× / min 1, metadata 3×	Shared media library (tv / film / serie)
ceph-bucket	Ceph RGW S3 (ObjectBucketClaim) via Rook	data 3× / min 1, metadata 3×	In-cluster S3 buckets
local-fio-etcd (static local PVs, no provisioner)	Node-local block	1× per node	Rancher management cluster's etcd, on a dedicated FusionIO partition per node

Node-local Performance Tiers

Beyond replica count, Longhorn's per-node disks are split into deliberate performance tiers, exposed as disk tags and selectable per StorageClass. Latency-sensitive workloads land on the fast tier while bulk data uses cheaper, higher-capacity disks.

Tier	Longhorn tag	Backing hardware	Role
FusionIO	fio, iomemory (+ large on the biggest disks)	ioMemory PCIe flash — direct PCIe, no RAID controller in the path	All Longhorn volumes now land here: etcd, databases, replicated block
SSD (P420i)	(not in Longhorn)	SSDs behind an HP Smart Array P420i	Reassigned to Ceph OSDs — the in-cluster bulk tier — and no longer a Longhorn tier
FusionIO (shared with Ceph)	(Ceph deviceClass: fio)	A shared metadata partition plus a second fio2 partition on every node, addressed via stable /dev/disk/by-partlabel/... paths rather than /dev/fioaN (unstable once a node has 2 cards)	6 of the 21 Ceph OSDs (2 per node) also live on FusionIO — Longhorn isn't the only consumer of the flash tier
Rotational	(not in Longhorn)	HP SAS 10K in RAID-5	Node OS and pod ephemeral storage — where a pod writes when it has no PVC

The longhorn-iomemory StorageClass pins a volume's replicas to the FusionIO tier; since the SSDs were handed to Ceph, every Longhorn volume effectively lives on FusionIO. etcd itself runs on a dedicated FusionIO partition on every control-plane node, so cluster-state writes never share a controller with bulk volume I/O.

Representative figures from production telemetry (node-exporter → VictoriaMetrics): the FusionIO tier commits durable writes in ~0.4 ms (etcd WAL fsync) and sustains low-latency I/O bursting past 1 GB/s — an order of magnitude faster, and far more predictable, than the controller-backed tiers. That predictability is the reason the most latency- and integrity-sensitive stores live there.

Longhorn Backup Policy

Longhorn snapshots ship to the longhorn-backup bucket on Ceph RGW, on a schedule driven by recurring-job group labels. A daily Windmill reconciler enforces the labels; Longhorn itself assigns any unlabelled volume to default.

Label	Applied to	Schedule
recurring-job-group.longhorn.io/default=enabled	Most PVCs	Monthly backup (1st @ 05:00) + daily filesystem-trim
recurring-job-group.longhorn.io/weekly=enabled	Selected namespaces	Weekly backup (Fri 01:00)
recurring-job-group.longhorn.io/daily=enabled	A few high-churn PVCs	Daily backup (03:40, retain 5)
recurring-job-group.longhorn.io/nobackup=enabled	Cache / metrics / re-acquirable-media PVCs	No backups; only filesystem-trim (47 15 * * *)

Weekly backup namespaces: appdaemon, bootstrap, envuassu, esphome, frigate, home-assistant, openldap, and others.

Object Storage — Ceph RGW

Every S3 bucket in the stack is served by the in-cluster Ceph RGW gateway (ceph-objectstore, in rook-ceph), with per-consumer scoped keys issued as CephObjectStoreUser resources. It carries both roles:

Role	Buckets
Live object stores	GitLab's full suite (container registry, CI artifacts, LFS, uploads, packages, Pages, dependency proxy, Terraform state, runner cache), OpenObserve's Parquet log store, Frigate's camera recordings, the zot OCI registry image store
Local backup tier	longhorn-backup, cnpg-* (one Barman store per database), gitlab-backups, config-backups, home-assistant-backups

Backing the object data with the multi-OSD Ceph cluster gives it real redundancy across the three nodes, and it benchmarked several times faster than the single-node object store it replaced on small-object workloads.

In-cluster is not the same as private

The three nodes are separate physical machines, so pod-to-pod traffic crosses the LAN on the overlay network. The backup tier's contents are not encrypted at rest — Longhorn's backupstore format and Barman's output are both plaintext — so the hop to RGW runs over TLS with a house-PKI leaf rather than the plain listener. The alternative would put every volume and every database on the wire in the clear.

Why this tier moved in-cluster

Until September 2026 the S3 tier was an external 3-node Garage cluster on the salt and pepper TrueNAS hosts, joined by a quorum-only in-cluster witness pod. The witness existed for a concrete reason worth keeping: with only two nodes and consistency_mode=consistent, CompleteMultipartUpload reads the parts list from the peer, and if the peer has not yet acknowledged the last UploadPart the read returns InvalidPart / NoSuchUpload — sequential uploads failed roughly 80% of the time. A third node removed the race.

Both hosts were retired in September 2026 and every bucket moved to RGW. The trade is stated plainly on Backups & DR: the old tier sat on separate hardware and survived a node-class failure of the cluster, the new one does not — so disaster coverage now rests entirely on the client-side-encrypted off-site channel.

Off-site Backup

The RGW backup tier holds the only local copy of several datasets that have no other home — Longhorn volume backups, the GitLab backups, the Barman stores and the daily config snapshots. A daily Windmill job (04:00 Europe/Zurich) rclone-syncs each of those buckets to two independent off-site destinations: a Backblaze B2 bucket and an off-site SFTP account.

VolSync's Ceph-PVC backups do not pass through this tier at all. restic encrypts them inside the cluster and writes them straight to B2, which is what lets them be the off-site copy of record rather than a replica of a plaintext store — see Backups & DR.

PV Reclaim Policy

Production PVCs default to reclaimPolicy: Delete — disk space here is finite and recovery from a deleted PVC almost always means restoring from backup anyway. Retain is reserved for the small set of PVCs where no backup path exists and accidental deletion would be unrecoverable.

A vmalert rule (PVMissingBackupStrategy) cross-checks every Delete-policy PV against a derived backup story — a Longhorn recurring-job group, a VolSync source, a CNPG Barman store, an etcd snapshot, or a documented app-level or exempt entry — and a Delete PV only stops alerting if it provably has a backup path. The Windmill pv_reclaim_policy_analysis flow publishes the same data as a Pushgateway gauge with a 36h staleness rule (PVReclaimPolicyAuditStale), so a silent failure of the audit script also fires.