Hardware & Network Topology¶

Physical Layout¶

Edge — Internet path¶

flowchart LR
    ext["External clients"]
    ont["FTTH SFP+ ONU\nFS XGS-SFP-ONT-MACI\n(XGS-PON, MaxLinear PRX126)"]
    bpir4["BPI-R4\nOpenWrt 25.12-tillo\nMT7988a, dnsmasq, haproxy,\ndnsdist, WireGuard"]
    bridge["br-lan trunk\n→ LAN / DMZ / ADLAN /\nwansub"]

    ext -->|"ISP /28\n31.3.128.48/28"| ont
    ont -->|"VLAN 11\nFTTH PPPoE"| bpir4
    bpir4 -->|"VLAN 10 (WAN trunk),\nLAN native (PVID 1)"| bridge

Cluster on the LAN — 192.168.1.0/24¶

flowchart LR
    bpir4["BPI-R4\n192.168.1.254\nLAN gateway"]
    techni["Technitium\n.54 DNS · .55 DHCP\nhome.tillo.ch zone"]
    mbt["mbptillo\n192.168.1.246\nsslh / mosh / OpenVPN"]

    subgraph k8s["mdapi-prod — Harvester HCI / RKE2"]
        qui["qui — bare metal"]
        quo["quo — bare metal"]
        qua["qua — bare metal"]
        vip191["ingress-nginx\nkube-vip 192.168.1.191"]
    end

    subgraph mgmt["Management"]
        rancher["Rancher"]
        ns["BIND9 (ns.mdapi.ch)\n192.168.1.53"]
    end

    bpir4 --> techni
    bpir4 --> mbt
    techni --> k8s
    mbt --> vip191
    rancher --> k8s

Switching fabric¶

Since 2026-08-05 the LAN core is a 2× Cisco Catalyst C9300-24T StackWise stack (c9300-stack, 192.168.1.250, IOS-XE 16.12 — 48× 1G copper plus 16× SFP+ via two C9300-NM-8X modules). It replaced the HP 2920 stack; a single 2920 (bomberos, 192.168.1.176) remains temporarily as a PoE leaf for the five WiFi APs, hanging off a 10G uplink (Te1/1/4 ↔ HP 1/A1, 10GBASE-T SFP+ modules both ends). It retires once the XikeStor 2.5G-PoE leaf takes the APs (planned uplink Te2/1/8).

Attachment	Ports	Link
`qui` / `quo` / `qua`	`Po20`–`Po22` = `Te1/1/1-3` + `Te2/1/1-3`	2×10G cross-chassis LACP per node
bpi-r4 (LAN gateway, STP root)	`Te1/1/8`	10G
`salt` / `pepper` (TrueNAS)	`Po10` / `Po11` = `Gi1/0/1-6` + `Gi2/0/1-6`	6×1G cross-chassis LACP each
`nastillo` (Synology)	`Po12` = `Gi1/0/13` + `Gi2/0/13`	2×1G cross-chassis LACP
`mbptillo`	`Te1/1/7` (10GBASE-T SFP+ module)	2.5G
iLOs, IMMs, small devices	`Gi1/0/7-16`	1G / 100M access, VLAN 1
HP AP leaf uplink	`Te1/1/4`	10G

Per-flow LACP spreading is enabled end to end: the stack hashes on src-dst-mixed-ip-port (global) and the Harvester nodes' mgmt-bo bonds use xmit_hash_policy layer3+4, persisted upgrade-safe in /oem/94_bond_xmit_hash.yaml — measured 18.1 Gbit/s aggregate between two nodes. The stack is pure L2 plus its management SVI; all inter-VLAN routing stays on the BPI-R4.

Node hardware¶

Host	Model	CPU / RAM	Storage
`qui` · `quo` · `qua`	HP ProLiant DL360p Gen8	2× Xeon E5-2640 v2 · ~96 GB	FusionIO PCIe flash (etcd + Longhorn's fast tier + a small CephFS-metadata OSD partition) · Smart Array P420i 558 GB logical volume (`sda`: OS, containerd, logs) · 5× SATA SSD (P420i pass-through, Ceph bulk-tier OSDs)
`salt` · `pepper`	IBM System x3550 M2	—	TrueNAS — nodes of the external Garage S3 cluster

Storage hosts¶

flowchart LR
    k8s["mdapi-prod\n(Longhorn + S3 clients)"]
    ceph["Rook Ceph\nin-cluster bulk tier\nqui · quo · qua SSD OSDs"]
    garage["Garage\ngarage.home.tillo.ch\n(3-node quorum)"]
    salt["salt — TrueNAS\nGarage\n+ NFS/iSCSI (idle)"]
    pepper["pepper — TrueNAS\nGarage"]
    witness["garage-witness pod\n(in-cluster, quorum-only)"]

    k8s --> ceph
    k8s -->|"S3 + Longhorn off-site"| garage
    garage --- salt
    garage --- pepper
    garage --- witness
    k8s -.->|"democratic-csi NFS/iSCSI\n(no live PVCs)"| salt

Auxiliary networks¶

flowchart LR
    bpir4["BPI-R4"]
    dmz["DMZ\n192.168.7.0/24\n(sfp-lan.20, VLAN 20)"]
    adlan["ADLAN\n192.168.77.0/24\n(sfp-lan.70, VLAN 70)"]
    wansub["wansub\n31.3.128.49/28\n(routed ISP /28)"]
    cm["CipherTrust Manager\ncm.mdapi.ch (DMZ)"]
    oob["Out-of-band / admin hosts"]

    bpir4 --> dmz --> cm
    bpir4 --> adlan --> oob
    bpir4 --> wansub
    cm -.->|"customer-fragment\nfor Akeyless ESO"| bpir4

VLAN ID conventions¶

VLAN	Tag	Trunk port	Network
LAN (native)	sometimes 1 (default PVID); untagged on `lan1`–`lan3`	also tagged 10 on `sfp-lan` and `wan`	`192.168.1.0/24` (br-lan)
WAN (LAN-side)	10	`sfp-lan.10`, `wan.10`	LAN traffic carried over the trunk to/from the WAN port group
FTTH PPPoE	11	`sfp-wan.11`	ISP-side PPPoE encapsulation toward the ONT
DMZ	20	`sfp-lan.20`	`192.168.7.0/24`
ADLAN	70	`sfp-lan.70`	`192.168.77.0/24`

External IP Map¶

ISP allocation: a /28 (31.3.128.48/28, usable .49–.62). The PPP endpoint sits on .49 (ftth-fixed); additional public IPs are configured as /32 aliases on pppoe-wan and DNAT'd (or haproxy-front-ended) into the cluster. The wansub interface (31.3.128.49/28) provides routed access to the rest of the /28 for hosts that need a real public IP rather than a DNAT target.

The authoritative scope of every public IP is the BPI-R4's dhcp.@host[*] reservation list (uci show dhcp). Each address has a human-readable name there.

The map below is IPv4. The edge is dual-stack: IPv6 clients reach the v4-only cluster through NAT64 on the BPI-R4, and public DNS terminates natively over v6 at dnsdist on the router — see Traffic → IPv6 → IPv4 (NAT64 via Jool).

IP	DHCP name	Inbound mechanism	Inbound target	Outbound SNAT match	Purpose
`31.3.128.49`	`ftth-fixed`	`wansub` interface	—	—	PPP endpoint
`31.3.128.50`	`mdapi-virtual-inbound`	DNAT	`192.168.1.191:80,443`	—	RKE2 ingress-nginx (all `*.mdapi.ch`)
`31.3.128.51`	`cm-virtual-inbound`	haproxy `frontend cm` :443, TLS termination	`cm.mdapi.ch` — CipherTrust Manager (DMZ)	—	Akeyless customer-fragment endpoint. haproxy holds the LE cert because CM has no ACME client of its own.
`31.3.128.52`	— (no `dhcp.@host` reservation)	DNAT	`192.168.1.62:443` (TCP+UDP) — jump-bastion pod, ns `jump`	—	ssh -p443 + mosh — access path of last resort, independent of mbptillo/bpi-r4
`31.3.128.53`	`cloud-envuassu-virtual-inbound`	DNAT	`192.168.1.40:3443` (TCP), `:4443` (QUIC) + `192.168.1.41:3478`	—	Nextcloud AIO (`envuassu`) + Talk
`31.3.128.54`	`mail-virtual`	haproxy TCP-mode :25/:465/:587/:993/:4190 with PROXY protocol to backend	mail stack in K8s	dest port = SMTP (TCP/25)	docker-mailserver. The Service runs `externalTrafficPolicy: Cluster` (kube-proxy SNATs to a node-internal address); PROXY protocol restores the real client IP. Outbound SMTP also SNATs to `.54` so PTR + SPF align with `mail-virtual`.
`31.3.128.55`	`mirror-virtual-inbound`	DNAT	`.49` (mirror :21/:80/:443/:873/:40000-40050), `.58:123` (ntppool), `.44:53` (opennic)	dest port = mirror NTP / DNS / rsync	Shared mirror + NTP pool + OpenNIC
`31.3.128.56`	`proxy-virtual-inbound`	DNAT	`192.168.1.50:443`	—	Squid HTTPS proxy
`31.3.128.57`	`znc-virtual`	DNAT	`192.168.1.51:113,443`	dest port = IRC	ZNC IRC bouncer
`31.3.128.58`	`mbptillo-virtual-inbound`	DNAT	`192.168.1.246:4443`	—	sslh on mbptillo (TLS / SSH / OpenVPN demux) + WireGuard `:51820`
`31.3.128.59`	`ns-virtual-inbound`	haproxy TCP :53 (`send-proxy-v2` → BIND's PROXY listener) + DNAT UDP :53	`192.168.1.53:53`	dest port = authoritative DNS	BIND9 (`ns.mdapi.ch`). Native IPv6 DNS terminates at dnsdist on the router instead.
`31.3.128.60`	— (no `dhcp.@host` reservation)	DNAT	`192.168.1.42` (spider3 SFTPGo) + `192.168.1.57` (tv SFTPGo) + `192.168.1.191` (ingress-nginx)	—	SFTPGo/FTP gateway for `*.files.mdapi.ch` — per-namespace control/data/passive ports DNAT to each SFTPGo backend, HTTPS UI DNAT to ingress-nginx
`31.3.128.62`	`vpn-virtual-inbound`	DNAT	`192.168.1.157:1194` (Firewalla VPN) + `192.168.1.46:32400` (Plex) + WireGuard `wg_iot` / `wg_s2s`	fallback — every flow that doesn't match a more specific SNAT rule	`pctillo.tillo.ch` for IoT/S2S VPNs and `egress.mdapi.ch`, the cluster's default outbound public IP.

Three edge mechanisms — DNAT, haproxy, SNAT¶

The /32 aliases on pppoe-wan serve traffic via three different patterns. They are not interchangeable — each exists because of a specific limitation on either side.

DNAT is the default. Most public IPs are plain destination-NAT rules: rewrite the destination to a LAN IP, forward, done. Used wherever the backend can speak the wire protocol directly (HTTP/2 with its own TLS, raw TCP, UDP). The cluster's RKE2 ingress on .50, sslh on .58, UDP DNS on .59, etc. all live here.

haproxy runs on the BPI-R4 (/etc/haproxy.cfg) and is used for three specific reasons:

The backend cannot do ACME itself. The CipherTrust Manager appliance (cm.mdapi.ch, in the DMZ) has no Let's Encrypt client — haproxy on .51:443 terminates TLS with the LE cert and relays to CM in HTTP/2 mode.
The backend needs the real client IP via PROXY protocol. docker-mailserver runs as a K8s workload behind a LoadBalancer Service with externalTrafficPolicy: Cluster, so kube-proxy SNATs every connection to a node-internal address. Without restoration, every connection looks like it comes from a cluster-internal address — which breaks per-IP rate-limiting, IP-based reputation, and fail2ban. haproxy on .54:25/465/587/993/4190 accepts the public connection, then opens the backend connection with PROXY protocol so docker-mailserver sees the original client.
The same, for DNS. Public IPv4 TCP/53 on .59 is fronted by haproxy in TCP mode with send-proxy-v2 into BIND's PROXY-protocol listener, so per-client rate-limiting and ACLs see real addresses; UDP/53 stays a plain DNAT. Native IPv6 DNS gets the identical treatment from dnsdist — see Traffic → Public DNS.

SNAT controls which public IP is used as the source for outbound traffic. The naive "SNAT by source LAN IP" approach does not work here: every K8s pod's outbound packet is already SNAT'd by the CNI to one of the node IPs (192.168.1.190, .192, .193), so the BPI-R4 cannot tell mail traffic apart from DNS or HTTP traffic by source alone. Instead, SNAT is keyed off destination port — the firewall marks each forwarded connection by its destination service, then chooses the public source IP per mark:

Outbound service	Destination port(s)	SNAT source
docker-mailserver (SMTP)	TCP/25	`.54` (so PTR + SPF align with `mail-virtual`)
BIND9 authoritative replies	UDP/53 sourced from `192.168.1.53`	`.59`
mirror NTP/DNS/rsync stack	per-port marks	`.55`
ZNC bouncer	IRC	`.57`
Everything else (LAN, DMZ, ADLAN, ONT, wansub)	—	`.62` (`egress.mdapi.ch`)

Source of truth: uci show dhcp (host names), uci show firewall (DNAT, SNAT marks, zone-default SNAT), /etc/haproxy.cfg (TLS/TCP frontends), and ip -4 addr show pppoe-wan (assigned aliases).

Internal IP Reference¶

Device	IP	Role
FTTH ONT	192.168.11.1	Fiber modem (integrated in BPI-R4)
BPI-R4	192.168.1.254	LAN gateway, DHCP relay, dnsmasq forwarder (DMZ/ADLAN)
split-horizon unbound	192.168.1.1	LAN split-horizon resolver (in-cluster, MetalLB VIP)
mbptillo	192.168.1.246	Workstation — sslh / mosh / OpenVPN endpoint
jump pod	192.168.1.62	In-cluster jump seat (`jump` ns) — ssh -p443 + mosh
qui iLO	192.168.1.170	BMC node 1
quo iLO	192.168.1.181	BMC node 2
qua iLO	192.168.1.182	BMC node 3
ingress-nginx (RKE2 builtin)	192.168.1.191	All HTTPS — `nginx` ingressClass; serves `*.mdapi.ch` and GitLab Pages
BIND9 nameserver	192.168.1.53	Authoritative public DNS (mdapi.ch, tillo.ch, …)
Technitium DNS	192.168.1.54	Internal DNS for `home.tillo.ch`
Technitium DHCP	192.168.1.55	Authoritative DHCP server (LAN/DMZ/ADLAN)
Garage S3	garage.home.tillo.ch	S3 object store (3-node quorum)

VLANs¶

The BPI-R4 routes three internal VLANs in addition to br-lan (the trunk on the cabled LAN). Each VLAN has BPI-R4 as gateway and uses Technitium for DHCP via relay.

VLAN	Subnet	Gateway	Purpose
`br-lan` (untagged)	192.168.1.0/24	192.168.1.254	Trusted LAN — servers, infra, workstations
`sfp-lan.20` (DMZ)	192.168.7.0/24	192.168.7.254	DMZ for limited-trust workloads
`sfp-lan.70` (ADLAN)	192.168.77.0/24	192.168.77.254	Administrative / out-of-band

Ingress Controller¶

The cluster runs a single nginx-ingress deployment — the RKE2 builtin rke2-ingress-nginx — exposed on kube-vip VIP 192.168.1.191 with IngressClass nginx and ModSecurity WAF enabled. It handles every *.mdapi.ch service including GitLab itself (gitlab.mdapi.ch, registry.mdapi.ch, kas.mdapi.ch) and GitLab Pages (*.pages.mdapi.ch + custom domains like docs.mdapi.ch).

GitLab Pages custom domains require an explicit Ingress with ingressClassName: nginx targeting the gitlab-gitlab-pages service on port 8090. The docs.mdapi.ch ingress lives at https://gitlab.mdapi.ch/mdapi/fleet/-/tree/main/docs (public mirror).

Load Balancer Architecture¶

Two components split the LoadBalancer job — MetalLB does IPAM (which Service gets which address), while Harvester's kube-vip in services mode does the announcement and interface plumbing:

MetalLB (metallb-system namespace) — assigns addresses from the pool 192.168.1.40–99 (metallb-pools/prod-pool.yml in Fleet), plus a dedicated single-address pool for the well-known LAN DNS VIP 192.168.1.1; L2Advertisements are scoped to mgmt-br. Strict opt-in via --lb-class=metallb: only Services with spec.loadBalancerClass: metallb are claimed; everything else is ignored, so MetalLB never fights another controller over an address.
kube-vip (Harvester-bundled, harvester-system/kube-vip DaemonSet) — binds each VIP as a /32 on mgmt-br and answers ARP. It also owns ingress-expose at 192.168.1.191 end-to-end: configured via the harvester-system/vip ConfigMap (mode: static, ip: 192.168.1.191) and Service annotations kube-vip.io/ignore-service-security: "true" + kube-vip.io/loadbalancerIPs: 192.168.1.191, leader-elected across control-plane nodes via the harvester-system/plndr-svcs-lock Lease. Failover is automatic on node loss; for a Service with externalTrafficPolicy: Local, the announcement comes from a node holding a local endpoint — preserving real source IP.

Each Service pins its IP via spec.loadBalancerIP: 192.168.1.<n>. The class field is immutable in Kubernetes — recreating a Service to change class requires a brief delete+apply.

25 LoadBalancer Services draw from these pools (several share a VIP); the principal VIPs:

VIP	Class	Namespace	Service	Policy
192.168.1.1	metallb	`split-horizon`	unbound (split-horizon DNS)	Cluster
192.168.1.40	metallb	`envuassu`	Nextcloud AIO Apache	Local
192.168.1.41	metallb	`envuassu`	Nextcloud AIO Talk	Local
192.168.1.42	metallb	`spider3`	SFTPGo	Local
192.168.1.43	metallb	`mqtt`	Eclipse Mosquitto	Local
192.168.1.44	metallb	`opennic`	OpenNIC tier-2	Local
192.168.1.45	metallb	`honeypot`	Trapeye	Local
192.168.1.46	metallb	`tv`	Plex Media Server	Local
192.168.1.48	metallb	`mail`	docker-mailserver	Cluster (real source IP via PROXY protocol from BPI-R4 haproxy)
192.168.1.49	metallb	`mirror`	Package mirror	Local
192.168.1.50	metallb	`squid`	Squid proxy	Local
192.168.1.51	metallb	`znc`	ZNC IRC bouncer	Local
192.168.1.52	metallb	`openldap`	OpenLDAP	Local
192.168.1.53	metallb	`nameserver`	BIND9 (public authoritative DNS)	Local
192.168.1.54	metallb	`technitium`	Technitium DNS (internal `home.tillo.ch`)	Cluster
192.168.1.55	metallb	`technitium`	Technitium DHCP (authoritative for LAN/DMZ/ADLAN)	Local
192.168.1.56	metallb	`tv`	Rsync	Local
192.168.1.57	metallb	`tv`	SFTPGo	Local
192.168.1.58	metallb	`ntppool`	Chrony NTP	Local
192.168.1.59	metallb	`garage-witness`	Garage RPC (in-cluster quorum witness)	Local
192.168.1.60	metallb	`cribl`	Cribl syslog receiver	Local
192.168.1.61	metallb	`windmill`	cert-host (Windmill cert delivery helper)	Local
192.168.1.62	metallb	`jump`	Jump pod (in-cluster jump seat — ssh -p443 + mosh)	Cluster
192.168.1.191	kube-vip	`kube-system`	`ingress-expose` (RKE2 ingress-nginx)	Local

tv also draws 192.168.1.47 (transmission-peer) alongside its .46/.56/.57 VIPs above. Free in pool: .63–.99.

ingress-expose data path

BPI-R4 DNATs external HTTPS (31.3.128.50:80,443) to 192.168.1.191:80,443. The kube-vip leader holds .191/32 on its mgmt-br. rke2-ingress-nginx runs as a DaemonSet binding hostPort: 80,443 on every node, so the DNATed packet on the leader's host is captured by the local nginx pod's port mapping (CNI portmap iptables DNAT), preserving the client source IP. ModSecurity then sees the real source.

DNS & DHCP Architecture¶

DNS is split across four components, each with a distinct role:

Component	Endpoint	Authoritative for	Used by
BIND9	`192.168.1.53` (ext: `31.3.128.59:53`)	Public zones — `mdapi.ch`, `tillo.ch`, etc.	External clients, cert-manager (RFC 2136 dynamic updates)
unbound (split-horizon)	`192.168.1.1` (in-cluster MetalLB VIP)	Override set — public hostnames answered with internal VIPs	LAN clients (DHCP option 6) + BPI-R4 dnsmasq upstream
BPI-R4 dnsmasq	`192.168.7.1`, `192.168.77.1`	— (pure forwarder, `no-resolv` → `192.168.1.1`)	DMZ / ADLAN clients (DHCP option 6)
Technitium	`192.168.1.54`	`home.tillo.ch` zone — auto-registers DHCP-leased hosts	unbound forwards (`home.tillo.ch` + every non-override query)
rke2-coredns	cluster-internal (`kube-dns` ClusterIP)	`cluster.local` and pod-side resolution	Every pod in the cluster

Resolution flow for an internal client:

Client queries the in-cluster split-horizon unbound at 192.168.1.1 (advertised via DHCP option 6; DMZ/ADLAN clients reach the same resolver through the router's dnsmasq forwarder).
If the hostname matches a split-horizon override (gitlab.mdapi.ch, notes.mdapi.ch, …) → unbound returns the internal VIP directly.
Everything else — including home.tillo.ch, which serves the live DHCP register — forwards to Technitium (192.168.1.54), which resolves external names via its upstream resolvers.

Resolution flow for a cluster pod:

CoreDNS is overridden (HelmChartConfig on the rke2-coredns chart) to two forward blocks rather than the default forward . /etc/resolv.conf:

The mdapi.ch and home.tillo.ch zones go to the in-cluster split-horizon unbound (forward . 192.168.1.1) with no fallback, to preserve split-horizon answers for our owned hostnames — a fallback here would return a technically-valid but wrong public answer, which is worse than none.
The default zone — everything else, including LAN reverse lookups — forwards to the same unbound resolver first, then falls through to NextDNS (45.90.28.16 / 45.90.30.16, paid filtering profile linked on the WAN IP) as the sole external fallback. policy sequential + health_check 5s + max_fails 2 means unbound handles the vast majority of queries and CoreDNS only reaches out to NextDNS on a real outage. Cloudflare (1.1.1.1) and Quad9 (9.9.9.9) are no longer in the chain — NextDNS is the cluster's only external resolver, same as the rest of the fleet.

This split keeps a router blip from cascading into a cluster-wide outage on container image pulls, runner clones, or any pod that talks to an external API — exactly the kind of incident the default single-upstream Corefile invites.

DHCP runs centrally on Technitium (192.168.1.55). BPI-R4 disables its own DHCP server on every VLAN and acts purely as a DHCP relay so all leases are registered and resolvable from a single source.

Lesson learned: the dnsmasq address directive is wildcard-based

address=/example.com/192.168.1.1 overrides the entire domain and every subdomain — it cannot be scoped to A-records only. For public domains (mdapi.ch, tillo.ch, …) the bpi-r4 dnsmasq sync only writes third-level entries (gitlab.mdapi.ch, never bare mdapi.ch); root-level entries would shadow MX/NS/TXT and break mail and registrar delegation.

Why Bare Metal + Harvester?¶

Harvester HCI runs RKE2 with KubeVirt integrated. iLO access on each node enables remote power management and out-of-band console access. The three-node setup provides etcd quorum; with most Longhorn volumes now 3-replica (one copy per node), a node can be drained for maintenance without a volume dropping below two live copies.