Monitoring & Alerting Architecture

Layers

Three planes, each owning a different signal class, all converging on VMAlertmanager:

graph TB
    subgraph metrics["Metrics plane (VictoriaMetrics)"]
        exp["node-exporter +<br/>kube-state-metrics<br/>(rancher-monitoring shell)"]
        bb["blackbox-exporter<br/>(synthetic probes)"]
        pg["Pushgateway<br/>(batch job heartbeat + gauges)"]
        vma["vmagent<br/>(VMServiceScrape / VMPodScrape)"]
        vms["VMSingle<br/>(3y retention · primary Grafana DS)"]
        valert["VMAlert<br/>(~400 rules)"]
        graf["Grafana 13<br/>(grafana.mdapi.ch)"]
    end

    subgraph logs["Log plane"]
        fb["fluent-bit DS<br/>(rancher-logging)"]
        flu["fluentd aggregator"]
        crib["Cribl LogStream<br/>(routing + filtering)"]
        relay["cribl-am-relay<br/>(Python; per-pattern detectors)"]
        o2["OpenObserve<br/>(logs.mdapi.ch, 7d on Ceph RGW)"]
    end

    subgraph alerting["Alert plane"]
        am["VMAlertmanager<br/>(send_resolved: true)"]
        adapt["statuspage adapter<br/>(webhook receiver)"]
        push["Pushover<br/>(severity-tiered)"]
        sp["statuspage.io<br/>(public)"]
        hc["healthchecks.io<br/>(dead-man switch)"]
    end

    exp & bb & pg -->|"vmagent scrape"| vma --> vms
    vms --> graf
    vms --> valert
    fb --> flu
    flu --> crib
    flu --> relay
    crib -->|"ES bulk"| o2

    valert -->|"rule firing"| am
    relay -->|"PostableAlert /api/v2/alerts"| am
    am -->|"team=mdapi route"| adapt --> sp
    am -->|"team=mdapi route"| push
    am -->|"alertname=WatchdogMdapi"| hc

Design principles

• VictoriaMetrics is the whole metrics plane. A vm-operator stack in the monitoring namespace: vmagent scrapes every target declared as a VMServiceScrape / VMPodScrape CR and writes into VMSingle (vmsingle-mdapi:8429, 3-year retention on Longhorn, the primary Grafana datasource). VMAlert evaluates ~400 alerting rules loaded from the PrometheusRule/VMRule CRs in the fleet repo's monitoring-rules/ bundle — plus the kube-prometheus recording rules (35-k8s-recording-rules.yml) that keep the upstream Kubernetes dashboards' pre-aggregated series alive — and fires into VMAlertmanager (vmalertmanager-mdapi-0). The rancher-monitoring chart's embedded Prometheus, Grafana and Alertmanager are gone; that Helm release survives only as an exporter shell — node-exporter, kube-state-metrics and dashboard ConfigMaps in cattle-monitoring-system.
• Cribl is a routing + filtering plane (cribl.mdapi.ch). fluent-bit + fluentd ship to Cribl's HTTP source; pipelines either (a) extract alert signals and POST to Alertmanager /api/v2/alerts via the cribl-am-relay, or (b) parse and forward into OpenObserve for indexed search. Real-time tail is via Cribl's Live Capture against the same stream.
• OpenObserve is the indexed-search layer. Single-node deployment in the openobserve namespace, writing Parquet to the in-cluster Ceph RGW S3 gateway (ceph-objectstore — the same multi-OSD object tier that carries GitLab's stores, chosen after a conclusive small-object benchmark). Streams are created on first ingest; retention is governed by ZO_DATA_RETENTION_DAYS (currently 7 days globally, per-stream override via PUT /api/<org>/streams/<name>/settings). Three streams are populated today — modsec, nat64 and ntp_clients (see Log search). Additional streams just need a Cribl route — or a small Windmill probe — to the same ES-compatible bulk endpoint with the desired index: value.
• GitOps via Fleet. Every scrape, rule and dashboard definition lives in the fleet repo. The handful of out-of-band touches that upstream charts own are documented inline so the recovery command is one copy-paste away if a chart upgrade resets them.

Grafana

Grafana runs as a standalone Deployment in the monitoring namespace (not a chart-bundled instance), published at grafana.mdapi.ch behind oauth2-proxy (auth.mdapi.ch). It is Grafana 13 with unified storage on CNPG Postgres — the grafana database on its own dedicated grafana-pg cluster — so dashboards, folders and preferences survive pod reschedules without a dashboard PVC.

Dashboards are provisioned two ways:

• The mdapi set from the fleet repo's monitoring-grafana/dashboards/*.json, packed into a ConfigMap by a regen script — dashboard changes are commits, reviewed and diffed like everything else.
• A k8s sidecar watches labelled ConfigMaps and provisions the Kubernetes / Rancher / Longhorn / Harvester folder sets shipped by their respective charts.

62 dashboards total, including /d/storage-stall (pressure-stall early warning), the VictoriaMetrics self-observability set and /d/ntppool-clients. A global 30-second minimum-refresh floor keeps auto-refreshing dashboards from hammering VMSingle.

Alert label convention

Every alert MUST carry these labels — the Alertmanager route, statuspage adapter, and Pushover template all key on them:

Label	Values	Purpose
severity	critical, warning, info	Maps to statuspage status: critical → major_outage, warning → degraded_performance
component	one of the 10 statuspage component names	Adapter looks this up in its component map and patches the matching statuspage component
team	mdapi	Distinguishes our alerts from imported upstream rule packs so VMAlertmanager routes can scope cleanly

Alert routing

Three destinations, keyed on the labels above:

• Pushover, severity-tiered. critical and warning page the operator; the info tier is recorded but not paged — auto-solving conditions stay info until they are actually stuck. Steady-state volume is ~10–15 messages/day.
• Public status page — the statuspage adapter (webhook receiver) patches component status on mdapi.statuspage.io so outsiders see outages without anyone touching the page by hand.
• healthchecks.io — the WatchdogMdapi dead-man route (see below).

Synthetic reachability itself comes from blackbox-exporter probes (next section), so the routing pipeline has signal even when the broken thing is the workload's own metrics endpoint.

Public statuspage components

The public status page is live at mdapi.statuspage.io — anyone can check service health or subscribe (email / Slack / RSS) for incident updates. Ten flat components (no groups, fits the free-tier ceiling), each mapping to one or more namespaces / probes:

Component	Covers
Mail	SMTP, IMAP, webmail, autoconfig
Sign-in (SSO)	Keycloak + OpenLDAP single sign-on (idp.mdapi.ch)
Platform	GitLab (source, CI, container registry), logging, automation, and shared platform tooling
Files & Documents	file sync (cloud.mdapi.ch), file gateways, notes (notes.mdapi.ch), document archive (documents.mdapi.ch), Debian/Ubuntu mirror
Backups	off-site backup pipelines + storage health (volumes, databases, config, Garage clusters, NAS)
Websites	personal + customer static sites
DNS	authoritative + recursive resolver + OpenNIC tier-2
Smart Home	Home Assistant, Frigate, MQTT, smart-plug integrations, remote-access VPN
Internet	WAN uplink, IPv6 + NAT64, BPI-R4 router
NTP	public NTP time server (mirror.mdapi.ch), pool.ntp.org contributor

Synthetic probes

Each statuspage component is backed by one or more VMProbe CRs targeting blackbox-exporter:

• HTTP /health/ready style endpoints where the app exposes one
• HTTP 200 | 301 | 302 | 401 | 403 (http_any_redirect module) for OIDC-gated services that 302 to login
• TCP banner check for SMTP, TCP+TLS handshake for IMAPS
• DNS A query for the authoritative resolver

probe_success == 0 for 5m → critical alert with the matching component label. Cert expiry < 14d on the probe_ssl_earliest_cert_expiry series → warning alert (backstop on cert-manager).

Workload health

Rule files in monitoring-rules/ keyed off kube-state-metrics:

• Deployment unavailable replicas > 0 for 5m → critical
• StatefulSet ready replicas < desired for 5m → critical
• Pod container restart rate > 0.5/min over 15m → warning
• Pod OOMKilled within last 1h → warning
• PVC fill > 85% → warning

Each rule sets team=mdapi and the appropriate component label so the routing pipeline does the right thing without rule-by-rule receiver wiring.

Beyond per-workload symptoms, pressure-stall information (PSI) metrics are collected on all three nodes. A NodeMemoryStallPSI rule fires as an early warning when memory stall time starts climbing — before the node visibly degrades — with /d/storage-stall as the companion dashboard.

Log-based alerts (cribl-am-relay)

A small Python relay (cribl-am-relay, in the cribl namespace) runs a chain of pattern detectors on log records, builds Alertmanager v2 PostableAlerts with proper labels + annotations + endsAt, and POSTs to /api/v2/alerts. Each detector is fed by its own narrow ClusterFlow that grep-prefilters at fluentd so only candidate lines hit the relay:

Detector	ClusterFlow (monitoring-rules/)	Pattern	Severity	Auto-resolve
ModSecBlockLikelyFalsePositive	15a-logging-flow-ingress-nginx.yml	ModSecurity: Access denied + client IP in trusted CIDR (192.168/16 except 192.168.164.2, 10/8, 100.64/10)	warning	5 min
PodIOError	14a-logging-flow-pod-io.yml	[Errno 5], blk_update_request.*I/O error, EXT4-fs error, XFS metadata/writeback error, buffer_io_error, EIO read/write, stale file handle, pcfg_openfile: Permission denied	warning	5 min
PostfixDeliveryFailure	14-logging-flow-postfix.yml	status=bounced|deferred, loops back, Connection refused in the mail namespace docker-mailserver pod	warning	5 min
RedisBGSAVEFailure	14b-logging-flow-redis-bgsave.yml	Background saving error, Failed opening the temp RDB file, Error saving DB on disk	warning	10 min

The relay tags each alert with team=mdapi plus a component derived from the namespace (Mail / Sign-in (SSO) / Platform / Files & Documents / Smart Home / Websites) so the existing Alertmanager route delivers it to statuspage + Pushover with no per-detector receiver wiring.

The all-logs catch-all ClusterFlow ships everything (except kube-system and cattle-logging-system) to Cribl for live search only (Live Capture against the same fluentd HTTP stream). It does not feed the relay — the catch-all volume saturates the relay's fluentd output buffer and triggers drop_oldest_chunk, so signal detection has to run from narrow grep-prefiltered flows.

The relay's modsec detector excludes 192.168.164.2 from the trusted-source check explicitly. That address is the Jool NAT64 v4 egress for external IPv6 clients (see external traffic on bpi-r4) — blocks from there are real attacks, not false positives. The companion address 192.168.164.3 (NAT64 v4 egress for internal LAN IPv6 clients) stays inside 192.168/16 and is treated as trusted, so blocks from there do alert as likely false positives.

Adding a new log-pattern detector

# In monitoring-cribl-am-relay/relay.yml, inside the server.py ConfigMap

MY_PATTERN_RE = re.compile(r"keyword1|keyword2", re.I)

def detect_my_signal(ev):
    line = (ev.get("log") or ev.get("message") or "").strip()
    if not line or not MY_PATTERN_RE.search(line):
        return None
    k = ev.get("kubernetes") or {}
    # (optional) scope by namespace / label
    if k.get("namespace_name") != "mynamespace":
        return None
    return {
        "labels": {
            "alertname": "MySignalDetected",
            "team":      "mdapi",
            "component": ns_to_component(k.get("namespace_name") or ""),
            "severity":  "warning",
            "signal":    "my_signal",
            "namespace": k.get("namespace_name", "?"),
            "pod":       k.get("pod_name", "?"),
        },
        "annotations": {
            "summary":     f"... {line[:120]}",
            "description": "...",
        },
        "endsAt": (datetime.now(timezone.utc) + timedelta(minutes=5)).isoformat().replace("+00:00", "Z"),
    }

DETECTORS = [detect_modsec, detect_pod_io, detect_postfix_failure, detect_redis_bgsave, detect_my_signal]

Steps:

1. Add detect_<name>(ev) and append to DETECTORS in relay.yml.
2. Add a narrow ClusterFlow under monitoring-rules/14<letter>-logging-flow-<name>.yml that grep-prefilters on the same pattern your detector matches, then ships to am-relay. Keep the grep regex aligned with the relay's regex — too narrow at the flow level = false negatives, too wide = relay buffer overflow. Do not widen the all-logs catch-all to feed the relay.
3. Commit + push.
4. Verify by injecting a synthetic event:

PAYLOAD='[{"log":"<line matching your pattern>","kubernetes":{"namespace_name":"<ns>","pod_name":"<pod>","container_name":"<c>"}}]'
kubectl -n windmill exec deploy/windmill-workers-default -- curl -s -X POST \
  http://cribl-am-relay.cribl.svc.cluster.local:9999/ingest \
  -H 'Content-Type: application/json' -d "$PAYLOAD"

# Then check the Alertmanager:
kubectl -n monitoring exec vmalertmanager-mdapi-0 -c alertmanager -- \
  wget -qO- 'http://localhost:9093/api/v2/alerts' | jq '.[] | select(.labels.alertname=="MySignalDetected")'

The relay's regex check (single substring before regex) runs at fluentd batch rate (~5s) for the ~6k events/min volume without measurable load.

Log search

Two surfaces, picked by time horizon.

Indexed historical search (up to retention) — OpenObserve at logs.mdapi.ch. SQL-over-Parquet on the in-cluster Ceph RGW object tier. Three streams are populated:

• modsec — every WAF audit JSON record and every nginx ModSecurity: Access denied line, with matched_rules, matched_messages, hostname, uri, client_ip, unique_id lifted to top-level columns by the Cribl pipeline.
• nat64 — per-session NAT64 forensics shipped from the router: session records from the Jool NAT64 gateway, so an external-IPv6 flow can be attributed to its v4-mapped session after the fact.
• ntp_clients — hourly aggregates of pool.ntp.org client activity against the stratum-1 server, pushed by a Windmill probe.

Default retention is 7 days; override per-stream when a longer horizon is justified. New streams just need a Cribl route — or a Windmill probe — pointing at the same ES-compatible bulk endpoint.

-- Where did this 403 come from? Root rule(s):
SELECT _timestamp, hostname, uri, method, http_code,
       matched_rules, matched_messages, client_ip
FROM modsec
WHERE hostname = 'rspamd.mdapi.ch' AND modsec_kind = 'audit'
ORDER BY _timestamp DESC LIMIT 10;

-- Which clients are being blocked the most right now?
SELECT client_ip, count(*) AS blocks
FROM modsec
WHERE modsec_kind = 'audit'
GROUP BY client_ip
ORDER BY blocks DESC LIMIT 10;

Real-time tail — Cribl Live Capture. Source / Route / Pipeline → Live Data tab (10-second sliding window of incoming events). Useful for ad-hoc investigations that are happening now.

• All logs from one namespace: kubernetes.namespace_name === '<ns>'
• Errors anywhere: /error|fail|exception|fatal|panic/i.test(log || '')
• Specific container: kubernetes.container_name === '<container>'

For aggregate metrics (pod restart counts, OOM rates, PVC fill), the kube_pod_* series are queried from VictoriaMetrics (3-year retention).

Batch-job metrics (Pushgateway)

Short-lived jobs — cluster sweeps, external API checks, anything that runs on a cron and exits — publish their results as Prometheus-format metrics through Pushgateway:

batch job ──push──▶ Pushgateway ──scrape──▶ vmagent ──▶ VMSingle ──VMAlert rule──▶ VMAlertmanager
                    (HTTP)                                        (statuspage + Pushover)

Each job pushes:

• <name>_last_run_timestamp_seconds — backstop "did it actually run?"
• <name>_last_run_success (1/0) — backstop "did it succeed?"
• Domain gauges produced by the check (backup age, days since last snapshot, S3 object count, …)

A meta BatchJobStale rule fires per-job at now - _last_run_timestamp_seconds > 2 × cadence_seconds, where cadence_seconds comes from windmill_schedule_interval_seconds published daily by f/probe/schedule_cadence_map. New scheduled probes are picked up automatically — there's no central regex list to maintain — but the probe's Pushgateway JOB label must equal basename(script_path) or the join silently misses. Each domain gauge has its own rule with the appropriate severity/component/team labels.

Pushgateway PUT replaces the whole job group

Probes pushing into a shared job group (i.e. anything that publishes to a /metrics/job/<name> where another script also publishes) must use POST. PUT wipes every metric in the group, including peers' _last_run_timestamp_seconds — which silently disables the BatchJobStale join. Use PUT only when the script owns its job group exclusively.

Dead-man switch

VMAlert evaluates an always-firing WatchdogMdapi rule. The VMAlertmanager config forwards it to a healthchecks.io ping URL every 4 minutes:

VMAlert emits WatchdogMdapi → AM matches alertname=WatchdogMdapi → webhook to hc-ping.com/<uuid>

If the alert plane is fundamentally broken (can't even deliver the keepalive), healthchecks.io sees the missed heartbeat and fires its own independent alert via email/Pushover. The ping URL is held in akeyless under /mdapi/pushover/healthchecks-watchdog-url and pulled via ExternalSecret.

Operational gotchas

Alert annotation templates aren't sprig

Alert annotations support Go's text/template plus a small set of Prometheus-style helpers (humanize, humanizeDuration, query, reReplaceAll, safeHtml, title). Sprig functions like default, lower, quote are NOT available — a rule using them is rejected with function "default" not defined.

fluentd flush_interval is silently ignored unless flush_mode: interval is set

Default flush_mode: lazy uses timekey-based flushing — events sit in buffer for ~10min+ before shipping. Set both options explicitly when you want responsive output.

logging-operator match: [{exclude: …}] doesn't route on its own

A ClusterFlow with only an exclude block produces a fluentd label_router route that never matches. Need an explicit select: {} first (match everything), then exclude. Symptom: fluentd_output_status_emit_count = 0 on the flow despite ClusterFlow status showing Active.

Resource Sizing & VPA

The Vertical Pod Autoscaler runs cluster-wide. Most workloads are in Auto mode (VPA mutates live pod requests on eviction), a smaller set stays in Off mode (recommendations are observed but not applied — used for spiky workloads where surprise eviction would be disruptive: GitLab webservice/sidekiq/gitaly, Frigate, Home Assistant, docker-mailserver, etc.).

For Off-mode workloads, requests and limits are set explicitly in the manifest from the VPA's target and upperBound recommendations. The scheduler then has real numbers to work with and noisy neighbours stay bounded.

VPA Auto preserves limit:request ratio

When VPA Auto bumps a request to its target, it scales the limit by the same factor. A Deployment written with request 200m / limit 2 (a 10× ratio) and a VPA target of 2 cores ends up with a 20-core live limit — enough to push a single node past 100% CPU-limit overcommit on its own and trip KubeCPUOvercommit, without any obvious culprit (the Deployment manifest still says limit: 2).

For workloads under VPA Auto, write request:limit close to 1:1. If a workload genuinely needs burst headroom (transcoder, batch job), keep its VPA in Off mode and size the limit explicitly — VPA's proportional scaling is the wrong tool for spiky workloads anyway. When debugging unexpected per-node CPU overcommit, compare the live pod spec against the Deployment spec — a divergence is the fingerprint of VPA mutation.

Longhorn backup monitoring thresholds

A single LonghornBackupStale alert covers every backed-up volume: it fires when longhorn_backup_age_days exceeds 40 days. The metric carries no per-group label, so there's no split by recurring-job cadence — one flat threshold, wide enough to absorb a single missed monthly cycle without a false-positive alert. LonghornBackupMissing (longhorn_backup_age_days == -1) separately covers a volume that has never completed a backup at all.

Stale Longhorn mount: detection + remediation

A class of failures manifests as filesystem EIO from a pod even though Longhorn reports the volume attached + healthy and dmesg on the node is clean. Symptoms:

• Pod up for many days, files in the mount have stale modification times
• Writes return EIO (I/O error, [Errno 5] Input/output error)
• /proc/mounts shows rw, perms are correct
• The Longhorn engine has restarted out from under the pod at some point and the pod's I/O session is stale

Fix is always a pod restart — the new pod gets a fresh mount and writes resume immediately. A Windmill flow does this automatically for CrashLoopBackOff + EIO-in-logs, and a companion LonghornVolumeReadOnlyRemount alert covers the silent Running-but-readonly case the log signature misses. See Longhorn → Mount-stall auto-recovery for the full picture.

The PodIOError detector in cribl-am-relay matches the EIO log signature across all namespaces and fires an alert tagged with namespace / pod / container / component, so an operator who wants to act before the auto-healer wakes up has the offending pod in the notification. A separate LonghornVolumeDegraded / LonghornVolumeFaulted rule keys on longhorn_volume_robustness so a degraded volume is alerted even before a workload tries to read or write to it.