Windmill Automation Platform

Windmill is an open-source workflow automation platform running in the windmill namespace. It serves as the operational brain of the homelab — running scheduled health checks, self-healing reconcilers, certificate delivery, and maintenance flows.

Workspace conventions

The workspace is organised by a folder taxonomy — the folder tells you what a script is allowed to do:

Folder	Role
f/probe/*	Read-only checks — observe, publish gauges to Pushgateway, never mutate
f/heal/*	Reconcilers — bounded, idempotent corrective actions (stalled Longhorn rebuilds, stuck replica sets, label enforcement)
f/config_backup/*	Daily config snapshots of appliances and services
f/security/*	CVE + secret-leak scanning and their ignore-list curators (see Supply Chain Security)
f/infra_certs/*	Certificate renewal + delivery to devices that can't run ACME themselves (APs, NAS, wireless controllers)
f/desk/*	Personal automation (inbox curation, review queues, watchers)
f/agent_tools/*	MCP-facing tools called by AI agents
f/lib/*	Shared library code — f/lib/mdapi (logging, state, metrics helpers)
f/portal/*, f/marketplace/*, f/envuassu/*	App backends with externally-bound paths — frozen: webhooks and app UIs reference them by path
f/secrets/*	f/secrets/get_secret — Akeyless-primary / OpenBao-fallback secret fetch
f/gitlab/*	GitLab upgrade pipeline + registry audits

Two conventions do a lot of quiet work:

• Schedule path == script path. Every schedule is named after the script it runs, so the cadence of anything is one lookup away and orphaned schedules are immediately visible.
• Pushgateway JOB label == script basename. Probes publish <name>_last_run_timestamp_seconds plus domain gauges; the BatchJobStale meta-rule joins on the basename, so every new scheduled probe is alert-covered automatically. The flip side: renaming a path is an API break — basenames are metric labels, and the frozen folders are wired into external webhooks.

Monday morning health train

All flows send Pushover alerts on findings. weekly_version_check and non_k8s_update_check actually run every morning at 07:00; the rest of the train — storage_health through resource_waste — is Monday-only, and runs sequentially to avoid parallel load on the cluster API.

flowchart LR
    daily["Cron scheduler\nDaily 07:00\nEurope/Zurich"]
    mon["Cron scheduler\nMon 07:30–08:00\nEurope/Zurich"]

    daily --> v["07:00\nweekly_version_check\nRKE2 / Rancher / Harvester\n+ 9 Helm charts"]
    daily --> n["07:00\nnon_k8s_update_check\nDSM (Synology) /\nTrueNAS firmware"]
    mon --> s["07:30\nstorage_health\nTrueNAS + Synology\npool health via REST/SSH"]
    mon --> b["07:45\nbpir4_health\nUptime / WAN\nkernel errors"]
    mon --> k["07:55\nkeel_update_log\nRecent image updates"]
    mon --> rw["08:00\nresource_waste\nUnused PVCs /\nzero-replica RSes"]

    v & n & s & b & k & rw --> pushover["Pushover\nnotification"]

Representative schedules

The full schedule list lives in the workspace itself (and is exported daily to metrics by f/probe/schedule_cadence_map); one representative example per folder family:

Cadence	Path	Type	Purpose
Every 5m	f/heal/longhorn_rebuild_stuck	script	Tracks active Longhorn replica rebuilds via Pushgateway gauge; auto-deletes a stalled WO replica after 30 min of no progress (cap 1 per run) so Longhorn re-schedules on a healthier node
Every 5m	f/probe/ntp_health	script	Functional NTP probe — queries the ntppool chrony server over NTP and publishes sync state to Pushgateway. Pod-level availability is not proof the daemon is serving time
Hourly	f/probe/ntppool_clients	script	Aggregates pool.ntp.org client activity into the ntp_clients OpenObserve stream
Hourly :15	f/probe/daily_infra_health	script	Fleet bundle readiness, cert expiry, Longhorn backup ages, node disk pressure — hourly so a transient bad value self-corrects within the hour
No active schedule	f/gitlab/upgrade_auto_flow	flow	Unattended GitLab patch / one-minor upgrade — see GitLab → Automated upgrades. Currently run on demand only; the flow carries no recurring Windmill schedule.
4×/day (02:25 · 08:25 · 14:25 · 20:25)	f/security/pod_image_cve_scan	script	Grype+Syft CVE scan of all running pod images, all registries (incl. docker.io)
Daily 03:00	f/config_backup/config_backup	script	Daily snapshot of mdapi configs
Daily	f/probe/schedule_cadence_map	script	Publishes windmill_schedule_interval_seconds{job=...} for every scheduled probe so the BatchJobStale rule can fire at 2 × interval per-job
Staggered, 05:23–06:47 daily	Seven per-device f/infra_certs/* scripts	script	Device-certificate renewals (APs, NAS, controllers) — independent, individually-staggered scripts, one per device; cert_renew_dispatch itself carries no active schedule
Wed 04:00	f/heal/backup_label_enforcer	script	Ensures all PVCs carry the correct Longhorn recurring-job labels

Version Checking Pattern

The version check flow compares running versions against upstream release APIs:

flowchart LR
    flow["weekly_version_check"]

    flow -->|"Rancher API"| k8s["K8s version\n(mdapi-prod + mdapi-rancher)"]
    flow -->|"GitHub Releases API"| gh["Rancher / Longhorn / cert-manager\n+ 6 other Helm charts"]
    flow -->|"SSH + REST"| nas["TrueNAS / Synology\nfirmware versions"]

    k8s & gh & nas --> compare["Compare running vs latest"]
    compare -->|"upgrade available"| alert["Pushover alert\nwith versions"]
    compare -->|"up to date"| silence["No notification"]

Only actionable upgrades generate alerts — no noise for services that are current.

Postgres Persistence

Windmill's Postgres runs as a StatefulSet with a 5 Gi harvester-longhorn-2replicas PVC. The PV reclaim policy is Retain (manually set — the default for dynamically provisioned PVs is Delete).

Fresh Windmill install = data loss

The default Helm chart does not configure persistence out of the box. A fresh install without a pre-existing PVC starts with an empty database — all workflows, schedules, and variables are lost. Always verify the PVC is bound and the PV reclaim policy is Retain before any Helm operation.