Windmill Automation Platform

Windmill is an open-source workflow automation platform running in the windmill namespace. It serves as the operational brain of the homelab — running scheduled health checks, self-healing reconcilers, certificate delivery, and maintenance flows.

Workspace conventions

The workspace is organised by a folder taxonomy — the folder tells you what a script is allowed to do:

Folder	Role
f/probe/*	Read-only checks — observe, publish gauges to Pushgateway, never mutate
f/heal/*	Reconcilers — bounded, idempotent corrective actions (stalled Longhorn rebuilds, stuck replica sets, label enforcement)
f/config_backup/*	Daily config snapshots of appliances and services
f/security/*	CVE + secret-leak scanning and their ignore-list curators (see Supply Chain Security)
f/infra_certs/*	Certificate renewal + delivery to devices that can't run ACME themselves (APs, NAS, wireless controllers)
f/desk/*	Personal automation (inbox curation, review queues, watchers)
f/agent_tools/*	MCP-facing tools called by AI agents
f/lib/*	Shared library code — f/lib/mdapi (logging, state, metrics helpers)
f/portal/*, f/marketplace/*, f/envuassu/*	App backends with externally-bound paths — frozen: webhooks and app UIs reference them by path
f/secrets/*	f/secrets/get_secret — Akeyless-primary / OpenBao-fallback secret fetch
f/gitlab/*	GitLab upgrade pipeline + registry audits

Two conventions do a lot of quiet work:

• Schedule path == script path, with a handful of legacy exceptions that add a _schedule suffix to disambiguate — the cadence of anything is still one lookup away.
• Pushgateway JOB label == script basename. Probes publish <name>_last_run_timestamp_seconds plus domain gauges; the BatchJobStale meta-rule joins on the basename, so every new scheduled probe is alert-covered automatically. The flip side: renaming a path is an API break — basenames are metric labels, and the frozen folders are wired into external webhooks.

Monday morning health train

All flows send Pushover alerts on findings. weekly_version_check and non_k8s_update_check actually run every morning at 07:00, and bpir4_health every 15 minutes; the rest of the train — storage_health and resource_waste — is Monday-only, and runs sequentially to avoid parallel load on the cluster API.

flowchart LR
    daily["Cron scheduler\nDaily 07:00\nEurope/Zurich"]
    freq["Cron scheduler\nEvery 15 min\nEurope/Zurich"]
    mon["Cron scheduler\nMon 07:30–08:00\nEurope/Zurich"]

    daily --> v["07:00\nweekly_version_check\nRKE2 / Rancher / Harvester\n+ 8 Helm charts"]
    daily --> n["07:00\nnon_k8s_update_check\nSynology DSM firmware"]
    freq --> b["bpir4_health\nUptime / WAN\nkernel errors"]
    mon --> s["07:30\nstorage_health\nSynology array +\nvolume health via SSH"]
    mon --> rw["08:00\nresource_waste\nUnused PVCs /\nzero-replica RSes"]

    v & n & s & b & rw --> pushover["Pushover\nnotification"]

Representative schedules

The full schedule list lives in the workspace itself (and is exported daily to metrics by f/probe/schedule_cadence_map); one representative example per folder family:

Cadence	Path	Type	Purpose
Every 5m	f/heal/longhorn_rebuild_stuck	script	Tracks active Longhorn replica rebuilds via Pushgateway gauge; auto-deletes a stalled WO replica after 30 min of no progress (cap 1 per run) so Longhorn re-schedules on a healthier node
Every 5m	f/probe/ntp_health	script	Functional NTP probe — queries the ntppool chrony server over NTP and publishes sync state to Pushgateway. Pod-level availability is not proof the daemon is serving time
Hourly	f/probe/ntppool_clients	script	Aggregates pool.ntp.org client activity into the ntp_clients OpenObserve stream
Hourly :15	f/probe/daily_infra_health	script	Fleet bundle readiness, cert expiry, Longhorn backup ages, node disk pressure — hourly so a transient bad value self-corrects within the hour
Weekly Sun 04:10	f/heal/ingress_worker_flush	script	Rolling restart of the ingress-nginx DaemonSet, one node at a time
4×/day (02:25 · 08:25 · 14:25 · 20:25)	f/security/pod_image_cve_scan	script	Grype CVE scan, rotating batch of 12 images/run across all registries (incl. docker.io) — full sweep of all running-pod images ~2-3 days
Daily 03:00	f/config_backup/config_backup	script	Daily snapshot of mdapi configs
Daily	f/probe/schedule_cadence_map	script	Publishes windmill_schedule_interval_seconds{job=...} for every scheduled probe so the BatchJobStale rule can fire at 2 × interval per-job
Staggered, 04:00–06:47 daily	Nine per-device f/infra_certs/* scripts	script	Device-certificate renewals (APs, NAS, controllers) — independent, individually-staggered scripts, one per device; cert_renew_dispatch itself carries no active schedule
Daily 04:00	f/heal/backup_label_enforcer	script	Ensures all PVCs carry the correct Longhorn recurring-job labels

Version Checking Pattern

The version check flow compares running versions against upstream release APIs:

flowchart LR
    flow["weekly_version_check"]

    flow -->|"Rancher API"| k8s["K8s version\n(mdapi-prod + mdapi-mgmt)"]
    flow -->|"GitHub Releases API"| gh["Rancher / cert-manager\n+ 7 other Helm charts"]

    k8s & gh --> compare["Compare running vs latest"]
    compare -->|"upgrade available"| alert["Pushover alert\nwith versions"]
    compare -->|"up to date"| silence["No notification"]

Only actionable upgrades generate alerts — no noise for services that are current.

Postgres Persistence

Windmill's Postgres runs as a StatefulSet with a 5 Gi harvester-longhorn-2replicas PVC. The PV reclaim policy is Retain (manually set — the default for dynamically provisioned PVs is Delete).

Fresh Windmill install = data loss

The default Helm chart does not configure persistence out of the box. A fresh install without a pre-existing PVC starts with an empty database — all workflows, schedules, and variables are lost. Always verify the PVC is bound and the PV reclaim policy is Retain before any Helm operation.

Database credential

The app and every worker group read DATABASE_URL from the Secret windmill-database-url. The ExternalSecret in the fleet bundle windmill/external-secrets builds it from the Akeyless item /mdapi/windmill/windmill-postgresql/postgres-password.

The role password lives in the database, not the Secret: a rotation writes a new Akeyless version, sets the role password in-database without exposing it in pg_stat_activity, then rolls the app and workers.

Port 5432 is reachable only from the app, the workers and openobserve/o2-dbm-collector. The Calico policy enforce.windmill-postgresql in the bundle windmill/netpol enforces this. o2_dbm has its own role.