Windmill Automation Platform

Windmill is an open-source workflow automation platform running in the windmill namespace. It serves as the operational brain of the homelab — running scheduled health checks, version monitors, and maintenance flows.

Infrastructure Monitoring Schedule

All flows send Pushover alerts on findings. The Monday morning cadence runs checks sequentially to avoid parallel load on the cluster API.

flowchart LR
    sched["Cron scheduler\nEvery Monday\n07:00–07:55\nEurope/Zurich"]

    sched --> v["07:00\nweekly_version_check\nRKE2 / Rancher / Harvester\n9 Helm charts"]
    sched --> h["07:15\nweekly_infra_health\nFleet bundles / certs\nLonghorn backup ages"]
    sched --> s["07:25\nstorage_health\nTrueNAS + Synology\npool health via REST/SSH"]
    sched --> r["07:30\nrancher_backup_check\nBackup age verification"]
    sched --> b["07:35\nbpir4_health\nRouter connectivity + uptime"]
    sched --> k["07:40\nkeel_update_log\nRecent image updates"]
    sched --> rw["07:45\nresource_waste\nUnused PVCs\nzero-replica RSes"]
    sched --> rc["07:50\nregistry_cleanup_audit\nGitLab stale image tags"]
    sched --> la["07:55\nbackup_label_enforcer\nLonghorn backup labels"]

    v & h & s & r & b & k & rw & rc & la --> pushover["Pushover\nnotification"]

Cadence	Flow	Purpose
Daily	pod_image_cve_scan	Grype+Syft CVE scan of all running pod images
Weekly (Mon 07:00)	non_k8s_update_check	OPNsense, DSM (Synology), TrueNAS firmware versions
On-demand	gitlab/upgrade_flow	Semi-automated GitLab minor version upgrade
On-demand	gitlab_minio_health	GitLab MinIO runner cache health

Version Checking Pattern

The version check flow compares running versions against upstream release APIs:

flowchart LR
    flow["weekly_version_check"]

    flow -->|"Rancher API"| k8s["K8s version\n(mdapi-prod + mdapi-rancher)"]
    flow -->|"GitHub Releases API"| gh["Rancher / Longhorn / cert-manager\n+ 6 other Helm charts"]
    flow -->|"SSH + REST"| nas["TrueNAS / Synology\nfirmware versions"]

    k8s & gh & nas --> compare["Compare running vs latest"]
    compare -->|"upgrade available"| alert["Pushover alert\nwith versions"]
    compare -->|"up to date"| silence["No notification"]

Only actionable upgrades generate alerts — no noise for services that are current.

Postgres Persistence

Windmill's Postgres runs as a StatefulSet with a 5 Gi harvester-longhorn-2replicas PVC. The PV reclaim policy is Retain (patched from the default Delete at first deploy).

Fresh Windmill install = data loss

The default Helm chart does not configure persistence out of the box. A fresh install without a pre-existing PVC starts with an empty database — all workflows, schedules, and variables are lost. Always verify the PVC is bound and the PV reclaim policy is Retain before any Helm operation.