Certificates & PKI

Every TLS endpoint in the estate — cluster ingresses, appliances, network gear, NAS boxes, the router — carries a monitored, automatically renewed certificate. Three issuance planes and one lifecycle-management brain cover it all.

flowchart TD
    subgraph issuance["Issuance"]
        le["Let's Encrypt\ncert-manager, DNS-01"]
        stream["Evertrust Stream\ninternal PKI (X509 + SSH + TSA)"]
    end
    horizon["Evertrust Horizon\ncertificate lifecycle management\n(discovery, grading, renewal driving)"]
    subgraph consumers["Consumers"]
        k8s["Kubernetes ingresses\n+ in-cluster services"]
        fleetdev["Appliance fleet\nAPs, switches, iLO, NAS, router, UPS"]
    end
    ct["certspotter\nCT-log monitoring"]

    le --> k8s
    stream --> fleetdev
    horizon --> le
    horizon --> stream
    horizon -.->|renewal webhooks| windmill["Windmill f/infra_certs\ndevice delivery scripts"]
    windmill --> fleetdev
    ct -.->|never-seen SANs alert| horizon

Public certificates — cert-manager

Cluster ingress certificates come from Let's Encrypt via cert-manager (letsencrypt-prod, DNS-01 against the authoritative BIND with TSIG-signed updates — internal home.tillo.ch ingresses use HTTP-01 instead). Subject convention is deliberately minimal: CN + O + C only. Full chains are served everywhere (one legacy exception is tracked: iLO serves leaf-only by device limitation).

Internal PKI — Evertrust Stream

Stream runs the internal PKI: X.509 CAs for devices and services, SSH certificates, and timestamping. Its key material sits behind a KMS — the Horizon/Stream vault is backed by CipherTrust Manager through a Tink KMS extension that was developed in-house and published to Maven Central.

Lifecycle management — Evertrust Horizon

Horizon is the brain: it discovers certificates across the estate (including a Kubernetes discovery feed), grades them against policy, tracks expiry, and drives renewals. Renewal events trigger Windmill delivery scripts.

Device delivery — the interesting part

Getting a renewed certificate onto an appliance is where fleets usually give up. Here every device class has an automated delivery path — Windmill for most, a standalone Kubernetes CronJob for iLO — each speaking the device's native management interface:

Device class	Delivery mechanism
Aruba APs (AOS-6 and Instant)	SSH CLI push + config commit
HP MSM430 AP	SSH CLI flow (P12 fetched from an ephemeral in-cluster URL)
Zyxel AP	zysh CLI with explicit config persist
Synology / NAS hosts	SSH deployment into the cert store
BPI-R4 router	ACME with nsupdate DNS-01 against BIND
cert-manager web endpoints	native
CipherTrust Manager (internal web UI)	Windmill enrolls from Horizon private-tls, native REST push
iLO (server management controllers)	Standalone Kubernetes CronJob (not Windmill)

A dispatch script routes Horizon renewal webhooks to the right delivery flow — the path is frozen as a documented API surface.

Monitoring

• certspotter watches Certificate Transparency logs and pages only on never-seen SANs or non-Let's-Encrypt issuers — silence is the steady state, and any surprise issuance for owned domains is a page.
• Horizon expiry metrics feed the monitoring plane; appliance certificates are probed in place (the device's live TLS endpoint, not just the CLM database).