CIS Kubernetes Benchmark

The CIS Kubernetes Benchmark is run against the production cluster weekly, by a CronJob in its own namespace, with the result published as metrics and alerted on regression.

The weekly cadence is the whole point. A benchmark run once is an audit; a benchmark run on a schedule, with a threshold, is a control. The question worth answering is not "is this cluster compliant today" — it is "did anything move since the last time we looked".

flowchart LR
    cron["CronJob<br/>weekly<br/>1 pod per node"] --> kb["kube-bench<br/>pinned image<br/>pinned profile"]
    kb --> jq["jq: totals + per-section"]
    jq -->|"POST"| pg["Pushgateway"]
    pg -->|"vmagent scrape"| vm["VictoriaMetrics"]
    vm --> alert["vmalert<br/>regression alert"]

What this page is, and is not

This page documents how the measurement is organised and how its results are interpreted. The findings themselves — which checks fail, what the policy sections measure, and what is being remediated — are an internal report and are deliberately not published here.

How a scanner result becomes a finding

The interesting part of running a benchmark is not the score. It is the triage, because a scanner verdict is a starting point, not a finding — in both directions.

A failing check can be an artefact of detection. Benchmark profiles encode where they expect configuration to live, and distributions move. This cluster's kubelet is started with a drop-in directory rather than a single config file — a newer RKE2 shape — so every check that expects the older path reports failure for settings it never managed to read. Reading the effective configuration shows those settings are exactly as the benchmark wants them.

A failing check can also be a literal deviation whose objective is met another way. A file-permission check compares a mode against a constant. If the directory above it is 0700 root:root, no unprivileged principal can reach the file regardless of its own bits, and if the file carries no credential there is nothing to protect anyway. That is worth recording as an accepted deviation with the reasoning — not as a backlog item pretending it will be fixed, and not as a "fix" applied to a file the distribution rewrites on every restart.

And a passing check proves less than it looks. A check can pass because it found nothing to object to rather than because the control exists.

The only reliable move is to read the effective configuration — the drop-in contents, the running process arguments, the live API — rather than trusting the benchmark's model of where things are.

Each result is therefore classified before it is acted on:

Class	Meaning	Action
Genuine finding	Control absent, objective not met	Remediate, or carry as a tracked risk
Accepted deviation	Literal check fails, objective met another way	Record with reasoning; revisit at review
Detection artefact	Check could not read what it was looking for	Verify by reading the real configuration

The scored sections and the policy sections are different work

Sections 1 to 4 are configuration checks against a distribution that ships mostly hardened. They automate well, and their value is regression detection rather than discovery.

Section 5 is entirely "Manual" items — RBAC shape, workload privilege, network segmentation, secret handling. kube-bench prints guidance and scores nothing, and that is not a gap in the tool: these are architecture questions, and a scanner cannot answer them.

So section 5 is answered separately, by querying the live cluster and turning each item into a number: how many bindings carry which roles, how many workloads request which privileges, how much of the namespace estate is covered by network policy. Those measurements feed the internal architecture review and the risk register, where they belong — alongside an owner, a treatment and an accepted residual.

Design decisions worth knowing

The image and the profile are pinned — and the pin still moves

Not tidiness. The same cluster, audited minutes apart with an unpinned image and a pinned one, produced different totals and a different verdict on three checks — different image, different benchmark definitions. Unpinned, the regression alert would fire on a tooling change rather than on anything that happened to the estate, and the on-call response to an alert that cries wolf is to stop trusting it.

But a pin is not a freeze. The image is pinned in git, and Renovate raises a merge request when a new release ships — which is exactly what should happen. So one day the regression alert fired, nothing on the estate had changed, and the cause was the tool: a merged version bump had changed what is measured. Running the old and the new release against the same nodes at the same moment proved it.

Two consequences, both now built in:

• The accepted baseline is recorded against the tool version, in the alert rule itself. The first question on a regression is whether the measuring tool moved, and the alert now asks it.
• When the tool moves, the baseline is re-derived from the list of checks, never from the total. A one-point change in the total hid a much larger churn underneath — checks retired, others introduced. A threshold on an aggregate can look stable while most of what it summarises has changed.

A re-derivation is a fresh review, not a rubber stamp: every newly reported check is verified against the node's effective configuration rather than the check's reading of it, and classified. Whatever turns out to be real goes to the internal risk register — which is also where reading the configuration directly had, in one case, already recorded a setting before the benchmark started testing for it.

The alert fires above the accepted baseline, not above zero

The threshold is the measured count of accepted-or-artefact failures, not zero. A threshold parked above the real count is worse than having no alert at all: it looks like coverage while being unable to fire. As items are genuinely remediated, the threshold comes down with them — which also means the baseline has to be re-derived rather than inherited.

It routes to an internal component, not a public one

Alert routing on this estate is driven by a component label, and some components are public — an alert carrying one degrades the public status page. A benchmark regression is a posture finding that no user can observe, so it carries an internal component: it reaches the operator and goes no further. See Monitoring → Alert label convention.

The same reasoning governs this page.

Why it has its own staleness rule

The house convention is that no job needs a per-job staleness alert, because a generic rule covers anything publishing a last-run timestamp. It does — for Windmill jobs. That rule joins against a cadence table published per Windmill script path. A Kubernetes CronJob publishes no cadence entry, so the join has no right-hand side and the generic alert can never fire for it — silently.

Hence a scoped staleness rule, with the reasoning recorded in the rule file so a future reader does not delete it as a duplicate. The general lesson: a convention that says "the generic rule covers this" is worth testing against the specific case, because a join that matches nothing looks exactly like a control that never trips.

One pod per node

The nodes are configured identically by the hypervisor platform — but "identical by construction" is an assumption, and auditing each node is what turns it into a measurement. The Job runs one pod per node via required anti-affinity.

The first run confirmed the assumption. It is worth having the measurement anyway, because the day they diverge is exactly the day nobody expects it.

The namespace is fenced, with a stated limit

The workload is privileged and mounts host paths to do its job, which makes it precisely the kind of namespace worth bounding. It carries a deny-by-default NetworkPolicy with narrow egress allowances for the few destinations it needs.

NetworkPolicy does not contain a privileged pod

It governs pod-network traffic and nothing else. A privileged + hostPID pod's access to the node itself is entirely outside its scope. The fence reduces lateral movement across the cluster network; it is not an isolation boundary for this workload, and treating it as one would be a mistake.

An egress rule written against a ClusterIP silently drops everything

Egress is evaluated after the ClusterIP is DNAT'd, so the destination that has to be allowed is the backing address, not the service IP. The rule looks correct and drops every call.

Encryption at rest is measured, not assumed

One benchmark item asks whether encryption providers are appropriately configured, and leaves it Manual. Configuration was checked — and then the stored bytes were checked, because enabling encryption does not encrypt retroactively: anything written before the provider was configured stays in the clear until something rewrites it.

Reading the objects straight out of the datastore and inspecting their stored prefixes is the only way to answer the question. The result is clean: every secret is stored encrypted, none in plaintext.

That measurement produces a large, sensitive artefact

A full dump of every secret object is hundreds of megabytes and, although encrypted, is still sensitive in bulk. Delete it as soon as the count is taken. This is a good example of a verification step that must not leave residue behind.

An implementation note that costs an afternoon

Pushgateway rejects a metrics body whose last line has no trailing newline with a bare 400 Bad Request, and wget --post-data=STR sends the string exactly as given. Shell command substitution strips trailing newlines, so every obvious way of appending one fails silently.

printf '%s\n' "$BODY" > /tmp/push.txt
wget -q -O- --post-file=/tmp/push.txt "$PUSHGATEWAY/metrics/job/<job>/node/$NODE_NAME"

--post-file is the form where the newline survives.

Published series cover the four result classes, a scored ratio, per-section counts, and the last-run timestamp/success pair that every batch job on this estate publishes — all labelled by node.