Runtime Detection (Falco)

Falco watches system calls on every node and reports when something unexpected happens inside a running container — a shell spawned where none should be, a binary dropped and executed, a sensitive file read.

It fills a gap the other layers leave by design. The WAF blocks at the edge, admission control refuses what should never start, network policy restricts what may talk to what — but none of them answers the question "did something unexpected just run in there?". That is a detection question, and it needs a detector.

flowchart LR
    k["Kernel<br/>syscalls"] -->|"modern eBPF<br/>(CO-RE)"| f["Falco<br/>DaemonSet<br/>1 per node"]
    f -->|"rule matches"| m["Metrics<br/>VictoriaMetrics"]
    f -->|"JSON events"| l["Logs<br/>OpenObserve"]
    m --> a["vmalert<br/>health + novelty"]
    m --> g["Grafana<br/>Security folder"]

Falco was introduced in September 2026, deployed through Fleet like everything else on the platform.

Why it fits this estate at all

The driver, and nothing else. The nodes run a 6.12 kernel with BTF type information available, which means Falco's modern eBPF driver (CO-RE) works without a kernel module.

That is the whole argument. The platform already carries one out-of-tree module, rebuilt against every new kernel by a dedicated DaemonSet (see Kubernetes Namespaces, cephfs-kmod). A second one — with the same rebuild-on-every-upgrade burden — would have been hard to justify. The eBPF driver avoids that entire class of problem.

The same choice keeps Falco least-privileged: instead of a privileged container it runs with a short list of explicit capabilities (BPF, perf monitoring, resource limits, ptrace), and it needs no host networking, so the estate's never-host-network rule survives intact.

Settings that are not the chart default, and why

• The ruleset does not update itself. The chart ships a sidecar that re-pulls the upstream rules every week and swaps them under a running Falco. That is a floating dependency inside an otherwise pinned, reviewed deployment: detection behaviour would change with no merge request, and afterwards nobody could say which rules were live when an event fired. Rules change when git says so.
• The minimum priority is raised. The default emits debug-level events — a firehose on a cluster that also runs virtual machines, and a cost paid in log volume for events nobody reads.
• Dropped system calls are alerted on. A dropped event is a blind window, not a performance curiosity.
• The container-runtime socket is pinned explicitly, even though the chart would probably find it on its own. It would find it by probe order, and the failure mode is silent: Falco keeps running and every event quietly loses its pod, namespace and image.

How it was rolled out

A detector nobody trusts gets muted, and a muted detector is worse than none. The rollout was staged so that trust was earned before anything could page.

Phase	What happened	Why in that order
1 — Observe	Deployed with no alerting and no exceptions	A threshold invented before the first measurement is a guess
2 — Tune	Exceptions written from what actually fired, scoped to the specific workload — never by disabling a rule	Writing exceptions first would suppress the very evidence they should be based on
3 — Alert	Health, coverage and novelty alerts; each one proven to fire by deliberately inverting it	An alert rule that has never fired is a hope, not a control
4 — Kubernetes audit source	Decided no	See below
5 — Automated response	Decided no	See below

Novelty is the alert worth dwelling on: it fires the first time a rule matches that has never matched before. After tuning, the steady state is quiet — so a rule firing for the first time is, by construction, the thing worth looking at.

Why the Kubernetes audit source stays off

Falco can also consume the Kubernetes API audit log. It was evaluated and deliberately left disabled. What that source can see depends entirely on the API server's audit policy, which on this estate is owned by the distribution: changing it means editing a platform-managed control-plane file that an upgrade would silently re-seed, and a malformed policy does not degrade — it stops the API server from starting.

Without that change, the source cannot see what most of its high-value rules depend on. An enabled-but-blind detection source is worse than an absent one, because it looks like coverage. The decision is revisited only if the distribution makes the audit policy a supported setting.

Why there is no automated response

Response automation is only as good as the detection it trusts, and this detector is new. The platform has already made the same call elsewhere — network policy is authored by hand rather than generated — and the reasoning carries: a wrong automated response is an outage you caused yourself, during an incident you were already having. It is revisited after the exception set has been stable for a quarter.

Cost, booked in advance

Falco requests a fixed amount of memory on every node, and that was accounted against the cluster's headroom before deployment rather than discovered afterwards.

It deliberately runs at the default priority class, not the shed-first one. Under memory pressure the lowest-priority workloads are evicted first — and a detector evicted under pressure is absent exactly when the cluster is behaving strangely.

Two surfaces, because neither is enough alone

Metrics count matches per rule and priority. They drive the dashboard in Grafana's Security folder and the alerts — but their only Kubernetes labels are Falco's own pod. They say that something fired, never where.

Events carry the attribution: pod, namespace, image, process, command line. They are collected with the rest of the platform's logs and searched in OpenObserve — which also means no log-platform credential had to be placed on pods that run with host-level visibility on every node.

Verify by behaviour, not by Ready

A Running pod proves nothing about whether the probe is attached. The checks that matter: the DaemonSet reports one pod per node, Falco reports the modern eBPF driver, no Falco kernel module is loaded, and a deliberate trigger in a scratch pod produces an event carrying the pod and namespace — not just a process ID.