Skip to content

ModSecurity WAF

ModSecurity 3.0.14 (connector v1.0.4) runs inline on every ingress-nginx instance, enforcing OWASP Core Rule Set (CRS) 4.22.0 on inbound HTTP/S traffic. Blocking is the default (SecRuleEngine On); the exceptions — a host where the WAF is switched off entirely, or one held in detection-only — are named per host in the global configuration rather than decided request by request.

How a Block is Diagnosed

Every WAF-enabled ingress writes the ModSecurity audit log as single-line JSON to stdout (SecAuditLogType Serial, SecAuditLogFormat JSON, SecAuditLog /dev/stdout, with SecAuditEngine RelevantOnly and SecAuditLogRelevantStatus "^(?:5|4(?!04))"). The status filter does not make this a block log: under RelevantOnly any transaction that matched at least one rule is recorded whatever its HTTP status — over one 7-day window the modsec stream held 352k HTTP-200 records carrying matched_rules, against 30k 403s. How much the engine sees is therefore computed from matched_rules, never from http_code = 403. Fluent Bit ships those records via Fluentd to Cribl, where a filter route forwards anything matching \"transaction\":{ or ModSecurity: Access denied into OpenObserve's modsec stream (7-day retention).

flowchart LR
    client["Client request"] -->|"HTTP 403"| nginx["ingress-nginx\nModSecurity"]
    nginx -->|"audit JSON to stdout"| fb["fluent-bit + fluentd"]
    fb --> cribl["Cribl\n(filter + parse)"]
    cribl --> o2["OpenObserve\nmodsec stream (7d)"]
    o2 --> sql["SQL: matched_rules"]
    sql --> fix["Apply exception\nin global modsec config"]

Rule 949110 is always the final aggregator. The root rule(s) live in the audit record's matched_rules array, which the Cribl pipeline lifts to a top-level column in OpenObserve.

# Find the root rule(s) for a recent block
O2_PASS=$(akeyless get-secret-value --name /mdapi/openobserve/o2/root-password | tr -d '\n"')
curl -sS -u "<your-o2-user>:$O2_PASS" -X POST -H "Content-Type: application/json" \
  -d "{\"query\":{\"sql\":\"SELECT _timestamp, hostname, uri, method, http_code, matched_rules, matched_messages, client_ip FROM modsec WHERE hostname = '<host>' AND modsec_kind = 'audit' ORDER BY _timestamp DESC LIMIT 10\",\"start_time\":$(( ($(date +%s) - 3600) * 1000000 )),\"end_time\":$(( $(date +%s) * 1000000 ))}}" \
  "https://logs.mdapi.ch/api/default/_search" | jq

Fallback for blocks older than the 7-day retention or when OpenObserve is unavailable: tail the live ingress-nginx error log directly with kubectl -n kube-system logs -l app.kubernetes.io/name=rke2-ingress-nginx --tail=-1 | grep ModSecurity (only ~1 hour of history due to kubelet rotation).

Common Rules

Rule Description Score
911100 HTTP method not allowed 5
941100 XSS via libinjection 5
942100 SQLi via libinjection 5
932100–932180 Unix shell / RCE detection 5
949110 Anomaly score exceeded (aggregator) —

Default anomaly thresholds: 10 inbound / 8 outbound.

Applying Exceptions

Exceptions live in a single global ModSecurity ConfigMap, which the ingress controller includes once:

nginx.ingress.kubernetes.io/modsecurity-snippet: |
  Include /etc/nginx/modsec-global/modsec-global.conf

Every exception inside that file is scoped to a specific host, so the active inventory stays deliberately bounded — roughly 30 scoped rule exceptions across the whole fleet, each tied to a specific host and path, reviewed as a set during WAF audits. One host-scoped file replaces per-ingress modsecurity-snippet annotations, so the whole exception surface stays reviewable in one place. Two patterns cover most cases:

Allow additional HTTP methods (rule 911100):

Use SecAction with setvar before the Include — the initialization rule only sets the default if the variable is unset, so your value takes precedence:

  SecAction "id:900200,phase:1,pass,nolog,setvar:tx.allowed_methods=GET HEAD POST OPTIONS PUT PATCH DELETE"
  Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf

The SecRuleEngine and SecAudit* settings that shape the audit record — what makes the block JSON reach OpenObserve — are configured once, globally, rather than repeated per ingress.

Disable specific rule IDs (after Include):

  Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf
  SecRuleRemoveById 941100 942100

Conditional removal (e.g. per URI) must come before Include, and only ONE ctl: per SecRule:

Two distinct rule-disable mechanisms with very different timing:

  • SecRuleRemoveById is a parse-time directive — its position relative to Include doesn't matter, the targeted rule never loads.
  • ctl:ruleRemoveById is a runtime action inside a SecRule. ModSecurity evaluates rules in declaration order within a phase, so the action only has effect if it has already executed by the time the CRS rule fires. Declared after the CRS Include, it silently no-ops.

A second, less obvious trap: multiple ctl:ruleRemoveById actions chained on a single SecRule action list only execute the first one in nginx-ingress. Always write one SecRule per rule ID being removed. The grouped ctl:ruleRemoveByTag=<tag> form does work in one shot when a whole tag family is in scope (e.g. attack-xss covers 941100 / 941110 / 941130 / …).

WebDAV and file-sync paths — a CRS rule-exclusions plugin, not a blanket engine-off:

A recursive rclone or a desktop sync client against OwnCloud sends arbitrary user file names, binary file bodies, and XML PROPFIND payloads. CRS request inspection on those paths matches the content, not an attack: file names trip the LFI and restricted-extension rules, PROPFIND XML trips libinjection's XSS heuristic (xmlns="DAV:"), binary PUT bodies trip the content-type rules. Carving out individual rule IDs is a losing game — every new file type finds a new rule, and several of them can trip on a single request against the 10-point anomaly threshold.

The maintainable fix is the upstream Nextcloud CRS rule-exclusions plugin (OwnCloud shares the OCS API surface and most WebDAV semantics), mounted into the ingress controller via a HelmChartConfig extraVolume. Included before the CRS Include, its ctl:ruleRemoveTargetById actions excise the specific rules that trip on WebDAV/sharing traffic — no blanket engine-off, and every other path on the host stays fully WAF-protected. One narrow gap the plugin doesn't cover (OwnCloud-specific PROPFIND/no-Content-Type sync behaviour on /remote.php/dav/) gets its own scoped carveout on top:

  # Nextcloud rule-exclusions plugin — Included before the CRS Include.
  Include /etc/nginx/modsec-plugins/nextcloud-rule-exclusions-before.conf
  # Narrow OwnCloud-specific gap the plugin doesn't cover:
  SecRule REQUEST_URI "@beginsWith /remote.php/dav/" \
    "id:1011,phase:1,pass,nolog,ctl:ruleRemoveByTag=OWASP_CRS"
  Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf

Reserve a scoped ctl:ruleRemoveByTag (or ruleRemoveById) carveout for cases the plugin doesn't reach — for example CRS 934110 (SSRF) on the RFC 8252 loopback redirect_uri of a native OAuth2 client, or CRS 942290 (NoSQL injection) on the OData $expand query parameter of a Graph API. One SecRule per path:

  SecRule REQUEST_URI "@beginsWith /index.php/apps/oauth2/" \
    "id:1008,phase:1,pass,nolog,ctl:ruleRemoveByTag=OWASP_CRS"
  SecRule REQUEST_URI "@beginsWith /graph/" \
    "id:1009,phase:1,pass,nolog,ctl:ruleRemoveByTag=OWASP_CRS"

Symptom of the wrong order (a carveout declared after the Include): requests are still blocked with HTTP 403 even though the exception is supposedly in place — the CRS rule already fired in the same phase before the ctl ran. Symptom of chained ctls: the first carveout works, the rest silently no-op.

Sidecar config staleness after a snippet change

Snippet changes roll the ingress-nginx controller — Reloader watches the global ModSecurity ConfigMap and restarts the DaemonSet one node at a time across the four nodes, which takes several minutes rather than an in-place reload. A fresh kubectl exec curl from a debug pod will show the new rules working while the long-lived rclone consume-puller is still being blocked. Restart the sidecar pod when verifying a phase-1 rule change — the controller-side check is the authoritative one, not the sidecar's behaviour.

Engine state is a per-host decision in the global config, not a per-ingress snippet.

Host-scoped SecRule gates run before the CRS include and sort every request into one of three states:

  • Off — a small, named list of hosts where a WAF is wrong rather than merely untested: the OCI registries and object store, gitlab.mdapi.ch, the LLM and MCP endpoints, the Rancher and Harvester UIs.
  • Detection-only — apps that legitimately generate high-scoring payloads: Joplin sync blobs, the grafana-embed d-solo panels, and Pushgateway's Prometheus text. Every entry has to carry its reason in the test suite, which fails on a host added without one.
  • Enforcing — every other named host. A Host that matches no Ingress falls through to detection-only rather than being enforced against a decision nobody made.

Steady-State Hygiene: Weekly Worker Restart

Every nginx configuration reload leaves the old worker processes running until their connections drain, and each worker carries its own loaded copy of the ModSecurity engine and CRS — so on a controller that reloads often, stale workers accumulate and their memory is never returned. The permanent mitigation is a weekly rolling restart of the ingress-nginx pods: one pod at a time, so ingress capacity never drops, flushing the accumulated stale workers before they matter. This is scheduled hygiene, not incident response.

Trusted-source false-positive alerting

The cribl-am-relay modsec detector only emits a ModSecBlockLikelyFalsePositive alert when the blocked client IP lies inside an internal CIDR — 192.168/16, 10/8, 100.64/10 (Tailscale) — so external scanners hitting blocked endpoints are silently dropped at the alert plane. One address inside 192.168/16 is explicitly excluded: 192.168.164.2, the Jool NAT64 v4 egress for external IPv6 clients (see external traffic on bpi-r4). The companion 192.168.164.3 egress, which carries internal LAN IPv6 traffic, stays trusted, so internal-user false positives still surface.

Critical: No Single Quotes in Snippets

Silent cluster-wide failure

nginx-ingress wraps the modsecurity-snippet value in modsecurity_rules '...'. A single quote (') inside the snippet terminates the nginx config string prematurely.

This causes nginx to fail to reload — silently blocking all new ingress configurations and certificate updates cluster-wide. The error only appears in the nginx-ingress-controller pod logs (unexpected "t" in /tmp/nginx/nginx-cfg…), not in the ingress object events.

Always use double quotes inside snippet content. And no apostrophes in English contractions inside # comments — aren't, doesn't, isn't, won't, let's, can't are all traps. Replace with the non-contracted form (are not, does not, …) or rephrase.