Skip to content

ModSecurity WAF

ModSecurity runs inline on every ingress-nginx instance, enforcing the OWASP Core Rule Set (CRS) on all inbound HTTP/S requests. Blocking is on by default (SecRuleEngine On); per-ingress exceptions are applied surgically when needed.

How a Block is Diagnosed

Every WAF-enabled ingress writes the ModSecurity audit log as single-line JSON to stdout (SecAuditLogType Serial, SecAuditLogFormat JSON, SecAuditLog /dev/stdout, with SecAuditEngine RelevantOnly and SecAuditLogRelevantStatus "^(?:5|4(?!04))" so only 4xx/5xx generate a record). Fluent Bit ships those records via Fluentd to Cribl, where a filter route forwards anything matching \"transaction\":{ or ModSecurity: Access denied into OpenObserve's modsec stream (7-day retention).

flowchart LR
    client["Client request"] -->|"HTTP 403"| nginx["ingress-nginx\nModSecurity"]
    nginx -->|"audit JSON to stdout"| fb["fluent-bit + fluentd"]
    fb --> cribl["Cribl\n(filter + parse)"]
    cribl --> o2["OpenObserve\nmodsec stream (7d)"]
    o2 --> sql["SQL: matched_rules"]
    sql --> fix["Apply exception\nin ingress annotation"]

Rule 949110 is always the final aggregator. The root rule(s) live in the audit record's matched_rules array, which the Cribl pipeline lifts to a top-level column in OpenObserve.

# Find the root rule(s) for a recent block
O2_PASS=$(akeyless get-secret-value --name /mdapi/openobserve/o2/root-password | tr -d '\n"')
curl -sS -u "<your-o2-user>:$O2_PASS" -X POST -H "Content-Type: application/json" \
  -d "{\"query\":{\"sql\":\"SELECT _timestamp, hostname, uri, method, http_code, matched_rules, matched_messages, client_ip FROM modsec WHERE hostname = '<host>' AND modsec_kind = 'audit' ORDER BY _timestamp DESC LIMIT 10\",\"start_time\":$(( ($(date +%s) - 3600) * 1000000 )),\"end_time\":$(( $(date +%s) * 1000000 ))}}" \
  "https://logs.mdapi.ch/api/default/_search" | jq

Fallback for blocks older than the 7-day retention or when OpenObserve is unavailable: tail the live ingress-nginx error log directly with kubectl -n kube-system logs -l app.kubernetes.io/name=rke2-ingress-nginx --tail=-1 | grep ModSecurity (only ~1 hour of history due to kubelet rotation).

Common Rules

Rule Description Score
911100 HTTP method not allowed 5
941100 XSS via libinjection 5
942100 SQLi via libinjection 5
932100–932180 Unix shell / RCE detection 5
949110 Anomaly score exceeded (aggregator) —

Default inbound anomaly threshold: 5 (one critical rule triggers a block).

Applying Exceptions

Exceptions live in the ingress annotation nginx.ingress.kubernetes.io/modsecurity-snippet. The active inventory is deliberately bounded — roughly 23 scoped rule exceptions across the whole fleet, each tied to a specific ingress and path, reviewed as a set during WAF audits. Two patterns cover most cases:

Allow additional HTTP methods (rule 911100):

Use SecAction with setvar before the Include — the initialization rule only sets the default if the variable is unset, so your value takes precedence:

nginx.ingress.kubernetes.io/modsecurity-snippet: |
  SecRuleEngine On
  SecAuditEngine RelevantOnly
  SecAuditLogRelevantStatus "^(?:5|4(?!04))"
  SecAuditLogParts ABIJDEFHZ
  SecAuditLogType Serial
  SecAuditLogFormat JSON
  SecAuditLog /dev/stdout
  SecAction "id:900200,phase:1,pass,nolog,setvar:tx.allowed_methods=GET HEAD POST OPTIONS PUT PATCH DELETE"
  Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf

The six SecAudit* lines are shared by every WAF-enabled ingress in the fleet — they're what makes the audit JSON record reach OpenObserve. Keep them in every per-ingress snippet (the controller-level modsecurity-snippet ConfigMap key is overridden by per-ingress snippets, so the audit config has to be repeated per-ingress).

Disable specific rule IDs (after Include):

nginx.ingress.kubernetes.io/modsecurity-snippet: |
  SecRuleEngine On
  Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf
  SecRuleRemoveById 941100 942100

Conditional removal (e.g. per URI) must come before Include, and only ONE ctl: per SecRule:

Two distinct rule-disable mechanisms with very different timing:

  • SecRuleRemoveById is a parse-time directive — its position relative to Include doesn't matter, the targeted rule never loads.
  • ctl:ruleRemoveById is a runtime action inside a SecRule. ModSecurity evaluates rules in declaration order within a phase, so the action only has effect if it has already executed by the time the CRS rule fires. Declared after the CRS Include, it silently no-ops.

A second, less obvious trap: multiple ctl:ruleRemoveById actions chained on a single SecRule action list only execute the first one in nginx-ingress. Always write one SecRule per rule ID being removed. The grouped ctl:ruleRemoveByTag=<tag> form does work in one shot when a whole tag family is in scope (e.g. attack-xss covers 941100 / 941110 / 941130 / …).

WebDAV and file-sync paths — a CRS rule-exclusions plugin, not a blanket engine-off:

A recursive rclone or a desktop sync client against OwnCloud sends arbitrary user file names, binary file bodies, and XML PROPFIND payloads. CRS request inspection on those paths matches the content, not an attack: file names trip the LFI and restricted-extension rules, PROPFIND XML trips libinjection's XSS heuristic (xmlns="DAV:"), binary PUT bodies trip the content-type rules. Carving out individual rule IDs is a losing game — every new file type finds a new rule, and the anomaly threshold is only 5, so any single hit blocks.

The maintainable fix is the upstream Nextcloud CRS rule-exclusions plugin (OwnCloud shares the OCS API surface and most WebDAV semantics), mounted into the ingress controller via a HelmChartConfig extraVolume. Included before the CRS Include, its ctl:ruleRemoveTargetById actions excise the specific rules that trip on WebDAV/sharing traffic — no blanket engine-off, and every other path on the host stays fully WAF-protected. One narrow gap the plugin doesn't cover (OwnCloud-specific PROPFIND/no-Content-Type sync behaviour on /remote.php/dav/) gets its own scoped carveout on top:

nginx.ingress.kubernetes.io/modsecurity-snippet: |
  SecRuleEngine On
  # Nextcloud rule-exclusions plugin — Included before the CRS Include.
  Include /etc/nginx/nextcloud-rule-exclusions-before.conf
  # Narrow OwnCloud-specific gap the plugin doesn't cover:
  SecRule REQUEST_URI "@beginsWith /remote.php/dav/" \
    "id:1011,phase:1,pass,nolog,ctl:ruleRemoveByTag=OWASP_CRS"
  Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf

Reserve a scoped ctl:ruleRemoveByTag (or ruleRemoveById) carveout for cases the plugin doesn't reach — for example CRS 934110 (SSRF) on the RFC 8252 loopback redirect_uri of a native OAuth2 client, or CRS 942290 (NoSQL injection) on the OData $expand query parameter of a Graph API. One SecRule per path:

  SecRule REQUEST_URI "@beginsWith /index.php/apps/oauth2/" \
    "id:1008,phase:1,pass,nolog,ctl:ruleRemoveByTag=OWASP_CRS"
  SecRule REQUEST_URI "@beginsWith /graph/" \
    "id:1009,phase:1,pass,nolog,ctl:ruleRemoveByTag=OWASP_CRS"

Symptom of the wrong order (a carveout declared after the Include): requests are still blocked with HTTP 403 even though the exception is supposedly in place — the CRS rule already fired in the same phase before the ctl ran. Symptom of chained ctls: the first carveout works, the rest silently no-op.

Sidecar config staleness after a snippet change

Snippet changes deploy through the nginx-ingress controller in ~2 minutes after the Ingress update lands, but a sidecar that already has a keep-alive connection to ingress-nginx keeps hitting the old worker until that connection is recycled. A fresh kubectl exec curl from a debug pod will show the new rules working while the long-lived rclone consume-puller is still being blocked. Restart the sidecar pod when verifying a phase-1 rule change — the controller-side check is the authoritative one, not the sidecar's behaviour.

Detection-only mode (no blocking):

Used for apps that legitimately generate high-scoring payloads (e.g. Joplin — users write shell command examples in Markdown notes):

nginx.ingress.kubernetes.io/modsecurity-snippet: |
  SecRuleEngine DetectionOnly
  Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf

Steady-State Hygiene: Weekly Worker Restart

Every nginx configuration reload leaves the old worker processes running until their connections drain, and each worker carries its own loaded copy of the ModSecurity engine and CRS — so on a controller that reloads often, stale workers accumulate and their memory is never returned. The permanent mitigation is a weekly rolling restart of the ingress-nginx pods: one pod at a time, so ingress capacity never drops, flushing the accumulated stale workers before they matter. This is scheduled hygiene, not incident response.

Trusted-source false-positive alerting

The cribl-am-relay modsec detector only emits a ModSecBlockLikelyFalsePositive alert when the blocked client IP lies inside an internal CIDR — 192.168/16, 10/8, 100.64/10 (Tailscale) — so external scanners hitting blocked endpoints are silently dropped at the alert plane. One address inside 192.168/16 is explicitly excluded: 192.168.164.2, the Jool NAT64 v4 egress for external IPv6 clients (see external traffic on bpi-r4). The companion 192.168.164.3 egress, which carries internal LAN IPv6 traffic, stays trusted, so internal-user false positives still surface.

Critical: No Single Quotes in Snippets

Silent cluster-wide failure

nginx-ingress wraps the modsecurity-snippet value in modsecurity_rules '...'. A single quote (') inside the snippet terminates the nginx config string prematurely.

This causes nginx to fail to reload — silently blocking all new ingress configurations and certificate updates cluster-wide. The error only appears in the nginx-ingress-controller pod logs (unexpected "t" in /tmp/nginx/nginx-cfg…), not in the ingress object events.

Always use double quotes inside snippet content. And no apostrophes in English contractions inside # comments — aren't, doesn't, isn't, won't, let's, can't are all traps. Replace with the non-contracted form (are not, does not, …) or rephrase.