Skip to content

ModSecurity WAF

ModSecurity runs inline on every ingress-nginx instance, enforcing the OWASP Core Rule Set (CRS) on all inbound HTTP/S requests. Blocking is on by default (SecRuleEngine On); per-ingress exceptions are applied surgically when needed.

How a Block is Diagnosed

Every WAF-enabled ingress writes the ModSecurity audit log as single-line JSON to stdout (SecAuditLogType Serial, SecAuditLogFormat JSON, SecAuditLog /dev/stdout, with SecAuditEngine RelevantOnly and SecAuditLogRelevantStatus "^(?:5|4(?!04))" so only 4xx/5xx generate a record). Fluent Bit ships those records via Fluentd to Cribl, where a filter route forwards anything matching \"transaction\":{ or ModSecurity: Access denied into OpenObserve's modsec stream (7-day retention).

flowchart LR
    client["Client request"] -->|"HTTP 403"| nginx["ingress-nginx\nModSecurity"]
    nginx -->|"audit JSON to stdout"| fb["fluent-bit + fluentd"]
    fb --> cribl["Cribl\n(filter + parse)"]
    cribl --> o2["OpenObserve\nmodsec stream (7d)"]
    o2 --> sql["SQL: matched_rules"]
    sql --> fix["Apply exception\nin ingress annotation"]

Rule 949110 is always the final aggregator. The root rule(s) live in the audit record's matched_rules array, which the Cribl pipeline lifts to a top-level column in OpenObserve.

# Find the root rule(s) for a recent block
O2_PASS=$(akeyless get-secret-value --name /mdapi/openobserve/o2/root-password | tr -d '\n"')
curl -sS -u "<your-o2-user>:$O2_PASS" -X POST -H "Content-Type: application/json" \
  -d "{\"query\":{\"sql\":\"SELECT _timestamp, hostname, uri, method, http_code, matched_rules, matched_messages, client_ip FROM modsec WHERE hostname = '<host>' AND modsec_kind = 'audit' ORDER BY _timestamp DESC LIMIT 10\",\"start_time\":$(( ($(date +%s) - 3600) * 1000000 )),\"end_time\":$(( $(date +%s) * 1000000 ))}}" \
  "https://logs.mdapi.ch/api/default/_search" | jq

Fallback for blocks older than the 7-day retention or when OpenObserve is unavailable: tail the live ingress-nginx error log directly with kubectl -n kube-system logs -l app.kubernetes.io/name=ingress-nginx --tail=-1 | grep ModSecurity (only ~1 hour of history due to kubelet rotation).

Common Rules

Rule Description Score
911100 HTTP method not allowed 5
941100 XSS via libinjection 5
942100 SQLi via libinjection 5
932100–932180 Unix shell / RCE detection 5
949110 Anomaly score exceeded (aggregator) —

Default inbound anomaly threshold: 5 (one critical rule triggers a block).

Applying Exceptions

Exceptions live in the ingress annotation nginx.ingress.kubernetes.io/modsecurity-snippet. Two patterns cover most cases:

Allow additional HTTP methods (rule 911100):

Use SecAction with setvar before the Include — the initialization rule only sets the default if the variable is unset, so your value takes precedence:

nginx.ingress.kubernetes.io/modsecurity-snippet: |
  SecRuleEngine On
  SecAuditEngine RelevantOnly
  SecAuditLogRelevantStatus "^(?:5|4(?!04))"
  SecAuditLogParts ABIJDEFHZ
  SecAuditLogType Serial
  SecAuditLogFormat JSON
  SecAuditLog /dev/stdout
  SecAction "id:900200,phase:1,pass,nolog,setvar:tx.allowed_methods=GET HEAD POST OPTIONS PUT PATCH DELETE"
  Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf

The six SecAudit* lines are shared by every WAF-enabled ingress in the fleet — they're what makes the audit JSON record reach OpenObserve. Keep them in every per-ingress snippet (the controller-level modsecurity-snippet ConfigMap key is overridden by per-ingress snippets, so the audit config has to be repeated per-ingress).

Disable specific rule IDs (after Include):

nginx.ingress.kubernetes.io/modsecurity-snippet: |
  SecRuleEngine On
  Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf
  SecRuleRemoveById 941100 942100

Conditional removal (e.g. per URI) must come before Include, and only ONE ctl: per SecRule:

Two distinct rule-disable mechanisms with very different timing:

  • SecRuleRemoveById is a parse-time directive — its position relative to Include doesn't matter, the targeted rule never loads.
  • ctl:ruleRemoveById is a runtime action inside a SecRule. ModSecurity evaluates rules in declaration order within a phase, so the action only has effect if it has already executed by the time the CRS rule fires. Declared after the CRS Include, it silently no-ops.

A second, less obvious trap: multiple ctl:ruleRemoveById actions chained on a single SecRule action list only execute the first one in nginx-ingress. Always write one SecRule per rule ID being removed. The grouped ctl:ruleRemoveByTag=<tag> form does work in one shot when a whole tag family is in scope (e.g. attack-xss covers 941100 / 941110 / 941130 / …).

WebDAV and file-sync paths — disable the engine, do not chase rule IDs:

A recursive rclone or a desktop sync client against OwnCloud sends arbitrary user file names, binary file bodies, and XML PROPFIND payloads. CRS request inspection on those paths matches the content, not an attack: file names trip the LFI and restricted-extension rules, PROPFIND XML trips libinjection's XSS heuristic (xmlns="DAV:"), binary PUT bodies trip the content-type rules. Carving out individual rule IDs is a losing game — every new file type finds a new rule, and the anomaly threshold is only 5, so any single hit blocks.

The maintainable fix is to disable the rule engine for the file-sync path prefixes only. The web UI, OAuth2 endpoints, and every other path on the host stay fully WAF-protected:

nginx.ingress.kubernetes.io/modsecurity-snippet: |
  SecRuleEngine On
  SecAction "id:900200,phase:1,pass,nolog,..."
  # MUST precede the Include — a phase:1 ctl runs in declaration order.
  SecRule REQUEST_URI "@rx ^/(public\.php/webdav|remote\.php/(dav|webdav))/" \
    "id:1001,phase:1,pass,nolog,ctl:ruleEngine=Off"
  Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf

ctl:ruleEngine=Off is the request-scoped equivalent of the SecRuleEngine Off directive — it switches the engine off for the matched request only, and obeys the same before-the-Include ordering rule as ctl:ruleRemoveById.

Reserve per-rule ctl:ruleRemoveById for the narrow cases where a single known rule fires on a path that otherwise deserves full inspection — for example CRS 934110 (SSRF) on the RFC 8252 loopback redirect_uri of a native OAuth2 client, or CRS 942290 (NoSQL injection) on the OData $expand query parameter of a Graph API. One SecRule per rule ID:

  SecRule REQUEST_URI "@beginsWith /index.php/apps/oauth2/" \
    "id:1008,phase:1,pass,nolog,ctl:ruleRemoveById=934110"
  SecRule REQUEST_URI "@beginsWith /graph/" \
    "id:1009,phase:1,pass,nolog,ctl:ruleRemoveById=942290"

Symptom of the wrong order (a carveout declared after the Include): requests are still blocked with HTTP 403 even though the exception is supposedly in place — the CRS rule already fired in the same phase before the ctl ran. Symptom of chained ctls: the first carveout works, the rest silently no-op.

Sidecar config staleness after a snippet change

Snippet changes deploy through the nginx-ingress controller in ~2 minutes after the Ingress update lands, but a sidecar that already has a keep-alive connection to ingress-nginx keeps hitting the old worker until that connection is recycled. A fresh kubectl exec curl from a debug pod will show the new rules working while the long-lived rclone consume-puller is still being blocked. Restart the sidecar pod when verifying a phase-1 rule change — the controller-side check is the authoritative one, not the sidecar's behaviour.

Detection-only mode (no blocking):

Used for apps that legitimately generate high-scoring payloads (e.g. Joplin — users write shell command examples in Markdown notes):

nginx.ingress.kubernetes.io/modsecurity-snippet: |
  SecRuleEngine DetectionOnly
  Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf

Trusted-source false-positive alerting

The cribl-am-relay modsec detector only emits a ModSecBlockLikelyFalsePositive alert when the blocked client IP lies inside an internal CIDR — 192.168/16, 10/8, 100.64/10 (Tailscale) — so external scanners hitting blocked endpoints are silently dropped at the alert plane. One address inside 192.168/16 is explicitly excluded: 192.168.164.2, the Jool NAT64 v4 egress for external IPv6 clients (see external traffic on bpi-r4). The companion 192.168.164.3 egress, which carries internal LAN IPv6 traffic, stays trusted, so internal-user false positives still surface.

Critical: No Single Quotes in Snippets

Silent cluster-wide failure

nginx-ingress wraps the modsecurity-snippet value in modsecurity_rules '...'. A single quote (') inside the snippet terminates the nginx config string prematurely.

This causes nginx to fail to reload — silently blocking all new ingress configurations and certificate updates cluster-wide. The error only appears in the nginx-ingress-controller pod logs (unexpected "t" in /tmp/nginx/nginx-cfg…), not in the ingress object events.

Always use double quotes inside snippet content. And no apostrophes in English contractions inside # comments — aren't, doesn't, isn't, won't, let's, can't are all traps. Replace with the non-contracted form (are not, does not, …) or rephrase.