Notify owners of 5 pre-existing inverted-paranoia TrafficProtectionPolicies (remediate blocking<=detection)

[ecv]

Summary

Five pre-existing TrafficProtectionPolicy (TPP) objects on prod carry an inverted OWASP CRS paranoia configuration (blocking > detection). All were created before the CEL admission rule that now rejects this shape, so they were grandfathered in. This issue tracks notifying their owners so they self-remediate.

All five are on networking.datumapis.com, generation: 1, detection: 1 / blocking: 2 (inverted).

Why inverted paranoia is a hazard

CRS rule 901500 fails closed when the blocking paranoia level exceeds the detection paranoia level. In Enforce mode this makes the coraza WAF return HTTP 500 on 100% of traffic to every route the policy attaches to — a hard outage for the affected hostname, not a partial degradation. See #242 for the full failure signature.

Current neutralization status (verified on prod, 2026-07-13)

Good news: none of the five are currently serving 500s. NSO v0.23.12 (which includes #252) is live, and the coraza data-plane delivery path is currently inert for tenant namespaces. Two independent facts establish this:

1. fix: flag inverted TrafficProtectionPolicy paranoia levels via status condition #252 skip-attachment is working for the newest/reconciled policies: NSO renders no tpp-<name> EnvoyPatchPolicy for them at all.
2. For the remaining three, NSO did render a tpp-<name> EnvoyPatchPolicy that still embeds the inverted paranoia_level=2 / paranoia_level=1 coraza SecRules — but that patch is Accepted=False / reason=Disabled on every edge checked (ae-north-1, us-central-1, us-east-1), because EnvoyPatchPolicy is disabled in the EnvoyGateway configuration that watches the projected ns-<uid> tenant namespaces. There are zero EnvoyExtensionPolicy objects cluster-wide, so there is no alternate live WAF path. The broken patch never reaches a running Envoy.

Latent risk: those three still carry the broken config in a rendered EnvoyPatchPolicy. If EnvoyPatchPolicy is ever re-enabled for tenant namespaces — or if the coraza delivery path migrates off the disabled flag — before these TPPs are fixed, they would immediately begin 500ing. They should be remediated regardless. (Propagation-lag caveats: see #266.)

Note: the TPP .status is stripped on both the karmada aggregated read and the edge projections, so the Accepted=False/Invalid ancestor condition written by #252 could not be confirmed directly from those views; the assessment above rests on data-plane attachment state.

The five policies

Namespace / name	Project (ProjectControlPlane)	Owner	mode	det/blk	Neutralization status
ns-1b46bde2…/tunnel-72tt5	test-proj-gyrkna	personal-org-ba109032 (unknown)	Observe	1/2	Neutralized — no EPP rendered (skipped) and mode: Observe (would not block)
ns-29e6c0fd…/tunnel-2252m	personal-project-cecba11a	personal-org-cecba11a (unknown)	Enforce	1/2	Inert — inverted EPP rendered but Disabled; latent
ns-46a5d45f…/pcep-lol-hnqr6o	personal-project-4fc7cf82 (hosts pcep.lol)	personal-org-4fc7cf82 (unknown)	Enforce	1/2	Inert — inverted EPP rendered but Disabled; latent
ns-46f4e730…/rfxn-cdn-t0jxi3	personal-project-02f2388c	personal-org-02f2388c (unknown)	Enforce	1/2	Inert — inverted EPP rendered but Disabled; latent
ns-c5f1cf4e…/kev-httpbin	default-ec6p8s	Kevin Williams	Enforce	1/2	Neutralized — no EPP rendered (skipped); created after v0.23.12

Owner mapping method: ns-<uid> → ProjectControlPlane (via the meta.datumapis.com/upstream-cluster-name label on the projected object) → resourcemanager.miloapis.com/owner-name annotation. These are all personal orgs; the human behind each personal-org-<hash> is not resolvable from the infra/karmada clusters (no organizations/users API is served there). kev-httpbin is the personal project of Kevin Williams (same project as kev1n-proxy).

What owners must do

Edit the offending TPP so detection >= blocking. Concretely, under each spec.ruleSets[].owaspCoreRuleSet.paranoiaLevels, set detection to be at least blocking (e.g. detection: 2 with blocking: 2, or lower blocking to 1). Once corrected, the policy will pass validation and can safely attach and enforce.

References

• TrafficProtectionPolicy Enforce mode: 500 with truncated/empty body on all traffic #242 — inverted-paranoia Enforce mode returns 500 on all traffic (the hazard)
• fix: flag inverted TrafficProtectionPolicy paranoia levels via status condition #252 — flag inverted TrafficProtectionPolicy paranoia levels via status condition (the fix, live in v0.23.12)
• Edge serves stale TrafficProtectionPolicy config during upstream→edge propagation with no lag visibility #266 — edge serves stale TPP config during upstream→edge propagation (propagation-lag caveat)

[ecv]

Triage note: severity is latent, not active

To make the priority explicit for anyone scanning this: none of the five are 500'ing today — this is not an active outage. Three of them (tunnel-2252m, pcep-lol-hnqr6o, rfxn-cdn-t0jxi3) are inert only because EnvoyPatchPolicy is disabled env-wide for tenant namespaces; the inverted config is still rendered and would begin 500'ing the instant that path is re-enabled or migrated off the disabled flag.

So treat this as low urgency now, high blast-radius if the environment flag changes — remediate before any change to the EnvoyPatchPolicy enablement, not as a fire. The two skip-attachment cases (tunnel-72tt5, kev-httpbin) are safe regardless.

[ecv]

Correction: two of these are 500'ing live in prod — retracting "none are serving 500s"

The neutralization assessment above analyzed the wrong delivery path. It reasoned from the tpp-<name> EnvoyPatchPolicy (which is disabled env-wide) and concluded the policies were inert. But the live WAF path on the alice edges is the datum-downstream-gateway coraza filter (extension-server), not EnvoyPatchPolicy — that path is active and enforcing across all 16 edge clusters (~134k blocks/24h fleet-wide).

Verified against prod VictoriaMetrics (coraza_envoy_filter_request_total), two of the five are actively returning HTTP 500 on benign traffic right now (coraza_outcome=blocked, http_status_code=500, mode=Enforce — the Mechanism-A inverted-paranoia / CRS 901500 fail-closed signature):

TPP	24h 500s	last 30m	clusters
kev-httpbin	53,516	~2,150	br-east-1, cl-central-1, de-central-1, gb-south-1, in-west-1, jp-east-1, nl-west-1, sg-central-1, us-east-1, za-central-1 (anycast, ~180/POP/30m)
pcep-lol-hnqr6o	17,243	357	us-east-1 (anycast-nearest)

The fixed policy kev1n-proxy ({2,2}) shows zero 500s over the same window — cross-validating that the 500s are caused by the inverted config, not the WAF generally.

Corrected status of the five

TPP	mode	corrected status
kev-httpbin	Enforce	ACTIVE OUTAGE — 500'ing fleet-wide now
pcep-lol-hnqr6o	Enforce	ACTIVE OUTAGE — 500'ing on us-east-1 now
tunnel-2252m	Enforce	no 500s observed (idle route or unattached)
rfxn-cdn-t0jxi3	Enforce	no 500s observed (idle route or unattached)
tunnel-72tt5	Observe	not blocking (Observe mode)

Consequence

#252 is not neutralizing kev-httpbin on the real (downstream-gateway) data-plane path — its "skip attachment" is not preventing the coraza filter from enforcing the inverted config. That is a gap in the shipped fix, tracked back to #242. The EnvoyPatchPolicy-disabled state is irrelevant to enforcement on these edges.

Severity for the two above is active customer outage, not latent. Remediation (set detection >= blocking) is now urgent for kev-httpbin and pcep-lol-hnqr6o. kev-httpbin is Kevin Williams' project; pcep-lol-hnqr6o is an unresolvable personal org needing operator escalation.

Refs: #242, #252, #266.

[ecv]

@kevwilliams — kev-httpbin in your project (default-ec6p8s) is the live outage here: it's returning HTTP 500 on ~100% of traffic across ~10 alice POPs right now (53,516 500s in the last 24h), because its TrafficProtectionPolicy has inverted OWASP paranoia levels — paranoiaLevels: {detection: 1, blocking: 2}. CRS rule 901500 fails closed when blocking > detection, so every request 500s before any attack eval (same Mechanism A as the #242 reopen).

Fix: set detection >= blocking on that TPP — e.g. detection: 2 (to match blocking: 2), or drop blocking: 1. Once corrected it will enforce normally. Same fix you already applied to kev1n-proxy (now {2,2}, zero 500s). This is your quickest path to stop the bleeding on kev-httpbin while we sort out why #252's skip-attachment isn't neutralizing it on the edge coraza path.

[ecv]

owners apparently resolveable, try fable

[ecv]

Rescoped and closing — outage mechanism fixed in code (v0.24.2)

The premise of this issue was an urgent outage notification: two inverted-paranoia TPPs (kev-httpbin, pcep-lol-hnqr6o) were returning HTTP 500 on benign traffic via the live edge coraza path, and #252's skip-attachment did not neutralize them there (see the correction comment above).

That gap is now closed. #273 (fix: withhold inverted TPP from coraza edge path, in v0.24.2) gates the extension server's directive emission on the same inverted-paranoia predicate the controller uses: computeCorazaDirectives returns no directives when InvertedParanoiaLevels() != nil, so the mutation layer withholds the policy and the data plane never enforces it. Both delivery paths now neutralize an inverted policy — #252 on the EnvoyPatchPolicy path, #273 on the extension-server/coraza edge path. Benign traffic is unaffected instead of 500'ing.

What remains (low priority — owner cleanup, not an outage)

All five policies are still inverted on prod (blocking > detection, generation 1, unremediated as of 2026-07-15). They are now inert rather than harmful: no 500s, but also no WAF protection — the policy silently does nothing. The only remaining value in reaching owners is a cleanup nudge so their WAF actually enforces:

Set detection >= blocking under spec.ruleSets[].owaspCoreRuleSet.paranoiaLevels (e.g. detection: 2 with blocking: 2) so the policy passes validation and attaches.

This is low-priority housekeeping, not incident response. Closing as the outage-notification framing is obsolete; owner cleanup can be tracked as routine follow-up if desired.

Refs: #273, #252, #242.