feat: cross ns policy attachment #8676
Conversation
Signed-off-by: Guy Daich <guy.daich@sap.com>
Signed-off-by: Guy Daich <guy.daich@sap.com>
Signed-off-by: Guy Daich <guy.daich@sap.com>
✅ Deploy Preview for cerulean-figolla-1f9435 ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #8676 +/- ##
==========================================
+ Coverage 74.59% 74.65% +0.05%
==========================================
Files 250 250
Lines 39751 40164 +413
==========================================
+ Hits 29654 29983 +329
- Misses 8048 8118 +70
- Partials 2049 2063 +14 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com>
Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com>
| // Selector selects namespaces when From is set to Selector. | ||
| // | ||
| // +optional | ||
| Selector *metav1.LabelSelector `json:"selector,omitempty"` |
There was a problem hiding this comment.
I’m a bit hesitant to add Selector support, as we don’t have a similar flexible ReferenceGrant API for it.
Or EG probably shouldn't require ReferenceGrant for cross-ns policy attachment at all. In the Gateway API semantics, ReferenceGrant is used for cross-namespace references to foreign objects, not for attachment relationships.
There is already precedent for this distinction in the Gateway API:
- cross-namespace Route-to-Gateway attachment is governed by
allowedRoutes, noReferenceGrantrequired. - cross-namespace ListenerSet-to-Gateway attachment is governed by
allowedListeners, noReferenceGrantrequired.
There was a problem hiding this comment.
My concern with allowing this without ReferenceGrant is that app teams could affect cluster-wide resources without the infra team’s knowledge or consent.
In Gateway API model, the affected resource (e.g. Gateway managed by the infra team) has the authority to control who can attach to it. In other words, consent comes from the target side.
By contrast, policy resources can be created by app teams, and a policy in one team’s namespace could target resources owned by other teams, without giving those teams any way to opt in or opt out.
That said, from the infra team’s perspective, this feature is very useful because a single policy can apply across resources in all namespaces.
So I think it's important to be clear about who should have that authority.
Would it make sense to add a toggle in EnvoyGateway to enable or disable this feature, or to allow it only for specific namespaces?
There was a problem hiding this comment.
Hi @kkk777-7 That's a good point! I agree the key question is who should have the authority to enable cross-namespace attachment.
One option is to require ReferenceGrant, but this might not be the right semantic fit, since Gateway API generally uses ReferenceGrant for cross-namespace object references rather than attachment-style relationships. A more Gateway API-native model would be target-side consent, for example something like an AllowedPolicies field on Gateways or Routes, though that would likely require upstream API work.
We've got a few options here:
- Use
ReferenceGrantto explicitly authorize cross-namespace policy attachment. - Disable cross-namespace policy attachment by default, and let operators enable it only for specific namespaces.
type CrossNamespacePolicyAttachment struct {
Enabled bool `json:"enabled,omitempty"`
AllowedPolicyNamespaces []string `json:"allowedPolicyNamespaces,omitempty"`
}- Pursue a more target-side consent model, though that likely needs upstream API support. This is not feasible if we want to land it in EG in the near future as it requires some time to discuss this upstream.
Option 1 provides more granular control, while option 2 is simpler and would allow a smoother transition to a more Gateway API-native target-side consent model, such as an AllowedPolicies attachment mechanism in the future.
If we go with option 1, we should also start with a simpler namespace allowlist first rather than a fully flexible Selector, since the latter makes the authorization and UX harder to reason about.
type TargetSelectorNamespaces struct {
// +kubebuilder:validation:Enum=Same;All;Names
From TargetNamespaceFrom `json:"from"`
MatchNames []string `json:"matchNames,omitempty"`
}In the meantime, we could pursue a more target-side consent model in the upstream Gateway API.
cc @envoyproxy/gateway-maintainers could you please weigh in on the options?
There was a problem hiding this comment.
thanks @zhaohuabing :)
I agree that option3 needs some times to discuss this upstream, so prefer option2.
There was a problem hiding this comment.
+1 for option1, since there isnt a direct parent child relationship for policy and target, similar to gateway-route or gateway-listenerSet
take a case where app team in namespace A tries to influence routes in namespace B using a policy A, we dont want this to be the default, and want team B to opt in
I do think we need to think of a way to opt out of explicit ReferenceGrants, to reduce the number of resources needed to author intent, that can be a future follow up
There was a problem hiding this comment.
Raised kubernetes-sigs/gateway-api#4799 in Gateway API to explore an “allow-from-target” approach, similar to Gateway.allowedListeners.
Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com>
479741a to
7d277a7
Compare
Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com>
Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com>
Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e507c223d1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
/retest |
Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com>
|
/retest |
| kind: Gateway | ||
| matchLabels: | ||
| policy: selected | ||
| namespaces: |
There was a problem hiding this comment.
since this is the selector case, should we skip targets with no refgrants or should we flag it as partially accepted in the status ?
There was a problem hiding this comment.
If two routes under the same Gateway are selected, and one is permitted by a ReferenceGrant while another is not, the status for that Gateway is set to Accepted=False in this PR:
status:
ancestors:
- ancestorRef:
group: gateway.networking.k8s.io
kind: Gateway
name: mixed-gateway
namespace: gateway-ns
sectionName: http
conditions:
- lastTransitionTime: null
message: Target HTTPRoute policy-target-b/denied-route is not permitted by
any ReferenceGrant.
reason: RefNotPermitted
status: "False"
type: AcceptedAnd the targets without RefrenceGrant are skipped during the policy stranslation.
We probably shouldn't silently skip targets without ReferenceGrants though, since users would otherwise have no way to understand why the policy is not taking effect on some routes.
A “partially accepted” status would be better, but right now the Accepted condition only supports True or False, and adding an addtional ParitallyAccepted condition for this feels like too much.
There was a problem hiding this comment.
How about:
-
If an ancestor has at least one effective target:
Accepted=True, -
if some selector-matched targets under that same ancestor were skipped due to missing
ReferenceGrant: addWarning=True, reasonRefNotPermitted.
Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com>
Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com>
Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 33fd9ab20c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if len(denied) > 0 { | ||
| policy, found := handledPolicies[policyName] | ||
| if !found { | ||
| policy = policyCopies[i] | ||
| handledPolicies[policyName] = policy | ||
| res = append(res, policy) | ||
| } | ||
| setPolicyTargetRefNotPermittedStatus(&policy.Status, denied, t.GatewayControllerName, policy.Generation) |
There was a problem hiding this comment.
Defer RefNotPermitted status until all targets are processed
Setting RefNotPermitted immediately after route selector evaluation can leave an ancestor permanently Accepted=False even when a later gateway target for the same ancestor is valid. In this flow, route denials are recorded first, then gateway processing runs afterward; because SetAcceptedForPolicyAncestor only sets Accepted when it is unset, the later successful attachment cannot flip the ancestor back to accepted. This produces incorrect status for mixed-grant policies (e.g., route denied but gateway allowed under the same parent).
Useful? React with 👍 / 👎.
| "Target %s %s/%s is not permitted by any ReferenceGrant.", | ||
| deniedMatch.Target.GetObjectKind().GroupVersionKind().Kind, | ||
| deniedMatch.Target.GetNamespace(), | ||
| deniedMatch.Target.GetName(), |
There was a problem hiding this comment.
Avoid exposing cross-namespace target identities in denial status
This message includes the concrete kind/namespace/name of denied cross-namespace targets discovered via selectors. That leaks object existence/details across namespace boundaries to a policy author without a matching ReferenceGrant, which is exactly the case being denied. For multi-tenant clusters, this becomes an information disclosure vector whenever broad selectors (for example namespaces.from: All) are used.
Useful? React with 👍 / 👎.
arkodg
left a comment
There was a problem hiding this comment.
LGTM thanks
Approving so this makes it into rc and the the current limitations can hopefully be addressed before the release
guydc
left a comment
There was a problem hiding this comment.
LGTM, thanks, this is super useful.
Some docs clarifying RefGrant requirements would be great in a follow-up PR.
|
/retest |
* api: cross-namespace policy target selectors Signed-off-by: Guy Daich <guy.daich@sap.com> * use gw-api style for namespace selector Signed-off-by: Guy Daich <guy.daich@sap.com> * fix json tag Signed-off-by: Guy Daich <guy.daich@sap.com> * update API Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * update API Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * implmentation Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * add e2e tests for cross ns policy attachment Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * update test Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * fix lint Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * update test Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * add referencegrants to the provider Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * fix test Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * fix test Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * simplify code Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * address comments Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * simplify code Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * simplify code Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * rename Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * fix race Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * remove unnecessary change Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * simplify code Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * rename Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * refactor Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * address comments Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * address comments Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * add referencegrants for extension server policies Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * Revert "add referencegrants for extension server policies" This reverts commit a0fbe68. Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * remove cross-ns policy attachment for extension server Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * fix gen Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * fix lint Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * fix test Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * address comments Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * remove unnecessary change Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * fix cel validation Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * add warning for partially accepted targets Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> --------- Signed-off-by: Guy Daich <guy.daich@sap.com> Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> Co-authored-by: Guy Daich <guy.daich@sap.com>
* api: cross-namespace policy target selectors Signed-off-by: Guy Daich <guy.daich@sap.com> * use gw-api style for namespace selector Signed-off-by: Guy Daich <guy.daich@sap.com> * fix json tag Signed-off-by: Guy Daich <guy.daich@sap.com> * update API Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * update API Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * implmentation Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * add e2e tests for cross ns policy attachment Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * update test Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * fix lint Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * update test Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * add referencegrants to the provider Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * fix test Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * fix test Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * simplify code Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * address comments Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * simplify code Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * simplify code Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * rename Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * fix race Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * remove unnecessary change Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * simplify code Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * rename Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * refactor Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * address comments Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * address comments Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * add referencegrants for extension server policies Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * Revert "add referencegrants for extension server policies" This reverts commit a0fbe68. Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * remove cross-ns policy attachment for extension server Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * fix gen Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * fix lint Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * fix test Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * address comments Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * remove unnecessary change Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * fix cel validation Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> * add warning for partially accepted targets Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> --------- Signed-off-by: Guy Daich <guy.daich@sap.com> Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> Co-authored-by: Guy Daich <guy.daich@sap.com>
Picking up #5145 and continuing the work from here. Thanks, @guydc, for the earlier work on this.
This PR adds cross-namespace policy attachment support for
ClientTrafficPolicy,BackendTrafficPolicy,EnvoyExtensionPolicy, andSecurityPolicy.Extension Server policies aren’t included yet since the issue is mostly around CTP, BTP, and EEP, and EG doesn’t have proper tests for Extension Server policies yet.
We can add support for Extension Server policies later in a follow-up PR if needed.
Implements: #5145