Feat: Native support for certificate fetching and rotation via SDS server

[raviverma-ai]

Description:

In continuation with Extension server policy propagation to PostTranslateModify. Although we are planning to make extension server and related sds server code Open-Source for others to use as needed. Though might be difficult for some users as it requires multiple components.

@guydc, @arkodg Will it be good to have this support natively with Envoy Gateway itself i.e. fetch mTLS certs via any external CA of choice and auto-rotate leveraging envoy's SDS capabilities (perhaps utilising BackendTLSPolicy and/or EnvoyProxy or new Envoy Gateway CRD)?

If it really comes handy to many users, I can start implementing it.

[arkodg]

can you elaborate a little more on what the workflow would look like for the platform dev and app dev ?
cc @muwaqar

[raviverma-ai]

Hi @arkodg , after going through multiple design options, I finalised this option to implement in simple and cleaner way.

We should go with new EG CRD, let's say, BackendmTLSPolicy (open for any suggestions), and here is a sample CRD of how it will look like.

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendmTLSPolicy
metadata:
  name: example-mtls-policy
  namespace: default
spec:
  targetRefs:
    - kind: Gateway 
       name: eg
       namespace: default
  sds:
    caType: CertManager # we can support two options, i.e. Istio and CertManager currently
    caAddress: cert-manager-istio-csr.cert-manager.svc:443 # OR istiod.istio-system.svc.cluster.local:15012
    secretName: default
  backendRefs: # Optional
    - kind: HTTPRoute # (two options, either HTTPRoute or Service) 
       name: test-httproute
       namespace: default

CRD explaination:

1. This CRD specifies the target gateways to attache to.
2. sds configuration species where to connect to get mtls certificates.
3. backendRefs is optional with following behaviour.
   a. No backendRefs specified - Generate mtls configurations for all connected HTTPRoutes.
   b. Kind specified as HTTPRoute - Generate mtls configurations for all paths specified for that HTTPRoute.
   c. kind is specified as Service - Generate mtls configuration for only those paths matching the destination service. (optional or open for any suggestions)

Flow:

1. Envoy Gateway controller would read this crd and generate sds related configurations for HTTPRoutes.
2. Pushes the configuration to respective gateway pod(s).
3. As soon as gateway pod(s) get the configuration, connects to SDS server to request and fetch certificate.
4. SDS server monitors and rotates certificates.

Architecture:

1. We have couple of options to run
   a. Run as separate pod. (More complex as another certificate needs to be maintained for SDS server communication)
   b. Run as separate container on Envoy Gateway pod. (Preferable option)
   c. Run as another process in same container as Envoy Gateway. (Not recommended for at least for initial releases).
2. SDS server opens uds socket (though it can support other connections), this is the easiest to make it secure with less overhead.
3. SDS server would implement Certificate CSRs for CertManager and Istio.
4. Envoy Gateway would request certificate from SDS server.
5. SDS server would create and send CSR request to configured CA and gets the certifcate.
6. SDS server provides the certificate to Envoy Gateway pod (and rotates).

[raviverma-ai]

cc @zirain, @muwaqar, @guydc

[guydc]

So far, the project avoided certificate LCM as a feature. I think that taking ownership of an SDS server that's integrated with a CA and auto-rotating certs is a major step away from the project's current position.

Can you maybe explain why this scenario cannot already be covered by native features that cert-manager offers?

• EG can read cert-manager-generated certs, cert-manager rotates certs, etc.
• For the istio case, I imagine that it's possible to build a custom cert-manager issuer that sends the cert-manager-generated CSRs to istio for signing.

The approach above would be useful also for users that don't have SDS and would probably provide a similar UX to the one that they have for server certs today.

[karaimin]

I have a similar use case with SPIRE (SPIFFE) integration on the edge gateway. With pure Envoy proxy + SDS (docs
), Envoy can fetch SPIFFE identities, establish mTLS for outbound calls, and validate peers against the root CA from SDS.

Would it make sense for Envoy Gateway to support this flow natively and configure the envoy proxy accordingly?

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[yuzisun]

+1 this becomes increasingly important for supporting agent identity which makes calls to ai gateway.

[zirain]

raise a design document for this https://docs.google.com/document/d/1BzfO7dZAxNy3Wkt43v9HbUJ8ZDypiU5nlhrR1m-4nyA/edit?tab=t.0#heading=h.i56e32ouyufv

cc @envoyproxy/gateway-maintainers PTAL

[zirain]

we discusssed this in the meeting today, we're agree to start with the SPIRE agent user case:

1. Prefer to start without a new API
2. The dataplane should talked to the (external) SDS provider/server directly.
3. The EG controller needs to handle for the case of kubernetest secrets/configmaps.

[baurmatt]

Not 100% sure if we match the usecase described in here but I wanted to bring it on since I saw mentions of Cert-Manager in this issue and the design document.

On the backend side, we implement secretless certificates with the help of Cert-Managers CSI driver. On Envoy, we don't seem to have such an options to have automatically rotated backend mTLS certificates without a intermediate Secret resources. It would be great if this usecase could be supported as well!