[Architecture] Migrate ingress-nginx to supported alternative (EOL March 2026)

[TineoC]

Purpose: TLDR of why we need this?

'ingress-nginx' reaches End of Life (EOL) at the end of March 2026. Continued use after this date poses security risks and lack of support for newer Kubernetes versions.

Functionality: TLDR of what will this change?

Replace the current 'ingress-nginx' controller with a supported alternative (e.g., a Gateway API-based controller like Envoy Gateway, or Traefik). This will involve updating ingress annotations and class names across all cluster manifests.

Affects: Where will this change?

The following files specifically use the nginx ingress class or related annotations:

• choose-native-plants/release-values.yaml
• echo-http.yaml
• cert-manager.issuers.yaml
• balancer/kustomization.yaml (Patch)
• sealed-secrets/release-values.yaml
• metabase/release-values.yaml
• paws-data-pipeline/release-values.yaml
• prevention-point/release-values.yaml
• Also potentially app/deploy/manifests/balancer/base/ingress.yaml (referenced by balancer/ lens).

Pros/Cons

Pros: Security compliance, modern ingress features.
Cons: Potential downtime during migration, need to update multiple manifests.

Additional context

The following files specifically use the nginx ingress class or specific annotations (like nginx.ingress.kubernetes.io/proxy-body-size) that will need to be translated to the new ingress controller's format:

• choose-native-plants/release-values.yaml
• paws-data-pipeline/release-values.yaml
• metabase/release-values.yaml
• prevention-point/release-values.yaml
• sealed-secrets/release-values.yaml
• balancer/kustomization.yaml (Patch)
• echo-http.yaml
• cert-manager.issuers.yaml (ACME challenge solvers)

[TineoC]

@themightychris I think for the long term, Gateway API would be the best choice. But at the same time there's some challenges for convenience for the rest of projects specially as not all devs have knowledge of k8s. Let me know your thoughts.

[themightychris]

Since Gateway API seems to be where most of the community momentum is going, my instinct is to start with that and take it for a spin on the sandbox cluster in parallel with ingress-nginx. I don't know if dev knowledge will be that big a deal as most people are just going to copy a template or use an AI agent to build their helm chart. We can create a skill plugin that includes templates and I expect most people will just use that going forward

[TineoC]

Cool, I would create a branch with these changes. I'll go with Envoy Gateway as it's what I have more experience with and it's been more widely used in the industry. What version of cert-manager is being used at the cluster?

[themightychris]

@TineoC you can see what's deployed here: https://github.com/CodeForPhilly/cfp-sandbox-cluster/blob/deploys/k8s-manifests/cert-manager/Deployment/cert-manager.yaml#L49

We could upgrade that though if needed

[TineoC]

@themightychris, how is cert-manager deployed in this project? Is it using plain manifests or Helm/Kustomize/Flux?

I need to add a flag to cert-manager my PR would need to merge with deploys/k8s-manifest?