feat(helm): add engine authentication and TLS support (FB-1943) (#23)

## Summary

Adds first-class chart support for:
- **Engine authentication** — bootstrap admin account + JWT signing
keys, with rotation support (first entry in `auth.signingKeys` is the
active signer; prepend to rotate).
- **TLS** on the gateway's client listener (`tls.gateway`) and the
engine's query listener (`tls.engine`), including the engine's own
self-health-check and the gateway's upstream TLS to engines.

Every secret artifact (admin password, signing keys, TLS certs) is
provisioned by exactly one of `existingSecret` (bring-your-own) or a
chart-rendered cert-manager `Certificate` — no Helm-generated
self-signed certs, no plaintext secret values anywhere.

The old `auth.{mode,local,oidc}` → `auth.json` scaffold is removed: it
was never enforced (`values.yaml` said so explicitly) and the engine
never read that file.

## Notable design points

- `auth.admin.password`, `tls.gateway`, `tls.engine`, and each
`auth.signingKeys[]` entry all share the same `existingSecret` /
`certManager` shape, resolved by a shared `fbinstance.secretSourceName`
helper that fails the render if a source sets both or neither.
- The engine's own internal health check dials its query listener over
HTTPS once `tls.engine.enabled`, and validates using the same file it
serves as `certificate_file` — this requires a full certificate chain,
not a leaf-only cert, so an init container concatenates `tls.crt` +
`ca.crt` from the resolved Secret.
- The gateway's Envoy `dynamic_forward_proxy` cluster gets `auto_sni` /
`auto_san_validation` on its upstream TLS context, since the Lua filter
rewrites `:authority` per request.
- `certificates.yaml` gates every rendered cert-manager `Certificate` on
its owning feature's `enabled` flag, so a stale `certManager` block on a
disabled feature never creates an unused cert-manager dependency.
- The `core-ui` dev sidecar's `FIREBOLT_CORE_URL` now follows
`tls.engine.enabled` (its nginx reverse proxy defaults
`proxy_ssl_verify` to `off`, so this needs no extra trust configuration
for the loopback connection).

## Caveat

The chart's pinned `appVersion` predates the `instance.auth` config
schema landing in the engine — an engine on that build crash-loops with
`Configuration entry does not exist` if `auth.enabled` or
`tls.engine.enabled` is set. This is documented as a prerequisite in
`docs/usage/authentication.mdx` and `docs/prerequisites.mdx`; the chart
code itself is verified correct against engine source and against a
`:dev` engine build. Merging as-is (docs-only) per discussion — the
feature activates automatically once a release image with the schema
ships.

One open judgment call left for reviewers: this PR does **not** add a
`BREAKING CHANGE:` footer even though it removes the old
(non-functional) `auth.*` values shape, since nothing could have relied
on it for real auth. Flag in review if a major version bump is wanted
instead of the default minor bump a `feat:` commit triggers.

Author: kop

.legitignore

@@ -0,0 +1,2 @@
+value_67A867BC754C44390E7B54969C75EEE8 # Example Kubernetes Secret name "firebolt-signing-key-2026-02" in helm/values.yaml and docs/usage/authentication.mdx — a resource reference, not a credential value.
+value_8FAFF5AF6AA62BB96EAF8C4A1C3ED99F # Example Kubernetes Secret name "firebolt-signing-key-2026-01" in docs/usage/authentication.mdx — a resource reference, not a credential value.

docs/docs.json

@@ -23,6 +23,8 @@
                 ]
               },
               "usage/external-postgres",
+              "usage/authentication",
+              "usage/tls",
               "usage/image-overrides"
             ]
           },

docs/known_pages.json

@@ -15,6 +15,8 @@
   "/troubleshooting/",
   "/usage",
   "/usage/",
+  "/usage/authentication",
+  "/usage/authentication/",
   "/usage/external-postgres",
   "/usage/external-postgres/",
   "/usage/image-overrides",
@@ -32,5 +34,7 @@
   "/usage/object-storage/s3-compatible",
   "/usage/object-storage/s3-compatible/",
   "/usage/single-engine",
-  "/usage/single-engine/"
+  "/usage/single-engine/",
+  "/usage/tls",
+  "/usage/tls/"
 ]

docs/prerequisites.mdx

@@ -114,3 +114,4 @@ AKS does not expose `LimitMEMLOCK` through its supported node configuration, so
 
 - **Prometheus Operator CRDs**. Required only if you set `podMonitor.engines.enabled` or `podMonitor.gateway.enabled`. The chart renders `monitoring.coreos.com/v1` `PodMonitor` resources gated on those values.
 - **A private image registry**. The chart pulls engine and metadata images from `ghcr.io/firebolt-db` by default. To mirror through your own registry, see [Image overrides](./usage/image-overrides).
+- **cert-manager**. Required only if you set a `certManager` block under `auth.signingKeys`, `tls.gateway`, or `tls.engine`. The chart renders `cert-manager.io/v1` `Certificate` resources gated on those blocks.

docs/security.mdx

@@ -59,6 +59,8 @@ Platform teams should apply `NetworkPolicy` objects covering at least the allowe
 
 Cross-engine traffic (replicas of *different* engines), gateway-to-metadata, and gateway-to-PostgreSQL are not required by any chart-emitted control flow and should be denied.
 
+`tls.gateway.enabled` and `tls.engine.enabled` change what's carried on the "External clients → Gateway" and "Gateway → Engine pods" flows above from plaintext HTTP to TLS; they don't change which flows are required. See [TLS](./usage/tls).
+
 ### Namespace-level resource ceilings
 
 A `ResourceQuota` capping the aggregate of `requests.cpu`, `requests.memory`, pod count, and PVC size across all engines in a namespace is a platform concern. The chart applies per-engine `resources.requests` and `resources.limits` from your values, but does not emit a `ResourceQuota` and does not validate the per-engine `resources` block against a Helm-side maximum. The per-namespace budget is a deployment-target decision (test cluster vs. multi-tenant production).
@@ -107,8 +109,16 @@ The chart never sees those cloud provider credentials. For an S3-compatible endp
 
 The chart does not currently expose `sslmode` or a CA-bundle mount for the Metadata Service's PostgreSQL connection. Encrypt the PostgreSQL connection at the network layer (VPC peering, sidecar TLS proxy, or service-mesh mTLS) until the chart grows native support.
 
+### Authentication and TLS
+
+`auth.enabled`, `auth.signingKeys`, and `tls.{gateway,engine}` are each provisioned by exactly one of an existing Secret you create yourself, or a chart-rendered [cert-manager](https://cert-manager.io) `Certificate`. The chart never generates or accepts a literal admin password. See [Authentication](./usage/authentication) and [TLS](./usage/tls) for the full setup, including signing-key rotation and the cert-manager option.
+
+When engine query-listener TLS is enabled (`tls.engine.enabled`), the gateway's upstream connection and its active engine health check both switch to TLS along with it; kubelet's own probes against the engine's dedicated health port continue over plain HTTP.
+
 ## See also
 
+- [Authentication](./usage/authentication): enabling engine authentication and JWT signing key rotation.
+- [TLS](./usage/tls): enabling TLS on the gateway and engine listeners.
 - [Monitoring](./monitoring): metrics surface and Prometheus Operator integration.
 - [Operator upgrade path](./operator-upgrade-path): when continuous admission, drift correction, and CR-status surfaces are required.
 - [`SECURITY.md`](https://github.com/firebolt-db/firebolt-instance-helm/blob/main/SECURITY.md): vulnerability reporting policy for this repository.

docs/usage/authentication.mdx

@@ -0,0 +1,108 @@
+---
+description: Enabling engine authentication, provisioning JWT signing keys, and rotating them.
+sidebarTitle: Authentication
+title: Authentication
+---
+
+This page shows how to enable authentication on the Firebolt Engine.
+
+`auth.enabled` turns on the engine's built-in authentication: a bootstrap administrator account and JWT signing keys for the engine's embedded authorization server. Everything beyond that — OIDC providers, `password_login`, JWT lifetime, just-in-time provisioning — is set through `customEngineConfig.instance.auth`, which the chart deep-merges alongside the admin account and signing keys it renders.
+
+The chart never accepts a literal password value for the admin account. Enabling auth always requires an existing Secret.
+
+### Values
+
+```yaml
+# my-values.yaml
+auth:
+  enabled: true
+  admin:
+    name: firebolt
+    password:
+      existingSecret:
+        secretRef: firebolt-admin-password # Secret with a `password` key
+
+  signingKeys:
+    - id: key-2026-01
+      existingSecret:
+        secretRef: firebolt-signing-key-2026-01 # Secret with a `tls.key` PEM private key
+```
+
+### Install
+
+Create the admin password Secret and a signing key Secret, then install the chart:
+
+```bash
+kubectl create secret generic firebolt-admin-password -n firebolt \
+  --from-literal=password='…'
+
+# tls.key holds a PEM RSA private key; the chart uses only the private key,
+# not a certificate, to sign JSON Web Tokens.
+openssl genrsa -out signing-key.pem 2048
+kubectl create secret generic firebolt-signing-key-2026-01 -n firebolt \
+  --from-file=tls.key=signing-key.pem
+
+helm install firebolt ./helm \
+  --namespace firebolt --create-namespace \
+  -f my-values.yaml
+```
+
+<Note>
+A multi-node engine needs an explicit signing key. Without one, each engine node generates its own development key on first startup, and tokens signed by one node fail validation on another. The chart's render fails immediately if `auth.enabled` is true and `signingKeys` is empty.
+</Note>
+
+Kubernetes health probes and the gateway's active engine health check keep working without a token — the engine exempts `/health/live` and `/health/ready` from authentication.
+
+### Signing key rotation
+
+`signingKeys` is an ordered list. The **first** entry is the active signer used for new tokens; every entry in the list remains valid for verifying tokens already issued under it. To rotate, prepend a new entry rather than replacing the list, and remove the old entry only once every token it signed has expired:
+
+```yaml
+auth:
+  signingKeys:
+    - id: key-2026-02 # now active
+      existingSecret:
+        secretRef: firebolt-signing-key-2026-02
+    - id: key-2026-01 # still verifies old tokens until they expire
+      existingSecret:
+        secretRef: firebolt-signing-key-2026-01
+```
+
+### cert-manager as a signing-key source
+
+As an alternative to an existing Secret, a signing key entry can request a [cert-manager](https://cert-manager.io) `Certificate` instead:
+
+```yaml
+auth:
+  signingKeys:
+    - id: key-2026-01
+      certManager:
+        algorithm: RSA
+        size: 2048
+        issuerRef:
+          name: internal-ca
+          kind: ClusterIssuer
+```
+
+The chart pins `privateKey.rotationPolicy: Never` on signing-key Certificates, so cert-manager's renewal timer never silently replaces an active signing key out from under already-issued tokens. Rotate by prepending a new `signingKeys` entry, the same as with an existing Secret.
+
+### OIDC and other auth settings
+
+OIDC providers, `password_login`, JWT lifetime, and just-in-time provisioning are not first-class values. Set them under `customEngineConfig.instance.auth`, which is deep-merged alongside the admin account and signing keys the chart renders:
+
+```yaml
+customEngineConfig:
+  instance:
+    auth:
+      password_login: admin_only
+      oidc:
+        providers:
+          - name: corporate_idp
+            discovery_url: https://idp.example.com/.well-known/openid-configuration
+            username_mapping: "{{ email }}"
+```
+
+## See also
+
+- [TLS](./tls): enabling TLS on the gateway's client listener and the engine's query listener.
+- [Security model](../security): pod-hardening baseline and secrets handling across the whole chart.

docs/usage/tls.mdx

@@ -0,0 +1,92 @@
+---
+description: Enabling TLS on the gateway's client listener and the engine's query listener.
+sidebarTitle: TLS
+title: TLS
+---
+
+This page shows how to enable TLS on the gateway and engine listeners.
+
+`tls.gateway` and `tls.engine` each enable TLS independently, provisioned the same way as `auth.signingKeys` (see [Authentication](./authentication)): an existing `kubernetes.io/tls` Secret, or a chart-rendered cert-manager `Certificate`.
+
+- `tls.gateway` terminates TLS on the gateway's client-facing listener (client → gateway).
+- `tls.engine` enables TLS on the engine's query listener and, correspondingly, the gateway's connection to engines — including the active engine health check, which runs over the same connection.
+
+They can be enabled independently or together.
+
+### Values
+
+```yaml
+# my-values.yaml
+tls:
+  gateway:
+    enabled: true
+    existingSecret:
+      secretRef: firebolt-gateway-tls # kubernetes.io/tls Secret: tls.crt, tls.key
+
+  engine:
+    enabled: true
+    existingSecret:
+      secretRef: firebolt-engine-tls # kubernetes.io/tls Secret: tls.crt, tls.key, ca.crt
+```
+
+<Note>
+`tls.engine`'s Secret requires all three keys — `tls.crt`, `tls.key`, and `ca.crt` — even for a self-signed certificate with no separate CA (set `ca.crt` to a copy of `tls.crt` in that case). The engine's own internal health check and the gateway's upstream connection both need the issuing CA to validate the certificate independently of `tls.crt`.
+
+The certificate's SANs must also cover every engine node's per-node hostname (`<release>-engine-<engine>-node-<i>-0.<release>-engine-<engine>-hl.<namespace>.svc.cluster.local`) and each engine's headless-service hostname (`<release>-engine-<engine>-hl.<namespace>.svc.cluster.local`) — both the engine's self-health-check and the gateway's `auto_san_validation` verify against these names. A missing SAN leaves the affected engine node permanently `NotReady` with no other symptom. `tls.engine.certManager` derives these automatically; a BYO certificate has to include them explicitly.
+</Note>
+
+### Install
+
+```bash
+kubectl create secret tls firebolt-gateway-tls -n firebolt \
+  --cert=gateway.crt --key=gateway.key
+
+# ca.crt is required in addition to the standard tls.crt / tls.key keys.
+kubectl create secret generic firebolt-engine-tls -n firebolt \
+  --from-file=tls.crt=engine.crt --from-file=tls.key=engine.key --from-file=ca.crt=ca.crt
+
+helm install firebolt ./helm \
+  --namespace firebolt --create-namespace \
+  -f my-values.yaml
+```
+
+### cert-manager as a certificate source
+
+```yaml
+tls:
+  engine:
+    enabled: true
+    certManager:
+      algorithm: RSA
+      size: 2048
+      issuerRef:
+        name: internal-ca
+        kind: ClusterIssuer
+
+  gateway:
+    enabled: true
+    certManager:
+      algorithm: RSA
+      size: 2048
+      dnsNames:
+        - firebolt.example.com # the chart can't infer the externally-visible hostname
+      issuerRef:
+        name: internal-ca
+        kind: ClusterIssuer
+```
+
+The engine's certificate's DNS names are derived automatically from every engine node's per-node hostname and each engine's internal service hostname — don't set `dnsNames` under `tls.engine.certManager`. The gateway's certificate has no such derivation; set `dnsNames` to whatever hostname clients use to reach the gateway.
+
+<Note>
+The issuer behind `tls.engine.certManager` must populate `ca.crt` in the Certificate's Secret, not just `tls.crt`/`tls.key`. CA and Vault issuers do this; a bare self-signed or ACME issuer may not. The chart's `tls-chain-setup` init container needs `ca.crt` to build the engine's own health-check trust bundle and fails the pod at startup if it's missing.
+</Note>
+
+### Caveats
+
+- The `core-ui` sidecar (`engineSpec.uiSidecar`) follows `tls.engine.enabled` automatically, proxying to the engine over HTTPS without verifying its certificate — the connection is over loopback within the pod, so this doesn't need a trusted CA.
+- cert-manager and its CRDs must already be installed in the cluster before `certManager` blocks are set on any value; the chart's `Certificate` resources otherwise fail to apply.
+
+## See also
+
+- [Authentication](./authentication): enabling engine authentication and JWT signing key rotation.
+- [Security model](../security): pod-hardening baseline and secrets handling across the whole chart.