|
| 1 | +# BoilStream Helm chart |
| 2 | + |
| 3 | +Deploys a BoilStream cluster on Kubernetes — StatefulSet of N pods sharing |
| 4 | +state via S3 (cluster coordination + per-user catalog backups), with optional |
| 5 | +Envoy Gateway SNI routing and pod-to-pod mTLS. |
| 6 | + |
| 7 | +## Prerequisites |
| 8 | + |
| 9 | +The chart **does not** install these — operators must provide them: |
| 10 | + |
| 11 | +| Prereq | Why | |
| 12 | +|---|---| |
| 13 | +| **cert-manager ≥ 1.13** | Issues the public wildcard cert and (optionally) the internal cluster-mTLS cert. Skip if you supply both via `tls.existingSecret` and `clusterTls.existingSecret`. | |
| 14 | +| **A `ClusterIssuer`** named in `tls.issuer.name` (default `boilstream-ca-issuer`) | cert-manager needs an issuer to mint certs. Self-signed CA, Let's Encrypt + DNS-01, or ACM Private CA all work. | |
| 15 | +| **Envoy Gateway ≥ 1.2** with a `GatewayClass` named in `gateway.className` (default `eg`) | The chart-managed `Gateway` + `TLSRoute` resources reference this class. Skip by setting `gateway.enabled: false` (clients then must reach the Pods/headless Service directly). | |
| 16 | +| **An S3-compatible bucket** at `s3.endpoint` (real S3, MinIO, RustFS, R2…) | Stores leader.json, broker registry, per-user catalog backups. | |
| 17 | + |
| 18 | +Quick local install of the prereqs (tested on OrbStack k8s): |
| 19 | + |
| 20 | +```bash |
| 21 | +helm install eg oci://docker.io/envoyproxy/gateway-helm --version v1.2.1 \ |
| 22 | + -n envoy-gateway-system --create-namespace |
| 23 | +helm install cert-manager jetstack/cert-manager --version v1.16.2 \ |
| 24 | + -n cert-manager --create-namespace --set crds.enabled=true |
| 25 | +kubectl apply -f - <<'YAML' |
| 26 | +apiVersion: cert-manager.io/v1 |
| 27 | +kind: ClusterIssuer |
| 28 | +metadata: { name: selfsigned-root } |
| 29 | +spec: { selfSigned: {} } |
| 30 | +--- |
| 31 | +apiVersion: cert-manager.io/v1 |
| 32 | +kind: Certificate |
| 33 | +metadata: { name: boilstream-ca, namespace: cert-manager } |
| 34 | +spec: |
| 35 | + isCA: true |
| 36 | + commonName: BoilStream Local CA |
| 37 | + secretName: boilstream-ca-tls |
| 38 | + privateKey: { algorithm: ECDSA, size: 256 } |
| 39 | + issuerRef: { name: selfsigned-root, kind: ClusterIssuer } |
| 40 | +--- |
| 41 | +apiVersion: cert-manager.io/v1 |
| 42 | +kind: ClusterIssuer |
| 43 | +metadata: { name: boilstream-ca-issuer } |
| 44 | +spec: { ca: { secretName: boilstream-ca-tls } } |
| 45 | +--- |
| 46 | +apiVersion: gateway.networking.k8s.io/v1 |
| 47 | +kind: GatewayClass |
| 48 | +metadata: { name: eg } |
| 49 | +spec: { controllerName: gateway.envoyproxy.io/gatewayclass-controller } |
| 50 | +YAML |
| 51 | +``` |
| 52 | + |
| 53 | +## Install |
| 54 | + |
| 55 | +**Local dev (OrbStack + RustFS, defaults):** |
| 56 | + |
| 57 | +```bash |
| 58 | +helm install boilstream ./charts/boilstream |
| 59 | +``` |
| 60 | + |
| 61 | +**EKS (AWS S3 + ACM/Let's Encrypt + Pod Identity):** |
| 62 | + |
| 63 | +```bash |
| 64 | +helm install boilstream ./charts/boilstream \ |
| 65 | + -f ./charts/boilstream/values-eks-example.yaml \ |
| 66 | + --set image.repository=<account>.dkr.ecr.<region>.amazonaws.com/boilstream \ |
| 67 | + --set image.tag=0.10.0 \ |
| 68 | + --set superadmin.existingSecret=boilstream-superadmin \ |
| 69 | + --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::... |
| 70 | +``` |
| 71 | + |
| 72 | +## Key values |
| 73 | + |
| 74 | +| Group | Key | Default | Notes | |
| 75 | +|---|---|---|---| |
| 76 | +| Replicas | `replicas` | `3` | Min 1, recommended ≥3 for quorum-style availability. | |
| 77 | +| Image | `image.repository`, `image.tag`, `image.pullPolicy`, `image.pullSecrets` | `boilstream:local-1.5.2`, `Never`, `[]` | Set `pullSecrets: [{name: regcred}]` for private registries. | |
| 78 | +| Naming | `namespace`, `namespaceCreate`, `nameOverride`, `fullnameOverride` | `boilstream`, `true` | Set `namespaceCreate: false` if using `--create-namespace` or pre-managed namespaces. | |
| 79 | +| Public TLS | `tls.existingSecret`, `tls.issuer.{name,kind}` | (chart-managed) | Provide `existingSecret` to use ACM-imported or pre-issued certs. | |
| 80 | +| Cluster mTLS | `clusterTls.enabled`, `clusterTls.issuer.*` | `false` | Pod-to-pod gRPC on `:8444`. Separate trust root from public edge. | |
| 81 | +| Gateway | `gateway.enabled`, `gateway.className`, `gateway.serviceAnnotations` | `true`, `eg`, `{}` | Annotations propagate to the LB Service Envoy Gateway provisions. | |
| 82 | +| ServiceAccount | `serviceAccount.{create,name,annotations}` | `true`, `boilstream`, `{}` | Set `eks.amazonaws.com/role-arn` for IRSA / Pod Identity. | |
| 83 | +| Resources | `resources.requests`, `resources.limits` | dev-sized | **Override for production** — see EKS overlay. | |
| 84 | +| Disruption | `podDisruptionBudget.{enabled,maxUnavailable\|minAvailable}` | `true`, `1` | Voluntary-disruption guard during drains/upgrades. | |
| 85 | +| Lifecycle | `terminationGracePeriodSeconds`, `preStopSleepSeconds` | `120`, `5` | Time for in-flight S3 flush + leader stepdown. | |
| 86 | +| S3 | `s3.{endpoint,bucket,region,prefix,forcePathStyle,accessKey,secretKey}` | RustFS dev defaults | Leave creds empty on EKS to use the IRSA role. | |
| 87 | +| Superadmin | `superadmin.password` *or* `superadmin.existingSecret` | dev literal | **Always override in prod** — use `existingSecret` from sealed-secrets / ESO / SSM. | |
| 88 | + |
| 89 | +See [`values.yaml`](values.yaml) for the full set with inline comments, and |
| 90 | +[`values-eks-example.yaml`](values-eks-example.yaml) for a production-shaped |
| 91 | +override. |
| 92 | + |
| 93 | +## Architecture summary |
| 94 | + |
| 95 | +- **StatefulSet** of `replicas` pods. Each pod has an emptyDir for hot-tier |
| 96 | + data (`./data/duckdb/`, `./data/tantivy/`); BoilStream uploads to S3 |
| 97 | + continuously, so pod loss is recoverable. |
| 98 | +- **Headless Service** gives every pod a stable per-pod DNS name |
| 99 | + `boilstream-N.boilstream-headless.<ns>.svc.cluster.local`. Used as |
| 100 | + `cluster_mode.advertised_host`. |
| 101 | +- **Per-pod ClusterIP Services** are the backends for SNI-routed TLSRoutes. |
| 102 | +- **Envoy Gateway** (when enabled) exposes one TLS-passthrough listener per |
| 103 | + protocol; SNI hostname `boilstream-N.<domain>` routes to the matching pod. |
| 104 | +- **cert-manager** issues `*.<domain>` (public) and the cluster-mTLS cert. |
| 105 | +- **S3** is the source of truth for cluster coordination |
| 106 | + (`<prefix>cluster_state/leader.json`, `<prefix>cluster_state/brokers/*.json`) |
| 107 | + and per-user catalog backups. |
| 108 | + |
| 109 | +## Verify |
| 110 | + |
| 111 | +After install: |
| 112 | + |
| 113 | +```bash |
| 114 | +kubectl -n <ns> rollout status statefulset/boilstream |
| 115 | +kubectl -n <ns> get pods,gateway,certificate,secret |
| 116 | +``` |
| 117 | + |
| 118 | +Production-readiness suite (from `tests/k8s/`): |
| 119 | + |
| 120 | +```bash |
| 121 | +cd tests/k8s |
| 122 | +pip install -r requirements.txt |
| 123 | +pytest -v -k "not destructive" |
| 124 | +``` |
| 125 | + |
| 126 | +## Upgrade |
| 127 | + |
| 128 | +```bash |
| 129 | +helm upgrade boilstream ./charts/boilstream -f my-values.yaml |
| 130 | +``` |
| 131 | + |
| 132 | +The StatefulSet selector is intentionally narrow (`app: boilstream`) to keep |
| 133 | +upgrade-compatible across chart-version bumps. Standard `app.kubernetes.io/*` |
| 134 | +labels are added on top for tooling. |
0 commit comments