Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -190,6 +190,7 @@ local-deploy: docker-build-local kind-load-operator manifests ## Build, load, an
--set image.pullPolicy=Never \
--set metrics.secure=false \
--set leaderElection.enabled=false \
--set telemetry.enabled=false \
--set logging.development=true \
--set-string podAnnotations.deploy-timestamp="$(shell date +%s)"
$(KUBECTL) rollout status deployment/firebolt-operator -n default --timeout=30s
Expand Down
23 changes: 22 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
# Firebolt Kubernetes Operator

[![Downloads](https://scarf.sh/installs-badge/firebolt-db/firebolt-operator?package-type=docker)](https://scarf.sh/)
[![Companies](https://scarf.sh/company-badge/firebolt-db/firebolt-operator?package-type=docker)](https://scarf.sh/)

A Kubernetes operator that manages Firebolt infrastructure: metadata services, an Envoy query-routing proxy, and compute engines with zero-downtime scaling via blue-green deployments.

## Overview
Expand Down Expand Up @@ -53,6 +56,20 @@ This is the fast path if you want the agent to drive the install for you. If you

For a step-by-step walkthrough, follow the quickstart guide in our [official documentation](https://docs.firebolt.io/self-managed/firebolt-operator/quickstart) or using the documentation source file at [`docs/quickstart.mdx`](docs/quickstart.mdx).

## Telemetry

The operator sends one anonymous, aggregate usage event to [Scarf](https://scarf.sh) at most once per day. This helps Firebolt understand how the community runs the operator and prioritize improvements.

The event contains the operator and engine versions, Kubernetes minor version, OS and architecture, and bucketed instance, engine, and replica counts. It does not contain names, stable identifiers, query data, schemas, connection parameters, secrets, or configuration. Counts are bucketed rather than exact. As with any network request, the source IP address is visible to Scarf; Scarf may use it to infer the company and does not store it.

Image and chart pulls through `oci.firebolt.io` additionally record the requested version and platform before redirecting to GitHub Container Registry.

You can opt out in any of these ways:

- Set `telemetry.enabled=false` in the Helm values.
- Set `DO_NOT_TRACK=1` or `SCARF_NO_ANALYTICS=1` in the operator environment.
- Pull images and charts directly from `ghcr.io` instead of `oci.firebolt.io`.

## Firebolt Operator flags

The Firebolt Operator supports these runtime flags. The binary default is what
Expand All @@ -78,6 +95,8 @@ the manager uses when you run it directly. The Helm chart default is what the
| `--engine-max-cpu` | `""` | Not set | Maximum allowed CPU request and limit on the engine container (`spec.template.spec.containers[name=engine].resources`). Empty disables the bound. |
| `--engine-max-memory` | `""` | Not set | Maximum allowed memory request and limit on the engine container. Empty disables the bound. |
| `--engine-max-ephemeral-storage` | `""` | Not set | Maximum allowed ephemeral-storage request and limit on the engine container. Empty disables the bound. |
| `--telemetry` | `true` | `true` | Send a once-daily anonymous aggregate usage event. |
| `--telemetry-endpoint` | `https://telemetry.firebolt.io/firebolt-operator` | Same as binary | Scarf Event Collection endpoint for anonymous aggregate usage events. |
| `--zap-devel` | `false` | Not set | Enable controller-runtime development logging defaults. |
| `--zap-encoder` | `json` | `json` | Log encoding. Valid values are `json` and `console`. |
| `--zap-log-level` | `info` | `info` | Minimum log level. Valid values include `debug`, `info`, `error`, and `panic`. |
Expand All @@ -95,4 +114,6 @@ make test-e2e # E2E tests (requires Kind cluster)
## Where to go next
- For **contributor** detail, conventions, and rules for making changes to this repo, see [`AGENTS.md`](AGENTS.md).
- The Helm chart for the operator lives in [helm/firebolt-operator](helm/firebolt-operator/README.md).
- The pure CRD chart for the operator lives in [helm/firebolt-operator-crds](helm/firebolt-operator-crds/README.md).
- The pure CRD chart for the operator lives in [helm/firebolt-operator-crds](helm/firebolt-operator-crds/README.md).

<img referrerpolicy="no-referrer-when-downgrade" src="https://px.firebolt.io/a.png?x-pxid=44d8e5cd-a225-4212-8705-a7639ab30398&page=operator-README" alt="" width="1" height="1" style="position:absolute; width:1px; height:1px; opacity:0; pointer-events:none;" />
18 changes: 18 additions & 0 deletions cmd/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ import (
computev1alpha1 "github.com/firebolt-db/firebolt-kubernetes-operator/api/v1alpha1"
"github.com/firebolt-db/firebolt-kubernetes-operator/internal/controller"
fireboltmetrics "github.com/firebolt-db/firebolt-kubernetes-operator/internal/metrics"
"github.com/firebolt-db/firebolt-kubernetes-operator/internal/telemetry"
// +kubebuilder:scaffold:imports
)

Expand Down Expand Up @@ -75,6 +76,8 @@ func main() {
var watchNamespacesArg string
var engineMaxCPUStr, engineMaxMemoryStr, engineMaxEphemeralStorageStr string
var gatewayWakeClusterRole string
var telemetryEnabled bool
var telemetryEndpoint string
var tlsOpts []func(*tls.Config)
flag.BoolVar(&showVersion, "version", false, "Print the version and exit.")
flag.StringVar(&metricsAddr, "metrics-bind-address", "0", "The address the metrics endpoint binds to. "+
Expand Down Expand Up @@ -115,6 +118,10 @@ func main() {
"a per-instance RoleBinding (so the gateway can stamp the wake annotation). "+
"Empty fails any FireboltInstance reconcile that requires operator-managed gateway RBAC; "+
"users supplying their own gateway ServiceAccount via spec.gateway.template.spec.serviceAccountName are unaffected.")
flag.BoolVar(&telemetryEnabled, "telemetry", true,
"Send a once-daily anonymous aggregate usage event. Set false to disable.")
flag.StringVar(&telemetryEndpoint, "telemetry-endpoint", telemetry.DefaultEndpoint,
"Scarf Event Collection endpoint for anonymous aggregate usage events.")
zapOpts := zap.Options{Development: false}
zapOpts.BindFlags(flag.CommandLine)
flag.Parse()
Expand Down Expand Up @@ -283,6 +290,17 @@ func main() {
os.Exit(1)
}
}

if err := mgr.Add(&telemetry.Reporter{
Client: mgr.GetClient(),
Config: mgr.GetConfig(),
OperatorVersion: version,
Endpoint: telemetryEndpoint,
Enabled: telemetryEnabled,
}); err != nil {
setupLog.Error(err, "unable to register telemetry reporter")
os.Exit(1)
}
// +kubebuilder:scaffold:builder

if err := mgr.AddHealthzCheck("healthz", healthz.Ping); err != nil {
Expand Down
12 changes: 9 additions & 3 deletions docs/installation.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ Install the CRD chart before installing the Firebolt Operator chart. This is the
allows you full control over the CRDs lifecycle.

```bash
helm upgrade --install firebolt-crds oci://ghcr.io/firebolt-db/helm-charts/firebolt-operator-crds
helm upgrade --install firebolt-crds oci://oci.firebolt.io/firebolt-db/helm-charts/firebolt-operator-crds
```

### Option 2: Firebolt Operator chart `crds/` directory
Expand All @@ -145,11 +145,17 @@ Prefer Option 1 for any deployment that will need to upgrade the operator over t
Install the Firebolt Operator controller:

```bash
helm upgrade --skip-crds --install firebolt-operator oci://ghcr.io/firebolt-db/helm-charts/firebolt-operator
helm upgrade --skip-crds --install firebolt-operator oci://oci.firebolt.io/firebolt-db/helm-charts/firebolt-operator
```

Omit `--skip-crds` if you chose the bundled `crds/` directory option above.

The `oci.firebolt.io` host is Firebolt's [Scarf](https://scarf.sh) gateway. It records the requested chart version and platform and redirects to GitHub Container Registry. To bypass the gateway, replace `oci://oci.firebolt.io` with `oci://ghcr.io`.

By default, the operator also sends one anonymous aggregate usage event to Scarf at most once per day. The event contains operator and engine versions, Kubernetes minor version, OS and architecture, and bucketed instance, engine, and replica counts. It contains no names, stable identifiers, query data, schemas, connection parameters, secrets, or configuration. As with any network request, the source IP is visible to Scarf; Scarf may use it to infer the company and does not store it.

Disable runtime telemetry with `--set telemetry.enabled=false`, or set `DO_NOT_TRACK=1` or `SCARF_NO_ANALYTICS=1` in the operator environment.

## Multi-tenant install: scope the Firebolt Operator to specific namespaces

By default the Firebolt Operator watches every namespace and the chart
Expand All @@ -160,7 +166,7 @@ them under `watchNamespaces`:

```bash
helm upgrade --skip-crds --install firebolt-operator \
oci://ghcr.io/firebolt-db/helm-charts/firebolt-operator \
oci://oci.firebolt.io/firebolt-db/helm-charts/firebolt-operator \
--set 'watchNamespaces={tenant-a,tenant-b}'
```

Expand Down
6 changes: 4 additions & 2 deletions docs/quickstart.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -41,10 +41,12 @@ firebolt-control-plane Ready control-plane 30s v1.35.0
Install the CRDs and operator with Helm:

```bash
helm upgrade --install firebolt-crds oci://ghcr.io/firebolt-db/helm-charts/firebolt-operator-crds
helm upgrade --skip-crds --install firebolt-operator oci://ghcr.io/firebolt-db/helm-charts/firebolt-operator
helm upgrade --install firebolt-crds oci://oci.firebolt.io/firebolt-db/helm-charts/firebolt-operator-crds
helm upgrade --skip-crds --install firebolt-operator oci://oci.firebolt.io/firebolt-db/helm-charts/firebolt-operator
```

The `oci.firebolt.io` host is Firebolt's [Scarf](https://scarf.sh) gateway. It records anonymous download statistics and redirects to GitHub Container Registry. To bypass the gateway, use `oci://ghcr.io/firebolt-db/helm-charts` instead.

For more installation options, see [Installation](./installation).

## Create a namespace
Expand Down
3 changes: 2 additions & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,13 @@ go 1.26.0

require (
github.com/distribution/reference v0.6.0
github.com/go-logr/logr v1.4.3
github.com/oklog/ulid/v2 v2.1.1
github.com/onsi/ginkgo/v2 v2.32.0
github.com/onsi/gomega v1.42.1
github.com/prometheus/client_golang v1.23.2
github.com/prometheus/client_model v0.6.2
github.com/scarf-sh/scarf-go v0.1.0
github.com/spf13/cobra v1.10.2
go.uber.org/zap v1.28.0
k8s.io/api v0.36.2
Expand Down Expand Up @@ -36,7 +38,6 @@ require (
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/fsnotify/fsnotify v1.9.0 // indirect
github.com/fxamacker/cbor/v2 v2.9.0 // indirect
github.com/go-logr/logr v1.4.3 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-logr/zapr v1.3.0 // indirect
github.com/go-openapi/jsonpointer v0.21.0 // indirect
Expand Down
2 changes: 2 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,8 @@ github.com/prometheus/procfs v0.19.2/go.mod h1:M0aotyiemPhBCM0z5w87kL22CxfcH05Zp
github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/scarf-sh/scarf-go v0.1.0 h1:A/o5+Q1T8ExmKNCa4D0c1iw5iB3EpS10jyycG9isWr0=
github.com/scarf-sh/scarf-go v0.1.0/go.mod h1:5Ll7/48YAVqo+GKDb+ptdZhwGat6TqPSbkGaZLVWtLI=
github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU=
github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4=
github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
Expand Down
1 change: 1 addition & 0 deletions helm/firebolt-operator/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ kubectl delete crd fireboltengines.compute.firebolt.io fireboltinstances.compute
| serviceAccount.name | string | `""` | The name of the ServiceAccount to use. If empty, a name is generated from the fullname template. |
| serviceMonitor.operator | object | `{"bearerTokenSecret":{"key":"token","name":""},"enabled":false,"interval":"15s"}` | Create a ServiceMonitor that scrapes the operator's controller-manager metrics Service (`<release>-metrics`). A ServiceMonitor is used (instead of a PodMonitor) because the secure path (`metrics.secure: true`) needs to authenticate to controller-runtime's built-in authn/authz, and only `ServiceMonitor.endpoint.authorization` exposes a credential reference. |
| serviceMonitor.operator.bearerTokenSecret | object | {} | Secret holding the bearer token Prometheus presents to the operator's metrics endpoint. Required when `metrics.secure: true`. Bring your own Secret (e.g. an ESO/Vault-projected one, a `kubernetes.io/service-account-token` Secret bound to your scrape SA, or whatever your Prometheus install already uses); leave `name` empty to render no authorization (the secure endpoint will reject scrapes). |
| telemetry.enabled | bool | `true` | Send a once-daily anonymous aggregate usage event to Scarf. The event contains operator/engine versions, Kubernetes minor, OS/architecture, and bucketed instance/engine/replica counts. It contains no names, stable identifiers, query data, schemas, or configuration. The source IP is visible to Scarf for company inference but is not stored. Set false to disable; DO_NOT_TRACK and SCARF_NO_ANALYTICS are also honored. |
| tolerations | list | `[]` | Tolerations for the operator pod. |
| topologySpreadConstraints | list | `[]` | Topology spread constraints for the operator pod. |
| watchNamespaces | list | `[]` | Namespaces the operator watches and the chart's manager RBAC is scoped to. Empty (the default) renders a cluster-wide install: one `ClusterRole` + `ClusterRoleBinding` for the manager, and the operator starts with `--namespaces=""` (controller-runtime watches every namespace). A non-empty list renders a `Role` + `RoleBinding` pair per listed namespace with the same rule set, and the operator starts with `--namespaces=<comma-list>` so its cache spans only those namespaces. Switching modes is a `helm upgrade` plus an operator restart. |
Expand Down
3 changes: 3 additions & 0 deletions helm/firebolt-operator/templates/deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,9 @@ spec:
- --engine-max-ephemeral-storage={{ . }}
{{- end }}
- --gateway-wake-cluster-role={{ include "firebolt-operator.gatewayWakeClusterRoleName" . }}
{{- if eq .Values.telemetry.enabled false }}
- --telemetry=false
{{- end }}
{{- if (.Values.logging.development | default false) }}
- --zap-devel=true
{{- else }}
Expand Down
9 changes: 9 additions & 0 deletions helm/firebolt-operator/values.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -220,6 +220,15 @@
"priorityClassName": {
"type": "string"
},
"telemetry": {
"type": "object",
"properties": {
"enabled": {
"type": "boolean"
}
},
"required": ["enabled"]
},
"additionalArgs": {
"type": "array",
"items": { "type": "string" }
Expand Down
9 changes: 9 additions & 0 deletions helm/firebolt-operator/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -209,6 +209,15 @@ logging:
# -- Priority class name for the operator pod.
priorityClassName: ""

telemetry:
# -- Send a once-daily anonymous aggregate usage event to Scarf. The event
# contains operator/engine versions, Kubernetes minor, OS/architecture, and
# bucketed instance/engine/replica counts. It contains no names, stable
# identifiers, query data, schemas, or configuration. The source IP is visible
# to Scarf for company inference but is not stored. Set false to disable;
# DO_NOT_TRACK and SCARF_NO_ANALYTICS are also honored.
enabled: true

# -- Additional CLI arguments passed to the operator binary.
additionalArgs: []

Expand Down
Loading