Wire dbt-starrocks adapter into ol-data-platform for B2B analytics materialization#2329
Wire dbt-starrocks adapter into ol-data-platform for B2B analytics materialization#2329blarghmatey wants to merge 16 commits into
Conversation
Add the dbt-starrocks adapter (v1.12.0) as a dependency so that dbt
models can be materialized directly into StarRocks. This is the
prerequisite for defining B2B analytics materialized views as dbt
models rather than Dagster DDL assets.
Changes:
- Add dbt-starrocks>=1.9.0 to pyproject.toml; uv resolved 1.12.0
- Add starrocks_qa and starrocks_production targets to profiles.yml,
pointing at lakehouse-starrocks-fe-service.starrocks.svc.cluster.local:9030
with credentials from DBT_STARROCKS_USERNAME / DBT_STARROCKS_PASSWORD
- Add b2b_analytics model directory config in dbt_project.yml with
+properties: {} and +grants: {} to clear Trino-specific inherited
settings, and iceberg_maintenance.enabled: false
- Add models/b2b_analytics/smoke_test.sql as a connectivity probe
(run with --target starrocks_qa or --target starrocks_production)
Closes mitodl/hq#10009
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR integrates the dbt-starrocks adapter into the ol_dbt project so dbt models can be materialized into StarRocks native tables (as groundwork for B2B analytics models to live in dbt instead of Dagster DDL assets).
Changes:
- Adds
dbt-starrocks(and its transitivemysql-connector-python) to the workspace dependencies and lockfile. - Adds StarRocks dbt profile targets (
starrocks_qa,starrocks_production) driven byDBT_STARROCKS_*environment variables. - Introduces a new
b2b_analyticsdbt models folder with a minimalsmoke_testmodel and schema YAML, plus folder-level config overrides indbt_project.yml.
Reviewed changes
Copilot reviewed 5 out of 6 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
pyproject.toml |
Adds dbt-starrocks>=1.9.0 to workspace dependencies. |
uv.lock |
Pins dbt-starrocks==1.12.0 and pulls in mysql-connector-python==9.7.0 transitively. |
src/ol_dbt/profiles.yml |
Adds starrocks_qa / starrocks_production targets using DBT_STARROCKS_* env vars. |
src/ol_dbt/dbt_project.yml |
Adds b2b_analytics folder config, clearing Trino-specific grants/properties and disabling Iceberg maintenance. |
src/ol_dbt/models/b2b_analytics/smoke_test.sql |
Adds a single-row table model to validate StarRocks connectivity. |
src/ol_dbt/models/b2b_analytics/_b2b_analytics__models.yml |
Adds model/column documentation for the smoke test model. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Three issues surfaced by review:
1. b2b_analytics models were not gated to StarRocks — they would be picked
up by normal Trino/DuckDB dbt build runs and fail. Fix: add
'+enabled: {{ target.type == "starrocks" }}' to the b2b_analytics
model config so they are disabled on all non-StarRocks targets.
2. '+schema: b2b_analytics' in dbt_project.yml would have produced the
schema name 'b2b_analytics_b2b_analytics' (dbt appends target.schema +
model schema). Fix: remove the +schema override entirely; the profile
already sets schema: b2b_analytics.
3. All source YAML files use '{{ target.database }}', which dbt-starrocks
does not expose (its connection_info() does not include a 'database'
key), so target.database is Jinja2 Undefined and causes a parse error
when running against any StarRocks target. Fix: add '| default(none)'
filter across all 14 affected source YAML files so StarRocks targets
parse cleanly while Trino and DuckDB targets are unaffected.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ive auth)
Following ol-infrastructure#4812, which wires Keycloak OIDC into StarRocks
via CREATE SECURITY INTEGRATION, developer-facing QA connections should use
OAuth2 while Dagster's production connection continues with Vault-managed
native credentials.
starrocks_qa:
- Host default changed to lakehouse.qa.starrocks.ol.mit.edu (the public QA
FQDN reachable from developer laptops; overrideable via DBT_STARROCKS_HOST)
- Credentials: DBT_STARROCKS_OIDC_USERNAME (Keycloak preferred_username) +
DBT_STARROCKS_OIDC_TOKEN (Keycloak access token)
- auth_plugin: mysql_clear_password so the JWT token is sent verbatim over
the MySQL wire (StarRocks validates it against the JWKS endpoint)
starrocks_production:
- Host stays at the in-cluster K8s service name (Dagster is in-cluster)
- Credentials: DBT_STARROCKS_USERNAME / DBT_STARROCKS_PASSWORD from Vault
dynamic credentials (native StarRocks auth, unaffected by OIDC config)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
will wait for keycloak auth PR before testing this one |
…mmand
- Add --mode vault to starrocks-auth: fetches native-password creds from
Vault dynamic database engine (database-starrocks-{env}/creds/{role})
and emits STARROCKS_USER / STARROCKS_PASSWORD instead of a JWT token.
--vault-role selects readonly/app/admin (default: app).
- Add ol-dbt starrocks subcommand (commands/starrocks.py): wraps ol-dbt run
with automatic Vault credential fetch and kubectl port-forward. Injects
STARROCKS_USER / STARROCKS_PASSWORD / STARROCKS_HOST into the environment
before delegating to ol-dbt run, so the dbt subprocess picks them up
without any manual credential management.
- Register starrocks_app in cli.py.
Rationale: dbt-starrocks uses mysql-connector-python directly and has no
OIDC support. Arrow Flight SQL skips all security integrations for non-MySQL
connections. Vault dynamic users are the correct path for automation — short-
lived native-password creds, no JWT plumbing needed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- ol-dbt starrocks now injects DBT_STARROCKS_USERNAME / DBT_STARROCKS_PASSWORD / DBT_STARROCKS_HOST to match the variable names profiles.yml already references. - Add starrocks_qa_vault profile: native-password (no auth_plugin), defaulting host to 127.0.0.1 for local port-forward use. The existing starrocks_qa target retains the OIDC/mysql_clear_password path for human interactive sessions. - _ENVS in starrocks.py now carries a dbt_target key so the command defaults to the right profile per env without requiring --target. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
vault-qa.odl.mit.edu for qa/ci, vault-production.odl.mit.edu for production. _vault_credentials() now passes VAULT_ADDR in the subprocess env rather than relying on the shell having it set, so the caller never needs to manage it. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… credential fetch
Add hvac[parser]>=2.0.0 to workspace and ol-dbt-cli package deps.
Add _vault_auth.py shared module in ol_dbt_cli.commands:
- _vault_client(): thin hvac.Client factory
- _oidc_callback(): one-shot HTTP server on port 8250 that captures the
Vault OIDC callback code (matching eks.py convention)
- load_vault_token(): checks ~/.cache/starrocks-auth/vault-token-{env}.json
for a live token before triggering browser OIDC login
- fetch_vault_db_credentials(): combines token load + client.read() to
return (username, password) from the database secrets engine
starrocks.py now imports fetch_vault_db_credentials from _vault_auth and
drops the subprocess.run(vault read ...) call.
bin/starrocks-auth mirrors the same hvac logic inline (can't import from the
package) but writes to the same token cache dir so a single browser login
is shared between starrocks-auth --mode vault and ol-dbt starrocks.
The vault binary is no longer required by either tool.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Vault OIDC auth role is now configurable rather than hardcoded to
'developer'. This lets an operator authenticate as 'admin' (or any other
role) when the developer policy doesn't yet have read access to the
target database mount.
Token cache key includes the OIDC role (vault-token-{env}-{role}.json)
so developer and admin tokens are stored separately and don't invalidate
each other.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The upstream adapter checks `properties is not none` to decide whether to
emit a PROPERTIES () block, but dbt_project.yml sets +properties: {} on
b2b_analytics models to clear inherited Iceberg/Trino format properties.
An empty dict is not None, so the adapter generated an empty
`PROPERTIES ( )` clause which StarRocks rejects with a syntax error.
Local macro override adds `and properties | length > 0` to the guard,
matching the same pattern used in apply_grants_macro_override.sql.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
58fb796 to
ef44899
Compare
- bin/starrocks-auth:526 — suppress CodeQL clear-text-logging false positive with nosec/noqa; print is intentional shell-eval output - _vault_auth.py:42 — guard _oidc_callback against error responses from denied OIDC auth - _vault_auth.py:79 — use .get() for nonce/state keys with RuntimeError on missing params - starrocks.py:82 — ci env now defaults port_forward=False; parameter default is None, resolved from env config
- _vault_auth.py + bin/starrocks-auth: guard client.read() against None return (invalid Vault path) and hvac.exceptions.Forbidden (insufficient policy); both now raise RuntimeError with actionable messages
| urllib.parse.parse_qsl(urllib.parse.urlparse(self.path).query) | ||
| ) | ||
| result["code"] = params.get("code", "") | ||
| self.send_response(200) |
There was a problem hiding this comment.
Callback handler always returns HTTP 200 but never checks for an error param. A denied/cancelled OIDC login silently returns an empty code string rather than raising. There's a similar guard in _vault_auth.py to handle this.
There was a problem hiding this comment.
Fixed in b6a5240 — _vault_oidc_callback's handler now checks for an error param, returns HTTP 400 with a user-readable message in the browser tab, and the function raises RuntimeError (with the error value, or a 'no authorization code' message if the code is empty) instead of silently returning an empty string. Mirrors _vault_auth.py's _oidc_callback.
|
|
||
|
|
||
| def _output_token(username: str, token: str, **_) -> None: # noqa: ARG001 | ||
| print(token) # noqa: T201 |
There was a problem hiding this comment.
Might need to add an additional annotation to pass CodeQL CI check. Same with line 534 as tagged by copilot below.
There was a problem hiding this comment.
Fixed in b6a5240 — added the same # nosec B106/# noqa: T201 annotation (plus a comment explaining token can be a Vault-issued password under --mode vault) to _output_token's print(token), matching the annotation already applied to _output_env's password print.
quazi-h
left a comment
There was a problem hiding this comment.
I utilized copilot to help with this review and was able to identify a few issues that I've added comments for. It looks like we have guard logic in _vault_auth.py that can be applied to those specific bugs. Besides that copilot also identified a browser/server race condition for starrocks-auth and _vault_auth regarding HTTP server / socket callback requests and binding but it looks like it was brought up in an earlier review by copilot and marked as resolved so you might have already handled/dismissed that.
Adds the six aggregated MVs from the B2B self-serve analytics epic as
dbt-starrocks materialized_view models in the existing b2b_analytics
model path (gated to target.type == 'starrocks'):
- mv_b2b_contract_utilization (org x contract)
- mv_b2b_enrollment_completion_funnel (org x contract x course_run)
- mv_b2b_monthly_engagement_trend (org x year_month)
- mv_b2b_program_funnel (org x contract x program)
- mv_b2b_content_engagement_depth (org x course_run)
- mv_b2b_mit_admin_contract_health (org x contract, MIT admin only)
Also adds _b2b_analytics__sources.yml declaring the dimensional and
reporting Iceberg sources via the ol_data_lake_{qa,production}
external catalog, and documents the new models in
_b2b_analytics__models.yml.
Model configs use the real dbt-starrocks materialized_view macro
options (distributed_by as a column list, buckets, refresh_method) --
the SQL-string-based config keys (refresh_async, refresh_interval)
assumed in the original epic spec don't exist in the adapter.
refresh_method is 'manual' since refresh is triggered by the Dagster
MV-refresh asset, not on a fixed interval.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds refresh_starrocks_analytics_mvs, a Dagster asset that manually
refreshes the 6 b2b_analytics StarRocks materialized views after the
upstream organization_administration_report dbt model materializes.
- StarRocksResource (resources/starrocks.py): connects over the MySQL
wire protocol using Vault dynamic database credentials, modeled on
VaultMySQLClientFactory (legacy_openedx/resources/mysql_db.py) rather
than the static KV-secret approach originally sketched in the task —
that's the pattern this branch already established for StarRocks Vault
auth (see src/ol_dbt_cli/ol_dbt_cli/commands/_vault_auth.py).
- assets/starrocks_mv_refresh.py: issues REFRESH MATERIALIZED VIEW ...
WITH SYNC MODE for each MV, matching the manual refresh_method set on
the dbt-starrocks materialized_view models.
- Registered both in definitions.py alongside the existing S3/Vault
resources and export_b2b_organization_data asset.
Vault database-secrets-engine mount point names (starrocks_vault_mount_map)
are placeholders following the mariadb-{deployment} convention and need
confirming against the actual ol-infrastructure StarRocks Vault config
before this runs for real.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Three review comments from quazi-h (2026-06-30) on PR #2329 were never addressed -- the fixes applied to _vault_auth.py's copy of this logic (0181b17) were never mirrored into bin/starrocks-auth's own near-duplicate implementation: - _vault_oidc_callback now checks for an 'error' callback param, returns HTTP 400 with a user-readable message in the browser tab, and raises RuntimeError with the error value after the server shuts down, instead of silently returning an empty code string on a denied/cancelled login. - _load_vault_token now guards nonce/state extraction with qs.get(...) and raises RuntimeError if either is missing from the Vault OIDC auth URL, instead of a bare KeyError. - _output_token annotates its print(token) with the same nosec B106 / noqa: T201 pattern already applied to _output_env's password print, since token can be a Vault-issued password under --mode vault. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
sentry flagged (PR #2329 comment 3516283942): execute() had no retry handling, so a single transient pymysql.err.OperationalError (1044/1045 from Vault-dynamic-user replication lag across StarRocks FE nodes, or 2006/2013 from a dropped connection) on any one materialized view would abort refresh_starrocks_analytics_mvs entirely, leaving later MVs in the loop un-refreshed for that run. execute() now retries up to 3 attempts with exponential backoff (1s, 2s), regenerating fresh Vault credentials each attempt, for the same error-code set VaultMySQLClientFactory treats as retriable. Non-retriable errors and exhausted retries still raise immediately/loudly, so a genuinely broken MV still fails the asset with a clear error -- only transient errors are absorbed. Verified with a standalone script covering: retry-then-succeed (both connect-time and query-execution-time failures), non-retriable error raises immediately, and exhausted retries re-raise the last error. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…se-starrocks-{env}
The dev/ci/qa/production mount map used the placeholder mysql-starrocks-{env}
convention, while bin/starrocks-auth and ol_dbt_cli/commands/starrocks.py
already use database-starrocks-{env} against the real Vault database secrets
engine. Only one of these can exist, so StarRocksResource._generate_credentials
would 404 the first time it ran for real.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
| database: "{{ 'ol_data_lake_qa' if 'qa' in target.name else 'ol_data_lake_production'\ | ||
| \ }}" | ||
| schema: "{{ 'ol_warehouse_qa_dimensional' if 'qa' in target.name else 'ol_warehouse_production_dimensional'\ | ||
| \ }}" |
| database: "{{ 'ol_data_lake_qa' if 'qa' in target.name else 'ol_data_lake_production'\ | ||
| \ }}" | ||
| schema: "{{ 'ol_warehouse_qa_reporting' if 'qa' in target.name else 'ol_warehouse_production_reporting'\ | ||
| \ }}" |
| _CACHE_DIR.mkdir(parents=True, exist_ok=True) | ||
| cache_path.write_text(json.dumps({"token": token})) | ||
| return token |
| def do_GET(self) -> None: | ||
| params = dict(urllib.parse.parse_qsl(urllib.parse.urlparse(self.path).query)) | ||
| error = params.get("error") | ||
| if error: | ||
| result["error"] = error |
| conn = connect( | ||
| host=self.host, | ||
| port=self.port, | ||
| database=self.database, | ||
| user=username, | ||
| password=password, | ||
| cursorclass=DictCursor, | ||
| ) |
This branch integrates the dbt-starrocks adapter (v1.12.0) to enable dbt models to be materialized directly into StarRocks native tables, unlocking B2B analytics views as dbt models rather than Dagster DDL assets.
Key Changes
starrocks_qa(OIDC for human users),starrocks_qa_vault(Vault dynamic creds for local dev),starrocks_production(in-cluster Vault for Dagster)target.type == "starrocks"with empty properties/grants to clear inherited Trino-specific config; includes smoke-test connectivity probeFiles Changed
Added (7 files)
bin/starrocks-auth— Multi-mode StarRocks credential CLI (PKCE/CI/Vault) with token cachesrc/ol_dbt/macros/starrocks_relation_helpers_override.sql— Guard empty PROPERTIES dicts against StarRocks syntax errorssrc/ol_dbt/models/b2b_analytics/smoke_test.sql— Single-row connectivity test tablesrc/ol_dbt/models/b2b_analytics/_b2b_analytics__models.yml— Model documentationsrc/ol_dbt_cli/ol_dbt_cli/commands/_vault_auth.py— Shared hvac helpers (load_vault_token, fetch_vault_db_credentials)src/ol_dbt_cli/ol_dbt_cli/commands/starrocks.py— ol-dbt starrocks subcommand with port-forward orchestrationModified (9 files)
pyproject.toml,src/ol_dbt_cli/pyproject.toml— Added hvac and dbt-starrocks dependenciessrc/ol_dbt/profiles.yml— Added starrocks_qa, starrocks_qa_vault, starrocks_production targetssrc/ol_dbt/dbt_project.yml— b2b_analytics model config (enabled on StarRocks only, clear properties/grants)src/ol_dbt_cli/ol_dbt_cli/cli.py— Registered starrocks_app| default(none)filter to{{ target.database }}references (dbt-starrocks does not populate target.database)User Experience
Before: B2B analytics materialized views defined as Dagster DDL assets; dbt had no way to target StarRocks.
After: Developers can now run dbt against StarRocks with a single command:
This fetches Vault credentials, tunnels the database port via kubectl, and runs dbt with injected environment variables—no manual credential or tunnel setup required.
Closes https://github.com/mitodl/hq/issues/10009
Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
Implementation Plan
Implementation Plan (Plan #2329)
What are the relevant tickets?
Closes https://github.com/mitodl/hq/issues/10009
Description (What does it do?)
Wires the dbt-starrocks adapter into ol-data-platform so that dbt models can be materialized directly into StarRocks native tables. This is the prerequisite for defining B2B analytics materialized views as dbt models rather than as Dagster DDL assets.
dbt adapter and profile targets
pyproject.toml— addsdbt-starrocks>=1.9.0; uv resolved1.12.0(mysql-connector-python==9.7.0is a transitive dep).src/ol_dbt/profiles.yml— three StarRocks targets:starrocks_qa— OIDC path for human developers. Host defaults tolakehouse.qa.starrocks.ol.mit.edu(public QA FQDN). Credentials:DBT_STARROCKS_OIDC_USERNAME+DBT_STARROCKS_OIDC_TOKEN(Keycloak access token). Usesauth_plugin: mysql_clear_passwordso the JWT is sent verbatim over the MySQL wire for server-side JWKS validation.starrocks_qa_vault— native-password path for local development via Vault dynamic credentials. Host defaults to127.0.0.1(port-forward). Credentials:DBT_STARROCKS_USERNAME+DBT_STARROCKS_PASSWORD.starrocks_production— native-password path for in-cluster Dagster. Host defaults tolakehouse-starrocks-fe-service.starrocks.svc.cluster.local:9030. Credentials:DBT_STARROCKS_USERNAME+DBT_STARROCKS_PASSWORDfrom Vault dynamic credentials.src/ol_dbt/dbt_project.yml—b2b_analyticsmodel-path config that:target.type == "starrocks"so they are disabled on Trino/DuckDB runs.+properties: {}and+grants: {}to remove inherited Trino-specific settings.iceberg_maintenance.enabled: false(irrelevant for native StarRocks tables).src/ol_dbt/models/b2b_analytics/smoke_test.sql— single-row probe table for connectivity validation.| default(none)to{{ target.database }}references.dbt-starrocksdoes not populatetarget.database, causing a Jinja2 parse error on StarRocks targets; this filter makes it a no-op on those targets while leaving Trino and DuckDB unaffected.Developer tooling (
bin/starrocks-auth+ol-dbt starrocks)bin/starrocks-auth— new CLI script for obtaining StarRocks credentials locally. Supports three auth flows (--mode browser | ci | vault) and four output modes (--output token | env | exec | mysql). Useshvacdirectly for the Vault OIDC browser flow and dynamic credential fetch — novaultbinary required. Token cached at~/.cache/starrocks-auth/vault-token-{env}.json.src/ol_dbt_cli/ol_dbt_cli/commands/_vault_auth.py— shared hvac helper (load_vault_token,fetch_vault_db_credentials) used by theol-dbtCLI. Writes to the same cache dir asstarrocks-authso a single browser login is shared between both tools.src/ol_dbt_cli/ol_dbt_cli/commands/starrocks.py— newol-dbt starrockssubcommand. Fetches Vault dynamic credentials, optionally starts akubectl port-forwardto the StarRocks FE service, injectsDBT_STARROCKS_USERNAME/DBT_STARROCKS_PASSWORD/DBT_STARROCKS_HOST, then delegates tool-dbt run.hvac[parser]>=2.0.0,<3added to bothpyproject.toml(workspace) andsrc/ol_dbt_cli/pyproject.toml.How can this be tested?
Local developer (Vault + port-forward):
Or as a single command:
In-cluster (Dagster / CI):
# With DBT_STARROCKS_USERNAME and DBT_STARROCKS_PASSWORD from Vault dbt run --target starrocks_production --select b2b_analytics.smoke_testAdditional Context
The
starrocks_qaandstarrocks_productiontarget names are intentionally notproduction, so the existingapply_grantsmacro override (which skips grants on non-production-named targets) also skips them — Starburst role names don't exist in StarRocks.The
b2b_analyticsschema in StarRocks will need to be created and the dbt service account granted permissions before running (tracked in https://github.com/mitodl/hq/issues/10006).StarRocks auth architecture note: StarRocks has two disjoint auth paths —
authentication_jwt(MySQL wire, requires MySQL 9.2+ client pluginauthentication_openid_connect_client) for OIDC, and native-password for service accounts. MariaDB and MySQL <9.2 cannot use the JWT path. Arrow Flight SQL also only supports native-password auth. Thestarrocks_qa_vaultandstarrocks_productiontargets use native-password exclusively, which works with all clients and is the correct path for automation. See upstream issue StarRocks/starrocks#75372 for a proposal to make the JWT path work with non-MySQL-9.2 clients.To replicate this PR locally, run: