docs(dbt): CI/QA hardening & data-trust technical spec

[blarghmatey]

What & why

Adds docs/specs/DBT_WAREHOUSE_CI_QA_SPEC.md — the spec-phase design for the dbt-warehouse CI/QA hardening & data-trust effort (project wp-dbt-warehouse-ci-qa-hardening-data-trust-021724, origin issue #2407, related to dimensional-migration epic #2072).

Motivation: the repo has strong test content (~3,100 dbt tests, heavy dbt_expectations) and a purpose-built ol-dbt CLI whose impact/validate commands already emit --format json "for CI pipelines" — but essentially nothing runs on dbt PRs today (only pre-commit sqlfluff). The biggest confidence win is wiring tools that already exist. This spec sequences that work by credential/engine access so each phase ships independently.

Contents

The doc turns the discovery-phase gap analysis into implementable designs with per-phase acceptance criteria, grounded in real file/line references:

• Phase 0 — ol-dbt diff (Add ol-dbt diff command to QA migrated mart/reporting models against pre-migration output #2407) — detailed: commands/diff.py signature, dbt_audit_helper in packages.yml, column-reconcile-first algorithm reusing sql_parser/manifest/yaml_registry, zero-copy via override_ref/override_source DuckDB views, JSON schema, exit codes, test_diff.py.
• Phase 1 — Fast PR CI (GitHub Actions, no creds) — dbt_pr_ci.yaml: parse, ol-dbt validate, ol-dbt impact PR comment, sqlfluff, pytest.
• Phase 1a — Dimensional-layering lint — manifest-based check enforcing [Epic] Migrate mart and reporting models to use the dimensional layer #2072 DoD (no mart/reporting → stg__/int__), with a checked-in baseline so today's 39 violators don't red-wall CI.
• Phase 2 / 2a — Concourse slim data CI + StarRocks schema_suffix namespacing — design altitude.
• Phase 3 — Runtime + org trust — Dagster asset checks, source freshness, OpenMetadata DBT test-case push, impact + OM lineage.

Section 9 has the task index/sequencing; section 10 lists open implementation questions.

Type

Documentation only — no code or behavior change. Individual phases land as their own implementation PRs.

Refs #2407, #2072.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds a technical specification document to guide a phased implementation of dbt CI/QA hardening and “data trust” improvements for src/ol_dbt and the ol-dbt CLI, with explicit acceptance criteria per phase and references to related issues/epics.

Changes:

• Introduces a spec document detailing Phase 0 (ol-dbt diff) through Phase 3 (runtime/org trust), including CI workflow design and linting proposals.
• Defines proposed interfaces/algorithms/output schemas for future implementation work (documentation-only; no behavior changes).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[blarghmatey]

Thanks @copilot — addressed all three in 25c70f4:

• driveable → drivable
• Slim-CI example collapsed onto a single inline-code line and named the concrete subcommand (dbt build) instead of the ambiguous dbt <sub>
• Clarified the gap-analysis reference is an internal witan agent-memory and noted the doc is self-contained (findings summarized in §1–§2).

[sentry]

Bug: The specification document incorrectly lists starrocks as an existing CLI sub-app, but it is not implemented in the codebase.
_{Severity: LOW}

Suggested Fix

Remove starrocks from the list of cyclopts.App sub-apps in the specification document to accurately reflect the current state of the codebase. It could also be noted elsewhere that starrocks is a dbt target profile, not a CLI sub-app.

Prompt for AI Agent

Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.

Location: docs/specs/DBT_WAREHOUSE_CI_QA_SPEC.md#L60

Potential issue: The specification document at `docs/specs/DBT_WAREHOUSE_CI_QA_SPEC.md`
incorrectly claims that `starrocks` is an existing `cyclopts.App` sub-app, alongside
`local`, `run`, and `generate`. A review of the codebase confirms that no
`starrocks_app` is defined, imported, or registered within the CLI application. This
factual error in the documentation could mislead developers, causing them to search for
non-existent code or misunderstand the current state of the CLI's implementation.

_{Did we get this right? 👍 / 👎 to inform future reviews.}