Skip to content

Commit 777a37b

Browse files
anandgupta42claude
andauthored
feat: automated dbt unit test generation (#674)
* feat: [AI-673] automated dbt unit test generation Add `dbt_unit_test_gen` tool and `dbt-unit-tests` skill for generating dbt unit tests from manifest + compiled SQL. Engine (`unit-tests.ts`): - Reuses `parseManifest()` for model info, deps, columns, descriptions - Reuses `dbtLineage()` for compiled SQL + column lineage mapping - Reuses `schema.inspect` for warehouse column enrichment - Reuses `sql.optimize` for anti-pattern detection - Keyword-based scenario detection (CASE, JOIN, GROUP BY, division, incremental) - Type-correct mock data generation with null/boundary/happy-path variants - Incremental tests include `input: this` mock for existing table state - Ephemeral deps correctly use `format: sql` even with no known columns - Cross-database support via `database` param in `schema.inspect` - YAML assembly via `yaml` library (not string concatenation) - Deterministic test names (index-based, not `Date.now()`) - Rich `UnitTestContext` with descriptions + lineage for LLM refinement Shared infrastructure: - Extract `helpers.ts` from `lineage.ts` (shared: `loadRawManifest`, `findModel`, `getUniqueId`, `detectDialect`, `buildSchemaContext`) - Manifest cache by path+mtime (avoids re-reading 128MB files) - Add `description` to `DbtModelInfo`/`DbtSourceInfo` - Add `adapter_type` to `DbtManifestResult` Skill (`dbt-unit-tests/SKILL.md`): - 5-phase workflow: Analyze -> Generate -> Refine -> Validate -> Write - Reference guides: YAML spec, edge-case patterns, incremental testing Tests: 32 tests covering manifest parsing, scenario detection, YAML round-trip validation, incremental `this` mock, ephemeral sql format, deterministic naming, descriptions, lineage context, source deps. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: [AI-673] address PR #674 review comments (CodeRabbit + cubic) Address review comments from CodeRabbit and cubic on PR #674. Bug fixes: - `resolveUpstream` now resolves seed.* and snapshot.* deps in addition to models and sources. Previously silently dropped, causing `dbt test` to fail for any model ref()ing a seed or snapshot. - Test name truncation no longer cuts off scenario suffix. Budget suffix length first, then truncate the model-name portion, so scenario names like `_null_handling_2` are always preserved even for long model names. - `findModel`/`getUniqueId` in helpers.ts now validate `resource_type === "model"` on the key lookup path, not just the name fallback. Prevents returning non-model nodes by unique_id. - Division detection regex now strips string literals AND comments before matching, so `'2024/01/15'` no longer triggers a false-positive boundary scenario. Documentation fixes: - `incremental-testing.md`: fix Jinja syntax — `{{ if is_incremental() }}` is invalid; use `{% if is_incremental() %}` for control flow. - `SKILL.md`: workflow header now shows all 5 phases (Analyze -> Generate -> Refine -> Validate -> Write). - `SKILL.md`: add language label to fenced code block for markdownlint. - `unit-test-yaml-spec.md`: show both top-level `tags` and nested `config.tags` forms explicitly. Infrastructure: - Add `seeds` and `snapshots` arrays to `DbtManifestResult` (previously only counts were returned). `parseManifest` now extracts full seed and snapshot info using the same shape as `DbtModelInfo`. - Tests migrate to shared `tmpdir()` fixture from `test/fixture/fixture.ts` for automatic cleanup (per project coding guidelines). - Wrap `DbtUnitTestGenTool` import/registration in `registry.ts` with `altimate_change` markers (upstream-shared file). New tests (4): - seed deps resolve via ref() not source() - snapshot deps resolve via ref() not source() - long model names preserve scenario suffix (no truncation collision) - division in string literals does not trigger boundary scenario Full suite: 2676 pass, 0 fail. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: [AI-673] address follow-up PR #674 review comments Follow-up fixes for additional CodeRabbit comments on PR #674. Performance: - `parseManifest()` now uses `loadRawManifest()` from helpers.ts, so it goes through the path+mtime cache. Previously, `generateDbtUnitTests` called `parseManifest()` (which parsed the manifest directly) and then `dbtLineage()` (which went through the cache) — meaning large 128MB manifests were still parsed twice on the first call. Now both paths hit the same cache; second call is a no-op. Type correctness: - Add `database?: string` to `SchemaInspectParams` interface. All call sites in `unit-tests.ts` pass it for cross-database support, but the interface didn't declare it. Wiring through to individual drivers is a separate refactor (the `Connector.describeTable` signature would need to change across all 10+ drivers); documented in the handler. Robustness: - `generateDbtUnitTests` now warns when any upstream dep in `model.depends_on` cannot be resolved by `resolveUpstream`. This catches future dbt resource types (e.g., `semantic_model.*`) and manifest inconsistencies that would otherwise silently drop deps from the generated `given` block. New test: verifies unresolvable deps (e.g., `semantic_model.*`) produce a warning instead of silently being dropped. Full suite: 2677 pass, 0 fail. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: [AI-673] document dbt_unit_test_gen tool and /dbt-unit-tests skill Update project docs to surface the new dbt unit test generation feature. - `docs/docs/data-engineering/tools/dbt-tools.md`: add `dbt_unit_test_gen` tool reference with parameters, scenario categories, dialect coverage, and example output. Add `/dbt-unit-tests` skill entry with workflow summary. - `docs/docs/data-engineering/tools/index.md`: bump dbt Tools row to "3 tools + 6 skills". - `docs/docs/configure/skills.md`: add `/dbt-unit-tests` to the skill reference table. - `packages/opencode/src/altimate/prompts/builder.txt`: add skill row to the dbt development table so the builder agent knows when to invoke `/dbt-unit-tests` vs `/dbt-test` (schema vs unit tests). - `CHANGELOG.md`: add Unreleased section documenting the feature, manifest parse cache, `description`/`adapter_type` additions. No functional changes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: [AI-673] address 3rd-round PR review + fix flaky CI test PR #674 follow-up review comments + CI failure on main branch. Review fixes: - **Division detection regex broadened** — previously `\b\w+\s*/\s*\w+` only caught bare identifiers like `a / b`. Now matches common SQL patterns: `SUM(amount) / COUNT(*)`, `CAST(a AS INT) / CAST(b AS INT)`, `COALESCE(x, 0) / COALESCE(y, 1)`, `a.col / b.col`, and parenthesized expressions. Still excludes `/*` (block comment) and `//`. - **Incremental scenario preserved across max_scenarios cap** — `buildTests()` sliced `scenarios` before processing, so SQL with enough non-incremental triggers (JOIN + CASE + division + happy_path) would push the incremental test out of the default `max_scenarios = 3` window, dropping the `input: this` mock entirely. Now the incremental scenario is always kept in the capped window (replaces the last non-happy-path scenario if missing). - **Ephemeral SQL column aliases now quoted** — the generated ephemeral `format: sql` block used raw identifiers (`AS order_id`), which breaks for reserved keywords (`select`, `order`, `group`), mixed case, or special characters. Now uses ANSI-standard double-quoted identifiers (`AS "order_id"`) via a new `quoteIdent()` helper that also escapes embedded double quotes. - **Tests use `await using tmpdir()` per-test** — replaced suite-level `beforeEach`/`afterEach` hooks and shared `tmp`/`manifestCounter` globals with per-test `await using tmp = await tmpdir()`. Each test now owns its own disposable tmpdir with automatic cleanup when the scope ends. Matches project coding guideline: "Always use `await using` syntax with `tmpdir()` for automatic cleanup when the variable goes out of scope." CI fix: - **Flaky `oauth-browser.test.ts`** — the "BrowserOpenFailed event is NOT published when open() succeeds" test used a fixed 600ms sleep that wasn't enough on slow CI runners to let the authenticate() flow reach the mocked `open()` call. Now polls for `openCalledWith` to be defined (5s budget) before the 500ms detection window assertion. Test passes locally in 2s, no more flaky CI failures. Full suite: 2677 pass, 0 fail. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: [AI-673] fix dbt-tools.md arithmetic error + add code fence labels PR #674 review fixes for dbt-tools.md documentation. - **Fix arithmetic error in example output** — the expected `order_total` for `order_id: 1` was `100`, but with `quantity: 3` and `unit_price: 100` the correct value is `3 × 100 = 300`. Flagged by both CodeRabbit (Critical) and cubic (P2). A wrong expected result in docs would mislead users about what the tool generates. - **Add `text` language specifier to two unlabeled code fences** — tool invocation example (line 79) and skill invocation example (line 174). Satisfies markdownlint MD040. No code changes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 02e2f66 commit 777a37b

File tree

22 files changed

+2496
-121
lines changed

22 files changed

+2496
-121
lines changed

.opencode/skills/dbt-test/SKILL.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,8 @@ altimate-dbt build --model <name> # build + test together
8181

8282
## Unit Test Workflow
8383

84+
**For automated unit test generation, use the `dbt-unit-tests` skill instead.** It analyzes model SQL, generates type-correct mock data, and assembles complete YAML automatically.
85+
8486
See [references/unit-test-guide.md](references/unit-test-guide.md) for the full unit test framework.
8587

8688
### Quick Pattern
Lines changed: 209 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,209 @@
1+
---
2+
name: dbt-unit-tests
3+
description: Generate dbt unit tests automatically for any model. Analyzes SQL logic (CASE/WHEN, JOINs, window functions, NULLs), creates type-correct mock inputs from manifest schema, and assembles complete YAML. Use when a user says "generate tests", "add unit tests", "test this model", or "test coverage" for dbt models.
4+
---
5+
6+
# dbt Unit Test Generation
7+
8+
## Requirements
9+
**Agent:** builder or migrator (requires file write access)
10+
**Tools used:** dbt_unit_test_gen, dbt_manifest, dbt_lineage, altimate_core_validate, altimate_core_testgen, bash (runs `altimate-dbt` commands), read, glob, write, edit
11+
12+
## When to Use This Skill
13+
14+
**Use when the user wants to:**
15+
- Generate unit tests for a dbt model
16+
- Add test coverage to an existing model
17+
- Create mock data for testing
18+
- Test-driven development (TDD) for dbt
19+
- Verify CASE/WHEN logic, NULL handling, JOIN behavior, or aggregation correctness
20+
- Test incremental model logic
21+
22+
**Do NOT use for:**
23+
- Adding schema tests (not_null, unique, accepted_values) -> use `dbt-test`
24+
- Creating or modifying model SQL -> use `dbt-develop`
25+
- Writing descriptions -> use `dbt-docs`
26+
- Debugging build failures -> use `dbt-troubleshoot`
27+
28+
## The Iron Rules
29+
30+
1. **Never guess expected outputs.** Compute them by running SQL against mock data when possible. If you cannot run SQL, clearly mark expected outputs as placeholders that need verification.
31+
2. **Never skip upstream dependencies.** Every ref() and source() the model touches MUST have a mock input. Miss one and the test won't compile.
32+
3. **Use sql format for ephemeral models.** Dict format fails silently for ephemeral upstreams.
33+
4. **Never weaken a test to make it pass.** If the test fails, the model logic may be wrong. Investigate before changing expected values.
34+
5. **Compile before committing.** Always run `altimate-dbt test --model <name>` to verify tests compile and execute.
35+
36+
## Core Workflow: Analyze -> Generate -> Refine -> Validate -> Write
37+
38+
### Phase 1: Analyze the Model
39+
40+
Before generating any tests, deeply understand the model:
41+
42+
```bash
43+
# 1. Ensure manifest is compiled
44+
altimate-dbt compile --model <name>
45+
46+
# 2. Read the model SQL
47+
read <model_sql_file>
48+
49+
# 3. Parse the manifest for dependencies
50+
dbt_unit_test_gen(manifest_path: "target/manifest.json", model: "<name>")
51+
```
52+
53+
**What to look for:**
54+
- Which upstream refs/sources does this model depend on?
55+
- What SQL constructs need testing? (CASE/WHEN, JOINs, window functions, aggregations)
56+
- What edge cases exist? (NULLs, empty strings, zero values, boundary dates)
57+
- Is this an incremental model? (needs `is_incremental` override tests)
58+
- Are any upstream models ephemeral? (need sql format)
59+
60+
### Phase 2: Generate Tests
61+
62+
The `dbt_unit_test_gen` tool does the heavy lifting:
63+
64+
```text
65+
dbt_unit_test_gen(
66+
manifest_path: "target/manifest.json",
67+
model: "fct_orders",
68+
max_scenarios: 5
69+
)
70+
```
71+
72+
This returns:
73+
- Complete YAML with mock inputs and expected outputs
74+
- Semantic context: model/column descriptions, column lineage, compiled SQL
75+
- List of anti-patterns that informed edge case generation
76+
- Warnings about ephemeral deps, missing columns, etc.
77+
78+
**If the tool reports missing columns** (placeholder rows in the YAML), discover them:
79+
```bash
80+
altimate-dbt columns --model <upstream_model_name>
81+
altimate-dbt columns-source --source <source_name> --table <table_name>
82+
```
83+
Then update the generated YAML with real column names.
84+
85+
### Phase 3: Refine Expected Outputs
86+
87+
**This is the critical step that differentiates good tests from bad ones.**
88+
89+
The tool generates placeholder expected outputs based on column types. You MUST refine them:
90+
91+
**Option A: Compute by running SQL (preferred)**
92+
```bash
93+
# Run the model against mock data to get actual output
94+
altimate-dbt test --model <name>
95+
# If the test fails, the error shows actual vs expected — use actual as expected
96+
```
97+
98+
**Option B: Manual computation**
99+
Read the model SQL carefully and mentally execute it against the mock inputs.
100+
For each test case:
101+
1. Look at the mock input rows
102+
2. Trace through the SQL logic (CASE/WHEN branches, JOINs, aggregations)
103+
3. Write the correct expected output
104+
105+
**Option C: Use the warehouse (most accurate)**
106+
```bash
107+
# Build a CTE query with mock data and run the model SQL against it
108+
altimate-dbt execute --query "WITH mock_stg_orders AS (SELECT 1 AS order_id, 100.00 AS amount) SELECT * FROM (<model_sql>) sub"
109+
```
110+
111+
### Phase 4: Validate
112+
113+
```bash
114+
# 1. Run the unit tests
115+
altimate-dbt test --model <name>
116+
117+
# 2. If tests fail, read the error carefully
118+
# - Compilation error? Missing ref, wrong column name, type mismatch
119+
# - Assertion error? Expected output doesn't match actual
120+
121+
# 3. Fix and retry (max 3 iterations)
122+
```
123+
124+
### Phase 5: Write to File
125+
126+
Place unit tests in one of these locations (match project convention):
127+
- `models/<layer>/_unit_tests.yml` (dedicated file)
128+
- `models/<layer>/schema.yml` (append to existing)
129+
130+
```bash
131+
# Check existing convention
132+
glob models/**/*unit_test*.yml models/**/*schema*.yml
133+
134+
# Write or append
135+
edit <yaml_file> # if file exists
136+
write <yaml_file> # if creating new
137+
```
138+
139+
## Test Case Categories
140+
141+
### Happy Path (always generate)
142+
Standard inputs that exercise the main logic path. 2 rows minimum.
143+
144+
### NULL Handling
145+
Set nullable columns to NULL in the last row. Verify COALESCE/NVL/IFNULL behavior.
146+
147+
### Boundary Values
148+
Zero amounts, empty strings, epoch dates, MAX values. Tests robustness.
149+
150+
### Edge Cases
151+
- Division by zero (if model divides)
152+
- Non-matching JOINs (LEFT JOIN with no match)
153+
- Single-row aggregation
154+
- Duplicate key handling
155+
156+
### Incremental
157+
For incremental models only. Use `overrides.macros.is_incremental: true` to test the incremental path.
158+
159+
## Common Mistakes
160+
161+
| Mistake | Fix |
162+
|---------|-----|
163+
| Missing a ref() in given | Parse manifest for ALL depends_on nodes |
164+
| Wrong column names in mock data | Use manifest columns, not guesses |
165+
| Wrong data types | Use schema catalog types |
166+
| Expected output is just mock input | Actually compute the transformation |
167+
| Dict format for ephemeral model | Use `format: sql` with raw SQL |
168+
| Not testing NULL path in COALESCE | Add null_handling test case |
169+
| Hardcoded dates with current_timestamp | Use overrides.macros to mock timestamps |
170+
| Testing trivial pass-through | Skip models with no logic |
171+
172+
## YAML Format Reference
173+
174+
```yaml
175+
unit_tests:
176+
- name: test_<model>_<scenario>
177+
description: "What this test verifies"
178+
model: <model_name>
179+
overrides: # optional
180+
macros:
181+
is_incremental: true # for incremental models
182+
vars:
183+
run_date: "2024-01-15" # for date-dependent logic
184+
given:
185+
- input: ref('upstream_model')
186+
rows:
187+
- { col1: value1, col2: value2 }
188+
- input: source('source_name', 'table_name')
189+
rows:
190+
- { col1: value1 }
191+
- input: ref('ephemeral_model')
192+
format: sql
193+
rows: |
194+
SELECT 1 AS id, 'test' AS name
195+
UNION ALL
196+
SELECT 2 AS id, 'other' AS name
197+
expect:
198+
rows:
199+
- { output_col1: expected1, output_col2: expected2 }
200+
```
201+
202+
## Reference Guides
203+
204+
| Guide | Use When |
205+
|-------|----------|
206+
| [references/unit-test-yaml-spec.md](references/unit-test-yaml-spec.md) | Full YAML specification and format details |
207+
| [references/edge-case-patterns.md](references/edge-case-patterns.md) | Catalog of edge cases by SQL construct |
208+
| [references/incremental-testing.md](references/incremental-testing.md) | Testing incremental models |
209+
| [references/altimate-dbt-commands.md](references/altimate-dbt-commands.md) | Full CLI reference |
Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
# altimate-dbt Command Reference
2+
3+
All dbt operations use the `altimate-dbt` CLI. Output is JSON to stdout; logs go to stderr.
4+
5+
```bash
6+
altimate-dbt <command> [args...]
7+
altimate-dbt <command> [args...] --format text # Human-readable output
8+
```
9+
10+
## First-Time Setup
11+
12+
```bash
13+
altimate-dbt init # Auto-detect project root
14+
altimate-dbt init --project-root /path # Explicit root
15+
altimate-dbt init --python-path /path # Override Python
16+
altimate-dbt doctor # Verify setup
17+
altimate-dbt info # Project name, adapter, root
18+
```
19+
20+
## Build & Run
21+
22+
```bash
23+
altimate-dbt build # full project build (compile + run + test)
24+
altimate-dbt build --model <name> [--downstream] # build a single model
25+
altimate-dbt run --model <name> [--downstream] # materialize only
26+
altimate-dbt test --model <name> # run tests only
27+
```
28+
29+
## Compile
30+
31+
```bash
32+
altimate-dbt compile --model <name>
33+
altimate-dbt compile-query --query "SELECT * FROM {{ ref('stg_orders') }}" [--model <context>]
34+
```
35+
36+
## Execute SQL
37+
38+
```bash
39+
altimate-dbt execute --query "SELECT count(*) FROM {{ ref('orders') }}" --limit 100
40+
```
41+
42+
## Schema & DAG
43+
44+
```bash
45+
altimate-dbt columns --model <name> # column names and types
46+
altimate-dbt columns-source --source <src> --table <tbl> # source table columns
47+
altimate-dbt column-values --model <name> --column <col> # sample values
48+
altimate-dbt children --model <name> # downstream models
49+
altimate-dbt parents --model <name> # upstream models
50+
```
51+
52+
## Packages
53+
54+
```bash
55+
altimate-dbt deps # install packages.yml
56+
altimate-dbt add-packages --packages dbt-utils,dbt-expectations
57+
```
58+
59+
## Error Handling
60+
61+
All errors return JSON with `error` and `fix` fields:
62+
```json
63+
{ "error": "dbt-core is not installed", "fix": "Install it: python3 -m pip install dbt-core" }
64+
```
65+
66+
Run `altimate-dbt doctor` as the first diagnostic step for any failure.

0 commit comments

Comments
 (0)