Commit 777a37b
feat: automated dbt unit test generation (#674)
* feat: [AI-673] automated dbt unit test generation
Add `dbt_unit_test_gen` tool and `dbt-unit-tests` skill for generating
dbt unit tests from manifest + compiled SQL.
Engine (`unit-tests.ts`):
- Reuses `parseManifest()` for model info, deps, columns, descriptions
- Reuses `dbtLineage()` for compiled SQL + column lineage mapping
- Reuses `schema.inspect` for warehouse column enrichment
- Reuses `sql.optimize` for anti-pattern detection
- Keyword-based scenario detection (CASE, JOIN, GROUP BY, division, incremental)
- Type-correct mock data generation with null/boundary/happy-path variants
- Incremental tests include `input: this` mock for existing table state
- Ephemeral deps correctly use `format: sql` even with no known columns
- Cross-database support via `database` param in `schema.inspect`
- YAML assembly via `yaml` library (not string concatenation)
- Deterministic test names (index-based, not `Date.now()`)
- Rich `UnitTestContext` with descriptions + lineage for LLM refinement
Shared infrastructure:
- Extract `helpers.ts` from `lineage.ts` (shared: `loadRawManifest`,
`findModel`, `getUniqueId`, `detectDialect`, `buildSchemaContext`)
- Manifest cache by path+mtime (avoids re-reading 128MB files)
- Add `description` to `DbtModelInfo`/`DbtSourceInfo`
- Add `adapter_type` to `DbtManifestResult`
Skill (`dbt-unit-tests/SKILL.md`):
- 5-phase workflow: Analyze -> Generate -> Refine -> Validate -> Write
- Reference guides: YAML spec, edge-case patterns, incremental testing
Tests: 32 tests covering manifest parsing, scenario detection, YAML
round-trip validation, incremental `this` mock, ephemeral sql format,
deterministic naming, descriptions, lineage context, source deps.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: [AI-673] address PR #674 review comments (CodeRabbit + cubic)
Address review comments from CodeRabbit and cubic on PR #674.
Bug fixes:
- `resolveUpstream` now resolves seed.* and snapshot.* deps in addition
to models and sources. Previously silently dropped, causing `dbt test`
to fail for any model ref()ing a seed or snapshot.
- Test name truncation no longer cuts off scenario suffix. Budget
suffix length first, then truncate the model-name portion, so
scenario names like `_null_handling_2` are always preserved even
for long model names.
- `findModel`/`getUniqueId` in helpers.ts now validate
`resource_type === "model"` on the key lookup path, not just the
name fallback. Prevents returning non-model nodes by unique_id.
- Division detection regex now strips string literals AND comments
before matching, so `'2024/01/15'` no longer triggers a false-positive
boundary scenario.
Documentation fixes:
- `incremental-testing.md`: fix Jinja syntax — `{{ if is_incremental() }}`
is invalid; use `{% if is_incremental() %}` for control flow.
- `SKILL.md`: workflow header now shows all 5 phases (Analyze -> Generate
-> Refine -> Validate -> Write).
- `SKILL.md`: add language label to fenced code block for markdownlint.
- `unit-test-yaml-spec.md`: show both top-level `tags` and nested
`config.tags` forms explicitly.
Infrastructure:
- Add `seeds` and `snapshots` arrays to `DbtManifestResult` (previously
only counts were returned). `parseManifest` now extracts full seed
and snapshot info using the same shape as `DbtModelInfo`.
- Tests migrate to shared `tmpdir()` fixture from `test/fixture/fixture.ts`
for automatic cleanup (per project coding guidelines).
- Wrap `DbtUnitTestGenTool` import/registration in `registry.ts` with
`altimate_change` markers (upstream-shared file).
New tests (4):
- seed deps resolve via ref() not source()
- snapshot deps resolve via ref() not source()
- long model names preserve scenario suffix (no truncation collision)
- division in string literals does not trigger boundary scenario
Full suite: 2676 pass, 0 fail.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: [AI-673] address follow-up PR #674 review comments
Follow-up fixes for additional CodeRabbit comments on PR #674.
Performance:
- `parseManifest()` now uses `loadRawManifest()` from helpers.ts, so it
goes through the path+mtime cache. Previously, `generateDbtUnitTests`
called `parseManifest()` (which parsed the manifest directly) and
then `dbtLineage()` (which went through the cache) — meaning large
128MB manifests were still parsed twice on the first call. Now both
paths hit the same cache; second call is a no-op.
Type correctness:
- Add `database?: string` to `SchemaInspectParams` interface. All call
sites in `unit-tests.ts` pass it for cross-database support, but the
interface didn't declare it. Wiring through to individual drivers is
a separate refactor (the `Connector.describeTable` signature would
need to change across all 10+ drivers); documented in the handler.
Robustness:
- `generateDbtUnitTests` now warns when any upstream dep in
`model.depends_on` cannot be resolved by `resolveUpstream`. This
catches future dbt resource types (e.g., `semantic_model.*`) and
manifest inconsistencies that would otherwise silently drop deps
from the generated `given` block.
New test: verifies unresolvable deps (e.g., `semantic_model.*`) produce
a warning instead of silently being dropped.
Full suite: 2677 pass, 0 fail.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs: [AI-673] document dbt_unit_test_gen tool and /dbt-unit-tests skill
Update project docs to surface the new dbt unit test generation feature.
- `docs/docs/data-engineering/tools/dbt-tools.md`: add `dbt_unit_test_gen`
tool reference with parameters, scenario categories, dialect coverage,
and example output. Add `/dbt-unit-tests` skill entry with workflow
summary.
- `docs/docs/data-engineering/tools/index.md`: bump dbt Tools row to
"3 tools + 6 skills".
- `docs/docs/configure/skills.md`: add `/dbt-unit-tests` to the skill
reference table.
- `packages/opencode/src/altimate/prompts/builder.txt`: add skill row
to the dbt development table so the builder agent knows when to
invoke `/dbt-unit-tests` vs `/dbt-test` (schema vs unit tests).
- `CHANGELOG.md`: add Unreleased section documenting the feature,
manifest parse cache, `description`/`adapter_type` additions.
No functional changes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: [AI-673] address 3rd-round PR review + fix flaky CI test
PR #674 follow-up review comments + CI failure on main branch.
Review fixes:
- **Division detection regex broadened** — previously `\b\w+\s*/\s*\w+`
only caught bare identifiers like `a / b`. Now matches common SQL
patterns: `SUM(amount) / COUNT(*)`, `CAST(a AS INT) / CAST(b AS INT)`,
`COALESCE(x, 0) / COALESCE(y, 1)`, `a.col / b.col`, and parenthesized
expressions. Still excludes `/*` (block comment) and `//`.
- **Incremental scenario preserved across max_scenarios cap** —
`buildTests()` sliced `scenarios` before processing, so SQL with enough
non-incremental triggers (JOIN + CASE + division + happy_path) would
push the incremental test out of the default `max_scenarios = 3`
window, dropping the `input: this` mock entirely. Now the incremental
scenario is always kept in the capped window (replaces the last
non-happy-path scenario if missing).
- **Ephemeral SQL column aliases now quoted** — the generated ephemeral
`format: sql` block used raw identifiers (`AS order_id`), which breaks
for reserved keywords (`select`, `order`, `group`), mixed case, or
special characters. Now uses ANSI-standard double-quoted identifiers
(`AS "order_id"`) via a new `quoteIdent()` helper that also escapes
embedded double quotes.
- **Tests use `await using tmpdir()` per-test** — replaced suite-level
`beforeEach`/`afterEach` hooks and shared `tmp`/`manifestCounter`
globals with per-test `await using tmp = await tmpdir()`. Each test
now owns its own disposable tmpdir with automatic cleanup when the
scope ends. Matches project coding guideline: "Always use `await
using` syntax with `tmpdir()` for automatic cleanup when the variable
goes out of scope."
CI fix:
- **Flaky `oauth-browser.test.ts`** — the "BrowserOpenFailed event is
NOT published when open() succeeds" test used a fixed 600ms sleep
that wasn't enough on slow CI runners to let the authenticate() flow
reach the mocked `open()` call. Now polls for `openCalledWith` to be
defined (5s budget) before the 500ms detection window assertion.
Test passes locally in 2s, no more flaky CI failures.
Full suite: 2677 pass, 0 fail.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs: [AI-673] fix dbt-tools.md arithmetic error + add code fence labels
PR #674 review fixes for dbt-tools.md documentation.
- **Fix arithmetic error in example output** — the expected
`order_total` for `order_id: 1` was `100`, but with `quantity: 3` and
`unit_price: 100` the correct value is `3 × 100 = 300`. Flagged by
both CodeRabbit (Critical) and cubic (P2). A wrong expected result
in docs would mislead users about what the tool generates.
- **Add `text` language specifier to two unlabeled code fences** —
tool invocation example (line 79) and skill invocation example
(line 174). Satisfies markdownlint MD040.
No code changes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>1 parent 02e2f66 commit 777a37b
File tree
22 files changed
+2496
-121
lines changed- .opencode/skills
- dbt-test
- dbt-unit-tests
- references
- docs/docs
- configure
- data-engineering/tools
- packages/opencode
- src
- altimate
- native
- connections
- dbt
- prompts
- tools
- tool
- test
- altimate
- mcp
22 files changed
+2496
-121
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
81 | 81 | | |
82 | 82 | | |
83 | 83 | | |
| 84 | + | |
| 85 | + | |
84 | 86 | | |
85 | 87 | | |
86 | 88 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
Lines changed: 66 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
0 commit comments