test: enforce spec coverage and add CI

- tests/spec_manifest.yml + tests/test_spec_coverage.py: a working manifest of the spec IDs that must have a case, enforced per entity so missing, orphan or misfiled cases fail loudly. Two hub IDs are tracked under partial_coverage.

- .github/workflows/tests.yml: run pip install -e .[test] + python -m pytest on Python 3.11 and 3.13.

- findings/testing_framework_status.md: framework overview, scope and known limitations.

Author: polat-deniz

.github/workflows/tests.yml

@@ -0,0 +1,35 @@
+name: tests
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+permissions:
+  contents: read
+
+jobs:
+  pytest:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.11", "3.13"]
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Install package + test deps
+        run: |
+          python -m pip install --upgrade pip
+          python -m pip install -e ".[test]"
+
+      - name: Run test suite
+        # `python -m` puts the repo root on sys.path; pytest's prepend mode puts
+        # tests/ on the path so the sibling `cases` package resolves.
+        run: python -m pytest -q

findings/testing_framework_status.md

@@ -0,0 +1,66 @@
+# datavault4sqlglot — Execution Testing Framework
+
+A data-driven test framework that executes each generator's SQL against an in-memory
+DuckDB and asserts on the resulting rows. A *case* is a YAML file carrying input data,
+the expected output and the generator wiring; the mechanics (seed → `to_sql()` → run →
+compare) live once in the runner, so a new case only adds data.
+
+## Layout
+
+- `tests/cases/case_schema.py` — pydantic schema + per-entity validation
+- `tests/cases/case_loader.py` — discovery/validation of `tests/cases/**/*.yml`
+- `tests/cases/dv_case_runner.py` — the engine (dispatch → seed → run → compare)
+- `tests/cases/<entity>/*.yml` — the cases
+- `tests/test_execution_cases.py` — one parametrized test per case (pytest id = spec id)
+- `tests/spec_manifest.yml` + `tests/test_spec_coverage.py` — the coverage gate
+- `tests/test_runner_internals.py` — unit tests for the comparison logic
+
+## Coverage (48 cases)
+
+| Entity | Cases | Source of the spec IDs |
+|---|---|---|
+| hub | 11 | Excel `TestCases` sheet |
+| link | 10 | hand-written Python oracle (the Excel has no link cases) |
+| satellite v0 | 24 | Excel `TestCases` sheet |
+| satellite v1 | 3 | derived from the datavault4dbt `sat_v1` macro semantics |
+
+The coverage gate enforces, per entity, that every manifest spec-ID has exactly one
+case (and vice-versa), so missing/orphan/misfiled cases fail loudly. Two hub IDs
+(`2.2.1.3`, `2.2.2.4`) are listed under `partial_coverage` — see Known limitations.
+
+## Scope
+
+- **In scope:** the execution behaviour of the loading generators — Hub, Link,
+  Satellite v0 and v1. The hand-written `test_execution_*.py` are kept as an
+  independent differential cross-check.
+- **Out of scope** (covered by other, existing tests): structural/AST checks, dialect
+  transpilation, configuration, hashing, and the **Stage** generator (which computes
+  hashes — a different mechanism).
+
+## Comparison modes
+
+`set` (default), `exact` (multiset), `subset`, `count`, `empty` — projected onto
+`key_columns`. On failure the runner reports the differing rows (count-aware for
+`exact`). A key column that exists in neither the expected rows nor the actual result
+fails loudly rather than passing silently.
+
+## Known limitations
+
+- **Hub same-LDTS tie-break:** the multi-source dedup orders only by `ldts` with no
+  secondary key, so on equal `ldts` across sources the surviving record source is not
+  deterministically defined. Cases `2.2.1.3` / `2.2.2.4` therefore assert only the
+  deterministic single-row dedup (`partial_coverage`).
+- **All-VARCHAR seeding:** values and row sets are verified; column data *types* are not.
+- Two `xfailed` tests in `tests/test_edge_cases.py` document pre-existing generator
+  limitations (Hub does not validate a non-empty business-key list; Stage can emit
+  `SELECT FROM <table>`); they are unrelated to this framework.
+
+## Run
+
+```bash
+python -m pip install -e ".[test]"
+python -m pytest -q                                   # full suite
+python -m pytest tests/test_execution_cases.py -v     # framework only (48 cases)
+python -m pytest tests/test_spec_coverage.py -v       # coverage gate
+python -m pytest "tests/test_execution_cases.py::test_execution_case[2.2.2.3]"   # one case
+```

tests/spec_manifest.yml

@@ -0,0 +1,88 @@
+# Spec-coverage manifest — the in-repo list of which spec-IDs must have an executable
+# YAML case. It follows the 'TestCases' worksheet as its source of truth (see
+# provenance below). `test_spec_coverage` fails on any drift between this list and
+# tests/cases/**/*.yml (missing case, orphan case, or a case filed under the wrong
+# entity), so coverage gaps can never pass silently — an enforced spec<->case
+# reconciliation.
+#
+# Lives in tests/ (not tests/cases/) on purpose: the case loader globs
+# tests/cases/**/*.yml and would otherwise try to parse this file as a Case.
+#
+# Provenance of the IDs:
+#   hub (2.x), satellite (4.x): leaf IDs from the 'TestCases' worksheet of the
+#                               'datavault4sqlglot test cases' workbook (the source
+#                               of truth this manifest follows). The older 'Test Cases' worksheet
+#                               (with a space) is superseded.
+#   link (3.x):                 from the hand-written Python oracle
+#                               (test_execution_link.py); the workbook has no link
+#                               cases yet (adding them to the spec is a future option).
+#   satellite_v1:               execution cases for the end-dated sat; no spec-IDs exist
+#                               in the workbook yet, derived from the datavault4dbt
+#                               sat_v1 macro semantics and tagged v1.*.
+
+hub:
+  - "2.1.1.1"
+  - "2.1.2.1"
+  - "2.1.2.2"
+  - "2.1.2.3"
+  - "2.2.1.1"
+  - "2.2.1.2"
+  - "2.2.1.3"
+  - "2.2.2.1"
+  - "2.2.2.2"
+  - "2.2.2.3"
+  - "2.2.2.4"
+
+link:
+  - "3.1.1.1"
+  - "3.1.2.1"
+  - "3.1.2.2"
+  - "3.1.2.3"
+  - "3.2.1.1"
+  - "3.2.1.2"
+  - "3.2.2.1"
+  - "3.2.2.2"
+  - "3.2.2.3"
+  - "3.2.3.1"
+
+satellite:
+  - "4.1.1.1.1"
+  - "4.1.1.2.1"
+  - "4.1.1.2.2"
+  - "4.1.1.2.3"
+  - "4.1.2.1.1"
+  - "4.1.2.1.2"
+  - "4.1.2.2.1"
+  - "4.1.2.2.2"
+  - "4.1.2.2.3"
+  - "4.1.2.2.4"
+  - "4.1.2.2.5"
+  - "4.1.2.2.6"
+  - "4.2.1.1.1"
+  - "4.2.1.2.1"
+  - "4.2.1.2.2"
+  - "4.2.1.2.3"
+  - "4.2.2.1.1"
+  - "4.2.2.1.2"
+  - "4.2.2.2.1"
+  - "4.2.2.2.2"
+  - "4.2.2.2.3"
+  - "4.2.2.2.4"
+  - "4.2.2.2.5"
+  - "4.2.2.2.6"
+
+satellite_v1:
+  - "v1.1.1"
+  - "v1.1.2"
+  - "v1.1.3"
+
+# Spec-IDs whose case EXISTS but only PARTIALLY asserts the spec, with the reason.
+# They still count as covered above (a case exists), and test_spec_coverage also
+# checks each ID here is a real case so the marker can't drift. This is the honest
+# counterpart to the coverage list: the limitation is a documented generator gap,
+# flagged on purpose — not a silent partial pass.
+partial_coverage:
+  "2.2.1.3": "Asserts only the deterministic single-row dedup; the spec's 'leading
+    source wins' tie-break is NOT verified. hub.py dedup orders by ldts with no
+    secondary key, so the surviving source is nondeterministic on ties (generator gap)."
+  "2.2.2.4": "Same generator gap as 2.2.1.3 (incremental variant)."

tests/test_spec_coverage.py

@@ -0,0 +1,67 @@
+"""
+Forced spec<->case coverage check.
+
+This test guarantees every spec-ID has an executable case: it loads the manifest
+(``spec_manifest.yml``) and the
+actual YAML cases, then asserts they match exactly *per entity*. A spec-ID without
+a case, a case without a manifest entry, or a case filed under the wrong entity all
+fail here — so coverage drift can never pass silently.
+
+Run with:  python -m pytest tests/test_spec_coverage.py -v
+"""
+from __future__ import annotations
+
+from collections import defaultdict
+from pathlib import Path
+
+import yaml
+
+from cases.case_loader import load_all_cases
+
+_MANIFEST = Path(__file__).parent / "spec_manifest.yml"
+
+
+# Top-level manifest keys that are NOT entity -> id lists.
+_NON_ENTITY_KEYS = {"partial_coverage"}
+
+
+def _load_manifest() -> dict:
+    return yaml.safe_load(_MANIFEST.read_text(encoding="utf-8"))
+
+
+def _manifest_ids(data: dict) -> dict[str, set[str]]:
+    return {k: set(v or []) for k, v in data.items() if k not in _NON_ENTITY_KEYS}
+
+
+def _case_ids() -> dict[str, set[str]]:
+    by_entity: dict[str, set[str]] = defaultdict(set)
+    for case in load_all_cases():
+        by_entity[case.entity.value].add(case.id)
+    return dict(by_entity)
+
+
+def test_spec_coverage_matches_manifest():
+    manifest = _manifest_ids(_load_manifest())
+    cases = _case_ids()
+
+    problems: list[str] = []
+    for entity in sorted(set(manifest) | set(cases)):
+        want = manifest.get(entity, set())
+        have = cases.get(entity, set())
+        missing = sorted(want - have)   # spec-ID with no case
+        orphan = sorted(have - want)    # case with no manifest entry
+        if missing:
+            problems.append(f"[{entity}] spec-IDs without a case: {missing}")
+        if orphan:
+            problems.append(f"[{entity}] cases without a manifest entry: {orphan}")
+
+    assert not problems, "Spec coverage drift:\n" + "\n".join(problems)
+
+
+def test_partial_coverage_entries_reference_real_cases():
+    # partial_coverage documents IDs whose case only PARTIALLY asserts the spec.
+    # Guard against drift: every ID listed there must correspond to a real case.
+    partial = _load_manifest().get("partial_coverage") or {}
+    case_ids = {case.id for case in load_all_cases()}
+    unknown = sorted(set(partial) - case_ids)
+    assert not unknown, f"partial_coverage lists IDs with no case: {unknown}"