spider2-dbt — duckdb_match verifier emitting binary reward.json#15

Merged

kentwelcome merged 26 commits into

mainfrom

spacedock-ensign/spider2-dbt-duckdb-match-verifier

Jun 18, 2026

Commits on Jun 18, 2026

feat(spider2): eval-spec + duckdb_match comparator (AC-1, AC-2)

kentwelcome
and
claude
committed
feat(spider2): verify.py CLI + materialize verifier assets/test.sh (AC-3)

kentwelcome
and
claude
committed
test(spider2): end-to-end emitted verifier writes harbor reward.json (AC-3)

kentwelcome
and
claude
committed
docs(spider2): implementation stage report

kentwelcome
and
claude
committed
docs(spider2): validation report — REJECTED (translate leakage regression + duckdb_match faithfulness)

kentwelcome
and
claude
committed
feedback: validation gate rejected (cycle 1) — duckdb_match faithfulness + leakage regression
kentwelcome
committed
fix(spider2): B2 — faithful duckdb_match (column-containment + 1e-2) + real eval-spec schema

kentwelcome
and
claude
committed
fix(spider2): B1 — scope leakage scan to agent-visible portion (exclude verify-time tests/)

kentwelcome
and
claude
committed
docs(spider2): implementation stage report (feedback cycle 1)

kentwelcome
and
claude
committed
docs(spider2): validation re-review (cycle 2) — PASSED (B2 faithful, B1 scoping sound)

kentwelcome
and
claude
committed
feedback: validation gate rejected (cycle 2) — fail-closed eval-spec validation
kentwelcome
committed
fix(spider2-dbt): fail closed on empty/malformed gold eval spec

kentwelcome
and
claude
committed
docs(spider2-dbt): cycle-2 implementation stage report (fail-closed)

kentwelcome
and
claude
committed
docs(spider2): validation re-review (cycle 3) — PASSED (fail-closed real + load-bearing)

kentwelcome
and
claude
committed
feedback: validation gate rejected (cycle 3) — honor eval-spec gold basename
kentwelcome
committed
fix(spider2-dbt): drive gold DB basename from evaluation.parameters.gold

kentwelcome
and
claude
committed
docs: cycle-3 implementation stage report (gold basename fix)

kentwelcome
and
claude
committed
docs(spider2): validation re-review (cycle-3 fix) — PASSED (gold basename real + load-bearing)

kentwelcome
and
claude
committed
fix(spider2-dbt): reject non-basename gold in load_eval_spec (path-traversal guard)

kentwelcome
and
claude
committed
fix(spider2-dbt): allowlist gold basename to block test.sh shell injection

kentwelcome
and
claude
committed
fix(spider2-dbt): shlex.quote both verifier args (predicted-db injection)

kentwelcome
and
claude
committed
fix(spider2-dbt): drop pandas from verifier; re-port comparator on duckdb-native

kentwelcome
and
claude
committed
fix(spider2-dbt): quote spec-supplied table names to block SQL injection

kentwelcome
and
claude
committed
fix(spider2-dbt): DECIMAL tolerance regression + fail closed on missing gold

kentwelcome
and
claude
committed
fix(spider2-dbt): guard verifier-asset copies against symlink write-through

kentwelcome
and
claude
committed
docs(spider2): final validation (cycles 1-10) — PASSED

kentwelcome
and
claude
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

spider2-dbt — duckdb_match verifier emitting binary reward.json#15

spider2-dbt — duckdb_match verifier emitting binary reward.json#15
kentwelcome merged 26 commits into
mainfrom
spacedock-ensign/spider2-dbt-duckdb-match-verifier

Commits on Jun 18, 2026

feat(spider2): eval-spec + duckdb_match comparator (AC-1, AC-2)

feat(spider2): verify.py CLI + materialize verifier assets/test.sh (AC-3)

test(spider2): end-to-end emitted verifier writes harbor reward.json (AC-3)

docs(spider2): implementation stage report

docs(spider2): validation report — REJECTED (translate leakage regression + duckdb_match faithfulness)

feedback: validation gate rejected (cycle 1) — duckdb_match faithfulness + leakage regression

fix(spider2): B2 — faithful duckdb_match (column-containment + 1e-2) + real eval-spec schema

fix(spider2): B1 — scope leakage scan to agent-visible portion (exclude verify-time tests/)

docs(spider2): implementation stage report (feedback cycle 1)

docs(spider2): validation re-review (cycle 2) — PASSED (B2 faithful, B1 scoping sound)

feedback: validation gate rejected (cycle 2) — fail-closed eval-spec validation

fix(spider2-dbt): fail closed on empty/malformed gold eval spec

docs(spider2-dbt): cycle-2 implementation stage report (fail-closed)

docs(spider2): validation re-review (cycle 3) — PASSED (fail-closed real + load-bearing)

feedback: validation gate rejected (cycle 3) — honor eval-spec gold basename

fix(spider2-dbt): drive gold DB basename from evaluation.parameters.gold

docs: cycle-3 implementation stage report (gold basename fix)

docs(spider2): validation re-review (cycle-3 fix) — PASSED (gold basename real + load-bearing)

fix(spider2-dbt): reject non-basename gold in load_eval_spec (path-traversal guard)

fix(spider2-dbt): allowlist gold basename to block test.sh shell injection

fix(spider2-dbt): shlex.quote both verifier args (predicted-db injection)

fix(spider2-dbt): drop pandas from verifier; re-port comparator on duckdb-native

fix(spider2-dbt): quote spec-supplied table names to block SQL injection

fix(spider2-dbt): DECIMAL tolerance regression + fail closed on missing gold

fix(spider2-dbt): guard verifier-asset copies against symlink write-through

docs(spider2): final validation (cycles 1-10) — PASSED