Skip to content

validate.md

Latest commit

 

History

History
60 lines (46 loc) · 2.67 KB

validate.md

File metadata and controls

60 lines (46 loc) · 2.67 KB
Raw

Validate

scripts/validate.py is the convention-driven numerical validation orchestrator. It runs the three stages of the accuracy loop — reference dump, C++ dump, compare — for a family, with every path derived from convention.

What it does

  validate.py ref     → runs dump_reference_<family>_*.py → writes build/validate/<family>/<variant>/<case>/ref/
  validate.py cpp     → runs transcribe-cli with TRANSCRIBE_DUMP_DIR → writes build/validate/.../cpp/
  validate.py compare → runs compare_tensors.py with tests/tolerances/<family>.json
  validate.py all     → ref + cpp + compare in sequence

One required input: --family <name>. Everything else is discovered:

Input Path
Manifest tests/golden/<family>/*.manifest.json
Dump script scripts/dump_reference_<family>_<reference>.py
Python env scripts/envs/<family>/
Tolerances tests/tolerances/<family>.json
Audio samples/jfk.wav (manifest can override)
GGUF models/<family>/ (first *.gguf, preferring *.bf16 / *.f32)
Reference output build/validate/<family>/<variant>/<case>/ref/
C++ output build/validate/<family>/<variant>/<case>/cpp/

CLI

uv run scripts/validate.py ref     --family cohere
uv run scripts/validate.py cpp     --family cohere
uv run scripts/validate.py compare --family cohere
uv run scripts/validate.py all     --family cohere

# Override model or GGUF
uv run scripts/validate.py all --family cohere --model /path/to/checkpoint
uv run scripts/validate.py cpp --family cohere --gguf models/cohere-transcribe-03-2026/cohere-transcribe-03-2026-BF16.gguf

Exit codes

  • 0 — all contract tensors within tolerance.
  • 1 — one or more contract tensors out of tolerance, missing, or shape-mismatched.

Exit-code driven, no interactive prompts (per project policy).

When to run

  • After any loader change in src/arch/<family>/weights.cpp.
  • After any graph change in src/arch/<family>/encoder.cpp or decoder.cpp.
  • After regenerating a GGUF with a new converter.
  • As the dev gate before running WER (which is slower).

WER is the user-facing gate. validate is the per-tensor dev gate that catches numerical regressions in seconds.