Skip to content

Commit 89ae91f

Browse files
committed
chore(justfile): add eval recipe for held-out MRR measurement
Adjacent to train / train-cpu. Requires models/neural/ to exist (from a prior `just train-cpu` or `just train` run). Appends eval_results.jsonl and exits non-zero if MRR < 0.66 cosine baseline. https://claude.ai/code/session_01YPqu7gti4azBach6ZvpRFJ
1 parent a8ae1d4 commit 89ae91f

1 file changed

Lines changed: 7 additions & 0 deletions

File tree

Justfile

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -349,6 +349,13 @@ train:
349349
train-cpu:
350350
ECHIDNA_MAX_PROOF_STATES=2000 ECHIDNA_NUM_EPOCHS=2 julia --project=src/julia src/julia/run_training_cpu.jl
351351

352+
# Evaluate trained model on held-out validation split (MRR / top-k).
353+
# Requires `just train-cpu` (or `just train`) to have produced models/neural/.
354+
# Appends one JSONL row to training_data/eval_results.jsonl and exits non-zero
355+
# if MRR < 0.66 (below the cosine baseline).
356+
eval:
357+
julia --project=src/julia src/julia/eval_held_out.jl
358+
352359
# End-to-end pipeline: provision → extract → merge → align → retrain.
353360
# Use `ECHIDNA_MAX_PROOF_STATES=0 just corpus-refresh` to lift the sample cap.
354361
corpus-refresh: provision-corpora extract-corpora merge-corpora align-premises retrain

0 commit comments

Comments
 (0)