Skip to content

Commit 7621ca6

Browse files
hyperpolymathclaude
andcommitted
feat(corpus): ingest echidnabot feedback deltas in merge_corpus.jl (7b-3)
Closes the schema-bridge half of the 7b-3 coordination ticket recorded in echidnabot/STATE.a2ml and echidnabot/docs/MCP-TEMPORARY-EXCEPTION.adoc. Echidnabot-mcp's CorpusDelta writer emits successful-proof rows to training_data/proof_states_echidnabot_<date>.jsonl in echidna's existing schema. Until now, merge_corpus.jl only iterated the static PER_PROVER_FILES whitelist, so those date-stamped feedback files never made it into proof_states_UNIFIED.jsonl during `just corpus-refresh`. Add a small glob-based ingest loop after the PER_PROVER_FILES loop that picks up `proof_states_echidnabot_*.jsonl`, logs counts, and appends to all_entries so dedup + balancing handle them identically. Glob over static-list addition because filenames are date-stamped; new days drop in automatically. Verified: `julia -e 'include(...)'` → "syntax OK". Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 365dde2 commit 7621ca6

1 file changed

Lines changed: 24 additions & 0 deletions

File tree

scripts/merge_corpus.jl

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -322,6 +322,30 @@ function main()::Int
322322
append!(all_entries, entries)
323323
end
324324

325+
# ------------------------------------------------------------------
326+
# 1b. Ingest echidnabot feedback corpus (7b-3 bridge).
327+
# Files are written by echidnabot-mcp's CorpusDelta on successful
328+
# proofs and follow proof_states_echidnabot_<date>.jsonl naming.
329+
# Adding by glob (rather than fixing one filename in the static
330+
# PER_PROVER_FILES list) lets new days drop into corpus-refresh
331+
# automatically without future Cargo-style edits.
332+
# ------------------------------------------------------------------
333+
if isdir(TRAINING_DIR)
334+
feedback_files = filter(
335+
f -> startswith(f, "proof_states_echidnabot_") && endswith(f, ".jsonl"),
336+
readdir(TRAINING_DIR; sort=false),
337+
)
338+
for fname in feedback_files
339+
fpath = joinpath(TRAINING_DIR, fname)
340+
entries = load_jsonl(fpath)
341+
file_counts[fname] = length(entries)
342+
if !isempty(entries)
343+
println(" Loaded $(lpad(string(length(entries)), 7)) feedback proofs from $fname")
344+
end
345+
append!(all_entries, entries)
346+
end
347+
end
348+
325349
println("\nTotal raw entries loaded: $(length(all_entries))")
326350

327351
# ------------------------------------------------------------------

0 commit comments

Comments
 (0)