Since 7711da2 the eval fns gate on apply_as_commit and record status=apply_fail instead of silently scoring a garbage HEAD~1..HEAD diff (previously these looked like "ok"). Sweep #64 interim data shows 215 such records.
Needed: sample ~20 of them from the run-64 checkpoints and classify —
- dataset noise (gold patch legitimately does not apply to base_commit, e.g. bad revision pin or pre-normalized patch) -> exclude those instance_ids upstream in
benchmarks/manifests/ and note in SPLIT_REPORT;
- strictness (
git apply --index rejects something a whitespace-tolerant apply would take) -> consider --whitespace=nowarn-class flags, NOT --3way (removed deliberately, see apply_as_commit docstring);
- mixed -> both.
Either way the denominator of every affected cell changes slightly; document the outcome in STATS_PLAN exclusions.
Since 7711da2 the eval fns gate on
apply_as_commitand record status=apply_fail instead of silently scoring a garbage HEAD~1..HEAD diff (previously these looked like "ok"). Sweep #64 interim data shows 215 such records.Needed: sample ~20 of them from the run-64 checkpoints and classify —
benchmarks/manifests/and note in SPLIT_REPORT;git apply --indexrejects something a whitespace-tolerant apply would take) -> consider--whitespace=nowarn-class flags, NOT--3way(removed deliberately, see apply_as_commit docstring);Either way the denominator of every affected cell changes slightly; document the outcome in STATS_PLAN exclusions.