Fix leakage risk in GSO

[hiydavid]

This is a public repository. Do not include customer names, workspace URLs, access tokens, or any customer-identifiable information. If your bug involves a customer environment, report it in #genie-workbench on Slack instead.

Describe the bug

Where the leakage happens:

1. Split exists — benchmarks are split 85/15 train/held_out (config.py:146, benchmarks.py:928). Curated benchmarks are forced into train.
2. Mining function — _mine_benchmark_example_sqls() at optimizer.py:7116 iterates over benchmarks passed in and takes each question's expected_sql as a ready-made example_sql proposal (lines 7170, 7237). There is no check to exclude questions that also appear in the eval set.
3. Called with train set — _run_enrichment() receives train_benchmarks and passes them straight to _mine_benchmark_example_sqls(...) (harness.py:2319). So training questions' expected SQL can be pasted into the space config.
4. Re-evaluated on the same train set — the lever loop calls run_evaluation(..., train_benchmarks, ..., "full", ...) at harness.py:4045. When Genie sees a question that matches an example_sql entry added from that same question, it has a trivial shortcut to the "right" answer — the accuracy delta is not evidence the space got better.
5. Lever 5 LLM prompt (LEVER_5_INSTRUCTION_PROMPT, config.py:1299–1392) is fed failures_context which contains the original question text and expected SQL. It tells the LLM "do not duplicate existing example SQL questions" but never "do not copy the benchmark question verbatim."
6. Held-out check is too late — the held-out evaluation (harness.py:6151) runs only after the lever loop has already accepted patches based on inflated
7. training scores.

Steps to reproduce

1. Go to '...'
2. Click on '...'
3. See error

Expected behavior

SQL examples should not have any overlap with benchmark questions.

Screenshots

If applicable, add screenshots to help explain the problem.

Additional context

Add any other context about the problem here.

[hiydavid]

Addressed in PR #202 (fix/gso-optimizer-correctness-and-leakage). Will auto-close on merge.

Branch is named for this exact issue. Key landings:

Source/split fixes (commit 238081f8 "PR 37: benchmark question isolation"):

• extract_genie_space_benchmarks reworked to use benchmarks.questions as primary and sample_questions as secondary; removed example_question_sqls as a benchmark source — the original leak channel.
• _filter_example_sql_mirrored_benchmarks drops benchmark rows whose normalised question key matches any entry in example_question_sqls.
• assign_splits removed the provenance-based curated-always-train override; all benchmarks participate in the same random held-out split.
• MAX_BENCHMARK_COUNT enforced before the record loop on both MLflow and raw Spark paths.

Single write-path firewall (commit 825c1016 "Close Bug #4 benchmark-leakage firewall"):

• New optimization/leakage.py (845 LOC) + tests/unit/test_leakage_firewall.py. is_benchmark_leak is called before every proposal is persisted.
• Combines canonicalize_sql (whitespace / alias / semicolon-normalised fingerprint) with per-patch-type shape-aware dispatch.
• BenchmarkCorpus precomputes shingles for train AND held-out so held-out leakage is caught even though held-out questions never enter a prompt — directly addresses point (6) of this issue.

Strict-mode default (commit b158ec31 "feat(leakage): strict-mode firewall blocks SQL pattern overlap by default"):

• Pattern-overlap blocking is now default-on.

Adjacent:

• optimization/_qid_extraction.py — canonical QID single source of truth across harness and GT-correction.
• optimization/afs.py — Abstracted Failure Signature; replaces raw question/expected_sql/generated_sql fields in cluster briefs with a closed schema derived entirely from judge ASI + P2.5 AST differ signals (no benchmark text reaches LEVER_5 prompts).
• feat: add leakage-safe RCA example synthesis bridge (commit ed7eaa23).