Commit a2742ca
Revert "Fix input validation issues and update multiple-choice options handling (microsoft#2501)" (microsoft#2509)
## Describe your changes
Reverts PR microsoft#2501 which broke vision VQA accuracy by changing option
labels from numbers (1/2/3/4) to letters (A/B/C/D) without updating the
ground truth conversion.
### Problem
PR microsoft#2501 changed option formatting to A/B/C/D but the ground truth was
still converted to 1-based numbers. The evaluator digit extraction took
priority over letter extraction, so predictions never matched ground
truth (~11% accuracy instead of ~66-75%).
### What this reverts
- `pre_process_data.py`: Restores 1-based numeric option labeling and
removes `extract_option_letter` field
- `olive_evaluator.py`: Removes the dead letter extraction branch,
restores digit-only extraction
### What is kept
The `graph_surgeries.py` change from microsoft#2501 (supporting `inputs_embeds`)
is intentionally **not** reverted as it is unrelated and correct.
### Testing
Validated on AI2D with Qwen3-VL-2B-Instruct — accuracy restored from
~11% to expected ~66%.
## Checklist
- [x] My code follows the style guidelines of this project
- [x] I have performed a self-review of my code
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>1 parent da09b4f commit a2742ca
2 files changed
Lines changed: 5 additions & 16 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
478 | 478 | | |
479 | 479 | | |
480 | 480 | | |
481 | | - | |
482 | 481 | | |
483 | 482 | | |
484 | | - | |
485 | | - | |
| 483 | + | |
486 | 484 | | |
487 | | - | |
488 | | - | |
489 | | - | |
490 | | - | |
| 485 | + | |
491 | 486 | | |
492 | 487 | | |
493 | 488 | | |
| |||
507 | 502 | | |
508 | 503 | | |
509 | 504 | | |
510 | | - | |
511 | 505 | | |
512 | 506 | | |
513 | 507 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
881 | 881 | | |
882 | 882 | | |
883 | 883 | | |
884 | | - | |
885 | 884 | | |
886 | 885 | | |
887 | 886 | | |
| |||
938 | 937 | | |
939 | 938 | | |
940 | 939 | | |
941 | | - | |
942 | | - | |
943 | | - | |
| 940 | + | |
| 941 | + | |
| 942 | + | |
944 | 943 | | |
945 | 944 | | |
946 | 945 | | |
| |||
953 | 952 | | |
954 | 953 | | |
955 | 954 | | |
956 | | - | |
957 | | - | |
958 | | - | |
959 | | - | |
960 | 955 | | |
961 | 956 | | |
962 | 957 | | |
| |||
0 commit comments