[Fix] MMMU-Pro cot_postproc: robustly extract the option letter (handles "Answer: $LETTER" echoes)

[sseanliu]

Summary

MMMUProDataset.cot_postproc mis-extracts the predicted option in two common cases, causing correct answers to be scored as wrong on the CoT settings of MMMU-Pro:

1. It counts uppercase letters across the entire Answer: line. The capital A in the word "Answer" is always counted, so len(counter) == 1 almost never holds and the clean-letter fast-path rarely fires.
2. When a model echoes the prompt's 'Answer: $LETTER' template literally — e.g. Answer: $B — the fallback returns the raw text " $B", which the downstream option matcher (exact_matching) cannot parse, so the item is scored incorrect.

build_prompt explicitly instructs "...format: 'Answer: $LETTER'...", and several models reproduce the $ literally, so this is hit frequently — especially in the Vision setting (MMMU_Pro_V), where the options live inside the image.

Fix

Inspect only the text after the last Answer: marker (and take the last Answer: line, in case the model emits more than one). This removes the spurious A from "Answer" and makes Answer: $B resolve to B.

if len(cands) >= 1:
    tail = cands[-1][len('Answer:'):]
    counter = defaultdict(lambda: 0)
    for ch in tail:
        if ch in string.ascii_uppercase:
            counter[ch] += 1
    if len(counter) == 1:
        return list(counter.keys())[0]
    else:
        return tail
return response

Validation

Eval-only rescoring of identical saved predictions (a 31B open model, greedy, --judge exact_matching), with only cot_postproc changed:

Setting	before	after
MMMU_Pro_V (COT)	18.55%	67.92%
MMMU_Pro_10c (COT)	~56%	~74%

In this run 77% of MMMU_Pro_V answers were written as Answer: $X (literal $). The fixed values match both the reference evaluate.py extraction (rfind("Answer:")) and the models' own stated answers, and are consistent with the public MMMU-Pro leaderboard for this model.

Single-function change, no new dependencies.