Is few_shot_eval prepared for base models?

[wenrizhou]

Hi, I have a question about the official few-shot evaluation pipeline for TheoremQA.

From few_shot_eval/run.py, it seems that model outputs are post-processed via:

answer = utils.answer_clean(
    ['The answer is:', 'The answer is', 'the answer is'],
    output
)

and thenanswer_clean(...)calls extract_theoremqa_answer(...) in few_shot_eval/utils.py.

At the same time, few_shot_eval/data_loader.py defines several model-specific prompt templates (short, step, alpaca, vicuna, gemma, mistral, etc.), which makes me think this pipeline may have been mainly designed for base models, with prompt formatting expected to stabilize the answer style.

I wanted to ask whether that understanding is correct.

Also, I noticed a possible extraction edge case in extract_theoremqa_answer(...):

if any(option in pred.lower() for option in ['yes', 'true']):
    pred = 'True'
elif any(option in pred.lower() for option in ['no', 'false']):
    pred = 'False'

Because this uses substring matching, a model output like:

which <A,B> represent Node A is connected to Node B.
becomes:

which <a,b> represent node a is connected to node b.
after lowercasing, and then "no" inside "node" causes the prediction to be extracted as "False".

So I would like to ask:

Was the few-shot eval pipeline mainly intended for base models whose outputs are expected to closely follow the prompt format?
Is the reliance on "The answer is" an intentional assumption in the official few-shot setting?
Would the "node" -> "no" -> False behavior be considered an expected limitation, or a bug in the current extraction logic?
Thank you very much.