Skip to content

fix: guard choices[0] and message=None before content access (41 sites, 32 files)#1332

Open
qizwiz wants to merge 1 commit into
EvolvingLMMs-Lab:mainfrom
qizwiz:fix/guard-unguarded-llm-choices
Open

fix: guard choices[0] and message=None before content access (41 sites, 32 files)#1332
qizwiz wants to merge 1 commit into
EvolvingLMMs-Lab:mainfrom
qizwiz:fix/guard-unguarded-llm-choices

Conversation

@qizwiz
Copy link
Copy Markdown

@qizwiz qizwiz commented May 17, 2026

Summary

OpenAI-compatible APIs can return empty choices on error or rate-limiting, causing IndexError at choices[0]. Gemini additionally returns choices[0].message = None on PROHIBITED_CONTENT safety filtering (HTTP 200 — no exception raised), causing AttributeError at .content.

This PR adds the comprehensive guard:

if not response.choices or response.choices[0].message is None:
    raise ValueError("LLM returned empty or filtered response")

before every bare choices[0].message.content access across 32 files, 41 sites.

Impact

Affected subsystems:

  • lmms_eval/llm_judge/providers/ — OpenAI and Azure OpenAI judge providers
  • lmms_eval/models/ — core model implementations (chat/openai, simple/openai, srt_api, gpt4v)
  • lmms_eval/tasks/ — 25+ task evaluation utilities across aime, babyvision, gpqa, live_bench, mathverse, mix_evals, mme_sci, openai_math, phyx, tomato, videoevalpro, videomme, and unig2u tasks
  • test/models/test_litellm.py

Verification

The Gemini choices[0].message = None crash is a documented production behavior on PROHIBITED_CONTENT safety filtering. Static analysis via pact (Z3 Fixedpoint datalog, llm_response_unguarded rule) found all 41 sites; post-fix scan returns 0 violations.

OpenAI-compatible APIs can return empty choices on error or content
filtering; Gemini returns choices[0].message=None on PROHIBITED_CONTENT
(HTTP 200). Both cause unhandled exceptions at runtime.

Adds `if not response.choices or response.choices[0].message is None:`
before every bare `choices[0].message.content` access across 32 files.
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this file got wrongly modified? The git diff looks not as expected as well as the other utils in mme_sci

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants