Skip to content

feat(gemini): migrate to google-genai SDK, keep gemini_api alias#1382

Open
Luodian wants to merge 3 commits into
mainfrom
feat/gemini-google-genai-sdk
Open

feat(gemini): migrate to google-genai SDK, keep gemini_api alias#1382
Luodian wants to merge 3 commits into
mainfrom
feat/gemini-google-genai-sdk

Conversation

@Luodian

@Luodian Luodian commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Gemini: migrate to the google-genai SDK

Public shipped the deprecated google-generativeai backend (gemini_api.py). This PR modernizes the Gemini integration to the current google-genai SDK.

What changed

  • New backend lmms_eval/models/simple/gemini.py (Gemini, registered as gemini): uses google.genai, with thread-safe video/audio file upload + cache, adaptive concurrency (via model_utils.concurrency_control), prefix-aware request queue, safety-block salvage, and token/usage accounting (model_utils.usage_metrics).
  • New chat backend lmms_eval/models/chat/gemini.py (Gemini(GeminiSimple), is_simple=False): converts the ChatMessages protocol to Gemini-native contents and extracts system messages into system_instruction.
  • Registry (lmms_eval/models/__init__.py): gemini registered as both a simple and a chat model; gemini_api kept as an alias of gemini (MODEL_ALIASES["gemini"] = ("gemini_api",)) so existing --model gemini_api commands keep working.
  • Removed the deprecated lmms_eval/models/simple/gemini_api.py (the CLI name survives via the alias).
  • pyproject.toml: gemini extra google-generativeai -> google-genai>=1.0.0.
  • New lmms_eval/models/model_utils/parallel.py: small generic ParallelPlan / resolve_local_rank_device launcher-parallelism helper (stdlib + lazy torch).

Compatibility

  • --model gemini and legacy --model gemini_api both resolve to the new Gemini class (verified via the registry manifest + alias resolution).
  • Backend switch: because gemini is now registered as a chat model too, --model gemini / --model gemini_api default to the chat backend (same convention as openai); pass --force_simple for the simple path. The interleave model_arg applies to the simple path only.
  • Default model: model_version defaults to gemini-3.5-flash (the old backend defaulted to the now-deprecated gemini-1.5-pro); pin model_version=... in --model_args if you need a specific model.
  • Concurrency default: num_concurrent=32 with linear backoff (the old backend was serial). On low-quota keys, lower it or set adaptive_concurrency=True.

Notes

Review pass (2026-07-06)

Two self-review commits:

  • Upload polling compared str(FileState.ACTIVE) ("FILESTATE.ACTIVE") against "ACTIVE" — it never matched, so every unique video/audio upload burned the full poll budget (~120s) and FAILED uploads were never surfaced. The loop now normalizes the enum via its name and raises on FAILED.
  • Audio file paths were always tagged mime_type="audio/wav". Both backends now map extensions to Gemini-canonical MIME types (.mp3 -> audio/mp3, .flac -> audio/flac, ..., fallback mimetypes.guess_type then audio/wav).

@Luodian Luodian marked this pull request as ready for review July 6, 2026 13:56
@Luodian Luodian requested review from kcz358 and pufanyi July 6, 2026 13:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant