feat(py/plugins/google-genai): Vertex multimodal embeddings via :predict#5649
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces support for Vertex AI multimodal embeddings by routing requests to the :predict endpoint when using multimodalembedding models. It handles the mapping of text, image, and video inputs, and parses the returned embeddings. Feedback on these changes includes addressing potential silent data loss when multiple media parts are provided in a single document, raising a ValueError for unsupported HTTP URLs in media references, and adding defensive checks to handle API prediction responses safely.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
hilariie
force-pushed
the
feat/py-googleai-gemini-embedding-2
branch
from
0d74bba to
0ac4107
Compare
hilariie
force-pushed
the
feat/py-googleai-embedder-fixes
branch
from
ed65017 to
a19b688
Compare
cabljac
left a comment
cabljac
left a comment
There was a problem hiding this comment.
Real fix and the test coverage on the rejection cases is thorough. A handful of questions below before this lands, mostly about the new :predict path's edges.
hilariie
force-pushed
the
feat/py-googleai-embedder-fixes
branch
from
d175056 to
0587208
Compare
…video embeddings
Route Vertex multimodalembedding@001 through the :predict endpoint
instead of embed_content, which silently drops non-text parts on Vertex.
Build {text,image,video} instances, map gs:// and data: URLs, and return
one embedding per image/text/video chunk. Advertise it from the curated
VERTEX_KNOWN_EMBEDDERS list (the Vertex catalog over-lists embedders and
returns supported_actions=None, so they are not discovered), with the
model's real default dimensions (1408). Mirrors the JS vertexai embedder.
…quests multimodalembedding@001 accepts only one instance per :predict call, but _generate_multimodal built one instance per input document, so embed_many (or any multi-document EmbedRequest) sent an invalid multi-instance payload that Vertex rejects with an opaque error. Reject len(input) > 1 up front with a clear ValueError until batching is implemented.
Vertex multimodal embedding calls the private _api_client.async_request / http_response.body seam, which is verified against google-genai 1.63.0. The prior >=1.7.0 floor was merely the then-current release when attribution headers landed (7283fc2), not a verified minimum for this code path.
…back - Restrict multimodalembedding@001 to Vertex AI via an explicit is_vertex flag threaded from the plugin; googleai/multimodalembedding@001 requests now raise instead of POSTing :predict on an API-key client. - Guard the private google-genai transport, raising a clear error naming google-genai>=1.63.0 when _api_client.async_request is unavailable. - Validate base64 data: URLs: require the ;base64, marker, else raise (fixes the comma-less IndexError and the silent non-base64 passthrough). - Read snake_case output_dimensionality only, matching the text path, and stop dropping a literal 0. - Document that one document fans out to multiple embeddings correlated by embedType metadata, not position. - Annotate _media_reference return as dict[str, Any] (holds videoSegmentConfig).
hilariie
force-pushed
the
feat/py-googleai-embedder-fixes
branch
from
0587208 to
72da35f
Compare
|
One post-approval change from the rebase: resolving conflicts with #5703, I initially dropped the |
2 participants
What
Adds working image and video embedding support for the Vertex AI
multimodalembedding@001model in thegoogle-genaiplugin, and registers correct output dimensions formultimodalembedding@001andtext-embedding-004.Why
The plugin advertised
multimodalembedding@001as accepting image and video input but routed every embed request throughembed_content. On Vertex AI that call drops non-text parts, so image and video requests failed at runtime with400 INVALID_ARGUMENT "Empty instances.". The advertised output dimension of 768 is also not valid for this model; the default is 1408.How
:predictendpoint instead ofembed_content.Embedder.generate()checks_is_multimodal()and dispatches to_generate_multimodal().{text, image, video}prediction instances per document. Image and video references mapgs://URLs togcsUrianddata:URLs tobytesBase64Encoded;http(s)URLs are rejected, since Vertex only acceptsgs://there. An optional video segment config is read from document metadata.predictions[]into oneEmbeddingper image, text, and video chunk, tagging each withmetadata.embedTypeand preserving video chunk offsets.multimodalembedding@001from the curatedVERTEX_KNOWN_EMBEDDERSlist rather than model discovery: the Vertex catalog over-lists embedders and returnssupported_actions=None, so Vertex embedders are curated, not discovered.multimodalembedding1408 (model default; valid values 128/256/512/1408) andtext-embedding-004768.Testing
uv run --directory py pytest plugins/google-genai/test/google_plugin_test.py plugins/google-genai/test/models/googlegenai_embedder_test.py(71 passed)uv run --directory py ruff checkandruff format --checkon the changed files (clean)New tests cover the
:predictrequest path and image/video result mapping, curated-list registration ofmultimodalembedding@001, and dimension and input metadata for bare and versioned model names.Limitations
multimodalembedding@001accepts a single instance per:predictcall, so requests with more than one document are rejected with a clear error until batching is implemented. Single document embeds work.Screenshots