feat(gemini): add multimodal embedding support (#944)

Co-authored-by: TJ Miller <tj@themillers.co>

Author: cyppe

docs/core-concepts/embeddings.md

@@ -1,6 +1,6 @@
 # Embeddings
 
-Transform your content into powerful vector representations! Embeddings let you add semantic search, recommendation systems, and other advanced features to your applications - whether you're working with text or images.
+Transform your content into powerful vector representations! Embeddings let you add semantic search, recommendation systems, and other advanced features to your applications - whether you're working with text, images, audio, video, or documents.
 
 ## Quick Start
 
@@ -24,7 +24,7 @@ echo $response->usage->tokens;
 
 ## Generating multiple embeddings
 
-You can generate multiple embeddings at once with all providers that support embeddings, other than Gemini:
+You can generate multiple embeddings at once with providers that support batch embeddings:
 
 ```php
 use Prism\Prism\Facades\Prism;
@@ -86,12 +86,12 @@ $response = Prism::embeddings()
 > [!NOTE]
 > Make sure your file exists and is readable. The generator will throw a helpful `PrismException` if there's any issue accessing the file.
 
-## Image Embeddings
+## Multimodal Embeddings
 
-Some providers support image embeddings, enabling powerful use cases like visual similarity search, cross-modal retrieval, and multimodal applications. Prism makes it easy to generate embeddings from images using the same fluent API.
+Some providers support multimodal embeddings, enabling powerful use cases like visual similarity search, cross-modal retrieval, and mixed media retrieval. Prism makes it easy to generate embeddings from images, audio, video, and documents using the same fluent API.
 
 > [!IMPORTANT]
-> Image embeddings require a provider and model that supports image input (such as CLIP-based models or multimodal embedding models like BGE-VL). Check your provider's documentation to confirm image embedding support.
+> Multimodal embeddings require a provider and model that supports the input modalities you send. Check your provider's documentation to confirm support for images, audio, video, documents, and grouped content.
 
 ### Single Image
 
@@ -131,25 +131,58 @@ foreach ($response->embeddings as $embedding) {
 }
 ```
 
-### Multimodal: Text + Image
+### Audio, Video, and Documents
 
-Combine text and images for cross-modal search scenarios. This is particularly useful for applications like "find products similar to this image that match this description":
+```php
+use Prism\Prism\Facades\Prism;
+use Prism\Prism\ValueObjects\Media\Audio;
+use Prism\Prism\ValueObjects\Media\Document;
+use Prism\Prism\ValueObjects\Media\Video;
+
+$response = Prism::embeddings()
+    ->using('provider', 'model')
+    ->fromAudio(Audio::fromLocalPath('/path/to/sample.mp3'))
+    ->fromVideo(Video::fromLocalPath('/path/to/sample.mp4'))
+    ->fromDocument(Document::fromLocalPath('/path/to/report.pdf'))
+    ->asEmbeddings();
+```
+
+### Grouped Multimodal Content
+
+Use `fromContent()` when you want a single embedding generated from multiple parts within the same content entry:
 
 ```php
 use Prism\Prism\Facades\Prism;
 use Prism\Prism\ValueObjects\Media\Image;
 
 $response = Prism::embeddings()
     ->using('provider', 'model')
-    ->fromInput('Find similar products in red')
-    ->fromImage(Image::fromBase64($productImage, 'image/png'))
+    ->fromContent([
+        'Find similar products in red',
+        Image::fromBase64($productImage, 'image/png'),
+    ])
+    ->asEmbeddings();
+```
+
+Use `fromContents()` when you want multiple embeddings in a single request:
+
+```php
+use Prism\Prism\Facades\Prism;
+use Prism\Prism\ValueObjects\Media\Image;
+
+$response = Prism::embeddings()
+    ->using('provider', 'model')
+    ->fromContents([
+        ['The dog is cute'],
+        [Image::fromLocalPath('/path/to/dog.png')],
+    ])
     ->asEmbeddings();
 ```
 
-You can chain `fromImage()` and `fromInput()` in any order - Prism handles both gracefully.
+You can still chain `fromInput()` and `fromImage()` in any order. Each chained call creates a separate content entry.
 
 > [!TIP]
-> The `Image` class supports multiple input sources: `fromLocalPath()`, `fromUrl()`, `fromBase64()`, `fromStoragePath()`, and `fromRawContent()`. See the [Images documentation](/input-modalities/images.html) for details.
+> Prism media value objects support multiple input sources. See the [Images documentation](/input-modalities/images.html) and related modality guides for details.
 
 ## Common Settings
 

docs/providers/gemini.md

@@ -395,6 +395,65 @@ $response = Prism::text()
 
 You can customize your Gemini embeddings request with additional parameters using `->withProviderOptions()`.
 
+### Gemini Embedding 2 Preview
+
+Gemini's `gemini-embedding-2-preview` model supports text, images, audio, video, and PDF documents in a unified embedding space. Use Prism's content entry API to control whether you want a single aggregated embedding or multiple embeddings in one request.
+
+### Single Modality Inputs
+
+```php
+use Prism\Prism\Enums\Provider;
+use Prism\Prism\Facades\Prism;
+use Prism\Prism\ValueObjects\Media\Audio;
+use Prism\Prism\ValueObjects\Media\Document;
+use Prism\Prism\ValueObjects\Media\Image;
+use Prism\Prism\ValueObjects\Media\Video;
+
+Prism::embeddings()
+    ->using(Provider::Gemini, 'gemini-embedding-2-preview')
+    ->fromImage(Image::fromLocalPath('/path/to/product.png'))
+    ->fromAudio(Audio::fromLocalPath('/path/to/example.mp3'))
+    ->fromVideo(Video::fromLocalPath('/path/to/example.mp4'))
+    ->fromDocument(Document::fromLocalPath('/path/to/report.pdf'))
+    ->asEmbeddings();
+```
+
+### Aggregated Multimodal Embeddings
+
+Use `fromContent()` to combine multiple parts into a single embedding:
+
+```php
+use Prism\Prism\Enums\Provider;
+use Prism\Prism\Facades\Prism;
+use Prism\Prism\ValueObjects\Media\Image;
+
+Prism::embeddings()
+    ->using(Provider::Gemini, 'gemini-embedding-2-preview')
+    ->fromContent([
+        'An image of a dog',
+        Image::fromLocalPath('/path/to/dog.png'),
+    ])
+    ->asEmbeddings();
+```
+
+### Batch Embeddings
+
+Use `fromContents()` to generate multiple embeddings in a single request:
+
+```php
+use Prism\Prism\Enums\Provider;
+use Prism\Prism\Facades\Prism;
+use Prism\Prism\ValueObjects\Media\Image;
+
+Prism::embeddings()
+    ->using(Provider::Gemini, 'gemini-embedding-2-preview')
+    ->fromContents([
+        ['The dog is cute'],
+        [Image::fromLocalPath('/path/to/dog.png')],
+    ])
+    ->asEmbeddings();
+```
+
 ### Title
 
 You can add a title to your embedding request. Only applicable when TaskType is `RETRIEVAL_DOCUMENT`

src/Embeddings/Content.php

@@ -0,0 +1,46 @@
+<?php
+
+declare(strict_types=1);
+
+namespace Prism\Prism\Embeddings;
+
+use InvalidArgumentException;
+use Prism\Prism\ValueObjects\Media\Media;
+use Prism\Prism\ValueObjects\Media\Text;
+
+readonly class Content
+{
+    /** @var array<int, Media|Text> */
+    private array $parts;
+
+    /**
+     * @param  array<int, Media|Text|string>  $parts
+     */
+    public function __construct(array $parts)
+    {
+        if ($parts === []) {
+            throw new InvalidArgumentException('Embeddings content must contain at least one part.');
+        }
+
+        $this->parts = array_map(
+            static fn (Media|Text|string $part): Media|Text => is_string($part) ? new Text($part) : $part,
+            $parts,
+        );
+    }
+
+    /**
+     * @param  array<int, Media|Text|string>  $parts
+     */
+    public static function make(array $parts): self
+    {
+        return new self($parts);
+    }
+
+    /**
+     * @return array<int, Media|Text>
+     */
+    public function parts(): array
+    {
+        return $this->parts;
+    }
+}

src/Embeddings/PendingRequest.php

@@ -9,23 +9,25 @@
 use Prism\Prism\Concerns\ConfiguresProviders;
 use Prism\Prism\Concerns\HasProviderOptions;
 use Prism\Prism\Exceptions\PrismException;
+use Prism\Prism\ValueObjects\Media\Audio;
+use Prism\Prism\ValueObjects\Media\Document;
 use Prism\Prism\ValueObjects\Media\Image;
+use Prism\Prism\ValueObjects\Media\Media;
+use Prism\Prism\ValueObjects\Media\Text;
+use Prism\Prism\ValueObjects\Media\Video;
 
 class PendingRequest
 {
     use ConfiguresClient;
     use ConfiguresProviders;
     use HasProviderOptions;
 
-    /** @var array<string> */
-    protected array $inputs = [];
-
-    /** @var array<Image> */
-    protected array $images = [];
+    /** @var array<Content> */
+    protected array $contents = [];
 
     public function fromInput(string $input): self
     {
-        $this->inputs[] = $input;
+        $this->contents[] = Content::make([$input]);
 
         return $this;
     }
@@ -35,7 +37,9 @@ public function fromInput(string $input): self
      */
     public function fromArray(array $inputs): self
     {
-        $this->inputs = array_merge($this->inputs, $inputs);
+        foreach ($inputs as $input) {
+            $this->fromInput($input);
+        }
 
         return $this;
     }
@@ -52,9 +56,7 @@ public function fromFile(string $path): self
             throw new PrismException(sprintf('%s contents could not be read', $path));
         }
 
-        $this->inputs[] = $contents;
-
-        return $this;
+        return $this->fromInput($contents);
     }
 
     /**
@@ -67,19 +69,98 @@ public function fromFile(string $path): self
      */
     public function fromImage(Image $image): self
     {
-        $this->images[] = $image;
+        $this->contents[] = Content::make([$image]);
 
         return $this;
     }
 
     /**
-     * Add multiple images for embedding generation.
-     *
      * @param  array<Image>  $images
      */
     public function fromImages(array $images): self
     {
-        $this->images = array_merge($this->images, $images);
+        foreach ($images as $image) {
+            $this->fromImage($image);
+        }
+
+        return $this;
+    }
+
+    public function fromAudio(Audio $audio): self
+    {
+        $this->contents[] = Content::make([$audio]);
+
+        return $this;
+    }
+
+    /**
+     * @param  array<Audio>  $audios
+     */
+    public function fromAudios(array $audios): self
+    {
+        foreach ($audios as $audio) {
+            $this->fromAudio($audio);
+        }
+
+        return $this;
+    }
+
+    public function fromVideo(Video $video): self
+    {
+        $this->contents[] = Content::make([$video]);
+
+        return $this;
+    }
+
+    /**
+     * @param  array<Video>  $videos
+     */
+    public function fromVideos(array $videos): self
+    {
+        foreach ($videos as $video) {
+            $this->fromVideo($video);
+        }
+
+        return $this;
+    }
+
+    public function fromDocument(Document $document): self
+    {
+        $this->contents[] = Content::make([$document]);
+
+        return $this;
+    }
+
+    /**
+     * @param  array<Document>  $documents
+     */
+    public function fromDocuments(array $documents): self
+    {
+        foreach ($documents as $document) {
+            $this->fromDocument($document);
+        }
+
+        return $this;
+    }
+
+    /**
+     * @param  array<int, Media|Text|string>  $parts
+     */
+    public function fromContent(array $parts): self
+    {
+        $this->contents[] = Content::make($parts);
+
+        return $this;
+    }
+
+    /**
+     * @param  array<int, array<int, Media|Text|string>>  $contents
+     */
+    public function fromContents(array $contents): self
+    {
+        foreach ($contents as $content) {
+            $this->fromContent($content);
+        }
 
         return $this;
     }
@@ -94,8 +175,8 @@ public function generate(): Response
 
     public function asEmbeddings(): Response
     {
-        if ($this->inputs === [] && $this->images === []) {
-            throw new PrismException('Embeddings input is required (text or images)');
+        if ($this->contents === []) {
+            throw new PrismException('Embeddings input is required (text, images, audio, video, documents, or content parts)');
         }
 
         $request = $this->toRequest();
@@ -112,11 +193,10 @@ protected function toRequest(): Request
         return new Request(
             model: $this->model,
             providerKey: $this->providerKey(),
-            inputs: $this->inputs,
-            images: $this->images,
             clientOptions: $this->clientOptions,
             clientRetry: $this->clientRetry,
-            providerOptions: $this->providerOptions
+            providerOptions: $this->providerOptions,
+            contents: $this->contents,
         );
     }
 }