feat: multimodal embeddings for bedrock titan and cohere#1130
Conversation
![matterai-app[bot]](https://avatars.githubusercontent.com/in/474549?s=60&v=4){kind=small}
There was a problem hiding this comment.
This PR adds multimodal embedding support for Bedrock Titan and Cohere providers, allowing both text and image inputs. The implementation is well-structured, but I've identified a few improvements that could enhance error handling and type safety.
|
Important PR Review SkippedPR review skipped as per the configuration setting. Run a manually review by commenting /matter review 💡Tips to use Matter AICommand List
|
|
Description# Summary By MatterAI
🔄 What ChangedThis Pull Request introduces support for multimodal embeddings for Bedrock Titan and Cohere models. The core changes involve updating the provider configuration ( 🔍 Impact of the ChangeThis change significantly enhances the capabilities of the Portkey API by enabling users to generate embeddings from both text and image inputs using Bedrock Titan and Cohere models. It aligns the API with multimodal embedding standards, providing greater flexibility and broader model support. The updated input parsing logic ensures compatibility with various client request formats. 📁 Total Files Changed2 files were changed in this pull request. 🧪 Test AddedManual testing was performed using
🔒Security VulnerabilitiesNo new security vulnerabilities were detected in the provided code changes. MotivationThis PR adds support for multimodal embeddings in compliance with the Portkey signature, expanding the functionality of the embeddings API to handle both text and image inputs for Bedrock Titan and Cohere models. Type of Change
How Has This Been Tested?
Screenshots (if applicable)N/A Checklist
Related IssuesN/A Tip Quality Recommendations
Sequence DiagramsequenceDiagram
actor User
participant PortkeyAPI as Portkey API
participant BedrockCohereConfig as BedrockCohereEmbedConfig
participant BedrockTitanConfig as BedrockTitanEmbedConfig
participant CohereConfig as CohereEmbedConfig
participant BedrockAPI as Bedrock API
participant CohereAPI as Cohere API
User->>PortkeyAPI: POST /v1/embeddings (model, input, dimensions, input_type, encoding_format, normalize)
PortkeyAPI->>PortkeyAPI: Determine Provider (Bedrock/Cohere)
alt Bedrock Cohere Model
PortkeyAPI->>BedrockCohereConfig: Apply config transformations (EmbedParams)
BedrockCohereConfig->>BedrockCohereConfig: Transform input: (string | object[]) -> texts (string[]) | images (base64 string[])
BedrockCohereConfig->>BedrockCohereConfig: Transform encoding_format: (string | string[]) -> embedding_types (string[])
BedrockCohereConfig-->>PortkeyAPI: Transformed Bedrock Cohere Params (texts, images, input_type, truncate, embedding_types)
PortkeyAPI->>BedrockAPI: Call Bedrock Cohere Embed API
else Bedrock Titan Model
PortkeyAPI->>BedrockTitanConfig: Apply config transformations (EmbedParams)
BedrockTitanConfig->>BedrockTitanConfig: Transform input: (string | object[]) -> inputText (string) | inputImage (base64 string)
BedrockTitanConfig->>BedrockTitanConfig: Transform dimensions: (number) -> dimensions (number) | embeddingConfig (object)
BedrockTitanConfig->>BedrockTitanConfig: Transform encoding_format: (string | string[]) -> embeddingTypes (string[])
BedrockTitanConfig-->>PortkeyAPI: Transformed Bedrock Titan Params (inputText, inputImage, dimensions, embeddingConfig, encoding_format, normalize)
PortkeyAPI->>BedrockAPI: Call Bedrock Titan Embed API
else Cohere Model
PortkeyAPI->>CohereConfig: Apply config transformations (EmbedParams)
CohereConfig->>CohereConfig: Transform input: (string | object[]) -> texts (string[]) | images (base64 string[])
CohereConfig->>CohereConfig: Transform encoding_format: (string | string[]) -> embedding_types (string[])
CohereConfig-->>PortkeyAPI: Transformed Cohere Params (texts, images, input_type, truncate, embedding_types)
PortkeyAPI->>CohereAPI: Call Cohere Embed API
end
BedrockAPI-->>PortkeyAPI: Embed Response
CohereAPI-->>PortkeyAPI: Embed Response
PortkeyAPI-->>User: EmbedResponse
|
|
Important PR Review SkippedPR review skipped as per the configuration setting. Run a manually review by commenting /matter review 💡Tips to use Matter AICommand List
|
|
Important PR Review SkippedPR review skipped as per the configuration setting. Run a manually review by commenting /matter review 💡Tips to use Matter AICommand List
|
2 participants
This PR adds support for multimodal embeddings in compliance with the portkey signature:
example requests: