Skip to content

Commit 3ac6966

Browse files
committed
SAS-1050: updated embeddings/imagegen operations
1 parent 18ba32b commit 3ac6966

1 file changed

Lines changed: 19 additions & 23 deletions

File tree

  • content/en/docs/appstore/use-content/platform-supported-content/modules/genai

content/en/docs/appstore/use-content/platform-supported-content/modules/genai/genai-commons.md

Lines changed: 19 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -305,7 +305,7 @@ Chat completions operations can be used by passing a [DeployedModel](#deployed-m
305305

306306
We recommend that you adapt to the same interface when developing custom chat completions or image generations operations, such as integration with different AI providers. The generic interfaces are described below. For more detailed information, refer to the documentation of the connector that you want to use, since it may expect specializations of the generic GenAI common entities as an input.
307307

308-
##### Chat Completions (Without History)
308+
##### Chat Completions (without history)
309309

310310
The `Chat Completions (without history)` operation supports scenarios where there is no need to send a list of (historic) messages comprising the conversation so far as part of the request.
311311

@@ -324,9 +324,9 @@ The `Chat Completions (without history)` operation supports scenarios where ther
324324
| --- | --- | --- |
325325
| `Response` | [Response](#response) | A `Response` object that contains the assistant's response.|
326326

327-
##### Chat Completions (With History)
327+
##### Chat Completions (with history)
328328

329-
The `Chat Completions (with history)` operation interface supports more complex use cases where a list of (historical) messages (for example, comprising the conversation or context so far) is sent as part of the request to the LLM.
329+
The `Chat Completions (with history)` operation supports more complex use cases where a list of (historical) messages (for example, comprising the conversation or context so far) is sent as part of the request to the LLM.
330330

331331
###### Input Parameters
332332

@@ -341,29 +341,29 @@ The `Chat Completions (with history)` operation interface supports more complex
341341
| --- | --- | --- |
342342
| `Response` | [Response](#response) | A `Response` object that contains the assistant's response. |
343343

344-
##### Image Generations
344+
##### Generate Image
345345

346-
The `Image Generations` operation interface supports the generation of images based on a `UserPrompt` passed as string. The returned `Response` contains a `FileContent` via `FileCollection` and `Message`. See microflows in the `Connector Building` folder to construct the output.
346+
The `Generate Image` operation supports the generation of images based on a `UserPrompt` passed as string. The returned `Response` contains a `FileContent` via `FileCollection` and `Message`. See microflows in the `Handle Response` folder to construct the output.
347347

348348
###### Input Parameters
349349

350350
| Name | Type | Mandatory | Description |
351351
| --- | --- | --- |--- |
352-
| `Connection` | [Connection](#connection) | mandatory | This is an object that contains specifications to interact with an AI provider. |
352+
| `DeployedModel` | [DeployedModel](#deployed-model) | mandatory | The DeployedModel entity replaces the Connection entity. It contains the name of the microflow to be executed for the specified model and other information relevant to connect to a model. The OutputModality needs to be Image. |
353353
| `UserPrompt` | String | mandatory | This is the description the image will be based on. |
354354
| `ImageOptions` | [ImageOptions](#imageoptions-entity) | optional | This can be used to pass optional request attributes. |
355355

356356
###### Return Value
357357

358358
| Name | Type | Description |
359359
| --- | --- | --- |
360-
| `Response` | [Response](#response) | A `Response` object that contains the assistant's response including a `FileContent` which will be used in [Get Generated Image (Single)](#image-get-single). |
360+
| `Response` | [Response](#response) | A `Response` object that contains the assistant's response including a `FileContent` which needs to be used in [Get Generated Image (Single)](#image-get-single) or [Get Generated Images (List)]({#image-get-list}).|
361361

362362
#### Text and Files: Build request {#text-build-request}
363363

364364
The following microflows help you construct the input request structures for the operations for text and files defined in GenAI Commons.
365365

366-
##### Chat: Create Request {#chat-create-request}
366+
##### Create Request {#chat-create-request}
367367

368368
This microflow can be used to create a request for a chat completion operation. This is the request object that contains the top-level functional input for the language model to generate a response.
369369

@@ -382,7 +382,7 @@ This microflow can be used to create a request for a chat completion operation.
382382
|--- |--- |--- |
383383
| `Request` |[Request](#request) | This is the created request object. |
384384

385-
##### Chat: Add Message to Request {#chat-add-message-to-request}
385+
##### Add Message to Request {#chat-add-message-to-request}
386386

387387
This microflow can be used to add a new [Message](#message) to the [Request](#request) object. A message represents the conversation text content and optionally has a collection of files attached that need to be taken into account when generating the response (such as images for vision). Make sure to add messages chronologically so that the most recent message is added last.
388388

@@ -399,7 +399,7 @@ This microflow can be used to add a new [Message](#message) to the [Request](#re
399399

400400
This microflow does not have a return value.
401401

402-
##### Chat: Add Stop Sequence {#chat-add-stop-sequence}
402+
##### Configure Stop Sequence {#chat-add-stop-sequence}
403403

404404
This microflow can be used to add an optional [StopSequence](#stopsequence) to the request. It can be used after the request has been created. If available for the connector and model of choice, stop sequences let models know when to stop generating text.
405405

@@ -451,7 +451,7 @@ Use this microflow to add a file to an existing [FileCollection](#filecollection
451451

452452
This microflow does not have a return value.
453453

454-
##### Image: Create ImageOptions {#imageoptions-create}
454+
##### Image Generation: Create ImageOptions {#imageoptions-create}
455455

456456
This microflow creates new [ImageOptions](#imageoptions-entity).
457457

@@ -512,7 +512,7 @@ This microflow does not have a return value.
512512

513513
The following microflows handle the response processing.
514514

515-
##### Get Model Response Text {#chat-get-model-response-text}
515+
##### Get Response Text {#chat-get-model-response-text}
516516

517517
This microflow can be used to get the content from the latest assistant message over association `Response_Message`. Use this microflow to get the response text from the latest assistant response message. In many cases, this is the main value needed for further logic after the operation or is displayed to the end user.
518518

@@ -580,22 +580,18 @@ This operation processes a response that was created by an image generations ope
580580

581581
#### Knowledge Bases and Embeddings: Embeddings {#embeddings}
582582

583-
To make use of embeddings in a Mendix app, GenAI Commons defines interfaces for embedding operations that connectors can adhere to. We recommend that you adapt to the same interface when developing custom embedding operations, such as integration with different AI providers. The generic interfaces are described below. For more detailed information, refer to the documentation of the connector that you want to use, since it may expect specializations of the generic GenAI common entities as an input.
583+
Embeddings operations can be used by passing a [DeployedModel](#deployed-model) object of the desired connector. The action calls the internally assigned microflow of the connector and returns the response. Operations from different connectors can be exchanged very easily without much additional development effort.
584584

585-
{{% alert color="info" %}}
586-
These operations are not implemented in this module. The module only describes the interface (microflow input parameters, return value, and expected behavior), and it is up to connectors that adhere to the principles of GenAI Commons to provide an implementation. For an implementation example, see the respective sections in the [OpenAI connector](/appstore/modules/genai/openai/) or the [Amazon Bedrock Connector](/appstore/modules/genai/bedrock/), or take a look at the [showcase app](https://marketplace.mendix.com/link/component/220475) where both connectors are implemented to decide at runtime whether to call the LLM through OpenAI or Amazon Bedrock.
587-
{{% /alert %}}
588-
589-
##### Embeddings (String)
585+
##### Generate Embeddings (String) {#embeddings-string}
590586

591-
The `Embeddings (String)` operation interface allows the invocation of the embeddings API with a String input and returns an `EmbeddingsResponse` object with token usage statistics, if applicable. The `EmbeddingsResponse_GetFirstVector` microflow from GenAI Commons can be used to retrieve the corresponding embedding vector in a String representation. This operation supports scenarios where the vector embedding of a single string must be generated, e.g. to perform a nearest neighbor search across an existing knowledge base.
587+
The `Generate Embeddings (String)` operation allows the invocation of the embeddings API with a String input and returns an `EmbeddingsResponse` object with token usage statistics, if applicable. The `EmbeddingsResponse_GetFirstVector` microflow from GenAI Commons can be used to retrieve the corresponding embedding vector in a String representation. This operation supports scenarios where the vector embedding of a single string must be generated, e.g. to perform a nearest neighbor search across an existing knowledge base.
592588

593589
###### Input Parameters
594590

595591
| Name | Type | Mandatory | Description |
596592
| --- | --- | ---| --- |
597593
| `InputText` | String | mandatory | Input text to create the embedding vector for. |
598-
| `Connection` | [Connection](#connection) | mandatory | Connection object that contains the required endpoint details and API credentials. Depending on the connector module, a specific specialization must be passed. |
594+
| `DeployedModel` | [DeployedModel](#deployed-model) | mandatory | The DeployedModel entity replaces the Connection entity. It contains the name of the microflow to be executed for the specified model and other information relevant to connect to a model. The OutputModality needs to be Embeddings. |
599595
| `EmbeddingOptions` | [EmbeddingsOptions](#embeddingsoptions-entity) | optional | Can be used to pass optional request attributes.|
600596

601597
###### Return Value
@@ -604,16 +600,16 @@ The `Embeddings (String)` operation interface allows the invocation of the embed
604600
| --- | --- | --- |
605601
| `EmbeddingsResponse` | [EmbeddingsResponse](#embeddingsresponse-entity) | A response object that contains the token usage statistics and the corresponding embedding vector as part of a ChunkCollection |
606602

607-
##### Embeddings (ChunkCollection)
603+
##### Generate Embeddings (Chunk Collection) {#embeddings-chunk-collection}
608604

609-
The `Embeddings (ChunkCollection)` operation interface allows the invocation of an embeddings API with a [ChunkCollection](#chunkcollection) and returns an [EmbeddingsResponse](#embeddingsresponse-entity) object with token usage statistics, if applicable. The response object is associated with the original [ChunkCollection](#chunkcollection) used as an input, and the [Chunk](#chunk-entity) (or [KnowledgeBaseChunk](#knowledgebasechunk-entity)) objects will be updated with their corresponding embedding vector retrieved from the Embeddings API within this microflow.
605+
The `Generate Embeddings (Chunk Collection)` operation allows the invocation of an embeddings API with a [ChunkCollection](#chunkcollection) and returns an [EmbeddingsResponse](#embeddingsresponse-entity) object with token usage statistics, if applicable. The response object is associated with the original [ChunkCollection](#chunkcollection) used as an input, and the [Chunk](#chunk-entity) (or [KnowledgeBaseChunk](#knowledgebasechunk-entity)) objects will be updated with their corresponding embedding vector retrieved from the Embeddings API within this microflow.
610606

611607
###### Input Parameters
612608

613609
| Name | Type | Mandatory | Description |
614610
| --- | --- | ---| --- |
615611
| `ChunkCollection` | [ChunkCollection](#chunkcollection) | mandatory | A ChunkCollection with Chunks for which an embedding vector should be generated. Use operations from GenAI commons to create a ChunkCollection and add Chunks or KnowledgeBaseChunks to it. |
616-
| `Connection` | [Connection](#connection) | mandatory | A Connection object that contains the required endpoint details and API credentials. Depending on the connector module, a specific specialization must be passed. |
612+
| `DeployedModel` | [DeployedModel](#deployed-model) | mandatory | The DeployedModel entity replaces the Connection entity. It contains the name of the microflow to be executed for the specified model and other information relevant to connect to a model. The OutputModality needs to be Embeddings. |
617613
| `EmbeddingOptions` | [EmbeddingsOptions](#embeddingsoptions-entity) | optional | Can be used to pass optional request attributes. |
618614

619615
###### Return Value

0 commit comments

Comments
 (0)