Skip to content

Commit 8c9af3d

Browse files
committed
tw review
1 parent 6a923c0 commit 8c9af3d

1 file changed

Lines changed: 16 additions & 16 deletions

File tree

  • content/en/docs/marketplace/genai/reference-guide/external-platforms

content/en/docs/marketplace/genai/reference-guide/external-platforms/gemini.md

Lines changed: 16 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ weight: 20
99

1010
## Introduction
1111

12-
The [Google Gemini Connector](https://marketplace.mendix.com/link/component/248276) allows you to integrate generative AI capabilities into your Mendix application. Since the Gemini API is compatible with [OpenAI API](https://platform.openai.com/), this module mainly focuses on Gemini specific UI while reusing the operations inside of the OpenAI connector.
12+
The [Google Gemini Connector](https://marketplace.mendix.com/link/component/254741) allows you to integrate generative AI capabilities into your Mendix application. Since the Gemini API is compatible with [OpenAI API](https://platform.openai.com/), this module mainly focuses on Gemini specific UI while reusing the operations inside of the OpenAI connector.
1313

1414
### Features {#features}
1515

@@ -41,7 +41,7 @@ To use this connector, you need to sign up for a Google AI Studio account and cr
4141

4242
Install all required modules from the Mendix Marketplace as listed in the [Dependencies](#dependencies) section above.
4343

44-
To import the [Google Gemini Connector](https://marketplace.mendix.com/link/component/248276) and the other modules into your app, follow the instructions in [How to Use Marketplace Content](/appstore/use-content/).
44+
To import the [Google Gemini Connector](https://marketplace.mendix.com/link/component/248276), and the other modules into your app, follow the instructions in [How to Use Marketplace Content](/appstore/use-content/).
4545

4646
## Configuration {#configuration}
4747

@@ -58,17 +58,17 @@ After you install the Gemini and OpenAI connector, you can find them in the **Ma
5858

5959
The following inputs are required for the Gemini configuration:
6060

61-
| Parameter | Value |
61+
| Parameter | Value |
6262
| ----------- | ------------------------------------------------------------ |
6363
| Display name | This is the name identifier of a configuration (for example, *MyConfiguration*). |
6464
| Endpoint | This is the API endpoint (for example, `https://generativelanguage.googleapis.com/v1beta/openai/`) |
65-
| Token | This is the access token to authorize your API call. <br />To get an API key, follow the steps mentioned in the [Quickstart](https://ai.google.dev/gemini-api/docs/quickstart). |
65+
| Token | This is the access token to authorize your API call. <br />To get an API key, follow the steps mentioned in the [Gemini API quickstart](https://ai.google.dev/gemini-api/docs/quickstart). |
6666

6767
#### Configuring the Gemini Deployed Models
6868

69-
A [Deployed Model](/appstore/modules/genai/genai-for-mx/commons/#deployed-model) represents a GenAI model instance that can be used by the app to generate text, embeddings, or images. For every model you want to invoke from your app, you need to create a `GeminiDeployedModel` record, a specialization of `DeployedModel` (and also a specialization of `OpenAIDeployedModel`). In addition to the model display name and a technical name or identifier, a Gemini deployed model contains a reference to the additional connection details as configured in the previous step. Currently only specific models for text generation are supported by the Google Gemini connector
69+
A [Deployed Model](/appstore/modules/genai/genai-for-mx/commons/#deployed-model) represents a GenAI model instance that can be used by the app to generate text, embeddings, or images. For every model you want to invoke from your app, you need to create a `GeminiDeployedModel` record, a specialization of `DeployedModel` (and also a specialization of `OpenAIDeployedModel`). In addition to the model display name and a technical name or identifier, a Gemini deployed model contains a reference to the additional connection details as configured in the previous step. Currently only specific models for text generation are supported by the Google Gemini connector.
7070

71-
1. Click the three dots ({{% icon name="three-dots-menu-horizontal" %}}) icon for a Gemini configuration and open **Manage Deployed Models**. It is possible to use a predefined generation method, where available models are created according to their capabilities.
71+
1. Click the three-dots ({{% icon name="three-dots-menu-horizontal-filled" %}}) icon for a Gemini configuration and open **Manage Deployed Models**. It is possible to use a predefined generation method, where available models are created according to their capabilities.
7272

7373
2. Close the **Manage Deployed Models** popup and test the configuration with the newly created deployed models.
7474

@@ -80,30 +80,30 @@ For more inspiration or guidance on how to use the microflow actions in your log
8080

8181
You can use the GenAI Commons toolbox actions to [create the required Request](/appstore/modules/genai/genai-for-mx/commons/#genai-request-building) and [handle the Response](/appstore/modules/genai/genai-for-mx/commons/#genai-response-handling) for your use case.
8282

83-
The internal chat completion logic supports [JSON mode](#chatcompletions-json-mode), [function calling](#chatcompletions-functioncalling), and [vision](#chatcompletions-vision) for Gemini. Make sure to check the actual compatibility of the available models with these functionalities, as this changes over time. The following sections list toolbox actions which are specifically for OpenAI compatible APIs (especially Gemini).
83+
The internal chat completion logic supports [JSON mode](#chatcompletions-json-mode), [Function Calling](#chatcompletions-functioncalling), and [Vision](#chatcompletions-vision) for Gemini. Make sure to check the actual compatibility of the available models with these functionalities, as this changes over time. The following sections list toolbox actions for OpenAI compatible APIs (especially Gemini).
8484

8585
#### Chat Completions
8686

87-
Operations for chat completions focus on the generation of text based on a certain input. In this context, system prompts and user prompts are two key components that help guide the language model in generating relevant and contextually appropriate responses. For more information on the type of prompts and message roles, see the [ENUM_MessageRole](/appstore/modules/genai/genai-for-mx/commons/#enum-messagerole) enumeration. To learn more about how to create the right prompts for your use case, see the [Read More](#read-more) section below
87+
Operations for chat completions focus on the generation of text based on a certain input. In this context, system prompts and user prompts are two key components that help guide the language model in generating relevant and contextually appropriate responses. For more information on the type of prompts and message roles, see the [ENUM_MessageRole](/appstore/modules/genai/genai-for-mx/commons/#enum-messagerole) enumeration. To learn more about how to create the right prompts for your use case, see the [Read More](#read-more) section below.
8888

89-
The `GeminiDeployedModel` is compatible with the two [Chat Completions operations from GenAI Commons](/appstore/modules/genai/genai-for-mx/commons/#genai-generate). While developing your custom microflow, you can drag and drop the following operations from the toolbox in Studio Pro. See category **GenAI (Generate)**:
89+
The `GeminiDeployedModel` is compatible with the two chat completions operations from GenAI Commons. While developing your custom microflow, you can drag and drop the following operations from the toolbox in Studio Pro. See category [GenAI (Generate)](/appstore/modules/genai/genai-for-mx/commons/#genai-generate):
9090

9191
* Chat Completions (with history)
9292
* Chat Completions (without history)
9393

9494
#### JSON Mode {#chatcompletions-json-mode}
9595

96-
When JSON mode is used, the model is programmatically instructed to return valid JSON. For the Google Gemini connector, you have to explicitly mention the necessity of a JSON structure in a message in the conversation, e.g. the system prompt. Additionally, after creating the request, but before passing it to the chat completions operation, use the toolbox action `Set Response Format` to set the required response format to JSON.
96+
When JSON mode is used, the model is programmatically instructed to return valid JSON. For the Google Gemini connector, you have to explicitly mention the necessity of a JSON structure in a message in the conversation, for example, the system prompt. Additionally, after creating the request, but before passing it to the chat completions operation, use the toolbox action `Set Response Format` to set the required response format to JSON.
9797

9898
#### Function Calling {#chatcompletions-functioncalling}
9999

100100
Function calling enables LLMs to connect with external tools to gather information, execute actions, convert natural language into structured data, and much more. Function calling thus enables the model to intelligently decide when to let the Mendix app call one or more predefined function microflows to gather additional information to include in the assistant's response.
101101

102102
Gemini does not call the function. The model returns a tool called JSON structure that is used to build the input of the function (or functions) so that they can be executed as part of the chat completions operation. Functions in Mendix are essentially microflows that can be registered within the request to the LLM​. The OpenAI connector takes care of handling the tool call response as well as executing the function microflows until the API returns the assistant's final response for Gemini.
103103

104-
This is all part of the implementation that is executed by the GenAI Commons chat completions operations mentioned before. As a developer, you have to make the system aware of your functions and what these do by registering the function(s) to the request. This is done using the GenAI Commons operation [Tools: Add Function to Request](/appstore/modules/genai/genai-for-mx/commons/#add-function-to-request) once per function before passing the request to the chat completions operation.
104+
This is all part of the implementation that is executed by the GenAI Commons chat completions operations. As a developer, make the system aware of your functions and what is done by registering the functions to the request. This is done using the GenAI Commons operation [Tools: Add Function to Request](/appstore/modules/genai/genai-for-mx/commons/#add-function-to-request) once per function before passing the request to the chat completions operation.
105105

106-
Function microflows can have none, a single, or multiple primitive input parameters such as Boolean, Datetime, Decimal, Enumeration, Integer or String. Additionally, they may accept the [Request](/appstore/modules/genai/genai-for-mx/commons/#request) or [Tool](/appstore/modules/genai/genai-for-mx/commons/#tool) objects as inputs. The function microflow must return a String value.
106+
Function microflows can have none, a single, or multiple primitive input parameters such as Boolean, Datetime, Decimal, Enumeration, Integer, or String. Additionally, they may accept the [Request](/appstore/modules/genai/genai-for-mx/commons/#request) or [Tool](/appstore/modules/genai/genai-for-mx/commons/#tool) objects as inputs. The function microflow must return a String value.
107107

108108
{{% alert color="warning" %}}
109109
Function calling is a very powerful capability and should be used with caution. Function microflows run in the context of the current user, without enforcing entity access. You can use `$currentUser` in XPath queries to ensure that you retrieve and return only information that the end-user is allowed to view; otherwise, confidential information may become visible to the current end-user in the assistant's response.
@@ -119,19 +119,19 @@ Adding knowledge bases to a call enables LLMs to retrieve information when a rel
119119

120120
Gemini does not directly connect to the knowledge resources. The model returns a tool call JSON structure that is used to build the input of the retrievals so that they can be executed as part of the chat completions operation. The OpenAI connector takes care of handling the tool call response for Gemini as well as executing the function microflows until the API returns the assistant's final response.
121121

122-
This functionality is part of the implementation executed by the GenAI Commons Chat Completions operations mentioned earlier. As a developer, you need to make the system aware of your indexes and their purpose by registering them with the request. This is done using the GenAI Commons operation [Tools: Add Knowledge Base](/appstore/modules/genai/genai-for-mx/commons/#add-knowledge-base-to-request), which must be called once per knowledge resource before passing the request to the Chat Completions operation.
122+
This functionality is part of the implementation executed by the GenAI Commons Chat Completions operations mentioned earlier. As a developer, make the system aware of your indexes and their purpose by registering them with the request. This is done using the GenAI Commons operation [Tools: Add Knowledge Base](/appstore/modules/genai/genai-for-mx/commons/#add-knowledge-base-to-request), which must be called once per knowledge resource before passing the request to the Chat Completions operation.
123123

124124
Note that the retrieval process is independent of the model provider and can be used with any model that supports function calling, as it relies on the generalized `GenAICommons.DeployedKnowledgeBase` input parameter.
125125

126126
#### Vision {#chatcompletions-vision}
127127

128-
Vision enables models to interpret and analyze images, allowing them to answer questions and perform tasks related to visual content. This integration of computer vision and language processing enhances the model's comprehension and makes it valuable for tasks involving visual information. To make use of vision with the Google Gemini connector, an optional [FileCollection](/appstore/modules/genai/genai-for-mx/commons/#filecollection) containing one or multiple images must be sent along with a single message.
128+
Vision enables models to interpret and analyze images, allowing them to answer questions, and perform tasks related to visual content. This integration of computer vision and language processing enhances the model's comprehension and makes it valuable for tasks involving visual information. To make use of vision with the Google Gemini connector, send an optional [FileCollection](/appstore/modules/genai/genai-for-mx/commons/#filecollection) containing one or multiple images along with a single message.
129129

130130
For `Chat Completions without History`, `FileCollection` is an optional input parameter.
131131

132-
For `Chat Completions with History`, `FileCollection` can optionally be added to individual user messages using [Chat: Add Message to Request](/appstore/modules/genai/genai-for-mx/commons/#chat-add-message-to-request).
132+
For `Chat Completions with History`, you can optionally add `FileCollection` to individual user messages using [Chat: Add Message to Request](/appstore/modules/genai/genai-for-mx/commons/#chat-add-message-to-request).
133133

134-
Use the two microflow actions from the OpenAI specific toolbox [Files: Initialize Collection with OpenAI File](#initialize-filecollection) and [Files: Add OpenAIFile to Collection](#add-file) to construct the input with either `FileDocuments` (for vision, it needs to be of type `Image`) or `URLs`. There are similar file operations exposed by the GenAI commons module that can be used for vision requests with the OpenAIConnector for Gemini. However, these generic operations do not support the optional OpenAI API-specific `Detail` attribute.
134+
Use the two microflow actions from the OpenAI specific toolbox [Files: Initialize Collection with OpenAI File](#initialize-filecollection) and [Files: Add OpenAIFile to Collection](#add-file) to construct the input with either `FileDocuments` (for vision, it must be of type `Image`) or `URLs`. There are similar file operations exposed by the GenAI commons module that you can use for vision requests with the OpenAIConnector for Gemini. However, these generic operations do not support the optional OpenAI API-specific `Detail` attribute.
135135

136136
For more information on vision, see [Gemini documentation](https://ai.google.dev/gemini-api/docs/openai#image-understanding).
137137

0 commit comments

Comments
 (0)