You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: content/en/docs/marketplace/genai/reference-guide/external-platforms/openai.md
+26-30Lines changed: 26 additions & 30 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,7 +11,7 @@ aliases:
11
11
12
12
## Introduction {#introduction}
13
13
14
-
The [OpenAI Connector](https://marketplace.mendix.com/link/component/220472) allows you to integrate generative AI into your Mendix app. It is compatible with [OpenAI's platform](https://platform.openai.com/)as well as [Azure's OpenAI service](https://oai.azure.com/).
14
+
The [OpenAI Connector](https://marketplace.mendix.com/link/component/220472) allows you to integrate generative AI into your Mendix app. It is compatible with [OpenAI's platform](https://platform.openai.com/)and [Microsoft Foundry](https://learn.microsoft.com/en-us/azure/ai-foundry/what-is-ai-foundry), where you can access OpenAI models.
* Creativity – Generate, edit, and iterate with end-users on creative and technical writing tasks, such as composing songs, writing screenplays, or learning an end-user’s writing style.
22
22
* Longer context – GPT-4 can handle over 25,000 words of text, allowing for use cases like long-form content creation, extended conversations, and document search and analysis.
23
23
24
-
Mendix provides dual-platform support for both [OpenAI](https://platform.openai.com/) and [Azure OpenAI](https://oai.azure.com/).
24
+
Mendix provides support for [OpenAI](https://platform.openai.com/) and [Microsoft Foundry](https://learn.microsoft.com/en-us/azure/ai-foundry/what-is-ai-foundry) (formerly known as Azure OpenAI or Cognitive Services). Microsoft Foundry is Microsoft's unified AI platform that provides access to OpenAI models.
25
25
26
26
With the current version, Mendix supports the Chat Completions API for [text generation](https://platform.openai.com/docs/guides/text-generation), the Image Generations API for [images](https://platform.openai.com/docs/guides/images), the Embeddings API for [vector embeddings](https://platform.openai.com/docs/guides/embeddings/what-are-embeddings), and indexes via [Azure AI Search](https://learn.microsoft.com/en-us/azure/search/) for knowledge base retrieval.
27
27
@@ -33,7 +33,7 @@ By integrating Azure AI Search, the OpenAI Connector enables knowledge base retr
33
33
34
34
### Prerequisites {#prerequisites}
35
35
36
-
To use this connector, you need to either sign up for an [OpenAI account](https://platform.openai.com/) or have access to deployments at [Azure OpenAI](https://oai.azure.com/).
36
+
To use this connector, you need to either sign up for an [OpenAI account](https://platform.openai.com/) or have access to a [Microsoft Foundry](https://learn.microsoft.com/en-us/azure/ai-foundry/what-is-ai-foundry) project with OpenAI models deployed.
37
37
38
38
### Dependencies {#dependencies}
39
39
@@ -74,33 +74,29 @@ The following inputs are required for the OpenAI configuration:
74
74
| Endpoint | This is the API endpoint (for example, `https://api.openai.com/v1`) |
75
75
| Token | This is the access token to authorize your API call. <br />To get an API, follow these steps:<ol><li>Create an account and sign in at [OpenAI](https://platform.openai.com/).</li><li> Go to the [API key page](https://platform.openai.com/account/api-keys) to create a new secret key. </li><li>Copy the API key and save this somewhere safe.</li></ol> |
| Display name | This is the name identifier of a configuration (for example, *MyConfiguration*). |
84
-
| API type | Select `AzureOpenAI`. |
85
-
| Endpoint | This is the API endpoint (for example, `https://your-resource-name.openai.azure.com/openai/deployments/`).<br />For details on how to obtain `your-resource-name`, see the [Obtaining Azure OpenAI Resource Name](#azure-resource-name) section below. |
84
+
| API type | Select `AzureOpenAI` for both Azure OpenAI and Microsoft Foundry deployments. |
85
+
| Endpoint | This is the API endpoint (for example, `https://your-resource-name.openai.azure.com/openai/deployments/`).<br />For details on how to obtain `your-resource-name`, see the [Obtaining Resource Name](#azure-resource-name) section below. |
86
86
| Azure key type | This is the type of token that is entered in the API key field. For Azure OpenAI, two types of keys are currently supported: Microsoft Entra token and API key. <br />For details on how to generate a Microsoft Entra access token, see [How to Configure Azure OpenAI Service with Managed Identities](https://learn.microsoft.com/en-gb/azure/ai-services/openai/how-to/managed-identity). Alternatively, if your organization allows it, you could use the Azure `api-key` authentication mechanism. For details on how to obtain an API key, see the [Obtaining Azure OpenAI API keys](#azure-api-keys) section below. For more information, see the [Technical Reference](#technical-reference) section. |
87
87
| Token / API key | This is the access token to authorize your API call. |
88
88
89
-
##### Obtaining the Azure OpenAI Resource Name {#azure-resource-name}
89
+
##### Obtaining the Resource Name {#azure-resource-name}
90
90
91
-
1. Go to the [Azure OpenAI portal](https://oai.azure.com/) and sign in.
92
-
2. In the upper-right corner, next to your Avatar, click on the scope dropdown.
93
-
3. The tab shows your Directory, Subscription, and Azure OpenAI resource.
94
-
4. Make sure the right Azure OpenAI resource is selected.
95
-
5. Use the copy icon ({{% icon name="copy" %}}) and use it as your resource name in the endpoint URL.
91
+
1. Go to the [Microsoft Foundry portal](https://ai.azure.com/) and sign in.
92
+
2. Select the right resource in the upper right corner.
93
+
3. The home page should show `Resource configuration` where you can find the `Azure OpenAI endpoint`
94
+
4. Use the copy icon ({{% icon name="copy" %}}) and use it as your resource name in the endpoint URL.
96
95
97
-
##### Obtaining the Azure OpenAI API Keys {#azure-api-keys}
96
+
##### Obtaining API Keys {#azure-api-keys}
98
97
99
-
1. Go to the [Azure OpenAI portal](https://oai.azure.com/) and sign in.
100
-
2. In the upper-right corner, next to your Avatar, click on the scope dropdown.
101
-
3. The tab shows your Directory, Subscription, and Azure OpenAI resource.
102
-
4. Make sure the right Azure OpenAI resource is selected.
103
-
5. You can now view ({{% icon name="view" %}}) and copy ({{% icon name="copy" %}}) the value of the **key1** or **key2** field as your API key while setting up the configuration. Note that these keys might not be visible for everyone in the Azure OpenAI Portal, depending on your organization's security settings.
98
+
1. On the same page where the resource name is located, you can find your API key information.
99
+
2. You can now view ({{% icon name="view" %}}) and copy ({{% icon name="copy" %}}) the value of the **key1** or **key2** field as your API key while setting up the configuration. Note that these keys might not be visible for everyone in the portal, depending on your organization's security settings.
104
100
105
101
##### Adding Azure AI Search Resources {#azure-ai-search}
106
102
@@ -119,15 +115,15 @@ Currently, the only supported authorization method for Azure AI Search resources
119
115
120
116
#### Configuring the OpenAI Deployed Models
121
117
122
-
A [Deployed Model](/appstore/modules/genai/genai-for-mx/commons/#deployed-model) represents a GenAI model instance that can be used by the app to generate text, embeddings, or images. For every model you want to invoke from your app, you need to create a `OpenAIDeployedModel` record, a specialization of `DeployedModel`. In addition to the model display name and a technical name/identifier, an OpenAI deployed model contains a reference to the additional connection details as configured in the previous step. For OpenAI, a set of common models will be prepopulated automatically upon saving the configuration. If you want to use additional models that are made available by OpenAI you need to configure additional OpenAI deployed models in your Mendix app. For Azure OpenAI no deployed models are created by default. The technical model names depend on the deployment names that were chosen while deploying the models in the [Azure Portal](https://oai.azure.com/resource/deployments). Therefore in this case you always need to configure the deployed models manually in your Mendix app.
118
+
A [Deployed Model](/appstore/modules/genai/genai-for-mx/commons/#deployed-model) represents a GenAI model instance that can be used by the app to generate text, embeddings, or images. For every model you want to invoke from your app, you need to create a `OpenAIDeployedModel` record, a specialization of `DeployedModel`. In addition to the model display name and a technical name/identifier, an OpenAI deployed model contains a reference to the additional connection details as configured in the previous step. For OpenAI, a set of common models can be created automatically using the designated button. If you want to use additional models that are made available by OpenAI you need to configure additional OpenAI deployed models in your Mendix app. For Microsoft Foundry the model names can be different. The technical model names depend on the deployment names that were chosen while deploying the models in the [Microsoft Foundry portal](https://ai.azure.com/). Therefore in this case you always need to configure the deployed models manually in your Mendix app.
123
119
124
120
1. If needed, click the three dots for an OpenAI configuration to open the "Manage Deployed Models" pop-up.
125
121
2. For every additional model, add a record. The following fields are required:
| Display name | This is the reference to the model for app users in case they have to select which one is to be used. |
130
-
| Deployment name / Model name | This is the technical reference for the model. For OpenAI this is equal to the [model aliases](https://platform.openai.com/docs/models#current-model-aliases). For Azure OpenAI this is the deployment name from the [Azure Portal](https://oai.azure.com/resource/deployments).
126
+
| Deployment name / Model name | This is the technical reference for the model. For OpenAI this is equal to the [model aliases](https://platform.openai.com/docs/models#current-model-aliases). For Microsoft Foundry this is the deployment name from the [Microsoft Foundry portal](https://ai.azure.com/).
131
127
| Output modality| Describes what the output of the model is. This connector currently supports Text, Embedding, and Image.
132
128
| Input modality| Describes what input modalities are accepted by the model. This connector currently supports Text and Image.
133
129
| Azure API version | Azure OpenAI only. This is the API version to use for this operation. It follows the `yyyy-MM-dd` format. For supported versions, see [Azure OpenAI documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/reference). The supported versions can vary depending on the type of model, so make sure to look for the right section (such as Chat Completions, Image Generation, or Embeddings) on that page. |
@@ -196,7 +192,7 @@ For `Chat Completions with History`, `FileCollection` can optionally be added to
196
192
Use the two OpenAI-specific microflow actions from the toolbox [Files: Initialize Collection with OpenAI File](#initialize-filecollection) and [Files: Add OpenAIFile to Collection](#add-file) to construct the input with either `FileDocuments` (for vision, it needs to be of type `Image`) or `URLs`. There are similar file operations exposed by the GenAI commons module that can be used for vision requests with the OpenAIConnector; however, these generic operations do not support the optional OpenAI-specific `Detail` attribute.
197
193
198
194
{{% alert color="info" %}}
199
-
OpenAI and Azure OpenAI for vision do not necessarily all models provide feature parity when it comes to combining functionalities. In other words, Azure OpenAI does not support the use of JSON mode and function calling in combination with image (vision) input for certain models, so make sure to check the [Azure Documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models).
195
+
OpenAI and Microsoft Foundry do not necessarily provide feature parity across all models when it comes to combining functionalities. In other words, Microsoft Foundry does not support the use of JSON mode and function calling in combination with image (vision) input for certain models, so make sure to check the [Azure Documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models).
200
196
201
197
When you use Azure OpenAI, it is recommended to set the optional `MaxTokens` input parameter; otherwise, the return output may be cut off.
202
198
{{% /alert %}}
@@ -212,7 +208,7 @@ For [Chat Completions (without history)](/appstore/modules/genai/genai-for-mx/co
212
208
You can send up to 100 pages across multiple files, with a maximum combined size of 32 MB per conversation. Currently, processing multiple files with OpenAI is not always guaranteed and can lead to unexpected behavior (for example, only one file being processed).
213
209
214
210
{{% alert color="info" %}}
215
-
Azure OpenAI does not currently support file input.
211
+
Microsoft Foundry does not currently support file input.
216
212
217
213
Note that the model uses the file name when analyzing documents, which may introduce a potential vulnerability to prompt injection. To reduce this risk, consider modifying the string or not passing it at all.
218
214
{{% /alert %}}
@@ -323,13 +319,13 @@ To check your JDK version and update it if necessary, follow these steps:
323
319
2. You may also need to update Gradle. To do this, go to **Edit** > **Preferences** > **Deployment** > **Gradle directory**. Click **Browse** and select the appropriate Gradle version from the Mendix folder. For Mendix 10.10 and above, use Gradle 8.5. For Mendix 10 versions below 10.10, use Gradle 7.6.3. Then save your settings by clicking **OK**.
324
320
3. Rerun the project.
325
321
326
-
### Chat Completions with Vision and JSON Mode (Azure OpenAI)
322
+
### Chat Completions with Vision and JSON Mode (Microsoft Foundry)
327
323
328
-
Azure OpenAI does not support the use of JSON mode and function calling in combination with image (vision) input and will return a `400 - model error`. Make sure the optional input parameters `ResponseFormat` and `FunctionCollection` are set to `empty` for all chat completion operations if you want to use vision with Azure OpenAI.
324
+
Microsoft Foundry does not support the use of JSON mode and function calling in combination with image (vision) input and will return a `400 - model error`. Make sure the optional input parameters `ResponseFormat` and `ToolCollection` are set to `empty` for all chat completion operations if you want to use vision with Microsoft Foundry.
329
325
330
-
### Chat Completions with Vision Response is Cut Off (Azure OpenAI)
326
+
### Chat Completions with Vision Response is Cut Off (Microsoft Foundry)
331
327
332
-
When you use Azure OpenAI, it is recommended to set the optional `MaxTokens` input parameter; otherwise, the response may be cut off. For more details, see the [Azure OpenAI Documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/gpt-with-vision?tabs=rest%2Csystem-assigned%2Cresource#call-the-chat-completion-apis).
328
+
When you use Microsoft Foundry, it is recommended to set the optional `MaxTokens` input parameter; otherwise, the response may be cut off. For more details, see the [Azure OpenAI Documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/gpt-with-vision?tabs=rest%2Csystem-assigned%2Cresource#call-the-chat-completion-apis).
333
329
334
330
### Attribute or Reference Required Error Message After Upgrade
335
331
@@ -342,9 +338,9 @@ If you encounter an error caused by conflicting Java libraries, such as `java.la
*[Introduction to Prompt Engineering – Microsoft Azure Documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/prompt-engineering)
346
-
*[Prompt Engineering Techniques – Microsoft Azure Documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/advanced-prompt-engineering?pivots=programming-language-chat-completions)
341
+
*[Introduction to Prompt Engineering – Microsoft Foundry Documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/prompt-engineering)
342
+
*[Prompt Engineering Techniques – Microsoft Foundry Documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/advanced-prompt-engineering?pivots=programming-language-chat-completions)
347
343
*[ChatGPT Prompt Engineering for Developers - DeepLearning.AI](https://www.deeplearning.ai/short-courses/chatgpt-prompt-engineering-for-developers)
0 commit comments