Skip to content

Commit 946721e

Browse files
Merge pull request mendix#9032 from EmilyvonBergen/HowTo-FunctionCalling
Improvement in GenAI Docs structure and addition of two documentations
2 parents 942d212 + 82f4d9b commit 946721e

29 files changed

Lines changed: 430 additions & 101 deletions

File tree

content/en/docs/appstore/use-content/platform-supported-content/modules/aws/amazon-bedrock.md

Lines changed: 12 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -107,7 +107,7 @@ You can follow a similar approach to implement any of the other operations in **
107107

108108
### Chatting with Large Language Models using the ChatCompletions Operation
109109

110-
A common use case of the Amazon Bedrock Connector is the development of chatbots and chat solutions. The **ChatCompletions (without history / with history)** operations offer an easy way to connect to most of the text-generation models available on Amazon Bedrock. The ChatCompletions operations are built on top of Bedrock's Converse API, allowing you to talk to different models without the need of a model-specific implementation. For more information on the ChatCompletion operations, see [GenAI Commons: Chat Completions](/appstore/modules/genai/commons/#genai-generate).
110+
A common use case of the Amazon Bedrock Connector is the development of chatbots and chat solutions. The **ChatCompletions (without history / with history)** operations offer an easy way to connect to most of the text-generation models available on Amazon Bedrock. The ChatCompletions operations are built on top of Bedrock's Converse API, allowing you to talk to different models without the need of a model-specific implementation. For more information on the ChatCompletion operations, see [GenAI Commons: Chat Completions](/appstore/modules/genai/genai-for-mx/commons/#genai-generate).
111111

112112
For an overview of supported models and model-specific capabilities and limitations, see [Amazon Bedrock Converse API](https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference.html#conversation-inference-supported-models-features) in the AWS documentation.
113113

@@ -187,14 +187,14 @@ To invoke a Bedrock agent for your Mendix app, do the following steps:
187187

188188
### Token Usage {#tokenusage}
189189

190-
[Token usage](/appstore/modules/genai/commons/#token-usage) monitoring is now possible for the following operations:
190+
[Token usage](/appstore/modules/genai/genai-for-mx/commons/#token-usage) monitoring is now possible for the following operations:
191191

192192
* Chat Completions with History
193193
* Chat Completion without History
194194
* Embeddings with Cohere Embed
195195
* Embeddings with Amazon Titan Embeddings
196196

197-
For more information about using this feature, refer to the [GenAI commons documentation](/appstore/modules/genai/commons/#token-usage).
197+
For more information about using this feature, refer to the [GenAI commons documentation](/appstore/modules/genai/genai-for-mx/commons/#token-usage).
198198

199199
## Technical Reference {#technical-reference}
200200

@@ -213,20 +213,20 @@ For additional information about available operations, refer to the sections bel
213213

214214
#### ChatCompletions (With History) and ChatCompletions (Without History) {#chat-completions}
215215

216-
The [ChatCompletions (with history)](/appstore/modules/genai/commons/#chat-completions-with-history) and [ChatCompletions (without history)](/appstore/modules/genai/commons/#chat-completions-without-history) activities can be used with a variety of supported LLMs.
216+
The [ChatCompletions (with history)](/appstore/modules/genai/genai-for-mx/commons/#chat-completions-with-history) and [ChatCompletions (without history)](/appstore/modules/genai/genai-for-mx/commons/#chat-completions-without-history) activities can be used with a variety of supported LLMs.
217217

218218
Some capabilities of the chat completions operations are currently only available for specific models:
219219

220-
* **Function Calling** - You can use function calling in all chat completions operations using a [supported model](https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference-supported-models-features.html) by adding a `ToolCollection` with a `Tool` via the [Tools: Add Function to Request](/appstore/modules/genai/commons/#add-function-to-request) operation. For more information about function calling, see the [Function Calling Documentation](/appstore/modules/genai/function-calling/).
220+
* **Function Calling** - You can use function calling in all chat completions operations using a [supported model](https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference-supported-models-features.html) by adding a `ToolCollection` with a `Tool` via the [Tools: Add Function to Request](/appstore/modules/genai/genai-for-mx/commons/#add-function-to-request) operation. For more information about function calling, see the [Function Calling Documentation](/appstore/modules/genai/function-calling/).
221221

222222
**Function calling microflows**: A microflow used as a tool for function calling must satisfy the following conditions:
223223

224224
1. One input parameter of type String or no input parameter.
225225
2. Return value of type String.
226226

227-
* **Vision** - This operation supports the *vision* capability for [supported models](https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference-supported-models-features.html). With vision, you can send image prompts, in addition to the traditional text prompts. You can use vision by adding a `FileCollection` with a `File` to the `Message` using the [Files: Initialize Collection with File](/appstore/modules/genai/commons/#initialize-filecollection) or the [Files: Add to Collection](/appstore/modules/genai/commons/#add-file-to-collection) operation. Make sure to set the `FileType` attribute to **image**.
227+
* **Vision** - This operation supports the *vision* capability for [supported models](https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference-supported-models-features.html). With vision, you can send image prompts, in addition to the traditional text prompts. You can use vision by adding a `FileCollection` with a `File` to the `Message` using the [Files: Initialize Collection with File](/appstore/modules/genai/genai-for-mx/commons/#initialize-filecollection) or the [Files: Add to Collection](/appstore/modules/genai/genai-for-mx/commons/#add-file-to-collection) operation. Make sure to set the `FileType` attribute to **image**.
228228

229-
* **Document Chat** - This operation supports the ability to chat with documents for [supported models](https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference-supported-models-features.html). To send a document to the model add a `FileCollection` with a `System.FileDocument` to the `Message` using the [Files: Initialize Collection with File](/appstore/modules/genai/commons/#initialize-filecollection) or the [Files: Add to Collection](/appstore/modules/genai/commons/#add-file-to-collection) operation. For Document Chat, it is not supported to create a `FileContent` from an URL using the above mentioned operations; Please use the `System.FileDocument` option. Make sure to set the `FileType` attribute to **document**.
229+
* **Document Chat** - This operation supports the ability to chat with documents for [supported models](https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference-supported-models-features.html). To send a document to the model add a `FileCollection` with a `System.FileDocument` to the `Message` using the [Files: Initialize Collection with File](/appstore/modules/genai/genai-for-mx/commons/#initialize-filecollection) or the [Files: Add to Collection](/appstore/modules/genai/genai-for-mx/commons/#add-file-to-collection) operation. For Document Chat, it is not supported to create a `FileContent` from an URL using the above mentioned operations; Please use the `System.FileDocument` option. Make sure to set the `FileType` attribute to **document**.
230230

231231
#### RetrieveAndGenerate {#retrieve-and-generate}
232232

@@ -270,25 +270,25 @@ The history can be enabled using the `SessionId` parameter on the RetrieveAndGen
270270
This activity was introduced in Amazon Bedrock Connector version 3.1.0.
271271
{{% /alert %}}
272272

273-
The [Generate Image](/appstore/modules/genai/commons/#generate-image) operation can be used to generate one or more images. Currently *Amazon Titan Image Generator G1* is the only supported model for image generation of the Amazon Bedrock Connector.
273+
The [Generate Image](/appstore/modules/genai/genai-for-mx/commons/#generate-image) operation can be used to generate one or more images. Currently *Amazon Titan Image Generator G1* is the only supported model for image generation of the Amazon Bedrock Connector.
274274

275-
`GenAICommons.ImageOptions` can be an empty object. If provided, it allows you to set additional options for Image Generation and can be created by using the [Image: Create Options](/appstore/modules/genai/commons/#imageoptions-create) operation of GenAI Commons.
275+
`GenAICommons.ImageOptions` can be an empty object. If provided, it allows you to set additional options for Image Generation and can be created by using the [Image: Create Options](/appstore/modules/genai/genai-for-mx/commons/#imageoptions-create) operation of GenAI Commons.
276276

277-
To retrieve actual image objects from the response, you can use the [Image: Get Generated Image (Single)](/appstore/modules/genai/commons/#image-get-single) or [Image: Get Generated Images (List)](/appstore/modules/genai/commons/#image-get-list) helper operations from GenAI Commons.
277+
To retrieve actual image objects from the response, you can use the [Image: Get Generated Image (Single)](/appstore/modules/genai/genai-for-mx/commons/#image-get-single) or [Image: Get Generated Images (List)](/appstore/modules/genai/genai-for-mx/commons/#image-get-list) helper operations from GenAI Commons.
278278

279279
For Titan Image models, the `Image Generation: Add Titan Image Extension` operation can be used to configure Titan image-specific values (currently only *NegativeText*).
280280

281281
#### Generate Embeddings (String) {#embeddings-single-string}
282282

283-
The [Generate Embeddings (String)](/appstore/modules/genai/commons/#embeddings-string) activity can be used to generate an embedding vector for a given input string with one of the Cohere Embed models or Titan Embeddings v2.
283+
The [Generate Embeddings (String)](/appstore/modules/genai/genai-for-mx/commons/#embeddings-string) activity can be used to generate an embedding vector for a given input string with one of the Cohere Embed models or Titan Embeddings v2.
284284

285285
For Cohere Embed and Titan Embeddings, the request can be associated to their respective EmbeddingsOptions extension object which can be created with the [Embeddings Options: Add Cohere Embed Extension](#add-cohere-embed-extension) or [Embeddings Options: Add Titan Embeddings Extension](#add-titan-embeddings-extension) operation. Through this extension, it is possible to tailor the operation to more specific needs.
286286

287287
Currently, embeddings are available for the Cohere Embed family and or Titan Embeddings v2.
288288

289289
#### Generate Embeddings (Chunk Collection) {#embeddings-chunk-collection}
290290

291-
The [Generate Embeddings (Chunk Collection)](/appstore/modules/genai/commons/#embeddings-chunk-collection) activity can be used to generate a collection of embedding vectors for a given collection of text chunks with one of the Cohere Embed models or Titan Embeddings v2.
291+
The [Generate Embeddings (Chunk Collection)](/appstore/modules/genai/genai-for-mx/commons/#embeddings-chunk-collection) activity can be used to generate a collection of embedding vectors for a given collection of text chunks with one of the Cohere Embed models or Titan Embeddings v2.
292292

293293
For each model family, the request can be associated to an extension of the EmbeddingsOptions object which can be created with either the [Embeddings Options: Add Cohere Embed Extension](#add-cohere-embed-extension) or the [Embeddings Options: Add Titan Embeddings Extension](#add-titan-embeddings-extension) operation. Through this extension, it is possible to tailor the operation to more specific needs.
294294

content/en/docs/appstore/use-content/platform-supported-content/modules/genai/concepts/_index.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
title: "Get Started with Generative AI"
33
url: /appstore/modules/genai/get-started/
44
linktitle: "Get Started with GenAI"
5-
weight: 10
5+
weight: 20
66
description: "Describes the concepts behind generative AI and what you might implement with it."
77
aliases:
88
- /appstore/modules/genai/using-gen-ai/

content/en/docs/appstore/use-content/platform-supported-content/modules/genai/concepts/prompt-engineering.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -37,9 +37,9 @@ A user prompt is another fundamental type. It is the user’s input, question, o
3737

3838
### Context Prompt
3939

40-
Depending on the project or use case, adding contextual information to the model may be necessary. Normally, this information, called context prompt or conversation history, is sent in the same interaction as the system and user prompt. It captures the historical information of the conversation to maintain coherence with the end user and be context aware. In the Mendix app chatbot setup, developers configure this within their application, and it is included in the request sent to the LLM using the [Chat Completions (with history)](/appstore/modules/genai/commons/#chat-completions-with-history) operation.
40+
Depending on the project or use case, adding contextual information to the model may be necessary. Normally, this information, called context prompt or conversation history, is sent in the same interaction as the system and user prompt. It captures the historical information of the conversation to maintain coherence with the end user and be context aware. In the Mendix app chatbot setup, developers configure this within their application, and it is included in the request sent to the LLM using the [Chat Completions (with history)](/appstore/modules/genai/genai-for-mx/commons/#chat-completions-with-history) operation.
4141

42-
To understand this concept, imagine a user interacting with a chatbot while asking, *How should I start?*. If in previous interactions, the user asked about Mendix, the LLM will understand that the question refers to the Mendix apps. In cases where the context is not needed, such as in command-based interactions where the inquiry could be: *Turn on the lights* and the LLM does not need any historical conversation, developers can use operations like [Chat Completions (without history)](/appstore/modules/genai/commons/#chat-completions-without-history).
42+
To understand this concept, imagine a user interacting with a chatbot while asking, *How should I start?*. If in previous interactions, the user asked about Mendix, the LLM will understand that the question refers to the Mendix apps. In cases where the context is not needed, such as in command-based interactions where the inquiry could be: *Turn on the lights* and the LLM does not need any historical conversation, developers can use operations like [Chat Completions (without history)](/appstore/modules/genai/genai-for-mx/commons/#chat-completions-without-history).
4343

4444
## Typical Components of a Prompt
4545

content/en/docs/appstore/use-content/platform-supported-content/modules/genai/concepts/rag-example-implementation.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ Every LLM will have its algorithm for generating vectors, but the convention is
2525

2626
#### Chunk
2727

28-
In the context of GenAI Commons in a Mendix app, embedding vectors are generated using a [Chunk](/appstore/modules/genai/commons/#chunk-entity). Each object represents a discrete piece of information and contains its original string representation, as well as (after the embedding operation) the vector representation of that string according to the LLM of choice.
28+
In the context of GenAI Commons in a Mendix app, embedding vectors are generated using a [Chunk](/appstore/modules/genai/genai-for-mx/commons/#chunk-entity). Each object represents a discrete piece of information and contains its original string representation, as well as (after the embedding operation) the vector representation of that string according to the LLM of choice.
2929

3030
#### Knowledge base
3131

@@ -35,11 +35,11 @@ In the context of GenAI Commons in a Mendix app, we use the [PgVector Knowledge
3535

3636
#### Knowledge base chunk
3737

38-
In most use cases, more information needs to be stored than just the original input string and its vector representation. A [KnowledgeBaseChunk](/appstore/modules/genai/commons/#knowledgebasechunk-entity) is an extension of [Chunk](/appstore/modules/genai/commons/#chunk-entity) that can hold additional information that is typically required for useful insertion and retrieval from a Mendix application.
38+
In most use cases, more information needs to be stored than just the original input string and its vector representation. A [KnowledgeBaseChunk](/appstore/modules/genai/genai-for-mx/commons/#knowledgebasechunk-entity) is an extension of [Chunk](/appstore/modules/genai/genai-for-mx/commons/#chunk-entity) that can hold additional information that is typically required for useful insertion and retrieval from a Mendix application.
3939

4040
#### Metadata
4141

42-
If additional conventional filtering is needed during similarity searches, such additional data can be stored in the knowledge base as well. [Metadata](/appstore/modules/genai/commons/#metadata-entity) objects are key-value pairs that are inserted along with the chunks and contain this additional information. The filtering is applied on an exact string-match basis for the key-value pair. Records are only retrieved if they match all records of the metadata in the collection provided as part of the search step.
42+
If additional conventional filtering is needed during similarity searches, such additional data can be stored in the knowledge base as well. [Metadata](/appstore/modules/genai/genai-for-mx/commons/#metadata-entity) objects are key-value pairs that are inserted along with the chunks and contain this additional information. The filtering is applied on an exact string-match basis for the key-value pair. Records are only retrieved if they match all records of the metadata in the collection provided as part of the search step.
4343

4444
{{% alert color="info" %}}The example described in the remainder of this document does not include the more advanced use case of metadata filtering nor does it cover the construction of complex input strings. If you want to see how this can work in practice, take a look at the *RAG with Semantic Search on Historical Data* example in the [GenAI Showcase app](https://marketplace.mendix.com/link/component/220475). {{% /alert %}}
4545

@@ -69,7 +69,7 @@ In summary, in the first step, you need to provide the private knowledge base, s
6969

7070
Before you start experimenting with the end-to-end process, make sure that you have access to a (remote) PostgreSQL database with the [pgvector](https://github.com/pgvector/pgvector) extension available. If you do not have one yet, [learn more](/appstore/modules/genai/pgvector-setup/) about how a PostgreSQL vector database can be set up to explore use cases with knowledge bases.
7171

72-
{{% alert color="info" %}}If you have access to an Amazon Web Services (AWS) account or Microsoft Azure account, Mendix recommends you use a setup described in the [Creating a PostgreSQL Database with Amazon RDS](/appstore/modules/genai/pgvector-setup/#aws-database-create) or [Managing a PostgreSQL Database with Microsoft Azure](/appstore/modules/genai/pgvector-setup/#azure-database) section. This is convenient, since these PostgreSQL databases in the cloud have the required pgvector extension available by default.{{% /alert %}}
72+
{{% alert color="info" %}}If you have access to an Amazon Web Services (AWS) account or Microsoft Azure account, Mendix recommends you use a setup described in the [Creating a PostgreSQL Database with Amazon RDS](/appstore/modules/genai/reference-guide/external-connectors/pgvector-setup/#aws-database-create) or [Managing a PostgreSQL Database with Microsoft Azure](/appstore/modules/genai/reference-guide/external-connectors/pgvector-setup/#azure-database) section. This is convenient, since these PostgreSQL databases in the cloud have the required pgvector extension available by default.{{% /alert %}}
7373

7474
### Steps {#steps}
7575

0 commit comments

Comments
 (0)