Skip to content

Commit a3eb14d

Browse files
guimachiavelliguimachiavelli
committed
add multimodal search basic documentation
1 parent d53b3c8 commit a3eb14d

4 files changed

Lines changed: 186 additions & 4 deletions

File tree

‎learn/resources/experimental_features_overview.mdx‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,3 +59,4 @@ Activating or deactivating experimental features this way does not require you t
5959
| [Search query embedding cache](/learn/self_hosted/configure_meilisearch_at_launch#search-query-embedding-cache) | Enable a cache for search query embeddings | CLI flag or environment variable |
6060
| [Uncompressed snapshots](/learn/self_hosted/configure_meilisearch_at_launch#uncompressed-snapshots) | Disable snapshot compaction | CLI flag or environment variable |
6161
| [Maximum batch payload size](/learn/self_hosted/configure_meilisearch_at_launch#maximum-batch-payload-size) | Limit batch payload size | CLI flag or environment variable |
62+
| [Multimodal search](/references/api/settings#indexingfragments) | Enable multimodal search | API route |

‎reference/api/experimental_features.mdx‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,8 @@ The experimental API route is not compatible with all experimental features. Con
2626
"containsFilter": false,
2727
"editDocumentsByFunction": false,
2828
"network": false,
29-
"chatCompletions": false
29+
"chatCompletions": false,
30+
"multimodal": false
3031
}
3132
```
3233

@@ -38,6 +39,7 @@ The experimental API route is not compatible with all experimental features. Con
3839
| **`editDocumentsByFunction`** | Boolean | `true` if feature is active, `false` otherwise |
3940
| **`network`** | Boolean | `true` if feature is active, `false` otherwise |
4041
| **`chatCompletions`** | Boolean | `true` if feature is active, `false` otherwise |
42+
| **`multimodal`** | Boolean | `true` if feature is active, `false` otherwise |
4143

4244
## Get all experimental features
4345

@@ -58,7 +60,8 @@ Get a list of all experimental features that can be activated via the `/experime
5860
"containsFilter": false,
5961
"editDocumentsByFunction": false,
6062
"network": false,
61-
"chatCompletions": false
63+
"chatCompletions": false,
64+
"multimodal": false
6265
}
6366
```
6467

@@ -87,6 +90,7 @@ Setting a field to `null` leaves its value unchanged.
8790
"containsFilter": false,
8891
"editDocumentsByFunction": false,
8992
"network": false,
90-
"chatCompletions": false
93+
"chatCompletions": false,
94+
"multimodal": false
9195
}
9296
```

‎reference/api/search.mdx‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,7 @@ By default, [this endpoint returns a maximum of 1000 results](/learn/resources/k
9797
| **[`vector`](#vector)** | Array of numbers | `null` | Search using a custom query vector |
9898
| **[`retrieveVectors`](#display-_vectors-in-response)** | Boolean | `false` | Return document vector data |
9999
| **[`locales`](#query-locales)** | Array of strings | `null` | Explicitly specify languages used in a query |
100+
| **[`media`](#media)** | Object | `null` | Perform AI-powered search queries with multimodal content |
100101

101102
### Response
102103

@@ -1283,3 +1284,63 @@ For full control over the way Meilisearch detects languages during indexing and
12831284
…
12841285
}
12851286
```
1287+
1288+
### Media <NoticeTag type="experimental" label="experimental" />
1289+
1290+
**Parameter**: `media`<br />
1291+
**Expected value**: Object<br />
1292+
**Default value**: `null`
1293+
1294+
<Note>
1295+
This is an experimental feature. Use the Meilisearch Cloud UI or the experimental features endpoint to activate it:
1296+
1297+
```sh
1298+
curl \
1299+
-X PATCH 'MEILISEARCH_URL/experimental-features/' \
1300+
-H 'Content-Type: application/json' \
1301+
--data-binary '{
1302+
"multimodal": true
1303+
}'
1304+
```
1305+
</Note>
1306+
1307+
Specifies data to populate search fragments when performing multimodal searches.
1308+
1309+
`media` must be an object whose fields must correspond to the data required by one search fragment. `media` must match a single search fragment. If `media` matches more than one fragment or no search fragments at all, Meilisearch will return an error.
1310+
1311+
It is mandatory to specify an embedder when using `media`.
1312+
1313+
#### Example
1314+
1315+
<CodeSamplesSearchParameterReferenceLocales1 />
1316+
1317+
```sh
1318+
curl -X POST 'MEILISEARCH_URL/indexes/INDEX_NAME/search' \
1319+
-H 'content-type: application/json' \
1320+
--data-binary '{
1321+
"hybrid": {
1322+
"embedder": "EMBEDDER_NAME"
1323+
},
1324+
"media": {
1325+
"FIELD_A": "VALUE_A",
1326+
"FIELD_B" : {
1327+
"FIELD_C": "VALUE_B"
1328+
"FIELD_D": "VALUE_C"
1329+
}
1330+
}
1331+
}'
1332+
```
1333+
1334+
```json
1335+
{
1336+
"hits": [
1337+
{
1338+
"id": 0,
1339+
"title": "DOCUMENT NAME",
1340+
…
1341+
}
1342+
…
1343+
],
1344+
…
1345+
}
1346+
```

‎reference/api/settings.mdx‎

Lines changed: 117 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2512,7 +2512,9 @@ These embedder objects may contain the following fields:
25122512
| **`binaryQuantized`** | Boolean | Empty | Once set to `true`, irreversibly converts all vector dimensions to 1-bit values |
25132513
| **`indexingEmbedder`** | Object | Empty | Configures embedder to vectorize documents during indexing |
25142514
| **`searchEmbedder`** | Object | Empty | Configures embedder to vectorize search queries |
2515-
| **`pooling`** | String | `"useModel"` | Pooling method for Hugging Face embedders |
2515+
| **`pooling`** | String | `"useModel"` | Pooling method for Hugging Face embedders |
2516+
| **`indexingFragments`** | Object | Empty | Configures a set of fields for multimodal embedding generation |
2517+
| **`searchFragments`** | Object | Empty | Configures a set of fields for use in multimodal search |
25162518

25172519
### Get embedder settings
25182520

@@ -2875,6 +2877,120 @@ Both fields must be an object and accept the same fields as a regular embedder,
28752877

28762878
`indexingEmbedder` and `searchEmbedder` are incompatible with all other embedder sources.
28772879

2880+
##### `indexingFragments` <NoticeTag type="experimental" label="experimental" />
2881+
2882+
<Note>
2883+
This is an experimental feature. Use the Meilisearch Cloud UI or the experimental features endpoint to activate it:
2884+
2885+
```sh
2886+
curl \
2887+
-X PATCH 'MEILISEARCH_URL/experimental-features/' \
2888+
-H 'Content-Type: application/json' \
2889+
--data-binary '{
2890+
"multimodal": true
2891+
}'
2892+
```
2893+
</Note>
2894+
2895+
Use `indexingFragments` and `searchFragments` to set up Meilisearch to perform multimodal searches.
2896+
2897+
`indexingFragments` specifies which fields in your documents should be used during multimodal embedding generation. It must be an object with the following structure:
2898+
2899+
```json
2900+
"FRAGMENT_NAME": {
2901+
"value": {
2902+
…
2903+
}
2904+
}
2905+
```
2906+
2907+
`FRAGMENT_NAME` can be any valid string. It must contain a single field, `value`. `value` must then follow your chosen model's specifications.
2908+
2909+
For example, for Voyage AI's text and image embedder, `value` must be an array containing a single object with a `type` field. Depending on `type`'s value, you must include either `text`, `image_url`, or `image_base64`:
2910+
2911+
```json
2912+
{
2913+
"FRAGMENT_NAME_A": {
2914+
"value": {
2915+
"content": [
2916+
{
2917+
"type": "text",
2918+
"text": "A document called {{doc.title}} that can be described as {{doc.description}}"
2919+
}
2920+
]
2921+
}
2922+
},
2923+
"FRAGMENT_NAME_B": {
2924+
"value": {
2925+
"content": [
2926+
{
2927+
"type": "image_url",
2928+
"image_url": "{{doc.image_url}}"
2929+
}
2930+
]
2931+
}
2932+
},
2933+
}
2934+
```
2935+
2936+
Use Liquid templates to interpolate document data into the fragment fields, where `doc` gives you access to all fields within a document.
2937+
2938+
`indexingFragments` is optional when using the `rest` source.
2939+
2940+
`indexingFragments` is incompatible with all other embedder sources.
2941+
2942+
Specifying a `documentTemplate` in an embedder using `indexingFragments` will result in an error.
2943+
2944+
You must specify at least one valid fragment in `searchFragments` when using `indexingFragments`.
2945+
2946+
##### `searchFragments` <NoticeTag type="experimental" label="experimental" />
2947+
2948+
<Note>
2949+
This is an experimental feature. Use the Meilisearch Cloud UI or the experimental features endpoint to activate it:
2950+
2951+
```sh
2952+
curl \
2953+
-X PATCH 'MEILISEARCH_URL/experimental-features/' \
2954+
-H 'Content-Type: application/json' \
2955+
--data-binary '{
2956+
"multimodal": true
2957+
}'
2958+
```
2959+
</Note>
2960+
2961+
`searchFragments` instructs Meilisearch how to parse fields present in a query's `media` search parameter. It must be an object following the same structure as the `indexerFragments` object:
2962+
2963+
```json
2964+
"FRAGMENT_NAME": {
2965+
"value": {
2966+
…
2967+
}
2968+
}
2969+
```
2970+
2971+
As with `indexingFragments`, the content of `value` should follow your model's specification.
2972+
2973+
Use Liquid templates to interpolate search query data into the fragment fields, where `media` gives you access to all fields specified in the query's `media` field:
2974+
2975+
```json
2976+
"SEARCH_FRAGMENT_A": {
2977+
"value": {
2978+
"content": [
2979+
{
2980+
"type": "image_base64",
2981+
"image_base64": "data:{{media.image.mime}};base64,{{media.image.data}}"
2982+
}
2983+
]
2984+
}
2985+
},
2986+
```
2987+
2988+
`searchFragments` is optional when using the `rest` source.
2989+
2990+
`searchFragments` is incompatible with all other embedder sources.
2991+
2992+
You must specify at least one valid fragment in `indexingFragments` when using `searchFragments`.
2993+
28782994
#### Example
28792995

28802996
<CodeSamplesUpdateEmbedders1 />

0 commit comments

Comments
 (0)