Skip to content

Commit b50db70

Browse files
qdequeleqdequele
authored
Merge branch 'main' into add-platform-documentation
2 parents 09927e6 + 3489d1f commit b50db70

11 files changed

Lines changed: 1581 additions & 1 deletion

File tree

‎config/navigation.json‎

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,22 @@
5757
"getting_started/integrations/postman",
5858
"getting_started/integrations/meilisearch_importer",
5959
"getting_started/integrations/mcp",
60-
"getting_started/integrations/langchain"
60+
"getting_started/integrations/langchain",
61+
{
62+
"group": "Kestra",
63+
"pages": [
64+
"getting_started/integrations/kestra/overview",
65+
"getting_started/integrations/kestra/postgresql",
66+
"getting_started/integrations/kestra/mongodb",
67+
"getting_started/integrations/kestra/amazon_s3",
68+
"getting_started/integrations/kestra/kafka",
69+
"getting_started/integrations/kestra/rest_api",
70+
"getting_started/integrations/kestra/elasticsearch",
71+
"getting_started/integrations/kestra/opensearch",
72+
"getting_started/integrations/kestra/rabbitmq",
73+
"getting_started/integrations/kestra/shopify"
74+
]
75+
}
6176
]
6277
}
6378
]

‎getting_started/integrations/kestra/amazon_s3.mdx‎

Lines changed: 151 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,151 @@
1+
---
2+
title: Connect Amazon S3 to Meilisearch with Kestra
3+
sidebarTitle: Amazon S3
4+
description: Backfill and event-driven sync of Amazon S3 objects into Meilisearch with Kestra.
5+
---
6+
7+
A huge amount of the world's data arrives as files in object storage: nightly exports, partner data drops, analytics dumps, catalog CSVs. Amazon S3 (and every S3-compatible store, such as MinIO, Cloudflare R2, or Google Cloud Storage in interop mode) is where they land. Meilisearch is where you want them searchable. This guide connects the two with [Kestra](https://kestra.io).
8+
9+
This guide covers both halves of the real problem: a one-shot load of an existing object, and then an **event-driven** pipeline where dropping a new file into a bucket makes its contents searchable within seconds. No manual step or polling script required.
10+
11+
## Why orchestrate the sync
12+
13+
Files arrive unpredictably and formats vary. You want a pipeline that reacts to new files automatically, converts whatever format they're in, indexes them reliably, and, crucially, processes each file exactly once. Kestra gives you a bucket trigger, format converters, and the Meilisearch task, wired together declaratively with full logging and retries.
14+
15+
## Prerequisites
16+
17+
A running Kestra with three plugins (Meilisearch, AWS, and the serdes plugin for CSV/JSON conversion), plus a [Meilisearch Cloud](https://www.meilisearch.com/cloud?utm_campaign=oss&utm_source=docs&utm_medium=kestra-integration) project. Only Kestra runs locally, since Meilisearch is managed:
18+
19+
```yaml
20+
services:
21+
kestra:
22+
image: kestra/kestra:latest
23+
command: server local
24+
ports: ["8080:8080"]
25+
environment:
26+
# your Meilisearch Cloud Default Admin API key, base64-encoded
27+
SECRET_MEILISEARCH_API_KEY: <base64 of your admin API key>
28+
```
29+
30+
```dockerfile
31+
FROM kestra/kestra:latest
32+
RUN /app/kestra plugins install \
33+
io.kestra.plugin:plugin-meilisearch:LATEST \
34+
io.kestra.plugin:plugin-aws:LATEST \
35+
io.kestra.plugin:plugin-serdes:LATEST
36+
```
37+
38+
<Note>
39+
**Get your Cloud credentials.** In the [Meilisearch Cloud](https://cloud.meilisearch.com) dashboard, create a project and copy its **Project URL** (the `url` in the flows below) and its **Default Admin API Key** (Settings, then API Keys). Store both AWS and Meilisearch credentials as Kestra secrets. This guide shows an S3-compatible endpoint (MinIO) with inline keys for clarity. For real AWS S3, drop `endpointOverride` / `compatibilityMode` / `forcePathStyle` and supply `accessKeyId` / `secretKeyId` (or an IAM role) via secrets.
40+
</Note>
41+
42+
The examples index a `games.csv` file with columns `id,title,platform,genre,rating`.
43+
44+
## Step 1: The first load (backfill)
45+
46+
Three steps: download the object, convert CSV to Kestra's ION format, and index it. The serdes plugin bridges the format gap: `DocumentAdd` speaks ION, and `CsvToIon` produces exactly that.
47+
48+
```yaml
49+
id: s3_csv_to_meilisearch
50+
namespace: company.search
51+
52+
variables:
53+
meilisearch_url: https://ms-xxxxxxxxxxxx-xxxx.meilisearch.io # your Meilisearch Cloud Project URL
54+
index: games
55+
56+
tasks:
57+
- id: download
58+
type: io.kestra.plugin.aws.s3.Download
59+
accessKeyId: minioadmin
60+
secretKeyId: minioadmin
61+
region: us-east-1
62+
endpointOverride: http://minio:9000 # omit for real AWS S3
63+
compatibilityMode: true # omit for real AWS S3
64+
forcePathStyle: true # omit for real AWS S3
65+
bucket: datasets
66+
key: games.csv
67+
68+
- id: to_ion
69+
type: io.kestra.plugin.serdes.csv.CsvToIon
70+
from: "{{ outputs.download.uri }}"
71+
72+
- id: index_documents
73+
type: io.kestra.plugin.meilisearch.DocumentAdd
74+
from: "{{ outputs.to_ion.uri }}"
75+
index: "{{ vars.index }}"
76+
url: "{{ vars.meilisearch_url }}"
77+
key: "{{ secret('MEILISEARCH_API_KEY') }}"
78+
```
79+
80+
<Warning>
81+
**S3-compatible storage gotcha.** For MinIO, R2, and friends you need both `compatibilityMode: true` and `forcePathStyle: true`. Without them the AWS SDK uses virtual-host addressing (`bucket.your-endpoint`) and fails on DNS resolution. On real AWS S3, leave all three lines out.
82+
</Warning>
83+
84+
Swap `CsvToIon` for `JsonToIon` or `AvroToIon` if your files arrive in those formats. The rest of the pipeline is identical.
85+
86+
One thing to know about CSV: `CsvToIon` emits every column as a **string** (`"rating":"96"`). If you want to filter or sort numerically in Meilisearch, either cast the values in a transform step, or configure the attribute accordingly and rely on Meilisearch's numeric handling.
87+
88+
## Step 2: Event-driven sync (files as they arrive)
89+
90+
The backfill indexes a file you name explicitly. The real workflow is: a new file lands in the bucket and gets indexed on its own. Kestra's S3 `Trigger` polls a prefix and starts an execution whenever new objects appear, and it can move or delete each object after it's handed off, giving you **exactly-once** processing.
91+
92+
Put incoming files under an `incoming/` prefix and let the trigger drain it:
93+
94+
```yaml
95+
id: s3_event_to_meilisearch
96+
namespace: company.search
97+
98+
variables:
99+
meilisearch_url: https://ms-xxxxxxxxxxxx-xxxx.meilisearch.io # your Meilisearch Cloud Project URL
100+
index: games
101+
102+
triggers:
103+
- id: on_new_file
104+
type: io.kestra.plugin.aws.s3.Trigger
105+
interval: PT10S # poll the prefix every 10 seconds
106+
accessKeyId: minioadmin
107+
secretKeyId: minioadmin
108+
region: us-east-1
109+
endpointOverride: http://minio:9000
110+
compatibilityMode: true
111+
forcePathStyle: true
112+
bucket: datasets
113+
prefix: incoming/
114+
action: DELETE # remove each object once handed to the flow
115+
116+
tasks:
117+
- id: to_ion
118+
type: io.kestra.plugin.serdes.csv.CsvToIon
119+
from: "{{ trigger.objects[0].uri }}"
120+
121+
- id: index_documents
122+
type: io.kestra.plugin.meilisearch.DocumentAdd
123+
from: "{{ outputs.to_ion.uri }}"
124+
index: "{{ vars.index }}"
125+
url: "{{ vars.meilisearch_url }}"
126+
key: "{{ secret('MEILISEARCH_API_KEY') }}"
127+
```
128+
129+
How it behaves: the trigger checks `incoming/` every ten seconds. When a file appears, it downloads it into Kestra's internal storage (available as `{{ trigger.objects[0].uri }}`), fires the flow, and then deletes the object from the bucket per `action: DELETE`. The flow converts and indexes it. Drop a CSV, and its rows are searchable seconds later, hands-off.
130+
131+
Prefer to keep an audit trail of processed files? Use `action: MOVE` with a `moveTo` destination to archive each object into a `processed/` prefix instead of deleting it.
132+
133+
## Handling updates and deletes
134+
135+
Object drops are naturally an **upsert** stream: because `DocumentAdd` is add-or-replace, a file re-exported with corrected rows overwrites the matching documents by primary key when it's dropped again. No special handling needed for updates.
136+
137+
Deletes are the one case files don't express well: a file that simply stops appearing can't tell Meilisearch to remove anything. Two options:
138+
139+
- Include a `deleted` marker column in your exports and add a branch that calls Meilisearch's `documents/delete-batch` endpoint for those ids (the pattern is shown in [Connect PostgreSQL to Meilisearch with Kestra](/getting_started/integrations/kestra/postgresql)).
140+
- For full-snapshot files, periodically re-index into a fresh index and swap it in with an index alias, so removed rows disappear.
141+
142+
## Going to production
143+
144+
- **Real AWS S3:** remove `endpointOverride`, `compatibilityMode`, and `forcePathStyle`. Authenticate with an IAM role or with `accessKeyId` / `secretKeyId` pulled from Kestra secrets.
145+
- **Large files:** `CsvToIon` and `DocumentAdd` stream through internal storage and batch automatically, so multi-gigabyte files work without tuning. Raise `DocumentAdd`'s `batchSize` if you want fewer, larger indexing tasks.
146+
- **Multiple files at once:** the trigger surfaces every matched object in `{{ trigger.objects }}`. Loop over them with an `EachSequential`/`ForEach` task if a poll can pick up more than one file.
147+
- **Retries:** add a `retry` block so a transient S3 or Meilisearch hiccup self-heals rather than failing the execution.
148+
149+
## Wrap-up
150+
151+
Two flows turn object storage into a live search source: a backfill for files already in the bucket, and an event-driven pipeline where new drops are converted and indexed automatically, each processed exactly once. It works identically on Amazon S3 and any S3-compatible store. Point the trigger at your bucket and let Kestra do the rest.

‎getting_started/integrations/kestra/elasticsearch.mdx‎

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
---
2+
title: Migrate Elasticsearch to Meilisearch with Kestra
3+
sidebarTitle: Elasticsearch
4+
description: Migrate an existing Elasticsearch index into Meilisearch with a safe, re-runnable Kestra workflow.
5+
---
6+
7+
You're running Elasticsearch for search, and it's become more than you need: a JVM cluster to babysit, relevance tuning that fights you, and a bill that grows with every shard. Meilisearch gives you instant, typo-tolerant, relevance-ranked search out of the box, and moving to it doesn't have to be a risky big-bang rewrite.
8+
9+
This guide shows a clean migration from an existing Elasticsearch index to Meilisearch using [Kestra](https://kestra.io): a one-flow backfill of your whole index, plus a safe cutover strategy that lets you run both engines in parallel until you're confident.
10+
11+
## Why do the migration in Kestra
12+
13+
You could write a script that scrolls Elasticsearch and pushes to Meilisearch, until it dies halfway through a ten-million-document index with no way to resume and no record of what transferred. Kestra makes the migration an observable, retryable workflow: the extract streams to disk, indexing batches and waits for completion, and every run is logged. You can re-run it safely as many times as you need during cutover.
14+
15+
## Prerequisites
16+
17+
A running Kestra with the Meilisearch and Elasticsearch plugins, plus a [Meilisearch Cloud](https://www.meilisearch.com/cloud?utm_campaign=oss&utm_source=docs&utm_medium=kestra-integration) project. Only Kestra runs locally, since Meilisearch is managed:
18+
19+
```yaml
20+
services:
21+
kestra:
22+
image: kestra/kestra:latest
23+
command: server local
24+
ports: ["8080:8080"]
25+
environment:
26+
# your Meilisearch Cloud Default Admin API key, base64-encoded
27+
SECRET_MEILISEARCH_API_KEY: <base64 of your admin API key>
28+
```
29+
30+
```dockerfile
31+
FROM kestra/kestra:latest
32+
RUN /app/kestra plugins install \
33+
io.kestra.plugin:plugin-meilisearch:LATEST \
34+
io.kestra.plugin:plugin-elasticsearch:LATEST
35+
```
36+
37+
<Note>
38+
**Get your Cloud credentials.** In the [Meilisearch Cloud](https://cloud.meilisearch.com) dashboard, create a project and copy its **Project URL** (the `url` in the flows below) and its **Default Admin API Key** (Settings, then API Keys). Store the key as the `MEILISEARCH_API_KEY` Kestra secret.
39+
</Note>
40+
41+
This guide migrates a `movies` index whose documents look like `{ "id": 1, "title": "Inception", "year": 2010, "genre": "sci-fi" }`.
42+
43+
## Step 1: Backfill the whole index
44+
45+
The Elasticsearch plugin's `Scroll` task walks an entire index using the scroll API and writes every document to an ION file in Kestra's internal storage, exactly the format the Meilisearch `DocumentAdd` task consumes. Two tasks move your whole index:
46+
47+
```yaml
48+
id: elasticsearch_to_meilisearch
49+
namespace: company.search
50+
51+
variables:
52+
meilisearch_url: https://ms-xxxxxxxxxxxx-xxxx.meilisearch.io # your Meilisearch Cloud Project URL
53+
index: movies
54+
55+
tasks:
56+
- id: scroll
57+
type: io.kestra.plugin.elasticsearch.Scroll
58+
connection:
59+
hosts:
60+
- http://elasticsearch:9200
61+
# for Elastic Cloud, add basicAuth:
62+
# basicAuth:
63+
# username: elastic
64+
# password: "{{ secret('ES_PASSWORD') }}"
65+
indexes:
66+
- movies
67+
request:
68+
query:
69+
match_all: {}
70+
71+
- id: index_documents
72+
type: io.kestra.plugin.meilisearch.DocumentAdd
73+
from: "{{ outputs.scroll.uri }}"
74+
index: "{{ vars.index }}"
75+
url: "{{ vars.meilisearch_url }}"
76+
key: "{{ secret('MEILISEARCH_API_KEY') }}"
77+
```
78+
79+
`Scroll` streams the result set to disk rather than holding it in memory, so this scales from five documents to fifty million without changing anything. `DocumentAdd` then batches the documents (1000 per request by default) and waits for Meilisearch to finish indexing, failing the run if any batch fails, so a partial migration surfaces loudly instead of silently.
80+
81+
Meilisearch uses each document's `id` field as its primary key. If your Elasticsearch documents keep their identifier only in `_id` (not inside `_source`), add a JSONata `TransformItems` step to copy it into a real field before indexing.
82+
83+
Run the flow once and your entire index is searchable in Meilisearch.
84+
85+
## Step 2: Cut over safely
86+
87+
The value of doing this in a workflow is a gradual migration rather than a leap. A safe cutover looks like:
88+
89+
1. **Backfill** into Meilisearch with the flow above.
90+
2. **Dual-run.** Point a copy of your search UI (or a feature-flagged path) at Meilisearch while production still serves from Elasticsearch. Compare relevance and latency on real queries.
91+
3. **Keep Meilisearch fresh during the overlap.** Re-run the backfill on a schedule, or narrow it to recent changes if your documents carry an `updated_at` field. Swap the `match_all` for a range query so each run only scrolls what changed:
92+
93+
```yaml
94+
request:
95+
query:
96+
range:
97+
updated_at:
98+
gte: "now-15m"
99+
```
100+
101+
Because `DocumentAdd` is add-or-replace, re-indexing the same document just overwrites it by primary key, so overlapping runs are always safe.
102+
4. **Flip the switch.** Once you trust the results, point production at Meilisearch and decommission the Elasticsearch cluster.
103+
104+
## A note on deletes
105+
106+
For a one-time migration, deletes don't matter, since you're taking a snapshot. If you dual-run for a while and documents get deleted in Elasticsearch during the overlap, the cleanest way to reconcile is to index each fresh backfill into a **new** Meilisearch index and then repoint an alias at it, so anything absent from the latest scroll simply disappears. Kestra can run that index-then-swap as a two-step flow.
107+
108+
## Going to production
109+
110+
- **Elastic Cloud / secured clusters:** add `basicAuth` (or an API key header) to the `connection`, with the password pulled from a Kestra secret. Set `trustAllSsl: true` only for self-signed dev clusters.
111+
- **Reshape while you migrate.** A migration is a good moment to clean up your schema. Use a JSONata `TransformItems` step between `scroll` and `index_documents` to rename fields, flatten nesting, or drop what search doesn't need.
112+
- **Configure Meilisearch settings first.** Define your searchable, filterable, and sortable attributes on the target index (an `http.Request` to the settings API) before the backfill, so the first indexing pass already ranks well.
113+
- **Retries.** Add a `retry` block to the tasks so a transient cluster hiccup during a long scroll is retried rather than failing the whole migration.
114+
115+
## Wrap-up
116+
117+
Migrating off Elasticsearch is two tasks (`Scroll` to export, `DocumentAdd` to index) wrapped in a workflow you can re-run safely while you dual-run and build confidence. Kestra handles the streaming, batching, and observability, so you get a gradual, reversible path from Elasticsearch to Meilisearch instead of a big-bang rewrite.

0 commit comments

Comments
 (0)