Add Kestra integration guides to Getting Started

config/navigation.json

@@ -57,7 +57,22 @@
                 "getting_started/integrations/postman",
                 "getting_started/integrations/meilisearch_importer",
                 "getting_started/integrations/mcp",
-                "getting_started/integrations/langchain"
+                "getting_started/integrations/langchain",
+                {
+                  "group": "Kestra",
+                  "pages": [
+                    "getting_started/integrations/kestra/overview",
+                    "getting_started/integrations/kestra/postgresql",
+                    "getting_started/integrations/kestra/mongodb",
+                    "getting_started/integrations/kestra/amazon_s3",
+                    "getting_started/integrations/kestra/kafka",
+                    "getting_started/integrations/kestra/rest_api",
+                    "getting_started/integrations/kestra/elasticsearch",
+                    "getting_started/integrations/kestra/opensearch",
+                    "getting_started/integrations/kestra/rabbitmq",
+                    "getting_started/integrations/kestra/shopify"
+                  ]
+                }
               ]
             }
           ]

getting_started/integrations/kestra/amazon_s3.mdx

@@ -0,0 +1,151 @@
+---
+title: Connect Amazon S3 to Meilisearch with Kestra
+sidebarTitle: Amazon S3
+description: Backfill and event-driven sync of Amazon S3 objects into Meilisearch with Kestra.
+---
+
+A huge amount of the world's data arrives as files in object storage: nightly exports, partner data drops, analytics dumps, catalog CSVs. Amazon S3 (and every S3-compatible store, such as MinIO, Cloudflare R2, or Google Cloud Storage in interop mode) is where they land. Meilisearch is where you want them searchable. This guide connects the two with [Kestra](https://kestra.io).
+
+This guide covers both halves of the real problem: a one-shot load of an existing object, and then an **event-driven** pipeline where dropping a new file into a bucket makes its contents searchable within seconds. No manual step or polling script required.
+
+## Why orchestrate the sync
+
+Files arrive unpredictably and formats vary. You want a pipeline that reacts to new files automatically, converts whatever format they're in, indexes them reliably, and, crucially, processes each file exactly once. Kestra gives you a bucket trigger, format converters, and the Meilisearch task, wired together declaratively with full logging and retries.
+
+## Prerequisites
+
+A running Kestra with three plugins (Meilisearch, AWS, and the serdes plugin for CSV/JSON conversion), plus a [Meilisearch Cloud](https://www.meilisearch.com/cloud?utm_campaign=oss&utm_source=docs&utm_medium=kestra-integration) project. Only Kestra runs locally, since Meilisearch is managed:
+
+```yaml
+services:
+  kestra:
+    image: kestra/kestra:latest
+    command: server local
+    ports: ["8080:8080"]
+    environment:
+      # your Meilisearch Cloud Default Admin API key, base64-encoded
+      SECRET_MEILISEARCH_API_KEY: <base64 of your admin API key>
+```
+
+```dockerfile
+FROM kestra/kestra:latest
+RUN /app/kestra plugins install \
+      io.kestra.plugin:plugin-meilisearch:LATEST \
+      io.kestra.plugin:plugin-aws:LATEST \
+      io.kestra.plugin:plugin-serdes:LATEST
+```
+
+<Note>
+**Get your Cloud credentials.** In the [Meilisearch Cloud](https://cloud.meilisearch.com) dashboard, create a project and copy its **Project URL** (the `url` in the flows below) and its **Default Admin API Key** (Settings, then API Keys). Store both AWS and Meilisearch credentials as Kestra secrets. This guide shows an S3-compatible endpoint (MinIO) with inline keys for clarity. For real AWS S3, drop `endpointOverride` / `compatibilityMode` / `forcePathStyle` and supply `accessKeyId` / `secretKeyId` (or an IAM role) via secrets.
+</Note>
+
+The examples index a `games.csv` file with columns `id,title,platform,genre,rating`.
+
+## Step 1: The first load (backfill)
+
+Three steps: download the object, convert CSV to Kestra's ION format, and index it. The serdes plugin bridges the format gap: `DocumentAdd` speaks ION, and `CsvToIon` produces exactly that.
+
+```yaml
+id: s3_csv_to_meilisearch
+namespace: company.search
+
+variables:
+  meilisearch_url: https://ms-xxxxxxxxxxxx-xxxx.meilisearch.io   # your Meilisearch Cloud Project URL
+  index: games
+
+tasks:
+  - id: download
+    type: io.kestra.plugin.aws.s3.Download
+    accessKeyId: minioadmin
+    secretKeyId: minioadmin
+    region: us-east-1
+    endpointOverride: http://minio:9000   # omit for real AWS S3
+    compatibilityMode: true               # omit for real AWS S3
+    forcePathStyle: true                  # omit for real AWS S3
+    bucket: datasets
+    key: games.csv
+
+  - id: to_ion
+    type: io.kestra.plugin.serdes.csv.CsvToIon
+    from: "{{ outputs.download.uri }}"
+
+  - id: index_documents
+    type: io.kestra.plugin.meilisearch.DocumentAdd
+    from: "{{ outputs.to_ion.uri }}"
+    index: "{{ vars.index }}"
+    url: "{{ vars.meilisearch_url }}"
+    key: "{{ secret('MEILISEARCH_API_KEY') }}"
+```
+
+<Warning>
+**S3-compatible storage gotcha.** For MinIO, R2, and friends you need both `compatibilityMode: true` and `forcePathStyle: true`. Without them the AWS SDK uses virtual-host addressing (`bucket.your-endpoint`) and fails on DNS resolution. On real AWS S3, leave all three lines out.
+</Warning>
+
+Swap `CsvToIon` for `JsonToIon` or `AvroToIon` if your files arrive in those formats. The rest of the pipeline is identical.
+
+One thing to know about CSV: `CsvToIon` emits every column as a **string** (`"rating":"96"`). If you want to filter or sort numerically in Meilisearch, either cast the values in a transform step, or configure the attribute accordingly and rely on Meilisearch's numeric handling.
+
+## Step 2: Event-driven sync (files as they arrive)
+
+The backfill indexes a file you name explicitly. The real workflow is: a new file lands in the bucket and gets indexed on its own. Kestra's S3 `Trigger` polls a prefix and starts an execution whenever new objects appear, and it can move or delete each object after it's handed off, giving you **exactly-once** processing.
+
+Put incoming files under an `incoming/` prefix and let the trigger drain it:
+
+```yaml
+id: s3_event_to_meilisearch
+namespace: company.search
+
+variables:
+  meilisearch_url: https://ms-xxxxxxxxxxxx-xxxx.meilisearch.io   # your Meilisearch Cloud Project URL
+  index: games
+
+triggers:
+  - id: on_new_file
+    type: io.kestra.plugin.aws.s3.Trigger
+    interval: PT10S            # poll the prefix every 10 seconds
+    accessKeyId: minioadmin
+    secretKeyId: minioadmin
+    region: us-east-1
+    endpointOverride: http://minio:9000
+    compatibilityMode: true
+    forcePathStyle: true
+    bucket: datasets
+    prefix: incoming/
+    action: DELETE            # remove each object once handed to the flow
+
+tasks:
+  - id: to_ion
+    type: io.kestra.plugin.serdes.csv.CsvToIon
+    from: "{{ trigger.objects[0].uri }}"
+
+  - id: index_documents
+    type: io.kestra.plugin.meilisearch.DocumentAdd
+    from: "{{ outputs.to_ion.uri }}"
+    index: "{{ vars.index }}"
+    url: "{{ vars.meilisearch_url }}"
+    key: "{{ secret('MEILISEARCH_API_KEY') }}"
+```
+
+How it behaves: the trigger checks `incoming/` every ten seconds. When a file appears, it downloads it into Kestra's internal storage (available as `{{ trigger.objects[0].uri }}`), fires the flow, and then deletes the object from the bucket per `action: DELETE`. The flow converts and indexes it. Drop a CSV, and its rows are searchable seconds later, hands-off.
+
+Prefer to keep an audit trail of processed files? Use `action: MOVE` with a `moveTo` destination to archive each object into a `processed/` prefix instead of deleting it.
+
+## Handling updates and deletes
+
+Object drops are naturally an **upsert** stream: because `DocumentAdd` is add-or-replace, a file re-exported with corrected rows overwrites the matching documents by primary key when it's dropped again. No special handling needed for updates.
+
+Deletes are the one case files don't express well: a file that simply stops appearing can't tell Meilisearch to remove anything. Two options:
+
+- Include a `deleted` marker column in your exports and add a branch that calls Meilisearch's `documents/delete-batch` endpoint for those ids (the pattern is shown in [Connect PostgreSQL to Meilisearch with Kestra](/getting_started/integrations/kestra/postgresql)).
+- For full-snapshot files, periodically re-index into a fresh index and swap it in with an index alias, so removed rows disappear.
+
+## Going to production
+
+- **Real AWS S3:** remove `endpointOverride`, `compatibilityMode`, and `forcePathStyle`. Authenticate with an IAM role or with `accessKeyId` / `secretKeyId` pulled from Kestra secrets.
+- **Large files:** `CsvToIon` and `DocumentAdd` stream through internal storage and batch automatically, so multi-gigabyte files work without tuning. Raise `DocumentAdd`'s `batchSize` if you want fewer, larger indexing tasks.
+- **Multiple files at once:** the trigger surfaces every matched object in `{{ trigger.objects }}`. Loop over them with an `EachSequential`/`ForEach` task if a poll can pick up more than one file.
+- **Retries:** add a `retry` block so a transient S3 or Meilisearch hiccup self-heals rather than failing the execution.
+
+## Wrap-up
+
+Two flows turn object storage into a live search source: a backfill for files already in the bucket, and an event-driven pipeline where new drops are converted and indexed automatically, each processed exactly once. It works identically on Amazon S3 and any S3-compatible store. Point the trigger at your bucket and let Kestra do the rest.

getting_started/integrations/kestra/elasticsearch.mdx

@@ -0,0 +1,117 @@
+---
+title: Migrate Elasticsearch to Meilisearch with Kestra
+sidebarTitle: Elasticsearch
+description: Migrate an existing Elasticsearch index into Meilisearch with a safe, re-runnable Kestra workflow.
+---
+
+You're running Elasticsearch for search, and it's become more than you need: a JVM cluster to babysit, relevance tuning that fights you, and a bill that grows with every shard. Meilisearch gives you instant, typo-tolerant, relevance-ranked search out of the box, and moving to it doesn't have to be a risky big-bang rewrite.
+
+This guide shows a clean migration from an existing Elasticsearch index to Meilisearch using [Kestra](https://kestra.io): a one-flow backfill of your whole index, plus a safe cutover strategy that lets you run both engines in parallel until you're confident.
+
+## Why do the migration in Kestra
+
+You could write a script that scrolls Elasticsearch and pushes to Meilisearch, until it dies halfway through a ten-million-document index with no way to resume and no record of what transferred. Kestra makes the migration an observable, retryable workflow: the extract streams to disk, indexing batches and waits for completion, and every run is logged. You can re-run it safely as many times as you need during cutover.
+
+## Prerequisites
+
+A running Kestra with the Meilisearch and Elasticsearch plugins, plus a [Meilisearch Cloud](https://www.meilisearch.com/cloud?utm_campaign=oss&utm_source=docs&utm_medium=kestra-integration) project. Only Kestra runs locally, since Meilisearch is managed:
+
+```yaml
+services:
+  kestra:
+    image: kestra/kestra:latest
+    command: server local
+    ports: ["8080:8080"]
+    environment:
+      # your Meilisearch Cloud Default Admin API key, base64-encoded
+      SECRET_MEILISEARCH_API_KEY: <base64 of your admin API key>
+```
+
+```dockerfile
+FROM kestra/kestra:latest
+RUN /app/kestra plugins install \
+      io.kestra.plugin:plugin-meilisearch:LATEST \
+      io.kestra.plugin:plugin-elasticsearch:LATEST
+```
+
+<Note>
+**Get your Cloud credentials.** In the [Meilisearch Cloud](https://cloud.meilisearch.com) dashboard, create a project and copy its **Project URL** (the `url` in the flows below) and its **Default Admin API Key** (Settings, then API Keys). Store the key as the `MEILISEARCH_API_KEY` Kestra secret.
+</Note>
+
+This guide migrates a `movies` index whose documents look like `{ "id": 1, "title": "Inception", "year": 2010, "genre": "sci-fi" }`.
+
+## Step 1: Backfill the whole index
+
+The Elasticsearch plugin's `Scroll` task walks an entire index using the scroll API and writes every document to an ION file in Kestra's internal storage, exactly the format the Meilisearch `DocumentAdd` task consumes. Two tasks move your whole index:
+
+```yaml
+id: elasticsearch_to_meilisearch
+namespace: company.search
+
+variables:
+  meilisearch_url: https://ms-xxxxxxxxxxxx-xxxx.meilisearch.io   # your Meilisearch Cloud Project URL
+  index: movies
+
+tasks:
+  - id: scroll
+    type: io.kestra.plugin.elasticsearch.Scroll
+    connection:
+      hosts:
+        - http://elasticsearch:9200
+      # for Elastic Cloud, add basicAuth:
+      #   basicAuth:
+      #     username: elastic
+      #     password: "{{ secret('ES_PASSWORD') }}"
+    indexes:
+      - movies
+    request:
+      query:
+        match_all: {}
+
+  - id: index_documents
+    type: io.kestra.plugin.meilisearch.DocumentAdd
+    from: "{{ outputs.scroll.uri }}"
+    index: "{{ vars.index }}"
+    url: "{{ vars.meilisearch_url }}"
+    key: "{{ secret('MEILISEARCH_API_KEY') }}"
+```
+
+`Scroll` streams the result set to disk rather than holding it in memory, so this scales from five documents to fifty million without changing anything. `DocumentAdd` then batches the documents (1000 per request by default) and waits for Meilisearch to finish indexing, failing the run if any batch fails, so a partial migration surfaces loudly instead of silently.
+
+Meilisearch uses each document's `id` field as its primary key. If your Elasticsearch documents keep their identifier only in `_id` (not inside `_source`), add a JSONata `TransformItems` step to copy it into a real field before indexing.
+
+Run the flow once and your entire index is searchable in Meilisearch.
+
+## Step 2: Cut over safely
+
+The value of doing this in a workflow is a gradual migration rather than a leap. A safe cutover looks like:
+
+1. **Backfill** into Meilisearch with the flow above.
+2. **Dual-run.** Point a copy of your search UI (or a feature-flagged path) at Meilisearch while production still serves from Elasticsearch. Compare relevance and latency on real queries.
+3. **Keep Meilisearch fresh during the overlap.** Re-run the backfill on a schedule, or narrow it to recent changes if your documents carry an `updated_at` field. Swap the `match_all` for a range query so each run only scrolls what changed:
+
+   ```yaml
+       request:
+         query:
+           range:
+             updated_at:
+               gte: "now-15m"
+   ```
+
+   Because `DocumentAdd` is add-or-replace, re-indexing the same document just overwrites it by primary key, so overlapping runs are always safe.
+4. **Flip the switch.** Once you trust the results, point production at Meilisearch and decommission the Elasticsearch cluster.
+
+## A note on deletes
+
+For a one-time migration, deletes don't matter, since you're taking a snapshot. If you dual-run for a while and documents get deleted in Elasticsearch during the overlap, the cleanest way to reconcile is to index each fresh backfill into a **new** Meilisearch index and then repoint an alias at it, so anything absent from the latest scroll simply disappears. Kestra can run that index-then-swap as a two-step flow.
+
+## Going to production
+
+- **Elastic Cloud / secured clusters:** add `basicAuth` (or an API key header) to the `connection`, with the password pulled from a Kestra secret. Set `trustAllSsl: true` only for self-signed dev clusters.
+- **Reshape while you migrate.** A migration is a good moment to clean up your schema. Use a JSONata `TransformItems` step between `scroll` and `index_documents` to rename fields, flatten nesting, or drop what search doesn't need.
+- **Configure Meilisearch settings first.** Define your searchable, filterable, and sortable attributes on the target index (an `http.Request` to the settings API) before the backfill, so the first indexing pass already ranks well.
+- **Retries.** Add a `retry` block to the tasks so a transient cluster hiccup during a long scroll is retried rather than failing the whole migration.
+
+## Wrap-up
+
+Migrating off Elasticsearch is two tasks (`Scroll` to export, `DocumentAdd` to index) wrapped in a workflow you can re-run safely while you dual-run and build confidence. Kestra handles the streaming, batching, and observability, so you get a gradual, reversible path from Elasticsearch to Meilisearch instead of a big-bang rewrite.