Prototype of Elasticsearch-based full text search solution (#1528)

* Renamed existing natural language service stuff

ElasticSearch first steps:
 - followed footsepts of natural language search components
 - makefile: ElasticSearch start, stop, and clean actions
 - achieved connectivity to started ElasticSearch cluster
 - achieved connectivity to self-launched child container service
 - test GQL endpoint, routing cluster health info for now
 - indexer: scaffold, no real indexing yet
 - search service API: scaffold, no real implementation yet

Full text search:
 - support framework for entity schemas registration (no versioning/migrations yet)
 - ES: first client calls for index check, registration, documents counting
 - feeding schemas for accounts and datasets from corresponding domains (very simplified fields now)
 - entire process happens via plugin style, partiular domains register provider components that are accessed by template shared process

ElasticSearch code structure extended:
- separated low level operations in `ElasticSearchClient`: deals with engine connection building, sending queries in the right format, interpreting responses, and dealing with API errors
- `ElasticSearchIndexMappings` handles creation of index mappings for the given schema + hashing it's content, this will be a future place to apply complex column properties depending on configs
- `ElasticSearchVersionedEntityIndex` manages indexes for entities and aliases, auto-registers indcies, validates schema metadata, automatically detects drifts without version modification, automatically applies breaking or reindsable upgrades
- main repository code stays at very high level

Basic shape of full ElasticSearch index re-indexing + sketched simplest indexing procedure for Datasets

Indexing owner-id in datasets index (for filters)

ElasticSearch indexing added for Accounts

Indexing creation time for Accounts/Datasets

Indexing dataset documents similarly to natural language seach: added schema fields, description, keywords, and attachments

More realstic field roles: hierarchical identifiers (account name,  dataset name, alias, schema field), prose (description, attachments), keywords (owner_id, keyword, dataset_kind) - with  corresponding analyzers and properties for ElasticSearch

Added "Title" field role, which is in between Prose and Identifier, using for account's display names for now.
Identifier fields get inner-ngrams (3..6) and wider edge-ngrams (2..10).

Account life cycle events update ElasticSearch index:
 - massaged events format a bit to satisfy new needs
 - new outbox event handler for account search index updates
 -  reorganized account schema code to encapsulate 1 document operations, while indexer and update handler use it's helpers
 -  issuing bulk insert, update, delete operations in ES for account events

Dummy implementation in e2e tests (until a better solution is found, as containerized ES starts for over 20s per each command, and that's affected by acocunt/dataset lifecycle events)

Implemented updates to ElasticSearch for dataset-related events: lifecycle, reference update, parent account rename/delete.
Fixed account deletion handler in datasets domain, no ReBAC/dangling checks should be executed during system event handling.

Datasets schema: better incremental re-indexings for partial updates

Hotfix: improved detection of invalid intervals in case of breaking changes in the dataset, when expected tail is ahead of head

First sketching of a search function:
 - simplest querying: query_string vs match_all, depending if non-empty query was received
 - support specifying list of indexes vs defaulting to all schemas
 - ES: sending search request, decoding response
 - trivial GQL endpoint support

Naive pagination support (size/from).
Requesting source fields in multiple modes: None, All, Particular, Complex (include+exclude patterns)

Next ElasticSearch steps:
- Search schemas constants moved and published by domains, so that GQL can reference those fields.
- Support flexible sorting of search results: N criterias, by field or relevance score, configurable direction.
- Each schema now provides a field that can be used for universal alphabetial sorting ("title" alias)

Support basic search filters (keyword = value, keyword in {values}) and compound from those (and, or, not).

Added convenience macros to specify compound filters and for sort specifications.

Search highlights for textual fields: displays best fragments explaining why certain document's field matched the query

On-demand "explain" option: outputs low-level ElasticSearch scoring computation formula

* ES: support unprocessed objects field (stored, but not indexed or searched)

* Supporting boolean fields + added a generic banning feature for ES indices (filter is auto-attached to "read alias").
Fix: search should always be directed to "read alias", never to "writable index".

* Merge corrections

* Corrections in ES client: use single bulk update operation with encoded comands, so that heterogenous ops  are possible in one bulk.

Simplified search context: we don't need account for now.

* Lock correction

* Backported ElasticSearch-focused changes from Molecule branch

* Deps correction

* Simplifying renames

* Unified account/dataset schemas to the style in Molecule branch

* Prototyped framework for integration tests with ElasticSearch involved:
 - EsTestContext: main facility, lazily initializes reusable ES client
 - a test proc-macro hiding the plumbing of the context
 - each test receives a dill::Catalog prefilled with ES client, ES repository impl, with unique randomly generated index prefix
 - a succesful test automatically cleans it's own indices, while a failing test keeps the indices available for inspection
 - on first ES client initialization, the potentially abandoned test indices from previous sessions are discarded automatically
 - written first couple tests for Accounts indexing

* Correct spelling of "Elasticsearch" brand name

* ElasticSearch test group on CI/CD

* More account indexing tests

* Initial test suite for datasets indexing.

Hardening es_client against async waiting issues: created index must be reachable, assigned alias must be reachable.

Makefile: automated cleaning of abandoned test artifacts from previous experiments. Not doing this in fixtures, as `cargo nextest run` creates races around it executing every test in separate process.

* Common searching harness + aligning dataset use case harness to be more pluggable

* Shared fixture for account use case tests + indexing tests.
Accounts indexing: testing predefined indexer

* Tests: predefined datasets indexing

* MT version of predefiend datasets indexing test

* MT version of incremental indexing

* Tests: renaming or deleting account affects index of it's datasets

* Udeps fixed

* Improvements and tests for detailed dataset content indexing (schema, setInfo, attachments).

Not indexing default vocabulary fields in schema, as those do not contribute to search relevance.

Indexing tests run with real outbox to maximize realism: forcing sync when necessary between test steps

* Added basic test suide for datasets searching: checking analyzers, filters, stemmers, ...

* Merged useful testability changes from Molecule prototype

* Elasticsearch: abiility to setup, connect to, and test with a server using HTTPS/TLS

* udeps fix

* Admin:
 - endpoint to force reset search indices
 - temporarily hidden querying full text GQL endpoint under admin's guard

* minor rename

* Makefiule actions to start Qdrant container

* QDrant parity with ES approaches:
 - dummy implementation
 - lazy init via background catalog
 - passing explicit context for queries

* changelog

* DEVELOPER.md notes on how to use Elasticsearch

* v0.256.0

Author: zaychenko-sergei

.config/nextest.toml

@@ -8,6 +8,7 @@ setup = { max-threads = 1 }  # Serialize the setup steps
 containerized = { max-threads = 8 }  # Don't use too much memory
 heavy-engine = { max-threads = 2 }  # Heavy engine tests are very memory-hungry
 database = { max-threads = 8 }  # Don't use too much memory
+elasticsearch = { max-threads = 4 }  # Elasticsearch tests can be parallelized moderately
 
 # NOTE: Periodic missing rows when the system is under load
 #       https://github.com/kamu-data/kamu-engine-risingwave/issues/7
@@ -31,6 +32,10 @@ test-group = "heavy-engine"
 filter = "test(::database::)"
 test-group = "database"
 
+[[profile.default.overrides]]
+filter = "test(::elasticsearch::)"
+test-group = "elasticsearch"
+
 [[profile.default.overrides]]
 filter = "test(::flaky::)"
 retries = { count = 3, backoff = "exponential", delay = "3s" }

.github/workflows/build.yaml

@@ -95,23 +95,27 @@ jobs:
             runs-on: ubuntu-latest
             cargo-flags: --profile ci --features ingest-ftp,web-ui
             nextest-flags: --cargo-profile ci --features ingest-ftp,web-ui
-            nextest-exclusions-main-set-tests: -E '!(test(::database::) | test(::spark::) | test(::flink::))'
+            nextest-exclusions-main-set-tests: -E '!(test(::database::) | test(::elasticsearch::) | test(::spark::) | test(::flink::))'
             nextest-exclusions-database-set-part: ''
+            nextest-exclusions-elasticsearch-set-part: ''
             container-runtime: podman
             ipfs: true
             database: true
+            elasticsearch: true
             maximize-space: true
           # TODO: M1 cannot run docker due to lack of nested virtualization
           # See: https://github.com/marketplace/actions/setup-docker-on-macos#arm64-processors-m1-m2-m3-series-used-on-macos-14-images-are-unsupported
           - name: MacOS / arm64
             runs-on: macos-15
             cargo-flags: --profile ci
             nextest-flags: --cargo-profile ci
-            nextest-exclusions-main-set-tests: -E '!(test(::containerized::) | test(::database::))'
+            nextest-exclusions-main-set-tests: -E '!(test(::containerized::) | test(::database::) | test(::elasticsearch::))'
             nextest-exclusions-database-set-part: '& !(test(::containerized::))'
+            nextest-exclusions-elasticsearch-set-part: '& !(test(::containerized::))'
             container-runtime: ''
             ipfs: false
             database: false
+            elasticsearch: false
             maximize-space: false
           # TODO: DataFusion tests are temporarily disabled
           # See: https://github.com/kamu-data/kamu-cli/issues/226
@@ -120,11 +124,13 @@ jobs:
             cargo-flags: --profile ci
             nextest-flags: --cargo-profile ci
             # platform(target) is a workaround for Windows only issue in cargo-nextest upon proc macro crates
-            nextest-exclusions-main-set-tests: -E 'platform(target) & !(test(::containerized::) | test(::datafusion::) | test(::database::))'
+            nextest-exclusions-main-set-tests: -E 'platform(target) & !(test(::containerized::) | test(::datafusion::) | test(::database::) | test(::elasticsearch::))'
             nextest-exclusions-database-set-part: '& !(test(::datafusion::))'
+            nextest-exclusions-elasticsearch-set-part: '& !(test(::datafusion::))'
             container-runtime: ''
             ipfs: false
             database: false
+            elasticsearch: false
             maximize-space: false
     name: Test / ${{ matrix.name }}
     runs-on: ${{ matrix.runs-on }}
@@ -144,6 +150,21 @@ jobs:
           MARIADB_DATABASE: kamu-test
         ports:
           - 3306:3306
+      elasticsearch:
+        image: ${{ matrix.elasticsearch && 'elasticsearch:9.2.1' || '' }}
+        env:
+          discovery.type: single-node
+          ES_JAVA_OPTS: -Xms1g -Xmx1g
+          xpack.security.enabled: "false"
+          xpack.security.http.ssl.enabled: "false"
+          xpack.security.transport.ssl.enabled: "false"
+          xpack.ml.enabled: "false"
+          ingest.geoip.downloader.enabled: "false"
+          node.store.allow_mmap: "false"
+          cluster.routing.allocation.disk.threshold_enabled: "false"
+        ports:
+          - 9200:9200
+          - 9300:9300
     steps:
       - name: Setup IPFS
         uses: ibnesayeed/setup-ipfs@master
@@ -194,18 +215,54 @@ jobs:
       - name: Run main set of tests
         run: cargo nextest run ${{ matrix.nextest-flags }} ${{ matrix.nextest-exclusions-main-set-tests }} -v
 
+      - name: Wait for Elasticsearch (stable)
+        if: matrix.elasticsearch
+        run: |
+          set -euo pipefail
+
+          # Require 5 consecutive successful responses.
+          ok=0
+          for i in {1..180}; do
+            if curl -fsS --max-time 2 "http://127.0.0.1:9200/_cluster/health?wait_for_status=yellow&timeout=1s" >/dev/null; then
+              ok=$((ok+1))
+              echo "ES health ok ($ok/5) at attempt $i"
+              if [ "$ok" -ge 5 ]; then
+                echo "ES is stable and healthy"
+                exit 0
+              fi
+            else
+              ok=0
+            fi
+            sleep 1
+          done
+
+          echo "ES did not become stable"
+          docker logs --tail 300 "${{ job.services.elasticsearch.id }}" || true
+          exit 1
+
+      - name: Elasticsearch tests
+        if: matrix.elasticsearch
+        env:
+          ELASTICSEARCH_URL: http://127.0.0.1:9200
+        run: |
+          # Show actual disk headroom ES sees
+          curl -fsS 'http://127.0.0.1:9200/_cat/nodes?v&h=name,ip,disk.used_percent,disk.avail,disk.total'
+
+          # Run ES related tests
+          cargo nextest run ${{ matrix.nextest-flags }} -E 'test(::elasticsearch::)' ${{ matrix.nextest-exclusions-elasticsearch-set-part }}
+
       - name: Postgres database tests
         if: matrix.database
         env:
           SQLX_OFFLINE: false
-          DATABASE_URL: postgres://root:root@localhost:5432/kamu-test
+          DATABASE_URL: postgres://root:root@127.0.0.1:5432/kamu-test
         run: cargo nextest run ${{ matrix.nextest-flags }} -E 'test(::postgres::) ${{ matrix.nextest-exclusions-database-set-part }}'
 
       - name: MariaDB database tests
         if: matrix.database
         env:
           SQLX_OFFLINE: false
-          DATABASE_URL: mariadb://root:root@localhost:3306/kamu-test
+          DATABASE_URL: mariadb://root:root@127.0.0.1:3306/kamu-test
         run: cargo nextest run ${{ matrix.nextest-flags }} -E 'test(::mysql::) ${{ matrix.nextest-exclusions-database-set-part }}'
 
       # Not running on windows due to line ending differences

.gitignore

@@ -7,3 +7,4 @@ target/
 
 .env
 /kamu.sqlite.db*
+.local/

CHANGELOG.md

@@ -11,6 +11,44 @@ Recommendation: for ease of reading, use the following order:
 - Fixed
 -->
 
+## [0.256.0] - 2026-01-08
+### Added
+- Prototyped full-text search capabilities via `Elasticsearch`:
+  - can be connected to external instance or launch an embedded container
+  - prototyped schema descriptors, supporting multiple field types with different search capabilities, such as :
+     - prose
+     - identifiers
+     - keywords
+     - dates
+     - numbers, 
+     - booleans
+     - ...
+  - designed search schemas for accounts and datasets
+  - automatic index creations and alias rotations for search schemas, with detection of mappings drift
+  - accounts and datasets are indexed automatically on first launch or after explicit reset,
+     as well as react to lifecycle changes incrementally (via handling outbox events)
+  - prototyped query language, allowing:
+     - textual prompts (simplistic, using `simple_query_string` algorithm for now)
+     - filter expressions: ==, !=, <, >, <=, >=, in {}, prefix
+     - filter combinations: AND, OR, NOT
+     - sorting criterias: by field or by relevance, any combination
+     - pagination controls
+  - supporting fields sourcing (returning original document)
+  - supporting highlights (explains which field matched)
+  - supporting debug explanations of scoring
+  - predefined capatilities regardless of schema:
+     - title-based sorting (each schema specifies a field that should act like a title)
+     - banning field (when set, the document is auto-excluded from read aliases matching search schemas)
+  - testing mechanisms with Elasticsearch:
+     - integration tests: creating temporary indices with a random prefix, cleaned at shutdown
+     - e2e tests
+     - CI support (separate `elasticsearch` test group, starting container on Github)
+     - written solid coverage for accounts & datasets indexing and searching logic
+  - temporary GQL endpoint for testing (gated with admin guard)
+  - GQL endpoint to force reindexing at run-time
+### Fixed
+- Qdrant initialization works similarly to Elasticsearch solution, and avoids excessive reindexing  
+
 ## [0.255.1] - 2026-01-05
 ### Changed
 - Upgraded `prost`, `tonic`, `flat`, `opentelemetry` to latest versions