Refactor semantic chunker embedder integration

Author: robodev-r2d2

infrastructure/rag/values.yaml

@@ -340,6 +340,12 @@ adminBackend:
     chunker:
       CHUNKER_MAX_SIZE: 1000
       CHUNKER_OVERLAP: 300
+      CHUNKER_SEMANTIC_BREAKPOINT_THRESHOLD_TYPE: "percentile"
+      CHUNKER_SEMANTIC_BREAKPOINT_THRESHOLD: "95"
+      CHUNKER_SEMANTIC_BUFFER_SIZE: "1"
+      CHUNKER_SEMANTIC_MIN_CHUNK_SIZE: "200"
+      CHUNKER_SEMANTIC_MAX_CHUNK_SIZE: "1200"
+      CHUNKER_SEMANTIC_TRIM_CHUNKS: "true"
     keyValueStore:
       USECASE_KEYVALUE_PORT: 6379
       USECASE_KEYVALUE_HOST: "rag-keydb"

libs/README.md

@@ -13,7 +13,8 @@ It consists of the following python packages:
   - [2.1 Requirements](#21-requirements)
   - [2.2 Endpoints](#22-endpoints)
   - [2.3 Replaceable parts](#23-replaceable-parts)
-  - [2.4 Summarizer retry behavior](#24-summarizer-retry-behavior)
+  - [2.4 Chunker configuration](#24-chunker-configuration)
+  - [2.5 Summarizer retry behavior](#25-summarizer-retry-behavior)
 - [`3. Extractor API lib`](#3-extractor-api-lib)
   - [3.1 Requirements](#31-requirements)
   - [3.2 Endpoints](#32-endpoints)
@@ -106,7 +107,7 @@ The default STACKIT embedder implementation (`StackitEmbedder`) uses the shared
 
 - Decorator: `rag_core_lib.impl.utils.retry_decorator.retry_with_backoff`
 - Base settings (fallback): [`RetryDecoratorSettings`](./rag-core-lib/src/rag_core_lib/impl/settings/retry_decorator_settings.py)
-- Per-embedder overrides: [`StackitEmbedderSettings`](./rag-core-api/src/rag_core_api/impl/settings/stackit_embedder_settings.py)
+- Per-embedder overrides: [`StackitEmbedderSettings`](./rag-core-lib/src/rag_core_lib/impl/settings/stackit_embedder_settings.py)
 
 How it resolves settings
 
@@ -189,7 +190,7 @@ The extracted information will be summarized using LLM. The summary, as well as
 | rag_api | [`admin_api_lib.rag_backend_client.openapi_client.api.rag_api.RagApi`](./admin-api-lib/src/admin_api_lib/rag_backend_client/openapi_client/api/rag_api.py) | [`admin_api_lib.rag_backend_client.openapi_client.api.rag_api.RagApi`](./admin-api-lib/src/admin_api_lib/rag_backend_client/openapi_client/api/rag_api.py) | Needs to be replaced if changes to the `/information_pieces/remove` or `/information_pieces/upload` of the [`rag-core-api`](#1-rag-core-api) are made. |
 | summarizer_prompt | `str` | [`admin_api_lib.prompt_templates.summarize_prompt.SUMMARIZE_PROMPT`](./admin-api-lib/src/admin_api_lib/prompt_templates/summarize_prompt.py) | The prompt used of the summarization. |
 | langfuse_manager | [`rag_core_lib.impl.langfuse_manager.langfuse_manager.LangfuseManager`](./rag-core-lib/src/rag_core_lib/impl/langfuse_manager/langfuse_manager.py) | [`rag_core_lib.impl.langfuse_manager.langfuse_manager.LangfuseManager`](./rag-core-lib/src/rag_core_lib/impl/langfuse_manager/langfuse_manager.py) | Retrieves additional settings, as well as the prompt from langfuse if available. |
-| summarizer |  [`admin_api_lib.summarizer.summarizer.Summarizer`](./admin-api-lib/src/admin_api_lib/summarizer/summarizer.py) | [`admin_api_lib.impl.summarizer.langchain_summarizer.LangchainSummarizer`](./admin-api-lib/src/admin_api_lib/impl/summarizer/langchain_summarizer.py) | Creates the summaries. Uses the shared retry decorator with optional per-summarizer overrides (see 2.4). |
+| summarizer |  [`admin_api_lib.summarizer.summarizer.Summarizer`](./admin-api-lib/src/admin_api_lib/summarizer/summarizer.py) | [`admin_api_lib.impl.summarizer.langchain_summarizer.LangchainSummarizer`](./admin-api-lib/src/admin_api_lib/impl/summarizer/langchain_summarizer.py) | Creates the summaries. Uses the shared retry decorator with optional per-summarizer overrides (see 2.5). |
 | untraced_information_enhancer |[`admin_api_lib.information_enhancer.information_enhancer.InformationEnhancer`](./admin-api-lib/src/admin_api_lib/information_enhancer/information_enhancer.py) | [`admin_api_lib.impl.information_enhancer.general_enhancer.GeneralEnhancer`](./admin-api-lib/src/admin_api_lib/impl/information_enhancer/general_enhancer.py) |  Uses the *summarizer* to enhance the extracted documents. |
 | information_enhancer |  [`rag_core_lib.chains.async_chain.AsyncChain[Any, Any]`](./rag-core-lib/src/rag_core_lib/chains/async_chain.py)| [`rag_core_lib.impl.tracers.langfuse_traced_chain.LangfuseTracedGraph`](./rag-core-lib/src/rag_core_lib/impl/tracers/langfuse_traced_chain.py) |Wraps around the *untraced_information_enhancer* and adds langfuse tracing. |
 | document_deleter |[`admin_api_lib.api_endpoints.document_deleter.DocumentDeleter`](./admin-api-lib/src/admin_api_lib/api_endpoints/document_deleter.py) | [`admin_api_lib.impl.api_endpoints.default_document_deleter.DefaultDocumentDeleter`](./admin-api-lib/src/admin_api_lib/impl/api_endpoints/default_document_deleter.py) |  Handles deletion of sources. |
@@ -198,7 +199,51 @@ The extracted information will be summarized using LLM. The summary, as well as
 | document_reference_retriever | [`admin_api_lib.api_endpoints.document_reference_retriever.DocumentReferenceRetriever`](./admin-api-lib/src/admin_api_lib/api_endpoints/document_reference_retriever.py) | [`admin_api_lib.impl.api_endpoints.default_document_reference_retriever.DefaultDocumentReferenceRetriever`](./admin-api-lib/src/admin_api_lib/impl/api_endpoints/default_document_reference_retriever.py) | Handles return of files from connected storage. |
 | file_uploader | [`admin_api_lib.api_endpoints.file_uploader.FileUploader`](./admin-api-lib/src/admin_api_lib/api_endpoints/file_uploader.py) | [`admin_api_lib.impl.api_endpoints.default_file_uploader.DefaultFileUploader`](./admin-api-lib/src/admin_api_lib/impl/api_endpoints/default_file_uploader.py) | Handles upload and extraction of files. |
 
-### 2.4 Summarizer retry behavior
+### 2.4 Chunker configuration
+
+The default dependency container exposes two chunking strategies via [`ChunkerSettings`](./admin-api-lib/src/admin_api_lib/impl/settings/chunker_settings.py):
+
+- `recursive` (default) wraps LangChain's `RecursiveCharacterTextSplitter`.
+- `semantic` wraps LangChain's `SemanticChunker` and requires an embeddings backend.
+
+You can switch between them and fine-tune their behaviour through environment variables:
+
+| Setting | Description | Default |
+|---------|-------------|---------|
+| `CHUNKER_MODE` | Selects the implementation (`recursive` or `semantic`). | `recursive` |
+| `CHUNKER_MAX_SIZE` | Maximum character count per recursive chunk. | `1000` |
+| `CHUNKER_OVERLAP` | Character overlap between recursive chunks. | `100` |
+| `CHUNKER_SEMANTIC_BREAKPOINT_THRESHOLD_TYPE` | Breakpoint heuristic (`percentile`, `standard_deviation`, `interquartile`). | `percentile` |
+| `CHUNKER_SEMANTIC_BREAKPOINT_THRESHOLD` | Threshold associated with the selected heuristic. | `95.0` |
+| `CHUNKER_SEMANTIC_BUFFER_SIZE` | Context buffer that is kept on both sides of a semantic breakpoint. | `1` |
+| `CHUNKER_SEMANTIC_MIN_CHUNK_SIZE` | Minimum size for semantic chunks. | `200` |
+| `CHUNKER_SEMANTIC_MAX_CHUNK_SIZE` | Optional maximum size for semantic chunks (`None` when omitted). | `1200` |
+| `CHUNKER_SEMANTIC_TRIM_CHUNKS` | Whether to strip whitespace around semantic chunks. | `true` |
+
+> 📌 The recursive chunker only uses the `CHUNKER_MAX_SIZE` and `CHUNKER_OVERLAP` knobs. The remaining keys are ignored unless `CHUNKER_MODE=semantic`.
+
+#### Embeddings backend for semantic chunking
+
+When `CHUNKER_MODE` is set to `semantic`, the dependency container selects embeddings using [`EmbedderClassTypeSettings`](./rag-core-lib/src/rag_core_lib/impl/settings/embedder_class_type_settings.py). Configure the backend via:
+
+- `EMBEDDER_CLASS_TYPE_EMBEDDER_TYPE`: choose one of `stackit`, `ollama`, or `fake`.
+
+Backend-specific options:
+
+- **STACKIT embeddings** (production default)
+  - `STACKIT_EMBEDDER_MODEL`
+  - `STACKIT_EMBEDDER_BASE_URL`
+  - `STACKIT_EMBEDDER_API_KEY` *(required)*
+  - Optional retry overrides: `STACKIT_EMBEDDER_MAX_RETRIES`, `STACKIT_EMBEDDER_RETRY_BASE_DELAY`, `STACKIT_EMBEDDER_RETRY_MAX_DELAY`, `STACKIT_EMBEDDER_BACKOFF_FACTOR`, `STACKIT_EMBEDDER_ATTEMPT_CAP`, `STACKIT_EMBEDDER_JITTER_MIN`, `STACKIT_EMBEDDER_JITTER_MAX`
+- **Ollama embeddings** (self-hosted)
+  - `OLLAMA_EMBEDDER_MODEL`
+  - `OLLAMA_EMBEDDER_BASE_URL`
+- **Fake embeddings** (testing)
+  - `FAKE_EMBEDDER_SIZE`
+
+In the Helm chart set `CHUNKER_*` keys under `adminBackend.envs.chunker`. The admin deployment reuses the embedder config maps from the backend release, so adjust `backend.envs.embedderClassTypes`, `backend.envs.stackitEmbedder`, `backend.envs.ollamaEmbedder`, or `backend.envs.fakeEmbedder` accordingly when switching embeddings for semantic chunking.
+
+### 2.5 Summarizer retry behavior
 
 The default summarizer implementation (`LangchainSummarizer`) now uses the shared retry decorator with exponential backoff from the `rag-core-lib`.
 

libs/admin-api-lib/src/admin_api_lib/dependency_container.py

@@ -2,13 +2,10 @@
 
 from admin_api_lib.impl.api_endpoints.default_file_uploader import DefaultFileUploader
 from dependency_injector.containers import DeclarativeContainer
-from dependency_injector.providers import (  # noqa: WOT001
-    Configuration,
-    List,
-    Selector,
-    Singleton,
-)
+from dependency_injector.providers import Configuration, List, Selector, Singleton
 from langchain.text_splitter import RecursiveCharacterTextSplitter
+from langchain_community.embeddings import OllamaEmbeddings
+from langchain_core.embeddings.fake import FakeEmbeddings
 from langfuse import Langfuse
 
 from admin_api_lib.extractor_api_client.openapi_client.api.extractor_api import (
@@ -29,6 +26,7 @@
 from admin_api_lib.impl.api_endpoints.default_documents_status_retriever import (
     DefaultDocumentsStatusRetriever,
 )
+from admin_api_lib.impl.chunker.semantic_text_chunker import SemanticTextChunker
 from admin_api_lib.impl.chunker.text_chunker import TextChunker
 from admin_api_lib.impl.file_services.s3_service import S3Service
 from admin_api_lib.impl.information_enhancer.general_enhancer import GeneralEnhancer
@@ -59,17 +57,26 @@
 from admin_api_lib.rag_backend_client.openapi_client.configuration import (
     Configuration as RagConfiguration,
 )
+from rag_core_lib.impl.embeddings.langchain_community_embedder import (
+    LangchainCommunityEmbedder,
+)
+from rag_core_lib.impl.embeddings.stackit_embedder import StackitEmbedder
 from rag_core_lib.impl.langfuse_manager.langfuse_manager import LangfuseManager
 from rag_core_lib.impl.llms.llm_factory import chat_model_provider
+from rag_core_lib.impl.settings.embedder_class_type_settings import (
+    EmbedderClassTypeSettings,
+)
+from rag_core_lib.impl.settings.fake_embedder_settings import FakeEmbedderSettings
 from rag_core_lib.impl.settings.langfuse_settings import LangfuseSettings
+from rag_core_lib.impl.settings.ollama_embedder_settings import OllamaEmbedderSettings
 from rag_core_lib.impl.settings.ollama_llm_settings import OllamaSettings
 from rag_core_lib.impl.settings.rag_class_types_settings import RAGClassTypeSettings
 from rag_core_lib.impl.settings.retry_decorator_settings import RetryDecoratorSettings
+from rag_core_lib.impl.settings.stackit_embedder_settings import StackitEmbedderSettings
 from rag_core_lib.impl.settings.stackit_vllm_settings import StackitVllmSettings
 from rag_core_lib.impl.tracers.langfuse_traced_runnable import LangfuseTracedRunnable
 from rag_core_lib.impl.utils.async_threadsafe_semaphore import AsyncThreadsafeSemaphore
 
-
 class DependencyContainer(DeclarativeContainer):
     """Dependency injection container for managing application dependencies."""
 
@@ -78,6 +85,10 @@ class DependencyContainer(DeclarativeContainer):
     # Settings
     s3_settings = S3Settings()
     chunker_settings = ChunkerSettings()
+    chunker_embedder_type_settings = EmbedderClassTypeSettings()
+    stackit_chunker_embedder_settings = StackitEmbedderSettings()
+    ollama_chunker_embedder_settings = OllamaEmbedderSettings()
+    fake_chunker_embedder_settings = FakeEmbedderSettings()
     ollama_settings = OllamaSettings()
     langfuse_settings = LangfuseSettings()
     stackit_vllm_settings = StackitVllmSettings()
@@ -96,7 +107,40 @@ class DependencyContainer(DeclarativeContainer):
         chunk_size=chunker_settings.max_size, chunk_overlap=chunker_settings.overlap
     )
 
-    chunker = Singleton(TextChunker, text_splitter)
+    semantic_chunker_embeddings = Selector(
+        chunker_embedder_type_settings.embedder_type,
+        stackit=Singleton(
+            StackitEmbedder,
+            stackit_chunker_embedder_settings,
+            retry_decorator_settings,
+        ),
+        ollama=Singleton(
+            LangchainCommunityEmbedder,
+            embedder=Singleton(
+                OllamaEmbeddings,
+                model=ollama_chunker_embedder_settings.model,
+                base_url=ollama_chunker_embedder_settings.base_url,
+            ),
+        ),
+        fake=Singleton(FakeEmbeddings, size=fake_chunker_embedder_settings.size),
+    )
+
+    semantic_chunker = Singleton(
+        SemanticTextChunker,
+        embeddings=semantic_chunker_embeddings,
+        breakpoint_threshold_type=chunker_settings.semantic_breakpoint_threshold_type,
+        breakpoint_threshold=chunker_settings.semantic_breakpoint_threshold,
+        buffer_size=chunker_settings.semantic_buffer_size,
+        min_chunk_size=chunker_settings.semantic_min_chunk_size,
+        max_chunk_size=chunker_settings.semantic_max_chunk_size,
+        trim_chunks=chunker_settings.semantic_trim_chunks,
+    )
+
+    chunker = Selector(
+        chunker_settings.mode,
+        recursive=Singleton(TextChunker, text_splitter),
+        semantic=semantic_chunker,
+    )
     extractor_api_configuration = Singleton(ExtractorConfiguration, host=document_extractor_settings.host)
     document_extractor_api_client = Singleton(ApiClient, extractor_api_configuration)
     document_extractor = Singleton(ExtractorApi, document_extractor_api_client)

libs/admin-api-lib/src/admin_api_lib/impl/chunker/semantic_text_chunker.py

@@ -0,0 +1,112 @@
+"""Semantic text chunker backed by LangChain's semantic splitter."""
+
+from __future__ import annotations
+
+from collections.abc import Iterable
+from inspect import signature
+from typing import Any, Type
+
+from langchain_core.documents import Document
+from langchain_core.embeddings import Embeddings
+from langchain_text_splitters import SemanticChunker as LangchainSemanticChunker
+
+from admin_api_lib.chunker.chunker import Chunker
+
+
+class SemanticTextChunker(Chunker):
+    """Wrap the LangChain semantic chunker behind the local ``Chunker`` interface."""
+
+    def __init__(
+        self,
+        embeddings: Embeddings,
+        *,
+        breakpoint_threshold_type: str | None = None,
+        breakpoint_threshold: float | None = None,
+        buffer_size: int | None = None,
+        min_chunk_size: int | None = None,
+        max_chunk_size: int | None = None,
+        trim_chunks: bool | None = None,
+        chunker_cls: Type[LangchainSemanticChunker] = LangchainSemanticChunker,
+    ) -> None:
+        """Initialise the semantic chunker.
+
+        Parameters
+        ----------
+        embeddings : Embeddings
+            The embeddings backend that powers semantic similarity detection.
+        breakpoint_threshold_type : str | None, optional
+            Strategy used to derive semantic breakpoints. Unsupported values are ignored.
+        breakpoint_threshold : float | None, optional
+            Threshold to apply for the selected breakpoint strategy. Unsupported values are ignored.
+        buffer_size : int | None, optional
+            Number of neighbouring sentences to include for context. Unsupported values are ignored.
+        min_chunk_size : int | None, optional
+            Minimum chunk size enforced by the chunker. Unsupported values are ignored.
+        max_chunk_size : int | None, optional
+            Maximum chunk size enforced by the chunker. Unsupported values are ignored.
+        trim_chunks : bool | None, optional
+            Whether to strip whitespace from chunk boundaries. Unsupported values are ignored.
+        chunker_cls : type[LangchainSemanticChunker], optional
+            Concrete semantic chunker implementation to instantiate. Defaults to
+            :class:`langchain_text_splitters.SemanticChunker`.
+        """
+
+        init_params = _supported_parameters(chunker_cls)
+        candidate_kwargs: dict[str, Any] = {
+            "breakpoint_threshold_type": breakpoint_threshold_type,
+            "breakpoint_threshold": breakpoint_threshold,
+            "buffer_size": buffer_size,
+            "min_chunk_size": min_chunk_size,
+            "max_chunk_size": max_chunk_size,
+            "trim_chunks": trim_chunks,
+        }
+        filtered_kwargs = {
+            key: value
+            for key, value in candidate_kwargs.items()
+            if value is not None and key in init_params
+        }
+
+        self._semantic_chunker = chunker_cls(
+            embeddings=embeddings,
+            **filtered_kwargs,
+        )
+
+    def chunk(self, documents: Iterable[Document]) -> list[Document]:
+        """Split documents using the configured semantic splitter.
+
+        Parameters
+        ----------
+        documents : Iterable[Document]
+            Documents to be chunked.
+
+        Returns
+        -------
+        list[Document]
+            Chunked documents produced by the semantic splitter.
+        """
+
+        documents_list = list(documents)
+        if not documents_list:
+            return []
+        return self._semantic_chunker.split_documents(documents_list)
+
+
+def _supported_parameters(chunker_cls: type) -> set[str]:
+    """Return constructor parameters supported by ``chunker_cls``.
+
+    Parameters
+    ----------
+    chunker_cls : type
+        Semantic chunker class whose constructor signature should be inspected.
+
+    Returns
+    -------
+    set[str]
+        Set of keyword-parameter names accepted by the constructor.
+    """
+
+    try:
+        params = signature(chunker_cls.__init__).parameters
+    except (TypeError, ValueError):  # pragma: no cover - defensive, should not occur
+        return set()
+    return {name for name in params if name != "self"}

libs/admin-api-lib/src/admin_api_lib/impl/settings/chunker_settings.py

@@ -1,23 +1,43 @@
 """Contains settings regarding the chunker."""
 
-from pydantic import Field
+from typing import Literal
+
+from pydantic import Field, model_validator
 from pydantic_settings import BaseSettings
 
 
 class ChunkerSettings(BaseSettings):
-    """Contains settings regarding the chunker.
-
-    Attributes
-    ----------
-    max_size (int): The maximum size of the chunks.
-    overlap (int): The overlap between the chunks.
-    """
+    """Contains settings regarding the chunker configuration."""
 
     class Config:
         """Config class for reading Fields from env."""
 
         env_prefix = "CHUNKER_"
         case_sensitive = False
 
-    max_size: int = Field(default=1000)
-    overlap: int = Field(default=100)
+    mode: Literal["recursive", "semantic"] = Field(default="recursive")
+    max_size: int = Field(default=1000, gt=0)
+    overlap: int = Field(default=100, ge=0)
+
+    semantic_breakpoint_threshold_type: Literal[
+        "percentile",
+        "standard_deviation",
+        "interquartile",
+    ] = Field(default="percentile")
+    semantic_breakpoint_threshold: float = Field(default=95.0, ge=0.0)
+    semantic_buffer_size: int = Field(default=1, ge=0)
+    semantic_min_chunk_size: int = Field(default=200, gt=0)
+    semantic_max_chunk_size: int | None = Field(default=1200, gt=0)
+    semantic_trim_chunks: bool = Field(default=True)
+
+    @model_validator(mode="after")
+    def _validate_min_max(self) -> "ChunkerSettings":
+        if self.mode != "semantic":
+            return self
+        if (
+            self.semantic_max_chunk_size is not None
+            and self.semantic_min_chunk_size > self.semantic_max_chunk_size
+        ):
+            msg = "semantic_min_chunk_size cannot exceed semantic_max_chunk_size"
+            raise ValueError(msg)
+        return self