This directory contains the core libraries of the STACKIT RAG template. These libraries provide comprehensive document extraction capabilities including support for files (PDF, DOCX, XML, EPUB), web sources via sitemaps, and Confluence pages. It consists of the following python packages:
With the exception of the RAG Core lib all of these packages contain an API definition and are easy to adjust for your specific use case.
Each of the packages defines the replaceable parts(1.3 Replaceable Parts, 2.3 Replaceable Parts, 3.3 Replaceable Parts), expected types and offer a brief description.
ⓘ INFO: If you replace parts it is important to keep the name of the component, otherwise the replacing-logic will not work.
This directory also contains a Dockerfile that is used to ensure proper linting and testing of the library packages.
For an example on how to use the packages, please consult the RAG Template project or the individual service implementations in the services directory.
The rag-core-api contains a default implementation of a RAG. For a default use case, no adjustments should be required.
The following endpoints are provided by the backend:
/chat/{session_id}: The endpoint for chatting./evaluate: Will start the evaluation of the RAG using the provided question-answer pairs./information_pieces/remove: Endpoint to remove documents from the vector database./information_pieces/upload: Endpoint to upload documents into the vector database. These documents need to have been parsed. For simplicity, a LangChain Documents like format is used.
All required python libraries can be found in the pyproject.toml file. In addition to python libraries, the following system packages are required:
build-essential
makeThis endpoint is used for chatting.
Will start the evaluation of the RAG using the provided question-answer pairs.
The file containing the dataset can be set by changing the RAGAS_DATASET_FILENAME environment variable, the default is test_data.json.
This path can be either an absolute path, or a path relative to the current working directory.
By default OpenAI is used by the evaluation. If you want to use the same LLM-class for the evaluation as is used for the chat you have to set the environment variable RAGA_USE_OPENAI to false and adjust the RAGAS_MODEL environment variable to the model-name of your choice.
📝 NOTE: Due to quality problems with open-source LLMs, it is recommended to use OpenAI for the evaluation.
Endpoint to remove documents from the vector database.
Endpoint to upload documents into the vector database. These documents need to have been parsed. For simplicity, a LangChain Documents like format is used. Uploaded documents are required to contain the following metadata:
document_urlthat points to a download link to the source document.- All documents of the type
IMAGErequire the content of the image encoded in base64 in thebase64_imagekey.
The default STACKIT embedder implementation (StackitEmbedder) uses the shared retry decorator with exponential backoff from the rag-core-lib.
- Decorator:
rag_core_lib.impl.utils.retry_decorator.retry_with_backoff - Base settings (fallback):
RetryDecoratorSettings - Per-embedder overrides:
StackitEmbedderSettings
How it resolves settings
- Each retry-related field in
StackitEmbedderSettingsis optional. When a field is provided (not None), it overrides the corresponding value fromRetryDecoratorSettings. - When a field is not provided (None), the embedder falls back to the value from
RetryDecoratorSettings.
Configuring via environment variables
- Embedder-specific (prefix
STACKIT_EMBEDDER_):STACKIT_EMBEDDER_MAX_RETRIESSTACKIT_EMBEDDER_RETRY_BASE_DELAYSTACKIT_EMBEDDER_RETRY_MAX_DELAYSTACKIT_EMBEDDER_BACKOFF_FACTORSTACKIT_EMBEDDER_ATTEMPT_CAPSTACKIT_EMBEDDER_JITTER_MINSTACKIT_EMBEDDER_JITTER_MAX
- Global fallback (prefix
RETRY_DECORATOR_): see section 4.2 for all keys and defaults. - Helm chart: set the same keys under
backend.envs.stackitEmbedderin infrastructure/rag/values.yaml.
The Admin API Library contains all required components for file management capabilities for RAG systems, handling all document lifecycle operations. It also includes a default dependency_container, that is pre-configured and should fit most use-cases.
The following endpoints are provided by the admin-api-lib:
/delete_document/{identification}: Deletes the file from storage (if applicable) and vector database. Theidentificationcan be retrieved from the/all_documents_statusendpoint./document_reference/{identification}: Returns the document./all_documents_status: Return theidentificationand status of all available sources./upload_file: Endpoint to upload files./upload_source: Endpoint to upload non-file sources.
All required python libraries can be found in the pyproject.toml file. In addition to python libraries, the following system packages are required:
build-essential
makeWill delete the document from the connected storage system and will send a request to the backend to delete all related Documents from the vector database.
Will return the source document stored in the connected storage system.
ⓘ INFO: Confluence pages are not stored in the connected storage system. They are only stored in the vector database and can't be retrieved using this endpoint.
Will return a list of all sources for the chat and their current status.
Files can be uploaded here. This endpoint will process the document in a background and will extract information using the document-extractor. The extracted information will be summarized using a LLM. The summary, as well as the unrefined extracted document, will be uploaded to the rag-core-api.
Loads all the content from an arbitrary non-file source using the document-extractor.
The type of the source needs to correspond to an extractor in the document-extractor. Supported types include confluence for Confluence pages and sitemap for web content via XML sitemaps.
The extracted information will be summarized using LLM. The summary, as well as the unrefined extracted document, will be uploaded to the rag-core-api. An is configured. Defaults to 3600 seconds (1 hour). Can be adjusted by values in the helm chart.
The default dependency container now exposes two chunking strategies which can be chosen by ChunkerClassTypeSettings:
recursive(default) wraps LangChain'sRecursiveCharacterTextSplitter.semanticwraps LangChain'sSemanticChunker, and considers minimum/maximum chunk size withnltk/RecursiveCharacterTextSplitter.
You can switch between them and fine-tune their behaviour through environment variables:
| Setting | Description | Default |
|---|---|---|
CHUNKER_MAX_SIZE |
Maximum character count per recursive chunk. | 1000 |
CHUNKER_OVERLAP |
Character overlap between recursive chunks. | 100 |
CHUNKER_BREAKPOINT_THRESHOLD_TYPE |
Breakpoint heuristic (percentile, standard_deviation, interquartile). |
percentile |
CHUNKER_BREAKPOINT_THRESHOLD_AMOUNT |
Threshold associated with the selected heuristic. | 95.0 |
CHUNKER_BUFFER_SIZE |
Context buffer that is kept on both sides of a semantic breakpoint. | 1 |
CHUNKER_MIN_SIZE |
Minimum size for semantic chunks. | 200 |
📌 The recursive chunker only uses the
CHUNKER_MAX_SIZEandCHUNKER_OVERLAPknobs. The remaining keys are ignored unlessCHUNKER_CLASS_TYPE_CHUNKER_TYPE=semantic.
Behavior details
- Recursive chunker enforces
CHUNKER_MAX_SIZEandCHUNKER_OVERLAPonly. - Semantic chunker uses embeddings to detect semantic breakpoints and can also enforce
min/maxsizes:- Oversized chunks are re-split with
RecursiveCharacterTextSplitter(auto-provisioned whenmax > min). - Trailing undersized chunks are rebalanced using sentence-aware splitting (NLTK Punkt when available, regex fallback otherwise) to avoid tiny tails while respecting
[min, max].
- Oversized chunks are re-split with
When CHUNKER_CLASS_TYPE_CHUNKER_TYPE is set to semantic, the dependency container selects embeddings using EmbedderClassTypeSettings. Configure the backend via:
EMBEDDER_CLASS_TYPE_EMBEDDER_TYPE: choose one ofstackit,ollama.
Backend-specific options:
- STACKIT embeddings (production default)
STACKIT_EMBEDDER_MODELSTACKIT_EMBEDDER_BASE_URLSTACKIT_EMBEDDER_API_KEY(required)- Optional retry overrides:
STACKIT_EMBEDDER_MAX_RETRIES,STACKIT_EMBEDDER_RETRY_BASE_DELAY,STACKIT_EMBEDDER_RETRY_MAX_DELAY,STACKIT_EMBEDDER_BACKOFF_FACTOR,STACKIT_EMBEDDER_ATTEMPT_CAP,STACKIT_EMBEDDER_JITTER_MIN,STACKIT_EMBEDDER_JITTER_MAX
- Ollama embeddings (self-hosted)
OLLAMA_EMBEDDER_MODELOLLAMA_EMBEDDER_BASE_URL
In the Helm chart set CHUNKER_* keys under adminBackend.envs.chunker. The admin deployment reuses the embedder config maps from the backend release, so adjust backend.envs.embedderClassTypes, backend.envs.stackitEmbedder, backend.envs.ollamaEmbedder, or backend.envs.fakeEmbedder accordingly when switching embeddings for semantic chunking.
The default summarizer implementation (LangchainSummarizer) now uses the shared retry decorator with exponential backoff from the rag-core-lib.
- Decorator:
rag_core_lib.impl.utils.retry_decorator.retry_with_backoff - Base settings (fallback):
RetryDecoratorSettings - Per-summarizer overrides:
SummarizerSettings
How it resolves settings
- Each field in
SummarizerSettingsis optional. When a field is provided (not None), it overrides the corresponding value fromRetryDecoratorSettings. - When a field is not provided (None), the summarizer falls back to the value from
RetryDecoratorSettings.
Configuring via environment variables
- Summarizer-specific (prefix
SUMMARIZER_):SUMMARIZER_MAX_RETRIESSUMMARIZER_RETRY_BASE_DELAYSUMMARIZER_RETRY_MAX_DELAYSUMMARIZER_BACKOFF_FACTORSUMMARIZER_ATTEMPT_CAPSUMMARIZER_JITTER_MINSUMMARIZER_JITTER_MAX
- Global fallback (prefix
RETRY_DECORATOR_): see section 4.2 for all keys and defaults. - Helm chart: set the same keys under
adminBackend.envs.summarizerin infrastructure/rag/values.yaml.
The Extractor Library contains components that provide document parsing capabilities for various file formats and web sources. It supports extracting content from PDF, DOCX, XML files, as well as web pages via sitemaps and Confluence pages. It also includes a default dependency_container, that is pre-configured and is a good starting point for most use-cases. This API should not be exposed by ingress and only used for internally.
The following endpoints are provided by the extractor-api-lib:
/extract_from_file: This endpoint extracts the information from files./extract_from_source: This endpoint extracts the information from a non-file source.
All required python libraries can be found in the pyproject.toml file. In addition to python libraries, the following system packages are required:
build-essential
make
ffmpeg
poppler-utils
tesseract-ocr
tesseract-ocr-deu
tesseract-ocr-engThis endpoint will extract the information from PDF,PTTX,WORD,XML files. It will load the files from the connected storage. The following types of information will be extracted:
TEXT: plain textTABLE: data in tabular form found in the document
This endpoint will extract data for non-file source.
The type of information that is extracted will vary depending on the source. Supported sources include confluence for Confluence pages and sitemap for web pages via XML sitemaps.
The following types of information can be extracted:
TEXT: plain textTABLE: data in tabular form found in the documentIMAGE: image found in the document
For Confluence sources, provide the instance url and API token and include either a space_key or a cql filter (empty values are ignored). Optional flags such as include_attachments, keep_markdown_format, and keep_newlines mirror the parameters supported by LangChain's ConfluenceLoader.
For sitemap sources, additional parameters can be provided, e.g.:
web_path: The URL of the XML sitemap to crawlfilter_urls: JSON array of URL patterns to filter pages (optional)header_template: JSON object for custom HTTP headers (optional)continue_on_failure: Whether to skip pages that fail to load instead of aborting the crawl (optional, default:true)
Technically, all parameters of the SitemapLoader from LangChain can be provided.
The HTML parsing logic can be tuned via the SITEMAP_PARSER environment variable (default: docusaurus; options: docusaurus, astro, generic). For Helm deployments, set extractor.envs.sitemap.SITEMAP_PARSER in infrastructure/rag/values.yaml. You can also override the parser per upload by passing a sitemap_parser key/value pair (same options) in the /upload_source request (available as a dropdown in the admin frontend).
| Name | Type | Default | Notes |
|---|---|---|---|
| file_service | extractor_api_lib.file_services.file_service.FileService |
extractor_api_lib.impl.file_services.s3_service.S3Service |
Handles operations on the connected storage. |
| database_converter | extractor_api_lib.table_converter.dataframe_converter.DataframeConverter |
extractor_api_lib.impl.table_converter.dataframe2markdown.DataFrame2Markdown |
Converts the extracted table from pandas.DataFrame to markdown. If you want the table to have another format, this would need to be adjusted. |
| pdf_extractor | extractor_api_lib.extractors.information_file_extractor.InformationFileExtractor |
extractor_api_lib.impl.extractors.file_extractors.pdf_extractor.PDFExtractor |
Extractor used for extracting information from PDF documents. |
| ms_docs_extractor | extractor_api_lib.extractors.information_file_extractor.InformationFileExtractor |
extractor_api_lib.impl.extractors.file_extractors.ms_docs_extractor.MSDocsExtractor |
Extractor used for extracting information from Microsoft Documents like *.docx, etc. |
| xml_extractor | extractor_api_lib.extractors.information_file_extractor.InformationFileExtractor |
extractor_api_lib.impl.extractors.file_extractors.xml_extractor.XMLExtractor |
Extractor used for extracting content from XML documents. |
| epub_extractor | extractor_api_lib.extractors.information_file_extractor.InformationFileExtractor |
extractor_api_lib.impl.extractors.file_extractors.epub_extractor.EPUBExtractor |
Extractor used for extracting content from EPUB documents. |
| file_extractors | dependency_injector.providers.List[extractor_api_lib.extractors.information_file_extractor.InformationFileExtractor] |
dependency_injector.providers.List(pdf_extractor, ms_docs_extractor, xml_extractor) |
List of all available file extractors. If you add a new type of file extractor you would have to add it to this list. |
| intern2external | extractor_api_lib.impl.mapper.internal2external_information_piece.Internal2ExternalInformationPiece |
extractor_api_lib.impl.mapper.internal2external_information_piece.Internal2ExternalInformationPiece |
Maps internal information pieces to external information pieces, converting between internal and external content types. |
| confluence_document2information_piece | extractor_api_lib.mapper.source_langchain_document2information_piece.SourceLangchainDocument2InformationPiece |
extractor_api_lib.impl.mapper.confluence_langchain_document2information_piece.ConfluenceLangchainDocument2InformationPiece |
Maps LangChain documents from Confluence to information pieces with Confluence-specific metadata handling. |
| sitemap_document2information_piece | extractor_api_lib.mapper.source_langchain_document2information_piece.SourceLangchainDocument2InformationPiece |
extractor_api_lib.impl.mapper.sitemap_document2information_piece.SitemapLangchainDocument2InformationPiece |
Maps LangChain documents from sitemap sources to information pieces with sitemap-specific metadata handling. |
| general_file_extractor | extractor_api_lib.api_endpoints.file_extractor.FileExtractor |
extractor_api_lib.impl.api_endpoints.general_file_extractor.GeneralFileExtractor |
Combines multiple file extractors and decides which one to use for the given file format. |
| confluence_extractor | extractor_api_lib.extractors.information_extractor.InformationExtractor |
extractor_api_lib.impl.extractors.confluence_extractor.ConfluenceExtractor |
Implementation of an extractor for the source confluence. |
| sitemap_extractor | extractor_api_lib.extractors.information_extractor.InformationExtractor |
extractor_api_lib.impl.extractors.sitemap_extractor.SitemapExtractor |
Implementation of an extractor for the source sitemap. Supports XML sitemap crawling with configurable parameters including URL filtering, custom headers, and crawling depth. Uses LangChain's SitemapLoader with support for custom parsing and meta functions via dependency injection. |
| sitemap_parsing_function | dependency_injector.providers.Factory[Callable] |
extractor_api_lib.impl.utils.sitemap_extractor_utils.custom_sitemap_parser_function |
Custom parsing function for sitemap content extraction. Used by the sitemap extractor to parse HTML content from web pages. Can be replaced to customize how web page content is processed and extracted. |
| sitemap_meta_function | dependency_injector.providers.Factory[Callable] |
extractor_api_lib.impl.utils.sitemap_extractor_utils.custom_sitemap_metadata_parser_function |
Custom meta function for sitemap content processing. Used by the sitemap extractor to extract metadata from web pages. Can be replaced to customize how metadata is extracted and structured from web content. |
| source_extractor | extractor_api_lib.api_endpoints.source_extractor.SourceExtractor |
extractor_api_lib.impl.api_endpoints.general_source_extractor.GeneralSourceExtractor |
Implementation of the /extract_from_source endpoint. Will decide the correct extractor for the source and handles available extractors for confluence and sitemap sources. |
The rag-core-lib contains components of the rag-core-api that are also useful for other services and therefore are packaged in a way that makes it easy to use.
Examples of included components:
- tracing for
LangChainchains usingLangfuse - settings for multiple LLMs and Langfuse
- factory for LLMs
ContentTypeenum of the Documents.- ...
All required python libraries can be found in the pyproject.toml file. In addition to python libraries the following system packages are required:
build-essential
makeThe rag-core-lib provides a reusable retry decorator with exponential backoff and rate‑limit awareness for both sync and async functions.
- Module:
rag_core_lib.impl.utils.retry_decorator.retry_with_backoff - Settings:
rag_core_lib.impl.settings.retry_decorator_settings.RetryDecoratorSettings - Works with: synchronous and asynchronous callables
- Rate-limit aware: optionally inspects HTTP status 429 and headers like
x-ratelimit-reset-requests/x-ratelimit-reset-tokens
Usage example
from rag_core_lib.impl.utils.retry_decorator import retry_with_backoff
from rag_core_lib.impl.settings.retry_decorator_settings import RetryDecoratorSettings
# Configure via code (env vars also supported, see below)
settings = RetryDecoratorSettings(
max_retries=3,
retry_base_delay=0.2,
)
@retry_with_backoff(settings=settings)
def fetch_something():
return "ok"
@retry_with_backoff(settings=settings)
async def fetch_async_something():
return "ok"Configuration
-
Environment variables (prefix
RETRY_DECORATOR_):RETRY_DECORATOR_MAX_RETRIES(default: 5)RETRY_DECORATOR_RETRY_BASE_DELAY(default: 0.5)RETRY_DECORATOR_RETRY_MAX_DELAY(default: 600)RETRY_DECORATOR_BACKOFF_FACTOR(default: 2)RETRY_DECORATOR_ATTEMPT_CAP(default: 6)RETRY_DECORATOR_JITTER_MIN(default: 0.05)RETRY_DECORATOR_JITTER_MAX(default: 0.25)
-
Helm chart (shared values): set the same keys under
shared.envs.retryDecoratorin infrastructure/rag/values.yaml to apply cluster‑wide defaults for backend/admin services.
Advanced
- Customize which exceptions trigger retries via
exceptionsandrate_limit_exceptionsparameters ofretry_with_backoff(). - Header‑based wait: When rate‑limited, the decorator will honor reset headers if present and add jitter.
For more examples, see tests in ./rag-core-lib/tests/retry_decorator_test.py.