Skip to content

Commit ecf5ade

Browse files
a-klosa-klos
committed
feat: add README for extractor-api-lib; update contributing instructions in existing READMEs for consistency
1 parent 0af1b3d commit ecf5ade

5 files changed

Lines changed: 98 additions & 4 deletions

File tree

‎libs/admin-api-lib/README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ Document lifecycle orchestration for the STACKIT RAG template. This library expo
44

55
It powers the `services/admin-backend` deployment and is the primary integration point for operators managing their document corpus.
66

7-
## Responsibilities in the RAG stack
7+
## Responsibilities
88

99
1. **Ingestion** – Accept files or external sources from the admin UI or API clients.
1010
2. **Extraction** – Call `extractor-api-lib` to obtain normalized information pieces.
@@ -89,7 +89,7 @@ Because components depend on interfaces defined here, downstream services can sw
8989

9090
## Contributing
9191

92-
Ensure new endpoints or adapters remain thin and defer to [`rag-core-lib`](../rag-core-lib/) for shared logic. Run `poetry run pytest` and the configured linters before opening a PR.
92+
Ensure new endpoints or adapters remain thin and defer to [`rag-core-lib`](../rag-core-lib/) for shared logic. Run `poetry run pytest` and the configured linters before opening a PR. For further instructions see the [Contributing Guide](https://github.com/stackitcloud/rag-template/blob/main/CONTRIBUTING.md).
9393

9494
## License
9595

‎libs/extractor-api-lib/README.md‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
# extractor-api-lib
2+
3+
Content ingestion layer for the STACKIT RAG template. This library exposes a FastAPI extraction service that ingests raw documents (files or remote sources), extracts and converts (to internal representations) the information, and hands output to [`admin-api-lib`](../admin-api-lib/).
4+
5+
## Responsibilities
6+
7+
- Receive binary uploads and remote source descriptors from the admin backend.
8+
- Route each request through the appropriate extractor (file, sitemap, Confluence, etc.).
9+
- Convert extracted fragments into the shared `InformationPiece` schema expected by downstream services.
10+
11+
## Feature highlights
12+
13+
- **Broad format coverage** – PDFs, DOCX, PPTX, XML/EPUB, Confluence spaces, and sitemap-driven websites.
14+
- **Consistent output schema** – Information pieces are returned in a unified structure with content type (`TEXT`, `TABLE`, `IMAGE`) and metadata.
15+
- **Swappable extractors** – Dependency-injector container makes it easy to add or replace file/source extractors, table converters, etc.
16+
- **Production-grade plumbing** – Built-in S3-compatible file service, LangChain loaders with retry/backoff, optional PDF OCR, and throttling controls for web crawls.
17+
18+
## Installation
19+
20+
```bash
21+
pip install extractor-api-lib
22+
```
23+
24+
Python 3.13 is required. OCR and computer-vision features expect system packages such as `ffmpeg`, `poppler-utils`, and `tesseract` (see `services/document-extractor/README.md` for the full list).
25+
26+
## Module tour
27+
28+
- `dependency_container.py` – Central dependency-injector wiring. Override providers here to plug in custom extractors, endpoints etc.
29+
- `api_endpoints/` & `impl/api_endpoints/` – Thin FastAPI endpoint abstractions and implementations for file and source (like confluence & sitemaps) extractors.
30+
- `apis/` – Extractor API abstractions and implementations.
31+
- `extractors/` & `impl/extractors/` – Format-specific logic (PDF, DOCX, PPTX, XML, EPUB, Confluence, sitemap) packaged behind the `InformationExtractor`/`InformationFileExtractor` interfaces.
32+
- `mapper/` & `impl/mapper/` – Abstractions and implementations to map langchain documents, internal and external information piece representations to each other.
33+
- `file_services/` – Default S3-compatible storage adapter; replace it if you store files elsewhere.
34+
- `impl/settings/` – Configuration settings for dependency injection container components.
35+
- `table_converter/` & `impl/table_converter/` – Abstractions and implementations to convert `pandas.DataFrame` to markdown and vice versa.
36+
- `impl/types/` - Enums for content-, extractor- and file types.
37+
- `impl/utils/` – Helper functions for hashed datetime and sitemap crawling, header injection, and custom metadata parsing.
38+
39+
## Endpoints provided
40+
41+
- `POST /extract_from_file` – Downloads the file from S3, extracts its contents, and returns normalized `InformationPiece` records.
42+
- `POST /extract_from_source` – Pulls from remote sources (Confluence, sitemap) using credentials and further optional kwargs.
43+
44+
Both endpoints stream their results back to `admin-api-lib`, which takes care of enrichment and persistence.
45+
46+
## How the file extraction endpoint works
47+
48+
1. Download the file from S3
49+
2. Chose suitable file extractor based on the filename ending
50+
3. Extract the content from the file
51+
4. Map the internal representation to the external schema
52+
5. Return the final output
53+
54+
## How the source extraction endpoint works
55+
56+
1. Chose suitable source extractor based on the source type
57+
2. Pull the source content using the provided credentials and parameters
58+
3. Extract the content from the source
59+
4. Map the internal representation to the external schema
60+
5. Return the final output
61+
62+
## Configuration overview
63+
64+
Two `pydantic-settings` models ship with this package:
65+
66+
- **S3 storage** (`S3Settings`) – configure the built-in file service with `S3_ACCESS_KEY_ID`, `S3_SECRET_ACCESS_KEY`, `S3_ENDPOINT`, and `S3_BUCKET`.
67+
- **PDF extraction** (`PDFExtractorSettings`) – adjust footer trimming or diagram export via `PDF_EXTRACTOR_FOOTER_HEIGHT` and `PDF_EXTRACTOR_DIAGRAMS_FOLDER_NAME`.
68+
69+
Other extractors accept their parameters at runtime through the request payload (`ExtractionParameters`). For example, the admin backend forwards Confluence credentials, sitemap URLs, or custom headers when it calls `/extract_from_source`. This keeps the library stateless and makes it easy to plug in additional sources without redeploying.
70+
71+
The Helm chart exposes the environment variables mentioned above under `documentExtractor.envs.*` so production deployments remain declarative.
72+
73+
## Typical usage
74+
75+
```python
76+
from extractor_api_lib.main import app as perfect_extractor_app
77+
```
78+
79+
`admin-api-lib` calls `/extract_from_file` and `/extract_from_source` to populate the ingestion pipeline.
80+
81+
## Extending the library
82+
83+
1. Implement `InformationFileExtractor` or `InformationExtractor` for your new format/source.
84+
2. Register it in `dependency_container.py` (append to `file_extractors` list or `source_extractors` dict).
85+
3. Update mapper or metadata handling if additional fields are required.
86+
4. Add unit tests under `libs/extractor-api-lib/tests` using fixtures and fake storage providers.
87+
88+
## Contributing
89+
90+
Ensure new endpoints or adapters remain thin and defer to [`rag-core-lib`](../rag-core-lib/) for shared logic. Run `poetry run pytest` and the configured linters before opening a PR. For further instructions see the [Contributing Guide](https://github.com/stackitcloud/rag-template/blob/main/CONTRIBUTING.md).
91+
92+
## License
93+
94+
Licensed under the project license. See the root [`LICENSE`](https://github.com/stackitcloud/rag-template/blob/main/LICENSE) file.

‎libs/extractor-api-lib/src/extractor_api_lib/impl/file_services/__init__.py‎

Whitespace-only changes.

‎libs/rag-core-api/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -84,7 +84,7 @@ Because components depend on interfaces defined here, downstream services can sw
8484

8585
## Contributing
8686

87-
Ensure new endpoints or adapters remain thin and defer to [`rag-core-lib`](../rag-core-lib/) for shared logic. Run `poetry run pytest` and the configured linters before opening a PR.
87+
Ensure new endpoints or adapters remain thin and defer to [`rag-core-lib`](../rag-core-lib/) for shared logic. Run `poetry run pytest` and the configured linters before opening a PR. For further instructions see the [Contributing Guide](https://github.com/stackitcloud/rag-template/blob/main/CONTRIBUTING.md).
8888

8989
## License
9090

‎libs/rag-core-lib/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -78,7 +78,7 @@ print(response.content)
7878

7979
## Contributing
8080

81-
Follow the repository conventions (Black, isort, Flake8, pytest). When adding new providers or utilities, include typed settings, update Helm values if required, and add unit tests under `libs/rag-core-lib/tests`.
81+
Follow the repository conventions (Black, isort, Flake8, pytest). When adding new providers or utilities, include typed settings, update Helm values if required, and add unit tests under `libs/rag-core-lib/tests`. For further instructions see the [Contributing Guide](https://github.com/stackitcloud/rag-template/blob/main/CONTRIBUTING.md).
8282

8383
## License
8484

0 commit comments

Comments
 (0)