Skip to content

Commit d8b500e

Browse files
dpomiandpomian
authored
Merge pull request #102 from Driss-Lalami99/my-dev-branch
feat: #33 add a sample end to end
2 parents 0abb967 + 9c8c48f commit d8b500e

10 files changed

Lines changed: 236 additions & 1 deletion

File tree

‎.gitignore‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -80,4 +80,7 @@ logs/indexer_skills.log
8080
src/logs/indexer_skills.log
8181

8282
# chroma index sqlite3
83-
src/localhost/chroma.sqlite3
83+
src/localhost/chroma.sqlite3
84+
85+
# Generated sample vector stores
86+
samples/**/output/

‎README.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,13 @@ ${OCI_ENGINE} run -it --rm \
4747

4848
# Documentation
4949

50+
If you are discovering `docs2vecs` for the first time, start with the [samples](samples/) folder. It contains clear end-to-end examples with local data, ready-to-run config files, and step-by-step README instructions so you can understand how the indexer works before adapting it to your own documents.
51+
52+
The current samples include:
53+
54+
- [PDF only](samples/pdf-only/) — index a local PDF file into a generated ChromaDB collection.
55+
- [Markdown only](samples/markdown-only/) — index a local Markdown file into a generated ChromaDB collection.
56+
5057
<details><summary>Expand me if you would like to find out how to vectorize your data</summary>
5158

5259
## Indexer sub-command

‎samples/README.md‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# docs2vecs Samples
2+
3+
This directory contains end-to-end samples that demonstrate how to use the `docs2vecs` indexer with local files.
4+
5+
## Available Samples
6+
7+
| Sample | Purpose |
8+
|--------|---------|
9+
| [PDF only](./pdf-only/) | Index a local PDF file into a generated ChromaDB collection |
10+
| [Markdown only](./markdown-only/) | Index a local Markdown file into a generated ChromaDB collection |
11+
12+
## Prerequisites
13+
14+
- Python 3.11 or higher
15+
- `uv`
16+
- A checkout of this repository
17+
18+
## Usage Pattern
19+
20+
Run sample commands from the repository root:
21+
22+
```bash
23+
uv run docs2vecs indexer --config samples/<sample-name>/config.yml
24+
```
25+
26+
Each sample writes a local ChromaDB database under its own `output/` directory. Those generated files are not committed to the repository; they are created when you run the sample.
27+
28+
The first run may take longer while the local embedding model is downloaded.

‎samples/markdown-only/README.md‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
# Markdown-Only Sample
2+
3+
This sample indexes a local Markdown document with the `docs2vecs indexer` pipeline.
4+
5+
## Files
6+
7+
| Path | Purpose |
8+
|------|---------|
9+
| `samples/markdown-only/input/sample-documentation.md` | Sample Markdown input document |
10+
| `samples/markdown-only/config.yml` | Ready-to-run indexer configuration |
11+
| `samples/markdown-only/output/` | Generated ChromaDB output directory created when the sample runs |
12+
13+
## Run
14+
15+
From the repository root:
16+
17+
```bash
18+
uv run docs2vecs indexer --config samples/markdown-only/config.yml
19+
```
20+
21+
## Expected Result
22+
23+
After the command completes, `samples/markdown-only/output/` should exist and contain a working ChromaDB database for the `markdown-sample-collection` collection.
24+
25+
You can verify the generated output by checking that:
26+
27+
- the command exits successfully,
28+
- `samples/markdown-only/output/` is created,
29+
- the ChromaDB files are present in that output directory.
30+
31+
Generated ChromaDB files are intentionally ignored by git.

‎samples/markdown-only/config.yml‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
definitions:
2+
- skill: &FileScanner
3+
type: file-scanner
4+
name: multi-file-scanner
5+
params:
6+
path: samples/markdown-only/input
7+
filter: ["*.md"]
8+
recursive: false
9+
10+
- skill: &FileReader
11+
type: file-reader
12+
name: multi-file-reader
13+
14+
- skill: &TextSplitter
15+
type: splitter
16+
name: recursive-character-splitter
17+
params:
18+
chunk_size: 1000
19+
overlap: 100
20+
21+
- skill: &FastEmbed
22+
type: embedding
23+
name: llama-fastembed
24+
25+
- skill: &ChromaDB
26+
type: vector-store
27+
name: chromadb
28+
params:
29+
db_path: samples/markdown-only/output
30+
collection_name: markdown-sample-collection
31+
32+
- skillset: &MarkdownSkillset
33+
- *FileScanner
34+
- *FileReader
35+
- *TextSplitter
36+
- *FastEmbed
37+
- *ChromaDB
38+
39+
indexer:
40+
skillset: *MarkdownSkillset

‎samples/markdown-only/input/sample-documentation.md‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
# Getting Started with Documentation
2+
3+
This sample Markdown document demonstrates how the `docs2vecs` indexer processes local Markdown files.
4+
5+
## Overview
6+
7+
Documentation often contains concepts, procedures, and examples that are useful in retrieval augmented generation workflows. Indexing the content into a vector store makes it searchable by semantic meaning.
8+
9+
## Key Concepts
10+
11+
### Vector Embeddings
12+
13+
Vector embeddings are numerical representations of text. They help compare pieces of text by meaning instead of exact wording.
14+
15+
### Document Chunking
16+
17+
Long documents are split into smaller chunks before embedding. Smaller chunks make retrieval more precise and keep each vector focused on a coherent idea.
18+
19+
### Local Vector Store
20+
21+
This sample stores generated vectors in a local ChromaDB database under the sample output directory.
22+
23+
## Example Workflow
24+
25+
1. Scan the input directory for Markdown files.
26+
2. Read Markdown content from each file.
27+
3. Split the text into chunks.
28+
4. Generate local embeddings.
29+
5. Store the chunks and embeddings in ChromaDB.
30+
31+
## Conclusion
32+
33+
This sample is intentionally small so new users can inspect the input, configuration, command, and generated output in one place.

‎samples/pdf-only/README.md‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
# PDF-Only Sample
2+
3+
This sample indexes a local PDF document with the `docs2vecs indexer` pipeline.
4+
5+
## Files
6+
7+
| Path | Purpose |
8+
|------|---------|
9+
| `samples/pdf-only/input/sample-guide.pdf` | Sample PDF input document |
10+
| `samples/pdf-only/config.yml` | Ready-to-run indexer configuration |
11+
| `samples/pdf-only/output/` | Generated ChromaDB output directory created when the sample runs |
12+
13+
## Run
14+
15+
From the repository root:
16+
17+
```bash
18+
uv run docs2vecs indexer --config samples/pdf-only/config.yml
19+
```
20+
21+
## Expected Result
22+
23+
After the command completes, `samples/pdf-only/output/` should exist and contain a working ChromaDB database for the `pdf-sample-collection` collection.
24+
25+
You can verify the generated output by checking that:
26+
27+
- the command exits successfully,
28+
- `samples/pdf-only/output/` is created,
29+
- the ChromaDB files are present in that output directory.
30+
31+
Generated ChromaDB files are intentionally ignored by git.

‎samples/pdf-only/config.yml‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
definitions:
2+
- skill: &FileScanner
3+
type: file-scanner
4+
name: multi-file-scanner
5+
params:
6+
path: samples/pdf-only/input
7+
filter: ["*.pdf"]
8+
recursive: false
9+
10+
- skill: &FileReader
11+
type: file-reader
12+
name: multi-file-reader
13+
14+
- skill: &TextSplitter
15+
type: splitter
16+
name: recursive-character-splitter
17+
params:
18+
chunk_size: 1000
19+
overlap: 100
20+
21+
- skill: &FastEmbed
22+
type: embedding
23+
name: llama-fastembed
24+
25+
- skill: &ChromaDB
26+
type: vector-store
27+
name: chromadb
28+
params:
29+
db_path: samples/pdf-only/output
30+
collection_name: pdf-sample-collection
31+
32+
- skillset: &PDFSkillset
33+
- *FileScanner
34+
- *FileReader
35+
- *TextSplitter
36+
- *FastEmbed
37+
- *ChromaDB
38+
39+
indexer:
40+
skillset: *PDFSkillset

‎samples/pdf-only/input/sample-guide.pdf‎

1.11 KB
Binary file not shown.

‎tests/test_config.py‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -94,3 +94,25 @@ def test_config():
9494
}
9595
assert actual_vector_store_config is not None
9696
assert actual_vector_store_config == expected_vector_store_config
97+
98+
99+
def test_sample_configs_are_valid():
100+
schema_file = Path("src/docs2vecs/subcommands/indexer/config/config_schema.yaml")
101+
102+
for config_file in (
103+
Path("samples/pdf-only/config.yml"),
104+
Path("samples/markdown-only/config.yml"),
105+
):
106+
config = Config(config_file, schema_file)
107+
assert config is not None
108+
109+
skill_names = [
110+
skill_config["name"] for skill_config in config.get_skills_config_dict()
111+
]
112+
assert skill_names == [
113+
"multi-file-scanner",
114+
"multi-file-reader",
115+
"recursive-character-splitter",
116+
"llama-fastembed",
117+
"chromadb",
118+
]

0 commit comments

Comments
 (0)