Skip to content

Commit d1d7bc6

Browse files
thiswillbeyourgithubthiswillbeyourgithub
committed
Merge branch 'dev'
2 parents 213d8ff + 13fef4d commit d1d7bc6

17 files changed

Lines changed: 875 additions & 38 deletions

File tree

‎ARCHITECTURE.md‎

Lines changed: 183 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,183 @@
1+
# WDOC Architecture
2+
3+
WDOC is a Retrieval-Augmented Generation (RAG) system that loads documents from 20+ source types, embeds them into a vector store, and supports querying, summarizing, searching, and parsing tasks via LLMs.
4+
5+
---
6+
7+
## Directory Structure
8+
9+
```
10+
wdoc/
11+
├── __init__.py # Package init, beartype setup
12+
├── __main__.py # CLI entry point (Google Fire)
13+
├── wdoc.py # Main orchestration class (~3000 lines)
14+
└── utils/
15+
├── env.py # Configuration (EnvDataclass, WDOC_* env vars)
16+
├── llm.py # LLM loading via LiteLLM, price tracking
17+
├── embeddings.py # Embedding model setup, FAISS vector store
18+
├── retrievers.py # Retriever creation (multi-query, parent, SVM, KNN)
19+
├── prompts.py # LLM prompt templates (named personas)
20+
├── batch_file_loader.py # Parallel multi-file loading with filetype inference
21+
├── load_recursive.py # Handles compound/recursive filetypes
22+
├── misc.py # Caching, hashing, token counting utilities
23+
├── errors.py # Custom exceptions
24+
├── logger.py # Loguru-based logging
25+
├── filters.py # Document filtering (regex, metadata)
26+
├── interact.py # Interactive CLI features
27+
├── loaders/ # Per-filetype document loaders
28+
│ ├── __init__.py # load_one_doc() dispatcher
29+
│ ├── pdf.py # PDF (15 parser backends)
30+
│ ├── url.py # URLs (jina, playwright, selenium)
31+
│ ├── youtube.py # YouTube videos
32+
│ ├── local_audio.py # Audio transcription (Deepgram)
33+
│ ├── local_video.py # Video extraction
34+
│ ├── epub.py # EPUB books
35+
│ ├── anki.py # Anki flashcards
36+
│ ├── logseq_markdown.py # Logseq notes
37+
│ ├── word.py # Word documents
38+
│ ├── powerpoint.py # PowerPoint
39+
│ └── json_dict.py # JSON entries
40+
├── tasks/ # Task implementations
41+
│ ├── types.py # wdocTask enum
42+
│ ├── query.py # Query: retrieve → evaluate → answer → combine
43+
│ ├── summarize.py # Recursive chunk summarization
44+
│ ├── search.py # Similarity search (no LLM answering)
45+
│ ├── parse.py # Raw document extraction
46+
│ └── shared_query_search.py
47+
└── customs/ # Custom implementations
48+
├── litellm_embeddings.py # LiteLLM-based embeddings
49+
├── binary_faiss_vectorstore.py # Binary FAISS (~32x compression)
50+
├── compressed_embeddings_cacher.py
51+
└── fix_llm_caching.py
52+
```
53+
54+
---
55+
56+
## Data Flow
57+
58+
```
59+
User Input (CLI / Python API)
60+
│
61+
▼
62+
CLI Parser (__main__.py) ── Google Fire argument parsing
63+
│ Piped input detection (JSON, TOML, text)
64+
▼
65+
wdoc.__init__() ── Configuration & validation
66+
│
67+
├──► Document Loading (batch_file_loader.py)
68+
│ 1. Infer filetype (regex rules + file magic)
69+
│ 2. Expand recursive types (directories, playlists, JSON arrays)
70+
│ 3. Parallel load via joblib (loaders/*.py)
71+
│ 4. Text splitting (RecursiveCharacterTextSplitter or semantic)
72+
│ 5. Metadata enrichment (hash, title, source, reading time)
73+
│
74+
├──► Embedding & Vector Store (embeddings.py)
75+
│ 1. Initialize embedding model (LiteLLMEmbeddings)
76+
│ 2. Cache-backed embedding layer
77+
│ 3. Build FAISS index (standard or binary-compressed)
78+
│
79+
└──► Task Execution
80+
│
81+
├─ query/search → Retrieval pipeline (see below)
82+
├─ summarize → Chunk-by-chunk summarization
83+
└─ parse → Return raw documents (no LLM)
84+
```
85+
86+
---
87+
88+
## Three LLM Roles
89+
90+
| Role | Default Model | Purpose |
91+
|------|--------------|---------|
92+
| **Main model** | `gemini-3.1-pro` | Answering, summarizing |
93+
| **Eval model** | `gemini-2.5-flash` | Document relevance checking (cheap/fast) |
94+
| **Embed model** | `text-embedding-3-small` | Dense vector embeddings |
95+
96+
All models are loaded through **LiteLLM**, supporting 100+ providers (OpenAI, Anthropic, Google, Mistral, Ollama, OpenRouter, etc.).
97+
98+
---
99+
100+
## Query Pipeline (RAG)
101+
102+
The query task uses named AI personas in sequence:
103+
104+
1. **Raphael (Rephraser)** — Expands user query into multiple alternative phrasings
105+
2. **Vector Store** — Embeds queries and retrieves top-k similar document chunks
106+
3. **Eve (Evaluator)** — Checks each chunk's relevance to the query (eval model, cheap)
107+
4. **Anna (Answerer)** — Extracts an answer from each relevant chunk (main model)
108+
5. **Carl (Combiner)** — Hierarchically clusters and combines intermediate answers into a final response
109+
110+
**Smart top-k expansion**: starts at top_k (default 200); if >90% of documents are relevant, automatically increases top_k and retries until diminishing returns.
111+
112+
---
113+
114+
## Summarization Pipeline
115+
116+
1. Split document into chunks (with overlap)
117+
2. **Sam (Summarizer)** summarizes each chunk, passing the previous chunk's summary as context
118+
3. If `summary_n_recursion > 0`, recursively summarize the summaries
119+
4. Returns a `wdocSummary` dataclass with the full result tree
120+
121+
---
122+
123+
## Configuration System
124+
125+
**`utils/env.py`** defines `EnvDataclass`, a frozen dataclass with 50+ fields loaded from `WDOC_*` environment variables. CLI arguments override env vars.
126+
127+
Key variables:
128+
- `WDOC_DEFAULT_MODEL` / `WDOC_DEFAULT_EMBED_MODEL` / `WDOC_DEFAULT_QUERY_EVAL_MODEL`
129+
- `WDOC_PRIVATE_MODE` — blocks outbound connections, redacts API keys
130+
- `WDOC_MAX_CHUNK_SIZE` — max tokens per chunk
131+
- `WDOC_FAISS_COMPRESSION` — enable binary embeddings
132+
133+
---
134+
135+
## Document Loaders
136+
137+
Each filetype has a dedicated loader in `utils/loaders/`. The dispatcher (`load_one_doc()`) dynamically imports the `load_{filetype}` function.
138+
139+
**Supported types**: `pdf`, `txt`, `word`, `powerpoint`, `epub`, `url`, `youtube`, `youtube_playlist`, `online_pdf`, `online_media`, `local_audio`, `local_video`, `anki`, `logseq_markdown`, `json_dict`, `recursive_paths`, `json_entries`, `toml_entries`, `string`, `ddg` (DuckDuckGo search).
140+
141+
**PDF parsing** is particularly robust: 15 different parser backends are evaluated and the best result is selected automatically.
142+
143+
**URL loading** has multiple fallbacks: Jina → Playwright → Selenium → others.
144+
145+
---
146+
147+
## Caching Layers
148+
149+
1. **LLM response cache** — `SQLiteCacheFixed` prevents duplicate LLM calls
150+
2. **Embedding cache** — `CacheBackedEmbeddings` avoids re-embedding identical content
151+
3. **Loader cache** — `joblib.Memory` caches parsed documents
152+
4. **Document hashing** — content_hash / file_hash / all_hash prevent reprocessing
153+
154+
---
155+
156+
## Error Handling
157+
158+
Custom exceptions in `utils/errors.py`:
159+
- `NoDocumentsRetrieved` — no documents found in vector store
160+
- `NoDocumentsAfterLLMEvalFiltering` — all retrieved docs deemed irrelevant
161+
- `NoRelevantIntermediateAnswers` — LLM found no useful answers
162+
- `TimeoutPdfLoaderError` — PDF parsing timeout
163+
164+
Fallback strategies: multiple parser/loader backends tried in sequence before failing.
165+
166+
---
167+
168+
## Key Dependencies
169+
170+
| Category | Libraries |
171+
|----------|-----------|
172+
| RAG framework | LangChain, LangChain-Community |
173+
| LLM abstraction | LiteLLM, LangChain-LiteLLM |
174+
| Vector store | FAISS |
175+
| Text splitting | Chonkie (semantic), LangChain (recursive) |
176+
| Document parsing | Unstructured, OpenParse, BeautifulSoup, lxml |
177+
| Media | yt-dlp, Deepgram-SDK, pydub |
178+
| Web scraping | Playwright, Selenium, Jina |
179+
| Parallel processing | Joblib |
180+
| Type checking | Beartype (runtime) |
181+
| Logging | Loguru, Rich |
182+
| Observability | Langfuse |
183+
| Caching/storage | SQLAlchemy, LMDB (PersistDict) |

‎README.md‎

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@ Created by a psychiatry resident who needed a way to get a definitive answer fro
3434
* **Web Search**: Preliminary web search support using [DuckDuckGo](https://en.wikipedia.org/wiki/DuckDuckGo) (via the [ddgs](https://pypi.org/project/ddgs/) library)
3535

3636
### Table of contents
37+
- [Comprehensive reference (SKILL.md)](#comprehensive-reference)
3738
- [Explanatory diagrams](#explanatory-diagrams)
3839
- [Ultra short guide for people in a hurry](#ultra-short-guide-for-people-in-a-hurry)
3940
- [Features](#features)
@@ -47,6 +48,10 @@ Created by a psychiatry resident who needed a way to get a definitive answer fro
4748
- [FAQ](#faq)
4849
- [Roadmap](#roadmap)
4950

51+
## Comprehensive reference
52+
53+
A single-page comprehensive reference covering every CLI argument, environment variable, filetype, and the full Python API can be found in **[SKILL.md](./SKILL.md)**.
54+
5055
## Explanatory diagrams
5156

5257
<p float="left" align="middle">
@@ -79,7 +84,7 @@ wdoc --path=$link --task=query --filetype="online_pdf" --query="What does it say
7984
4. Use those embeddings to search through all chunks of the text and get the 200 most appropriate documents
8085
5. Pass each of those documents to the smaller LLM (default: openrouter/google/gemini-2.5-flash) to tell us if the document seems appropriate given the user query
8186
6. If More than 90% of the 200 documents are appropriate, then we do another search with a higher top_k and repeat until documents start to be irrelevant OR we it 500 documents.
82-
7. Then each relevant doc is sent to the strong LLM (by default, openrouter/google/gemini-2.5-pro) to extract relevant info and give one answer per relevant document.
87+
7. Then each relevant doc is sent to the strong LLM (by default, openrouter/google/gemini-3.1-pro-preview) to extract relevant info and give one answer per relevant document.
8388
8. Then all those "intermediate" answers are 'semantic batched' (meaning we create embeddings, do hierarchical clustering, then create small batch containing several intermediate answers of similar semantics, sort the batch in semantic order too), each batch is combined into a single answer per batch of relevant doc (or after: per batch of batches).
8489
9. Rinse and repeat steps 7+8 (i.e. gradually aggregate batches) until we have only one answer, that is returned to the user.
8590

@@ -92,7 +97,7 @@ wdoc --path=$link --task=summarize --filetype="online_pdf"
9297
```
9398
* This will:
9499
1. Split the text into chunks
95-
2. pass each chunk into the strong LLM (by default openrouter/google/gemini-2.5-pro) for a very low level (=with all details) summary. The format is markdown bullet points for each idea and with logical indentation.
100+
2. pass each chunk into the strong LLM (by default openrouter/google/gemini-3.1-pro-preview) for a very low level (=with all details) summary. The format is markdown bullet points for each idea and with logical indentation.
96101
3. When creating each new chunk, the LLM has access to the previous chunk for context.
97102
4. All summary are then concatenated and returned to the user
98103

@@ -196,7 +201,11 @@ Refer to [examples.md](https://github.com/thiswillbeyourgithub/wdoc/blob/main/wd
196201
* Install the `wdoc[full]` version except if you have specific constraints.
197202
* try to install pdftotext with `pip install -U wdoc[pdftotext]` as well as add fasttext support with `pip install -U wdoc[fasttext]`.
198203
* If you plan on contributing, you will also need `wdoc[dev]` for the commit hooks.
199-
2. Add the API key for the backend you want as an environment variable: for example `export OPENAI_API_KEY="***my_key***"`
204+
* **Claude Code users**: to give Claude Code knowledge of `wdoc`'s CLI and Python API, install the [SKILL.md](./SKILL.md) reference file:
205+
```bash
206+
mkdir -p ~/.claude/skills/wdoc && wget -O ~/.claude/skills/wdoc/SKILL.md https://raw.githubusercontent.com/thiswillbeyourgithub/wdoc/main/SKILL.md
207+
```
208+
2. Add the API key for the backend you want as an environment variable: for example `export ANTHROPIC_API_KEY="***my_key***"`
200209
3. Launch is as easy as using `wdoc --task=query --path=MYDOC [ARGS]`
201210
* If for some reason this fails, maybe try with `python -m wdoc`. And if everything fails, try with `uvx wdoc@latest`, or as last resort clone this repo and try again after `cd` inside it? Don't hesitate to open an issue.
202211
* To get shell autocompletion: if you're using zsh: `eval $(cat shell_completions/wdoc_completion.zsh)`. Also provided for `bash` and `fish`. You can generate your own with `wdoc -- --completion MYSHELL > my_completion_file"`.

0 commit comments

Comments
 (0)