Merge branch 'dev'

Author: thiswillbeyourgithub

ARCHITECTURE.md

@@ -89,12 +89,14 @@ wdoc.__init__() ── Configuration & validation
 
 | Role | Default Model | Purpose |
 |------|--------------|---------|
-| **Main model** | `gemini-3.1-pro` | Answering, summarizing |
-| **Eval model** | `gemini-2.5-flash` | Document relevance checking (cheap/fast) |
+| **Main model** | `deepseek-v4-pro` | Answering, summarizing |
+| **Eval model** | `deepseek-v4-flash` | Document relevance checking (cheap/fast) |
 | **Embed model** | `text-embedding-3-small` | Dense vector embeddings |
 
 All models are loaded through **LiteLLM**, supporting 100+ providers (OpenAI, Anthropic, Google, Mistral, Ollama, OpenRouter, etc.).
 
+The defaults are duplicated across `wdoc/utils/env.py`, the docs, `README.md`, `SKILL.md`, this file, and `docker/env.example`. Use `./bump_default_models.sh` at the repo root to change them in one shot (dry-run by default, `--apply` to write).
+
 ---
 
 ## Query Pipeline (RAG)

CLAUDE.md

@@ -0,0 +1,116 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+wdoc is a production-grade RAG system for document summarization, searching, and querying across 20+ file types. It serves as both a CLI tool (via Google Fire) and a Python library (`from wdoc import wdoc`). All LLM calls go through LiteLLM (100+ providers). See `ARCHITECTURE.md` for detailed data flow and component diagrams.
+
+## Commands
+
+### Install
+```bash
+pip install -e .[full]
+```
+
+### Test
+```bash
+# All tests
+pytest tests --quiet
+
+# By category (basic = no API keys needed, api = needs credentials)
+pytest -m basic tests/test_wdoc.py
+pytest -m api tests/test_wdoc.py
+
+# Single test
+pytest tests/test_wdoc.py::test_wdoc_version
+
+# Parallelized
+pytest -n auto -m basic tests/test_parsing.py
+
+# CLI integration tests
+cd tests && ./test_cli.sh api
+```
+
+Test environment variables: `PYTEST_IS_TESTING_WDOC=true`, `WDOC_TYPECHECKING=crash`, `WDOC_DISABLE_EMBEDDINGS_CACHE=true`.
+
+### Lint / Format
+```bash
+ruff format wdoc/
+pre-commit run --all-files   # runs ruff-format + pytest (on pre-merge)
+```
+
+## Architecture Essentials
+
+**Entry points**: `__main__.py` (CLI via Google Fire) → `wdoc.py` (main orchestration class, ~3000 lines).
+
+**Named AI personas** drive the RAG pipeline in sequence:
+- **Raphael** — rephrases user query into alternatives
+- **Eve** — evaluates chunk relevance (cheap eval model)
+- **Anna** — extracts answers from relevant chunks (main model)
+- **Carl** — combines intermediate answers hierarchically
+- **Sam** — summarizes chunks (summarization task)
+
+Each persona can be customized via `WDOC_{NAME}_INSTRUCTIONS` env vars.
+
+**Configuration**: `utils/env.py` defines `EnvDataclass` — a frozen dataclass loading 50+ `WDOC_*` environment variables. CLI args override env vars.
+
+**Document loading**: `utils/loaders/` has per-filetype loaders dispatched by `load_one_doc()`. PDF loading tries 15 parser backends and picks the best. URL loading has Jina → Playwright → Selenium fallback chain.
+
+**Caching**: four layers — LLM responses (SQLite), embeddings (CacheBackedEmbeddings), parsed documents (joblib.Memory), and document content hashing.
+
+**Type checking**: Beartype runtime checking, controlled by `WDOC_TYPECHECKING` env var (crash/warn/disabled).
+
+## Adding New Settings
+
+Any new setting (CLI argument or environment variable) **must** be documented in:
+- `wdoc/docs/help.md` — describe the setting, its type, default value, and accepted values.
+- `wdoc/docs/examples.md` — add usage examples where appropriate.
+
+New settings should be either:
+- A **CLI argument** (defined in `wdoc.py`'s main class), or
+- A **`WDOC_*` environment variable** (defined in `wdoc/utils/env.py`'s `EnvDataclass`).
+
+**Environment variables are re-read on every access, not just at declaration.** The `EnvDataclass.__getattribute__` method checks `os.environ` at access time when the dataclass is frozen, so changing an env var between wdoc instantiations (or even at runtime) takes effect without reimporting.
+
+## Variables in `wdoc/utils/misc.py` to Keep Updated
+
+When adding new CLI arguments or loader-specific parameters, update these dicts in `wdoc/utils/misc.py`:
+- `filetype_arg_types` — maps loader-specific argument names to their types (e.g. `"whisper_lang": str`).
+- `extra_args_types` — maps extra wdoc instantiation arguments to their types. It merges `filetype_arg_types` automatically.
+- `DocDict.allowed_keys` — derives from `filetype_arg_types` keys automatically, but verify new keys appear.
+
+## Adding Support for a New Filetype
+
+1. **Create a loader** in `wdoc/utils/loaders/` — add a file (e.g. `myformat.py`) containing a function `load_myformat(path, file_hash, ...) -> List[Document]`. Use the `@debug_return_empty` and `@optional_strip_unexp_args` decorators (see `txt.py` for a minimal example).
+2. **Register the filetype** — add `"myformat"` to the `LOADABLE_FILETYPE` list in `wdoc/utils/loaders/__init__.py`.
+3. **Add loader-specific args** (if any) to `filetype_arg_types` in `wdoc/utils/misc.py`.
+4. **Document it** — add the filetype and its arguments to `wdoc/docs/help.md`, and add examples to `wdoc/docs/examples.md`.
+5. **Auto-detection** (optional) — if the filetype corresponds to a file extension, add a mapping in the auto-detection logic so `--filetype=auto` can infer it.
+
+## Bumping the Default Models
+
+The two default LLM identifiers (`WDOC_DEFAULT_MODEL` and `WDOC_DEFAULT_QUERY_EVAL_MODEL`) are duplicated across `wdoc/utils/env.py`, `wdoc/docs/help.md`, `SKILL.md`, `README.md`, `ARCHITECTURE.md`, and `docker/env.example`. Use the repo-root helper instead of editing each file by hand:
+
+```bash
+./bump_default_models.sh <NEW_STRONG_MODEL> <NEW_EVAL_MODEL>           # dry-run preview
+./bump_default_models.sh <NEW_STRONG_MODEL> <NEW_EVAL_MODEL> --apply   # write
+```
+
+`env.py` is the source of truth for the current values. The script replaces both the full id (`provider/path/name`) and the basename, and re-syncs the `KEY=VALUE` lines in `docker/env.example` independently (so it recovers from prior drift). It never commits; review with `git diff` and commit yourself.
+
+## Building Documentation
+
+To regenerate the Sphinx autodoc files, run from the repo root:
+```bash
+sphinx-apidoc -o docs/source/ wdoc --force
+```
+
+## Key Conventions
+
+- `utils/misc.py` is a large utility module (~53KB) — search it before creating new helpers.
+- Piped input to CLI is auto-detected as JSON, TOML, URLs, or file paths (`__main__.py`).
+- `--private` / `WDOC_PRIVATE_MODE=true` blocks all outbound connections and redacts API keys.
+- Binary FAISS embeddings (`WDOC_FAISS_COMPRESSION`) give ~32x compression; implemented in `utils/customs/binary_faiss_vectorstore.py`.
+- Semantic batching clusters intermediate answers before combining (scipy hierarchical clustering).
+- Use `python` not `python3` in commands.

README.md

@@ -1,4 +1,5 @@
 [![PyPI version](https://badge.fury.io/py/wdoc.svg)](https://badge.fury.io/py/wdoc)
+[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/thiswillbeyourgithub/wdoc)
 
 # wdoc
 
@@ -82,9 +83,9 @@ wdoc --path=$link --task=query --filetype="online_pdf" --query="What does it say
     2. cut the text into chunks and create embeddings for each
     3. Take the user query, create embeddings for it ('basic') AND ask the default LLM to generate alternative queries and embed those
     4. Use those embeddings to search through all chunks of the text and get the 200 most appropriate documents
-    5. Pass each of those documents to the smaller LLM (default: openrouter/google/gemini-2.5-flash) to tell us if the document seems appropriate given the user query
+    5. Pass each of those documents to the smaller LLM (default: openrouter/deepseek/deepseek-v4-flash) to tell us if the document seems appropriate given the user query
     6. If More than 90% of the 200 documents are appropriate, then we do another search with a higher top_k and repeat until documents start to be irrelevant OR we it 500 documents.
-    7. Then each relevant doc is sent to the strong LLM (by default, openrouter/google/gemini-3.1-pro-preview) to extract relevant info and give one answer per relevant document.
+    7. Then each relevant doc is sent to the strong LLM (by default, openrouter/deepseek/deepseek-v4-pro) to extract relevant info and give one answer per relevant document.
     8. Then all those "intermediate" answers are 'semantic batched' (meaning we create embeddings, do hierarchical clustering, then create small batch containing several intermediate answers of similar semantics, sort the batch in semantic order too), each batch is combined into a single answer per batch of relevant doc (or after: per batch of batches).
     9. Rinse and repeat steps 7+8 (i.e. gradually aggregate batches) until we have only one answer, that is returned to the user.
 
@@ -97,7 +98,7 @@ wdoc --path=$link --task=summarize --filetype="online_pdf"
 ```
 * This will:
     1. Split the text into chunks
-    2. pass each chunk into the strong LLM (by default openrouter/google/gemini-3.1-pro-preview) for a very low level (=with all details) summary. The format is markdown bullet points for each idea and with logical indentation.
+    2. pass each chunk into the strong LLM (by default openrouter/deepseek/deepseek-v4-pro) for a very low level (=with all details) summary. The format is markdown bullet points for each idea and with logical indentation.
     3. When creating each new chunk, the LLM has access to the previous chunk for context.
     4. All summary are then concatenated and returned to the user
 
@@ -282,6 +283,10 @@ FAQ
         no argument but returning a `List[Document]`. Take a look at the `OpenparseDocumentParser`
         class for an example.
 
+* **Can `wdoc` add source citations to summaries?**
+    * Yes! When summarizing documents that have page metadata (like PDFs), `wdoc` automatically adds `[p.N]` citations to bullet points tracking which page the information came from. For multi-file summaries, citations include the filename: `[p.N, file.pdf]`. You can also use `--citation_url_template` to turn these into clickable markdown links pointing to your own document server (e.g. `--citation_url_template="https://my-site.com/docs/{source}#page={page}"`). This feature was developed with Claude Code.
+    * For the query task, source documents are referenced with clickable anchor links `[N](#document-N)` in the final answer.
+
 * **What should I do if I keep hitting rate limits?**
     * The simplest way is to add the `debug` argument. It will disable multithreading,
         multiprocessing and LLM concurrency. A less harsh alternative is to set the

SKILL.md

@@ -254,8 +254,8 @@ These load multiple documents and can combine different sources:
 
 | Variable | Default | Description |
 |----------|---------|-------------|
-| `WDOC_DEFAULT_MODEL` | `openrouter/google/gemini-3.1-pro-preview` | Default strong LLM |
-| `WDOC_DEFAULT_QUERY_EVAL_MODEL` | `openrouter/google/gemini-2.5-flash` | Default eval LLM |
+| `WDOC_DEFAULT_MODEL` | `openrouter/deepseek/deepseek-v4-pro` | Default strong LLM |
+| `WDOC_DEFAULT_QUERY_EVAL_MODEL` | `openrouter/deepseek/deepseek-v4-flash` | Default eval LLM |
 | `WDOC_DEFAULT_EMBED_MODEL` | `openai/text-embedding-3-small` | Default embedding model |
 | `WDOC_DEFAULT_EMBED_DIMENSION` | `none` | Embedding dimensions to request |