You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
All models are loaded through **LiteLLM**, supporting 100+ providers (OpenAI, Anthropic, Google, Mistral, Ollama, OpenRouter, etc.).
97
99
98
-
The defaults are duplicated across `wdoc/utils/env.py`, the docs, `README.md`, `SKILL.md`, this file, and `docker/env.example`. Use `./bump_default_models.sh` at the repo root to change them in one shot (dry-run by default, `--apply` to write).
100
+
The defaults are duplicated across `wdoc/utils/env.py`, the docs, `README.md`, `wdoc-skill/REFERENCE.md`, this file, and `docker/env.example`. Use `./bump_default_models.sh` at the repo root to change them in one shot (dry-run by default, `--apply` to write).
99
101
100
102
---
101
103
@@ -138,7 +140,7 @@ Key variables:
138
140
139
141
Each filetype has a dedicated loader in `utils/loaders/`. The dispatcher (`load_one_doc()`) dynamically imports the `load_{filetype}` function.
Copy file name to clipboardExpand all lines: CLAUDE.md
+25-2Lines changed: 25 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -66,6 +66,7 @@ Each persona can be customized via `WDOC_{NAME}_INSTRUCTIONS` env vars.
66
66
Any new setting (CLI argument or environment variable) **must** be documented in:
67
67
-`wdoc/docs/help.md` — describe the setting, its type, default value, and accepted values.
68
68
-`wdoc/docs/examples.md` — add usage examples where appropriate.
69
+
-`wdoc-skill/REFERENCE.md`: the comprehensive user-facing reference; mirror the new setting in the relevant table (see [Keeping the `wdoc-skill/` reference in Sync](#keeping-the-wdoc-skill-reference-in-sync)).
69
70
70
71
New settings should be either:
71
72
- A **CLI argument** (defined in `wdoc.py`'s main class), or
@@ -82,15 +83,21 @@ When adding new CLI arguments or loader-specific parameters, update these dicts
82
83
83
84
## Adding Support for a New Filetype
84
85
86
+
Adding a filetype is intentionally self-contained: it touches a new loader module plus a couple of registration points, never the core RAG pipeline. The recent `zotero` and `karakeep` types are good references for the two routes available:
87
+
-**Standalone loader** (most types): write a `load_*` function that returns `List[Document]` and register it (steps below).
88
+
-**Recursive fan-out** (`zotero`, `karakeep`, `recursive_paths`, ...): resolve a selector/path into many sub-documents and delegate parsing to *existing* loaders via DocDicts, so you reuse the existing parsers instead of writing new parsing code. Register the function in `recursive_types_func_mapping` in `wdoc/utils/load_recursive.py`.
89
+
90
+
Standalone loader steps:
91
+
85
92
1.**Create a loader** in `wdoc/utils/loaders/` — add a file (e.g. `myformat.py`) containing a function `load_myformat(path, file_hash, ...) -> List[Document]`. Use the `@debug_return_empty` and `@optional_strip_unexp_args` decorators (see `txt.py` for a minimal example).
86
93
2.**Register the filetype** — add `"myformat"` to the `LOADABLE_FILETYPE` list in `wdoc/utils/loaders/__init__.py`.
87
94
3.**Add loader-specific args** (if any) to `filetype_arg_types` in `wdoc/utils/misc.py`.
88
-
4.**Document it** — add the filetype and its arguments to `wdoc/docs/help.md`, and add examples to `wdoc/docs/examples.md`.
95
+
4.**Document it** — add the filetype and its arguments to `wdoc/docs/help.md`, add examples to `wdoc/docs/examples.md`, add the filetype + its loader args to the tables in `wdoc-skill/REFERENCE.md`, and a shell example to `wdoc-skill/EXAMPLES.md` (see [Keeping the `wdoc-skill/` reference in Sync](#keeping-the-wdoc-skill-reference-in-sync)).
89
96
5.**Auto-detection** (optional) — if the filetype corresponds to a file extension, add a mapping in the auto-detection logic so `--filetype=auto` can infer it.
90
97
91
98
## Bumping the Default Models
92
99
93
-
The two default LLM identifiers (`WDOC_DEFAULT_MODEL` and `WDOC_DEFAULT_QUERY_EVAL_MODEL`) are duplicated across `wdoc/utils/env.py`, `wdoc/docs/help.md`, `SKILL.md`, `README.md`, `ARCHITECTURE.md`, and `docker/env.example`. Use the repo-root helper instead of editing each file by hand:
100
+
The two default LLM identifiers (`WDOC_DEFAULT_MODEL` and `WDOC_DEFAULT_QUERY_EVAL_MODEL`) are duplicated across `wdoc/utils/env.py`, `wdoc/docs/help.md`, `wdoc-skill/REFERENCE.md`, `README.md`, `ARCHITECTURE.md`, and `docker/env.example`. Use the repo-root helper instead of editing each file by hand:
@@ -99,6 +106,22 @@ The two default LLM identifiers (`WDOC_DEFAULT_MODEL` and `WDOC_DEFAULT_QUERY_EV
99
106
100
107
`env.py` is the source of truth for the current values. The script replaces both the full id (`provider/path/name`) and the basename, and re-syncs the `KEY=VALUE` lines in `docker/env.example` independently (so it recovers from prior drift). It never commits; review with `git diff` and commit yourself.
101
108
109
+
## Keeping the `wdoc-skill/` reference in Sync
110
+
111
+
The user-facing reference lives in the `wdoc-skill/` directory (a Claude Code skill), split into three files:
112
+
-`wdoc-skill/SKILL.md`: concise orientation (frontmatter with trigger description, quick start, the four tasks, core mechanics). Keep it short.
113
+
-`wdoc-skill/REFERENCE.md`: the comprehensive tables (Installation, all CLI argument groups, Filetypes, Recursive Filetypes, Loader-Specific Arguments, Filtering, Environment Variables, and the full Python API).
114
+
-`wdoc-skill/EXAMPLES.md`: copy-pasteable shell and Python examples.
115
+
116
+
These are **not** auto-generated, so they drift unless updated by hand. Whenever you change user-visible behavior, keep them in sync:
117
+
- New CLI argument, env var, or loader-specific argument: add it to the relevant table in `REFERENCE.md` (Global/Model/Embedding/Query/Summary Arguments, Environment Variables, or Loader-Specific Arguments).
118
+
- New filetype: add a row to the Filetypes or Recursive Filetypes table in `REFERENCE.md`, plus its loader args, and ideally a shell example in `EXAMPLES.md`.
119
+
- Changed default, accepted values, or behavior of an existing setting: update the matching row in `REFERENCE.md`.
120
+
- New install extra: update the Installation section in `REFERENCE.md` (and the quick start in `SKILL.md` if relevant).
121
+
- On a version bump, update the `> Written for **wdoc vX.Y.Z**` header line at the top of all three files.
122
+
123
+
Do not use em-dashes anywhere in these files. `wdoc/docs/help.md` and `wdoc/docs/examples.md` are the source the reference mirrors; if you edited those, mirror the change here too.
124
+
102
125
## Building Documentation
103
126
104
127
To regenerate the Sphinx autodoc files, run from the repo root:
Copy file name to clipboardExpand all lines: README.md
+9-4Lines changed: 9 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -35,7 +35,7 @@ Created by a psychiatry resident who needed a way to get a definitive answer fro
35
35
***Web Search**: Preliminary web search support using [DuckDuckGo](https://en.wikipedia.org/wiki/DuckDuckGo) (via the [ddgs](https://pypi.org/project/ddgs/) library)
-[Ultra short guide for people in a hurry](#ultra-short-guide-for-people-in-a-hurry)
41
41
-[Features](#features)
@@ -51,7 +51,7 @@ Created by a psychiatry resident who needed a way to get a definitive answer fro
51
51
52
52
## Comprehensive reference
53
53
54
-
A single-page comprehensive reference covering every CLI argument, environment variable, filetype, and the full Python API can be found in **[SKILL.md](./SKILL.md)**.
54
+
A comprehensive reference covering every CLI argument, environment variable, filetype, and the full Python API lives in the **[`wdoc-skill/`](./wdoc-skill/)** directory, packaged as a Claude Code skill. It has three files: [`SKILL.md`](./wdoc-skill/SKILL.md) (a quick orientation), [`REFERENCE.md`](./wdoc-skill/REFERENCE.md) (the full argument, env var, filetype, and Python API tables), and [`EXAMPLES.md`](./wdoc-skill/EXAMPLES.md) (copy-pasteable shell and Python recipes).
***recursive_paths**: turns a path, a regex pattern and a filetype into all the files found recurisvely, and treated a the specified filetype (for example many PDFs or lots of HTML files etc).
183
183
***toml_entries**: read a .toml file. An example can be found in `docs/toml_entries_example.toml`.
184
184
***youtube playlists**: get the link for each video then process as **youtube**
185
+
***zotero**: load documents straight from a [Zotero](https://www.zotero.org/) library via [pyzotero](https://github.com/urschrei/pyzotero). The `--path` selects a collection, tag, item keys, saved search or the whole library, and each attachment is fed to the regular loaders. Requires the `zotero` extra. See [help.md](https://github.com/thiswillbeyourgithub/wdoc/docs/help.md) for the selector syntax and options.
186
+
***karakeep**: load bookmarks straight from a [Karakeep](https://karakeep.app/) instance via [karakeep-python-api](https://github.com/thiswillbeyourgithub/karakeep_python_api). The `--path` selects a list, tag, search, bookmark ids or the whole library, and each bookmark's stored content (crawled html, text, or a stored pdf asset) is fed to the regular loaders without re-fetching the live url. Requires the `karakeep` extra. See [help.md](https://github.com/thiswillbeyourgithub/wdoc/docs/help.md) for the selector syntax and options.
185
187
186
188
## Walkthrough and examples
187
189
@@ -208,9 +210,12 @@ Refer to [examples.md](https://github.com/thiswillbeyourgithub/wdoc/blob/main/wd
208
210
*`wdoc[full]` is a shortcut that includes all the loader extras above (excluding `fasttext` and `pdftotext`, which need special handling). If unsure, use `wdoc[full]` and don't worry about filetypes.
209
211
* If you have problems with pdftotext or fasttext, try `uvx wdoc[full,pdftotext,fasttext]`.
210
212
* If you plan on contributing, you will also need `wdoc[dev]` for the commit hooks.
211
-
***Claude Code users**: to give Claude Code knowledge of `wdoc`'s CLI and Python API, install the [SKILL.md](./SKILL.md) reference file:
213
+
***Claude Code users**: to give Claude Code knowledge of `wdoc`'s CLI and Python API, install the [`wdoc-skill/`](./wdoc-skill/) skill (three files: `SKILL.md`, `REFERENCE.md`, `EXAMPLES.md`):
Copy file name to clipboardExpand all lines: setup.py
+11-3Lines changed: 11 additions & 3 deletions
Original file line number
Diff line number
Diff line change
@@ -129,7 +129,7 @@ def run(self):
129
129
130
130
setup(
131
131
name="wdoc",
132
-
version="5.1.0",
132
+
version="5.1.1",
133
133
description="A perfect AI powered RAG for document query and summary. Supports ~all LLM and ~all filetypes (url, pdf, epub, youtube (incl playlist), audio, anki, md, docx, pptx, or any combination!)",
134
134
long_description=long_description,
135
135
long_description_content_type="text/markdown",
@@ -205,7 +205,7 @@ def run(self):
205
205
"blake3>=1.0.8", # faster than sha256
206
206
"pandas >= 2.3.3",
207
207
"trio >= 0.31.0", # for some reason older versions of trio, when present are used and cause issues on python 3.11: https://github.com/python-trio/trio/issues/2317
208
-
"unstructured >= 0.18.15", # base package only, used by pdf loader for clean_extra_whitespace. The heavy [all-docs] extra is in [office].
208
+
"unstructured >= 0.18.15,<0.18.31", # base package only, used by pdf loader for clean_extra_whitespace. The heavy [all-docs] extra is in [office].
209
209
# PDF loading (default, since pdf is the most common filetype)
210
210
"openparse[ml] >= 0.5.7", # pdf with table support
@@ -35,6 +36,28 @@ if [[ -z "${WDOC_WHISPER_API_KEY:-}" ]]; then
35
36
echo"ERROR: WDOC_WHISPER_API_KEY env var is not set but is required to test whisper transcription. Set WDOC_WHISPER_API_KEY before running tests.">&2
36
37
exit 1
37
38
fi
39
+
# Zotero loader: the api test needs real credentials (or a reachable local
40
+
# Zotero) plus a small selector to fan out. Mirror the whisper check so a
41
+
# missing setup crashes early instead of silently skipping the test.
42
+
if [[ -z"${ZOTERO_API_KEY:-}" ]];then
43
+
echo"ERROR: ZOTERO_API_KEY env var is not set but is required to test the zotero loader. Set ZOTERO_API_KEY (and ZOTERO_LIBRARY_ID) before running tests.">&2
44
+
exit 1
45
+
fi
46
+
if [[ -z"${ZOTERO_COLLECTION_NAME:-}" ]];then
47
+
echo"ERROR: ZOTERO_COLLECTION_NAME env var is not set but is required to exercise the zotero loader api test. Set it to the name of a (preferably small) collection in your library.">&2
48
+
exit 1
49
+
fi
50
+
# Karakeep loader: the api test creates its own temporary bookmark and deletes
51
+
# it, so it only needs real credentials (endpoint + api key). Mirror the zotero
52
+
# check so a missing setup crashes early instead of silently skipping the test.
53
+
if [[ -z"${KARAKEEP_PYTHON_API_KEY:-}" ]];then
54
+
echo"ERROR: KARAKEEP_PYTHON_API_KEY env var is not set but is required to test the karakeep loader. Set KARAKEEP_PYTHON_API_KEY (and KARAKEEP_PYTHON_API_ENDPOINT) before running tests.">&2
55
+
exit 1
56
+
fi
57
+
if [[ -z"${KARAKEEP_PYTHON_API_ENDPOINT:-}" ]];then
58
+
echo"ERROR: KARAKEEP_PYTHON_API_ENDPOINT env var is not set but is required to test the karakeep loader. Set it to your instance URL including /api/v1/.">&2
59
+
exit 1
60
+
fi
38
61
39
62
# cleanup previous
40
63
[[ "$(type deactivate)"=="deactivate is a shell function"* ]] && deactivate
0 commit comments