Skip to content

Release 5.1.1

Choose a tag to compare

View all tags
@thiswillbeyourgithub thiswillbeyourgithub released this 23 Jun 14:18
5.1.1
6feb5f1

What's new

PLACEHOLDER

Commits details since the last release

bumpver.toml
docs/source/conf.py
setup.py
wdoc/wdoc.py

setup.py

  • [ed81e2a] by @thiswillbeyourgithub, 7 days ago:
    docs: restructure SKILL.md into the wdoc-skill/ skill directory
    Split the monolithic root SKILL.md into a Claude Code skill directory
    following the write-a-skill conventions:
  • wdoc-skill/SKILL.md: concise orientation with frontmatter + trigger
    description, quick start, the four tasks, and core mechanics (73 lines)
  • wdoc-skill/REFERENCE.md: full CLI args, env vars, filetypes, loader
    options, and the complete Python API tables
  • wdoc-skill/EXAMPLES.md: copy-pasteable shell and Python recipes

Polished wording, added the skill frontmatter, and removed em-dashes from
the reference content. Updated every reference to the old root SKILL.md
(bump_default_models.sh, README.md, CLAUDE.md, ARCHITECTURE.md) and the
Claude Code install instructions to the new three-file layout.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

ARCHITECTURE.md
CLAUDE.md
README.md
bump_default_models.sh
wdoc-skill/EXAMPLES.md
wdoc-skill/REFERENCE.md
wdoc-skill/SKILL.md

  • [af68f46] by @thiswillbeyourgithub, 9 days ago:
    docs: note how self-contained adding a new filetype is
    Add a compact lead-in to CLAUDE.md's "Adding Support for a New Filetype"
    section describing the two routes (standalone loader vs recursive fan-out)
    and citing the recent zotero/karakeep types, plus a short note in SKILL.md
    after the recursive filetypes table pointing contributors at CLAUDE.md.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

CLAUDE.md
SKILL.md

  • [a5473f4] by @thiswillbeyourgithub, 9 days ago:
    fix(summarize): dump LLM output on empty content, fix recursion dedup
    Follow-up to the previous empty-summary fix, addressing review feedback.

  • Empty message content: instead of retrying (which just reproduces the
    same blank when the cause is structural), raise a ValueError that dumps
    every attribute of the LLM generations and llm_output. For some
    reasoning models litellm leaves the message content empty and puts the
    answer in additional_kwargs['reasoning_content'], so the dump reveals
    when we are simply reading the wrong field
    (generations[0].text == message.content).

  • Recursion dedup bug: if summary_text not in recursive_summaries tested
    membership against the dict's integer level keys, so it was always True
    and forced recursion to stop after the first pass. Now compares against
    recursive_summaries.values(), which also makes the "identical summary"
    warning accurate.

  • doc_reading_length: default to 0.0 (float) instead of 0 (int) when a
    doc has no doc_reading_time, otherwise wdocSummary construction crashes
    under beartype (WDOC_TYPECHECKING=crash).

Tests: the empty-content test now asserts the generation dump is included;
adds a recursion test proving summarization continues past the first pass
(fails as 2 != 3 with the old predicate).

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

tests/test_wdoc.py
wdoc/utils/tasks/summarize.py

  • [7eb0a61] by @thiswillbeyourgithub, 9 days ago:
    fix(summarize): retry then raise on empty LLM completion
    A reasoning model can return an empty completion (all of its budget spent
    on reasoning tokens) with finish_reason "stop". _summarize previously
    turned that into a silently empty summary, which only surfaced much later
    as a confusing assert 'monkey' in '' in
    test_summary_tim_urban_default_model.

Now each chunk generation retries once while bypassing the cache on an
empty completion, and raises an explicit ValueError if it is still empty.
Token usage from every attempt is accumulated so the cost accounting
stays correct.

Adds a basic regression test that drives _summarize with a fake LLM
returning empty content and asserts the explicit error is raised.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

tests/test_wdoc.py
wdoc/utils/tasks/summarize.py

  • [10c3b40] by @thiswillbeyourgithub, 9 days ago:
    docs: drop em-dashes introduced in the SKILL.md sync edits
    Replace the em-dashes I added (CLAUDE.md SKILL.md bullet, SKILL.md
    install comments) with colons, per the project's no-em-dash convention.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

CLAUDE.md
SKILL.md

  • [85289a9] by @thiswillbeyourgithub, 9 days ago:
    docs(skill): document the citation_url_template argument
    It was a real but undocumented CLI/constructor argument (turns page
    citations into clickable links in summaries). Add it to the Summary
    Arguments table and the Python constructor signature.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

SKILL.md

  • [8db96a1] by @thiswillbeyourgithub, 9 days ago:
    docs(skill): document youtube subtitle changes and kebab-case flags
    Reflect changes since the reference was written: --youtube_language now
    auto-detects the original-language (-orig) subtitle track when unset,
    --yt_* flags are rewritten to --youtube_*, and CLI flags accept
    kebab-case (dashes normalized to underscores).

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

SKILL.md

  • [65b619c] by @thiswillbeyourgithub, 9 days ago:
    docs(skill): document the zotero and karakeep filetypes
    Both were added since the reference was last written. Add them to the
    Recursive Filetypes table, document their loader-specific arguments and
    --path selector syntax in the DocDict section, and add shell examples.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

SKILL.md

  • [f1b2693] by @thiswillbeyourgithub, 9 days ago:
    docs(skill): bump reference to v5.1.0 and list modular install extras
    The reference was pinned to v5.0.0; the repo is now v5.1.0. Also expand
    the Installation section to document the modular extras introduced when
    install_requires was split (youtube/audio/anki/office/logseq/zotero/
    karakeep/...), since the base package only ships pdf + url loaders.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

SKILL.md

  • [36db392] by @thiswillbeyourgithub, 9 days ago:
    docs(claude): require keeping SKILL.md in sync with user-facing changes
    SKILL.md is a hand-maintained comprehensive reference that drifts unless
    updated by hand. Add it to the "Adding New Settings" and "Adding Support
    for a New Filetype" checklists and add a dedicated "Keeping SKILL.md in
    Sync" section so future changes update it.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

CLAUDE.md

  • [43e7e8b] by @thiswillbeyourgithub, 10 days ago:
    test(run_all_tests): validate MISTRAL_API_KEY before the suite runs
    test_mistral_embeddings hardcoded "mistral/mistral-embed", so the
    existing _check_provider_key "mistral/" MISTRAL_API_KEY guard in
    run_all_tests.sh never matched anything and the key was never checked
    up-front: a missing MISTRAL_API_KEY only surfaced late, mid-suite.

Make the embed model overridable via WDOC_TEST_MISTRAL_EMBED_MODEL
(default mistral/mistral-embed, full id incl. provider) and add it to
_ALL_TEST_MODELS so the guard now matches and crashes early.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

tests/run_all_tests.sh
tests/test_wdoc.py

  • [1d4ea7a] by @thiswillbeyourgithub, 10 days ago:
    test(karakeep): pad the roundtrip bookmark text past the reading-length guard
    A bare marker string is too short and trips wdoc's 'total reading length
    is suspiciously low' assertion (> 0.1 min). Repeat a deterministic
    sentence around the unique marker so the loaded doc clears the guard
    while the assertions still match on the marker.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

tests/test_karakeep.py

  • [14cb323] by @thiswillbeyourgithub, 10 days ago:
    Merge branch 'karakeep_source' into dev

  • [3ce4a93] by @thiswillbeyourgithub, 11 days ago:
    test(karakeep): make the api test self-contained via a bookmark lifecycle
    Instead of depending on a pre-existing library and a KARAKEEP_TEST_SELECTOR env
    var, the api test now creates its own temporary text bookmark (deterministic,
    unlike a link whose html is crawled asynchronously), loads it through wdoc's
    loader, asserts the round-trip, then deletes it (mirrors karakeep_python_api's
    own create/delete lifecycle tests). It also asserts the live bookmark carries
    the structural keys the FakeKarakeep fixtures encode, so the fast basic tests
    cannot silently drift from the real schema. Drop the now-unneeded
    KARAKEEP_TEST_SELECTOR guard from run_all_tests.sh.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

tests/run_all_tests.sh
tests/test_karakeep.py

  • [b3add40] by @thiswillbeyourgithub, 11 days ago:
    docs(karakeep): document the karakeep filetype
    Add the karakeep filetype, its --path selector syntax and karakeep_* args to
    help.md, usage examples to examples.md, and short mentions to README.md and
    ARCHITECTURE.md. Built with the help of Claude Code.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

ARCHITECTURE.md
README.md
wdoc/docs/examples.md
wdoc/docs/help.md

tests/run_all_tests.sh

  • [5de76cb] by @thiswillbeyourgithub, 11 days ago:
    test(karakeep): cover selector parsing, fan-out, resilience and caching
    A fake Karakeep client drives the basic tests (no network): selector parsing,
    link/text/asset fan-out, the native-source skip path, the loading_failure=warn
    resilience guarantee, and the doc_loaders_cache hit. Adds an api-gated real
    instance test driven by KARAKEEP_TEST_SELECTOR.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

tests/test_karakeep.py

  • [02df505] by @thiswillbeyourgithub, 11 days ago:
    feat(karakeep): add parse_karakeep recursive fan-out type
    Resolves a Karakeep selector to bookmarks, materializes each bookmark's stored
    artifact to a temp file via the cached resolver, and emits one local_html/txt/pdf
    DocDict per bookmark (loading_failure=warn so one bad bookmark does not abort the
    selection). Registered in recursive_types_func_mapping.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

wdoc/utils/load_recursive.py

  • [fa6ecbc] by @thiswillbeyourgithub, 11 days ago:
    feat(karakeep): add Karakeep API helper module
    Client construction (with private-mode guard), selector parsing, bookmark
    resolution across lists/tags/search/ids/library with cursor pagination, a
    metadata header builder, and a doc_loaders_cache-backed per-bookmark content
    resolver that prefers Karakeep's stored artifacts (html/text/pdf asset) and
    never re-fetches the live URL.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

wdoc/utils/loaders/karakeep.py

setup.py
wdoc/utils/misc.py

  • [d03299d] by @thiswillbeyourgithub, 11 days ago:
    Merge branch 'zotero_source' into dev

  • [3df03dc] by @thiswillbeyourgithub, 12 days ago:
    test(zotero): drive the api test from ZOTERO_COLLECTION_NAME
    Replace the bespoke ZOTERO_TEST_SELECTOR env var with ZOTERO_COLLECTION_NAME,
    which sits alongside the standard pyzotero ZOTERO_* credentials, so the api
    test setup is consistent with the other env-driven tests. The test now
    parametrizes over both attachment-text routes (fulltext and wdoc) against the
    named collection, and a new basic test asserts the fan-out marks every
    sub-document loading_failure='warn'. run_all_tests.sh guards on
    ZOTERO_COLLECTION_NAME accordingly.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

tests/run_all_tests.sh
tests/test_zotero.py

  • [6515753] by @thiswillbeyourgithub, 12 days ago:
    fix(zotero): make the fan-out resilient to per-item load failures
    A selection that spans a whole library inevitably contains items that fail
    wdoc's loaders: a sparse bibliographic entry whose metadata document is too
    short for the minimum reading-length check, a corrupt or DRM'd attachment, an
    unsupported attachment type, etc. Inherited from parse_doc, every emitted
    sub-document carried loading_failure='crash', so a single bad item aborted the
    entire selection.

Emit each zotero sub-document with loading_failure='warn' so a failure is
logged and skipped while the rest load; the batch still raises only if every
sub-document fails. Found while parsing a real 17-item collection where one
abstract-less book entry crashed the whole run.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

wdoc/utils/load_recursive.py

  • [1b1c113] by @thiswillbeyourgithub, 12 days ago:
    docs(zotero): trim the supported-types and README mentions
    Drop the parenthetical after zotero in ARCHITECTURE.md's supported-types
    line and shorten the README recursive-filetype entry to match the house
    style, deferring the selector syntax and options to help.md.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

ARCHITECTURE.md
README.md

  • [cd4e5bd] by @thiswillbeyourgithub, 12 days ago:
    test(zotero): exercise both attachment routes against a real library
    The api test now parametrizes over the fulltext and wdoc attachment-text
    routes so both the Zotero indexed-fulltext path and the download-then-parse
    path are covered, and asserts every emitted document carries a title.

run_all_tests.sh now actually runs test_zotero.py (basic + api) and crashes
early, mirroring the whisper key guard, when ZOTERO_API_KEY or
ZOTERO_TEST_SELECTOR are missing.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

tests/run_all_tests.sh
tests/test_zotero.py

  • [c711ada] by @thiswillbeyourgithub, 12 days ago:
    fix(loaders): make title a first-class load_one_doc arg
    Passing a title via a DocDict crashed any loader that does not declare a
    title parameter (txt, pdf, ...) with MissingDocdictArguments, because title
    was validated as a loader argument. Only load_url accepts title, so every
    other filetype was affected. The zotero fan-out hit this whenever it labelled
    its emitted txt/pdf sub-documents.

title is now an explicit parameter of load_one_doc: it is consumed for
document metadata and still forwarded to loaders that declare it (url), but no
longer rejected by loaders that do not. Adds a basic regression test that
parses a txt file with a title kwarg.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

tests/test_parsing.py
wdoc/utils/loaders/init.py

  • [f8c5575] by @thiswillbeyourgithub, 2 weeks ago:
    fix(zotero): pin pyzotero to a real version floor
    1.5.18 was not a real release; pin to >= 1.6.0 (well past local-API support;
    current pyzotero is 1.13.1).

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

setup.py

  • [f056008] by @thiswillbeyourgithub, 2 weeks ago:
    docs(zotero): document the zotero filetype
    Document the new zotero recursive filetype and its --path selector + zotero_*
    arguments in help.md, add CLI examples (collection/library/tag, fulltext
    backend, notes) to examples.md, and list it in the README filetypes and
    ARCHITECTURE.md. Implemented with Claude Code.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

ARCHITECTURE.md
README.md
wdoc/docs/examples.md
wdoc/docs/help.md

  • [34ebdfa] by @thiswillbeyourgithub, 2 weeks ago:
    test(zotero): cover selector parsing and fan-out with a fake client
    Add tests/test_zotero.py: basic tests for the --path selector parser, the
    recursive registration, DocDict acceptance of zotero_*/title keys, and the
    fan-out behaviour for each backend (wdoc pdf, fulltext, notes on/off, metadata
    off, tag selector) using a fake pyzotero client (no network). Plus an api-marked
    test gated on ZOTERO_TEST_SELECTOR for a real library.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

tests/test_zotero.py

  • [dee47a5] by @thiswillbeyourgithub, 2 weeks ago:
    feat(zotero): add parse_zotero recursive fan-out type
    Register a zotero recursive filetype that expands a Zotero selector (passed via
    --path: collection path, tag:, items:, search:, or library) into one DocDict per
    loadable sub-document: each attachment becomes a pdf/auto/url doc handed to
    wdoc's existing loaders (or a txt doc with Zotero's indexed fulltext when
    zotero_attachment_text requests it), plus an always-on per-item metadata/abstract
    doc and optional note docs. This reuses wdoc's loaders/caching with no
    duplicated extraction.

Also allow title as a DocDict key so per-document titles set by recursive
loaders reach load_one_doc's metadata enrichment (it already reads
kwargs["title"]).

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

wdoc/utils/load_recursive.py
wdoc/utils/misc.py

  • [8a74e36] by @thiswillbeyourgithub, 2 weeks ago:
    feat(zotero): add Zotero API helper module
    New wdoc/utils/loaders/zotero.py with the Zotero-facing helpers used by the
    upcoming parse_zotero recursive type: client connection (local API first, web
    API fallback, private-mode guard), selector parsing (collection path / tag /
    item keys / saved search / whole library), collection-tree resolution,
    attachment materialisation (download or linked file/url), indexed fulltext
    fetch, and bibliographic header / metadata-document formatting.

Attachments are handed to wdoc's own pdf/auto loaders rather than re-extracting
text, so no PDF parsing logic is duplicated.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

wdoc/utils/loaders/zotero.py

  • [38bb00f] by @thiswillbeyourgithub, 2 weeks ago:
    feat(zotero): add zotero extra and loader arg types
    Add the zotero optional dependency (pyzotero) to setup.py and include it
    in the full aggregate. Register the zotero_* loader arguments in
    filetype_arg_types so they flow into extra_args_types and DocDict.allowed_keys.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

setup.py
wdoc/utils/misc.py

README.md

Contributors

thiswillbeyourgithub
Assets 2
Loading