feat(zotero): add Zotero connector

[thiswillbeyourgithub]

Summary

Adds a Zotero connector to oikb. It syncs the text of the PDF attachments of items in a Zotero collection (and its subcollections) into an Open WebUI Knowledge Base, mapping the Zotero collection hierarchy onto KB directories. It is read-only with respect to Zotero: it never modifies, deletes, or adds anything to the library.

This is a Claude Code port of an earlier standalone tool I wrote, https://github.com/thiswillbeyourgithub/openwebui-knowledge-zotero-sync, reworked to fit oikb's connector interface.

Zotero connector

• New zotero:<hierarchy> source scheme. %% separates collection names
  (e.g. zotero:Research%%Machine Learning); a bare zotero: syncs every top-level collection.
• Text extraction prefers Zotero's indexed fulltext API, falling back to PyMuPDF on the downloaded PDF.
• Collection hierarchy maps to KB directories; an item that lives in several subcollections is handled.
• Change detection via ZOTERO_CHECKSUM: version (cheap, hashes the Zotero item version, no download, default) or content (hashes the extracted text).
• ZOTERO_EXCLUDE skips subcollections, relative to the synced root.
• Auth/options via env: ZOTERO_LIBRARY_ID, ZOTERO_API_KEY, ZOTERO_LIBRARY_TYPE, ZOTERO_CHECKSUM, ZOTERO_EXCLUDE.
• Optional dependency group: pip install oikb[zotero] (pyzotero + pymupdf).
• README and docs/guide.md updated.

Sync change: unavailable source files are warnings, not fatal errors

A file the source advertises in its manifest but cannot provide content for (e.g. a Zotero attachment whose bytes aren't in storage: a web link, a linked file, or a WebDAV-only item) is a source-side data gap, not an oikb failure. Previously any such case landed in result.errors and exited the whole sync with code 1.

This PR adds a SourceFileUnavailable exception to the connector base. run_sync catches it specifically (no retry) and routes the file to a new result.warnings
list instead of result.errors, so a sync whose only problems are unavailable source files still succeeds (exit 0). Any other read_file() exception still fails the run exactly as before. The CLI and daemon surface warnings separately from errors; only result.errors affects the exit code / success status.

Tests

tests/test_sync_warnings.py covers all three paths: the Zotero missing-file mapping to SourceFileUnavailable, warning routing in run_sync, and that a generic read failure is still an error.

Notes

• Opening as a draft for early feedback on the connector shape and on whether the warnings-vs-errors sync change is acceptable as written.
• Implemented with Claude Code.
• I initially wanted to add support for using only the yaml instead of env variables to simplify setup but it resuired some fundamental changes I felt could go beyond the scope of a single PR. Let me know if you'd like me to streamline this.

[psuckrow-routine]

Nice work on this! One thing that'd make it even more useful: an option to also pull in notes and highlights/annotations, not just the PDF body text. A lot of the value in Zotero libraries lives in child notes and PDF annotations, and right now those get skipped since _collect_items only keeps itemType == "attachment" children.

Maybe something like a ZOTERO_INCLUDE_NOTES / ZOTERO_INCLUDE_ANNOTATIONS flag that appends note content and annotation text/comments to (or alongside) the extracted PDF text?

[psuckrow-routine]

Hey, got the connector working with a Docker setup. Works great for items in collections but I noticed items sitting directly in "My Library" (unfiled, not in any collection) never get picked up regardless of whether you use zotero: bare or a named hierarchy.

Makes sense given build_manifest() walks the collection tree, but would be nice to also sweep root library items when no hierarchy is specified. Could land them in a virtual _unfiled/ directory to keep things tidy. Or maybe a zotero:* source mode that opts into full-library sync.

[thiswillbeyourgithub]

Hi @psuckrow-routine , thanks for giving it a try and thanks for the nice words.

I pushes a few things. Can you try and see if it solves the issues you raised?

[psuckrow-routine]

Hey, thanks so much for the work. Yes I will try it out and let you know !

[psuckrow-routine]

When oikb sync fails to upload a file, the error is silently swallowed — the CLI exits with code 1 and the summary count is off, but neither the default nor verbose (-v) output reveals which file failed or why.

Steps to reproduce:

1. Run oikb sync with a Zotero source
2. One file fails to upload (transient network error / API timeout)
3. Output shows e.g. Done: 4 added when 5 were expected — exit code 1

Running -v produces identical output to the default. --dry-run correctly lists the stuck file as pending, but re-running sync still adds 0 with no error shown.

Expected: per-file error messages — when a file fails, show the filename and the actual error (HTTP status / exception), e.g.:

✗ _unfiled/Large Language Models: A Survey.txt — 500 Internal Server Error

Actual:

Done: 4 added   ← no indication of which file or why
exit code 1

[psuckrow-routine]

WebDAV storage support

When Zotero is configured to use WebDAV storage instead of Zotero's native cloud storage, oikb sync fails with a 404 on the file endpoint:

Zotero has no downloadable file for attachment 7AHE5BCC (
Code: 404
URL: https://api.zotero.org/users/.../items/7AHE5BCC/file

This is expected — Zotero's API doesn't proxy WebDAV files. But oikb currently has no way to reach the WebDAV server directly.

Feature request: add ZOTERO_WEBDAV_URL + ZOTERO_WEBDAV_USER / ZOTERO_WEBDAV_PASSWORD env vars so oikb can fetch files from WebDAV storage when the Zotero API file endpoint returns 404. The WebDAV setting should only override how the file content is fetched — annotations, notes, and metadata should continue to be retrieved via the Zotero API as normal (since those live in Zotero's own database regardless of storage backend).

In the meantime, notes and annotations (ZOTERO_INCLUDE_NOTES / ZOTERO_INCLUDE_ANNOTATIONS) still sync correctly for WebDAV items — only the full PDF text extraction is blocked.

[psuckrow-routine]

Daemon interval sync not firing

When running oikb daemon, the scheduled interval sync never fires automatically. The daemon starts and reports the correct number of configured sources, but no syncs run until triggered manually via oikb sync.

Tested with interval: 1min in .oikb.yaml — after waiting several minutes with the daemon running, GET /history returns {"entries":[]}. Only a manual oikb sync call produces a history entry.

Expected: the daemon should run the first sync shortly after startup and then repeat on the configured interval.