feat: optional multimodal ingest + visual retrieval (CLIP cross-modal + text-page coupling)#283
Conversation
Adds a tiny hook at the end of the document-loading pipeline that forwards
the original file to an HTTP webhook whenever text extraction produced
effectively-empty pages (e.g. scanned PDFs). The webhook is expected to
return `{"text": "...", "provider": "..."}`; its output replaces the
extraction and is tagged with `ocr_used=True` so downstream pipelines can
tell text-backed chunks from OCR-backed chunks.
Disabled by default — enables only when PRE_EXTRACTION_WEBHOOK_URL is set.
Configurable threshold (PRE_EXTRACTION_WEBHOOK_MIN_CHARS, default 100 chars
per page) and timeout (PRE_EXTRACTION_WEBHOOK_TIMEOUT, default 60 s).
Webhook failures never break ingest — on any error we fall back to the
original extraction with a warning log.
This keeps the core simple while letting external services participate in
ingest (OCR, translation, custom chunking, …) without subclassing loaders.
Tests: 6 new pytest cases covering the disabled path, the threshold
branches, HTTP failures, empty-response fallback, and the empty-documents
edge case.
Adds a feature-flagged visual pipeline alongside the existing text path:
when VISUAL_EMBED_URL points at a CLIP-style image-embed service, PDFs
are additionally rendered per-page via pdftoppm, each page PNG is sent
to the sidecar, and the resulting 768-dim vectors are persisted in a
new pgvector table (visual_chunks). The same sidecar's text endpoint
is used at query time so a text query can match against image vectors
in the shared cross-modal space.
/query gains an opt-in `include_visual: bool`. When true the response
is wrapped as `{"chunks": [...], "visual_matches": [{file_id,
page_number, image_path, score}]}`; default remains the legacy flat
list so existing callers are unaffected. Scores below a configurable
cosine threshold are dropped to keep cross-modal noise out.
Everything is soft-fail by design — missing pdftoppm, unreachable
sidecar or DB issues log a warning and continue, so the text ingest
path is never broken by the visual one.
Env vars (all empty / unset by default):
- VISUAL_EMBED_URL enables the pipeline (e.g. http://127.0.0.1:8002/embed/image)
- VISUAL_TEXT_EMBED_URL optional; defaults to the image URL with /embed/image → /embed/text
- VISUAL_PAGE_DPI=100 pdftoppm DPI (CLIP normalises anyway)
- VISUAL_STORAGE_ROOT=/var/rag-visual
- VISUAL_EMBED_TIMEOUT=30 HTTP timeout per page
- VISUAL_SCORE_THRESHOLD=0.25 cosine cutoff at retrieval time
- VISUAL_QUERY_TOP_K=3
Tests: 12 new pytest cases for the ingest helper (disabled-is-noop,
non-PDF-is-noop, pdftoppm-missing soft-fail, sidecar HTTP, DB upsert,
threshold filter, happy-path end-to-end, single-page failure keeps
going). Two new FastAPI cases for the /query response shape with
include_visual=True, including the empty-text-chunks case.
CLIP-style embeddings weight text-inside-images, so pages with matching
headlines can outrank pages with the actually-relevant photo. Fixture B
(IRZ_JB_2024.pdf) made this unworkable: page 75 (Workshop group photo
under "Länderberichte") was unreachable for any query phrasing because
page 76 shouted "Vietnam" in its headline.
Couple visual retrieval to the text pipeline's decisions: when /query
text-search hits pages {X, Y, Z}, attach those pages' visuals as
primary visual_matches regardless of CLIP score. CLIP remains as a
secondary signal for pages the text pipeline missed.
Config:
VISUAL_TEXT_COUPLED=true (default on)
VISUAL_TEXT_COUPLED_MAX_PAGES=4 (cap the primary signal)
Merge semantics: dedup by (file_id, page_number), primary wins, synthetic
score=1.0 on coupled hits to stay above VISUAL_SCORE_THRESHOLD filters.
Soft-fails everywhere — if the DB lookup breaks, we fall back to the
Phase-3 CLIP-only path.
Tests:
- unit: fetch_visual_chunks_for_pages (only-requested-pages, missing)
- integration: coupling wins, disabled falls back, max-pages cap
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The text-coupling code in _pages_by_file_from_text_docs originally only looked up 'page_number', but LangChain's PyPDFLoader (which rag_api uses for PDFs) stores it as 'page'. Prod smoke test against Fixture B (IRZ_JB_2024.pdf) showed all visual_matches came back with source='clip' and page 75 was missing — because text-coupling silently no-oped on every chunk. Fix: prefer 'page_number' (custom loaders), fall back to 'page' (PyPDFLoader). Unit test covers both keys, deduplication, and rank preservation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 829afb3656
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| visual = await _fetch_visual_matches_for_file_ids( | ||
| request, | ||
| body.query, | ||
| [body.file_id], | ||
| text_documents=authorized_documents, |
There was a problem hiding this comment.
When include_visual=true, this route calls visual retrieval even if authorized_documents is empty due to a user/file ownership mismatch. In that case _fetch_visual_matches_for_file_ids still runs CLIP search by file_id and can return image_path/page data for another user's file, creating a data-leak path that only requires knowing a valid file_id and having visual retrieval enabled.
Useful? React with 👍 / 👎.
| if page_number is None: | ||
| page_number = metadata.get("page") |
There was a problem hiding this comment.
The text-coupling fallback reads metadata['page'] directly, but PyPDFLoader emits zero-based page indices while visual ingest persists one-based page numbers (i + 1 in render output). Without normalizing here, text-coupled lookups query the wrong rows in visual_chunks, so primary visual matches for PDF text hits are missed or shifted by one page.
Useful? React with 👍 / 👎.
| except RuntimeError as exc: | ||
| logger.warning("visual ingest: pdftoppm step failed for %s: %s", file_id, exc) |
There was a problem hiding this comment.
This handler only catches RuntimeError, but render_pdf_pages can raise other OSError subclasses (for example PermissionError from out_dir.mkdir) before it wraps exceptions. Those errors will bubble out of maybe_embed_visuals and fail /embed requests, which breaks the intended soft-fail behavior whenever VISUAL_STORAGE_ROOT is misconfigured or not writable.
Useful? React with 👍 / 👎.
2 participants
Problem
Text-only RAG is blind to visual content: layout, color, typography, image placement, charts. For our use-case (feedback on direct-mail and print collateral) that's where the expertise lives — a vision-capable model asked "how is the layout on page 2 composed?" has nothing to work with.
Solution
Optional, env-gated parallel visual pipeline alongside the existing text path:
Three new env vars, all optional, all off by default:
VISUAL_EMBED_URLVISUAL_TEXT_COUPLEDtrueonceVISUAL_EMBED_URLis set: attach the visuals of pages that text-RAG retrieved, regardless of CLIP score. Bypasses CLIP's well-known weakness on conceptual queries.VISUAL_TEXT_COUPLED_MAX_PAGES/queryreturns the newvisual_matchesshape only when the client setsinclude_visual=true; legacy clients see the existing flat list.Why text-coupling matters (UAT receipt)
We tested with a 100-page mixed business PDF: a German query "first photo in the Vietnam section" matched the page with a big "Vietnam" headline (CLIP weighted text-in-image), not the page with the actual group photo (smaller mention). The actual group-photo page was not in CLIP's top-7 for ANY phrasing we tried. Text-RAG, by contrast, immediately retrieved the right pages from the section text.
So we merge the signals: text-RAG decides which pages are relevant, CLIP supplies additional candidates. Primary always wins on dedup.
What this PR does NOT do
nomic-ai/nomic-embed-vision-v1.5+nomic-embed-text-v1.5(shared 768-dim space → cross-modal). Anyone can plug in OpenCLIP, original CLIP, or any service exposing/embed/imageand/embed/text.Soft-fail policy
Every step soft-fails to text-only: missing renderer dependency, sidecar down, DB unreachable, malformed response, page-rendering failure for one of N pages. A broken visual pipeline must NEVER break the existing text RAG.
Tests
Includes 14+ pytest cases covering: text-coupling primary/secondary merge, dedup by
(file_id, page_number), max-pages cap, soft-fail paths,/queryresponse shape (legacy flat-list and new{chunks, visual_matches}), threshold filtering.Why upstream
CLIP-style visual retrieval is increasingly relevant for design-heavy / scanned-document workloads, but every install will use a different embedding model — hence the HTTP-sidecar split. The text-coupling refinement is independent of model choice and addresses a real CLIP failure mode we hit in production.
Stacks on top of #282 (the pre-extraction webhook PR) — that one merges first, this one would rebase to ~3 commits.
Co-authored-by: Claude (Anthropic)