Fix FTS arm recall for natural-language queries (OR lexemes, not AND)

[JSv4]

Why

CoreAnnotationVectorStore already defaults to hybrid retrieval (pgvector + Postgres full-text, fused with RRF), and its _fuse_results already returns whichever arm is non-empty — so the intended "embeddings improve results, FTS still answers when they're missing" behavior is the design. But it didn't actually degrade gracefully, because the FTS arm could not match real questions.

_run_fts_query built its query with SearchQuery(query_text, config=...), whose default plainto_tsquery ANDs every lexeme. A natural-language question references multiple concepts at once, and no single annotation contains all of them — so the FTS arm returned zero rows for any realistic query. When a corpus has no embeddings, hybrid degraded to that empty FTS arm and the agent answered "I found nothing."

Measured on a 53-document SEC-filing corpus:

Query passed to the FTS arm	Matches
"What are the main risks or obligations across these documents?" (plain AND — what the agent sends)	0
same, search_type="websearch"	0
OR of content lexemes (this fix)	8

What

_run_fts_query now tokenizes the query and OR-combines per-token plainto_tsquery sub-queries:

• Any term match qualifies (recall), while SearchRank still ranks annotations matching more terms higher (precision preference preserved), and RRF + the optional reranker clean up ordering downstream.
• Per-token plainto_tsquery safely elides stopwords without raising a tsquery syntax error, so no manual stopword list or raw-tsquery escaping (and no injection surface) is needed.
• It's the single FTS entry point for the hybrid, fts-only, and global_search paths, so the recall fix applies everywhere at once.

This does not make a zero-embedding corpus as good as an embedded one (FTS can't do synonyms — "liabilities" won't hit "obligations"), but it turns "I found nothing" into "here are the keyword matches" — the graceful degradation hybrid was supposed to provide.

Tests

• New regression test test_hybrid_search.py::TestHybridSearch::test_fts_matches_natural_language_query_spanning_annotations — a natural-language query whose three concepts are spread across three annotations. RED before the change (0 matches, AND semantics); GREEN after (all three, OR semantics). Written test-first.
• test_hybrid_search.py — 37 passed.
• Related suites exercising the shared FTS arm (test_django_annotation_vector_store, test_corpus_isolation_vector_store, test_subtree_group_vector_search, test_version_aware_vector_store, test_conversation_search) — 122 passed.
• pre-commit (black/isort/flake8/mypy + changelog validation) clean.

Notes

Independent of the Corpus Intelligence home PR (#1969) — that PR surfaces the cross-document "ask" UI; this fixes the retrieval recall behind it. Found while verifying #1969 against a freshly-embedded corpus.

[claude]

Code Review

Overview

This PR fixes a real, measurable regression: the FTS arm of hybrid search was silently returning zero results for any natural-language query because plainto_tsquery (Django's default SearchQuery) ANDs all lexemes — no single annotation can contain all terms in a question like "What are the main risks or obligations across these documents?". The fix OR-combines per-token plainto_tsquery sub-queries, trading off a small amount of precision for dramatically better recall, which is the correct trade for a first-pass retrieval arm that feeds into RRF + optional reranking downstream.

The motivation is clear, the implementation is minimal, and the regression test was written first. This is a well-executed targeted fix.

---

Code quality

core_vector_stores.py — the fix itself

The implementation is clean. The else branch (empty/non-word query_text) degrades to the prior behavior rather than crashing — good defensive guard. The comment correctly calls out the non-obvious constraint (AND semantics ⇒ zero recall) and is appropriate per project conventions. operator.or_ on two SearchQuery instances emits tsquery_a | tsquery_b, which PostgreSQL handles natively.

One substantive concern — token count with long inputs

re.findall(r"\w+", query_text) creates one SearchQuery per word. Agent queries are typically one or two sentences (safe), but _run_fts_query is also the global_search entry point. If a caller passes a multi-paragraph document excerpt as query_text, reduce will build a very deep OR chain — O(N) SearchQuery objects and an equally wide tsquery expression sent to PostgreSQL. While Postgres can handle this, there is no guard against it and the cost grows linearly with input length.

Suggested fix — cap tokens before reducing:

MAX_FTS_TOKENS = 30  # or a constant in opencontractserver/constants/search.py
tokens = re.findall(r"\w+", query_text)[:MAX_FTS_TOKENS]

This is non-breaking (callers already accept approximate recall) and prevents a latent slowdown on long inputs.

Minor — duplicate tokens

If the query repeats a word ("obligations and obligations"), two identical SearchQuery objects are OR'd. Harmless (Postgres deduplicates), but list(dict.fromkeys(tokens)) before the reduce would make the generated SQL cleaner.

---

Test coverage

The new test test_fts_matches_natural_language_query_spanning_annotations is the right test to write — three semantically distinct annotations, one cross-cutting question, isolated to mode="fts" so no embedding mock is needed. The assertEqual on the full result set (not just "results > 0") is a good choice: it would also catch the reverse regression of over-matching unrelated annotations.

A few edge cases not currently covered that would be nice to add in a follow-up:

1. All-stopword query (query_text="the and or"): every per-token plainto_tsquery returns an empty tsquery. OR-combining empty tsqueries may produce unexpected SQL depending on Django/PostgreSQL version. Worth a quick test to confirm empty-result behavior rather than a crash.

2. Single-content-word query (query_text="indemnification"): the tokenization path should behave identically to the old single-SearchQuery path for a single content word. This implicitly validates the refactor doesn't regress single-term behavior.

---

Security

No new injection surface — plainto_tsquery escapes its input, and the per-token path is exactly as safe as the original single-call path. The PR correctly notes this in the comment.

---

Summary

Correctness	Correct. Fixes the reported zero-recall issue.
Risk	Low. Narrow change to a single private static method; else branch preserves prior behavior for the empty-input edge case.
Performance	Acceptable for typical agent queries. A token cap ([:MAX_FTS_TOKENS]) is worth adding to protect the global_search path from long inputs.
Tests	Good regression coverage for the reported bug. All-stopword and single-word edge cases would improve confidence.
Conventions	Changelog fragment present and well-written. Imports clean.

The token cap is the only change I'd call out before merge; the dedup and extra test cases are nice-to-haves for a follow-up.