tests: eliminate NLTK stopwords download race under pytest-xdist

[vishal-bala]

Motivation

TextQuery, HybridQuery, and AggregateHybridQuery default to stopwords="english" and lazily download the NLTK corpus the first time they need it. Under pytest-xdist, several workers can hit the missing corpus at the same moment and call nltk.download() concurrently — their unzips clobber each other in the shared nltk_data directory, and a worker then reads a half-written corpus. It surfaces only on Python 3.14 because there sentence-transformers is skipped, so nothing warms the corpus serially before the parallel tests start; on 3.10–3.13 the HuggingFace path fetches it first and the race never opens. It's intermittent (only some 3.14 jobs, across redis:8.2 and redis:latest alike), which is the giveaway that Redis has nothing to do with it.

Fix

Two complementary changes, following the pattern the unit tests already use (they pass stopwords=None to avoid NLTK entirely):

Pre-download the corpus once, before the workers race for it. A new session-scoped autouse fixture fetches stopwords a single time, serialized across xdist workers with a file lock, so exactly one process does the download and the rest find it already present. This covers every test that legitimately needs english stopwords — including the ones that assert on stopword filtering (test_empty_query_string, test_hybrid_query_stopwords) and the count-sensitive TextQuery tests in test_query.py, whose result counts depend on filtering and so must keep real stopwords. Adds filelock (already present transitively) as an explicit dev dependency.

Decouple the tests that don't actually need stopwords. The hybrid tests in test_aggregation.py match text with an optional ~@field clause, so the result set is driven by the vector and filters, not the text tokens — dropping stopwords there changes nothing about the assertions. Passing stopwords=None on those constructions removes NLTK from that file except the two tests that specifically exercise the feature, shrinking the surface that can ever race.

Verification

Ran the affected tests under -n 8 with the stopwords corpus deleted beforehand to simulate a cold runner. The fixture downloads once, no worker sees a partial corpus, and all tests pass — including the count-sensitive TextQuery tests, confirming english-stopword filtering is still intact.

Independent of the redis:latest WORKERS fix (#641); this addresses a separate, pre-existing flake.

🤖 Generated with Claude Code

---

Note

Low Risk
Test-only and dev-dependency changes; no production library behavior is modified.

Overview
Fixes intermittent CI failures when parallel pytest workers concurrently download the NLTK stopwords corpus into a shared nltk_data directory.

A new session-scoped autouse fixture ensure_nltk_stopwords in conftest.py ensures the corpus exists once before tests run; under pytest-xdist, a filelock serializes the download so only one worker fetches it. filelock is added as an explicit dev dependency in pyproject.toml / uv.lock.

Hybrid aggregation integration tests that do not assert on stopword behavior now pass stopwords=None on AggregateHybridQuery so those cases skip NLTK entirely; tests that exercise stopword filtering keep default or explicit stopword lists.

^{Reviewed by Cursor Bugbot for commit 17f5435. Bugbot is set up for automated code reviews on this repo. Configure here.}

[jit-ci]

🛡️ Jit Security Scan Results

✅ No security findings were detected in this PR

---

^{Security scan by Jit}

[nkanu17]

lgtm

[applied-ai-release-bot]

🚀 PR was released in v0.23.0 🚀