Drop nltk (#645)

# Drop NLTK dependency; serve stopwords statically

## Why

RedisVL only used NLTK for one thing: default stopword lists in
full-text
query building. Pulling in the whole `nltk` package for that has cost us
repeatedly — packaging/support friction and a recurring source of test
flakiness (parallel pytest-xdist workers racing on `nltk.download()`,
most
recently patched in #644).

The stopword lists are small, static, and change rarely. This PR bundles
them
directly and removes `nltk` entirely.

## What changed

- **New static data** — `redisvl/utils/stopwords.json` (33 languages,
~11k
words, ~120KB), extracted once from NLTK's `stopwords` corpus so the
lists
are byte-identical to what NLTK returned. **No behavior change** for any
  supported language.
- **New loader** — `redisvl/utils/stopwords.py` exposes
`get_stopwords(language)`,
  which lazily reads and caches the JSON via `importlib.resources`.
- **Rewired consumers** — `full_text_query_helper.py` and `query.py` now
call
`get_stopwords()` instead of lazily importing NLTK and downloading the
corpus.
  The download/race/retry blocks are gone.
- **Removed vestigial imports** — `aggregate.py` had unused `nltk`
lazy-imports
  (it already routes through the helper); deleted.
- **Dropped the dependency** — removed the `nltk` extra and its entry in
the
  `all` extra from `pyproject.toml`.
- **Simplified tests** — deleted the `ensure_nltk_stopwords` session
fixture
from `tests/conftest.py`. The xdist download race it guarded against no
  longer exists: there is no download.
- **Spellcheck** — added `stopwords.json` to codespell's `--skip` list
(it's a
  multi-language word-list data file, not prose).

## Backward compatibility

The public API is unchanged. All existing usages still work:

- `stopwords="english"` (default), `"german"`, `"french"`, etc. — any of
the
  33 bundled languages
- `stopwords={"custom", "set"}` / list / tuple
- `stopwords=None` (no filtering)

Error behavior is preserved: an unknown language string raises
`ValueError`
(now with the list of available languages), and a
non-string/non-collection
raises `TypeError`.

The only user-visible change: `pip install redisvl[nltk]` no longer
resolves
(the extra was removed). Worth a changelog note.

## Verification

- All stopword / query / aggregation / hybrid unit tests pass (945
passed);
`german` still yields `der`/`die`/`und`, invalid language →
`ValueError`,
  invalid type → `TypeError`.
- Confirmed `stopwords.json` ships in the built wheel
  (`redisvl/utils/stopwords.json` present in `redisvl-*.whl`).
- pre-commit (format, isort, mypy, codespell) passes.

## Downstream follow-up (separate, not in this PR)

`redis-ai-resources` has two notebooks
(`python-recipes/vector-search/01_redisvl.ipynb`,
`02_hybrid_search.ipynb`) that carry `nltk` in their `%pip install`
lines and
one stale markdown reference. Nothing breaks (they never `import nltk`),
but
the `nltk` install is now dead weight. That cleanup should land
**after** this
release and bump the notebooks' `redisvl` floor to the version that
bundles
stopwords.

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> Touches default full-text tokenization for all string `stopwords`
language codes; lists are meant to match NLTK but any drift or packaging
omission of `stopwords.json` would change search behavior. Removing the
`nltk` extra is a breaking install change for users who relied on it.
> 
> **Overview**
> **Removes the `nltk` optional dependency** and ships default full-text
stopword lists as bundled `redisvl/utils/stopwords.json` (~33
languages), loaded via new `get_stopwords()` in
`redisvl/utils/stopwords.py`.
> 
> **Full-text query paths** (`query.py`, `full_text_query_helper.py`)
now call `get_stopwords()` instead of lazy NLTK imports and runtime
`nltk.download()`; unused NLTK imports in `aggregate.py` are removed.
> 
> **Packaging and CI:** `nltk` is dropped from `pyproject.toml`
extras/`all` and `uv.lock`; codespell skips `stopwords.json`. The pytest
session fixture that pre-downloaded NLTK stopwords for xdist is deleted
from `tests/conftest.py`.
> 
> Public stopword API behavior for supported languages and custom sets
is intended to stay the same; `pip install redisvl[nltk]` no longer
applies.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
23a8c335ea20a879319cdccababa91b0cabe5be6. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

---------

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Author: rbs333

.pre-commit-config.yaml

@@ -13,5 +13,5 @@ repos:
         name: Check spelling
         args:
           - --write-changes
-          - --skip=*.pyc,*.pyo,*.lock,*.git,*.mypy_cache,__pycache__,*.egg-info,.pytest_cache,docs/_build,env,venv,.venv
+          - --skip=*.pyc,*.pyo,*.lock,*.git,*.mypy_cache,__pycache__,*.egg-info,.pytest_cache,docs/_build,env,venv,.venv,redisvl/utils/stopwords.json
           - --ignore-words-list=enginee

pyproject.toml

@@ -40,7 +40,6 @@ mcp = [
 ]
 mistralai = ["mistralai>=1.0.0,<2"]
 openai = ["openai>=1.1.0"]
-nltk = ["nltk>=3.8.1,<4"]
 cohere = ["cohere>=4.44"]
 voyageai = ["voyageai>=0.2.2"]
 sentence-transformers = ["sentence-transformers>=3.4.0,<4"]
@@ -62,7 +61,6 @@ sql-redis = [
 all = [
     "mistralai>=1.0.0,<2",
     "openai>=1.1.0",
-    "nltk>=3.8.1,<4",
     "cohere>=4.44",
     "voyageai>=0.2.2",
     "sentence-transformers>=3.4.0,<4",

redisvl/query/aggregate.py

@@ -8,10 +8,6 @@
 from redisvl.redis.utils import array_to_buffer
 from redisvl.schema.fields import VectorDataType
 from redisvl.utils.full_text_query_helper import FullTextQueryHelper
-from redisvl.utils.utils import lazy_import
-
-nltk = lazy_import("nltk")
-nltk_stopwords = lazy_import("nltk.corpus.stopwords")
 
 
 class Vector(BaseModel):

redisvl/query/query.py

@@ -6,14 +6,12 @@
 from redisvl.query.filter import FilterExpression
 from redisvl.redis.utils import array_to_buffer
 from redisvl.utils.log import get_logger
+from redisvl.utils.stopwords import get_stopwords
 from redisvl.utils.token_escaper import TokenEscaper
-from redisvl.utils.utils import denorm_cosine_distance, lazy_import
+from redisvl.utils.utils import denorm_cosine_distance
 
 logger = get_logger(__name__)
 
-nltk = lazy_import("nltk")
-nltk_stopwords = lazy_import("nltk.corpus.stopwords")
-
 # Type alias for sort specification
 # Can be:
 # - str: single field name (ASC by default)
@@ -1406,21 +1404,7 @@ def _set_stopwords(self, stopwords: str | set[str] | None = "english"):
         if not stopwords:
             self._stopwords = set()
         elif isinstance(stopwords, str):
-            try:
-                # Try loading first; only download if not already present.
-                # This avoids race conditions when parallel workers (e.g.
-                # pytest-xdist) call nltk.download() concurrently.
-                try:
-                    self._stopwords = set(nltk_stopwords.words(stopwords))
-                except LookupError:
-                    nltk.download("stopwords", quiet=True)
-                    self._stopwords = set(nltk_stopwords.words(stopwords))
-            except ImportError:
-                raise ValueError(
-                    f"Loading stopwords for {stopwords} failed: nltk is not installed."
-                )
-            except Exception as e:
-                raise ValueError(f"Error trying to load {stopwords} from nltk. {e}")
+            self._stopwords = get_stopwords(stopwords)
         elif isinstance(stopwords, (set, list, tuple)) and all(
             isinstance(word, str) for word in stopwords
         ):

redisvl/utils/full_text_query_helper.py

@@ -1,9 +1,6 @@
 from redisvl.query.filter import FilterExpression
+from redisvl.utils.stopwords import get_stopwords
 from redisvl.utils.token_escaper import TokenEscaper
-from redisvl.utils.utils import lazy_import
-
-nltk = lazy_import("nltk")
-nltk_stopwords = lazy_import("nltk.corpus.stopwords")
 
 
 def _parse_text_weights(weights: dict[str, float] | None) -> dict[str, float]:
@@ -88,21 +85,7 @@ def _get_stopwords(self, stopwords: str | set[str] | None = "english") -> set[st
         if not stopwords:
             return set()
         elif isinstance(stopwords, str):
-            try:
-                # Try loading first; only download if not already present.
-                # This avoids race conditions when parallel workers (e.g.
-                # pytest-xdist) call nltk.download() concurrently.
-                try:
-                    return set(nltk_stopwords.words(stopwords))
-                except LookupError:
-                    nltk.download("stopwords", quiet=True)
-                    return set(nltk_stopwords.words(stopwords))
-            except ImportError:
-                raise ValueError(
-                    f"Loading stopwords for {stopwords} failed: nltk is not installed."
-                )
-            except Exception as e:
-                raise ValueError(f"Error trying to load {stopwords} from nltk. {e}")
+            return get_stopwords(stopwords)
         elif isinstance(stopwords, (set, list, tuple)) and all(
             isinstance(word, str) for word in stopwords
         ):