fix: replace NLTK with spaCy to remediate CVE-2025-14009

[lawrence-u10d]

Summary

• CVE-2025-14009 (CVSS 10.0) affects NLTK's downloader — zipfile.extractall() without path validation enables RCE via malicious packages. No patch exists.
• Replaces NLTK with spaCy (en_core_web_sm) across the entire codebase, eliminating the vulnerable downloader entirely since spaCy models install as pip packages.
• The only POS-tag consumer is contains_verb() which checks for VB* tags — both NLTK and spaCy use Penn Treebank verb tags, so behavior is preserved.

Changes

• unstructured/nlp/tokenize.py — rewritten to use spacy.load("en_core_web_sm")
• requirements/base.in + base.txt — swap nltk for spacy + en-core-web-sm
• Dockerfile + docker/rockylinux-9.2/Dockerfile — remove NLTK_DATA, add spacy download
• Makefile — rename install-nltk-models → install-spacy-model
• All CI workflows — remove NLTK_DATA env vars/cache paths, update make targets
• typings/nltk/ — deleted (6 stub files)
• test_unstructured/nlp/mock_nltk.py → mock_nlp.py

Test plan

• CI passes: test_unstructured/nlp/test_tokenize.py
• CI passes: test_unstructured/partition/test_text_type.py
• Docker image builds successfully
• Verify contains_verb behavior matches (Penn Treebank VB* tags)
• If any POS tag edge-case diffs surface in tests, update expected values (behavioral parity, not a bug)

🤖 Generated with Claude Code

---

Note

Medium Risk
Touches core NLP tokenization and POS-tagging behavior used in text classification, so small output differences can cascade into partitioning/fixture changes. Dependency/model loading changes may also impact environments that don’t install en_core_web_sm correctly.

Overview
Replaces NLTK-based tokenization/POS tagging with spaCy (en_core_web_sm) to remove reliance on NLTK’s downloader and mitigate CVE-2025-14009.

This updates runtime/CI/Docker install flows to stop downloading NLTK data (NLTK_DATA, make install-nltk-models) and instead ship the spaCy model as a dependency (via pyproject.toml + uv source URL), removes NLTK typing stubs, bumps to 0.20.9, and refreshes tests/expected fixtures to match the new tokenization/classification outputs.

^{Written by Cursor Bugbot for commit b753ddc. This will update automatically on new commits. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.}

[cursor]

Uncached _process causes triple spaCy pipeline execution per text

Medium Severity

The _process helper runs the full spaCy pipeline (tokenizer, tagger, parser, NER) but is not cached, while word_tokenize, pos_tag, and _tokenize_for_cache each have their own lru_cache and call _process independently. In the hot path (is_possible_narrative_text), all three are called for the same text, so the full spaCy pipeline runs three times per unique text element. NLTK's equivalent calls were orders of magnitude lighter. Adding @lru_cache to _process (safe since callers only read the Doc) would eliminate the redundant pipeline runs.

Additional Locations (1)

• unstructured/nlp/tokenize.py#L30-L41

 

[cursor]

Non-PyPI dependency breaks pip installation of published package

High Severity

en-core-web-sm is listed in [project.dependencies] but is not published on PyPI — it's only available from GitHub releases. The [tool.uv.sources] URL override works for local uv sync, but is not propagated to the built wheel's metadata. When the package is published to PyPI (via twine), pip install unstructured will fail because pip cannot resolve en-core-web-sm>=3.8.0 from any index. This is a regression from nltk, which was fully PyPI-resolvable.

Additional Locations (1)

• pyproject.toml#L183-L185

 

[badGarnet]

I would rather try to get nltk/nltk#3502 in (beg the maintainer 😅 )
spacy adds a lot of complexity into the code and it is a relatively massive package