Merge pull request #1309 from PyThaiNLP/copilot/remove-auto-download-corpus

Add `PYTHAINLP_OFFLINE` support, `is_offline_mode()`, and `PYTHAINLP_DATA` env var to corpus path handling

Author: bact

CHANGELOG.md

@@ -40,6 +40,16 @@ See PR for prompt and details.
 - Add BLEU, ROUGE, WER, and CER metrics to pythainlp.benchmarks #1295
 - Add Attaparse engine to dependency parser
   (`dependency_parsing`, engine="attaparse") #1303
+- `get_corpus_path()` now respects `PYTHAINLP_OFFLINE` env var (same semantics
+  as `HF_HUB_OFFLINE`): when set, raises `FileNotFoundError` if the corpus is
+  not already cached locally; when unset, auto-downloads as before #1306
+- Added `pythainlp.is_offline_mode()` helper function (mirrors
+  `huggingface_hub.is_offline_mode()`) #1306
+- `PYTHAINLP_DATA` is now the preferred env var for the data directory
+  (same pattern as `NLTK_DATA`); `PYTHAINLP_DATA_DIR` is deprecated
+  and will emit a `DeprecationWarning` #1306
+- Callers raise `FileNotFoundError` with download instructions when a corpus
+  path cannot be resolved (e.g. download failed) #1306
 - Improved documentation; code cleanup; more tests
 
 ## Version 5.1.2 -> 5.2.0

README.md

@@ -77,14 +77,27 @@ please inspect the `[project.optional-dependencies]` section of
 PyThaiNLP downloads data (see the data catalog `db.json` at
 [pythainlp-corpus](https://github.com/PyThaiNLP/pythainlp-corpus))
 to `~/pythainlp-data` by default.
-Set the `PYTHAINLP_DATA_DIR` environment variable to override this location.
+Set the `PYTHAINLP_DATA` environment variable to override this location.
+(`PYTHAINLP_DATA_DIR` is still accepted but deprecated.)
 
 When using PyThaiNLP in distributed computing environments
-(e.g., Apache Spark), set the `PYTHAINLP_DATA_DIR` environment variable
+(e.g., Apache Spark), set the `PYTHAINLP_DATA` environment variable
 inside the function that will be distributed to worker nodes.
 See details in
 [the documentation](https://pythainlp.org/dev-docs/notes/installation.html).
 
+### Offline mode
+
+Set `PYTHAINLP_OFFLINE=1` to disable automatic corpus downloads.
+When this variable is set and a corpus is not already cached locally,
+a `FileNotFoundError` is raised instead of attempting a network download.
+Use `pythainlp.is_offline_mode()` to check the current state programmatically.
+
+```python
+import pythainlp
+print(pythainlp.is_offline_mode())  # True if PYTHAINLP_OFFLINE=1
+```
+
 ## Testing
 
 We test core functionalities on all officially supported Python versions.

README_TH.md

@@ -109,10 +109,11 @@ pip install "pythainlp[extra1,extra2,...]"
 PyThaiNLP ดาวน์โหลดข้อมูล (ดูแค็ตตาล็อกข้อมูล `db.json` ที่
 [pythainlp-corpus](https://github.com/PyThaiNLP/pythainlp-corpus))
 ไปที่ `~/pythainlp-data` ตามค่าเริ่มต้น
-ตั้งค่า environment variable `PYTHAINLP_DATA_DIR` เพื่อเปลี่ยนตำแหน่งนี้
+ตั้งค่า environment variable `PYTHAINLP_DATA` เพื่อเปลี่ยนตำแหน่งนี้
+(`PYTHAINLP_DATA_DIR` ยังคงใช้ได้แต่เลิกใช้แล้ว)
 
 เมื่อใช้ PyThaiNLP ในสภาพแวดล้อมการคำนวณแบบกระจาย
-(เช่น Apache Spark) ให้ตั้งค่า environment variable `PYTHAINLP_DATA_DIR`
+(เช่น Apache Spark) ให้ตั้งค่า environment variable `PYTHAINLP_DATA`
 ภายในฟังก์ชันที่จะถูกกระจายไปยัง worker nodes
 ดูรายละเอียดใน[เอกสาร](https://pythainlp.org/dev-docs/notes/installation.html)
 

docs/notes/installation.rst

@@ -89,7 +89,8 @@ Key considerations
 
 2. **Use a writable local directory**: The default data directory (``~/pythainlp-data``) may not be writable on executor nodes. Use a local directory like ``./pythainlp-data`` instead.
 
-3. **Set ``PYTHAINLP_DATA_DIR`` before data access**: Always set the ``PYTHAINLP_DATA_DIR`` environment variable before the first call that reads or writes PyThaiNLP data on each worker.
+3. **Set ``PYTHAINLP_DATA`` before data access**: Always set the ``PYTHAINLP_DATA`` environment variable before the first call that reads or writes PyThaiNLP data on each worker.
+   (``PYTHAINLP_DATA_DIR`` is also accepted for backward compatibility but is deprecated.)
 
 Example usage with Apache Spark
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -104,7 +105,7 @@ Basic example using PySpark RDD::
 
     def tokenize_thai(text):
         import os
-        os.environ['PYTHAINLP_DATA_DIR'] = './pythainlp-data'
+        os.environ['PYTHAINLP_DATA'] = './pythainlp-data'
         from pythainlp.tokenize import word_tokenize
         return word_tokenize(text)
 
@@ -123,7 +124,7 @@ Example using PySpark DataFrame API::
     @udf(returnType=ArrayType(StringType()))
     def tokenize_udf(text):
         import os
-        os.environ['PYTHAINLP_DATA_DIR'] = './pythainlp-data'
+        os.environ['PYTHAINLP_DATA'] = './pythainlp-data'
         from pythainlp.tokenize import word_tokenize
         return word_tokenize(text)
 
@@ -141,13 +142,30 @@ Note that while the code itself is thread-safe, you still need to configure the
 Runtime configurations
 ----------------------
 
-.. envvar:: PYTHAINLP_DATA_DIR
+.. envvar:: PYTHAINLP_DATA
 
    Specifies the location where downloaded data and the corpus database are stored. If the directory does not exist, PyThaiNLP will create it.
 
    By default this is a directory named ``pythainlp-data`` in the user's home directory.
 
-   Run ``thainlp data path`` at the command line to display the current `PYTHAINLP_DATA_DIR`.
+   Run ``thainlp data path`` at the command line to display the current data directory.
+
+.. envvar:: PYTHAINLP_DATA_DIR
+
+   .. deprecated::
+      Use :envvar:`PYTHAINLP_DATA` instead. Setting ``PYTHAINLP_DATA_DIR`` triggers a
+      :class:`DeprecationWarning` at runtime. If both ``PYTHAINLP_DATA`` and ``PYTHAINLP_DATA_DIR``
+      are set simultaneously, PyThaiNLP raises :exc:`ValueError`.
+
+.. envvar:: PYTHAINLP_OFFLINE
+
+   When set to a truthy value (``1``, ``true``, ``yes``, ``on``), PyThaiNLP operates in
+   *offline mode*: corpus downloads are disabled, and :func:`pythainlp.corpus.get_corpus_path`
+   raises :exc:`FileNotFoundError` for any corpus that is not already cached locally.
+
+   Use :func:`pythainlp.is_offline_mode` to check the current state programmatically.
+
+   This follows the same convention as ``HF_HUB_OFFLINE`` in `huggingface_hub`.
 
 .. envvar:: PYTHAINLP_READ_MODE
 
@@ -158,11 +176,11 @@ Installation FAQ
 
 Q: How do I set environment variables on each executor node in a distributed environment?
 
-A: When using PyThaiNLP in distributed computing environments like Apache Spark, you need to set the ``PYTHAINLP_DATA_DIR`` environment variable inside the function that will be distributed to executor nodes. For example::
+A: When using PyThaiNLP in distributed computing environments like Apache Spark, you need to set the ``PYTHAINLP_DATA`` environment variable inside the function that will be distributed to executor nodes. For example::
 
     def tokenize_thai(text):
         import os
-        os.environ['PYTHAINLP_DATA_DIR'] = './pythainlp-data'
+        os.environ['PYTHAINLP_DATA'] = './pythainlp-data'
         from pythainlp.tokenize import word_tokenize
         return word_tokenize(text)
 

pythainlp/__init__.py

@@ -55,6 +55,7 @@
 __all__: list[str] = [
     "collate",
     "correct",
+    "is_offline_mode",
     "pos_tag",
     "romanize",
     "spell",
@@ -76,5 +77,6 @@
     subword_tokenize,
     word_tokenize,
 )
+from pythainlp.tools.path import is_offline_mode
 from pythainlp.transliterate import romanize, transliterate
 from pythainlp.util import collate, thai_strftime

pythainlp/augment/word2vec/ltw2v.py

@@ -32,10 +32,12 @@ def tokenizer(self, text: str) -> list[str]:
 
     def load_w2v(self) -> None:  # insert substitute
         """Load LTW2V's word2vec model"""
-        if self.ltw2v_wv is None:
-            raise ValueError(
-                "LTW2V word2vec model not found. "
-                "Please download it first using pythainlp.corpus.download('ltw2v_wv')"
+        if not self.ltw2v_wv:
+            raise FileNotFoundError(
+                "corpus-not-found name='ltw2v_wv'\n"
+                "  Corpus 'ltw2v_wv' not found.\n"
+                "    Python: pythainlp.corpus.download('ltw2v_wv')\n"
+                "    CLI:    thainlp data get ltw2v_wv"
             )
         self.aug: Word2VecAug = Word2VecAug(
             self.ltw2v_wv, self.tokenizer, type="binary"

pythainlp/augment/word2vec/thai2fit.py

@@ -33,10 +33,12 @@ def tokenizer(self, text: str) -> list[str]:
 
     def load_w2v(self) -> None:
         """Load Thai2Fit's word2vec model"""
-        if self.thai2fit_wv is None:
-            raise ValueError(
-                "Thai2Fit word2vec model not found. "
-                "Please download it first using pythainlp.corpus.download('thai2fit_wv')"
+        if not self.thai2fit_wv:
+            raise FileNotFoundError(
+                "corpus-not-found name='thai2fit_wv'\n"
+                "  Corpus 'thai2fit_wv' not found.\n"
+                "    Python: pythainlp.corpus.download('thai2fit_wv')\n"
+                "    CLI:    thainlp data get thai2fit_wv"
             )
         self.aug: Word2VecAug = Word2VecAug(
             self.thai2fit_wv, self.tokenizer, type="binary"

pythainlp/cli/data.py

@@ -33,7 +33,8 @@ def __init__(self, argv: Sequence[str]) -> None:
                 "Current data path:\n\n"
                 f"{get_pythainlp_data_path()}\n\n"
                 "To change PyThaiNLP data path, set the operating system's\n"
-                "PYTHAINLP_DATA_DIR environment variable.\n\n"
+                "PYTHAINLP_DATA environment variable.\n"
+                "(PYTHAINLP_DATA_DIR is also accepted but deprecated.)\n\n"
                 "For more information about corpora that PyThaiNLP use, see:\n"
                 "https://github.com/PyThaiNLP/pythainlp-corpus/\n\n"
                 "--"