Add is_offline_mode(), PYTHAINLP_DATA env var, fix PYTHAINLP_DATA_DIR deprecation

Co-authored-by: bact <128572+bact@users.noreply.github.com>

Author: Copilot

CHANGELOG.md

@@ -43,6 +43,11 @@ See PR for prompt and details.
 - `get_corpus_path()` now respects `PYTHAINLP_OFFLINE` env var (same semantics
   as `HF_HUB_OFFLINE`): when set, raises `FileNotFoundError` if the corpus is
   not already cached locally; when unset, auto-downloads as before #1306
+- Added `pythainlp.is_offline_mode()` helper function (mirrors
+  `huggingface_hub.is_offline_mode()`) #1306
+- `PYTHAINLP_DATA` is now the preferred env var for the data directory
+  (same pattern as `NLTK_DATA`); `PYTHAINLP_DATA_DIR` is deprecated
+  and will emit a `DeprecationWarning` #1306
 - Callers raise `FileNotFoundError` with download instructions when a corpus
   path cannot be resolved (e.g. download failed) #1306
 - Improved documentation; code cleanup; more tests

README.md

@@ -77,14 +77,27 @@ please inspect the `[project.optional-dependencies]` section of
 PyThaiNLP downloads data (see the data catalog `db.json` at
 [pythainlp-corpus](https://github.com/PyThaiNLP/pythainlp-corpus))
 to `~/pythainlp-data` by default.
-Set the `PYTHAINLP_DATA_DIR` environment variable to override this location.
+Set the `PYTHAINLP_DATA` environment variable to override this location.
+(`PYTHAINLP_DATA_DIR` is still accepted but deprecated.)
 
 When using PyThaiNLP in distributed computing environments
-(e.g., Apache Spark), set the `PYTHAINLP_DATA_DIR` environment variable
+(e.g., Apache Spark), set the `PYTHAINLP_DATA` environment variable
 inside the function that will be distributed to worker nodes.
 See details in
 [the documentation](https://pythainlp.org/dev-docs/notes/installation.html).
 
+### Offline mode
+
+Set `PYTHAINLP_OFFLINE=1` to disable automatic corpus downloads.
+When this variable is set and a corpus is not already cached locally,
+a `FileNotFoundError` is raised instead of attempting a network download.
+Use `pythainlp.is_offline_mode()` to check the current state programmatically.
+
+```python
+import pythainlp
+print(pythainlp.is_offline_mode())  # True if PYTHAINLP_OFFLINE=1
+```
+
 ## Testing
 
 We test core functionalities on all officially supported Python versions.

README_TH.md

@@ -109,10 +109,11 @@ pip install "pythainlp[extra1,extra2,...]"
 PyThaiNLP ดาวน์โหลดข้อมูล (ดูแค็ตตาล็อกข้อมูล `db.json` ที่
 [pythainlp-corpus](https://github.com/PyThaiNLP/pythainlp-corpus))
 ไปที่ `~/pythainlp-data` ตามค่าเริ่มต้น
-ตั้งค่า environment variable `PYTHAINLP_DATA_DIR` เพื่อเปลี่ยนตำแหน่งนี้
+ตั้งค่า environment variable `PYTHAINLP_DATA` เพื่อเปลี่ยนตำแหน่งนี้
+(`PYTHAINLP_DATA_DIR` ยังคงใช้ได้แต่เลิกใช้แล้ว)
 
 เมื่อใช้ PyThaiNLP ในสภาพแวดล้อมการคำนวณแบบกระจาย
-(เช่น Apache Spark) ให้ตั้งค่า environment variable `PYTHAINLP_DATA_DIR`
+(เช่น Apache Spark) ให้ตั้งค่า environment variable `PYTHAINLP_DATA`
 ภายในฟังก์ชันที่จะถูกกระจายไปยัง worker nodes
 ดูรายละเอียดใน[เอกสาร](https://pythainlp.org/dev-docs/notes/installation.html)
 

docs/notes/installation.rst

@@ -89,7 +89,8 @@ Key considerations
 
 2. **Use a writable local directory**: The default data directory (``~/pythainlp-data``) may not be writable on executor nodes. Use a local directory like ``./pythainlp-data`` instead.
 
-3. **Set ``PYTHAINLP_DATA_DIR`` before data access**: Always set the ``PYTHAINLP_DATA_DIR`` environment variable before the first call that reads or writes PyThaiNLP data on each worker.
+3. **Set ``PYTHAINLP_DATA`` before data access**: Always set the ``PYTHAINLP_DATA`` environment variable before the first call that reads or writes PyThaiNLP data on each worker.
+   (``PYTHAINLP_DATA_DIR`` is also accepted for backward compatibility but is deprecated.)
 
 Example usage with Apache Spark
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -104,7 +105,7 @@ Basic example using PySpark RDD::
 
     def tokenize_thai(text):
         import os
-        os.environ['PYTHAINLP_DATA_DIR'] = './pythainlp-data'
+        os.environ['PYTHAINLP_DATA'] = './pythainlp-data'
         from pythainlp.tokenize import word_tokenize
         return word_tokenize(text)
 
@@ -123,7 +124,7 @@ Example using PySpark DataFrame API::
     @udf(returnType=ArrayType(StringType()))
     def tokenize_udf(text):
         import os
-        os.environ['PYTHAINLP_DATA_DIR'] = './pythainlp-data'
+        os.environ['PYTHAINLP_DATA'] = './pythainlp-data'
         from pythainlp.tokenize import word_tokenize
         return word_tokenize(text)
 
@@ -141,13 +142,30 @@ Note that while the code itself is thread-safe, you still need to configure the
 Runtime configurations
 ----------------------
 
-.. envvar:: PYTHAINLP_DATA_DIR
+.. envvar:: PYTHAINLP_DATA
 
    Specifies the location where downloaded data and the corpus database are stored. If the directory does not exist, PyThaiNLP will create it.
 
    By default this is a directory named ``pythainlp-data`` in the user's home directory.
 
-   Run ``thainlp data path`` at the command line to display the current `PYTHAINLP_DATA_DIR`.
+   Run ``thainlp data path`` at the command line to display the current data directory.
+
+.. envvar:: PYTHAINLP_DATA_DIR
+
+   .. deprecated::
+      Use :envvar:`PYTHAINLP_DATA` instead. Setting ``PYTHAINLP_DATA_DIR`` triggers a
+      :class:`DeprecationWarning` at runtime. If both ``PYTHAINLP_DATA`` and ``PYTHAINLP_DATA_DIR``
+      are set simultaneously, PyThaiNLP raises :exc:`ValueError`.
+
+.. envvar:: PYTHAINLP_OFFLINE
+
+   When set to a truthy value (``1``, ``true``, ``yes``, ``on``), PyThaiNLP operates in
+   *offline mode*: corpus downloads are disabled, and :func:`pythainlp.corpus.get_corpus_path`
+   raises :exc:`FileNotFoundError` for any corpus that is not already cached locally.
+
+   Use :func:`pythainlp.is_offline_mode` to check the current state programmatically.
+
+   This follows the same convention as ``HF_HUB_OFFLINE`` in `huggingface_hub`.
 
 .. envvar:: PYTHAINLP_READ_MODE
 
@@ -158,11 +176,11 @@ Installation FAQ
 
 Q: How do I set environment variables on each executor node in a distributed environment?
 
-A: When using PyThaiNLP in distributed computing environments like Apache Spark, you need to set the ``PYTHAINLP_DATA_DIR`` environment variable inside the function that will be distributed to executor nodes. For example::
+A: When using PyThaiNLP in distributed computing environments like Apache Spark, you need to set the ``PYTHAINLP_DATA`` environment variable inside the function that will be distributed to executor nodes. For example::
 
     def tokenize_thai(text):
         import os
-        os.environ['PYTHAINLP_DATA_DIR'] = './pythainlp-data'
+        os.environ['PYTHAINLP_DATA'] = './pythainlp-data'
         from pythainlp.tokenize import word_tokenize
         return word_tokenize(text)
 

pythainlp/__init__.py

@@ -55,6 +55,7 @@
 __all__: list[str] = [
     "collate",
     "correct",
+    "is_offline_mode",
     "pos_tag",
     "romanize",
     "spell",
@@ -76,5 +77,6 @@
     subword_tokenize,
     word_tokenize,
 )
+from pythainlp.tools.path import is_offline_mode
 from pythainlp.transliterate import romanize, transliterate
 from pythainlp.util import collate, thai_strftime

pythainlp/cli/data.py

@@ -33,7 +33,8 @@ def __init__(self, argv: Sequence[str]) -> None:
                 "Current data path:\n\n"
                 f"{get_pythainlp_data_path()}\n\n"
                 "To change PyThaiNLP data path, set the operating system's\n"
-                "PYTHAINLP_DATA_DIR environment variable.\n\n"
+                "PYTHAINLP_DATA environment variable.\n"
+                "(PYTHAINLP_DATA_DIR is also accepted but deprecated.)\n\n"
                 "For more information about corpora that PyThaiNLP use, see:\n"
                 "https://github.com/PyThaiNLP/pythainlp-corpus/\n\n"
                 "--"

pythainlp/corpus/core.py

@@ -18,6 +18,7 @@
 from pythainlp import __version__
 from pythainlp.corpus import corpus_db_path, corpus_db_url, corpus_path
 from pythainlp.tools import get_full_data_path
+from pythainlp.tools.path import is_offline_mode
 
 if TYPE_CHECKING:
     from typing import Any, Optional
@@ -30,19 +31,6 @@
 )
 
 
-def _is_offline() -> bool:
-    """Return True if ``PYTHAINLP_OFFLINE`` env var is set to a truthy value.
-
-    Truthy values: any non-empty string other than ``"0"``, ``"false"``,
-    ``"no"``, and ``"off"`` (case-insensitive).
-
-    This follows the same convention as ``HF_HUB_OFFLINE`` in
-    `huggingface_hub`.
-    """
-    val = os.getenv("PYTHAINLP_OFFLINE", "")
-    return val.strip().lower() not in ("", "0", "false", "no", "off")
-
-
 class _ResponseWrapper:
     """Wrapper to provide requests.Response-like interface for urllib response."""
 
@@ -343,7 +331,7 @@ def get_corpus_path(name: str, version: str = "") -> Optional[str]:
     corpus_db_detail = get_corpus_db_detail(name, version=version)
     if not corpus_db_detail:
         # Corpus not in local catalog; download it unless in offline mode
-        if _is_offline():
+        if is_offline_mode():
             raise FileNotFoundError(
                 f"Corpus '{name}' not found locally. "
                 f"PYTHAINLP_OFFLINE is set; automatic downloading is disabled. "
@@ -364,7 +352,7 @@ def get_corpus_path(name: str, version: str = "") -> Optional[str]:
         return path
 
     # File is registered in catalog but missing from disk
-    if _is_offline():
+    if is_offline_mode():
         raise FileNotFoundError(
             f"Corpus '{name}' expected at '{path}' but file not found. "
             f"PYTHAINLP_OFFLINE is set; automatic re-downloading is disabled. "
@@ -663,7 +651,7 @@ def download(
     if _CHECK_MODE == "1":
         print("PyThaiNLP is read-only mode. It can't download.")
         return False
-    if _is_offline():
+    if is_offline_mode():
         print(
             "PYTHAINLP_OFFLINE is set. Cannot download. "
             "To enable downloading, unset PYTHAINLP_OFFLINE."

pythainlp/tools/__init__.py

@@ -6,6 +6,7 @@
     "get_full_data_path",
     "get_pythainlp_data_path",
     "get_pythainlp_path",
+    "is_offline_mode",
     "safe_print",
     "warn_deprecation",
 ]
@@ -16,4 +17,5 @@
     get_full_data_path,
     get_pythainlp_data_path,
     get_pythainlp_path,
+    is_offline_mode,
 )

pythainlp/tools/path.py

@@ -21,6 +21,41 @@
 PYTHAINLP_DEFAULT_DATA_DIR: str = "pythainlp-data"
 
 
+def is_offline_mode() -> bool:
+    """Return whether PyThaiNLP is operating in offline mode.
+
+    Offline mode is activated by setting the ``PYTHAINLP_OFFLINE``
+    environment variable to a truthy value (e.g. ``"1"``).
+    Falsy values (``""``, ``"0"``, ``"false"``, ``"no"``, ``"off"``)
+    keep online mode active.
+
+    This follows the same convention as ``HF_HUB_OFFLINE`` in
+    `huggingface_hub`.
+
+    When offline mode is active, :func:`pythainlp.corpus.get_corpus_path`
+    raises :exc:`FileNotFoundError` for any corpus that is not already
+    cached locally, and :func:`pythainlp.corpus.download` refuses to
+    fetch anything from the network.
+
+    :return: ``True`` if PyThaiNLP is in offline mode, ``False`` otherwise.
+    :rtype: bool
+
+    :Example:
+    ::
+
+        import os
+        from pythainlp import is_offline_mode
+
+        os.environ["PYTHAINLP_OFFLINE"] = "1"
+        print(is_offline_mode())  # True
+
+        os.environ["PYTHAINLP_OFFLINE"] = "0"
+        print(is_offline_mode())  # False
+    """
+    val = os.getenv("PYTHAINLP_OFFLINE", "")
+    return val.strip().lower() not in ("", "0", "false", "no", "off")
+
+
 def get_full_data_path(path: str) -> str:
     """This function joins path of :mod:`pythainlp` data directory and the
     given path, and returns the full path.
@@ -40,11 +75,24 @@ def get_full_data_path(path: str) -> str:
 
 
 def get_pythainlp_data_path() -> str:
-    """Returns the full path where PyThaiNLP keeps its (downloaded) data.
-    If the directory does not yet exist, it will be created.
-    The path can be specified through the environment variable
-    :envvar:`PYTHAINLP_DATA_DIR`. By default, `~/pythainlp-data`
-    will be used.
+    """Return the full path where PyThaiNLP keeps its (downloaded) data.
+
+    The directory is created if it does not yet exist.
+
+    The path is resolved in the following order:
+
+    1. ``PYTHAINLP_DATA`` environment variable (preferred).
+    2. ``PYTHAINLP_DATA_DIR`` environment variable
+       (deprecated; shows a warning).
+    3. If **both** variables are set, the function raises
+       :exc:`ValueError` because the conflict must be resolved
+       explicitly.
+    4. If neither is set, ``~/pythainlp-data`` is used.
+
+    .. deprecated::
+        ``PYTHAINLP_DATA_DIR`` is deprecated.
+        Use ``PYTHAINLP_DATA`` instead (follows the same pattern as
+        ``NLTK_DATA``).
 
     :return: full path of directory for :mod:`pythainlp` downloaded data
     :rtype: str
@@ -57,10 +105,27 @@ def get_pythainlp_data_path() -> str:
         get_pythainlp_data_path()
         # output: '/root/pythainlp-data'
     """
-    pythainlp_data_dir = os.getenv(
-        "PYTHAINLP_DATA_DIR", os.path.join("~", PYTHAINLP_DEFAULT_DATA_DIR)
-    )
-    path = os.path.expanduser(pythainlp_data_dir)
+    import warnings
+
+    data_dir = os.getenv("PYTHAINLP_DATA")
+    data_dir_legacy = os.getenv("PYTHAINLP_DATA_DIR")
+
+    if data_dir and data_dir_legacy:
+        raise ValueError(
+            "Both PYTHAINLP_DATA and PYTHAINLP_DATA_DIR are set. "
+            "Please use PYTHAINLP_DATA only and unset PYTHAINLP_DATA_DIR."
+        )
+
+    if data_dir_legacy and not data_dir:
+        warnings.warn(
+            "PYTHAINLP_DATA_DIR is deprecated; use PYTHAINLP_DATA instead.",
+            DeprecationWarning,
+            stacklevel=2,
+        )
+        data_dir = data_dir_legacy
+
+    resolved = data_dir or os.path.join("~", PYTHAINLP_DEFAULT_DATA_DIR)
+    path = os.path.expanduser(resolved)
     os.makedirs(path, exist_ok=True)
     return path