Reduce peak memory on import by 62x

[what-in-the-nim]

What does this changes

Refactor by moving expensive loading variable into LRU cached function

What was wrong

Most of the variables, tokenizer and tries, are initialized in the module scope, so when we do import pythainlp, this will unnecessarily load everything into the RAM. This also affect the import time too.

How this fixes it

I move everything inside the function to prevent sudden execution (lazy) and wrap it with functools.lru_cache to cache the result for future access.

This is the minimal script to test the behavior

# test.py
import time
import tracemalloc

tracemalloc.start()
start = time.perf_counter()

import pythainlp

end = time.perf_counter()

# Capture snapshot AFTER imports
snapshot = tracemalloc.take_snapshot()

current, peak = tracemalloc.get_traced_memory()
tracemalloc.stop()

print(f"Version: {pythainlp.__version__}")

# Top memory users by file
top_stats = snapshot.statistics("filename")

print("\nTop 10 memory allocations by file:")
for stat in top_stats[:10]:
    print(f"{stat.traceback[0].filename}: {stat.size / 1e6:.3f} MB")

print(f"\nCurrent memory: {current / 1e6:.3f} MB")
print(f"Peak memory: {peak / 1e6:.3f} MB")
print(f"Import time: {end - start:.4f}s")

Branch dev

Version: 5.2.0

Top 10 memory allocations by file:
/home/nim/Project/pythainlp/pythainlp/util/trie.py: 186.240 MB
/home/nim/Project/pythainlp/pythainlp/corpus/core.py: 11.781 MB
/home/nim/Project/pythainlp/pythainlp/corpus/tnc.py: 3.074 MB
<frozen importlib._bootstrap_external>: 1.900 MB
/usr/lib/python3.12/collections/__init__.py: 0.961 MB
/home/nim/Project/pythainlp/pythainlp/util/wordtonum.py: 0.506 MB
<frozen importlib._bootstrap>: 0.304 MB
/usr/lib/python3.12/typing.py: 0.184 MB
/home/nim/Project/pythainlp/pythainlp/util/emojiconv.py: 0.137 MB
<frozen abc>: 0.075 MB

Current memory: 205.733 MB
Peak memory: 222.194 MB
Import time: 2.6567s

This PR

Version: 5.2.0

Top 10 memory allocations by file:
<frozen importlib._bootstrap_external>: 1.906 MB
<frozen importlib._bootstrap>: 0.305 MB
/usr/lib/python3.12/typing.py: 0.185 MB
/home/nim/Project/pythainlp/pythainlp/util/emojiconv.py: 0.137 MB
/usr/lib/python3.12/re/_compiler.py: 0.137 MB
/home/nim/Project/pythainlp/pythainlp/corpus/core.py: 0.110 MB
<frozen abc>: 0.074 MB
/usr/lib/python3.12/ast.py: 0.065 MB
/usr/lib/python3.12/sysconfig.py: 0.036 MB
/usr/lib/python3.12/re/_parser.py: 0.035 MB

Current memory: 3.402 MB
Peak memory: 3.559 MB
Import time: 0.1471s

Import time is ~18x faster and peak RAM is reduced by ~62x

Your checklist for this pull request

• Passed code styles and structures
• Passed code linting checks and unit test

[Copilot]

Pull request overview

This PR refactors the PyThaiNLP library to reduce memory usage and import time by converting eagerly-loaded module-level variables (tokenizers and tries) into lazily-loaded, LRU-cached functions. This prevents expensive data structures from being loaded immediately when the module is imported.

Changes:

• Converted global tokenizer and trie constants to lru_cache decorated functions that load on-demand
• Extracted Thai2Fit tokenizer initialization into a separate module with lazy loading
• Updated all references throughout the codebase to call the new lazy-loading functions
• Modified function signatures to accept Optional parameters instead of using the old constants as defaults

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
pythainlp/tokenize/init.py	Converted DEFAULT_WORD_DICT_TRIE, DEFAULT_SYLLABLE_DICT_TRIE, and THAI2FIT_TOKENIZER to lazy-loaded functions
pythainlp/tokenize/thai2fit.py	New module containing lazy-loaded Thai2Fit tokenizer function
pythainlp/tokenize/core.py	Updated to use lazy-loaded trie functions; changed custom_dict parameter default to None
pythainlp/tokenize/newmm.py	Updated to call word_dict_trie() function and handle Optional[Trie] parameter
pythainlp/tokenize/multi_cut.py	Updated to call word_dict_trie() function and handle Optional[Trie] parameter
pythainlp/tokenize/longest.py	Updated to call word_dict_trie() function and handle Optional[Trie] parameter
pythainlp/util/wordtonum.py	Converted module-level tokenizers to lazy-loaded functions
pythainlp/util/time.py	Converted module-level tokenizer to lazy-loaded function
pythainlp/util/spell_words.py	Converted module-level tokenizer to lazy-loaded function
pythainlp/util/phoneme.py	Converted module-level tokenizer to lazy-loaded function
pythainlp/word_vector/core.py	Updated to call thai2fit_tokenizer() function
pythainlp/ulmfit/core.py	Changed tok_func parameter default from callable to None with lazy initialization
pythainlp/ulmfit/tokenizer.py	Updated to call thai2fit_tokenizer() function
pythainlp/augment/word2vec/thai2fit.py	Updated to call thai2fit_tokenizer() function
tests/core/test_tokenize.py	Updated test to call word_dict_trie() function
tests/extra/testx_tokenize.py	Updated tests to call word_dict_trie() function

Comments suppressed due to low confidence (1)

pythainlp/tokenize/core.py:883

• Debug print statement should be removed. This appears to be leftover debugging code that will print "HEHE" every time a Tokenizer is instantiated.

        self.__trie_dict = Trie([])

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated no new comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[coveralls]

coverage: 52.793% (-0.2%) from 52.947%
when pulling f91a977 on what-in-the-nim:reduce-peak-memory
into 8819cfe on PyThaiNLP:dev.

[bact]

Thank you for this PR.

The library use case was primarily tokenization back at the time, so it kinda made sense to load the dictionary immediately when import, to simplify things and because it will eventually be loaded anyway (as it is the main use case).

Since the library is growing and offers more functionality. That assumption no longer holds.
So yes, let's refactor and do the lazy loading when it makes sense.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[bact]

Merged. Thank you @what-in-the-nim