fix(newmm): resolve exponential BFS path explosion in ambiguous tokenization

[Copilot]

newmm would hang or run extremely slowly on text with many ambiguous breaking points — e.g., repeated "ด้านหน้า" (splittable as "ด้าน"+"หน้า" or "ด้านหน้า", with "หน้าด้าน" also valid). Tokenizing 1,000 repetitions took ~1.4 s before; after this fix it takes ~9 ms.

What do these changes do

• Add visited set to _bfs_paths_graph (O(2^n) → O(V+E))
• graph.clear() after each commit point in _onecut (both the ambiguous and no-candidate branches), replacing unbounded accumulation
• Remove the now-unnecessary graph[begin_pos].append(end_pos) in the no-candidate branch
• Trie.prefixes(text, start=0) gains an optional start offset — avoids creating an O(n) string copy (text[begin_pos:]) on every dictionary lookup
• Switch _PAT_NONTHAI.match(text[pos:]) → _PAT_NONTHAI.match(text, pos) for the same reason
• Add test_newmm_ambiguous_performance regression test: "ด้านหน้า" * 1000 must complete in < 5 s

What was wrong

Three compounding bugs in newmm._onecut / _bfs_paths_graph:

1. No visited set in BFS — full paths for every possible route were stored in the queue. With 2 valid split choices per ambiguous word, queue depth grows as 2^n. Measured: 31 pops for n=5, 1 023 for n=10, 57 848 for n=100.
2. graph never cleared — edges from all prior commit steps accumulated indefinitely, growing the BFS search space over the lifetime of a _onecut call.
3. text[begin_pos:] slice per Trie lookup — created an O(n) string copy on every call, making the per-character work O(n) and the total O(n²) for large inputs.

How this fixes it

The visited set makes BFS O(V+E); with _MAX_GRAPH_SIZE = 50, that is effectively O(1) per commit. Clearing the graph after each commit keeps V and E small. Passing a start offset to Trie.prefixes() eliminates the string-copy overhead entirely.

Trie.prefixes() change is backward-compatible (default start=0 preserves the existing API):

# Before — allocates entire remaining string on each call
for word in custom_dict.prefixes(text[begin_pos:]):
    ...

# After — no allocation; iterates from offset in the original string
for word in custom_dict.prefixes(text, begin_pos):
    ...

To fix #893

Your checklist for this pull request

• Passed code styles and structures
• Passed code linting checks and unit test

Original prompt

Analyses the issue of newmm tokenization engine as reported in #893

Download the text file in the comment to test the tokenizer. See where is the problem.

May need to divide and conquer, split the very long text into smaller parts to find the root cause.
But this may not guaranteed. The issue may not be directly about the content of the string but the length or other combinations of character types that occur in particular sequence.

Try to investigate to find the cause and fix it or avoid it (if can't fix entirely) or at least document about it so the user can aware of the limitations.

Or if it can be confirmed that the issue is no longer there, it's good, so we can close the issue.

This work can takes long time, iterate until can find the solution or can know that there is no reasonable solution.

---

🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. Learn more about Advanced Security.

[coveralls]

coverage: 65.546% (+0.01%) from 65.535%
when pulling 98511de on copilot/investigate-tokenization-issue
into f0d2f71 on dev.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[bact]

From a simple test, run locally, this PR speed up the tokenization. But newmm in normal mode (safe_mode = False) consumes way too much memory.

[newmm(safe_mode=False), dev branch]
  Input size     : 0.1 MB
  Tokens         : 4,823
  Wall-clock time: 0.552 s
  Peak memory    : 81.9 MB
.
[newmm(safe_mode=True), dev branch]
  Input size     : 0.1 MB
  Tokens         : 4,836
  Wall-clock time: 0.100 s
  Peak memory    : 0.4 MB
.
----------------------------------------------------------------------


[newmm(safe_mode=False), 5.2 branch]
  Input size     : 0.1 MB
  Tokens         : 4,823
  Wall-clock time: 0.759 s
  Peak memory    : 0.5 MB
.
[newmm(safe_mode=True), 5.2 branch]
  Input size     : 0.1 MB
  Tokens         : 4,836
  Wall-clock time: 0.566 s
  Peak memory    : 0.5 MB
.
----------------------------------------------------------------------

Trying to fix this with PR #1323

[bact]

Got it. The real "issue" is that the dictionary loading was made lazy by #1186 so at the first run of the tokenizer, it needs to load the dictionary.

Try switching the order of two tokenizers (safe_mode=False, then safe_mode=True) yields similar result. The first run will got the higher peak memory, because of the dictionary loading.

So this is not a performance bug, but by design.