Skip to content

docs: convert :Example: RST code blocks to proper Python doctest format - #1392

Merged
bact merged 30 commits into
devfrom
copilot/update-documentation-and-doctests
Apr 1, 2026
Merged

docs: convert :Example: RST code blocks to proper Python doctest format#1392
bact merged 30 commits into
devfrom
copilot/update-documentation-and-doctests

Conversation

Copilot AI commented Mar 31, 2026

Copy link
Copy Markdown
Contributor

What do these changes do

Convert :Example: docstring sections from RST :: code-block format to runnable Python doctest format (>>>) across all pythainlp/ modules.

What was wrong

Example sections used RST literal-block syntax (:: + indented code + # output: ... comments), which Sphinx renders but Python's doctest module cannot execute or validate:

# Before — not runnable as doctest
:Example:
::

    from pythainlp.util import remove_dangling

    remove_dangling("๊ก")
    # output: 'ก'

How this fixes it

Converted all discoverable :Example: blocks in the affected files to proper doctest format with verified expected outputs (run against compact + extra dependencies). Examples requiring optional dependencies or corpus downloads are marked # doctest: +SKIP. Several incorrect expected outputs in the original comments were also fixed in the process.

# After — runnable doctest
:Example:

    >>> from pythainlp.util import remove_dangling
    >>> remove_dangling("๊ก")
    'ก'

Files converted:

pythainlp/util/:

  • collate, digitconv, keyboard, normalize, numtoword, wordtonum — initial batch
  • trie, syllable, morse, encoding, lcs, emojiconv — second batch
  • thaianalyze_thai_text example converted per review feedback

pythainlp/soundex/:

  • udom83, lk82, metasound, prayut_and_somchaip, core — live doctests with verified outputs
  • sound — marked +SKIP (requires optional panphon dependency)

pythainlp/morpheme/:

  • thaiwordcheck — flattened sub-section structure into standard doctest block
  • word_formation — converted assert statements to >>> / expected-output pairs

pythainlp/transliterate/:

  • core (romanize, transliterate, pronunciate) — royin/iso_11940 run live; engine-specific examples marked +SKIP
  • spoonerism, wunsen — marked +SKIP (optional deps)

pythainlp/ancient/:

  • aksonhan, currency — live doctests with verified outputs

pythainlp/braille/:

  • core — live doctests with verified outputs

pythainlp/tokenize/:

  • core, _utils — core examples live; ML-engine examples marked +SKIP

pythainlp/corpus/:

  • common, core, phupha, util — marked +SKIP (require corpus downloads)

pythainlp/tools/:

  • misspell, path — marked +SKIP (non-deterministic or environment-specific)

pythainlp/tag/:

  • locations, named_entity, pos_tag, thai_nner, thainer, tltk — marked +SKIP (require NER/POS models)

pythainlp/translate/:

  • core, en_th, small100, th_fr, zh_th — marked +SKIP (require translation models)

pythainlp/augment/:

  • lm/phayathaibert, lm/wangchanberta, word2vec/bpemb_wv, word2vec/ltw2v, word2vec/thai2fit, wordnet — marked +SKIP (require neural models)

pythainlp/benchmarks/:

  • metrics — live doctests with verified outputs (pure computation)

pythainlp/chat/, pythainlp/chunk/, pythainlp/classify/, pythainlp/coref/, pythainlp/el/, pythainlp/generate/, pythainlp/khavee/, pythainlp/lm/, pythainlp/parse/, pythainlp/phayathaibert/, pythainlp/spell/, pythainlp/summarize/, pythainlp/ulmfit/, pythainlp/word_vector/, pythainlp/wsd/:

  • All examples marked +SKIP (require ML models or optional dependencies)

Your checklist for this pull request

  • Passed code styles and structures
  • Passed code linting checks and unit test

@bact bact added the documentation improve documentation and test cases label Mar 31, 2026
Copilot AI and others added 5 commits March 31, 2026 14:41
- Fix formatting, typos, and reStructuredText issues
- Fix doctest continuation line format (use ...)
- Use parallel language in descriptions
- Remove duplicate documentation
- Update outdated information

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: bact <128572+bact@users.noreply.github.com>
- complete_soundex.py: fix expected output for complete_soundex("ปุญญา")
  from 'ปป4G0น-ยย1B0--*' to 'ปป1B0น-*' (actual output)
- complete_soundex.py: fix expected output for complete_soundex("ปันนา")
  from 'ปป1A0น-นน1B0--' to 'ปป1Bน0-' (actual output)
- metasound.py: fix typo (extra closing paren) in docstring example
- _tag_perceptron.py: fix multiline list doctest formatting with '...'
  continuation lines so the doctest runs correctly
- pos_tag.py: remove duplicate 'tdtb' corpus entry in docstring
- pos_tag.py: fix missing closing quote in pos_tag_sents example

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: bact <128572+bact@users.noreply.github.com>
- corpus/common.py: fix wrong URL filenames for provinces and ORST
  words; fix grammar ("if it isn't exist"); fix rtype List->list
- corpus/core.py: convert @param/@return Javadoc style to RST :param:
  in _download, _check_hash, _is_within_directory, _safe_extract_tar,
  _safe_extract_zip
- corpus/tnc.py: convert Credit: inline attribution to RST :See Also:
- corpus/util.py: fix period typo in example (str. ... -> str, ...);
  fix undefined 'wordlist' variable in example -> 'base_words'; fix
  wrong :return: text in revise_wordset/revise_newmm_default_wordset;
  fix rtype Set->set; fix grammar
- spell/pn.py: add missing SPDX file header; fix :param str custom_dict:
  (not a str) to split :param:/:type: format
- tools/path.py: remove outdated Python 3.6 reference in example output
- transliterate/core.py: fix missing closing quote 'thai2rom_onnx,' in
  engine list
- transliterate/iso_11940.py: fix wrong param name (text->word); add
  summary line and :rtype:
- transliterate/lookup.py: improve follow_rtgs docstring; clean inline
  comments in _romanize
- transliterate/thai2rom.py: add summary line and :rtype: to romanize

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: bact <128572+bact@users.noreply.github.com>
- summarize/core.py: fix :func: spacing, long lines, missing :rtype:,
  and improve extract_keywords docstring clarity
- benchmarks/word_tokenization.py: shorten long docstring lines, fix
  param name mismatch (samples vs raw_sample)
- khavee/core.py: fix capitalization in check_marttra docstring and
  fix wrong :param name in handle_karun_sound_silence
- morpheme/word_formation.py: remove backslash continuations from
  nighit docstring, rewrite more clearly
- ancient/aksonhan.py: remove backslash continuation in docstring
- ancient/currency.py: fix typo 'is not support' -> 'is not supported'
- util/strftime.py: fix typo 'datatime.strrftime' -> 'datetime.strftime'
- util/morse.py: remove backslash continuation in morse_decode docstring
- util/time.py: remove backslash continuation in time_to_thaiword docstring
- util/phoneme.py: remove backslash continuations, shorten comment block
- util/spell_words.py: fix wrong param name 's' -> 'text' in docstring
- util/keywords.py: remove backslash continuations, fix misaligned comment
- util/abbreviation.py: shorten long :return: line in docstring

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: bact <128572+bact@users.noreply.github.com>
Copilot AI changed the title [WIP] Update documentation, comments, and doctests Fix doctests, docstring formatting, typos, and outdated content across all modules Mar 31, 2026
Copilot finished work on behalf of bact March 31, 2026 15:17
Copilot AI requested a review from bact March 31, 2026 15:17
Copilot AI changed the title Fix doctests, docstring formatting, typos, and outdated content across all modules docs: convert :Example: RST code blocks to proper Python doctest format Mar 31, 2026
Copilot finished work on behalf of bact March 31, 2026 16:37
@coveralls

coveralls commented Mar 31, 2026

Copy link
Copy Markdown

Coverage Status

coverage: 66.633%. remained the same
when pulling 7892663 on copilot/update-documentation-and-doctests
into 9dfae5a on dev.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR focuses on improving PyThaiNLP’s documentation quality by converting docstring :Example: sections (primarily under pythainlp/util/) into runnable Python doctests and by cleaning up docstring formatting/typos across the codebase.

Changes:

  • Convert many :Example: docstring blocks from RST literal blocks (:: + # output:) to doctest prompts (>>>) with concrete expected outputs.
  • Standardize and clarify docstrings (params/return types, wording, typos, references) across multiple modules.
  • Add/refresh a CHANGELOG.md “[Unreleased]” entry describing the documentation/doctest cleanup.

Reviewed changes

Copilot reviewed 55 out of 55 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
pythainlp/word_vector/core.py Removes redundant __init__ docstring (class docstring remains).
pythainlp/wangchanberta/init.py Adds module docstring.
pythainlp/util/wordtonum.py Converts :Example: blocks to doctest; docstring wording updates.
pythainlp/util/trie.py Converts :Example: block to doctest.
pythainlp/util/time.py Refines parameter docs for time_data.
pythainlp/util/thai.py Docstring refactors; examples formatting changed (see comments).
pythainlp/util/syllable.py Converts :Example: blocks to doctest.
pythainlp/util/strftime.py Fixes incorrect function reference in docs.
pythainlp/util/spell_words.py Docstring param/return type cleanup.
pythainlp/util/phoneme.py Improves reference formatting and clarifies comments.
pythainlp/util/numtoword.py Converts :Example: blocks to doctest.
pythainlp/util/normalize.py Converts :Example: blocks to doctest; docstring typing/deprecation markup updates.
pythainlp/util/morse.py Converts :Example: blocks to doctest; docstring cleanup.
pythainlp/util/lcs.py Converts :Example: block to doctest.
pythainlp/util/keywords.py Docstring formatting fixes for examples.
pythainlp/util/keyboard.py Converts examples to doctest and updates parameter docs.
pythainlp/util/encoding.py Fixes encoding cookie; converts examples to doctest.
pythainlp/util/emojiconv.py Converts :Example: block to doctest.
pythainlp/util/digitconv.py Converts :Example: blocks to doctest; docstring restructuring.
pythainlp/util/date.py Fixes incorrect example output comment.
pythainlp/util/collate.py Converts :Example: block to doctest.
pythainlp/util/abbreviation.py Updates return type docs to match typing.
pythainlp/ulmfit/tokenizer.py Docstring rewrite/cleanup and doctest formatting alignment.
pythainlp/ulmfit/core.py Docstring cleanup + corrected exception type documentation.
pythainlp/transliterate/thai2rom.py Adds short summary + missing :rtype:.
pythainlp/transliterate/lookup.py Docstring clarity improvements.
pythainlp/transliterate/iso_11940.py Docstring clarity improvements + :rtype:.
pythainlp/transliterate/core.py Fixes typo in documented engine list.
pythainlp/translate/core.py Docstring formatting fix + clearer return typing docs.
pythainlp/tools/path.py Updates example output path to be less version-specific.
pythainlp/tokenize/tcc.py Fixes return-type description mismatch (set vs list).
pythainlp/tokenize/longest.py Docstring wording/formatting cleanup.
pythainlp/tokenize/etcc.py Fixes duplicated :return: and corrects :rtype:.
pythainlp/tokenize/core.py Extensive docstring formatting/typing improvements across APIs.
pythainlp/tag/pos_tag.py Removes duplicated/incorrect option docs; fixes example typo.
pythainlp/tag/_tag_perceptron.py Fixes doctest continuation formatting for list literal.
pythainlp/summarize/core.py Docstring standardization + adds missing :rtype:.
pythainlp/spell/pn.py Adds SPDX headers; improves docstring typing/formatting.
pythainlp/soundex/metasound.py Fixes typo in example ())).
pythainlp/soundex/complete_soundex.py Updates documented expected outputs; doctest examples present (see comments).
pythainlp/morpheme/word_formation.py Docstring clarity improvements.
pythainlp/lm/text_util.py Docstring clarity improvements.
pythainlp/khavee/core.py Docstring grammar fixes.
pythainlp/generate/core.py Docstring clarity improvements + corrected :rtype:.
pythainlp/corpus/util.py Docstring corrections + example fixes.
pythainlp/corpus/tnc.py Docstring formatting and reference cleanup.
pythainlp/corpus/core.py Converts ad-hoc @param docs to Sphinx-style :param:.
pythainlp/corpus/common.py Updates corpus file references + docstring return typing cleanup.
pythainlp/cli/misspell.py Adds module docstring.
pythainlp/cli/benchmark.py Adds module docstring.
pythainlp/classify/param_free.py Docstring expansions for save/load.
pythainlp/benchmarks/word_tokenization.py Docstring clarity improvements + minor signature docs fixes.
pythainlp/ancient/currency.py Fixes grammar in error message (“supported”).
pythainlp/ancient/aksonhan.py Docstring clarity/grammar improvements.
CHANGELOG.md Adds “[Unreleased]” entry describing the docs/doctest cleanup.
Comments suppressed due to low confidence (1)

pythainlp/util/thai.py:316

  • The PR aims to convert :Example: blocks to runnable doctests, but this example is still written as an RST literal block (::) with print(...) and # output: comments. doctest won’t execute/verify it as-is; please convert to >>> style (and avoid print) or mark it as skipped if it relies on optional dependencies.

Comment thread pythainlp/util/thai.py
Comment thread pythainlp/soundex/complete_soundex.py Outdated
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copilot finished work on behalf of bact March 31, 2026 18:02
Copilot AI requested a review from bact March 31, 2026 22:13
@bact

bact commented Mar 31, 2026

Copy link
Copy Markdown
Member

Some strftime doctests are wrong (the 1:40 AM ones). Need to fix that.
Need more reviews on others as well.

@bact

bact commented Apr 1, 2026

Copy link
Copy Markdown
Member

All wrong doctests fixed. doctests enabled as much as possible. Still skipped those required optional dependencies.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 98 out of 98 changed files in this pull request and generated 5 comments.

Comment thread pythainlp/util/time.py
Comment thread pythainlp/util/time.py
Comment thread pythainlp/util/pronounce.py
Comment thread pythainlp/benchmarks/word_tokenization.py Outdated
Comment thread pythainlp/transliterate/core.py Outdated
bact and others added 2 commits April 1, 2026 14:08
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Comment thread pythainlp/transliterate/core.py Outdated
bact added 2 commits April 1, 2026 14:10
Updated doctest examples to remove SKIP directive.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 98 out of 98 changed files in this pull request and generated 1 comment.

Comment thread pythainlp/util/remove_trailing_repeat_consonants.py Outdated
@bact
bact merged commit 0207d40 into dev Apr 1, 2026
21 of 22 checks passed
@bact bact added this to PyThaiNLP Apr 1, 2026
@github-project-automation github-project-automation Bot moved this to Done in PyThaiNLP Apr 1, 2026
@bact bact added this to the 5.3.4 milestone Apr 1, 2026
@sonarqubecloud

sonarqubecloud Bot commented Apr 1, 2026

Copy link
Copy Markdown

@bact
bact deleted the copilot/update-documentation-and-doctests branch April 1, 2026 14:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation improve documentation and test cases

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

4 participants