Defensive file loading - #1327
Conversation
Check row completeness before append to dictionary Check types before evaluate Signed-off-by: Arthit Suriyawongkul <arthit@gmail.com>
bact
added
refactoring
There was a problem hiding this comment.
Pull request overview
Improves robustness of built-in corpus loaders in pythainlp.corpus.common by skipping malformed records and warning instead of failing, aligning with the goal of defensive file loading.
Changes:
- Add warning-based skipping for malformed province CSV lines.
- Refactor
thai_dict(),thai_wsd_dict(), andthai_synonyms()to accumulate validated rows before assigning globals. - Add parsing/shape checks for
thai_wsd_dict()entries.
You can also share your feedback on Copilot code review. Take the survey.
Signed-off-by: Arthit Suriyawongkul <arthit@gmail.com>
There was a problem hiding this comment.
Pull request overview
Adds defensive, warning-based validation when loading certain corpus resources, preventing malformed rows from crashing corpus helpers and providing test coverage for these edge cases.
Changes:
- Add validation +
UserWarningemissions for malformed/blank rows inprovinces(),thai_dict(),thai_wsd_dict(), andthai_synonyms(). - Refactor dictionary/synonyms loaders to accumulate validated rows before caching globals.
- Add unit tests covering skipped rows and warning messages.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
pythainlp/corpus/common.py |
Implements defensive row validation and warning-based skipping for corpus loaders. |
tests/core/test_corpus.py |
Adds tests to verify skipping behavior and warnings for malformed/empty fields. |
You can also share your feedback on Copilot code review. Take the survey.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
|
Check row completeness and validate data before append to dictionary
Skip invalid entry, raise warning, and continue on next entry
Passed code styles and structures
Passed code linting checks and unit test