Skip to content

test: expand GoJieba fulltext BVT coverage#25897

Draft
Ariznawlll wants to merge 4 commits into
mainfrom
test/gojieba-bvt-coverage
Draft

test: expand GoJieba fulltext BVT coverage#25897
Ariznawlll wants to merge 4 commits into
mainfrom
test/gojieba-bvt-coverage

Conversation

@Ariznawlll

@Ariznawlll Ariznawlll commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

What changed

  • add focused GoJieba tokenizer BVT coverage for exact tokens, UTF-8 positions, DocLen, mixed text, repeated terms, token-length limits, and multi-column boundaries
  • cover the remaining SQL-level tokenizer gaps: dictionary-only/HMM-disabled segmentation plus whitespace-only, punctuation-only, empty, and NULL input
  • cover NULL/empty-value indexing, duplicate-term and cross-document negatives, empty boolean-pattern rejection, non-indexed-column updates, and TRUNCATE/reinsert visibility
  • run the deterministic DISTINCT, GROUP BY, HAVING, CTE, derived-table, and ordinary-predicate query shapes through a GoJieba index
  • retain unsupported query shapes as standard issue-tagged expected failures

Why

These small deterministic contracts were primarily exercised by nightly regression, so regressions could be detected later than necessary. The new BVT cases move fast functional coverage into the MatrixOne repository without duplicating existing JOIN, UNION, ranking, JSON, primary-key update, large-data, or transaction scenarios.

Impact

Test-only change. No production code or transaction cases are modified.

Validation

  • mo-tester: 53 total, 51 passed, 2 expected issue skips, 0 failed, 0 abnormal
  • go test ./pkg/sql/plan -run FullText -count=1
  • git diff --check

Closes #25896

@matrix-meow matrix-meow added size/L Denotes a PR that changes [500,999] lines and removed size/M Denotes a PR that changes [100,499] lines labels Jul 20, 2026
@Ariznawlll
Ariznawlll marked this pull request as ready for review July 20, 2026 09:02
@Ariznawlll
Ariznawlll requested a review from heni02 as a code owner July 20, 2026 09:02
@qodo-code-review

Copy link
Copy Markdown

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

@gouhongshen gouhongshen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex automated review

One non-blocking BVT determinism issue found. Tokenizer unit tests pass; broader tests were blocked by missing local C headers.

P2 - Order exact tokenizer results explicitly (test/distributed/cases/fulltext/gojieba_precise.sql:10)

The six select f.pos, f.word queries assert a specific row sequence, including __DocLen last, but have no ORDER BY. SQL does not guarantee table-function result order, so plan or distributed-execution changes can make this BVT flaky without changing tokenization. Add ORDER BY f.pos to each exact-token query.

@Ariznawlll
Ariznawlll marked this pull request as draft July 20, 2026 11:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/L Denotes a PR that changes [500,999] lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Test]: expand GoJieba fulltext BVT coverage

3 participants