Commit 3ace752
committed
OPENNLP-1885: Add WordpieceEncoder and fold the unreleased BertTokenizer into it
WordpieceEncoder in opennlp-api runs the full BERT tokenization pipeline
as a SubwordTokenizer: every piece carries its vocabulary id and the span
of the original text, surviving the normalization steps that change,
insert, and remove characters. Content is computed with the same library
calls the previous pipeline made; offsets come from a per-code-point
rerun, with contextual case mappings (Greek final sigma) falling back to
word-wide spans that widen but never misplace. List and map constructors
cover line-number and explicit-id vocabularies.
BertTokenizer, unreleased and superseded, is removed. The dl tokenizer
creation builds on the encoder behind the existing Tokenizer plumbing via
a package-private adapter with unchanged special-token selection, and a
vocabulary missing its special tokens now fails at construction instead
of at the first id mapping, pinned by a test.
Parity is enforced twice: a differential suite against the reference
pipeline (kept test-only as ReferenceBertPipeline) over a curated corpus
plus 800 randomized inputs, and the removed class's reference token
sequences ported case for case. WordpieceTokenizer is untouched.1 parent 01863a8 commit 3ace752
12 files changed
Lines changed: 872 additions & 284 deletions
File tree
- opennlp-api/src/main/java/opennlp/tools/tokenize
- opennlp-core
- opennlp-ml/opennlp-dl/src
- main/java/opennlp/dl
- doccat
- namefinder
- vectors
- test/java/opennlp/dl
- opennlp-runtime/src/test/java/opennlp/tools/tokenize
Lines changed: 1 addition & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
22 | | - | |
| 22 | + | |
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
| |||
Lines changed: 0 additions & 211 deletions
This file was deleted.
0 commit comments