Skip to content

Commit 3ace752

Browse files
committed
OPENNLP-1885: Add WordpieceEncoder and fold the unreleased BertTokenizer into it
WordpieceEncoder in opennlp-api runs the full BERT tokenization pipeline as a SubwordTokenizer: every piece carries its vocabulary id and the span of the original text, surviving the normalization steps that change, insert, and remove characters. Content is computed with the same library calls the previous pipeline made; offsets come from a per-code-point rerun, with contextual case mappings (Greek final sigma) falling back to word-wide spans that widen but never misplace. List and map constructors cover line-number and explicit-id vocabularies. BertTokenizer, unreleased and superseded, is removed. The dl tokenizer creation builds on the encoder behind the existing Tokenizer plumbing via a package-private adapter with unchanged special-token selection, and a vocabulary missing its special tokens now fails at construction instead of at the first id mapping, pinned by a test. Parity is enforced twice: a differential suite against the reference pipeline (kept test-only as ReferenceBertPipeline) over a curated corpus plus 800 randomized inputs, and the removed class's reference token sequences ported case for case. WordpieceTokenizer is untouched.
1 parent 01863a8 commit 3ace752

12 files changed

Lines changed: 872 additions & 284 deletions

File tree

opennlp-api/src/main/java/opennlp/tools/tokenize/BertNormalization.java

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@
1919

2020
/**
2121
* Character classifications and text transforms of the reference BERT
22-
* {@code BasicTokenizer}, shared by {@link BertTokenizer} and
22+
* {@code BasicTokenizer}, shared by {@link WordpieceEncoder} and
2323
* {@link WordpieceTokenizer}.
2424
*/
2525
final class BertNormalization {

opennlp-api/src/main/java/opennlp/tools/tokenize/BertTokenizer.java

Lines changed: 0 additions & 211 deletions
This file was deleted.

0 commit comments

Comments
 (0)