Strip Japanese voiced-kana marks in BasicTokenizer (#352, Bug 2)

[apocryphx]

Addresses Bug 2 of #352.

BasicTokenizer.maybeStripAccents used .folding(options: .diacriticInsensitive, locale: nil), which strips Latin diacritics but does not strip the U+3099 / U+309A combining sound marks that Japanese voiced kana carry. Precomposed ザ (U+30B6) and で (U+3067) therefore reached WordPiece intact, while BERT-family vocabularies only contain the dakuten-stripped forms (##サ, ##て) — ##ザ and ##で are missing entries. A single missing continuation forced the WordPiece greedy match to fall back to [UNK] for the entire word, so long Japanese inputs containing any voiced kana returned essentially nothing useful (the issue document's "long katakana → single [UNK]" symptom is this root cause).

Switch to NFD-decompose-then-Mn-filter, matching HF Python's _run_strip_accents. This strips every nonspacing-mark scalar regardless of Unicode block, so Latin diacritics and Japanese dakuten/handakuten are handled identically.

Test plan

• swift test --filter TokenizerTests — all 46 tests pass (no regressions).
• New test bertJapaneseDakuten() uses BAAI/bge-small-en-v1.5 and a multi-script Japanese input that exercises hiragana, ideographs, and katakana including the voiced kana ザ and で. The expected IDs match HF Python transformers==4.57.1 byte-for-byte.

🤖 Generated with Claude Code

[apocryphx]

One observation for your review: there's an alternative location for this fix that may be architecturally cleaner — widening BertNormalizer.stripAccents in Normalizer.swift:238-244 from "filter U+0300..U+036F" to "filter all Mn." That single change handles both the Japanese voiced-kana case in this PR and the Hangul case that #353 addresses from a different angle (the Hangul jamo are Lo, not Mn, so they survive the widened filter and reach WordPiece exactly as needed).

Concretely, the alternative would be:

private func stripAccents(text: String) -> String {
    String(
        String.UnicodeScalarView(
            text.decomposedStringWithCanonicalMapping.unicodeScalars.filter {
                $0.properties.generalCategory != .nonspacingMark
            }
        )
    )
}

…and BasicTokenizer.maybeStripAccents could keep using .folding(options: .diacriticInsensitive) (or be removed entirely as redundant — the BertNormalizer pass runs first when tokenizer.json declares a BertNormalizer, which it does for BAAI/bge-small-en-v1.5 and most BERT-family models).

The current PR fixes the bug in BasicTokenizer.maybeStripAccents because that mirrors HF Python's BertTokenizer.basic_tokenizer._run_strip_accents — which is the right reference for the slow Python tokenizer. The fast (Rust) tokenizer that transformers==4.57.1 actually loads under use_fast=True uses the normalizer pipeline only, no separate BasicTokenizer step, so for the byte-identity contract against AutoTokenizer.from_pretrained(...), fixing BertNormalizer.stripAccents is arguably the more on-target location.

Either fix passes the regression test. Let me know which location you'd prefer and I'll repush.

For what it's worth, the pure-Foundation Obj-C port of this same tokenizer pipeline (https://github.com/apocryphx/ObjCTokenizer) consolidates this into a single Mn-filter pass at the BertNormalizer layer (OCTNormalizer.m:234-282) and produces byte-identical output to HF Python on the same multilingual corpus shipped at https://github.com/apocryphx/ObjCTokenizer/tree/main/Conformance — happy data point if you want to compare architecture choices.

[pcuenca]

Very nice, thanks for the in-depth explanations and breadcrumbs.

I verified the Rust implementation and it indeed relies on the normalizer fix alone.

I would suggest we apply the fix to the Normalizer, as you pointed out, but also keep maybeStripAccents as you already pushed here. This would be a defensive case against someone using BertTokenizer directly – I think it's rare, but possible. For example, a test like the following added to BertTokenizerTests would expose the problem:

    @Test("Direct BertTokenizer strips Japanese voiced-kana dakuten")
    func bertTokenizerStripsDakuten() {
        let vocab: [String: Int] = [
            "[UNK]": 0,
            "[CLS]": 1,
            "[SEP]": 2,
            "サ": 3,
            "##サ": 4,
        ]
        let tokenizer = BertTokenizer(vocab: vocab, merges: nil)

        #expect(tokenizer.tokenize(text: "ザ") == ["サ"])
        #expect(tokenizer.tokenize(text: "ザザ") == ["サ", "##サ"])
    }

---

Going forward, I think we should probably deprecate public use of BertTokenizer like above, and just rely on the from(pretrained:) path as a parallel to the "fast" tokenizers path, which is the maintained approach from transformers v5.

[pcuenca]

Fixed the linter whinings, merging now. Thanks a lot @apocryphx!