Thanks for adding this — the in-memory Create(vocab, ...) overload is clean and the ID-preservation (JSON vocab index = token id) is the right call. I implemented this same JSON-only Unigram capability recently against real Hugging Face models, and hit two correctness issues that this PR's single test model happens to mask. Details inline; summary here.

Bugs (untested by the current suite)

1. BOS/EOS positional fallback corrupts real pieces. FindSpecialTokenId(pieces, "<s>", 1) / ("</s>", 2) fall back to positions 1/2 when the vocab has no piece literally named <s>/</s>. Many HF Unigram tokenizers don't use those names — e.g. minishlab/potion-multilingual-128M (bge-m3 family) has unk_id=1, vocab [0]="[PAD]", [1]="[UNK]", [2]=",". There eosId→2 marks "," as Control and drops it from the Viterbi trie (it can never be emitted), and bosId→1 collides with unkId and clobbers the unknown entry. This is structural (independent of addBos/addEos).

2. Normalizer steps beyond Precompiled are silently dropped. ExtractPrecompiledCharsMap extracts only the charsmap and discards sibling normalizers. Real Unigram models often have a richer chain (potion/bge-m3: Sequence[Precompiled, Replace(punctuation spacing), Replace("\\s+"->" "), Strip]), which SentencePieceNormalizer cannot reproduce — so CreateFromTokenizerJson silently yields different tokens than HF. Since the charsmap must run before those Replace steps, they can't just be reordered into SP; at minimum this should throw on unrecognized normalizer types rather than silently ignore them.

Why the test stays green

Paraphrase-multilingual-MiniLM-L12-v2 names its specials (<s>=0, </s>=2) so the fallback never fires, and its normalizer is a single Precompiled, so the dropped-sibling path is never hit. Recommend adding fixtures that (a) place specials at non-conventional positions / omit <s>/</s>, and (b) use a Sequence normalizer with Replace/Strip, asserting against HF reference ids.

Minor

• removeExtraWhitespaces is hard-coded true in both factories rather than derived from the JSON.
• added_tokens from the JSON aren't auto-wired; correctness depends on the caller passing specialTokens. Worth documenting or reading them.

(Posting as comments only — not an approval or change request.)

Pull request overview

This PR extends SentencePieceTokenizer to support Hugging Face JSON-only Unigram tokenizers by adding new public factory APIs that can construct a Unigram tokenizer from either an in-memory vocab list or a tokenizer.json stream, avoiding the current requirement for a SentencePiece .model protobuf.

Changes:

• Add SentencePieceTokenizer.Create(IEnumerable<(string Piece, float Score)> vocab, ...) for constructing a Unigram tokenizer directly from a vocab list.
• Add SentencePieceTokenizer.CreateFromTokenizerJson(Stream tokenizerJsonStream, ...) for parsing HF tokenizer.json (Unigram) including vocab, unk_id, precompiled charsmap, and Metaspace settings.
• Add internal constructors/refactoring to build a SentencePieceUnigramModel from vocab pieces and config values, plus new tests covering these creation paths.

Copilot's findings

• Files reviewed: 4/4 changed files
• Comments generated: 4

Copilot's findings

• Files reviewed: 6/6 changed files
• Comments generated: 7

Copilot's findings

• Files reviewed: 6/6 changed files
• Comments generated: 3

Copilot's findings

• Files reviewed: 6/6 changed files
• Comments generated: 4

Copilot's findings

• Files reviewed: 6/6 changed files
• Comments generated: 1

Copilot's findings

• Files reviewed: 6/6 changed files
• Comments generated: 3

Copilot's findings

• Files reviewed: 6/6 changed files
• Comments generated: 3

Copilot's findings

• Files reviewed: 6/6 changed files
• Comments generated: 0 new

Left minor comment, LGTM otherwise.