Defined in: modules/natural_language_processing/TokenizerModule.ts:11
Module for Tokenizer functionalities.
new TokenizerModule():
TokenizerModule
TokenizerModule
nativeModule:
any
Defined in: modules/natural_language_processing/TokenizerModule.ts:15
Native module instance
decode(
tokens,skipSpecialTokens?):Promise<string>
Defined in: modules/natural_language_processing/TokenizerModule.ts:61
Converts an array of token IDs into a string.
number[]
Array of token IDs to be decoded.
boolean = true
Whether to skip special tokens during decoding (default: true).
Promise<string>
The decoded string.
encode(
input):Promise<number[]>
Defined in: modules/natural_language_processing/TokenizerModule.ts:51
Converts a string into an array of token IDs.
string
The input string to be tokenized.
Promise<number[]>
An array of token IDs.
getVocabSize():
Promise<number>
Defined in: modules/natural_language_processing/TokenizerModule.ts:75
Returns the size of the tokenizer's vocabulary.
Promise<number>
The vocabulary size.
idToToken(
tokenId):Promise<string>
Defined in: modules/natural_language_processing/TokenizerModule.ts:84
Returns the token associated to the ID.
number
ID of the token.
Promise<string>
The token string associated to ID.
load(
tokenizer,onDownloadProgressCallback?):Promise<void>
Defined in: modules/natural_language_processing/TokenizerModule.ts:23
Loads the tokenizer from the specified source.
tokenizerSource is a string that points to the location of the tokenizer JSON file.
Object containing tokenizerSource.
(progress) => void
Optional callback to monitor download progress.
Promise<void>
tokenToId(
token):Promise<number>
Defined in: modules/natural_language_processing/TokenizerModule.ts:93
Returns the ID associated to the token.
string
The token string.
Promise<number>
The ID associated to the token.