Skip to content

TokenizerProps.md

Latest commit

 

History

History
31 lines (16 loc) · 1.14 KB

TokenizerProps.md

File metadata and controls

31 lines (16 loc) · 1.14 KB

Interface: TokenizerProps

Defined in: types/tokenizer.ts:8

Parameters for initializing and configuring a Tokenizer instance.

Properties

preventLoad?

optional preventLoad: boolean

Defined in: types/tokenizer.ts:19

Boolean that can prevent automatic model loading (and downloading the data if you load it for the first time) after running the hook.

tokenizer

tokenizer: object

Defined in: types/tokenizer.ts:14

Object containing:

tokenizerSource - A ResourceSource that specifies the location of the tokenizer.

tokenizerSource

tokenizerSource: ResourceSource