diff --git a/docs-website/reference/haystack-api/token_counters_api.md b/docs-website/reference/haystack-api/token_counters_api.md new file mode 100644 index 0000000000..ab33c3731e --- /dev/null +++ b/docs-website/reference/haystack-api/token_counters_api.md @@ -0,0 +1,220 @@ +--- +title: "Token Counters" +id: token-counters-api +description: "Estimate how many tokens a conversation occupies, for features that need a size before sending it to a model." +slug: "/token-counters-api" +--- + + +## approximate_counter + +### ApproximateTokenCounter + +Bases: TokenCounter + +Estimates tokens from text length using a flat ratio of characters to tokens. + +## Usage Example: + +```python +from haystack.dataclasses import ChatMessage +from haystack.token_counters import ApproximateTokenCounter + +counter = ApproximateTokenCounter(chars_per_token=4.0) +messages = [ + ChatMessage.from_user("Hello, how are you?"), + ChatMessage.from_assistant("I'm good, thank you! How can I assist you today?") +] +token_count = counter.count(messages) +print(f"Estimated token count: {token_count}") +``` + +#### __init__ + +```python +__init__( + chars_per_token: float = 4.0, + tokens_per_image: int = 85, + tokens_per_file: int = 1000, +) -> None +``` + +Initialize the counter. + +**Parameters:** + +- **chars_per_token** (float) – How many characters to treat as one token. +- **tokens_per_image** (int) – Tokens to charge per image, which has no text to measure. The default is what + OpenAI charges for a small image; raise it if you send large ones. +- **tokens_per_file** (int) – Tokens to charge per file. A rough stand-in for a short document, since the real + cost depends on the page count; raise it if you send long ones. + +**Raises:** + +- ValueError – If `chars_per_token` is not positive. + +#### count + +```python +count(messages: list[ChatMessage], tools: ToolsType | None = None) -> int +``` + +Return the estimated number of tokens the given messages occupy. + +**Parameters:** + +- **messages** (list\[ChatMessage\]) – The messages to measure. +- **tools** (ToolsType | None) – Tools whose schemas are sent alongside the messages, and so consume tokens too. + +**Returns:** + +- int – The estimated token count, or `0` when there is nothing to measure. + +#### to_dict + +```python +to_dict() -> dict[str, Any] +``` + +Serialize the counter. + +**Returns:** + +- dict\[str, Any\] – A dictionary representation of the counter. + +## tiktoken_counter + +### TiktokenCounter + +Bases: TokenCounter + +Counts tokens locally with `tiktoken`, OpenAI's byte-pair encoder. + +Counting is an estimate, and two limits are worth knowing before relying on it: + +- **It is text-only**, so images and files get the flat `tokens_per_image` / `tokens_per_file` estimate rather + than a real count. +- **It is OpenAI's encoder.** Other providers tokenize differently, so expect the count to drift on them. + +## Usage Example: + +```python +from haystack.dataclasses import ChatMessage +from haystack.token_counters import TiktokenCounter + +counter = TiktokenCounter(encoding="o200k_base") +messages = [ + ChatMessage.from_user("Hello, how are you?"), + ChatMessage.from_assistant("I'm good, thank you! How can I assist you today?") +] +token_count = counter.count(messages) +print(f"Token count: {token_count}") +``` + +#### __init__ + +```python +__init__( + encoding: str = "o200k_base", + tokens_per_image: int = 85, + tokens_per_file: int = 1000, +) -> None +``` + +Initialize the counter. + +**Parameters:** + +- **encoding** (str) – The `tiktoken` encoding to count with. The default, `o200k_base`, is what current OpenAI + models use. +- **tokens_per_image** (int) – Tokens to charge per image, which the tokenizer cannot measure. The default is what + OpenAI charges for a small image; raise it if you send large ones. +- **tokens_per_file** (int) – Tokens to charge per file. A rough stand-in for a short document, since the real + cost depends on the page count; raise it if you send long ones. + +**Raises:** + +- ImportError – If `tiktoken` is not installed. + +#### warm_up + +```python +warm_up() -> None +``` + +Load the encoder, downloading its vocabulary if it is not already cached. + +#### count + +```python +count(messages: list[ChatMessage], tools: ToolsType | None = None) -> int +``` + +Return the estimated number of tokens used by the given messages. + +**Parameters:** + +- **messages** (list\[ChatMessage\]) – The messages to measure. +- **tools** (ToolsType | None) – Tools whose schemas are sent alongside the messages, and so consume tokens too. + +**Returns:** + +- int – The estimated token count, or `0` when there is nothing to measure. + +#### to_dict + +```python +to_dict() -> dict[str, Any] +``` + +Serialize the counter. + +**Returns:** + +- dict\[str, Any\] – A dictionary representation of the counter. + +## types/protocol + +### TokenCounter + +Bases: Protocol + +Estimates the number tokens used by a list of messages. + +Implement `to_dict` so the counter's settings survive serialization. The default `from_dict` passes them straight +back to the constructor, which is enough for plain values; override it when `to_dict` emitted something that has to +be rebuilt first, such as a `Secret` or a nested component. + +#### count + +```python +count(messages: list[ChatMessage], tools: ToolsType | None = None) -> int +``` + +Return the estimated number of tokens in the given messages. + +**Parameters:** + +- **messages** (list\[ChatMessage\]) – The messages to measure. +- **tools** (ToolsType | None) – Tools whose schemas are sent alongside the messages, and so consume tokens too. Pass them to have + them counted; leave as None to measure the messages alone. + +**Returns:** + +- int – The estimated token count. + +#### to_dict + +```python +to_dict() -> dict[str, Any] +``` + +Serialize the counter to a dictionary. + +#### from_dict + +```python +from_dict(data: dict[str, Any]) -> TokenCounter +``` + +Deserialize the counter from a dictionary.