docs: sync Haystack API reference on Docusaurus

docs-website/reference/haystack-api/token_counters_api.md

@@ -0,0 +1,220 @@
+---
+title: "Token Counters"
+id: token-counters-api
+description: "Estimate how many tokens a conversation occupies, for features that need a size before sending it to a model."
+slug: "/token-counters-api"
+---
+
+
+## approximate_counter
+
+### ApproximateTokenCounter
+
+Bases: <code>TokenCounter</code>
+
+Estimates tokens from text length using a flat ratio of characters to tokens.
+
+## Usage Example:
+
+```python
+from haystack.dataclasses import ChatMessage
+from haystack.token_counters import ApproximateTokenCounter
+
+counter = ApproximateTokenCounter(chars_per_token=4.0)
+messages = [
+    ChatMessage.from_user("Hello, how are you?"),
+    ChatMessage.from_assistant("I'm good, thank you! How can I assist you today?")
+]
+token_count = counter.count(messages)
+print(f"Estimated token count: {token_count}")
+```
+
+#### __init__
+
+```python
+__init__(
+    chars_per_token: float = 4.0,
+    tokens_per_image: int = 85,
+    tokens_per_file: int = 1000,
+) -> None
+```
+
+Initialize the counter.
+
+**Parameters:**
+
+- **chars_per_token** (<code>float</code>) – How many characters to treat as one token.
+- **tokens_per_image** (<code>int</code>) – Tokens to charge per image, which has no text to measure. The default is what
+  OpenAI charges for a small image; raise it if you send large ones.
+- **tokens_per_file** (<code>int</code>) – Tokens to charge per file. A rough stand-in for a short document, since the real
+  cost depends on the page count; raise it if you send long ones.
+
+**Raises:**
+
+- <code>ValueError</code> – If `chars_per_token` is not positive.
+
+#### count
+
+```python
+count(messages: list[ChatMessage], tools: ToolsType | None = None) -> int
+```
+
+Return the estimated number of tokens the given messages occupy.
+
+**Parameters:**
+
+- **messages** (<code>list\[ChatMessage\]</code>) – The messages to measure.
+- **tools** (<code>ToolsType | None</code>) – Tools whose schemas are sent alongside the messages, and so consume tokens too.
+
+**Returns:**
+
+- <code>int</code> – The estimated token count, or `0` when there is nothing to measure.
+
+#### to_dict
+
+```python
+to_dict() -> dict[str, Any]
+```
+
+Serialize the counter.
+
+**Returns:**
+
+- <code>dict\[str, Any\]</code> – A dictionary representation of the counter.
+
+## tiktoken_counter
+
+### TiktokenCounter
+
+Bases: <code>TokenCounter</code>
+
+Counts tokens locally with `tiktoken`, OpenAI's byte-pair encoder.
+
+Counting is an estimate, and two limits are worth knowing before relying on it:
+
+- **It is text-only**, so images and files get the flat `tokens_per_image` / `tokens_per_file` estimate rather
+  than a real count.
+- **It is OpenAI's encoder.** Other providers tokenize differently, so expect the count to drift on them.
+
+## Usage Example:
+
+```python
+from haystack.dataclasses import ChatMessage
+from haystack.token_counters import TiktokenCounter
+
+counter = TiktokenCounter(encoding="o200k_base")
+messages = [
+    ChatMessage.from_user("Hello, how are you?"),
+    ChatMessage.from_assistant("I'm good, thank you! How can I assist you today?")
+]
+token_count = counter.count(messages)
+print(f"Token count: {token_count}")
+```
+
+#### __init__
+
+```python
+__init__(
+    encoding: str = "o200k_base",
+    tokens_per_image: int = 85,
+    tokens_per_file: int = 1000,
+) -> None
+```
+
+Initialize the counter.
+
+**Parameters:**
+
+- **encoding** (<code>str</code>) – The `tiktoken` encoding to count with. The default, `o200k_base`, is what current OpenAI
+  models use.
+- **tokens_per_image** (<code>int</code>) – Tokens to charge per image, which the tokenizer cannot measure. The default is what
+  OpenAI charges for a small image; raise it if you send large ones.
+- **tokens_per_file** (<code>int</code>) – Tokens to charge per file. A rough stand-in for a short document, since the real
+  cost depends on the page count; raise it if you send long ones.
+
+**Raises:**
+
+- <code>ImportError</code> – If `tiktoken` is not installed.
+
+#### warm_up
+
+```python
+warm_up() -> None
+```
+
+Load the encoder, downloading its vocabulary if it is not already cached.
+
+#### count
+
+```python
+count(messages: list[ChatMessage], tools: ToolsType | None = None) -> int
+```
+
+Return the estimated number of tokens used by the given messages.
+
+**Parameters:**
+
+- **messages** (<code>list\[ChatMessage\]</code>) – The messages to measure.
+- **tools** (<code>ToolsType | None</code>) – Tools whose schemas are sent alongside the messages, and so consume tokens too.
+
+**Returns:**
+
+- <code>int</code> – The estimated token count, or `0` when there is nothing to measure.
+
+#### to_dict
+
+```python
+to_dict() -> dict[str, Any]
+```
+
+Serialize the counter.
+
+**Returns:**
+
+- <code>dict\[str, Any\]</code> – A dictionary representation of the counter.
+
+## types/protocol
+
+### TokenCounter
+
+Bases: <code>Protocol</code>
+
+Estimates the number tokens used by a list of messages.
+
+Implement `to_dict` so the counter's settings survive serialization. The default `from_dict` passes them straight
+back to the constructor, which is enough for plain values; override it when `to_dict` emitted something that has to
+be rebuilt first, such as a `Secret` or a nested component.
+
+#### count
+
+```python
+count(messages: list[ChatMessage], tools: ToolsType | None = None) -> int
+```
+
+Return the estimated number of tokens in the given messages.
+
+**Parameters:**
+
+- **messages** (<code>list\[ChatMessage\]</code>) – The messages to measure.
+- **tools** (<code>ToolsType | None</code>) – Tools whose schemas are sent alongside the messages, and so consume tokens too. Pass them to have
+  them counted; leave as None to measure the messages alone.
+
+**Returns:**
+
+- <code>int</code> – The estimated token count.
+
+#### to_dict
+
+```python
+to_dict() -> dict[str, Any]
+```
+
+Serialize the counter to a dictionary.
+
+#### from_dict
+
+```python
+from_dict(data: dict[str, Any]) -> TokenCounter
+```
+
+Deserialize the counter from a dictionary.