Error 404 on input detections when using an openai server other than vLLM

[mdevino]

Describe the bug

When using either the completions or chat completions endpoints and the openai server configured is not a vLLM instance (e.g. Ollama), a request with input detections returns 404:

Request

curl --location 'http://localhost:8033/api/v2/text/completions-detection' \
--header 'Content-Type: application/json' \
--data '{
    "model": "qwen3:0.6b",
    "prompt": "I hate aliens",
    "detectors": {
        "input": {
            "hap": {}
        }
    }
}'

Response

{"code":404,"details":"tokenize request failed for `qwen3:0.6b`: unknown error occurred"}

This is caused because the /tokenize is invoked when there are input detections to gather usage data (.e.g here). However, this endpoint is not part of the openai API, it's strict to vLLM.

Discussion

I'm wondering what would be the best approach in this scenario. I've thought of two ideas, but both have limitations:

1. When invoking the /tokenize endpoint returns 404, return usage as empty. - the limitation with this idea is that it might not be obvious if no warning about this is provided (and I think there was an ongoing discussion to deprecate the warnings field in the orchestrator response.
2. Create an additional config to accept 404 responses from /tokenize - this would have the same behavior as the previous option, except that it would only happen if this additional config parameter would be set to allow such responses. The drawback here is that there would be an extra config for the orchestrator.

[RobGeada]

A possible third option might be to have a fallback tokenize option that, in the event of 404'd /tokenize, does a best effort approximation of token count. A dumb example would be to just split on whitespace and use that as the token count.

[mdevino]

I'm not sure if trying to approximate the token count would be a good idea.

Scenario:

1. User enters a prompt and gets input detections with guessed tokens.
2. User sends the same prompt, but relaxes detection threshols, so that there's no longer an input detection.

This could cause confusion (and even suspicion) if the token count is different in these scenarios.

[declark1]

Adding one more option to the list. For this input detection scenario, we could potentially integrate and use tiktoken (OpenAI's BPE tokenizer) to count tokens using encodings for one of the OpenAI models, e.g. o200k_base or cl100k_base.

https://docs.rs/tiktoken-rs/latest/tiktoken_rs/#counting-token-length
https://github.com/openai/openai-cookbook/blob/66b988407d8d13cad5060a881dc8c892141f2d5c/examples/How_to_count_tokens_with_tiktoken.ipynb