refactor(tokenization): drop dead offsets return from Render

[mayur-tolexo]

What

Removes the []types.Offset return value from the Tokenizer.Render(string) interface method and its implementation.

Why

Since the move to the renderer service (RenderChatCompletion RPC), Render has always returned a nil offset slice — the underlying RenderCompletion RPC doesn't produce character offsets. Every caller already throws it away with _, so the return value is dead weight that just makes the signature noisier and invites confusion about whether offsets are ever populated.

Closes #462.

Changes

• Interface Tokenizer.Render is now Render(string) ([]uint32, error).
• UdsTokenizer.Render drops the always-nil offset return; doc comment updated to note the renderer service doesn't produce offsets.
• Updated the single production caller (pool.go), the MockTokenizer, and the unit + e2e tests.

Encode is intentionally left alone — it still returns real offsets from the Tokenize RPC and the e2e tests assert on them.

Testing

• go build ./... and go vet ./... clean
• gofmt -l clean
• go test ./pkg/tokenization/... passes

No behavior change — this is a pure signature cleanup.

[github-actions]

Unsigned commits detected! Please sign your commits.

For instructions on how to set up GPG/SSH signing and verify your commits, please see GitHub Documentation.

[Copilot]

Pull request overview

This PR refactors the pkg/tokenization.Tokenizer API by removing the unused []types.Offset return value from Render(string), since offsets are never produced by the renderer service and all callers already discarded them. This reduces signature noise without changing runtime behavior.

Changes:

• Updated Tokenizer.Render to return only ([]uint32, error) and adjusted UdsTokenizer.Render accordingly.
• Updated the production call site in the tokenization pool to match the new signature.
• Updated mocks and unit/e2e tests to remove offset handling for Render while leaving Encode offsets intact.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
tests/e2e/uds_tokenizer/uds_e2e_test.go	Updates e2e tests to use the new Render(prompt) ([]uint32, error) signature.
pkg/tokenization/uds_tokenizer.go	Removes the dead offsets return from UdsTokenizer.Render and updates its doc comment accordingly.
pkg/tokenization/uds_tokenizer_test.go	Updates unit tests to match the new Render signature and removes the offsets assertion.
pkg/tokenization/tokenizer.go	Updates the Tokenizer interface to drop the offsets return from Render.
pkg/tokenization/pool.go	Updates the pool’s Render call site to match the new signature.
pkg/tokenization/pool_test.go	Updates the mock Tokenizer and related tests to return only tokens + error for Render.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[yankay]

HI @mayur-tolexo

Thanks, this cleanup makes sense to me. Since this changes a public Go API signature, could we mention it in the release notes as a breaking API cleanup?

I checked llm-d-router: the default vLLM HTTP render path is not affected, but the deprecated UDS adapter still uses the old three-return-value UdsTokenizer.Render call and will need a small compile-time update when router bumps kv-cache:
https://github.com/llm-d/llm-d-router/blob/main/pkg/epp/framework/plugins/requestcontrol/dataproducer/tokenizer/uds.go#L55

/ok-to-test

[mayur-tolexo]

HI @mayur-tolexo

Thanks, this cleanup makes sense to me. Since this changes a public Go API signature, could we mention it in the release notes as a breaking API cleanup?

I checked llm-d-router: the default vLLM HTTP render path is not affected, but the deprecated UDS adapter still uses the old three-return-value UdsTokenizer.Render call and will need a small compile-time update when router bumps kv-cache: https://github.com/llm-d/llm-d-router/blob/main/pkg/epp/framework/plugins/requestcontrol/dataproducer/tokenizer/uds.go#L55

/ok-to-test

Thanks @yankay. Agreed it's a breaking API change - Render is exported, so importers need a compile-time update — though it's source-only, no behavior change. Could you flag it as a breaking cleanup in the release notes on your side?

[sagearc]

/lgtm

[vMaroon]

lets hold with this, we're bringing them back:

• [API] Add token offsets to render endpoints (/v1/.../render) vllm-project/vllm#44226