feat(api): add /v1/detokenize endpoint

[Dennisadira]

Summary

Closes #1649. Mirror of /v1/tokenize for the inverse direction: take a list of token IDs and return the detokenized text, requested by @benniekiss in the issue thread for "complete API workflow" use cases that need to turn token IDs back into text without local processing.

The proto/handler shape was discussed in #1649 (comment). @benniekiss reacted positively; landing this with the strict-mirror-of-tokenize precedent in mind. Happy to adjust if the proto naming or response shape should differ.

What's added

• Proto (backend/backend.proto): new Detokenize(DetokenizeRequest) returns (DetokenizeResponse) RPC, with DetokenizeRequest{repeated int32 tokens} and DetokenizeResponse{string content}. The Go bindings are regenerated by make protogen-go (gitignored as usual).
• llama.cpp backend (backend/cpp/llama-cpp/grpc-server.cpp): handler that calls common_token_to_piece per token and concatenates — the same primitive TokenizeString already uses internally at the same file.
• Other backends: inherit the default Unimplemented from pkg/grpc/base.Base — same pattern as Detect, Rerank, etc. Backends can opt in later.
• Go plumbing: pkg/grpc/{interface,server,backend,client,embed}.go + pkg/grpc/base/base.go updated alongside their TokenizeString counterparts.
• HTTP: POST /v1/detokenize in core/http/endpoints/localai/detokenize.go and core/http/routes/localai.go. Request {"model": "...", "tokens": [...]}, response {"content": "..."}.
• Auth: entry in RouteFeatureRegistry gated by the existing FeatureTokenize — no new feature flag.
• Discovery: added under ai_functions in the routes index.
• Swagger regenerated; authentication.md updated to list the new endpoint.

Test plan

• make protogen-go regenerates clean
• go build ./core/... ./pkg/grpc/... clean
• go vet ./core/... ./pkg/grpc/... clean
• go test -c -o /dev/null ./core/services/nodes/... clean (the existing testcontainers-based suite needs Docker; only updated the two interface mocks so the test package still compiles)
• make swagger regenerates with the new endpoint visible
• Manual round-trip: POST /v1/tokenize → POST /v1/detokenize returns the original text on a llama.cpp model

Assisted-by: Claude:claude-opus-4-7

[mudler]

this requires a test addition to our e2e-backend test suite where we exercise a mocked backend via api

[Dennisadira]

Done — added a Detokenize method to the mock gRPC backend and two e2e tests in the MockBackend suite (0024a9c): one that POSTs known token IDs and asserts a non-empty content response, and a round-trip that tokenizes first then detokenizes the returned IDs.

[Dennisadira]

Rebased onto current master (05e8e1e). The only conflicts were in the generated swagger files — upstream had added Diarization types; I merged both sets in alphabetical order. All other files applied cleanly. Ready for another look when you have a moment.

[Dennisadira]

Hi @mudler — just checking in on this one. The e2e mock backend tests you requested are in (commit 349c9d2), and the branch was rebased onto current master on May 21. Happy to address any further feedback whenever you get a chance to take another look.

[Dennisadira]

Rebased onto current master — branch now has 2 commits (the endpoint + e2e tests) cleanly on top of master with no unrelated changes.

Also checked: no other PR has landed a /v1/detokenize implementation in master. The feature is still unaddressed upstream (related open issue: #1649), so this PR is still relevant.

[Dennisadira]

Hi @mudler — just flagging that the e2e tests requested in the review have been added in the second commit (c168bcba). The test suite adds a Detokenize method to the mock gRPC backend and covers both a basic POST with known token IDs and a tokenize→detokenize round-trip.

Happy to make any further changes if needed.

[mudler]

sorry @Dennisadira for some reason I've lost the comments notifications and now the PR has conflicts

[Dennisadira]

Thanks for the rebase, @mudler 🙏 Merges cleanly now and the e2e mock-backend tests are in. Just waiting on CI / your merge — let me know if anything else is needed.

[benniekiss]

Hi all! Sorry for the ping, but I was wondering if there was anything left other than resolving merge conflicts to get this merged?

@Dennisadira, are you still able to see this PR through? If not, I am happy to try and pick up the work where you left off.

[Dennisadira]

Thanks for the nudge @benniekiss — yes, I'm still happy to see this through. Just rebased onto current master (90355cd4); the branch is conflict-free again.

Two conflicts, both mechanical:

• core/http/routes/localai.go — master added voice_profiles to the ai_functions discovery map; kept both entries.
• tests/e2e/mock-backend/main.go — master added Score / extractRouteHint / extractRouteLabel in the same spot as the mock Detokenize; kept both.

Two more things had drifted underneath the PR that merged cleanly but no longer compiled, worth flagging since they're easy to miss:

• pkg/grpc/client.go — the watchdog helper changed shape: wdUnMark is gone and the idiom is now defer c.wdMark()(). Detokenize still used the old wdMark() / defer wdUnMark() pair.
• core/backend/detokenize.go — trace.InitBackendTracingIfEnabled gained a second parameter (TracingMaxBodyBytes); now matches what tokenize.go does.

Verified locally after the rebase: go build and go vet clean across ./core/..., ./pkg/grpc/..., ./tests/e2e/...; core/http/auth and core/schema unit tests pass; the e2e suites compile. I couldn't run the mock-backend e2e specs locally (no Docker on this box), so CI will be the first real run of them.

Still two commits — the endpoint, and the mock-backend e2e tests added from the earlier review. @mudler let me know if anything else is needed.

[localai-org-maint-bot]

Reviewed the refreshed head db857dc620edd1c19d2442f5aa6d5acfd6760814. The endpoint is wired through the HTTP/auth/discovery surfaces, gRPC backend interface, llama.cpp implementation, Swagger/docs, and the requested mock-backend e2e coverage. git diff --check passes and the branch is based on current master. I found no blocking issue. Focused Go tests could not start in this fresh worktree because the repository does not contain the generated pkg/grpc/proto package; CI/local verification after make protogen-go remains authoritative. @mudler this looks good to merge once the fresh CI checks complete.

[Dennisadira]

Follow-up: I went back through the checks on the previous head and found lint was red (the other 8 were green). It was errcheck on this PR's own new lines:

pkg/grpc/client.go:436                        conn.Close       not checked
tests/e2e/mock_backend_test.go:400,423,449    Body.Close       not checked

Since .golangci.yml uses new-from-merge-base: origin/master, the surrounding pre-existing defer conn.Close() calls are baseline and don't fail, but the ones added here are held to the full linter set. Switched all four to the form already used elsewhere in the same files (defer func() { _ = conn.Close() }(), as at client.go:547 and mock_backend_test.go:257). Just pushed.

Verified locally with golangci-lint v2.11.4 (the version the lint workflow pins), running make lint after make protogen-go and stubbing core/http/react-ui/dist the same way the workflow does — 0 issues. I also confirmed the check is meaningful by reverting one fix and watching it fail again, rather than trusting a green run.

One thing that needs you @mudler: all 9 workflow runs on the new head are sitting at action_required, so none of them have actually started. They'll need approval before there's anything real to look at — in particular the mock-backend e2e specs have never run anywhere yet.

[Dennisadira]

Pushed 8db6ef6f. One real failure on the last head, and it was mine — the rest are infra. Splitting them out so the red X isn't misleading:

The genuine break: the Rust backend didn't compile.

error[E0046]: not all trait items implemented, missing: `detokenize`
  --> src/service.rs:72:1
72 | impl Backend for KokorosService {
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ missing `detokenize` in implementation

Adding the Detokenize RPC grows the tonic-generated Backend trait, and Rust has nothing to inherit a default from — every method must be listed explicitly. Go backends pick up the Unimplemented default from base.Base (pkg/grpc/base/base.go) and the generated C++/Python servicer bases default to UNIMPLEMENTED, which is why Build test and tests stayed green and only Rust broke. backend/rust/kokoros is the only Rust crate in the tree (one Cargo.toml, one impl Backend), so that's the full extent of it.

Fixed with Err(Status::unimplemented("Not supported")), matching how the same file already gates tokenize_string and ~20 other unsupported RPCs. That clears tests-kokoros and backend-jobs-singlearch-4 (-cpu-kokoros).

Verified locally rather than trusting CI: cargo check --release on the crate with rustup stable (1.97.1, same as Dockerfile.rust installs) and protoc, after replicating the Makefile's proto copy. Clean. I also reverted the fix and watched E0046 come back, to confirm the check was actually exercising the trait impl rather than passing vacuously.

The other five are not related to this diff:

• -cpu-llama-cpp, -gpu-nvidia-cuda-13-magpie-tts-cpp, -cpu-depth-anything-cpp — all three died pulling the base image: Get "https://registry-1.docker.io/v2/": context deadline exceeded
• backend-jobs-darwin (nemo) — ModuleNotFoundError: No module named 'maturin' in the darwin builder
• tests-vibevoice-cpp-grpc-transcription — suite timeout, 3600s, stalled in downloadFile in BeforeAll before the backend ever answered Health

I can't give you a master baseline for those: the path filters mean detect-changes / generate-matrix skip that whole matrix on master pushes, so those jobs simply don't run there. What I can say is that a detokenize RPC can't cause a registry timeout, a missing Python build tool, or a model-download stall. The ~21 cancelled jobs are fail-fast cascade from the above.

One ask: the runs need approving again. Every workflow on db857dc6 ended action_required, and the previous head only executed as run_attempt: 2 — so this push is sitting unstarted until you approve it.

Assisted-by: Claude:claude-opus-5