Skip to content

feat(tokens): add in-process text token counting - #11247

Closed
localai-org-maint-bot wants to merge 1 commit into
masterfrom
maint/issue-9537-token-counter
Closed

feat(tokens): add in-process text token counting#11247
localai-org-maint-bot wants to merge 1 commit into
masterfrom
maint/issue-9537-token-counter

Conversation

@localai-org-maint-bot

Copy link
Copy Markdown
Collaborator

Summary

  • add a public pkg/tokens helper for approximate in-process token counting
  • resolve known tiktoken model encodings with a conservative cl100k_base fallback
  • count plain and multimodal text content while rejecting unsupported shapes
  • promote tiktoken-go to a direct dependency and add hermetic Ginkgo coverage

Testing

  • go test -count=1 ./pkg/tokens
  • go vet ./pkg/tokens
  • gofmt -l pkg/tokens/*.go
  • git diff --check origin/master...HEAD

Documentation

No user-facing runtime behavior or configuration surface changes; the new API is documented with Go comments.

Closes #9537

Assisted-by: Codex:gpt-5
@mudler

mudler commented Jul 31, 2026

Copy link
Copy Markdown
Owner

@localai-org-maint-bot code isn't used. I don't like to introduce code which is not used

@mudler mudler closed this Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RFC: Add pkg/tokens — in-process token counter helper (tiktoken-go wrapper)

2 participants