Search results

FR: Add GPT-5 Tokenizer

Status: Open.

#41 In dqbd/tiktokenizer;

· lcarnegie opened on Nov 5, 2025

FR: Add Qwen3 tokenizer

Status: Open.

#39 In dqbd/tiktokenizer;

· gemnioo opened on Apr 29, 2025

Latest Next.js Issue: tiktoken doesn't work in App Router.

Status: Open.

#37 In dqbd/tiktokenizer;

· alfonsusac opened on Apr 4, 2025

website can not visit

Status: Open.

#36 In dqbd/tiktokenizer;

· 9a-aaaaaaaa opened on Mar 30, 2025

Failure to tokenize after " 私" for Meta-Llama-3-8B

Status: Open.

#32 In dqbd/tiktokenizer;

· davidquarel opened on Jan 21, 2025

Can we automatically add input templates for our models?

Status: Open.

#28 In dqbd/tiktokenizer;

· tcxia opened on Jul 25, 2024

TOKENS 切割功能

Status: Open.

#22 In dqbd/tiktokenizer;

· EllisMorrow opened on Apr 19, 2024

Mind if the OpenAI Cookbook repo points to your Tiktokenizer app?

Status: Open.

#13 In dqbd/tiktokenizer;

· ted-at-openai opened on Jul 24, 2023

[bug] When I use a custom name and then change my mind and try to use one of the default ones, it doesn't update in the textarea below

Status: Open.

#11 In dqbd/tiktokenizer;

· Evertt opened on Jul 3, 2023