Issues

• FR: Add GPT-5 Tokenizer

  dqbd/tiktokenizer#41 · Filter by author lcarnegie opened on Nov 5, 2025
  1

• FR: Add Qwen3 tokenizer

  dqbd/tiktokenizer#39 · Filter by author gemnioo opened on Apr 29, 2025

• Latest Next.js Issue: tiktoken doesn't work in App Router.

  dqbd/tiktokenizer#37 · Filter by author alfonsusac opened on Apr 4, 2025

• website can not visit

  dqbd/tiktokenizer#36 · Filter by author 9a-aaaaaaaa opened on Mar 30, 2025

• Failure to tokenize after " 私" for Meta-Llama-3-8B

  dqbd/tiktokenizer#32 · Filter by author davidquarel opened on Jan 21, 2025

• Can we automatically add input templates for our models?

  dqbd/tiktokenizer#28 · Filter by author tcxia opened on Jul 25, 2024
  1

• TOKENS 切割功能

  dqbd/tiktokenizer#22 · Filter by author EllisMorrow opened on Apr 19, 2024

• Mind if the OpenAI Cookbook repo points to your Tiktokenizer app?

  dqbd/tiktokenizer#13 · Filter by author ted-at-openai opened on Jul 24, 2023
  3

• [bug] When I use a custom name and then change my mind and try to use one of the default ones, it doesn't update in the textarea below

  dqbd/tiktokenizer#11 · Filter by author Evertt opened on Jul 3, 2023