Release: ttok 1.0

I released ttok 0.4, ran uv tool upgrade ttok, piped a file into the new version... and realized that it was defaulting to the GPT-4 tokenizer when it should very clearly default to GPT-5/GPT-6 instead!

I figured switching the default was a reasonable excuse to finally ship a 1.0.

OpenAI haven't actually confirmed that GPT-6 uses the same tokenizer as the GPT-5 family yet - there's an angry issue about it - but I found this commit by William Liu which reports on an experiment he ran confirming that the tokenizers are likely the same:

All seven GPT models (5.5, 5.6 Sol/Terra/Luna, 6 Astra/Sol/Luna) report 44,794 tokens and match each other on every one of the 31 fixtures. GPT-6 introduces no input-count change on this corpus.

Tags: projects, ai, openai, generative-ai, llms, tokenization

Release: ttok 0.4

ttok is my CLI tool for counting tokens, using OpenAI's open source tiktoken library.

It hasn't been in updated in a couple of years, but I finally fixed a Click warning, updated CI, and added a --list-models command to list available models.

It works with uvx, so you can count tokens in anything like this:

cat file.txt | uvx ttok

Tags: projects, ai, openai, generative-ai, llms, tokenization

Sam Rose explains how LLMs work with a visual essay

Sam Rose is one of my favorite authors of explorable interactive explanations - here's his previous collection.

Sam joined ngrok in September as a developer educator. Here's his first big visual explainer for them, ostensibly about how prompt caching works but it quickly expands to cover tokenization, embeddings, and the basics of the transformer architecture.

The result is one of the clearest and most accessible introductions to LLM internals I've seen anywhere.

Tags: ai, explorables, generative-ai, llms, sam-rose, tokenization