Skip to content
ChatGPT & LLMs

Tokenization Guide: Why LLMs Fail Simple Tasks

Tokenization turns text into LLM tokens. Most guides skip the gotchas that break counting, math, and costs. Here's the practical fix from real tests.

5 min readBeginner

Tokenization Is Why Your LLM Looks Dumb

Most people treat tokenization like plumbing. Boring, necessary, ignore it. I used to. Then I watched GPT-4 confidently claim “strawberry” has two R’s. Three times. That was the moment I stopped blaming the model and started staring at the tokens.

Models never see letters. Your string gets chopped into chunks, each chunk becomes an integer ID, and only those IDs go forward. Next-token prediction runs on numbers. Hide the letters inside opaque subwords and counting is pure memorization, not inspection.

Tutorials stop at definitions and pretty “Hello world” lists. They skip the daily failures: long numbers that fragment, non-English text that burns context, one trailing space that rewrites a few-shot label, boundaries that shift after special tokens.

The Inspection-First Approach to Tokenization

Rule I actually follow: never ship a prompt you haven’t tokenized yourself. Tokenizer first. Model second.

Install tiktoken – OpenAI’s fast open-source BPE library. Run critical strings through it. Current GPT-4o-class models map to o200k_base (~200k vocab) as of 2025; earlier GPT-4 / 3.5-class stacks used cl100k_base (~100k). Encodings change – confirm with the docs before you bill a batch. The tiktoken repo makes it a one-liner:

import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o") # or get_encoding("o200k_base")
ids = enc.encode("strawberry")
print(ids)
print([enc.decode([i]) for i in ids])
print(len(ids))

Decode the IDs back to pieces. You’ll see subword chunks, not characters. The model receives a handful of opaque integers. No standalone “r”. That’s the strawberry failure mode community tests keep hitting on current encodings.

Spaces travel with words. “dog” and ” dog” are different tokens. Capitalization flips IDs too. OpenAI’s help center states this outright; tokenizer UIs make it obvious in color. One stray space after a system message can move every later boundary – and still get billed.

Why BPE Became the Default (And What It Still Misses)

Byte-Pair Encoding, adapted for subwords by Sennrich, Haddow and Birch in 2016 (arXiv:1508.07909). Start from bytes or characters. Merge the most frequent adjacent pair. Repeat until you hit a target vocab size.

You get something between pure characters (huge sequences) and pure words (giant vocab, constant unknowns). Hugging Face’s LLM course walks the full pipeline if you want the architecture view: tokenizers chapter.

Modern OpenAI vocabs sit near 100k-200k entries. o200k_base roughly doubled cl100k. Multilingual text improves a bit. English still gets cheap single-token packing; other scripts pay a fertility tax – more tokens per word, less effective context, higher bills. Long numbers split into inconsistent multi-digit pieces, so arithmetic loses clean place-value signals and falls back to pattern matching.

Pro tip: Always decode IDs back to strings and print them beside the original. The visual mismatch is where the bugs live. One afternoon of this beats a week of prompt wrestling.

Real Prompt That Broke Until I Looked at the Tokens

Classification prompt. Perfect in the playground. Dead in production. Diff? A trailing newline plus a space my joiner added between few-shot examples. After tokenization that space glued onto the next label – expected ” B” became bare “B” after a boundary the model barely ever saw. Accuracy dropped. Delete the invisible character, problem gone.

Same family of landmine: JSON mode, code fences, anything where whitespace after special tokens or trailing spaces rewrite the next ID. Assumptions about few-shot formatting and even KV-cache-friendly prefixes quietly rot.

Quick Wins Once You Can See the Tokens

The catch is boring and mechanical:

  • Strip or normalize trailing whitespace before every API call.
  • Use newer encodings first (o200k vs older) for heavy non-English work – still English-biased, just less painful.
  • Count tokens with the exact encoding for your model before you estimate cost or context headroom. OpenAI’s rough rule – 1 token ≈ 4 characters ≈ ¾ of a word for common English; 100 tokens ≈ 75 words – lives on their tokens help page. English-only. Measure real samples.
  • Letter-level or precise string tasks: force spell-outs or tools. Don’t trust raw generation to “see” characters it never received.

Will models go fully byte-level or character-level and ditch these quirks? Some research tries. Production stacks still tokenize first. Ignoring that layer is optional. Paying for it isn’t.

Open the OpenAI tokenizer tool or fire up tiktoken. Paste the last prompt you sent. Look at the colored chunks. Start there.

FAQ

How many tokens is a typical English sentence?

Use OpenAI’s estimate: about 4 characters per token, or ~75 words per 100 tokens for common English. Measure your exact string with the model encoding. Guesses lie.

Why does the same text cost more in other languages?

Vocabs grew up on English-heavy data. Common English subwords pack tight; other scripts and morphologies fragment. More tokens for the same meaning → shorter usable context and a bigger bill. I once compared a support macro in English vs the same intent in a morphologically rich language: the second ate context like a leak. Newer large vocabs (o200k-class) help. They don’t erase the gap. Test fertility on real user samples per language before you price a global feature.

Do I need to retrain a tokenizer for my domain?

No. Stock tokenizer is fine for most apps – inspect, then adapt prompts. People jump to a custom vocab too early. Domain tokenizers (code, medical, low-resource languages) earn their keep when you’re pre-training or doing heavy fine-tunes and fertility is actually killing throughput. Even then: measure how the existing encoding treats your data first. Building a new one is a last resort, not a badge.

Related Tutorials