GigaToken: 1000x Faster Tokenization – How to Actually Use It

Here’s the end state you’re aiming for: one command, and you’re tokenizing text at multiple gigabytes per second on your own laptop. No install, no config, no waiting on pip to compile Rust.

GigaToken – released by Marcel Rød in late July 2026 and within 24 hours sitting at the top of Hacker News – is a Rust-based tokenizer that claims 500-1000x speedups over HuggingFace and 100x over tiktoken. Stanford’s Tatsu Hashimoto called itan order-of-magnitude tokenizer speedup. The r/LocalLLaMA thread lit up fast. This post walks you through actually running it, then explains where the marketing numbers stop matching reality.

The 30-second version

If you have uv installed, you don’t need to install GigaToken at all. Run this:

uvx gigatoken bench 'openai-community/gpt2' owt_train.txt --validate --doc-separator "<|endoftext|>"

Swap in any HuggingFace model repo name and any text file you’ve got lying around. You’ll see three numbers: MB/s for gigatoken, MB/s for HuggingFace, and the ratio between them. On a modern laptop you should see a three-to-four-digit multiplier. That’s the article, basically. The rest is why it works, when it doesn’t, and how to plug it into a real pipeline.

Why the existing tokenizers are the bottleneck nobody talks about

Tokenization is supposed to be the boring part. You take text, split it into subwords, map each subword to an integer ID, feed it to the model. According to the LoPT paper, this is typically done with BPE or WordPiece against a pre-trained vocabulary – well-understood algorithms that have been around for years.

But when you’re pretraining a model, tokenization can eat serious wall-clock time. HuggingFace Tokenizers is already Rust under the hood. Tiktoken is already Rust under the hood. Both are multithreaded. And they still get crushed here.

The speedup, per Rød’s own HN comment, comes from three places: replacing the regex engine used for pretokenization with hand-written SIMD, minimizing branches in the hot loop, and aggressive caching of pretoken-to-token mappings so repeated words are basically free lookups. He notes that caching is genuinely hard here because pretoken distributions are extremely long-tailed – the cache blows up fast if you’re not careful.

Installing it and running your first real benchmark

If you liked the 30-second version and want to actually use GigaToken in Python, install it normally:

pip install gigatoken

Then the minimum viable script – this is straight from the official README, adapted:

import gigatoken as gt

tokenizer = gt.Tokenizer("Qwen/Qwen3-8B")
file_source = gt.TextFileSource(
 ["owt_train.txt"],
 separator=b"<|endoftext|>"
)
tokens = tokenizer.encode_files(file_source)

Two things to notice. First, gt.Tokenizer accepts HuggingFace model names directly, so you don’t need to hunt down vocab files. Second – and this is the part every quick-start blog post glosses over – you’re using TextFileSource and encode_files, not passing a Python list of strings. That matters. A lot.

The gotcha every launch tweet skipped

The 1000x number is real, but it’s conditional. From the README, in the author’s own words: passing Python data structures through the API still incurs the overhead of reading from Python, and while you’ll still see faster performance, it won’t be the headline 1000x.

Watch out: If your dataset is already on disk as one or more text files with a document separator, feed the file paths directly to TextFileSource. The moment you’re doing tokenizer.encode([str1, str2, str3, ...]) from a Python list, you’re paying Python object-marshalling costs on every string, and the speedup collapses toward the HuggingFace baseline. This is a design choice, not a bug.

Two more edges worth knowing before you benchmark:

• SentencePiece tokenizers are the slowest. Per the README, SentencePiece-based tokenizers are not well optimized in Gigatoken yet. That includes the Llama, T5, and Gemma families. If your model uses SentencePiece, you’ll still see a speedup, just not the eye-popping one. BPE-based tokenizers (GPT-2, Qwen, most modern OpenAI-adjacent stuff) are where the numbers shine.
• macOS lies to you on the first run. The README explicitly warns: You might need to run your commands twice on macOS to get a good reading, because Gatekeeper does a one-time security scan of the Rust binary. If you benchmark once and see disappointing numbers on an M-series Mac, run it again before you tweet about it.

What the benchmarks actually look like

Here are the two headline runs from the official repo, plus one independent reproduction so you know what to expect on modest hardware (all figures from the July 2026 launch period – check the README for updates):

The independent KrabArena reproduction on a small 4-vCPU cloud VM landed at 26.2x over tiktoken – a fraction of the marketing number, but still a huge win for a $30/month box. Directionally the claims hold. The absolute speedup you personally see will scale with core count and CPU generation.

At the EPYC rate, you could tokenize all of Common Crawl – roughly 130 trillion tokens, often described as the entire scraped internet – in under 6.5 hours on one machine. That’s the line that made the launch go viral, and it’s not marketing fluff, it’s arithmetic on the throughput.

A real-world example: prepping a pretraining shard

Say you’re training a small model on a 40 GB slice of OpenWebText and you want tokenized shards saved to disk. With HuggingFace this is an overnight job. With GigaToken on any half-serious workstation, it’s coffee-break-length.

import gigatoken as gt
import numpy as np

tokenizer = gt.Tokenizer("openai-community/gpt2")

# Point it at your raw text files - do NOT read them into Python first
source = gt.TextFileSource(
 ["shard_00.txt", "shard_01.txt", "shard_02.txt"],
 separator=b"<|endoftext|>",
)

tokens = tokenizer.encode_files(source)
np.save("tokens.npy", tokens)

Notice what’s not in this code: no open(), no .read(), no for line in file. The Rust side is opening and streaming those files itself. That’s the whole point. The moment you insert a Python read loop, you’re back to being bottlenecked by Python.

Should you swap it in today?

Honest answer: it depends on what you’re doing.

If you’re training a model from scratch or re-tokenizing a large corpus, yes – the speedup pays for itself in hours of your life. If you’re building an inference-time API that tokenizes one prompt at a time, no. Tokenizing a 500-token prompt was never the bottleneck. You’d be optimizing something that already runs in microseconds.

The interesting middle ground is data preprocessing for RAG or fine-tuning. If you’re ingesting a few GB of PDFs or documentation nightly, GigaToken turns that from a batch job into a real-time job. Whether that unlocks anything new for you is a product question, not a tokenizer question.

FAQ

Does GigaToken produce the exact same token IDs as HuggingFace?

Yes – the README confirms outputs match HuggingFace Tokenizers exactly in tested settings. Your model weights and vocab stay identical when you swap it in.

What if I get weird results on a Llama or Gemma model?

If you benchmarked on a Llama-3 checkpoint and the speedup looked underwhelming – say, 8x instead of the advertised 1000x – that’s expected. Those models use SentencePiece, which (as covered in the gotcha section above) isn’t well optimized in GigaToken yet. Low double-digit multipliers on SentencePiece vocab, four-digit multipliers on GPT-2/Qwen BPE. Hardware matters too – see the 26.2x vs 1353x contrast in the table above.

Can I use this for inference-time tokenization in production?

You can, but it’s the wrong problem to solve. Single-prompt tokenization at inference is already sub-millisecond with tiktoken. GigaToken’s design assumes you’re throwing gigabytes at it and want the Rust runtime to own the I/O loop. For an API endpoint that tokenizes one user message, stick with what you have.

Next step: pick any HuggingFace tokenizer you actually use, grab a 100 MB text file, and run the uvx gigatoken bench command from the top of this post. See what number you get on your own hardware – that’s the only benchmark that matters.