QWE AI Academy George Hotz published “I love LLMs, I hate hype” on July 12, 2026. By the next morning it had 276+ points on Hacker News and a comment section that was mostly people agreeing loudly with slightly different versions of the same frustration. His actual complaint is narrower than the title suggests: he calls out “negative valence hype” – the window-closing, perpetual-underclass framing – and the strawman leap from “fancy autocomplete” all the way to “owning the light cone.”
The post isn’t a tutorial. But it points at a skill worth having, especially if you’re new to LLMs and every week brings another launch post claiming the previous best model is now obsolete: how do you tell real progress from marketing? This guide is that tutorial. Not a recap of the drama – a repeatable routine you can run every time a new model drops.
Why the post struck a nerve
A top comment in the HN thread nailed the pricing mechanic: at $100-$200/month for a bounded token budget, frontier models are a no-brainer for personal use – but at raw token rates, often 10x to 100x higher for the same models, the economics flip completely. That gap between demo price and production price is where a lot of hype hides.
The other reason it landed: benchmarks stopped being useful the moment everyone started teaching to the test. Is that getting worse, not better? That’s an open question – but the data suggests it’s been broken for a while already.
The core skill: separating capability from claim
Every LLM claim comes in three flavors. Spot them and you’ve already won half the battle.
- Benchmark claim: “Model X scored 92% on MMLU.” Sounds objective. Often isn’t (more on why in Step 2).
- Vibe claim: “It feels smarter.” Real signal, but personal and hard to verify.
- Existential claim: “This changes everything / you’ll be left behind.” This is the one geohot is punching at.
When you see a claim, ask which flavor it is before you react. Existential claims deserve the most skepticism and the least of your emotional bandwidth.
Your 5-minute hype filter
A routine, not a philosophy. Run it whenever a shiny new model shows up in your feed.
Step 1 – Check whose benchmark it is
If the only source for “state of the art on X” is the lab that trained the model, downgrade it in your head. Frontier models now cluster above 88% on MMLU (per Label Studio’s LLM evaluation guide, as of early 2026). When everyone gets an A, the test stopped mattering. A self-reported top score is the academic equivalent of grading your own exam.
Step 2 – Look for the contamination footnote
Turns out, 29.1% of MMLU test items showed signs of contamination – meaning models likely saw the answers during training, per a NAACL 2024 measurement by Johns Hopkins researchers. Apply an inference-time decontamination technique and 22.9% of the inflated score disappears. A model bragging about MMLU without addressing this is either naive or hoping you are.
There’s a subtler version of the same problem: the pretraining corpus can be genuinely clean, but teams train dozens of checkpoints and ship whichever one posted the highest MMLU. No cheating, technically – but the benchmark number was still the selection filter, per llm-stats.com’s contamination guide.
Step 3 – Compare against a private-set number
73.4%. That’s what GPT-4o scored on MMLU-CF – Microsoft’s privately rebuilt test set, not the public one. Lower than its public MMLU score. When a private-set benchmark exists for the same model, prefer it. If none exists, note that gap and move on.
Step 4 – Do the token math, not the subscription math
Ask: “What does this cost if I ship it to 10,000 users, not if I use it personally?” The answer is almost always a different order of magnitude than the landing page price suggests.
Step 5 – Run your own three prompts
Keep three prompts you’ve written yourself – from your actual work – that no model could have memorized. A weird edge case from your job. A paragraph you actually wrote, asking for a rewrite. A debugging question about your own code. Run them against every new model. Mine took about 20 minutes to write once and I’ve reused them for six months. First time I ran them on a “revolutionary” new model, it fumbled the code question worse than the previous version. That was more useful than any benchmark number I’d read that week.
The signal comes from consistency, not cleverness. Write your three prompts once and don’t change them for at least six months. If model B clearly answers your unchanged prompts better than model A did in January, that’s real progress. Everything else is marketing.
Three traps beginners keep falling into
Trap 1 – Trusting the leaderboard rank. As of early 2026, there are 239 models on major leaderboards – and nearly 30% of teams don’t test their models at all before deploying (Pranava Kailash, Incremys, 2026). The rank is a starting hypothesis, not a verdict. The contamination stats from Step 2 explain why the number itself can be gamed even without obvious cheating.
Trap 2 – Confusing feel with fit. A model can feel brilliant in a five-minute chat and be useless for your actual task. Public leaderboards measure capability in a vacuum – a model that aces quantum physics questions (illustrative example) might still botch a standard meeting summary. Your three-prompt personal test catches this in about ninety seconds.
Trap 3 – Falling for negative-valence hype. “You’ll be left behind” is engineered to bypass your evaluation brain and go straight for your gut. When a post is trying to make you feel bad rather than tell you something true – skip it. That’s the whole thesis of geohot’s post, distilled.
Hype filter vs. official benchmarks vs. gut feel
Side by side:
| Method | Speed | Reliability | What it misses |
|---|---|---|---|
| Reading official benchmarks | Slow (need context) | Medium – inflated by contamination | Real-world fit for your task |
| Gut feel from a demo | Fast | Low – cherry-picked | Cost, edge cases, boring tasks |
| 5-minute hype filter (this guide) | ~5 min | High for personal use | Doesn’t generalize to other people’s tasks |
The filter isn’t better than a proper eval suite. It’s better than the two things you were probably doing before – which is enough.
An honest admission
I don’t know whether the current wave of models will keep improving at this pace or plateau by next spring. Nobody does. The reason to build a hype filter isn’t that you’ll always get the right answer – it’s that you stop outsourcing the answer to whoever writes the loudest launch post. A cheap, repeatable habit that keeps working when the labels on the boxes keep changing. That’s the thing worth stealing from geohot’s post, whatever you think of his other opinions.
Frequently asked questions
Is geohot’s post anti-AI?
No. He’s explicitly enthusiastic about LLMs, coding agents, and self-driving. The complaint is about the framing, not the technology.
If MMLU is contaminated, which benchmark should I actually trust for picking a model?
Trust a mix. A reasonable starting point: one public leaderboard for a shortlist, then a private-set version if it exists (MMLU-CF is the clearest example for GPT-4o – its 73.4% private score vs. the higher public number shows exactly how much inflation to expect). Then run your own three prompts against the top two candidates. The private-set gap tells you how inflated the public score is for that specific model. Your prompts tell you which one is actually better at your work. Label Studio’s explainer on why static benchmarking and real evaluation are different disciplines is worth ten minutes if you want to go deeper.
Do I need any of this if I’m just using ChatGPT to write emails?
Honestly? No. Casual, personal use – ignore the leaderboards entirely and pick based on price and what feels right. The hype filter matters when you’re about to spend real money, build a product, or make a career move based on a claim. Below that bar, skip it. Don’t turn a five-minute habit into a thirty-minute obsession for something that doesn’t warrant it. The filter is a tool, not a ritual.
Next action: Open a notes file right now and write down your three personal prompts. Bookmark this page. Next time a model launches, come back, run the five steps, and see if you feel any calmer about it than you did today.