QWE AI Academy Two ways to react to the Habsburg-jaw frog benchmark that just hit Hacker News: laugh at the screenshots and close the tab, or copy the format for yourself. The first is more fun for ten minutes. The second gives you a repeatable way to sanity-check every new model release without reading a single MMLU chart.
This tutorial goes with option two – but only after covering why the specific prompt works, where it breaks, and how to keep it useful once the labs notice.
Why anyone cares about a frog with a chin
One prompt. Every model. Three tries a month. August 2026: 14 models, 42 runs, 42 produced an SVG (frogs.vaguespac.es). No leaderboard math, no ELO, no vibes-based star ratings. You just look at the frogs.
The direct ancestor is Simon Willison’s older experiment. On October 25, 2024, Willison introduced the pelican-on-a-bicycle test – exact prompt: “generate an SVG of a pelican riding a bicycle” – and ran it against 16 models from OpenAI, Anthropic, Google, and Meta (simonwillison.net). The frog swaps the pelican for something with a specific anatomical requirement. A jaw the model has to actively deform, not just draw.
That one-sentence prompt tests a lot at once: instruction-following, syntactically valid SVG output, object recognition, spatial reasoning, composition, and restraint. No image generator, no diffusion – just text tokens that have to compile into a working vector file. Think of it like a unit test that takes 30 seconds to run and produces something you can paste into Slack.
The reader scenario: you’re picking a model this week
Three models. Marketing pages all say SOTA. You have twenty minutes.
HumanEval requires a use you don’t have set up. Your own code gives results your teammate won’t trust. A single silly-SVG prompt lands differently: 30 seconds per model, visual output that’s self-explanatory, and – if you pick your own weird prompt – almost certainly not memorized verbatim.
Don’t copy “Habsburg jaw frog” for your own testing once it’s viral. Pick something equally specific but yours – “an SVG of a giraffe wearing sunglasses on a unicycle,” “an octopus playing chess.” Change the animal, the anatomy quirk, or the impossible action. The whole diagnostic depends on the prompt being rare in training data.
How to run it yourself
Open three chat windows – Claude, ChatGPT, Gemini. Paste the same prompt. Save the raw SVG text to three files. Open in a browser.
<!-- save as frog-claude.html -->
<!DOCTYPE html>
<html><body style="background:#222">
<!-- paste model output below -->
<svg ...>...</svg>
</body></html>No Python, no API keys. If the model wrapped the SVG in a code fence, strip it. If it added commentary above or below the code, delete that too – but note it, because “couldn’t just return the code” is itself a signal.
Score by asking three questions
- Does it render? Blank page = malformed or truncated SVG. That’s a fail before you even look at frog anatomy.
- Is it a frog? Green blob with eyes on top counts. Feathers or four upright legs don’t.
- Did it try the jaw? Protruding lower mandible = pass. Generic cartoon frog mouth = the model ignored the specific instruction and pattern-matched on “frog.”
The prompt design lesson buried in the joke
Why “Habsburg jaw” instead of “sad frog” or “deformed frog”? The Habsburg dynasty’s centuries of intermarriage produced a well-documented cluster of physical traits – most famously mandibular prognathism, the protruding lower jaw now called the Habsburg jaw. That specificity forces the model to: (a) know the historical reference, (b) translate mandibular prognathism into vector geometry, and (c) apply it to non-human anatomy.
A vague prompt lets the model retrieve a cached frog. A weirdly specific prompt forces real composition. That’s the whole design principle worth stealing for your own benchmarks.
What the results actually mean
| Observation | What it probably means |
|---|---|
| Clean, closed SVG that renders | Model handled long-form structured output without truncation |
| Recognizably frog-shaped | Reasonable visual concept mapping |
| Jaw is genuinely exaggerated | Model parsed the specific modifier, not just “frog” |
| Model refused or hedged | Alignment tuning tripped on “Habsburg” as historical/genetic content |
| Output includes commentary explaining the joke | Instruction-following is weak – you asked for code, got an essay |
The site also lets models annotate their own outputs. Per the site copy, those annotations include some editorializing – phrases like “massive protruding mandible” and “recessed upper lip.” Treat a model’s self-description as a second signal: the one that describes the anatomy correctly usually drew it best.
Honest limitations
1. The output token trap
SVGs run long. A detailed frog can hit thousands of tokens of path data. In one recorded run, Claude Sonnet 5 burned its entire 64K output budget and still never closed the SVG. Another run hit a 16K cap with the same result – unclosed tag, blank render (per the Playcode MacBook SVG benchmark). As of August 2026, Claude Sonnet 5 supports up to 128K max output tokens, but that ceiling doesn’t prevent truncation if a lower cap is set by the API caller.
If your frog renders blank, check whether the last character is </svg> before concluding the model can’t draw frogs. Retry with a higher max_tokens or a simpler prompt asking for fewer path details.
2. Adaptive thinking eats your budget invisibly
This one shows up on the invoice, not the screen. Claude Sonnet 5 has adaptive thinking on by default (as of August 2026, per Anthropic’s documentation). Extended thinking means the model works through the problem internally before producing a visible response – and those thinking tokens are billed separately. They don’t appear in the SVG you receive. A frog that looks cheap in character count can cost several times more than expected once thinking tokens are included. If you’re running many models in a batch comparison, check your billing dashboard, not just the response length.
Why does this happen? Extended thinking is a separate compute pass. The architecture keeps it hidden from the output stream by design – useful for users who don’t want reasoning noise, but it means the token meter is running somewhere you can’t see.
3. Once it’s viral, the benchmark starts dying
Here’s where it gets uncomfortable. Turns out Dylan Castillo tested this at scale: he generated 1,008 SVGs across 7 frontier models specifically to check whether AI labs train on benchmark prompts. His July 2026 conclusion was careful but pointed – the more plausible story for the leader’s advantage is SVGmaxxing at Google/DeepMind, and other labs might be doing it more quietly. The experiment can’t say who, only that the pattern is there (dylancastillo.co, July 2026).
The frog benchmark is fresh as of August 2026, when the story hit HN. That won’t last. Which is exactly why running a private prompt nobody else has seen matters more than which model wins the public leaderboard this month.
4. The judging problem
No correct-answer key exists for “frog with Habsburg jaw.” The public site uses model self-annotation, and those annotations editorialize. Formal academic SVG benchmarks like UniSVG use CLIP scoring, SSIM, and human evaluation to sidestep this – on that benchmark, Claude 3.7 led non-finetuned baselines with a score of 0.722 (as of the paper’s publication). For your five-minute Slack test, you’re the judge. Own that. Don’t pretend it’s objective.
FAQ
Is this actually a benchmark or just a joke?
Both. Non-quantitative, easily gamed – but the prompt genuinely stresses syntax, reasoning, instruction-following, and composition at once. Smell test, not leaderboard.
Which model wins on the current site?
Go look – results shift monthly and a screenshot I include today will be wrong by next week. What I’d say instead: don’t pick a model based on one frog. Run the frog prompt as one of five weird prompts you designed yourself, score them together. A model that nails the frog but fumbles your other four isn’t the model you want for your actual work. The single-prompt winner changes; the model that handles all five consistently is the interesting finding.
Can I use DALL-E or Midjourney for this?
No – and that’s the whole point. Image generators would trivially produce a frog with a big jaw by drawing pixels. This test is specifically about whether a language model can express visual composition through structured text output. Sending it to an image generator isn’t a shortcut; it’s a different test entirely.
Your next 15 minutes
Pick something weird that isn’t the frog and isn’t the pelican. “An SVG of a hedgehog conducting an orchestra.” “An octopus wearing a tie.” Write it in a text file – that’s your private eval now. Run it against whichever three models you use most this week. Save the outputs. Run it again in three months. That’s a real benchmark. No lab can train on it because nobody else has seen it.