QWE AI Academy A matched pair of 5B models. Same architecture, same token budget, same Qwen3-style recipe. One saw a full educational web mix. The other saw only text filtered to U.S. Common Core K-5. On Grade-8 MathCAMPS, even at pass@1024, the filtered model solves fewer than half as many items as its twin. That ceiling is the whole experiment behind LittleLearner – paper, weights, and live demo that hit HN hard after the project page circulated.
If you train or fine-tune small models, this is the cleanest public sandbox for “did pretraining already draw the map?” You can chat in 30 seconds and copy the probes below.
You’re debugging a kid’s homework helper that invents the next unit
Picture this: you fine-tune a small model on elementary science so it can help 10-year-olds. It nails “how does rain form?” A parent then asks something one grade past the books you used – multi-step fractions with an extra constraint, or a modern physics term – and the answers turn into confident nonsense. Broken SFT? Or was the ceiling locked before you touched a gradient?
Researchers built the control. They filtered FineWeb-Edu into an 88B-token LittleCurriculum aligned to Common Core K-5, dropping concepts, facts, and vocab taught later. Fresh 0.6B / 1.3B / 5B runs from scratch, each with an Unfiltered twin on the same recipe. Paper: arXiv:2608.13545 (submitted 13 Aug 2026; MPI-IS, ELLIS Institute Tübingen, ETH Zürich).
HN’s shorthand was “fifth-grade genius.” Fair for in-scope fluency. Useless as a promise that scale or light post-training will invent missing school years.
Practical setup: live demo, then weights
Fast path – no install:
- Open the hosted 5B chatty demo from the project page (link above; URLs move – start there).
- Run a Grade-5 word problem, then a Grade-8-style multi-step problem in the same session.
- Optional: same prompts in any normal model side-by-side.
Weights sit under the littlelearner org (base, grpo, chatty, plus unfiltered twins). Chatty 5B sketch:
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "littlelearner/littlelearner-5b-chatty"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(
repo, torch_dtype="bfloat16", device_map="auto"
)
msgs = [{"role": "user", "content": "If Sarah has 12 apples and gives 5 to Tom, how many left?"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=128)
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))The catch is on the littlelearner-5b-chatty card: messages = user only. No system prompt. Default or custom system text drops accuracy – these checkpoints were SFT’d without one. On vLLM, set the stop_token_ids from the card; tokenizer defaults overrun the turn.
Pro tip: second process with the matching Unfiltered 5B. Same prompt, same seed if you can. Only the pretraining filter should differ.
Boundary probes you can copy this afternoon
| Probe type | Example prompt | What you’re measuring |
|---|---|---|
| In-scope floor | “How does rain form? Use short sentences.” | Basic science + style control |
| Boundary math | Grade-5 vs Grade-6/8 word problems (MathCAMPS-style) | Where accuracy falls off |
| Out-of-scope science | A concept taught after Grade 5, with and without “explain like I’m 10” | Confabulation vs refusal |
| Few-shot rescue | 3 worked Grade-8 examples + new Grade-8 question | Whether ICL invents missing procedures |
Paper baselines worth treating as ground truth before you trust a lucky sample:
- Scaling 0.6B→5B lifts K-5 a lot and nudges the immediate boundary (Grades 6-7). Grade 8 stays near the floor for LittleLearner.
- At 0.6B, LittleLearner can beat its Unfiltered control on pure K-5 math – capacity isn’t split across advanced topics. That specialty edge fades at larger sizes; Beyond-K-5 stays crushed either way.
- SFT+GRPO boosts in-scope hard. Give LittleLearner the same out-of-scope GRPO data that lifts Unfiltered and the Beyond-K-5 gap still doesn’t close.
- Hand-written CoT few-shots steer format. They don’t enable curriculum the pretraining filter never showed.
Qualitative checks in the paper (their Table 1) use simple gravity vs a famous quantum thought experiment – filtered model stays in fifth-grade register and confabulates; Unfiltered states the real setup. Use that pair if you want the authors’ demo script. Don’t stop there; the math cliff is sharper for measurement.
Skill order inside the model also refuses to copy the human syllabus. Sometimes multi-digit division lands stronger than the single-digit case that’s “supposed” to come first (MathCAMPS grade labels vs actual model ordering, paper §3.3.2). Boundary is real. Internal roadmap is not your pacing guide.
If your own fine-tunes feel stuck on topics the base never saw, this is the uncomfortable map: post-training mostly amplifies what’s already inside the prior.
Sandbox limits (skip the hype)
Filter is precision-first. Held-out checks: ~0% Beyond-K-5 retained, only ~35-42% of true K-5 documents kept (paper Figure 2 / §3.1.6). Clean experimental wall. Incomplete elementary corpus if you wanted a full K-5 library.
As of the release wave (project page + HF org at publish time of that wave), checkpoints are up; public datasets on the org page: 0. LittleCurriculum itself isn’t a one-click download for custom ablations – you’d re-run the multi-stage pipeline.
It’s 88B tokens of filtered educational text, not a child. No embodied years, no classroom social loop. Chatty 5B card numbers for context: ~5.04B params, 4096 context, custom 64k BPE; MathCAMPS K-5 about pass@64 78.7 / pass@1 31.4 after behavior SFT – solid in-scope, still a wall past it.
FAQ
Can I prompt a normal LLM to “answer like a fifth grader” and get the same effect?
No. Style prompts change surface form. LittleLearner never stored the later concepts. Failures are missing knowledge, not tone.
Is the 5B chatty model usable as a kids’ tutor today?
Straight K-5 facts and simple word problems: yes – stays in register and is pleasant. Curriculum-edge multi-step work or any post-Grade-5 term: it invents. Pair with retrieval or a stronger model past the wall. One parent-style check: feed the exact worksheet from the backpack, then ask “why does that rule exist?” one grade higher and watch the drop.
Does this prove post-training can never create new capabilities?
It proves that under the budgets and methods they ran (SFT+GRPO on their data mixes, plus the ICL setups), the pretraining filter dominated. Not a verdict on every future recipe. The authors built a sandbox so harder RL, continual learning, or discovery methods can attribute new skills cleanly – the prior is known. That’s the invitation. Try your method; the Unfiltered twin is the control.
Open the demo. Run Grade-5 vs Grade-8 math in one thread. Load chatty 5B with user-only messages and one few-shot rescue probe. Screenshot both. Hardest clean picture you’ll get this week of a pretraining ceiling that scale and light GRPO don’t erase.