QWE AI Academy Most of us were taught neural nets are black boxes of tangled polysemantic knots. Nick Cammarata’s essay Neural Networks Are Already Clean (Goodfire, ~Oct 15 2026) says the opposite for modern models: you can often read the parts directly. His X post sat around 218k views; LessWrong and Reddit threads treated it as potentially huge for alignment if the audits hold – with a mountain of mapping work still left.
Years of sparse-autoencoder tooling assumed you had to untangle first. If most neurons already have one main job, debugging, steering, and trust checks change. Below is a beginner path: what the claim actually is, how to probe a neuron with free tools under a clean-first assumption, and where residual messiness still bites.
What “clean” means (and what it doesn’t)
Cammarata (ex-OpenAI Circuits, Zoom In co-author) went back to the canonical “mess.” Turns out the famous cat-and-car unit is a V-shape detector – flames, water jets, pointed fox faces – with a negative pole on rounds. The dog-car story reads as shared pose generalization, not arbitrary high-dimensional packing. That reanalysis sits against the older Zoom In examples.
His 2023 OpenAI automated interpretability run labeled only ~1.7-2% of neurons as interpretable (neuron explainer writeup). Preliminary Qwen3-32B audits in the 2026 essay land differently: roughly 90% of neuron poles show one main job (often with small related offshoots), ~7% two related jobs, 2-4% still opaque – as of those audits, and similar on denser/MoE stacks up toward Kimi K3 scale. Modernized InceptionV1 (residuals + LayerNorm, same size/data) hits 99-100% easily readable last-layer units; the original sat near zero. Coarser concepts when you shrink capacity, but the legibility pattern holds across a wide size range.
What still looks polysemantic is often recursive generalization: a feature grows like tree branches. An “iy” detector expands toward “diy” then “do it yourself” (without needing the substring). Same motif showed up in Multimodal Neurons (2021). One learning pressure – generalization – instead of a separate superposition tax.
Picture a rainforest, not a knot board. Leaves and vines look chaotic from far away; once you track growth rings and branch order, the layout stops feeling random. That ecology frame is the real mental swap.
Pro tip: When a unit looks messy, check training-checkpoint growth or the weak-activation tail before you declare unrelated packing. Branches show up there first.
Step-by-step: inspect a neuron assuming cleanliness
You do not need Goodfire Silico (their agentic runner that plans experiments and maps circuits on large models) to begin. Public tools are enough. Goal: one feature, one coherent-job hypothesis, a steer test.
- Load an open model in TransformerLens or browse pre-computed activations on Neuronpedia. Start small – GPT-2-small, Gemma, or a Qwen variant if the host has it.
- Cache activations on a short, varied prompt set. Mid-layer MLP neuron or an SAE feature is fine. Sort top activating examples. Skip the noun-detector bias; plenty of units fire on punctuation or the token before a feeling/context lands.
- Write a one-sentence hypothesis. Test it: ablate or amplify, watch the output delta, read the negative pole, glance upstream/downstream. If it fans into related idioms (weak MAGA-hat acts on a Trump-ish unit), catalog the branches – don’t stamp “dirty.”
- Steer: add the direction into activations and generate. Behavior match?
Skeleton if you want live hooks:
from transformer_lens import HookedTransformer
model = HookedTransformer.from_pretrained("gpt2-small")
logits, cache = model.run_with_cache("The DIY kit worked better than...")
# Inspect neuron activations / top examples, then
# model.run_with_hooks(..., fwd_hooks=[(hook_name, steering_fn)])Conjecture → test → revise. Agents accelerate the loop; the first dozen passes are doable by hand. Understanding compounds like a crossword – once two neighbors lock in, the third option space shrinks fast.
Common pitfalls when you assume they’re clean
Thresholds lie. Weak activations drag in broad cousins (Titanic-ish unit → voyages, steam, crossings). Pick the rule before you score: strong acts only, or the full tail? Essay footnotes note strong-only samples can look cleaner – closer to ~97% non-arbitrary in the slices they checked – while tools that only look backward miss forward cues like “boat” after Titanic context.
Old vision feature-viz still reward-hacks (dogs as the easy path). Prefer natural top images over optimized dreams. SAEs can smear double-negative SwiGLU regimes that raw neuron reads catch; starting every project with a decomposition sometimes hides the motif you wanted. And scale still blocks full coverage: trillion-param maps stay preliminary (companion work called out as forthcoming). The 2-4% opaque bucket can move.
Architectural overflow plus weak tools made early InceptionV1 last layers look hopeless – not pure superposition law. Residuals and LayerNorm fixed legibility to 99-100% without growing the net; retrain the old recipe and you still land messier (~70% range in the essay’s contrast). Clean-first is a prior, not a guarantee on every checkpoint.
Clean-first vs superposition-first
| Approach | Old (superposition-first) | New (clean-first) |
|---|---|---|
| First step | Train SAE / decomposition | Native neurons + circuits |
| Default read | Arbitrary packing under low capacity | Broader concept or recursive branches |
| Tooling bias | Reconstruction objectives | Microscopes, steering, agentic normal science |
| Modern-model read | Low % “monosemantic” | ~90%+ one main job (as-of audits) |
SAEs still earn their keep for dense feature dictionaries. They are no longer the mandatory untangle gate. Direct study surfaces growth dynamics sooner. Closed models block weight peeks, but the prior still helps: expect legible structure and build evals/monitors that look for it rather than treating every failure as irreducible mush.
FAQ
Is every neuron now 100% human-interpretable?
No. Audits still leave a few percent opaque, and “one job” has fuzzy edges on offshoots. Clean enough for ordinary science to compound – not finished taxonomy.
Does this kill sparse autoencoders?
No. Use them when you need a wide dictionary or a reconstruction target. On a fresh open model, try native top-activation reads first; fall back if density or isolation demands it. A practical mix beats either religion. Double-negative SwiGLU-style messes are one place direct inspection still wins.
What should I change in LLM work tomorrow?
Opaque behavior? Probe mid-layer activations or a known hosted feature before you only prompt harder. Open weights: ten-line TransformerLens top-act check beats assuming superposition. Alignment or red-team passes: hunt recursive social-cue / feeling branches – they show up often and steer cleanly. Grab Neuronpedia hosted data or a tiny HookedTransformer script. Label three units with a growth-branch lens after you skim the essay. That single afternoon recalibrates how “illegible” the stack feels.
Open the essay. Pick one public model. Tag three neurons. Stop when the third hypothesis survives a steer test.