Astra Reads Qwen Weights: How to Apply the Fix

End state you want: a local Qwen3.5-0.8B that stops eating the 32k token budget on math, finishes answers, and scores far higher on MATH-500 – without a fine-tune run. That’s the practical payoff behind the line people repeated this week: Astra can read and write weights to improve Qwen.

MIT CSAIL’s Nov 2, 2026 post (Takehi, Teoh, Zhang, Khattab) is where the numbers live. GPT-6 Astra changed 131,072 weights (0.016%) of Qwen3.5-0.8B – no gradients, no training loop. Per their Experiment 8 table: MATH-500 7.9% → 48.1%, GSM8K 13.8% → 67.1%, IFEval 21.0% → 62.3%, token-limit hits 93.0% → 33.1% at the 50-pass edit. HN’s read was blunt: this is closer to ablating a bad direction than teaching new math.

You probably don’t have Astra quota. You do have the published diffs. Here’s how to use them.

What actually changed in the weights

Qwen’s own model card already flags that 0.8B thinking mode loves to loop under the recommended sampler. Astra treated that as one direction written hardest by the MLPs of layers 15 and 19, then subtracted it from 64 MLP output columns in each of those layers. Same family of move as single-direction edits in interpretability – see Arditi et al. on refusal directions (arXiv:2406.11717).

Read it the way the authors frame it: Qwen already had more math than the scores showed. The loop hid it. The edit unmasks; it doesn’t install a new calculator.

Those rows are from the official Weight-Space Programming table (as of the Nov 2, 2026 drop). The cliff is the story: 10- and 25-pass budgets barely move MATH-500 (12.3% / 14.1%). Only the ~50-pass artifact unlocks the jump, with token-limit hits called out for base vs 50-pass only in the published summary.

Hands-on: apply Astra’s Qwen weight edit

Goal in 15-30 minutes on a machine that can load 0.8B in bf16/fp16.

1. Grab base weights: Qwen/Qwen3.5-0.8B (text path used in the logs) from Hugging Face.
2. Clone the experiment dump: RikiyaT/lm_understand-logs, folder exp8_math/.
3. Use the run that matches the headline numbers: runs/p50c/. Final edit lives under that run’s edit/manifest paths; the README points at code/build_ckpt.py to cast submitted tensors to bfloat16 and splice them over the base checkpoint.
4. Build the patched checkpoint, then load with Transformers or vLLM the same way you load any Qwen3.5.
5. Match eval sampling when you sanity-check: temperature 0.6, top_p 0.95, top_k 20, presence_penalty 1.5, thinking on, long max tokens (they used up to 32,768).

# sketch - follow exp8_math/code/build_ckpt.py for exact tensor names
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

base = "Qwen/Qwen3.5-0.8B" # or local snapshot used in logs
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.bfloat16, device_map="auto")
tok = AutoTokenizer.from_pretrained(base)

# After build_ckpt.py (or equivalent) writes patched weights:
# model = AutoModelForCausalLM.from_pretrained("./qwen35-0.8b-astra-p50c", ...)

prompt = "What is 17 * 19? Please reason step by step, and put your final answer within \boxed{}."
messages = [{"role": "user", "content": prompt}]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=True)
inputs = tok(text, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=4096, temperature=0.6, top_p=0.95, top_k=20)
print(tok.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

You’re looking for fewer copy-paste reasoning spirals and an actual boxed{} before the limit. Full MATH-500 needs their eval use; a handful of hard word problems is enough to feel the difference.

Pro tip: Don’t re-derive the direction unless you’re doing research. The p50c diffs are the product of a long metered forward-pass budget (max 50 passes in the use). Applying the release is the beginner path; re-running Astra is a different, expensive project.

Common pitfalls when you touch the checkpoint

• Wrong run folder. p10c and p25c are other edits with weaker table scores. Only p50c is the 64-column × layers 15+19 story behind the viral MATH-500 jump.
• Sampler mismatch. Loop rate is sampler-sensitive. Crank temperature or shorten max tokens and you can fake a “fix” that is just early cutoff.
• Editing the forbidden set. Per the exp8_math README, save_edit refused final-norm, embedding/output layer, and whole-tensor constant multiplies – blind global scaling is blocked by design. Stay in MLP output columns like the release.
• bf16 cast. Submitted tensors get cast before merge. Mixing dtypes when you splice can silently noop or NaN.

Think of the weight file less like clay you sculpt from scratch and more like a circuit board where one noisy trace keeps the rest of the board from finishing the job.

Performance you should actually expect

Published eval only (Astra never saw the test benchmarks while editing, per the use notes): big gains on the loop-heavy MATH/GSM8K/IFEval suite above; apply-from-diff cost is minutes of download + merge, then normal 0.8B inference. Pattern fits “stop thrashing,” not “become a bigger model.”

When NOT to use this

Skip it if the model doesn’t loop. Their Gemma 3 270M control rarely hit the limit (~8.5% cut share vs Qwen’s 93%), and edits never moved MATH-500 by more than about a point (base ~6.8% → max ~7.7% across 5-50 passes). No single switch-off feature, no win.

Also skip if you need capability the base weights never had, if you can’t verify on held-out math, or if you’re on a different architecture/size and hoping the exact column indices transfer – they won’t. For those cases stay with SFT/RL or classic interp tooling. If you go deeper, start with direction-ablation papers and small-model loop debugging on Qwen3.5 – not this exact patch file.

FAQ

Do I need GPT-6 Astra to benefit?

No. Use the released p50c diffs and build_ckpt.py in the logs repo. The patch is a static artifact.

Is this the same as fine-tuning Qwen on more math?

No. Fine-tuning updates many weights from data and can add (or forget) behavior. This subtracts one looping direction from 0.016% of parameters with zero training loss. If you need new domain knowledge, fine-tune. If your 0.8B spins until max tokens, try the patch first.

Can I repeat the method on my own 7B?

Maybe in spirit – find a failure direction, ablate it – but the blog is early sparks, not a productized auto-editor. You’d need a strong model-in-the-loop, a metered forward-pass use (practice GSM8K/MATH train only; no backward/autograd), and patience. Measure loop rate under a fixed sampler first; if it’s low, don’t expect a Qwen-sized jump. As of the drop day, independent full Astra+Qwen replications weren’t a solved commodity; the practical path is applying the published diffs, not re-deriving them.

Next action: clone lm_understand-logs, open exp8_math/README.md, run the p50c merge against a fresh Qwen3.5-0.8B snapshot, and A/B three MATH-style prompts with the sampler above. Keep the base checkpoint; compare base vs patched, same seeds.