QWE AI Academy Black-box distillation wins for most teams. Teacher writes answers (and short rationales); you SFT a smaller student on those pairs. White-box logits or hidden-state matching can carry richer signal, sure – but you need open weights, compatible shapes, and more plumbing. API teachers are enough for black-box. That is what ships.
Hinton, Vinyals, and Dean’s 2015 paper framed modern distillation around soft targets and temperature – the “dark knowledge” in relative probabilities. LLM practice usually drops the fancy loss on day one: generate synthetic completions, then supervised fine-tune. Hard outputs dominate; logits stay optional when you truly own both models (Llama’s distillation guide says the same).
Quick context: what you actually buy
Strong teacher. Narrow task prompts. Collect completions. LoRA-tune a smaller open student. Baseline first, train, re-eval. The student will not stay a generalist. On that specialty it can sit close to teacher quality for much less latency and spend – if the teacher already solves the task cleanly.
Think of it like a head chef plating the same dish a hundred times while a junior only watches those plates, not the whole restaurant. You transfer one station, not the cuisine.
Old encoder proof point still gets quoted: DistilBERT held about 97% of BERT-base GLUE with ~40% fewer parameters and ~60% faster inference. For generative LLMs, practical runs and platform docs more often land in an 80-95% task-retention band when the job is scoped (classification, extraction, tool calling, RAG-style flows). Amazon Bedrock’s distillation docs report up to 5× faster and up to 75% cheaper students with under 2% accuracy loss in their published RAG-style examples (same-family pairs; you still pay teacher inference for synth data plus student fine-tune rates – verify current pricing, figures can move).
Hands-on black-box distillation tutorial
One concrete task only: intent, FAQ answers, structured extraction, or simple tool calling. Synthetic data + LoRA. No white-box requirement.
1. Define the task tightly and collect seeds
One paragraph. Exact I/O schema. Edge cases. Negatives. Then 10-50 seed pairs you would trust in production. Contradictory seeds do not “average out” – they teach mush. Hold out 50-250 test rows that look like real traffic, not like your prompt template.
Under-specified instructions are the quiet killer. The teacher invents a policy; the student memorizes the invention. Failure write-ups keep hitting the same pattern: vague task text or conflicting seeds → noisy synth → “training succeeded,” metrics still bad.
2. Generate synthetic data with the teacher
Seeds plus paraphrases, hard negatives, awkward phrasings. Answer + short rationale when the rationale actually constrains the label. Keep sampling conservative so format stays tight. Batch API calls or a synth-data kit; stop at a few hundred to a couple thousand rows you are willing to read samples from.
Filter like you mean it – wrong schema, hedges that dodge the task, confident nonsense. Run a cheap teacher-only score on the holdout before any student GPU time. Teacher weak on your distribution? Fix the brief. Do not train yet.
Pro tip: twenty minutes of spot-checking teacher rows beats an overnight LoRA on garbage. The student inherits the mess; it does not correct it.
3. Choose and baseline the student
Pick for deploy limits: small Llama, Qwen, Phi-class, similar. Zero-shot or few-shot holdout number first. That baseline is the only honest “did distillation help?” line. Climb size only when quality stalls.
4. Fine-tune with LoRA (or QLoRA)
PEFT + TRL, torchtune, or the stack you already use. Freeze base; train adapters on (prompt, teacher_completion) pairs.
from peft import LoraConfig, get_peft_model
# after loading student
lora_config = LoraConfig(
r=16,
lora_alpha=32,
target_modules=["q_proj", "v_proj"], # model-specific
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM"
)
model = get_peft_model(model, lora_config)
# SFTTrainer on filtered synth pairs
# a few epochs, lr ~1e-4-2e-4, small batch + grad accumSingle datacenter GPU is plenty for many 1B-8B adapter runs; QLoRA fits tighter boxes. Save adapter checkpoints. If you later own both models and chase generative quality harder, research lines like MiniLLM (reverse-KL style objectives) exist to blunt exposure bias – optional, not step one.
5. Evaluate properly
Aim near ≥90% of teacher on the scoped metric. Student vs teacher vs baseline. Exact match, F1, structured validators, or LLM-as-judge when paraphrases are valid – exact match alone understates good rewrites. Log latency, memory, cost per 1k tokens too. Short of target? More filtered data beats blind epoch spam. Still short? One size bump.
Common pitfalls to avoid
- Skipping teacher eval and only discovering failure after student training.
- Open-ended creative or broad chat goals – the small model lacks the capacity; keep the big model there.
- Proprietary ToS and anti-distillation defenses (summarized chains, traffic classifiers). Providers have reported large cuts to illicit distillation traffic; prefer open teachers, self-host, or compliant same-family platform flows.
- Holdout that does not match production – pretty lab scores, rough live traffic.
- Student too small for the skill depth, or too few clean seeds to cover the decision surface.
Subtler: coherent-but-wrong teacher traces – a distillation trap. The student copies fluent garbage. Filter hard. Prefer teachers that already score well on your distribution. Overdoing self-distill style loops can also hurt out-of-distribution stretch; treat OOD checks as mandatory, not optional polish.
The catch is data hygiene, not the LoRA config.
Performance and results you can expect
Scoped tasks: most of the teacher’s accuracy, far less serve cost. Bedrock’s published figures (link above) show the 5× / ~75% cost story with small accuracy gaps in friendly RAG-style setups – re-check their docs before you budget. Open pipelines with 1B-8B students and strong teachers often report that 80-95% retention band when filtering is real (practical distillation write-ups describe the same 10-50 seeds → hundreds/thousands of pairs → LoRA pattern).
White-box matching can squeeze a bit more when weights align. Engineering cost jumps. Skip it until black-box plateaus.
When NOT to use distillation
Abundant clean human labels already? Plain fine-tuning may be enough and avoids teacher quirks. Teacher fails your eval? Stop. Legal or ToS wall on outputs? Stop. One-off low traffic? Call the big model. Distribution that shifts weekly? A frozen student rots. Broad world knowledge and free-form writing stay with larger models – compression likes specialisms.
Is a small quality gap fine if inference gets several times cheaper? For a lot of classifiers, extractors, and tool routers, yes. For a brand voice chatbot that must improvise, usually no.
FAQ
Is black-box distillation the same as regular fine-tuning?
No. Distillation’s training signal is the stronger teacher’s outputs – style, refusals, formatting ticks included. Human labels are optional spice, not the core.
How many synthetic examples do I need?
A few hundred clean pairs from 10-50 seeds is a sane start. Many solid runs sit around 1k-5k after filters. Example: 20 crisp seeds → ~800 kept rows → adapter trains; metric jumps past few-shot baseline, then flattens – so you re-generate hard slices instead of dumping another 40k noisy lines. Quality first. Scale only after the teacher already clears the holdout.
Can the student ever beat the teacher?
On the narrow in-domain metric, sometimes. Filtering can drop the teacher’s bad days and leave a tighter specialist than a generalist teacher. That is not “the 1B model became smarter than GPT-class everywhere.” Broad skills stay with the teacher. Lean too hard on short “correct” traces and OOD behavior can get brittle – keep a messy real-traffic slice in eval so you notice.
This week: one production task, tight brief, ~20 seeds, ~500 teacher completions, ruthless filter, LoRA on a small open student, holdout score before any fancier white-box work. That single loop tells you if distillation pays rent.