QWE AI Academy You’ve asked ChatGPT the same careful question twice. Once it refuses. Once it lectures. Once it invents a confident answer. That gap is the everyday face of alignment – whether the model’s behavior matches what you (and its builders) actually want, not just next-token statistics from the web.
Two common fixes sit on the table. Approach A: stack system prompts, refusals, and clever wording at inference time. Approach B: change the model with preference data so preferred answers become more likely by default. Approach B wins for durable behavior. Prompts are cheap and reversible; they also get ignored, jailbroken, or diluted as the context fills. Preference training (RLHF, DPO, and cousins) rewires the distribution. For anything you ship, you want that second path – or a vendor who already did it.
The problem alignment is actually solving
A base LLM is trained to predict the next token. That objective doesn’t care about your intent, honesty, or safety. It will complete toxic patterns, invent plausible facts, or follow a harmful instruction if those patterns showed up often enough in pretraining. IBM’s LLM alignment overview treats those failure modes as structural, not one-off bugs.
Most teams aim at Anthropic’s HHH target: helpful, honest, harmless. They fight each other. Max help on a borderline request can cut harmlessness. Blunt honesty about uncertainty can feel cold. Alignment work is mostly negotiating those trade-offs in data and losses – not finding one magic principle.
Why prompt-only and pure SFT fall short
System prompts and instruction-style SFT get you partway. SFT teaches format and “answer like this demo.” It does not teach relative preference when two fluent answers disagree on tone, caution, or truthfulness. Prompt wrappers sit outside the weights. Users override them. Long chats weaken them. Injection treats them as text, not law.
Outer alignment – the usual post-training layer – has a known brittleness. Research summarized by IBM notes that post-hoc methods don’t truly unlearn base tendencies. Small fine-tunes, even on harmless math sets like GSM8K, have eroded safety behaviors. If your “solution” is only a thin refusal layer, plan for it to crack under domain shift or further training.
Recommended approach: preference alignment after SFT
The practical stack that made assistants usable looks like this.
- Supervised fine-tuning (SFT) on high-quality prompt→response demos so the model follows instructions at all.
- Preference data: for the same prompt, humans (or AIs under a constitution) pick which of two or more completions is better.
- Optimization so the policy prefers the chosen answers without drifting too far from the SFT reference (KL-style regularization in classic setups).
Classic RLHF, popularized by OpenAI’s InstructGPT work, trains an explicit reward model on those rankings, then runs RL (often PPO) to maximize reward. Human evals on their prompt distribution showed a 1.3B InstructGPT preferred over 175B GPT-3 – alignment quality beat raw scale for instruction following (Ouyang et al., arXiv:2203.02155; also in OpenAI’s instruction-following post).
DPO (Direct Preference Optimization) skips the separate reward model and online RL loop. Rafailov et al. turned the same preference pairs into a classification-style loss on the policy versus a frozen reference – no reward model, no PPO sampling loop. The DPO paper (arXiv:2305.18290) is lighter and often matches or beats PPO-style RLHF on dialogue and summarization-style tasks at moderate scale.
Constitutional AI swaps much of the human harmlessness labeling for AI critique against a written principle list, then preference-trains on that signal (RLAIF). Recipe: Bai et al., arXiv:2212.08073. Someone still has to write the constitution; helpfulness labels usually stay partly human.
| Method | Feedback source | Extra models / loop | Best known for |
|---|---|---|---|
| RLHF (PPO-style) | Human (or mixed) rankings | Reward model + RL | Production chat baselines |
| DPO | Same preference pairs | Policy + reference only | Simpler open fine-tunes |
| Constitutional / RLAIF | Principles + AI judges | Critique + preference stage | Scalable harmlessness |
Ji et al. (arXiv:2310.19852) frames alignment research with RICE goals – Robustness, Interpretability, Controllability, Ethicality – and splits “forward” training from “backward” assurance and governance. Shipping a product? Forward preference training is the lever you feel first. Backward evals tell you whether it stuck.
Real-world example: base GPT-3 vs InstructGPT-style chat
Before preference tuning, API users often got completion-style rambling: the model continued the prompt instead of treating it as a task. After SFT + RLHF, the same family started answering questions, admitting limits more often, and cutting some toxicity – enough that people preferred a much smaller tuned model to a giant untuned one on OpenAI’s evals. “Just scale pretraining” stopped being the full product story.
ChatGPT-class systems still rest on variants of that stack, mixed with AI feedback, tool policies, and written specs. The user-visible win is boring reliability: fewer random genre shifts, more refusals on clear harm, more “I don’t know” when appropriate. Residual failures – sycophancy, overconfident tone, occasional concealment in agent settings – are why labs still publish misalignment case studies.
Pro tip: When you evaluate a vendor model, don’t only red-team jailbreaks. Also score pairwise preferences on your domain: two answers, blind rank, track helpfulness vs. hedging vs. fabrication. That’s closer to how the model was trained than a single pass/fail refusal test.
Pro tips that survive contact with real data
- Write the conflicts down. If helpfulness and harmlessness clash on your use case (e.g., dual-use chemistry Q&A), decide the priority order before you collect labels. Annotators will invent one otherwise.
- Watch length and flattery. Preference models love polished, agreeable, long answers. Cap length or add explicit criteria for concision and pushback – or you train a people-pleaser that sounds sure while wrong.
- Keep a frozen reference. PPO-style KL or DPO’s reference policy both exist because unconstrained optimization drifts into reward hacking. Follow-on work still sees over-optimization even with “direct” methods.
- Ask who labeled. Small rater pools encode culture, politics, and corporate policy – not abstract “human values.” InstructGPT’s own discussion of labeler sources is still the right mental model: you are aligning to that pool, not to humanity at large.
- Re-check safety after every train. Treat outer alignment as thin. Domain fine-tunes can strip refusals; run the same harm evals again before you ship the new checkpoint.
Think of alignment less like installing a moral chip and more like teaching taste with lots of graded examples. Taste generalizes until the situation is new, the incentives flip, or someone retrains the student on a different exam.
FAQ
Is alignment the same as “the model refuses bad stuff”?
No. Refusal is one harmlessness slice. You also need intent-following and honest uncertainty – not a brick wall that never helps.
Should I use DPO or full RLHF for a small open model?
Most indie and mid-size fine-tunes: strong SFT, then DPO on clean preference pairs. You skip reward-model training and PPO hyperparameter drama. Reach for classic RLHF-style loops when you need online sampling, multi-objective reward shaping, or lab-scale infrastructure. Bad preference data still beats a clever optimizer.
If ChatGPT already feels aligned, is the problem solved?
Everyday chat got far more usable than base LLMs. That is not deep value alignment under tools, long horizons, or adversarial pressure. As of 2025-2026, OpenAI’s model misalignment reporting framework and related lab disclosures still treat residual cases (concealment, unauthorized tool use, and similar) as open. Consumer polish ≠solved oversight.
Next action: pick ten real prompts from your product, generate two answers each from the model you use today, rank them with a short written rubric (helpfulness, honesty, harm), and note where the winner still fails your rules. That ranking file is the seed of your own alignment dataset – or your vendor scorecard.