Train a 4B Model for 81% Faster Postgres Plans

End state, as reported in Rohan Bansal’s qorl writeup: a small open policy finds plans with a 1.81× geometric-mean speedup on 113 Join Order Benchmark queries (about 81% faster than default Postgres on that metric), 1.81× workload speedup, 68 wins, 0 regressions, and a 44.7% drop in summed JOB latency. Code is public at github.com/polyphilz/qorl. This walkthrough starts from measurement, not from a research plot dump.

I didn’t start by renting H100s. I started with a meaner question: how did a 4B checkpoint that only produced 14/113 valid plan candidates at 0.85× S_geo later beat the default planner on a join-heavy workload? Teach the tool interface with supervised traces, then reinforce on wall-clock time. Miss the timing discipline and you’re just ranking cache luck.

Why a tiny policy can win on repeated analytic SQL

Postgres chooses the lowest estimated cost. Join ordering is NP-hard. Cardinality math still assumes independence that IMDb-style correlations smash. Leis et al. (JOB / VLDB) showed large leftover gains years ago; the same class of errors still shows up in later retrospectives cited by the writeup. Checking a candidate is simple: execute and time it. That scalar is a clean RL signal.

Skipping the live planner for every ad-hoc query is the point. Inference plus exploration is too slow there. What pays off is offline search on SQL you rerun constantly – discover better physical plans once, pin them with hints, amortize the spend across many executions.

Steering uses pg_hint_plan. The agent does not rewrite query logic. It emits PlanAction JSON (join order, join method, scan type) that compiles into /*+ ... */ comments Postgres honors.

Base weights, per the writeup and model card: empero-ai/Qwen3.8-4B-Distill. Trainable surface was a LoRA on the order of ~21.2M params / ~42.5MB against a ~4.66B full model. Reported spend in the writeup: about $1,200 total (~$800 for ~95h on 2×H100 SXM from Lambda, ~$400 OpenAI API for Astra teacher demos) – figures as published there; cloud pricing moves.

Stand up qorl before you chase the headline multiplier

Clone the repo. Load IMDb + JOB. Run calibrate until no-op timings look boring. People who skip that step “train” on page-cache lottery tickets.

1. Repo baseline: Python 3.12, Linux x86-64 for training; uv sync --frozen (add --extra gpu on the GPU box).
2. IMDb via scripts/imdb/; JOB/CEB under benchmarks/. Fixture id imdb. JOB is 113 queries over 33 templates; CEB is the larger training pool (~13.6k queries) called out in the writeup.
3. Create a calibration experiment before any policy run:

uv run qorl experiment create --name buffer-study --method calibrate 
 --tasksets 'test=job' 
 --postgres-config docker/postgres/configs/000-pgconf-default 
 --pool-config docker/worker_pool/configs/002-poolconf-4x8

That drops experiments/NNN-buffer-study/ with config, tasks, and a thin run.py. Reference measurement protocol from the writeup: four Postgres containers for parallelism, warm until shared hit/read blocks stay within ~2%, then 20 timed runs. Author-style rig detail called out alongside the work: on the order of 4 cores / 8GB per container, shared_buffers often 128MB, IMDb slice ~8.5GB on disk.

Think of unwarmed timings like ranking race cars while each heat starts with a random head-start. Medians still wobble. Until no-op candidates stop “winning,” the reward model is lying.

4. Tools the policy must learn: inspect_relation, get_column_stats, get_plan, evaluate_candidate, keep_default, finish. evaluate_candidate validates a PlanAction, compiles hints, runs SQL, returns speedup vs default.
5. SFT: teacher trajectories first (writeup path: on the order of ~100 train / 20 val Astra traces packed into 382 rows), --method sft on a frozen base revision. Vanilla 4B is a non-starter for pure RL – 14 valid candidates and sub-1× speedups – so grammar and tool use have to land before rewards get noisy.
6. RL: merge the SFT adapter, fresh LoRA, multi-rollout rewards against measured time (calibrate|eval|sft|rl are the experiment methods in the README). Eval with --method eval. For the writeup’s headline table, selection used best feedback across three trajectories (up to 15 candidates), not only the model’s single finish action.

Pro tip: Best-feedback selection is an offline evaluation / search lens. Training still has to live with the policy’s own finish decision. Production pinning should freeze a measured hint – hint table or app layer – not re-roll an LLM on every request.

Where runs go wrong when the reward is wall time

Page cache is the quiet failure mode. Two containers thrashing the same ~8.5GB dataset steal warm pages; shared_buffers eviction makes “identical” plans disagree on clocks. Author calibration detail: shared hit/read blocks must stabilize within 2% across warmups, and mean no-op error plus p90 fool rates drove the rig. Without that, no-op candidates can look like wins a large fraction of the time and RL chases ghosts.

• Multi-run after warmup. One shot is not a reward.
• SFT before RL. Invalid JSON/plan actions waste rollouts.
• Budget eyes open. Writeup total was ~$1,200 API+GPU as published – not zero.

Scope limit, stated once: numbers are warmed, read-only SELECTs on a dataset that fits in memory with constrained buffers. Cold cache, mixed OLTP, and writes are out of frame. HN pushback tracks that. The writeup’s own framing is repeated analytic queries, not dominating one-off planning.

Against EXPLAIN grinding and classic LQO stacks

One ugly dashboard query? Fix stats and indexes first. Sticky JOB-like batches already in Dockerized Postgres workers? Then the agent loop is a real lever: inspect stats, propose hints, keep only what runs faster.

FAQ

Is 1.81× the policy alone?

No. Best-of across three trajectories. The trained policy’s own picks sit nearer ~1.40-1.41× geo mean on JOB in the writeup tables.

Can I drop a 4B model into production Postgres tomorrow?

Picture a nightly job that rewrites the same 40 reporting queries. You run the loop offline, store winning hints, re-check when data drifts. Always-on planning? Latency and variance are wrong for that. Useful during app testing to show where Postgres leaves time on the table – not as a live dependency.

Do I need a full teacher API budget to learn anything?

Full reproduction follows the writeup stack: big-GPU training, four local Postgres workers, Astra teacher demos in the published spend. For learning the system, start with calibrate + eval on a smaller taskset and a local 4B under the same 2%-stability / multi-run protocol. You will hit cache noise before you hit fancy RL math – and that measurement habit is what transfers. Fewer teacher traces can still teach PlanAction shape; dishonest rewards cannot be papered over with hardware.

Next: clone polyphilz/qorl, run a JOB calibrate experiment until no-op fool rates go dull, then read the full notes at rohanbansal.com/qorl with your own timing plots beside the published tables.