Install TRL v1.13.0 for RLHF Open Source

Deploy Hugging Face TRL v1.13.0 for RLHF open source post-training. Exact pip, Docker, extras, verify steps, and install fixes that actually work.

Casey Morgan2026-09-116 min readIntermediate

TRL v1.13.0 on the box, version-checked, ready for open-source post-training (SFT → DPO/GRPO/reward). That is the job if you searched RLHF open source tooling: a working Hugging Face TRL environment, not another alignment essay.

As of 10 Sep 2026, v1.13.0 is the stable tag. Install first. Train second.

Most writeups still open with “what is RLHF?” then paste a demo trainer. Useful once. Useless when import already fails. This page treats the environment as the product.

System requirements before you touch pip

TRL ships as OS-independent Python. Training does not. Match the machine to the workload.

Component	Minimum (library)	Practical for 7-8B LoRA/QLoRA
Python	≥3.10 (`requires-python` in pyproject)	3.11 or 3.12
Core libs	transformers≥4.56.2, accelerate≥1.4.0, datasets≥4.7.0	Same + a CUDA PyTorch build that matches the GPU driver
GPU VRAM	CPU smoke only	16-24 GB class (as of community deployment notes tied to 7B QLoRA)
System RAM	–	32 GB+ in those same notes
CUDA	Whatever your torch wheel expects	12.x on current official images

Official docs never publish a hard VRAM floor. The 16-24 GB band is where 7B-class QLoRA stops feeling cursed; full FT and heavy online RL still want data-center memory.

Official download sources

PyPI wheel (default path):pypi.org/project/trl – pip install trl
Install docs:huggingface.co/docs/trl/en/installation
Source:github.com/huggingface/trl
Docker Hub:huggingface/trl – hub.docker.com/r/huggingface/trl
Breaking changes:v1.13.0 release notes

Skip forks and old posts that still center PPO as the happy path.

Install TRL v1.13.0

Virtualenv only. System Python fights you later.

python -m venv .venv
source .venv/bin/activate # Windows: .venvScriptsactivate
pip install --upgrade pip
pip install "trl==1.13.0"

uv works the same way (both show up in the install docs):

uv pip install "trl==1.13.0"

Core pull: accelerate, datasets≥4.7.0, transformers≥4.56.2. Torch follows your transformers/torch stack – land the right CUDA wheel before you blame TRL for a CPU-only device.

Extras and the floors that break silent upgrades

As of v1.13.0 the optional floors are the gotcha, not the extra names:

# LoRA / QLoRA on consumer GPUs
pip install "trl[peft,quantization]" # peft>=0.13.0

# Online RL generation (GRPO/RLOO)
pip install "trl[vllm]" # vllm>=0.19.1,<=0.28.0

# Multi-GPU ZeRO
pip install "trl[deepspeed]" # deepspeed>=0.18.6

# Fused kernels
pip install "trl[liger]" # liger-kernel>=0.8.2

# Common combo
pip install "trl[peft,quantization,liger]==1.13.0"

pip install -U trl alone leaves old peft/deepspeed behind; trainers then die on missing APIs. The vllm extra stays out of the dev bundle on purpose – CUDA/resolution fights with unpinned latest vLLM plus current transformers show up constantly in community notebooks.

The catch is psychological: people treat extras like flavor flags. Here they are version contracts.

Docker

docker pull huggingface/trl
docker run --gpus all -it --rm huggingface/trl

Repo Dockerfile base: pytorch/pytorch:2.8.0-cuda12.8-cudnn9-devel, image preinstalls trl[liger,peft,vlm]. Pin a digest from Docker Hub when the node must be frozen.

From source

git clone https://github.com/huggingface/trl.git
cd trl
pip install -e .
# contributors:
pip install -e ".[dev]"

One-liner: pip install git+https://github.com/huggingface/trl.git. Prefer trl==1.13.0 wheels on training boxes; editable trees are for debugging TRL or grabbing a commit PyPI does not have yet.

Minimum config

No config file required to import. For a real run:

Gated model/dataset → huggingface-cli login or export HF_TOKEN=...
nvidia-smi, then optional CUDA_VISIBLE_DEVICES=0
Multi-GPU later → accelerate config once

Enough for the smoke checks below. Dataset schemas and trainer kwargs wait until import trl is boringly reliable.

People sleep on this: cold import trl still drags transformers with it. A few seconds on a laptop is normal. Not proof the install is broken.

Verify the install works

python -c "import trl; print(trl.__version__)"
trl --help

You want 1.13.0 (or a git describe string on editable installs) and a CLI banner that lists sft, dpo, grpo, reward, and friends. Entry point is trl = trl.cli:main per packaging scripts.

python -c "from trl import SFTTrainer, DPOTrainer, GRPOTrainer; print('trainers ok')"

If those imports succeed, the RLHF open-source toolchain is alive. Save long trl sft runs for after pins and CUDA are settled – no need to clone the docs demo to prove pip worked.

Common install errors and fixes

1. from trl import PPOTrainer → ImportError
Gone. v1.13.0 release notes remove PPOTrainer, PPOConfig, and value-head modeling. Top-level from trl import PPOTrainer was already dead since v1.10. Move to GRPO/DPO/RLOO, or pin a pre-removal wheel if a legacy script is non-negotiable.

2. Resolver explodes on trl[vllm]
The extra’s upper bound (≤0.28.0) exists because free-floating latest vLLM and current transformers often refuse to solve. Install the extra as declared; read the current release notes for supported pairs instead of stacking random pins.

3. pip install git+... on Windows → WinError 1314
Symlink privilege when setup links examples/scripts (still reported, including GH #1461-class failures). Developer Mode or elevated shell – or skip source and take the PyPI wheel.

4. Upgrade-only pip install -U trl → PEFT/DeepSpeed AttributeErrors at train time
Floors moved. Reinstall the extras you actually use so peft≥0.13.0 and deepspeed≥0.18.6 arrive together:

pip install -U "trl[peft,deepspeed]==1.13.0"

5. Preference data blows up in SFT/DPO/Reward preprocessing after a clean install
datasets is now ≥4.7.0, and MIGRATION.md dropped automatic None stripping on nested examples. Older preference dumps (pre-Json dtype) can inject Nones. Call remove_none_values yourself before training – the library will not babysit that path anymore.

Upgrade / uninstall

pip install -U "trl[peft,quantization]==1.13.0"
# migration reference:
# https://github.com/huggingface/trl/blob/main/MIGRATION.md

Coming from earlier v1: GRPO/RLOO default vllm_mode flipped server → colocate; SFT packing value bfd-requeue renamed bfd_split. Those are silent behavior changes, not install failures – they show up as “why is my old script slower/weirder?” after a clean wheel upgrade.

pip uninstall trl -y
deactivate
rm -rf .venv

Docker cleanup: docker rmi huggingface/trl when you want the image gone too.

FAQ

Is TRL still the right RLHF open source library after PPO removal?

Yes. Maintained path is SFT + reward modeling + DPO/KTO/GRPO/RLOO. Pin old TRL only if classic PPO scripts are mandatory.

Do I need vLLM on day one?

No. SFT and offline DPO are fine without it. Add trl[vllm] when online methods generate completions every step (GRPO/RLOO) and you care about sample speed. Wait until that day – the pin window is where installs go sideways, and you do not need the pain during a first SFT smoke.

Can I train a 70B model after this install?

Same package. Different machine. Feasibility is VRAM + PEFT/DeepSpeed/FSDP, not a second TRL flavor. Multi-GPU QLoRA with trl[peft,quantization,deepspeed] is what most large-model runs use; full FT needs cluster-class memory. Prove import trl and the trainer imports on a tiny config first, then scale model size and parallel settings. Do not invent a parallel install path just because the checkpoint got bigger.

Activate the venv. Run python -c "import trl; print(trl.__version__)". When it prints 1.13.0 and the trainer imports hold, point your real model and dataset at SFT/DPO/GRPO and start training.