Deploy vLLM v0.23.0: LLM Serving Framework Setup Guide

Hot take: if you’re still typing pip install vllm, you’re already behind the official docs. The vLLM team now recommends uv as the installer of record – not just a style preference. uv handles the PyTorch index auto-selection that pip routinely gets wrong, which is the single most common reason a fresh install fails on import.

This guide covers deploying vLLM v0.23.0 as an LLM serving framework. We’ll skip the PagedAttention marketing pitch (it’s real, it works, Kwon et al. SOSP 2023 if you want the paper) and go straight to where deployments actually break: KV cache math, the CUDA-graph trap, and the v0.8.0 upgrade gotcha that silently changed your model’s defaults.

What you’re actually deploying

vLLM wraps Hugging Face model weights and exposes them as an OpenAI-compatible HTTP server. Binds to http://localhost:8000 by default. Implements /v1/chat/completions and /v1/completions – your existing OpenAI SDK code can point at it unchanged. The official quickstart confirms the defaults; host and port are configurable via --host and --port.

The throughput gains come from block-based KV cache management – sequences share cache pages instead of each holding a private allocation. More concurrent sequences fit in the same VRAM budget. That’s the why. Now the how.

System requirements

Most tutorials list “NVIDIA GPU” and stop. The vLLM releases page (as of mid-2026) is tighter:

AMD? ROCm support landed in v0.14.0; as of v0.21.0, vLLM ships against ROCm 7.2.2. Check the releases page before assuming your card is covered – the supported matrix shifts between minor versions.

Installing vLLM v0.23.0

Install uv first:

curl -LsSf https://astral.sh/uv/install.sh | sh

Then create an isolated environment and install. The --torch-backend=auto flag inspects your installed CUDA driver and picks the matching PyTorch wheel – skip it and pip will grab whatever wheel it finds first, which is usually wrong.

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto

Verify:

python -c "import vllm; print(vllm.__version__)"
# Should print 0.23.0 (as of late 2025 / early 2026 - check PyPI if this diverges)

Prefer Docker? The official image is vllm/vllm-openai:latest on Docker Hub. Per the vLLM GPU installation docs:

docker run --runtime nvidia --gpus all 
 -v ~/.cache/huggingface:/root/.cache/huggingface 
 --env "HF_TOKEN=<your_token>" 
 -p 8000:8000 --ipc=host 
 vllm/vllm-openai:latest 
 --model Qwen/Qwen2.5-1.5B-Instruct

Older CUDA drivers on datacenter GPUs? Add -e VLLM_ENABLE_CUDA_COMPATIBILITY=1 to the docker run command. This enables forward-compat CUDA libs – the GPU installation docs note it only works on select datacenter hardware, not consumer cards.

First boot: the configuration that actually matters

Start with the model from the official quickstart. That way any error is yours, not a model compatibility quirk:

vllm serve Qwen/Qwen2.5-1.5B-Instruct 
 --gpu-memory-utilization 0.85 
 --max-model-len 4096

Three flags do most of the work. --gpu-memory-utilization caps how much VRAM vLLM may claim (default is greedy). --max-model-len sets the context length ceiling. --max-num-seqs caps concurrent requests. The third one trips people up most.

Here’s the math tutorials skip: KV cache memory ≈ 2 × num_layers × hidden_size × max-model-len × max-num-seqs × dtype_size. Set --max-model-len 32768 with --max-num-seqs 256 on a 7B model and you’ve pre-reserved 20+ GB for cache before generating a single token. Most applications don’t need 32K context or 256 concurrent sequences. Pick what your workload actually uses.

Start conservative: --max-model-len 2048, --gpu-memory-utilization 0.85. Once the server boots, raise both incrementally. The model’s documented context window is the ceiling, not the target.

There’s a question worth sitting with here: how much of your GPU budget should go to KV cache vs. leaving headroom for OS and driver overhead? The answer changes depending on whether you’re running a dedicated inference box or sharing a machine. No formula covers both – this is where you need to instrument and observe, not just configure and hope.

Verifying the install

curl http://localhost:8000/v1/models

curl http://localhost:8000/v1/chat/completions 
 -H "Content-Type: application/json" 
 -d '{
 "model": "Qwen/Qwen2.5-1.5B-Instruct",
 "messages": [{"role":"user","content":"ping"}]
 }'

JSON response with a choices array? Done. Connection refused? Server is still loading weights – small models take seconds, a 70B can take several minutes on first download.

Errors nobody warns you about

OOM when nvidia-smi shows free VRAM. vLLM pre-records a static CUDA graph for the decode phase to speed up repeated kernel launches. That graph reserves a contiguous memory slab up front. When it can’t get that slab – even if total free VRAM looks fine – it crashes. Add --enforce-eager to skip CUDA graphs entirely. Turns out this frees roughly 1.5-2.0 GB on a 24 GB card (per community testing documented in Markaicode’s vLLM OOM guide, May 2026), at the cost of some decode throughput. Worth it for debugging; reconsider before production.

Sampling defaults changed silently in v0.8.0. Before that release, vLLM used its own neutral defaults for temperature, top-p, and so on. After v0.8.0, it pulls those defaults from the model’s generation_config.json instead (PR #12622, per the troubleshooting docs). For most models, better outputs. For some, noticeably worse. If an upgrade broke your output quality without any other change, pass --generation-config vllm to restore the old behavior.

NCCL_CUMEM_ENABLE=0 in old deployment scripts. vLLM versions 0.4.3 through 0.10.1.1 set this env var to work around an NCCL memory bug. The bug was fixed in NCCL 2.22.3 and the override was removed. But if your Kubernetes manifests or systemd units still export it, you’re disabling NCCL performance optimizations for no reason. Audit and remove.

PTX toolchain mismatch.CUDA error: the provided PTX was compiled with an unsupported toolchain means your driver is older than the CUDA version the wheels were built against. In Docker: add -e VLLM_ENABLE_CUDA_COMPATIBILITY=1. Datacenter GPUs only – the official docs are explicit on that.

Upgrading and uninstalling

Patch releases ship roughly every two weeks. Upgrade in place:

uv pip install --upgrade vllm --torch-backend=auto

Between minor versions, read the changelog before swapping a production tag. Behavior under tensor parallelism has shifted more than once across recent releases. Test against your actual workload – not a benchmark you found online.

Clean uninstall:

uv pip uninstall vllm
deactivate
rm -rf .venv
rm -rf ~/.cache/huggingface/hub # only if you want to clear downloaded weights too

The Hugging Face cache is the biggest disk consumer. A few large models will quietly fill a drive. Check the size before deleting anything you’d need to re-download on a slow connection.

FAQ

Can I run vLLM on a consumer GPU like an RTX 4090?

Yes. 24 GB VRAM handles 7B models comfortably in FP16. Below 16 GB, reduce --max-model-len and stick to smaller models – the VRAM math catches up fast.

Why is the first request so slow?

Two things stack on top of each other. First, if weights aren’t cached yet, vLLM downloads them from Hugging Face (stored in ~/.cache/huggingface afterward). Then it warms up CUDA graphs by running dummy forward passes to lock in kernel selections – this happens every cold start. After warm-up, steady-state throughput kicks in. If you just need a quick test and don’t care about maximum throughput, --enforce-eager skips graph compilation and cuts startup time. The trade-off: decode is slower per request once serving.

How does vLLM compare to Ollama or TGI?

Different tools, genuinely different targets. Ollama is for single-user local inference with GGUF models – not trying to serve concurrent traffic. Hugging Face TGI is the closest real competitor; the raw throughput gap has narrowed. The case for vLLM: broader quantization format support in a single engine (FP8, MXFP4, NVFP4, GPTQ, AWQ, GGUF) and the OpenAI API drop-in. If your stack already expects OpenAI-shaped requests, vLLM is the path of least resistance. If you’re running a personal dev setup, Ollama is simpler and that’s fine.

Next: spin up the Qwen2.5-1.5B example above, hit it with the curl request, and watch nvidia-smi dmon -s mu in a second terminal while you do. Five minutes watching the memory curve live teaches more about KV cache behavior than any explanation of the math.