ONNX Setup Guide: Install v1.21 + Runtime v1.28 Right

“Why does pip install onnxruntime-gpu succeed but CUDAExecutionProvider silently falls back to CPU?” That question shows up on GitHub every few weeks, and the answer is almost never the one tutorials give. It’s usually a cuDNN version mismatch, an opset trap, or – in one memorable case – Microsoft shipping the wrong CUDA binaries entirely.

This guide covers installing ONNX v1.21.0 (released 27 March 2026) and ONNX Runtime v1.28.0 (the current PyPI release as of mid-2026), then untangles the failure modes that break real deployments. If you just want the model format for saving and loading, you only need onnx. If you want to run inference, you need onnxruntime too. Different packages, different install paths.

Why the pairing matters before you install anything

ONNX is a spec. ONNX Runtime is one implementation of that spec. They ship on separate release schedules – ONNX targets a four-month cadence, and ONNX Runtime ships approximately every quarter. That desync is the root cause of most “why doesn’t my model load” errors.

The rule to remember: your model’s opset ≤ what your runtime supports. The ORT compatibility docs confirm that all ORT versions support opsets from ONNX v1.2.1 onwards (opset 7+), but the ceiling moves with each ORT release. Export at opset 21 into a runtime that maxes at opset 20 and you get a hard load failure, not a warning.

System requirements for ONNX Runtime v1.28

Requirements split by execution provider:

The CUDA row is the one that bites people. Per the official CUDA EP docs, ORT built against CUDA 12.8 works with any CUDA 12.x, and ORT built against 13.0 works with any CUDA 13.x – but the two ranges are not cross-compatible. Same story for cuDNN: a build linked against cuDNN 8.x won’t work with cuDNN 9.x and vice versa. Get this wrong and you’ll see a LoadLibrary failed error at session creation.

Install ONNX Runtime v1.28.0 – the fast path

CPU inference (works everywhere, smallest download):

# CPU-only
pip install onnx==1.21.0
pip install onnxruntime==1.28.0

GPU inference on modern NVIDIA cards – pin explicitly, because the package name alone doesn’t tell you which CUDA it was compiled against:

# GPU with CUDA 13.x + cuDNN 9.x
pip install onnx==1.21.0
pip install onnxruntime-gpu==1.28.0

Old codebase still on CUDA 12? You have one more release window. The ORT v1.28 release notes on GitHub confirm that support for CUDA 12 was removed in 1.27.0 – the CUDA 13 track is now mainline. If you can’t migrate yet, pin onnxruntime-gpu==1.26.0 and plan the upgrade.

Verification: confirm you got what you think you installed

A working install is one that reports the provider you actually want. Run this before loading any model:

import onnx, onnxruntime as ort
print("ONNX:", onnx.__version__)
print("ORT:", ort.__version__)
print("Providers:", ort.get_available_providers())

Expected output on a healthy GPU install:

ONNX: 1.21.0
ORT: 1.28.0
Providers: ['CUDAExecutionProvider', 'CPUExecutionProvider']

If CUDAExecutionProvider is missing from that list, the GPU package can’t locate its CUDA/cuDNN dependencies. Fix it now – debugging a model later when the real problem is the install is a multi-hour trap.

Heads up: When creating your InferenceSession, pass providers explicitly: ort.InferenceSession("model.onnx", providers=["CUDAExecutionProvider", "CPUExecutionProvider"]). Without this, ORT silently falls back to CPU and you’ll be wondering why “GPU inference” runs at CPU speeds.

Real errors from the issue tracker

1. LoadLibrary failed with error 126 ... onnxruntime_providers_cuda.dll

Classic cuDNN mismatch. In issue #21769, a user upgraded from onnxruntime-gpu 1.18.1 – which ran fine on CUDA 11.7 – to 1.19.0 and hit a hard requirement for cuDNN 9.x and CUDA 12.x. No middle ground exists: cuDNN 8 and cuDNN 9 don’t coexist for the same ORT build. Fix: install matching cuDNN 9, or downgrade the runtime.

2. Opset X is under development and support for this is limited

Misleading message. It doesn’t mean the opset is experimental – it means your model was exported at an opset version newer than what your ORT build understands. The fix lives at export time: pass opset_version=<runtime_ceiling> to torch.onnx.export. Community threads (including diagnosis on the drdroid blog and the sherpa-onnx GitHub tracker) confirm this is the correct read – “under development” is the runtime saying “I don’t know this opset yet,” not a warning about the opset itself.

3. not a supported wheel on this platform

You’re on aarch64 – Jetson or similar – and pip is trying to install an x86_64 wheel. Official onnxruntime-gpu wheels target x86_64 and specific Python versions only. For Jetson devices you need the JetPack-specific build from NVIDIA’s developer resources, or you build from source. Don’t fight pip; it’s telling the truth.

The 1.21.0 gotcha nobody documented properly

The onnxruntime-gpu==1.21.0 Windows package was accidentally built against CUDA 11 libraries instead of CUDA 12, despite the official docs claiming CUDA 12 support. If you’re pinning to 1.21.0 on Windows and CUDA won’t initialize, that’s the reason. Move to a later build.

This kind of slip is a good reminder of why CUDA versioning in the ML toolchain is genuinely hard: ORT, PyTorch, and cuDNN all have their own release cycles, and the intersection of compatible versions is smaller than the docs imply. A shipping build can pass its own test suite and still fail in your environment. The four-line verification check above is the only reliable signal.

Upgrading, downgrading, and clean uninstall

Moving from an older ORT to 1.28? Uninstall first. pip’s dependency resolver won’t cleanly swap CUDA build variants on its own:

pip uninstall -y onnxruntime onnxruntime-gpu onnxruntime-directml
pip cache purge
pip install onnxruntime-gpu==1.28.0

The onnxruntime-training package line was deprecated after 1.19.2 per the ORT v1.20 release notes – move to the standard runtime plus a separate training pipeline. And ORT 1.28 changed how CUDA libs load: cuDNN and cuFFT are now optional at runtime for the CUDA EP, and nvrtc is no longer linked at all. That cuts the CUDA redistributable footprint you need to ship.

Full removal if you want a clean slate:

pip uninstall -y onnx onnxruntime onnxruntime-gpu
# Optional: clear cached models
rm -rf ~/.cache/onnx ~/.onnx

Where to go next

ORT loads and reports the right provider. The next task is exporting a model at the correct opset. Grab a PyTorch or Hugging Face model, export with torch.onnx.export(..., opset_version=20), then run it through the four-line check. If InferenceSession loads without complaint, you’re production-ready.

FAQ

Do I need both onnx and onnxruntime, or just one?

Running inference only? Just onnxruntime – it bundles its own ONNX parser and doesn’t need the separate onnx package at runtime. Install both only if you’re also inspecting or modifying the graph.

Can I use onnxruntime-gpu 1.28 alongside PyTorch in the same environment?

Yes, but they must share the same major CUDA and cuDNN versions. The cleanest approach: install PyTorch first (it pulls in a specific cuDNN), then install ORT with the matching CUDA build. If PyTorch bundles cuDNN 9.x, you want ORT built for cuDNN 9. The CUDA EP docs spell out the compatibility model. Critical point: mismatches don’t surface at install time – they only appear when you actually create a session.

Why is my inference slower on GPU than on CPU?

Check whether the model has ops that the CUDA EP can’t handle. When that happens, ORT falls back per-op to CPU, which means data is copying between CPU and GPU memory at every unsupported layer. That overhead can easily make GPU inference slower than a pure CPU run on small models. Use session.get_providers() to confirm which provider is active, then check the ORT op coverage list for your specific ops.