Install DeepSpeed 0.19.5 for Distributed Training

A 7B model in full precision can OOM on a 24GB card before epoch one finishes. DeepSpeed 0.19.5 is the practical way to shard optimizer state, gradients, and parameters across GPUs so the job fits – without rewriting the model.

This is a deploy path pinned to the current PyPI release, not a ZeRO concepts tour. Install from official channels, drop a minimal config, prove it with ds_report, and fix the log lines that actually show up.

System requirements before you touch pip

PyTorch first. DeepSpeed will not replace a broken CUDA wheel. Per the PyPI requirements for DeepSpeed (as of the 0.19.5 listing), install a CUDA/ROCm build of torch before DeepSpeed, with PyTorch ≥ 2.0 if you want full features – ideally the stable build that matches your driver.

Windows: pip can pull prebuilt ops and many train/infer paths work; AIO and GDS stay unsupported, so NVMe offload plans belong on Linux. macOS is not a GPU target here.

Where 0.19.5 actually comes from

• PyPI:pypi.org/project/deepspeed – pin 0.19.5
• GitHub tag:deepspeedai/DeepSpeedv0.19.5
• Install docs:advanced install (CUDA mismatch, arch list, JIT cache)

ZeRO is the memory technique under the hood. Research pointer if you need it for a design doc: Rajbhandari et al., arXiv:1910.02054.

Install DeepSpeed 0.19.5

Same env where import torch already sees GPUs. No second “training” env with a different CUDA story.

python -c "import torch; print(torch.__version__, torch.version.cuda, torch.cuda.is_available())"
nvcc --version # major should match torch.version.cuda major

pip install deepspeed==0.19.5

Build isolation sometimes hides torch while setup.py imports it. Then you get ModuleNotFoundError: No module named 'torch' mid-pip. Fix:

pip install deepspeed==0.19.5 --no-build-isolation

Want ops compiled during install instead of on first training step?

DS_BUILD_OPS=1 pip install deepspeed==0.19.5 --no-build-isolation

Default path JIT-builds into ~/.cache/torch_extensions/. Fine on a laptop. On a shared node it means every mismatched job can recompile – or worse, reuse someone else’s binary.

Think of that cache like a single junk drawer for every roommate’s power tools. Python 3.10 + CUDA 12 in the morning, 3.11 + CUDA 11 at night – same drawer, broken handles. Official advanced-install docs call out setting a per-env TORCH_EXTENSIONS_DIR before the first JIT build when you juggle multiple virtualenvs.

export TORCH_EXTENSIONS_DIR=$HOME/.cache/torch_extensions_$CONDA_DEFAULT_ENV
pip install deepspeed==0.19.5 --no-build-isolation

Source only if you need a commit past the pin:

git clone https://github.com/deepspeedai/DeepSpeed.git
cd DeepSpeed
git checkout v0.19.5
pip install . --no-build-isolation

Minimum config that boots

JSON drives behavior; the deepspeed entry point sets ranks. ds_config.json:

{
 "train_batch_size": 8,
 "gradient_accumulation_steps": 1,
 "optimizer": {
 "type": "Adam",
 "params": { "lr": 0.00015 }
 },
 "fp16": { "enabled": true },
 "zero_optimization": {
 "stage": 2
 }
}

Engine wiring (pattern from DeepSpeed’s getting-started guide):

import deepspeed

model_engine, optimizer, _, _ = deepspeed.initialize(
 args=cmd_args,
 model=model,
 model_parameters=model.parameters()
)

loss = model_engine(batch)
model_engine.backward(loss)
model_engine.step()

deepspeed --num_gpus=2 train.py --deepspeed --deepspeed_config ds_config.json

Multi-node: hostfile lines like worker-1 slots=4, then deepspeed --hostfile=myhostfile .... Cloud images without SSH meshes use the launcher’s no-SSH / rank / master-addr flags on each node – same binary, different process bootstrap.

Actually – before you scale hosts, prove one node.

Verify the install

ds_report
# same idea:
python -m deepspeed.env_report

You want version 0.19.5, a populated torch CUDA version, and ops you care about marked compatible (installed if you used DS_BUILD_OPS=1). Smoke path: tiny nn.Linear + deepspeed.initialize with the JSON above under the launcher. If backward/step return without op-builder exceptions, the stack is alive.

Common install errors and fixes

These are the paste-bin strings – not “update your drivers” hand-waving.

• Installed CUDA version X does not match the version torch was compiled with Y – Align toolkit major (nvcc --version) with torch.version.cuda, or reinstall a matching torch wheel. Majors block compile; minors often only warn. Escape hatch from the advanced-install mismatch section: DS_SKIP_CUDA_CHECK=1 (runtime bugs possible).
• CUDA_HOME does not exist, unable to compile CUDA op(s) – torch.cuda.is_available() only proves runtime. Op builds need a toolkit. Export CUDA_HOME to the install root and confirm $CUDA_HOME/bin/nvcc exists.
• RuntimeError: CUDA error: no kernel image is available for execution on the device – extensions built for other arches. Rebuild with a tuned list, e.g. TORCH_CUDA_ARCH_LIST="7.5;8.0;8.6;9.0" pip install --force-reinstall --no-cache-dir deepspeed==0.19.5 --no-build-isolation.
• ModuleNotFoundError: No module named 'torch' during pip – isolation hid torch. --no-build-isolation after torch is already in the env.

Here’s the mental model that saves hours: driver + PyTorch wheel can look perfect while DeepSpeed still refuses to build ops. Runtime CUDA ≠ compile toolchain. The op builder probes nvcc/headers at install or first use; if that probe fails, training never starts – even though a plain tensor on GPU works fine.

Upgrade, migrate, uninstall

pip install -U deepspeed==0.19.5

After a torch or DeepSpeed jump, stale .so files lie:

rm -rf ~/.cache/torch_extensions/
# or your custom TORCH_EXTENSIONS_DIR

pip uninstall deepspeed -y
rm -rf ~/.cache/torch_extensions/

Older tutorial JSON usually still parses. If a flag is ignored, check the current config JSON reference for stage-3 offload and optimizer keys. Same install pairs with Transformers/Accelerate DeepSpeed integration or Lightning’s DeepSpeed strategy – different launcher flags, not a second pip stack.

FAQ

Do I need DS_BUILD_OPS=1 every time?

No. Skip it for quick trials; JIT builds what you touch. Use it on golden images where first-step latency and identical binaries matter.

Can I run DeepSpeed on Windows for multi-GPU LLM finetuning?

Scenario: one workstation, Windows, two consumer GPUs, “just finetune.” pip install deepspeed often works for single-GPU train/infer with prebuilt ops; source builds want Visual C++ tools and build_win.bat. AIO/GDS still unavailable (plan offload on Linux). Multi-GPU Windows remains thinner than Linux – for multi-GPU distributed jobs, use Linux or WSL2 with a full CUDA toolkit.

Why does ds_report show an op as compatible but not installed?

“Compatible” means the machine can build it. “Installed” means it was precompiled into the wheel or a prior local build. People read the green compatible column and assume zero compile risk mid-job – that’s the trap. JIT still needs nvcc the first time that op runs. If surprise compile time is unacceptable, reinstall with DS_BUILD_OPS=1 (or the matching DS_BUILD_* flags), wipe a dirty extensions cache if you changed torch, rerun ds_report, and only then submit the cluster job.

Next: fresh venv → torch for your CUDA build → pip install deepspeed==0.19.5 --no-build-isolation → ds_report → one ZeRO-2 step on two GPUs. Hostfiles later.