WhisperX v3.8.6: Speaker Diarization Install Guide

Most WhisperX tutorials still tell you to accept the license for pyannote/speaker-diarization-3.1. That’s wrong for the current version. WhisperX v3.8.6 (released May 25, 2025) switched its diarization backend to speaker-diarization-community-1, and if you only accept the old model’s terms, your script runs to completion – with empty speaker labels. No error, no warning. Just silence where speaker names should be. This guide installs the current version, gets diarization working, and flags the gotchas no one’s updated their blog posts for.

The stack: faster-whisper handles transcription at up to 70x realtime with large-v2, wav2vec2 aligns words to sub-100ms timestamps, and pyannote assigns speaker identities. You need all three working together. One broken link – usually the HF token or the torch version – and you get either a crash or a silently degraded output that looks fine until you check the speaker labels.

System requirements (as of v3.8.6)

Diarization quality and speed depend heavily on hardware. Here’s what works in practice:

CUDA 12.8 isn’t a suggestion. Per the official README, GPU acceleration requires CUDA toolkit 12.8 installed before WhisperX – skip this only for CPU-only runs. On a correctly sized card, faster-whisper’s backend needs under 8 GB of VRAM for large-v2 with beam_size=5.

The Pascal GPU trap – read this before running pip install

GTX 10xx series? The default pip install whisperx will probably break at runtime with a CUDA capability error. The reason isn’t in any official WhisperX doc.

The pip resolver pulls in torch 2.8 (CUDA 12.8), which dropped Pascal-architecture support. Your install completes cleanly. Then you run it and hit CUDA error: no kernel image is available for execution on the device. A community-maintained WhisperX service documents this exact upstream behavior in its v0.3.2 changelog – it exposes explicit torch version build args and re-pins torch after the WhisperX install to restore Pascal and Hopper compatibility.

Fix for Pascal hardware (10xx, P100):

pip install whisperx
pip install --force-reinstall torch==2.7.1 torchaudio==2.7.1 
 --index-url https://download.pytorch.org/whl/cu126

RTX 50xx (Blackwell) users go the other direction – keep torch 2.8 / cu128. Turing through Hopper (20xx-H100) is fine with defaults.

There’s something worth sitting with here: three generations of GPU architecture, three different torch pins, and WhisperX itself has no awareness of any of it. The tool just asks for the latest torch and lets the GPU ecosystem sort out the casualties. Knowing this upfront saves an hour of staring at CUDA error codes that feel like your whole environment is broken when it’s really just one version flag.

Install WhisperX v3.8.6

One version history note first: PyPI versions 3.1.1-3.1.6 are yanked. Third-party unofficial releases, per the PyPI package page. Don’t pin those. Install from the current release.

Step 1 – System prerequisites

# Ubuntu/Debian
sudo apt update && sudo apt install -y ffmpeg python3.11-venv

# macOS
brew install ffmpeg [email protected]

# Verify
ffmpeg -version

Step 2 – Virtual environment + WhisperX

python3.11 -m venv whisperx-env
source whisperx-env/bin/activate
pip install --upgrade pip
pip install whisperx

One command. Pulls in faster-whisper, ctranslate2, pyannote.audio, torch – the whole chain. If you’re on Pascal, run the torch re-pin from above immediately after this step.

Step 3 – The HuggingFace token (where most tutorials send you to the wrong page)

Diarization in v3.8.6 requires accepting the license for the new model. Not the 3.1 page.

1. Create a free HuggingFace account at huggingface.co.
2. Generate a read token at huggingface.co/settings/tokens.
3. Go to huggingface.co/pyannote/speaker-diarization-community-1 and accept the terms. Not the 3.1 page.
4. Export the token: export HF_TOKEN=hf_xxxxxxxxxxxx

Why the switch? Almost two years after pyannote.audio 3.1, pyannoteAI released pyannote.audio 4.0 alongside community-1 – its latest open-source diarization model, now licensed under CC-BY-4.0. WhisperX adopted it as the default backend. The old 3.1 model still exists on HuggingFace; accepting its license does nothing for v3.8.x.

Heads up: If you had a working WhisperX setup and diarization suddenly returns empty speaker labels after an upgrade, the community-1 license is almost certainly the cause. Accept the terms with the same HF account that issued your token. Tokens scoped to organizations you don’t own won’t work.

First-run verification

Drop a test WAV in your working directory – a 2-speaker, ~2-minute clip works well:

whisperx --version

whisperx test.wav 
 --model large-v3 
 --diarize 
 --min_speakers 2 --max_speakers 2 
 --hf_token $HF_TOKEN 
 --compute_type float16 
 --output_format srt

First run downloads models into ~/.cache/huggingface (a few GB). Output should be an SRT with lines prefixed [SPEAKER_00] and [SPEAKER_01]. Both labels appearing and roughly tracking the right speakers = install works.

Pass --min_speakers / --max_speakers whenever you know the speaker count. With known speaker counts, diarization accuracy on multi-speaker recordings improves noticeably – pyannote’s auto-detection tends to under-count when voices overlap, and you pay for that in merged speaker segments. The WhisperX docs call these flags out specifically for this reason.

Audio prep nobody mentions

community-1 expects mono 16kHz. Stereo gets averaged – both channels blended into one. That auto-downmix sounds harmless until you think about what podcast stereo actually contains.

Speaker A panned hard left, Speaker B hard right. Channel-averaging throws that separation away before pyannote sees a single frame. The model then has to separate two voices from a blended mono signal instead of each getting a clean track. Convert manually:

# Control the mix instead of letting the model average it
ffmpeg -i podcast.wav -ac 1 -ar 16000 -af "pan=mono|c0=0.5*c0+0.5*c1" mono16k.wav

Or – better, when you have isolated tracks – feed each channel as a separate file and diarize per-track. The community-1 model card confirms the averaging behavior; it’s documented, not a bug, but it’s a quality trap for multi-track recordings.

Also: strip music intros. WhisperX’s VAD handles silence well. It doesn’t know your bumper music isn’t speech.

Common errors

• No module named 'whisperx' – wrong package name. It’s whisperx, not whisper-x.
• Diarization returns SPEAKER_UNKNOWN – token invalid or community-1 license not accepted. The 3.1 license alone does nothing in v3.8.x. See Step 3.
• CUDA error: no kernel image is available – Pascal + torch 2.8 conflict. Re-pin to torch 2.7.1+cu126 as described above.
• 'soundfile' backend is not available – install libsndfile: conda install -c conda-forge libsndfile or apt install libsndfile1.
• OOM on a 12 GB card – drop --batch_size from 16 to 4, or switch to --compute_type int8 for ~30% VRAM reduction with minor accuracy impact.
• Language detection wrong – force it with --language en. Auto-detection on short clips is unreliable.

Upgrading and cleanup

In-place upgrades across minor versions are risky. PyTorch/CUDA pins drift between WhisperX releases and the failure mode is usually silent – diarization breaks without a crash. Full venv rebuild is cleaner:

# Quick upgrade attempt
pip install --upgrade whisperx

# Full rebuild (safer when crossing minor versions)
deactivate && rm -rf whisperx-env
python3.11 -m venv whisperx-env && source whisperx-env/bin/activate
pip install whisperx

# Uninstall + clear model cache
pip uninstall whisperx faster-whisper ctranslate2 pyannote.audio
rm -rf ~/.cache/huggingface/hub/models--pyannote--*
rm -rf ~/.cache/huggingface/hub/models--Systran--faster-whisper-*

That HuggingFace cache is the disk hog – several GB. Keep it if you’ll reinstall soon. Nuke it if you’re moving to a different ASR stack.

For the technical side of why this pipeline gets sub-100ms word timestamps where vanilla Whisper drifts by seconds, the WhisperX paper (arXiv:2303.00747, INTERSPEECH 2023) walks through the forced-alignment approach. Worth reading if you’re tuning for legal transcription or subtitle accuracy where timestamp drift matters.

FAQ

Does WhisperX work on Apple Silicon?

Yes – with --device cpu --compute_type int8. MPS acceleration for the faster-whisper backend isn’t reliable as of v3.8.6, so M-series users run CPU mode and accept slower processing than a modern NVIDIA card.

Why does diarization label two distinct people as the same speaker?

Most common cause: no --min_speakers / --max_speakers flags, and pyannote under-counted. Set them explicitly and re-run. If that doesn’t fix it, the two speakers may have genuinely similar voice embeddings – same gender, age range, accent, similar mic – and community-1 clusters by acoustic similarity, so it can’t separate them reliably. The practical workaround is recording each person on a separate track and diarizing per-track. That sidesteps the problem entirely rather than trying to untangle voices that were blended at capture.

Can I run diarization without WhisperX, just using pyannote directly?

Yes. pyannote.audio is standalone. You lose word-level alignment and integrated SRT output, but for speaker turns only, it’s a lighter install.

Next step: grab a 5-minute clip with known speakers, run the verification command, confirm the SRT has correct speaker labels. If the test fails, the failure mode tells you exactly which fix above applies.