QWE AI Academy Here’s the unpopular take: for most people, running DeepSeek V4 Flash on a single AMD MI300X is a worse idea than paying DeepSeek’s API – and it’s still worth doing anyway. Not for cost. For the control you get when someone hands you a 284B-parameter open model and one GPU that can actually hold it.
The bring-up went viral on Hacker News because it’s the first genuinely credible “single-card, whole-model” recipe for a frontier-class MoE. It’s also a landmine field of half-merged PRs, contradictory release notes, and FP8 dialect mismatches nobody warns you about. This guide walks the working path.
The key takeaway (read this first)
You can serve DeepSeek V4 Flash from one MI300X today. You cannot do it with stock vllm/vllm-openai-rocm:v0.21.0. You need an overlay of unmerged PRs plus a very specific FP8 KV cache dtype. Skip either and you’ll waste a day.
If you’re doing this to save money on your own inference – don’t. If you’re doing it to learn the ROCm stack, resell tokens, or de-risk your dependency on closed APIs – read on.
What just dropped, briefly
DeepSeek V4 Flash is a 284-billion parameter sparse MoE with only 13B parameters active per forward pass, a native 1M-token context window, and three reasoning-effort modes. Benchmarks from the DeepSeek-V4 technical report put it at MMLU 88.7 (5-shot), MMLU-Pro 68.3, and AGIEval 82.6 – strong numbers for a model activating roughly a third of its total parameters on any given token.
The MI300X ships 192 GB of HBM3 per card – 2.4× the H100 SXM5’s 80 GB, per the Ryan Zhou repo README. FP8 weights land at ~156 GB. On an H100 you’d be doing 4-way tensor parallelism. On an MI300X it’s just… loaded.
Before you rent anything: verify your target provider actually gives you the whole card. Some virtualized MI300X instances split the GPU – you’ll see “MI300X VF” in
rocm-smi– and that changes what fits.
Method A vs Method B: stock vLLM or the overlay repo
There are two ways people are trying this. One of them is a trap.
Method A – Stock vLLM ROCm image
Pull vllm/vllm-openai-rocm:v0.21.0, run the standard DeepSeek recipe, expect it to work because the release notes said “DeepSeek V4: AMD/ROCm support.” It won’t. The same project’s canonical recipes YAML marks mi300x, mi325x, and mi355x as unsupported for both DSv4-Flash and DSv4-Pro – release notes and recipes contradict each other in the same repo.
What’s happening: PR #41946 (“[ROCm] [DSV4] Add aiter mhc support”) was merged to main on 2026-05-13 but did not make the v0.21.0 cut, and the remaining AMD fallbacks live in open PR #41136 (both tracked in vLLM issue #42876). The cost, per the issue author, is about half a day of triage on real hardware before you figure it out.
Method B – Ryan Zhou’s overlay repo
The ryanzhou/deepseek-v4-flash-mi300x repo pins the exact vLLM image SHA, model revision, and carries the unmerged AMD fixes as overlays. It builds on Fergus Finn’s June 2026 bring-up worklog at Doubleword – the one that surfaced the FNUZ-vs-OCP FP8 dialect issue, AITER gaps on gfx942, HIP-graph hazards, and MoE routing bugs.
Which wins?
| Stock vLLM | Overlay repo | |
|---|---|---|
| Works today (as of Aug 2026) | No | Yes |
| Long-term maintenance | Yours, once it merges | Manual rebase |
| Setup time | ~4 hrs of debugging + failure | ~5 min of model load |
| KV cache dtype handled | Wrong default | fp8_ds_mla configured |
Method B. Not close.
The actual walkthrough (Method B)
1. Get the hardware right
235 GiB of CPU RAM – that’s the part that catches people off guard. The GPU (one MI300X, gfx942, 304 CUs, 192 GiB HBM) holds the model weights fine. But the CPU KV tier the recipe assumes needs that RAM headroom on top. Add ~500 GB disk (model cache alone is ~156 GB), the AMD kernel driver, and recent Docker Compose. If your rental instance has 128 GB of RAM, this recipe will not start. Check before you commit to a rental window.
2. Pull the pinned image and model
VLLM_IMAGE='vllm/vllm-openai-rocm@sha256:e68d18b2ba50298661bfc49baf01158fbf036645c2362cccf3e8a7a79fe6c69a'
MODEL='deepseek-ai/DeepSeek-V4-Flash-0731'
REVISION='7872f01b1d1fe23eabc4c98b48bffcef5a386062'
docker pull "$VLLM_IMAGE"
docker run --rm --entrypoint hf
-v /root/.cache/huggingface:/root/.cache/huggingface
"$VLLM_IMAGE" download "$MODEL" --revision "$REVISION"Pinning the SHA matters. The stock :latest tag rolls forward and reintroduces bugs the overlay is patching around.
3. Verify overlays, then start
Copy the Caddyfile, chmod the entrypoint, then run sha256sum -c SHA256SUMS to verify the overlay files match before you boot. If the verify fails, stop – you have the wrong branch checked out.
cp Caddyfile.example Caddyfile
chmod +x vllm-entrypoint.sh
sha256sum -c SHA256SUMS
docker compose config -q
docker compose up -d
docker compose logs -f inference4. Confirm a healthy start
Healthy start: about 5 minutes, logs show Model loading took 156.67 GiB, DSpark draft model loading, and a non-zero GPU KV cache size. If model loading reports noticeably under 156 GiB, you’re loading a quantized fork by accident.
What’s actually running: fp8_ds_mla KV cache (UE8M0 block-scaled FP8 – not generic unscaled FP8), 256-token block size, DSpark speculative decoding with num_speculative_tokens=7. The FP8 dtype choice isn’t aesthetic. It’s the difference between correct output and garbage tokens on MI300X. More on that below.
Edge cases nobody warns you about
The FP8 dialect trap
AMD’s FP8 is FNUZ. Most FP8 checkpoints in the wild are OCP FP8. They are not interchangeable. If a tutorial tells you to pass a generic --kv-cache-dtype fp8 flag, you’ll get silent garbage output on MI300X – not a crash, garbage tokens. Fergus Finn’s June 2026 bring-up log at Doubleword is where this was first publicly documented for this specific model-hardware combination. The overlay repo’s fp8_ds_mla setting handles it. Any deviation from that specific string is a bug.
The CPU-KV cross-stream sync gap
The overlay carries an unmerged vLLM patch (community-tracked in the overlay repo, not yet in a tagged release as of August 2026). It addresses a missing cross-stream synchronization between the CPU KV load path and compute. Symptom in production: intermittent wrong-token generation under heavy concurrency, no error message. If you fork this repo and drop that overlay to “clean things up,” you’re reintroducing the bug.
SGLang is not the escape hatch
People tried. SGLang’s DSv4 rocm-dev image on MI300X currently fails with assert self.store_dtype == torch.uint8 (issue #25118, filed May 2026, persisting in the latest version at time of writing). The vLLM overlay path is the more reliable route on MI300X right now – the opposite of what’s true on Blackwell.
Which raises an honest question worth sitting with: AMD’s GPU hardware is genuinely strong on paper here. The software gaps are real but they’re closing – PRs are open, worklogs are public, the community is engaged. Whether ROCm tooling catches up to CUDA on serving workloads within the next 6-12 months is the actual bet you’re making when you invest time in this stack.
Does the math actually work?
$2.99/hr on Hot Aisle, $2.19/hr on RunPod – those are the MI300X on-demand rates as of July-August 2026 per GPUPerHour. Pinned full-time that’s about $2,153/month. A commenter in the HN thread on this setup estimates $3-$4/hr of token revenue at usable per-user throughput, implying roughly 100% margin when reselling.
Single developer? DeepSeek’s API is cheaper by an order of magnitude. Small team wanting data privacy and predictable cost? The number gets closer. Anyone building a bring-your-own-model inference service on ROCm without touching NVIDIA? This is the cheapest legitimate on-ramp that exists right now.
FAQ
Do I really need 235 GB of CPU RAM?
Yes, for this specific recipe. Technically you can disable the CPU offload path, but then concurrency drops and long-context requests OOM the GPU cache instead. Different trade-off, not a free lunch.
Will this recipe still work in a month?
Probably not without touching it. The overlay repo pins a specific vLLM SHA and a specific model revision – once PRs #41946 and #41136 fully land in a tagged vLLM release, the overlays become obsolete and the recipe should collapse into a vanilla docker run. Watch the vLLM release notes, and cross-check against recipes.vllm.ai before trusting them – that’s the exact contradiction that started this whole mess.
Can I fine-tune on this setup too?
No. Inference only. A 284B MoE needs room for optimizer state alongside weights – one MI300X doesn’t have it regardless of software.
Next step: Rent one MI300X for one hour (roughly $3), clone the overlay repo, and run the health-check logs. Don’t try to serve production traffic on your first boot. Just watch the model load, hit /v1/chat/completions once, and confirm you get coherent tokens. Everything else – autoscaling, quotas, resale – is a decision you make after you’ve seen it work.