Skip to content
ChatGPT & LLMs

DeepSeek Just Open-Sourced Its Inference Engine: Run DeepInfer Now

DeepSeek open-sourced DeepInfer on Oct 20, 2026. Build it, run offline V4.1-Flash inference, and dodge the three hardware/env traps that kill most first attempts.

6 min readIntermediate

What you’ll have running by the end

Goal: a working python examples/offline_inference.py on DeepSeek-V4.1-Flash. Not a repo tour. Clone → build against SM100 or Ascend 950 → one real offline generation from the HF checkpoint.

DeepSeek shipped the engine October 20, 2026. Apache-2.0. Repo: deepseek-ai/DeepInfer. It is the compute path behind their V4.1-Flash services – offline only in this drop.

Why the split looks the way it does

Earlier in 2025 they called a full engine dump impractical. This cut is cleaner: kernels, scheduler, parallelism, and the offline runner are public. KV cache, full VE/Prefill/Decode disaggregation glue, and DP replica load balancing stay internal. That is intentional, not an accident.

Open-sourcing a production inference stack is weirdly intimate. You get the ugly env vars and the force-pinned NCCL version – the stuff polished API blogs never show. Whether that intimacy is useful depends entirely on whether you can touch the listed silicon.

Build and run: the path that actually finished

Clean Linux box. Right GPUs. Empty Python 3.12 venv. Everything else fought me.

1. Prerequisites (hard gate)

  • Linux
  • GCC >= 13
  • Python 3.12 exactly, fresh empty venv
  • CUDA 13 on NVIDIA SM100, or CANN 9.2 on Huawei Ascend 950

No SM100 (Blackwell datacenter, e.g. B200) and no Ascend 950? Stop here. Consumer cards, SM120, earlier datacenter gens, laptops – unsupported per the README hardware section. Rent the box or use another runtime.

2. Clone and build

git clone https://github.com/deepseek-ai/DeepInfer.git
cd DeepInfer
# fresh Python 3.12 env already activated
export DEEPINFER_ACCELERATOR=nvidia # or ascend
./build.sh

build.sh pulls accelerator requirements, force-reinstalls NCCL 2.32.3 on the NVIDIA path, compiles vendored kernels (FlashMLA, DeepGEMM, DeepEP, DeepSelect), then DeepInfer itself. Leave DEEPINFER_ACCELERATOR exported in every shell that touches the engine – drop it and the wrong backend loads.

Submodule fetch dies on flaky mirrors more often than the compile does. git submodule update --init --recursive before retrying. The script rewrites git@ → https; that alone is not enough on a bad day.

3. Offline inference

Default checkpoint: deepseek-ai/DeepSeek-V4.1-Flash-1017 on Hugging Face (~510GB, MIT weights, multimodal, up to 1M context, ~552B MoE backbone with 8B active prefill / 16B decode). First run lands in the HF cache. After that, point DEEPINFER_HF_MODEL_DIR at a local copy.

export WORLD_SIZE=1 RANK=0
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 # match your SM100 count
python examples/offline_inference.py --ep 8
# mixes also work: --ep 2 --dp 8 or --ep 2 --cp 8

Script builds the LLM object, runs the sample prompts shipped under examples/, and can check structured outputs plus PPL baselines. Full flag list: python examples/offline_inference.py --help – see also the examples README.

4. Knobs that change behavior

Actually, the knobs worth touching before you chase throughput:

# DSpark speculative decoding (production defaults)
export DEEPINFER_ENABLE_MTP_VARLEN=1
export DEEPINFER_USE_NEXTN_DECODE=5 # set 0 to disable

# SWA bounded replay - trims decoder prefill work
export DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact
# pair with DEEPINFER_YOCO_* keep-token settings from README

# Batch-invariant / deterministic path - NVIDIA only
export DEEPINFER_BATCH_INVARIANT=1
# fixed seq_seed in SamplingParams; MTP off for full determinism

Structured output goes through SamplingParams(so_args={...}) – regex, JSON schema, or EBNF. There is a dedicated example file for it. Async scheduler stays on by default.

Common pitfalls that waste hours

Wrong Python minor or GCC < 13: C++ extension build dies immediately.

Lost DEEPINFER_ACCELERATOR export between terminals: silent wrong backend.

Parallelism flags that do not line up with visible devices and WORLD_SIZE: wrong sharding or hard crash. Read the examples README before inventing a topology.

Treating the repo as online-serving vLLM. It is not. Offline inference only. Bring your own KV cache system, disaggregation glue, and DP load balancers if you want a service.

Numbers DeepSeek published

As of the Oct 2026 release, their internal Q&A (doc/inference_deployment_qa.md) states one production cluster moves 155 trillion tokens per day and clears about US$38 revenue per GPU-hour at valley pricing. Those are DeepSeek’s figures on release day – treat them as dated until independent runs show up.

Repo charts cover prefill/decode scale-up on SM100. DSpark defaults to nextn=5 with confidence-scheduled verification; public discussion assumes high draft acceptance. Turn it off with DEEPINFER_USE_NEXTN_DECODE=0 if you are debugging correctness, not speed.

Acknowledgments in the README are blunt: built primarily on vLLM ideas, with SGLang and TensorRT-LLM inspiration. That lineage shows in the flag surface.

When you should skip DeepInfer

No SM100 or Ascend 950 access → skip. Use vLLM, SGLang, llama.cpp, or a hosted API for V4.1-Flash. The hardware gate is not a suggestion.

You need a chat endpoint today → official DeepSeek API (or any host already serving the model) is faster to stand up. DeepInfer is for the exact offline production path, or for reading the kernels and scheduler.

You wanted every DeepSeek checkpoint → only V4.1-Flash is listed. Future models are promised for this repo; they are not here yet.

What most readers actually gain is not a day-one full engine run. It is the DSpark verification schedule, the SWA replay truncation pattern, and the async batcher – then porting those ideas into a stack you already trust.

FAQ

Is DeepInfer the exact engine DeepSeek runs in production?

It powers V4.1-Flash services on the compute path. KV cache, full disagg, and cross-replica balancing stayed private. Real engine core – not a 1:1 service clone.

Can I run this on a single RTX 4090 or a Mac?

No. SM100-class or Ascend 950 only. Checkpoint alone is hundreds of GB. For a laptop experiment, grab a distilled or quantized community build and run it under vLLM or llama.cpp instead – you will learn more in an hour than fighting an unsupported backend all weekend.

How do I match official API tokenization behavior?

People assume DeepInfer handles encode/decode the same way the public API does. It does not; it focuses on model execution. Same-day companion package deepseek-recipe implements the production token encode/decode path (special tokens, streaming alignment). Install that alongside if third-party serving must stay bit-compatible with DeepSeek’s API behavior.

Next step: Python 3.12 venv on a machine that actually has the listed GPUs. Export DEEPINFER_ACCELERATOR. Run ./build.sh. Execute offline_inference with a small –ep. Publish the first successful generation log – independent runs are still scarce.

Related Tutorials