DeepInfer Guide: Run DeepSeek V4.1-Flash Offline

Key takeaway: DeepInfer is not a cheap “any DeepSeek model” box. As of the October 2026 open release, it is DeepSeek’s own offline engine for one checkpoint family member – DeepSeek-V4.1-Flash – and only on NVIDIA SM100 or Huawei Ascend 950. Wrong silicon? You’re reading a design note, not starting a weekend install.

That undercuts the usual open-source reflex. A new inference repo drops and people smell a free API. GitHub noise was real (the repo cleared 10k stars in community coverage around 2026-10-20), yet the README stays blunt: production kernels and scheduling for offline runs, not a hosted stack you can point DNS at.

What actually shipped

Apache 2.0. Offline inference only. Default weights: deepseek-ai/DeepSeek-V4.1-Flash-1017 (override with DEEPINFER_HF_MODEL_DIR). Build bar from the DeepInfer README: Linux, GCC ≥ 13, Python 3.12, CUDA 13 or CANN 9.2.

~155 trillion tokens/day on one cluster and about US$38 revenue per GPU-hour at valley pricing show up in DeepSeek’s own inference deployment Q&A. Vendor claim; not third-party numbers. Same note sketches wide expert-parallel decode on the order of 64-128 GPU decode pools.

Tokenizer/protocol glue lives next door: deepseek-recipe (MIT) turns Chat Completions / Messages / Responses into DeepSeek-aligned prompts. DeepInfer runs tokens. Recipe shapes the chat.

Method A vs Method B

Method A – keep vLLM, SGLang, or TensorRT-LLM. Multi-model fleets, broader cards, OpenAI-shaped servers, plugins. DeepInfer’s README acknowledgments say it was built primarily on vLLM with ideas from SGLang and TensorRT-LLM. Mixed or older GPUs? A still wins.

Method B – DeepInfer offline, V4.1-Flash only. You trade coverage for DeepSeek’s wired path on supported accelerators: DSpark speculative decode (confidence-scheduled verification), SWA bounded replay, batch-invariant mode, deterministic sampling (NVIDIA), structured outputs via so_args on SamplingParams, and the EP/DP/CP layouts in the examples.

Closest match to DeepSeek’s V4.1-Flash serving path on the listed iron → B. Below is B only.

Offline path that ships

Empty Python 3.12 env. Old vLLM site-packages will fight you.

git clone https://github.com/deepseek-ai/DeepInfer.git
cd DeepInfer
export DEEPINFER_ACCELERATOR=nvidia # ascend on Ascend 950
# leave this set in every shell that runs DeepInfer - build-only export is a silent miss
./build.sh

First run pulls the default HF checkpoint. Air-gapped weights:

export DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017
python examples/offline_inference.py

Layouts called out in examples/README.md:

• --ep 8 – 8-way expert parallel on one node
• --ep 2 --dp 8 – expert + data parallel mix
• --ep 2 --cp 8 – context parallel split

DSpark varlen path when you want confidence-scheduled draft verification:

export DEEPINFER_ENABLE_MTP_VARLEN=1
export DEEPINFER_USE_NEXTN_DECODE=5 # max draft length; README default 5

V4.1-default SWA bounded replay:

export DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact
export DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140

Structured outputs: so_args={"schema_type": ..., "schema": ...} on SamplingParams (see examples/_structured_output.py). Sanity flag on the offline script: --check_structured_output for regex / JSON / EBNF on one LLM instance.

Pro tip: README Pareto curves used 4 SM100 for prefill and 64 for decode in one scale-up domain. Replay via examples/sweep/run_sweep.sh. Chart peaks often assume full draft acceptance – scale by the accept rate you measure before capacity math.

Batch-invariant mode: DEEPINFER_BATCH_INVARIANT=1 (NVIDIA only per README).

Edge cases tutorials skip

Want bit-stable samples? README rules are strict: DEEPINFER_BATCH_INVARIANT=1, DEEPINFER_ENABLE_MTP_VARLEN=0 (DSpark varlen off), fixed seq_seed on SamplingParams – NVIDIA only. Speculative max throughput and deterministic evals do not stack.

SWA keep-tokens is a correctness gate, not a free speed dial. Retained suffix must cover the sliding-window span plus max backtracking. Undershoot and the decoder literally sees a different history. V4.1 default pair in docs: compact mode + 140 tokens.

Upstream “not included” list is short and sharp: no KV cache system, no VE/prefill/decode disaggregation orchestration, no load balancing across DP replicas. Deployment Q&A pegs a practical efficient unit near 64 prefill + 64 decode GPUs when you chase their wide decode layout. Eight consumer cards in one chassis is not that unit.

Boring trap, still real: DEEPINFER_ACCELERATOR must stay exported at runtime. Fresh tmux panes drop it; failures look like mysterious backend misses, not a clean error.

Is withholding the storage fabric incomplete product – or a clean line between reusable compute and proprietary ops? You’ll feel that answer when you price the cache tier yourself.

FAQ

Can I replace the official DeepSeek API with DeepInfer alone?

No. Offline V4.1-Flash generation on listed accelerators only. HTTP, long-lived KV, multi-replica balance – you build those. Pair deepseek-recipe if you need protocol parity on the front door.

I only have H100/A100 nodes. Install anyway?

No – as of the public README, targets are SM100 and Ascend 950. Unsupported cards mean build pain. Stay on a general engine; skim DeepInfer docs for DSpark/SWA ideas worth porting later.

DSpark vs “just turn on MTP”?

DSpark’s varlen path drafts tokens and uses a confidence head so verification length can move instead of always checking fixed next-n. Flip DEEPINFER_ENABLE_MTP_VARLEN=1; cap drafts with DEEPINFER_USE_NEXTN_DECODE (default 5). Vendor write-ups talk fewer verified drafts per step at their concurrency. Your curve follows prompt mix and batch shape – don’t trust the Pareto operating point until you measure on your traffic.

Have SM100 or Ascend 950? Clean 3.12 env. Export DEEPINFER_ACCELERATOR. ./build.sh. Then offline inference with local weights via DEEPINFER_HF_MODEL_DIR and an EP layout from the examples. No hardware? Clone deepseek-recipe and keep the stack you already trust.