Deploy Moonshot Open Source Kimi K2-0905

End state first: an OpenAI-compatible HTTP endpoint on port 8000 serving Moonshot open sourceKimi-K2-Instruct-0905 (September 2025 K2 drop, 256K context) from a 16-GPU FP8 cluster under vLLM. Curl it. Point LangChain or OpenWebUI at it. Wire tools. Hardware and flags come after that picture – not a museum tour of MoE trivia.

Kimi K2 is Moonshot AI’s 1T-total / 32B-active MoE line (384 experts, 8 active per token, MLA attention). The training story sits in the Kimi K2 tech report (arXiv:2507.20534) – about 15.5T tokens with MuonClip if you care why the agentic behavior shows up. Instruct-0905 is the self-host checkpoint when you want this family, not the chase into later multimodal K2.x or K3. Weights: Hugging Face. Code and deploy notes: GitHub MoonshotAI/Kimi-K2.

What you’re actually standing up

Target stack as of late 2025 deploy docs (engines move – recheck pins before you freeze an image):

• Model:moonshotai/Kimi-K2-Instruct-0905 (~1.03 TB block-FP8 on disk)
• Engine: vLLM ≥ v0.10.0rc1 – official floor for K2
• Shape: 16× H200 / H20 / H800-class for full long-context FP8 (TP16 or DP+EP)
• API: OpenAI chat completions, served name kimi-k2
• Tools:--enable-auto-tool-choice --tool-call-parser kimi_k2

SGLang, KTransformers, and TensorRT-LLM also work per the official deployment guide. This piece stays on one deep vLLM path. Four half-finished install recipes help nobody at 2 a.m.

System requirements (honest minimums)

Shorter --max-model-len (8k-32k) and heavy community quants can shrink the box. Tool-call fidelity and long-context quality aren’t free when you leave FP8 – budget a real re-eval, not a vibes check.

Download the official weights

Grab the 0905 repo. The older 128K Instruct card is the wrong tree (HF banners 0905 as newer).

# Fast transfers help; optional
pip install -U "huggingface_hub[hf_transfer]" hf_transfer
export HF_HUB_ENABLE_HF_TRANSFER=1

huggingface-cli download moonshotai/Kimi-K2-Instruct-0905 
 --local-dir /models/Kimi-K2-Instruct-0905 
 --local-dir-use-symlinks False

# or snapshot_download in Python if you prefer

Long pull. Before you burn GPU time, confirm config.json shows "model_type": "kimi_k2", tokenizer files exist, and the safetensors shards are complete.

Install vLLM and stand up the server

Fresh venv. Serve line adapted from Moonshot’s guidance for the 0905 path and a two-node TP16 layout.

python -m venv .venv && source .venv/bin/activate
pip install -U pip
# Prefer current stable vLLM with CUDA backend matching your box
pip install -U vllm --torch-backend auto
pip install blobfile # do not skip - see errors section

export MODEL_PATH=/models/Kimi-K2-Instruct-0905
# Start Ray across nodes if TP spans hosts, then on the head:

vllm serve $MODEL_PATH 
 --port 8000 
 --served-model-name kimi-k2 
 --trust-remote-code 
 --tensor-parallel-size 16 
 --enable-auto-tool-choice 
 --tool-call-parser kimi_k2

Want higher batch with DP+EP? Install DeepEP/DeepGEMM and follow the multi-node sample in the official guide – --data-parallel-size 16, local ranks, --enable-expert-parallel, --gpu-memory-utilization 0.85.

Pro tip: Cap context while validating. --max-model-len 65536 (or lower) plus a sane --max-num-batched-tokens (8k-32k) saves activation memory and fails faster than a full 256K cold start. Raise only after health checks pass.

Single-node 8-GPU experiments show up in community write-ups (pipeline parallel, FP8 KV, heavy quant). Different reliability profile than Moonshot’s 16-GPU FP8 baseline – don’t confuse the two.

Client settings that matter

Once logs say “Application startup complete”:

1. Base URL http://<head-node>:8000/v1
2. Model name exactly kimi-k2 (or your --served-model-name)
3. Temperature 0.6 – HF/docs default for Instruct
4. Dummy API key if the client refuses empty
5. Default system line from the card is fine: “You are Kimi, an AI assistant created by Moonshot AI.”

License before production traffic: Modified MIT is permissive, but if the product or service crosses 100 million MAU or $20M monthly revenue, the UI must prominently show “Kimi K2” (repo LICENSE). Legal/product copy belongs in the launch checklist – not the week after a growth spike.

Verify it works

curl http://127.0.0.1:8000/v1/models

curl http://127.0.0.1:8000/v1/chat/completions 
 -H "Content-Type: application/json" 
 -d '{
 "model": "kimi-k2",
 "temperature": 0.6,
 "max_tokens": 128,
 "messages": [
 {"role": "system", "content": "You are Kimi, an AI assistant created by Moonshot AI."},
 {"role": "user", "content": "Reply with exactly: kimi-k2-ok"}
 ]
 }'

HTTP 200. Completion contains the marker. GPUs not stuck at zero. Then one tool-enabled request through your real agent use – parser bugs love to hide until then.

Common install errors and fixes

ImportError: blobfile is not installed during tokenizer init (tiktoken BPE path inside tokenization_kimi). Fresh vLLM images hit this often. Fix: pip install blobfile in the same env, restart serve.

invalid tool call parser: kimi_k2 means your build predates the parser. Docs floor is v0.10.0rc1+; older trees need an upgrade or a current source build. Without the parser, agent scaffolds mangle tool JSON.

CUDA OOM at startup or first long prompt. Drop --max-model-len, nudge --gpu-memory-utilization (down to survive, up toward 0.90-0.95 once stable), cut --max-num-seqs / batched tokens, or add nodes. Full 256K on an undersized cluster is the usual culprit.

Framework doesn’t know kimi_k2. Official workaround from deploy guidance “Others”: temporarily set "model_type": "deepseek_v3" in config.json (architecture reuses DeepSeekV3CausalLM shapes) and parse tools yourself when no kimi parser exists.

Upgrade, swap checkpoints, uninstall

Not an apt upgrade. Moving original Instruct → 0905 (or a later open K2.x you choose): download the new snapshot to a fresh directory, stop vLLM, repoint MODEL_PATH, restart with the same flags. Keep the old tree until smoke tests pass – multi-TB re-downloads hurt.

Engine bumps: upgrade vLLM in the venv, reinstall blobfile on clean images, re-hit /v1/models and a tool call. Read the vLLM changelog for MoE/EP flag renames before you copy last quarter’s scripts.

Cleanup: stop the process or systemd unit, rm -rf the model dir when you’re sure, drop the venv, reclaim Ray/temp under /tmp and the Hugging Face cache if disk is tight. No special Moonshot uninstaller.

Sixteen high-end GPUs for FP8 is a real bill. If your traffic shape is spiky or light, hosted Kimi routes or newer open checkpoints with friendlier quant paths may win on TCO. Run the spreadsheet before you renew the cluster – not after.

FAQ

Which checkpoint should I deploy in 2026 – K2-0905 or a later Kimi?

If you need a pure K2 Instruct line with known vLLM recipes, 0905 is the K2 instruct drop with 256K and stronger coding. Later Moonshot open models add multimodality, different hardware stories, or other licenses. Pick from your eval suite. Blog headlines are not an eval suite.

Can I run Moonshot open source Kimi K2 on one GPU?

Not full FP8 0905 at useful context. Community offload paths exist for poking around. Bad production agent backends.

Do I still need Moonshot’s API key for self-host?

No. Local vLLM serves your weights. Keys matter only for platform.moonshot.ai / kimi.com hosted routes. As of 2026 platform notes, older K2 API model IDs were on a retirement path – self-host is how you keep this exact checkpoint alive when the hosted ID goes away (platform quickstart).

Reserve the 16-GPU box. Start the HF download tonight. Land curl ... kimi-k2-ok before you wire agents.