Fixing Tool-Call Accuracy in vLLM Guide

One flag. Schema-valid tool calls you can ship.

As of vLLM 0.31 (late 2025), --tool-strict-level parameter is the line that moves newer Kimi and DeepSeek serves from the old ~92% schema-valid rate to 100% on the same style of checks Ubicloud ran. Malformed arguments stop landing in multi-turn agent history.

Unconstrained sampling is a lot like letting someone free-type SQL against a live DB: most statements work, a few quietly poison the next turn. Constrained decoding does not make the model “smarter” – it removes the keys that cannot appear if the output must stay inside the tool envelope and JSON schema.

Serve line (Kimi first)

vllm serve moonshotai/Kimi-K2-Instruct 
 --enable-auto-tool-choice 
 --tool-call-parser kimi_k2 
 --tool-strict-level parameter 
 --trust-remote-code
# add -tp N, --max-model-len, reasoning-parser as needed

Pin parser + chat template + model commit as one deploy artifact. Template drift still wrecks accuracy; the strict flag never fixes a bad prompt build.

parameter treats every tool as if the client set strict: true on the full JSON schema. function only locks the envelope and name. auto stays the default. Values and behavior are documented in the vLLM tool calling docs (as of latest 0.31 docs).

DeepSeek-family checkpoints need their matching tool-call parser and template from current vLLM recipes – not a copy-paste of the Kimi flags. Client-side, define tools the OpenAI way with additionalProperties: false and required/nullable properties spelled out. The server floor still helps when your agent kit never sets per-tool strict.

Verify before you roll traffic

1. Clone K2-Vendor-Verifier.
2. Point it at the new endpoint.
3. Run the ToolCall suite (~2000 requests per the project). Target: schema accuracy at or near 100%, plus strong trigger similarity to the vendor API.

Numbers still low? Dump raw completions. Empty content promoted to a list, or a missing add_generation_prompt, mangles the prompt before constrained decoding runs – so the flag looks “on” while the model never saw a clean tool turn. That mismatch is called out across the vLLM Kimi K2 accuracy write-up and the official tool-calling template notes.

Four pitfalls that still fail requests

• Speculative decoding + grammar – draft tokens get rejected and whole requests die. Ubicloud’s write-up on fixing tool-call accuracy in vLLM lists this interaction; disable speculation or re-test end-to-end.
• Kimi history IDs – past tool_call ids must look like functions.func_name:idx. Raw ids such as search:2 hit the parser’s strict split path, throw, and drop valid calls. Normalize on the client before multi-turn send (same Kimi accuracy post).
• Wrong parser/template pair – HTTP 200 with an empty tool_calls array. Match family-specific parser and tool-aware jinja; do not assume one flag set travels across model lines.
• parameter on the wrong family – forced schema tokens some Qwen 3.x, GLM, and gpt-oss-style stacks would not sample. Trigger quality drops even when JSON is “valid.” Stay on auto and retry in the agent loop for those.

Is 100% schema adherence enough if the model still picks the wrong tool half the time? Schema fixes the wire format; trigger similarity is a separate score on K2VV – watch both before you call the rollout done.

What the numbers actually were

100% schema accuracy on the newer Kimi/DeepSeek path with the 0.31 enforcer – that is the headline from Ubicloud’s production check, up from 92% (2 of 25 failed) on Kimi K2.6 under stock vLLM 0.28. Overhead? Negligible once XGrammar-style structural-tag masks sit precomputed on GPU.

Default stays off/auto on purpose: earlier default-on attempts broke model-specific paths and fought other features. Your checkpoint and history normalization still move the needle.

When to leave the flag alone

Skip parameter (and maybe function) when the model already drifts hard from its documented tool markup – early reports fingered certain Qwen, GLM, and gpt-oss variants. Skip if heavy speculative decoding is non-negotiable and you have not re-validated. Skip if tools are intentionally free-form and you want invented fields. Otherwise, for current Kimi/DeepSeek-style agent serves, add the flag, run K2VV, then ship.

FAQ

Does –tool-strict-level work with tool_choice=”auto”?

Yes. function / parameter turn on structural tags even when the client never sets per-tool strict. Named or required tool_choice already forced schemas before.

I set the flag but still see bad JSON – what next?

Picture a green deploy dashboard and agents still coughing syntax errors. Walk this order once: parser matches the model family; chat template is the tool-aware build (recent enough on Kimi); generation is not truncating mid-object. Re-run K2VV on a fixed seed set. Weird history ids? Normalize client-side before you blame the enforcer.

Is the overhead really zero?

For practical serving, Ubicloud’s numbers and the XGrammar integration story say yes – masks apply on GPU with almost no steady-state cost. The first request that builds a new grammar can hitch once on compile; after that the grammar is cached. If you run extreme batch sizes, measure your own p99 instead of trusting the brochure claim. Turns out “zero” means “you will not notice it next to model compute,” not “the profiler shows a flat line at all batch shapes.”

Next step: vLLM ≥0.31, --tool-strict-level parameter (or function) on models that tolerate it, K2-Vendor-Verifier against the new endpoint before traffic. An hour of bench time beats another week of silent context pollution.