The JSON tool schema is wrapped in <|im_system|>tool_declare tokens before being fed to the model.
VLLM TOOL-CALL ACCURACY FIXED WITH CONSTRAINED DECODING
*Senior engineers expose a 92% success rate for Kimi K2.6 tool calls on vLLM 0.28 and show how vLLM 0.31 restores perfect schema adherence. The fix matters for any multi‑turn AI agent.*
On October 20, 2026, senior engineers Jesse Donkervliet and Junhao Li published a stark assessment of tool‑call accuracy in open‑weight LLM deployments. They documented that serving the Kimi K2.6 model with vLLM version 0.28 yielded the correct tool‑call schema only 92 % of the time—equating to two failures in every twenty‑five attempts. In agentic workloads, where multi‑turn interactions depend on precise tool usage, such slip‑ups corrupt context and drive down output quality. The authors traced the problem to a lack of schema enforcement in the inference engine, a shortfall shared by many providers. Their remedy arrives in vLLM 0.31, which introduces a server‑side flag for constrained decoding, a technique that forces every generated call to match the declared JSON schema. Enabling the flag restores 100 % schema adherence for newer Kimi and DeepSeek models, promising a decisive step toward reliable AI‑driven tooling. The post also outlines how the JSON tool description is wrapped in LLM‑specific tokens before inference, ensuring the model receives the schema in the format it was trained on.
Why did Kimi K2.6 miss tool calls on vLLM 0.28?
The authors measured tool‑call schema success at 92% when serving Kimi K2.6 with vLLM version 0.28. That translates to two failed calls for every twenty‑five attempts. The shortfall stems from the inference engine’s inability to enforce the JSON schema during generation. Without a guard, the model can drift into malformed output, especially when the prompt includes complex tool declarations. The problem is not unique to Kimi; many providers and widely‑used inference engines exhibit similar gaps in tool‑call accuracy tests.
The cost of inaccurate tool calls
When tool calls stray from the declared schema, the model builds on nonsensical context. In agentic workloads that rely on multi‑turn sessions, each erroneous call contaminates the dialogue history, degrading answer relevance and coherence. Developers observe lower‑quality results, higher retry rates, and wasted compute cycles. The authors note that such failures erode the promised performance of open‑weight models, making them unreliable for production‑grade AI assistants.
Constrained decoding: how it guarantees schema adherence
vLLM introduced constrained decoding as a technique to lock generation to the provided JSON schema. Released as a server‑side flag in vLLM 0.31, the feature forces every token sequence to match the declared tool‑call structure. Enabling the flag lifts schema accuracy from 92% to 100% for newer Kimi and DeepSeek models. The approach eliminates the drift that caused earlier failures, delivering deterministic, schema‑compliant tool calls across all inference requests.
Steps to enable the fix in vLLM 0.31
To activate perfect schema adherence, operators must set the constrained decoding flag on the vLLM server running version 0.31. The workflow begins by sending a JSON array that describes each external tool, such as the webfetch function with its URL, format, and timeout parameters. The inference engine then wraps this JSON in LLM‑specific tokens—<|im_system|>tool_declare<|im_middle|>…<|im_end|>—before feeding it to the model. With the flag on, the model can only emit calls that satisfy the JSON schema, guaranteeing 100% accuracy for each request.
Frequently asked questions
What tool-call accuracy did Kimi K2.6 achieve on vLLM 0.28?
It achieved 92% schema accuracy, meaning two out of every twenty‑five tool calls failed.
How does constrained decoding improve schema adherence in vLLM?
Constrained decoding forces generated output to match the declared JSON schema, guaranteeing 100% schema accuracy when the server‑side flag in vLLM 0.31 is enabled.
Which vLLM version introduced the server‑side flag for constrained decoding?
The server‑side flag for constrained decoding was released in vLLM version 0.31.
The fix is a clear signal that tool‑call reliability can be engineered, not left to chance. By adopting constrained decoding via vLLM 0.31, providers can eliminate the 8% error margin that has plagued open‑weight agents. The move sets a new baseline for AI safety in production environments and forces the broader ecosystem to prioritize schema enforcement as a core feature.
Sources: [Fixing Tool-Call Accuracy in vLLM – Ubicloud Blog, October 20, 2026]