AIdataNewsGlobal LLMs Report: vLLM 0.31 Constrained Decoding Lifts Kimi K2.6 Tool-Call Schema Accuracy From 92% to 100%

An agent that cannot address its own tools cannot work — and in production the line between a dependable deployment and a flaky one is often a single server flag rather than a bigger model. The open-weight serving numbers now on the record make that concrete: running Kimi K2.6 on vLLM version 0.28 reproduced the expected tool-call schema in 92% of responses, i.e. 2 invalid calls for every 25 attempts, until constrained decoding was switched on as the server-side option shipped in vLLM 0.31, a capability that can bring schema accuracy to 100% for newer Kimi and DeepSeek checkpoints.

Read this first: the measured signal

Dimension	Free sampling (vLLM 0.28-era default)	Constrained decoding (server-side flag in 0.31)
Tool-call schema accuracy, Kimi K2.6	92% — 2 of every 25 calls malformed	100% for newer Kimi and DeepSeek models
Failure handling	Error text accumulates in the context window; agent retries	Invalid generation paths are blocked before they occur
Nature of the guarantee	Best-effort behaviour inherited from training	Runtime invariant enforced during sampling
Cost profile	Retry spend grows with session length	Negligible overhead when token masks are precomputed
Regression observed in the field	91% after a patched fork was replaced by upstream defaults	Restored to 100% once strictness was explicitly enabled

Every figure above is taken from the K2VV ToolCall benchmark workload and from the serving-side engineering account discussed later in this report; no external estimates have been introduced.

How a tool call reaches the model — and back

Exposing external functions to an LLM requires two moves: telling the model which calls exist, and telling it how to format them. In the OpenCode example, the harness sends Kimi a JSON array describing the available tools; one entry is webfetch, which retrieves web page content.

Within that array, each schema declares parameters, their types and required fields. The webfetch definition points to https://json-schema.org/draft/2020-12/schema, permits text, markdown or html, defaults to markdown, and documents an optional timeout in seconds capped at 120.

Before the array reaches the model, the inference engine wraps it in model-specific tokens so the shape matches what was seen in training — for Kimi, system tool-declaration tokens around the array.

Execution then runs as a loop. Asked to fetch a page, the agent may emit tokens such as functions.webfetch:0 followed by a JSON argument object. The inference server's parser extracts the call and rewrites it into an assistant message object carrying tool_calls. OpenCode or a comparable harness then executes webfetch and returns the output to the next turn.

Why 92% is a reliability problem, not a rounding error

The naive approach trusts post-training: let the model write calls freely and assume syntax holds. It does hold most of the time, because contemporary imitations of structured patterns are strong — and "most of the time" is precisely where agentic workloads bleed.

Each malformed call becomes an error message in the transcript, which is context the model must later reason around. Quality falls, latency and token spend rise because the agent retries one or more times, and multi-turn sessions amplify both effects.

Moonshot AI published the K2VV ToolCall benchmark so providers can demonstrate that they serve Kimi at original quality, valid tool-call generation included. Its 2000-request workload probes when an LLM decides to trigger a call and whether the payload is syntactically correct. Two metrics come out of it: trigger similarity, which compares a service's call-triggering decisions against Moonshot AI's own service, and schema accuracy, which measures the share of calls that conform to the schema and can be invoked successfully. A 50% score means half the generated calls are unusable.

The misses trace back to decoding mechanics: models rank candidate next tokens by likelihood rather than by running a compiler or type checker, so malformed calls can surface even under a clear prompt. A second driver is drift between training corpora and the live harness. Checkpoints absorb many tool datasets, each with its own tools, schemas and conventions, so a learned name can appear where it does not exist in production — search_web instead of web_search, or city_name instead of location.

Constrained decoding: making validity a precondition

Ordinary decoding computes probabilities for each possible next token and samples by weighted choice. Constrained decoding intervenes at that step, blocking tokens that would render the call syntactically invalid, so the engine guides generation stage by stage instead of sampling first and validating afterwards.

What counts as allowed is derived from the supplied tool schema. vLLM implements this with structural tags from XGrammar, which define a valid tool-call structure: where the call begins, where the index appears, where arguments start, and where schema-constrained JSON must be produced.

Legal generation therefore follows one path: Begin tool call → Tool name → Tool call index → Begin arguments → Schema-constrained JSON → End tool call. During the index phase, sampling is restricted to integer tokens and everything else is masked. The figure below, reproduced from XGrammar's article, visualizes that invalid-token masking.

Image source: Dong, Yixin, et al. "XGrammar: Flexible and efficient structured generation engine for large language models." Proceedings of Machine Learning and Systems 7 (2025).

The consequence is a category shift: schema compliance stops being best-effort behaviour of the weights and becomes a runtime guarantee. The model still chooses a token, but from a restricted, syntactically correct set. Because per-phase token masks are precomputed and applied efficiently on the GPU during generation, the enforcement adds negligible performance overhead.

Defaults decide the outcome: three classes of breakage

Moonshot AI recommends keeping constrained decoding on, and its own API behaves that way. Ubicloud first added the technique in its own fork of vLLM; once the capability landed upstream in v0.25, the team moved to the upstream implementation for Kimi and DeepSeek — and its tool-call accuracy score slid back to 91%.

Analysis showed vLLM's constrained-decoding implementation itself remained correct. What mattered was the changed default for requests that did not explicitly enable it, a posture driven by the engine's need to support a broad model catalogue. Supporting constrained decoding is a per-model task, and the problems fall into three camps:

• General model issues. For some models, unconstrained output diverges sharply from the tool schema or tool-calling format, so the constraint forces tokens the model would not otherwise select and leads it off course. Users of gpt-oss-20b, Qwen 3.6 and GLM 5.1 reported problems after the feature became a default.
• Grammar translation bugs. Translating a model's tool schema or calling format into that model's grammar can fail inside the engine, producing invalid tool calls, failed requests, silently dropped constrained decoding, or lower tool-calling accuracy.
• Feature interaction bugs. Constraints can interact incorrectly with other serving-engine features; a bug involving speculative decoding caused speculative tokens to be wrongly rejected by the grammar, leading to failing requests.

Because of these issues, vLLM maintainers reported seeing problems with several models once constrained decoding was enabled, and leaving it off by default became the safer option for a model-agnostic serving engine. The --tool-strict-level flag was later added so users can enable tool-call strictness for their own deployments.

A specialised host weighs the trade-off differently. Ubicloud's stated focus is serving Kimi and DeepSeek models at the quality users expect from first-party providers; after enabling constrained decoding, re-running the K2VV ToolCall benchmark and verifying 100% tool-call schema accuracy across all served model versions, the change was deployed to production. DeepSeek V4 Flash was among the deployments where visible quality degradation caused by tool-call schema inaccuracy had been observed.

Common misconceptions

• "Benchmark scores transfer to my stack." Observed LLM quality depends on model and inference engine together; changing the engine or its settings changes output quality.
• "A well-written prompt fixes formatting." Likelihood-based sampling can still yield malformed calls even when the schema is spelled out clearly.
• "Strictness costs throughput." With precomputed masks applied on the GPU, the overhead is described as negligible.
• "Flipping the flag is enough." Defaults, per-request parameters and per-model grammar support determine whether the constraint is actually active.
• "Retries are cheap." Each retry injects error text into the context window and depresses the turns that follow.

Action recommendations for teams serving open-weight models

1. Establish a tool-call baseline before tuning: run the K2VV ToolCall benchmark or an equivalent harness and record trigger similarity separately from schema accuracy.
2. Audit your engine version and its strictness defaults, and confirm whether constrained decoding is applied per request rather than assumed from a global setting.
3. When accuracy undershoots expectation, enable the explicit control (in vLLM, the --tool-strict-level flag) and re-measure before rewriting prompts.
4. Test model by model: one strictness default across a mixed catalogue can repair Kimi-class deployments while derailing other families.
5. Treat post-upgrade regressions as expected, and re-run the benchmark whenever you move from a patched fork to upstream code.
6. Monitor symptoms downstream, not only accuracy — retry volume, error strings accumulating in transcripts, and per-session token spend.

FAQ

What does constrained decoding change at inference time?

It removes tokens that would break the schema from the candidate set at each step, so generation proceeds only along paths the supplied JSON schema permits.

Does 92% schema accuracy simply mean 8% worse output?

No. That figure describes only the share of tool-call payloads that parse and execute. Because failures pile up as context inside multi-turn agent sessions, the observable degradation can exceed the headline gap.

Which configuration reached 100% in the reported tests?

The constrained-decoding path exposed as a server-side flag in vLLM 0.31; with it enabled for Kimi and DeepSeek, schema accuracy reached 100%, matching the direction of Moonshot AI's first-party API.

Should strictness be on by default for every model?

Not necessarily. It was disabled by default because of general model issues, grammar translation bugs and feature interaction bugs; per-deployment control exists for exactly that reason.

Where do the cited numbers originate?

From the K2VV ToolCall benchmark published by Moonshot AI, whose 2000-request workload measures trigger similarity and schema accuracy, plus the serving-side engineering account quoted in this report.

Bottom line

For platform teams and infrastructure buyers, the episode is a reminder that open-weight deployment quality is a systems property, not a checkpoint property. The same workload moved from 2 failures per 25 attempts to a clean pass through engine behaviour and one explicit setting — and the only way to know which side of that line a service sits on is to measure the schema, not merely the answers.

Written by David Nakamura for the LLMs desk at AIdataNewsGlobal (AI & Data News). Editorial contact: editorial@aidatanews.com.