Cloudflare Workers AI GLM-5.3 KV Cache Leak: 155 Responses, Two Root Causes, and a Hardening Checklist for Inference Operators — AIdataNewsGlobal General

An inference-layer fault inside a public AI platform has produced one of the season's most closely read infrastructure disclosures. The post-mortem, published on October 26, 2026 alongside a matching write-up from the SGLang project, describes a tenant-isolation defect in the GLM-5.3 Workers AI endpoint (@cf/zai-org/glm-5.3) that was live from August 27, 2026 until remediation on September 9. The company's own accounting puts the damage at 155 responses carrying material from other customers' sessions — every one of them a reply to a prompt longer than 681,280 tokens — with no sign that anyone deliberately exploited the flaw. Read past the headline count, the report is a compact lesson in how silently an inference engine can break the isolation guarantees a platform promises.

Case file at a glance

Item	Figure stated in the disclosure
Model in scope	GLM 5.3, served as @cf/zai-org/glm-5.3 on Workers AI; no other hosted model affected
Exposure window	August 27, 2026 to September 9, present from launch day
Affected output	155 responses
Trigger condition	Prompts above 681,280 tokens
Advertised context	1,048,576-token context window
Actual prefill KV pool	681,280 tokens
Model disabled after incident declared	within 40 minutes
Leak reproduced internally	within 25 minutes, using two independent accounts
Pool after the configuration fix	about 2.04 million tokens, roughly twice the 1,048,576-token context length
Signs of abuse	None found

How the fault surfaced and was closed

The chain starts outside the company. Gadi Evron, a security researcher at Knostic, filed a report through the vendor's HackerOne programme on September 8, 2026, stating that very long prompts sent to @cf/zai-org/glm-5.3 sometimes returned text belonging to other users' sessions. The bug was therefore caught by a bounty channel rather than by routine platform monitoring.

Engineers reproduced the behaviour the next morning. Within 40 minutes of declaring a security incident the model was taken offline, a corrected configuration and a patched inference engine were rolled out, and service resumed the same day, September 9. Stripping out the researcher's own activity and Cloudflare's internal test traffic, the remaining requests matched ordinary long-context use in both timing and volume, and the review turned up no evidence of deliberate exploitation. Because the write-up was released as a coordinated disclosure together with a post from SGLang, the open-source inference engine underneath the service, it covers the causing bug, the response timeline and the hardening applied to production SGLang deployments, plus continuing work with upstream maintainers on this and similar defects.

Two overlapping issues behind one leak

The report attributes the cross-request KV cache exposure to two overlapping issues — a configuration mismatch on top of an engine defect. For readers running their own serving stacks, the pairing is the most transferable part of the story.

Issue one: a KV cache pool smaller than the advertised context

Z.ai's GLM is a text-generation model aimed at agentic coding workflows, and it is one of many models reachable through Workers AI. Cloudflare deployed GLM 5.3 with a 1,048,576-token context window, while the KV cache pool on its prefill GPUs had room for just 681,280 tokens. The mismatch was known before launch: the expectation was that the engine would simply return an error for any prompt it could not hold, and long-context benchmark runs came back with successful HTTP status codes and streaming bodies containing no error frames, so the gap did not look like a problem. The chosen pool size is printed by SGLang during startup and also exposed as a metric.

The company now calls this a critical gap in its own testing: response codes and engine performance at long context were validated, but model output and coherence were not. The stated correction is to measure KV pool size against context length directly from the running engine before any disaggregated deployment change ships, instead of inferring agreement from status codes.

Issue two: a rejected request that still moved cache pages

The second fault lay in how SGLang treated over-length requests in disaggregated prefill/decode mode. A prompt that did not fit the pool was internally marked as rejected and swapped for a one-token placeholder; but rather than aborting and returning a client error, prefill generated the KV cache for that placeholder prompt and transferred it to the decode node. The decode node received a "transfer complete" message and assumed prefill had overwritten earlier cache contents with the full prompt for this request, when in fact most of the previous state was left intact. Generation then attended over stale cache pages that still held data from earlier requests — which is how replies to prompts longer than 681,280 tokens could carry content from other users' GLM 5.3 sessions.

Inside the engine, the rejection path behaved differently between single-node and split deployments. In v0.5.21 the SGLang team addressed the handling of prompts longer than the KV pool; before that, the scheduler rejected the over-length prompt and the rejection path (Req.set_finish_with_abort) recorded the error meant for the client while swapping in the placeholder so the request could drain through normal batch machinery without allocating real memory. On a single-node deployment that worked: the placeholder ran, the batch ended, the client got an error. In the disaggregated setup Cloudflare operates, the admission queues on prefill and decode tested whether a request was finished rather than whether it was rejected, and a rejected request reported itself as live. The published diagram traces the sequence:

1. Bootstrap. The router tells the decode node the prompt length — 8 pages in the diagram — and the decode node reserves 8 KV pages for the request.
2. Prefill admission. The prefill admission queue accepted anything whose req.finished() was False. A rejected request stashed its status in to_finish rather than finished_reason, so finished() still returned False and the rejected request was admitted.
3. Prefill forward pass. The prompt had already been replaced by the placeholder, so prefill ran a trivial forward pass on that single token.
4. KV transfer. Prefill sent one page of KV to decode and signalled that the transfer was complete.
5. Decode admission. Decode's admission queue carried the same finished() check, so it accepted the handoff.
6. Decode attention. Generation began and attention read all 8 reserved pages. Only page P0 had been written by this request; pages P1 to P7 still held what earlier requests left there — other users' prompts and partial generations.
7. Response. The caller got HTTP 200 instead of HTTP 400, prompt_tokens matched the long prompt and the reply read coherently. It arrived faster than an 8-page prefill should take, and it contained content from other users' KV pages.

Every request over the threshold behaved this way. GLM 5.3 was the only model hit, because every other disaggregated deployment on Workers AI has KV pools larger than its context length on both prefill and decode, so no request could reach the rejection path. The engine fix and the hardening described below have since been applied to every model served with SGLang, and deployment checks now require them for every new deployment.

Architecture context: shared pages and no zeroing

Workers AI runs GLM 5.3 on SGLang in a disaggregated configuration: a pool of prefill GPUs processes the prompt and produces the model's KV cache, then hands that cache over remote direct memory access (RDMA), via Mooncake, to a pool of decode GPUs that generate the response — an architecture the company had described earlier in its posts on high-performance LLMs and on smaller, faster, safer models.

The KV cache in GPU memory is a shared pool of fixed-size pages: each request is assigned pages on admission and releases them on completion, and pages are not currently zeroed between requests, since the next request to receive a page simply overwrites it. That approach is common among inference engines and is necessary for peak performance, but it makes the engine responsible for tracking ownership of those memory pages — handling shared ownership within a node, where KV cache offload and token generation touch the same memory on different lifetimes, and ownership that is technically shared with the remote prefill node, which holds a simultaneous mutable reference to the page. Pool size is not a configuration value either: SGLang sizes it at startup from whatever GPU memory remains after the model weights load, while the advertised context length is a separate setting, and nothing forces the two to agree. Checking that they do falls to the operator.

Disclosure timeline (all times UTC)

When	What happened
Aug 27, 2026	GLM 5.3 went live on Workers AI carrying a prefill KV pool smaller than its context window; the exposure was present from launch.
Sep 8, 2026 - 23:11	The HackerOne report arrived.
Sep 9, 2026 - 08:00	Verification of the report began.
Sep 9, 2026 - 13:22	Security incident declared; Product Security recreated the leak with two independent accounts within 25 minutes.
Sep 9, 2026 - 14:00	Scope confirmed, GLM 5.3 set to private, requests answered with HTTP 400, status page updated.
Sep 9, 2026 - 14:35	Root cause pinned to the prefill pool being smaller than the context length; a canary prefill node with a larger pool proved not susceptible.
Sep 9, 2026 - 14:52	All prefill nodes rolled to the corrected configuration.
Sep 9, 2026 - 19:04	Full deployment rebuilt with the corrected configuration plus a Cloudflare-authored engine patch that returns proper errors for requests rejected by the prefill scheduler; checked against the researcher's reproduction script.
Sep 9, 2026 - 19:40	Model restored to public.
Sep 10, 2026 - 23:30	Meeting with the SGLang maintainers — RadixArk's CTO and core engineers — to agree upstream fixes.
Sep 11-19, 2026	Additional hardening rolled out to all production SGLang deployments.
Sep 16-Oct 12, 2026	Upstream SGLang merges the admission fix, per-transfer checksums and abort-ACK preservation, and turns deferred KV release on by default.
Oct 2, 2026	SGLang v0.5.21 released.
Oct 19, 2026	SGLang v0.5.22 released.
Oct 26, 2026	Coordinated publication with the researcher and RadixArk.

Three steps taken before public access resumed

• Configuration fix. The pool was made large enough to hold the full context: aligning the GLM 5.3 setup with the GLM 5.2 configuration freed sufficient GPU memory that the prefill KV pool now holds about 2.04 million tokens, roughly twice the 1,048,576-token context length. A prompt that fits the context window now always fits the pool, so a valid request never reaches the rejection path — on its own, enough to keep the bug unreachable.
• Engine fix. The rejection path was also made safe, because relying on pool size alone is fragile: any change to quantization, tensor parallelism, CUDA graph sizes or a future model version shifts the pool in ways that are easy to miss, and other conditions could let the prefill scheduler reject a request and trigger the same bug. The patched engine retires a request carrying a pending rejection at the disaggregation admission queue, with its original error, before any bootstrap, allocation or KV transfer happens. This was proposed upstream as #38704, which additionally had the decode node verify that the number of tokens transferred matches the number it reserved. The equivalent fix was already on SGLang main as #36988, merged September 6 for a related failure path but not yet released; maintainers also merged #38935, which rejects these requests earlier, following the same approach.
• Validation. Both changes were checked against the researcher's reproduction script and Cloudflare's own canary-marker tests before the model was restored.

A wider bug class: KV cache memory safety

The report places this fault within a class of KV cache memory-safety bugs the company has been working for several months, each of them a case of a page being freed, reused or read while something else still has a claim on it. Two causes recur.

• Page lifetime inside a single node. Several tasks on a decode node can hold the same page at once — token generation, offload to host memory, the KV transfer receiver, and the abort and error paths. If one frees the page while another is still using it, a new request can be handed that page mid-read or mid-write. Earlier repairs for decode offload (#24226, #30899) and double release on grammar errors (#30937) were of this kind.
• Races across nodes. When a request is aborted while prefill is still writing its KV cache to decode over RDMA, the writes already sent cannot be recalled; if decode frees those pages immediately, they can be given to another request and the late writes land in that request's KV cache. Cloudflare contributed to fixing this race through proposed fixes (#31357, #38961) and a reproduction script (#32564). Both patterns end the same way: corrupted output, or one user's data in another user's response, with no error raised. The bug in this incident differed in that a rejected request went through the handoff every time rather than by a timing accident, but the result was the same — decode reading pages this request never wrote. None of the earlier issues covered this failure mode, where prefill rejected the request and decode never learned of it.

Hardening shipped since the incident

According to the disclosure, all of the following have been released to disaggregated production deployments:

• Deferred release extended to hold prefill's source pages as well, and to cover every failure type rather than aborts only — proposed upstream as #38961, superseded by #40645, #41402 and #41404.
• Bounded Mooncake transfer batches (#32758, from the SGLang community), so no single transfer call can outlast the release timeout.
• A load-balancer change so that a client disconnect no longer aborts a request mid-transfer: the prefill connection is held until prefill finishes and the decode connection until the first token, then both are closed.
• A gate on every disaggregated deployment change: KV pool size measured against context length, deferred release on both roles, transfer batch bounds inside the release timeout, and all required patches applied.

Upstream work with the SGLang team

Cloudflare met the SGLang maintainers at RadixArk on September 10, the day after the incident, and agreed that RadixArk would investigate the reported race conditions, take the integrity-checking idea upstream and build reproduction tests for this class of bug. The company notes that the same fault had already been reported to SGLang by another user as issue #30233 on July 6, 2026 — noticed while investigating this incident — and that this type of problem had been flagged with the project since v0.5.10; the security incident sped up collaboration that had already begun. The patch for this issue was released in SGLang v0.5.20, with follow-ups in v0.5.21 and v0.5.22, and is available to every other operator dependent on SGLang.

The items below have landed on SGLang main to address this incident and related KV cache race conditions; all but one are included in v0.5.22, and operators running disaggregated SGLang are advised to update to v0.5.22 or backport the patches as needed.

Category	Pull request	Release	What it does
Incident	#36988	v0.5.20	Aborted prefill requests are retired before their result reaches decode, which stops the exposure behind this incident.
Incident	#38935	v0.5.21	Rejects such requests at admission, ahead of any bootstrap, allocation or KV transfer, following Cloudflare's draft patch #38704.
Race	#39461, #41612	v0.5.21, v0.5.22	sgl-router cancels both halves of a request on client disconnect or prefill failure.
Race	#37454, #41023	v0.5.20, v0.5.21	Deferred KV release is enabled only on backends that support it, then switched on by default.
Race	#40645, #41402	v0.5.21	Recovers cases where the drain confirmation was lost, so releases happen on confirmation rather than waiting out the timeout.
Race	#41404	v0.5.21	Holds pages on every transfer failure, not only on decode-initiated aborts.
Race	#41618	v0.5.21	Fixes a regression in the abort notification path.
Race	#43719	v0.5.22	A failed Mooncake send no longer reports its writes as drained.
Race	#37077	v0.5.22	Tags abort messages per request so a reused ID cannot consume a stale confirmation; adds MORI support.
Race	#36612	v0.5.20	Shares the prefill-to-decode failure notification across Mooncake, NIXL and MORI.
Race	#32758	v0.5.20	Optional cap on Mooncake transfer batch size, so one timed-out call cannot leave an unbounded number of writes in flight.
Lifetime	#30899	v0.5.21	Decode waits for an in-progress offload to host memory before retracting a request and freeing its pages; authored by Cloudflare.
Lifetime	#43562	v0.5.22	Keeps tensor-parallel ranks in agreement when a host-memory cache restore fails.
Observability	#39500	v0.5.21	Adds --disaggregation-enable-kv-checksum so decode verifies a checksum of the transferred KV before generating; cleanup in #39729, merged and awaiting release.
Observability	#38596, #40642	v0.5.20, v0.5.21	Repairs to SGLang's KV canary, which detects and logs potential corruption.
Observability	#41128	v0.5.22	Metrics for deferred release: count, duration, and whether each hold ended on drain confirmation or timeout.

Follow-ups that remain works in progress:

• #39729 — additional cleanup for --disaggregation-enable-kv-checksum; merged, awaiting release.
• Zeroing KV pages on release, draft proposal #46435.
• Extending KV checksum validation from the prefill-to-decode handoff to every decode step, part of the "integrity by construction" line of work.
• Broader KV Canary and KV Checksum model and feature support.

Checklist for operators of disaggregated inference

The disclosure's advice to other operators can be reduced to a short checklist:

• Measure your KV pool and compare it to your context length on every prefill and decode node, reading the engine's startup log, after every change to weights, quantization, parallelism or memory fraction.
• If the pool is smaller, lower the context length to fit; do not rely on a proxy to enforce the limit.
• Update SGLang dependencies to v0.5.22 or later, or backport the listed patches.
• Validate model output and coherence at long context, not only HTTP status codes and performance — the gap Cloudflare itself identifies as the reason the mismatch survived pre-launch testing.

FAQ: what readers ask about the Workers AI leak

Which service and model were affected? Only GLM 5.3, served on Workers AI as @cf/zai-org/glm-5.3, during the August 27 to September 9 window. Cloudflare confirmed that no other model hosted on Workers AI was affected.

How much cross-customer exposure was there? 155 responses in total, all of them replies to prompts longer than 681,280 tokens, containing data from other customers' requests to the same model shortly before theirs.

Was the bug exploited? No evidence of deliberate exploitation was found. Excluding the researcher's activity and Cloudflare's own testing, the identified requests matched ordinary long-context use in timing and volume.

Why did affected customers see a normal-looking success response? Because prefill sent a "transfer complete" signal for the placeholder cache and decode assumed the pages had been overwritten, so the client received HTTP 200 rather than HTTP 400, with a coherent reply that arrived faster than an 8-page prefill should take.

Is self-hosted SGLang exposed? Deployments running disaggregated prefill/decode where the KV pool is smaller than the context length are in scope. The problem had been reported to the project as #30233 on July 6, 2026, similar issues had been flagged since v0.5.10, and the fix shipped in v0.5.20 with follow-ups in v0.5.21 and v0.5.22.

How fast was the incident contained? The report was reproduced the next morning, the model was made private within 40 minutes of the incident being declared, and public access returned the same day after configuration and engine patches were validated; Product Security had confirmed the leak using two independent accounts within 25 minutes.

Is the work finished? Not entirely. Deferred release, transfer batching and admission checks are in production, while page zeroing (#46435) and checksum validation across every decode step are still in progress upstream.

For an AI-industry audience the takeaway sits in the mechanics rather than the count: tenant isolation in modern LLM serving rests on invariants that no component enforces on the operator's behalf — pool size against context length, page ownership spread across two machines, and error paths that must stay error paths instead of quietly producing cache pages. The upstream list shows how much of that work is still being negotiated between a platform operator and an open-source engine project.

Reported for AI & Data News by David Nakamura. Corrections and tips: editorial@aidatanews.com.