Your AI Prompt May Be Leaking Through Response Time

Access controls can stop one employee from opening another department's documents. They can prevent one customer from retrieving another customer's data. But those protections do not automatically cover every trace that sensitive information leaves behind while an AI model processes it.

One of those traces is time.

Large language models, or LLMs, do an enormous amount of repeated computation. To avoid performing the same work again and again, inference systems store intermediate results in a key-value cache, usually shortened to KV cache. This makes responses arrive faster, allows the same hardware to serve more requests, and reduces the cost of processing long prompts.

The security problem appears when different users, applications, or customers share the same inference infrastructure. If one request can reuse cached work created by another, a carefully timed request may reveal whether particular text was processed earlier. The attacker does not directly read the other user's prompt. Instead, the attacker looks for a difference between a fast response that reused cached work and a slower response that had to recompute it.

This kind of leak is called a timing side channel. Information is exposed through the system's behavior rather than through normal data access.

The important question is therefore not simply whether KV caching is enabled. It is who is allowed to benefit from the same cached computation, what that shared state reveals, and whether the cache crosses boundaries that the rest of the application is supposed to enforce.

When an LLM answers a request, it handles the prompt in two broad stages.

During prefill, the model processes the input text and calculates the internal attention states needed to understand the prompt. During decode, it generates the response one token at a time. A token is a small unit of text, such as a word, part of a word, or punctuation mark.

The model stores intermediate key and value tensors from this process in the KV cache. In plain English, the cache is a record of work the model has already completed. Reusing that record means the system does not need to recalculate the same prompt content for every generated token.

Some serving engines extend this idea across separate requests. Automatic Prefix Caching allows a new request to reuse cached work when its beginning matches text that was processed earlier.

That is useful because many production prompts contain large repeated sections. A company may attach the same system instructions to every request, ask repeated questions about the same document, or carry a long conversation history from one turn to the next. Reprocessing all of that text wastes GPU time.

Serving frameworks such as vLLM divide cached content into blocks that can be matched and reused later. Prompt construction may include much more than the user's visible question. It can also contain system instructions, retrieved documents, account information, tool output, dates, and conversation history.

That improves efficiency, but it also means the cache may contain traces of sensitive material.

Consider a healthcare system that sends the following prompt to a shared LLM service:

An attacker who uses the same inference environment may already know the surrounding template and have a short list of possible diagnoses. The attacker submits several versions of the prompt, changing only the suspected private condition.

If one version matches cached text from the earlier request, the system may reuse part of its previous computation. That version could begin producing an answer slightly faster than the others.

The attacker still does not see the victim's cache entry. The difference in response time simply acts as a clue. Security researchers sometimes call an observable clue like this an oracle because it helps answer a hidden yes-or-no question: did this candidate text match something already in the cache?

Repeated carefully, that process can become a prompt-reconstruction attack. Research on Automatic Prefix Caching has shown that attackers can test candidate continuations and use patterns of cache hits and misses to recover private fields or portions of a prompt.

The attack is easier when several conditions are present:

A hypothetical enterprise example makes the architectural issue clearer. Suppose a human-resources department uses a shared internal LLM cluster to analyze an unreleased restructuring plan. Another employee cannot retrieve the HR document because role-based access controls work correctly.

Both employees' requests, however, reach the same inference workers and prefix cache.

If the employee can reproduce likely fragments from the HR prompt and repeatedly measure the time to first token, the cache may reveal whether those fragments were processed recently. The document system did not grant access. The inference system leaked a clue through shared computation.

Encryption and conventional authorization still matter, but they protect different parts of the system. Encryption protects stored data and network traffic. Authorization controls who may retrieve or act on information. A timing side channel leaks information after an authorized request has already reached the model.

The risk should not be exaggerated.

An attacker generally needs access to an inference environment that shares the relevant cache state. The signal can also be weakened by network delays, unpredictable routing, high system load, continuous batching, cache eviction, different hardware, or requests landing on different workers.

The threat is therefore most credible in environments where an attacker can repeatedly probe the same or closely related infrastructure under reasonably stable conditions. Examples include shared enterprise clusters, adjacent containers, research clouds, or multi-tenant services with predictable routing and cache-sharing behavior.

This does not mean any internet user can extract arbitrary private prompts from every commercial AI service. It means infrastructure teams should not assume that application-level isolation automatically extends into the inference engine.

Retrieval-augmented generation, usually called RAG, lets an AI application search external documents and insert relevant passages into the model's prompt before it answers.

Traditional prefix-cache attacks require an exact match from the beginning of the prompt. A single different token near the start can prevent a cache hit. That limitation can make attacks more difficult in RAG systems because each user's prompt may begin with different system instructions, conversation history, or private context.

Newer architectures are trying to reuse cached document chunks even when those chunks appear later in the prompt. Frameworks such as LMCache and techniques such as cache blending can reuse a long retrieved document despite differences in the text that comes before it.

This can make RAG systems much more efficient. It also creates new timing behavior.

A June 2026 arXiv paper describes an attack called SpliceLeak against this kind of non-prefix cache reuse. In the researchers' controlled setup, fixed-size chunk boundaries and selective recomputation created step-like changes in latency. The attack first estimated the length of a hidden prefix, then manipulated cache boundaries to test candidate tokens.

The paper reported extraction success of up to 100 percent in bounded-entropy scenarios, meaning cases where the secret came from a limited or structured set of possibilities. Under one candidate-generation setup, the attack required as few as 63 requests per token. That figure came from testing 21 candidates three times each.

Those results are significant, but they need context. Open-ended text can have an enormous number of possible continuations. Larger candidate sets require many more requests, and real production noise can make the timing signal difficult or impossible to observe.

SpliceLeak is best understood as evidence of a structural weakness in certain cache-fusion designs, not as proof that every RAG service is currently leaking complete prompts.

Treating KV caching as an all-or-nothing choice leads to two poor outcomes. One is unrestricted sharing across users who should not trust one another. The other is disabling a valuable performance optimization everywhere.

A better approach is to decide where reuse is allowed and apply stronger defenses when cache state crosses a meaningful security boundary.

The clearest defense is to prevent unrelated users or customers from reusing one another's cache entries.

A trust domain is a group of requests that the system intentionally allows to share state. It might represent one customer, one organization, one application, one session, or another explicitly approved group.

Isolation can be implemented with separate caches, tenant-bound cache keys, session partitions, or an additional value included in the cache identity. Current vLLM documentation supports an optional per-request cache_salt. Requests can reuse blocks only when they provide the same salt, allowing reuse within an approved group while preventing cross-group cache hits.

This does not eliminate caching. Requests within the same tenant, application, session, conversation, or trusted group can still benefit from repeated content.

The tradeoff is lower reuse across the entire platform. Infrastructure teams should measure that cost rather than assume that strict tenant isolation makes caching impractical.

A March 2026 research paper proposed a system called CacheSolidarity. Instead of separating every user's cache entries at all times, it tracks which users interact with cached prefixes and isolates prefixes that appear to be involved in probing.

The authors reported up to 70 percent more cache reuse and 30 percent lower inference latency than defenses based on complete user-level isolation in their evaluated workloads. Those figures are experimental comparisons, not universal guarantees.

The design also has limitations. It cannot protect the first appearance of a prefix because there is no earlier interaction to flag. It may also fail when an attacker guesses a secret correctly on the first attempt. Its detection logic depends on an administrator-defined threshold for deciding when timing differences are suspicious enough to require isolation.

Selective isolation may preserve more performance, but it is not equivalent to keeping unrelated tenants separate from the start.

Some non-prefix cache systems may need defenses deeper in the scheduler or memory-management layer.

The SpliceLeak researchers proposed two mechanisms under the name SpliceDefense. Quantized Chunk Padding hides exact prefix lengths by aligning work to cache boundaries. Constant-Time Boundary Fusion makes boundary operations take roughly the same amount of time whether candidate text matches or not.

The goal is straightforward: a cache hit and a cache miss should not produce a timing difference that helps the attacker test a guess.

In the paper's evaluation, the defense reduced the exploitable timing difference to approximately zero across the tested extraction tasks. Padding added an average of 7.5 tokens per request and less than 5 milliseconds of mean latency on the evaluated NVIDIA A40 setup, which the authors reported as less than one percent of total prefill time.

These results are promising, but SpliceDefense remains a recent research proposal. Teams should not assume that their inference engine, cache connector, or managed provider already offers equivalent protection.

Any shared inference environment that processes sensitive prompts, retrieved documents, conversation histories, or tool output should include KV-cache behavior in its threat model.

A practical review should answer six questions.

Determine whether cached work is shared across requests, users, applications, departments, customers, containers, or physical GPU workers. Do not treat the phrase "multi-tenant isolation" as sufficient without knowing whether it includes the inference cache.

Confirm whether reuse is tied to a tenant, session, workload, application, or unguessable cache salt. A billing account or API credential does not automatically prove that cache entries are isolated at the serving layer.

Traditional prefix caching and arbitrary-position RAG chunk reuse behave differently. A defense designed for one may not address the other.

Test low-load periods, long repeated contexts, predictable templates, repeated candidate probes, different model sizes, and realistic co-located attackers. High average production load may add noise, but noise is not a dependable security control.

Near-duplicate requests, systematic changes to one token or field, unusual repetition, and latency-sensitive testing may support rate limits or detection. Monitoring should supplement isolation, not replace it.

Managed-inference customers should ask whether cache entries cross customer boundaries, whether isolation is enabled by default, which components participate in cache sharing, and whether timing-side-channel testing is part of the provider's security assurance process.

The central design question is simple:

If the answer is broader than the application's authorization model, the inference layer has created a second and weaker trust boundary.

KV caching is a practical part of modern LLM infrastructure. Long prompts and retrieved documents are expensive to process repeatedly, and caching makes useful AI services faster and more affordable.

The failure occurs when an implementation treats computational reuse as harmless even though it crosses boundaries that the rest of the system is designed to enforce.

Application authorization decides who may retrieve sensitive information. Cache policy must decide who may learn from the traces that processing leaves behind.

Shared caches deserve the same architectural scrutiny as shared databases, message queues, object stores, and memory pools. Performance state is still state. When one tenant can observe another tenant's state through timing, that state can carry secrets.