Prompt Caching and Context Optimization in Coding Agents
How structural optimizations are scaling agentic workflows.
What Is Prompt Caching
I was checking my usage dashboard in Cursor recently and noticed a distinct metric for “cached read”
Agentic coding tools pass massive amounts of context to language models—system instructions, workspace rules, file contents, and conversation history. Since LLMs are inherently stateless, they normally process this entire payload from scratch on every single request.
Prompt caching is a mechanism that saves the intermediate computations—specifically the output of the transformer stage. When a subsequent request contains the exact same sequence, the model retrieves the cached states instead of recalculating them. This optimization applies strictly to input tokens. The time for generating output tokens remain unchanged, as the model still has to predict the response sequentially.
How Input Is Processed in an LLM
To understand where caching fits in, you have to look at how coding tools construct their payloads. They do not just send your immediate question to the model; they build a large composite prompt. Tools structure these prompts by placing the most static information at the top and the most volatile information at the bottom.
The structure typically looks something like this:
<system_instructions> You are an expert coding assistant... </system_instructions>
<project_rules> Use functional React components... </project_rules>
<file_context_1> ... </file_context_1>
<file_context_2> ... </file_context_2>
<conversation_history> ... </conversation_history>
<user_input> Refactor this function to handle edge cases. </user_input>
Once this sequence is submitted, the transformer architecture takes over. The text is tokenized, and the model begins calculating the attention matrices. For every token, it computes the Key (K) and Value (V) states, mapping how each token relates to every other token in the sequence. This process is quadratic in complexity which means processing a massive context window of file contents requires significantly more compute than a simple, short query.
It is this fully processed, “attention-mapped” KV state that the model then relies on for output generation. The model does not reference the raw input prompt while formulating an answer; it refers to these transformer output matrices to predict and generate the response, one token at a time.
While LLM providers generally charge more for output token generation, the sheer size of the input context that modern day AI Coding Agents are providing makes it worthwhile to consider any possible cost saving on the input stage as well.
How Caching Helps
When you ask a follow-up question in the same session, the system instructions, project rules, and file context remain identical. The prefix matches. Instead of recalculating the entire attention matrix from scratch, the model hits the cache for the bulk of the payload. Only the new user input appended at the very end requires fresh computation.
Most of the LLM providers are implicitly optimizing for this compute overhead. This structural optimization makes long-context interactions financially viable for tool builders and noticeably faster for users, drastically reducing the time to first token on heavy requests.
For reference, here is Google AI Studios documentation on how their models use context caching : https://docs.cloud.google.com/vertex-ai/generative-ai/docs/context-cache/context-cache-overview
Rules For Caching And What Breaks It
The reliance on prefix stability introduces clear limitations. As soon as the system encounters a token that diverges from the cached version, the cache breaks, and all subsequent tokens must be processed normally.
Here is what breaks the cache in practice:
Dynamic injections: If a tool dynamically injects a varying variable like a timestamp at the top of the prompt, the prefix changes immediately, rendering caching useless.
Context reordering: Constantly changing the sequence of loaded context files prevents prefix matching.
Upstream file edits: If you edit a file that is placed early in the prompt structure, the cache is invalidated for that specific file and every single token that follows it.
Note that in most systems, caching works on exact token-prefix matches, though some platforms expose explicit cache handles or partial reuse mechanisms, like in the Google docs I linked above
An interesting gotcha: Why appending doesn’t break the cache
At first glance, caching seems structurally impossible. If attention maps every token to every other token, shouldn’t appending a new user prompt at the bottom ripple backwards and change the attention weights of the system instructions at the top?
It would, if models looked in both directions. But modern generative LLMs are decoder-only and use causal (masked) self-attention. Tokens can only look backwards. Information only flows left to right.
Because of this, we are not actually caching a fragile, easily broken global attention matrix. We are caching the Key (K) and Value (V) tensors of the prefix sequence (the KV Cache). The K and V states of the early context are calculated once and never change, regardless of what user prompt you append at the end. The model simply takes your new tokens, generates their Queries, and runs them against the heavily cached Keys of the past.
Ending Note
Prompt caching is just one part of the cost optimization playbook. Instead of stuffing every file into the context window and hoping for a cache hit, coding agents also need ways to selectively retrieve only the most relevant code. As I myself try to optimize the cost of vibe coding, next week I will look at how these tools use RAG and embeddings to build queryable views of a codebase and what sort of cost implications does that have. See you on the next one!
By the way, if you want to understand transformer architecture better, I would strongly recommend going through this article by Grant Sanderson. It has some amazingly well crafted videos to help cement your understanding of what is actually a pretty complex subject
