If you've ever built something with ChatGPT, Claude, or Gemini's API, you've probably run into the term "prompt caching" in the pricing docs and skimmed past it. It sounds like a backend detail that only matters to infrastructure engineers. It isn't. Understanding it changes how you design AI applications, and it can cut your API bill by 50–90%.
This article explains what prompt caching actually is, why it exists, and how it works — with real examples, no assumed AI background required. Toward the end, we'll go one level deeper into the actual computation, for readers who want the engineering picture too.
Start with a problem you've probably felt
Imagine you're building a customer support chatbot. Every time a user sends a message, you don't just send that one message to the AI model — you send the entire conversation so far, plus a system prompt describing how the bot should behave, plus maybe a list of tools it can use.
So by message 10 of a conversation, your request to the AI looks something like this:
System prompt (500 words: "You are a support agent for Acme Corp...")
+ Tool definitions (300 words: refund_order, check_shipping, etc.)
+ Message 1 (user)
+ Message 1 (assistant reply)
+ Message 2 (user)
+ Message 2 (assistant reply)
...
+ Message 10 (user) ← the only actually new partHere's the catch: AI models have no memory. Every single request is treated as brand new. The model doesn't "remember" message 9 — you have to physically resend it, every time, forever, for the conversation to make sense. This means that by message 10, you're re-sending (and the model is re-reading) 9 previous exchanges just to add one new line.
Now imagine 10,000 users doing this simultaneously with your chatbot, all sharing the same 500-word system prompt and the same tool definitions. That's a lot of repeated reading happening constantly, for text that never changes.
This is exactly the waste that prompt caching is designed to eliminate.
The analogy that makes it click
Think about how you'd study for an exam using a textbook.
The first time you read Chapter 1, it takes real effort — you're parsing every sentence, building understanding from scratch. But if someone asks you a follow-up question about Chapter 1 five minutes later, you don't re-read the whole chapter again. You already processed it. It's fresh in your head. You just recall it instantly and answer.
Prompt caching gives the AI model a similar shortcut. If you send it the same chunk of text (like your system prompt) that it has very recently already processed, it doesn't need to "re-read and re-understand" that chunk from zero. It reuses the work it already did, and only spends real effort on the genuinely new part of your message.
The important nuance: the model isn't "remembering" in a human sense. It's not storing an opinion or a memory of the conversation. What's being reused is purely the computational effort of processing that specific block of text — like reusing a highlighted, annotated textbook instead of a blank one.
So what's actually being "cached"?
When an AI model reads your prompt, it doesn't process it all at once as a blob. It processes it token by token (a token is roughly a word or word-fragment), and for every token, it performs a chunk of internal math to figure out "how does this word relate to every other word around it." This step is called attention, and it's genuinely the most expensive part of running these models.
Here's the key fact that makes caching possible: for any block of text, once the model has done this "relate this word to its surroundings" math, the result never changes — as long as nothing before that block changes. If your system prompt is always identical, the model's internal understanding of that system prompt is always identical too. So instead of redoing that expensive math every single time, the AI provider's servers can save the result of that math the first time, and simply reuse it on every future request — as long as the text is the same.
That saved intermediate result is what's technically called the KV cache (short for "Key-Value cache" — a reference to the internal math structure, which we won't need to unpack for this article, but it's worth knowing the name if you read pricing docs).
Two important things about this cache:
It's prefix-based. The cached part has to be the beginning of your prompt, matching exactly, word for word. If you change even one sentence in your system prompt, everything from that point onward has to be recomputed — the cache only helps for the part that's identical.
It's temporary. Providers don't keep this saved forever. Typically it lives for about 5–10 minutes of inactivity, though some providers now offer longer retention (up to an hour or more) for an extra cost. If nobody hits that exact same prefix again within that window, the cache quietly expires and gets recomputed fresh next time.
Okay, but what is the model actually doing with that text? (The deeper technical layer)
This section goes a level deeper into the mechanics. You don't need it to use prompt caching effectively, but it makes the "why" click in a way that sticks.
Every word gets three "roles" assigned to it
When the model reads a sentence, it doesn't just see plain words. For every single word (technically, "token"), the model computes three different mathematical fingerprints of it, called Query, Key, and Value — Q, K, V for short.
Here's what that looks like as a diagram, for one single word:
Token: "cat"
|
[ embedding vector ]
|
┌─────────────────┼─────────────────┐
| | |
× W_Query × W_Key × W_Value
| | |
▼ ▼ ▼
Query("cat") Key("cat") Value("cat")
"what am I "what do I "what info do
looking for?" have to offer?" I actually hand
over if picked?"Every single token in your prompt goes through this same three-way split — one word in, three different-purpose vectors out. None of them are the "real" version of the word; they're three different lenses on it, each built for a different job in the next step.
A relatable way to picture this: imagine a room full of people at a networking event, and every person wears three name tags:
- Query tag — "Here's what I'm currently looking for in this conversation."
- Key tag — "Here's what I have to offer, as a topic someone else might be looking for."
- Value tag — "Here's the actual information I'll share if someone finds me relevant."
When it's your turn to speak, you look around the room holding up your Query tag, comparing it against everyone else's Key tag, to figure out who in the room is actually relevant to what you're trying to say right now. Then, you pull the actual content — the Value — from the people who matched best, and blend it into what you say next.
That's, quite literally, what happens inside every layer of the model, for every word, all at once. Take the sentence "The cat sat on the mat because it was tired." When the model processes the word "it," its Query is essentially asking "who am I referring to?" It compares that Query against the Keys of every earlier word — "cat," "sat," "mat," "tired" — and finds that "cat" is the strongest match. So it pulls "cat's" Value into its understanding of "it." This comparison-and-blend process, repeated for every word against every other word, is what people mean when they say a model is "paying attention" to context. It's not a metaphor — it's the literal mechanism, running as matrix multiplication under the hood.
Here's that same example as a diagram — the word "it" comparing its Query against every earlier word's Key, then blending in their Values by how well each one matched:
Sentence so far: "The cat sat on the mat because it ..."
|
Query("it")
|
┌───────────────┬───────────────┬────────────┴──────────────┐
| | | |
Key("cat") Key("sat") Key("mat") Key("tired")
| | | |
match: 0.82 match: 0.06 match: 0.05 match: 0.07
(strong!) (weak) (weak) (weak)
| | | |
▼ ▼ ▼ ▼
Value("cat") ×0.82 + Value("sat")×0.06 + Value("mat")×0.05 + Value("tired")×0.07
└────────────────────────────┬────────────────────────────────┘
▼
New meaning of "it" = mostly "cat",
a little bit of everything elseThe percentages (0.82, 0.06, 0.05, 0.07 — always adding up to 1.0) come from a step called softmax, which just turns raw "how well do these match" scores into clean, comparable weights. The point of the diagram: "it" ends up meaning "mostly cat" not because the model looked up a dictionary definition, but because "cat's" Key was the best match for "it's" Query, so "cat's" Value dominated the blend.
Why this gets expensive as conversations grow
Here's the relatable part: imagine a party where every new guest who arrives has to shake hands and exchange a "who are you, what do you know" conversation with every single person already in the room.
- 5 people at the party → 10 handshakes needed to fully connect everyone.
- 50 people → over 1,200 handshakes.
- 500 people → over 124,000 handshakes.
That's roughly how attention scales if you did it the naive way for every new word generated: each new word has to compare itself against every previous word, and as the conversation grows, this blows up quadratically — technically written as O(n²), meaning if the conversation doubles in length, the work more than doubles (it roughly quadruples). This is the mathematical reason very long conversations start to feel slower and more expensive per message — there's simply more "comparing against everyone in the room" happening.
Here's what that growth looks like laid out side by side, guest-by-guest, with and without a saved guest list:
WITHOUT caching (everyone re-introduces themselves to the whole room, every time)
Guest count: 5 50 500
Handshakes: 10 1,225 124,750 ← grows as n² (explodes)
WITH KV cache (only the newest guest introduces themselves; everyone else's
tag is already written down on the list)
Guest count: 5 50 500
New handshakes: 4 49 499 ← grows as n (steady, linear)
▲ ▲ ▲
(only handshakes involving the newest arrival)The gap between those two rows — 124,750 vs. 499 at 500 guests — is the entire reason this optimization matters at real-world scale. It's not a minor tweak; it's the difference between a conversation that gets dramatically slower the longer it runs, and one that stays roughly steady per new message.
Why K and V specifically get "saved," and why that fixes the problem
Now here's the trick that makes caching possible, and it comes from one simple fact: once a word's Key and Value tags are made, they never change, no matter what gets said afterward. "Cat's" Key and Value tag, computed the moment "cat" was said, stay exactly the same whether the sentence ends there or continues for another 500 words. Nothing that happens later in the conversation can retroactively change what "cat" is.
Query, on the other hand, is different — it's only relevant for the person speaking right now, in the current moment, and gets thrown away once that turn is done.
So the engineering shortcut becomes obvious: instead of making everyone re-introduce themselves (recompute Key and Value) every single time a new person speaks, you keep a running guest list with everyone's Key/Value tags already written down. When a new word/token arrives, it only has to write its own tag once, then it can instantly "read" everyone else's tags off the list, without asking them to reintroduce themselves. That guest list is the KV cache. It's the reason the actual, real-world scaling of generating each new word is closer to O(n) in practice — mostly linear, one new lookup per existing guest, not a repeated round of handshakes for the entire room every time.
Prompt caching (the API feature we've been discussing) is this exact same idea, just stretched across separate requests instead of within one ongoing generation. If your system prompt is the same 500-word block every time, its Key and Value tags never need to be rewritten — the server keeps that "guest list" saved for a few minutes and simply hands it back the moment it sees the same block walk in the door again.
REQUEST 1 (first message of the conversation)
[ System prompt ][ Tools ][ Message 1 ]
| | |
▼ ▼ ▼
compute K/V compute K/V compute K/V ← full cost, everything is new
└───────────────┴──────────┘
|
(saved to cache, ~5-10 min)
REQUEST 2 (next message, moments later)
[ System prompt ][ Tools ][ Message 1 ][ Reply 1 ][ Message 2 ]
| | | | |
▼ ▼ ▼ ▼ ▼
CACHE HIT CACHE HIT CACHE HIT CACHE HIT compute K/V
(reused, (reused, (reused, (reused, ← only the new
~10% cost) cheap) cheap) cheap) part is paid
for in full
└───────────────┴──────────┴───────────┴────────────┘
|
same list, just one new guest added,
saved again for next timeEverything shaded as "reused" above didn't need to be re-read or re-understood by the model — it was pulled straight from what got saved a moment ago. Only the newest sliver of the conversation, at the far right, costs full price. That's the entire mechanism, just repeated every time you send a new message.
Putting it together with a number
Say your model has, roughly, dozens of these Q/K/V comparisons happening per layer, and dozens of layers stacked on top of each other (this is genuinely how large models are structured — 30 to 80+ layers is common). Every one of those layers is running this "compare Query against everyone's Key, blend in everyone's Value" process, for every single token. Multiply that across a 3,000-token system prompt, and you can see why re-doing it on every single chat message — for every single user — becomes a real, measurable cost. And why not re-doing it, via caching, is worth a 90% price cut from providers: they're genuinely saving that much repeated computation.
A concrete before/after example
Let's say your chatbot has:
- A 2,000-token system prompt describing the bot's personality and rules
- A 1,000-token block of tool definitions
- A growing conversation history
Without caching, every single message the user sends forces the model to re-read all 3,000+ tokens of setup, plus the whole history, before it can even start thinking about the new message.
With caching, the first message pays that full cost once. Every message after that only pays for:
- The new user message
- Reusing the already-processed system prompt + tools + prior history (near-instant, much cheaper)
Providers price this difference dramatically. As an illustrative example, one major provider charges roughly a small premium the first time a chunk gets cached (since storing it costs them something), but then charges around 90% less for every subsequent request that reuses it. If your app has any repeated structure at all — a shared system prompt across users, a long document being asked about repeatedly, a multi-turn conversation — the savings compound fast.
This is also why caching improves speed, not just cost. Skipping the re-processing step means the model can start generating its answer sooner — providers report cutting time-to-first-word by up to 80%+ on long prompts.
Where this really pays off (real scenarios)
- Customer support / chat assistants — same system prompt and tool list on every single user turn, across every single user. Massive repeated prefix.
- "Chat with your PDF" tools — you paste in a 20-page document once, then ask multiple questions about it. The document text becomes the cached prefix; only your questions change.
- Coding assistants — a large codebase or set of instructions gets reused across many small follow-up edits.
- Agents that call tools repeatedly — every tool definition gets resent on every step of a multi-step task; caching keeps that from being re-billed every step.
Where it does not help: one-off requests, prompts that are different every time with no shared prefix, or conversations where the system prompt/instructions change on every single call.
Common misconception, worth stating clearly
People sometimes assume prompt caching means the model is becoming "stateful" — that it's starting to genuinely retain memory of you between sessions. It isn't. This is purely a performance optimization sitting underneath the model, invisible to it. The model's actual behavior, its answer, is (in principle) identical whether the cache was used or not — caching only changes how fast and how cheaply that identical answer gets produced. You still have to manage and resend your own conversation history and context every time; caching doesn't remove that responsibility, it just makes the resending cheaper when the resent part is unchanged.
It's also worth knowing that this cache is kept private and isolated — your organization's cached data isn't shared with, or accessible to, other customers, even if by coincidence they happen to use a very similar prompt.
For the applied AI engineers reading this: a bit more depth
If you're actually building with these APIs, here's what matters practically:
It's prefix-order sensitive. Put the stuff that never changes first (system prompt, tool definitions, static reference documents), and put the stuff that changes every turn (the newest user message) last. If you interleave static and dynamic content, you break the matching prefix and lose the cache benefit for everything after the change.
There's a minimum size. Most providers require a cached block to be at least ~1,000 tokens before it's worth caching — trying to cache a 50-token system prompt won't do much.
Explicit vs automatic caching varies by provider. Some providers cache automatically behind the scenes with no configuration needed. Others require you to explicitly mark where the cacheable portion of your prompt ends, using a parameter in your API call. Check your specific provider's docs — this changes how much control (and how much savings) you actually get.
Cache lifetime is short by default. If your app has bursty, spaced-out traffic (e.g., a user comes back every 20 minutes), you may fall outside the default cache window and pay full price again. Some providers let you pay a bit more for extended retention if this matters for your use case.
It composes with your own conversation-loop design. If you're manually managing multi-turn context (appending history, tool results, etc. and re-sending it), structuring that history to stay as a stable, append-only prefix (rather than editing/reordering past messages) is what keeps cache hits high turn after turn.
The one-sentence summary
Prompt caching is the AI provider saying: "If you send me the same beginning again, I won't waste time and money re-understanding it from scratch — I'll just remember the work I already did, for a little while, and pick up right where that left off."
It doesn't make the model smarter, and it doesn't give it real memory — it just makes repeated, structured conversations dramatically cheaper and faster, which is exactly the shape of workload most real AI products actually have.