StackLift
All terms

Glossary

Prompt caching

Reusing the already-processed start of a prompt across requests to cut latency and cost.

Prompt caching is a feature of many model APIs that reuses the work already done on an identical prompt prefix, such as a long system prompt or a large document, so repeated requests are faster and cheaper. The cached part must be the same, and must come first in the prompt.

It helps most when many requests share a large, stable block: instructions, tool definitions, a reference manual or a long conversation. Put stable content at the start and the changing question at the end, so the prefix stays identical.

Providers differ in how caching is switched on, how long entries last and how cached tokens are billed, so check the current documentation. Cached tokens are usually priced differently; see the LLM pricing page.

Start in 60 seconds

Put this to work on your project.

Get an AI-assisted, architect-reviewed scope, cost range, and timeline. Free, no login.

The complete guide

Prompt caching: the complete guide

Updated November 18, 2026 ยท Published by , Founder, StackLift AI. Written with AI assistance; how we make content.

Short answer

Prompt caching lets a model API reuse the work it already did on an identical start of a prompt, such as a long system prompt, tool definitions or a large document, so repeat requests are faster and cheaper. The reused part must be exactly the same and come first. It pays off when many requests share a big stable block.

What is prompt caching?

Every time you call a language model, it reads the whole prompt before it writes a reply. If your prompt starts with the same long block each time, such as instructions, a policy manual or tool descriptions, the model repeats the same reading work on every call. Prompt caching stores the result of that work so the next request can skip it.

More precisely, the provider saves the internal state computed for a prompt prefix. When a later request begins with exactly the same prefix, the provider loads that saved state instead of recomputing it. You are then billed less for those cached tokens, and the response usually starts sooner.

This is different from caching the answer. The model still generates a new reply to your new question. Only the processing of the unchanged start is reused. That keeps the answers fresh while cutting the repeated cost.

How does prompt caching work?

  1. You send a request whose prompt begins with a large, stable block, followed by the part that changes.
  2. The provider processes the prompt and, where caching applies, stores the state for the stable prefix.
  3. A later request arrives with an identical prefix, within the time the cache entry stays valid.
  4. The provider recognizes the match, reuses the stored state and processes only the new part.
  5. The response reports how many tokens were read from the cache, and you are billed at the cached rate for them.
  6. If the prefix differs even slightly, or the entry has expired, the provider processes the full prompt again.

The key word is prefix. A match is checked from the start of the prompt, token by token. Change one character near the top and everything after it is a miss. That is why prompt layout matters so much.

How should you structure a prompt for caching?

Order the prompt from the most stable content to the least. Everything that never changes goes first, content that changes occasionally goes in the middle, and the user's question and other per-request data go last.

Prompt layout for good cache hits
PositionContentHow often it changes
1 (top)System instructions and roleRarely, when you ship a new version
2Tool and function definitionsRarely
3Reference material, policies, examplesOccasionally
4Earlier conversation turnsGrows by appending, never edited
5 (end)The new user message and request-specific dataEvery request
  • Keep timestamps, user names, request IDs and random values out of the stable prefix; put them at the end.
  • Keep the order of tools and examples fixed. Reordering counts as a change.
  • Append to a conversation instead of rewriting earlier turns.
  • Avoid regenerating the system prompt from a template that inserts variable text near the top.

The most common reason caching does not work is an innocent-looking change near the top, such as a current date in the first line.

When does prompt caching help most?

It helps when requests share a large prefix and arrive close enough together for the cache to still be warm. The bigger and more stable the shared block, the more you save.

Use cases and fit
Use caseWhat is cachedFit
Support assistant with a long policy manualInstructions plus the manualStrong
Agent with many tool definitionsSystem prompt and tool schemasStrong
Chat with a long documentThe document, reused across questionsStrong
Long multi-turn conversationEarlier turnsGood, since each turn extends the same prefix
Coding assistant over a repositoryRepository contextGood when the context is stable
One-off requests with short promptsNothing worth cachingWeak
Prompts that differ from the first lineNo shared prefixNone

Agents benefit especially, because each step repeats the same instructions and tools while adding a little new information. See AI agents and tool calling.

Prompt caching vs. other ways to cut cost and latency

Several techniques reduce what you spend on a model. They solve different problems and often work together.

Prompt caching compared with related techniques
TechniqueWhat it doesTrade-off
Prompt cachingReuses processing of an identical prompt prefixNeeds a stable prefix; entries expire
Response cachingStores full answers to repeated identical questionsOnly works when questions repeat; answers can go stale
Retrieval (RAG)Sends only relevant passages instead of a whole libraryAdds retrieval design; quality depends on search
Model routingSends easy requests to cheaper modelsNeeds a way to judge difficulty
Shorter promptsRemoves unneeded textTakes careful testing so quality holds
Batch processingRuns non-urgent jobs togetherSlower results; only for offline work

A system can use all of these: retrieve a few passages, route to a suitable model, and cache the stable instructions. The LLM cost optimization playbook puts them in order of effort and payoff.

How do providers differ?

Major providers, including Anthropic, OpenAI and Google, offer some form of caching, but the details differ and change. Check current documentation before you design around any of them.

  • Activation: some caches apply automatically to long enough prompts, while others need you to mark which parts of the prompt to cache.
  • Minimum size: there is often a minimum prefix length below which caching does not apply.
  • Lifetime: entries last for a limited time and may be extended when reused. Some providers offer longer-lived options.
  • Pricing: cached reads are usually billed at a different rate than normal input, and some providers charge for writing to the cache.
  • Scope: caches are typically limited to your own account or organization, not shared with others.
  • Reporting: responses usually include usage fields that show how many tokens were cached.

Current model prices, including cached-token rates where published, are tracked on the LLM pricing page.

How do you know whether caching is working?

Do not assume. Read the usage data that each response returns, and log it. If cached-token counts stay at zero on requests that should match, something is changing the prefix.

  • Cache hit rate: how often requests reuse a cached prefix.
  • Cached tokens per request compared with total input tokens.
  • Time to first token for cached and uncached requests.
  • Cost per task before and after the change.
  • Misses over time: a sudden drop often follows a prompt change or a deploy.

When hits are lower than expected, work through a short checklist: is the prefix long enough, is anything variable near the top, is the order of tools or examples stable, and are requests arriving often enough before the entry expires? Add these metrics to your AI observability dashboards.

Common prompt caching mistakes

  • Putting the date, time or user name in the first line of the system prompt.
  • Editing earlier messages in a conversation, which breaks the shared prefix.
  • Expecting a cache to persist for hours when entries expire after a short period.
  • Caching a prefix that is too short to qualify.
  • Changing tool order or whitespace between deploys and wondering why hits dropped.
  • Ignoring write costs on providers that charge to populate the cache, then finding that rarely reused prefixes cost more.
  • Putting sensitive per-user data in a shared prefix, which mixes user data across requests.

Cached content is still sent to the provider, so caching does not change your data handling duties. Treat it under the same rules as any other prompt content.

What does a prompt caching setup look like in practice?

Take a support assistant for an online store. The system prompt holds tone rules and escalation rules, the tool list covers order lookup and refunds, and a returns policy of several pages sits in the prompt. Every customer message reuses all of that and adds one short question.

Without caching, every message pays to process the full policy again. With the policy, rules and tools placed first and the customer's message last, the first request fills the cache and the following ones within the cache lifetime read it. The customer's order details and name go at the end, so they never break the match.

The team then watches the usage fields. If a deploy changes the wording of one rule, the hit rate drops until the new prefix is warmed, which is expected. If it stays low, they look for a variable that crept into the top of the prompt.

How does caching work in agents and long conversations?

Agents loop: they call the model, run a tool, add the result to the history and call the model again. Each call repeats everything before it. Because the history only grows by appending, the earlier part forms a stable prefix, and caching covers most of every step after the first.

  • Append tool results and new turns at the end; never rewrite earlier turns.
  • Trim or summarize history only at planned points, knowing that each trim resets the cache for that conversation.
  • Keep tool definitions identical across steps and across users where possible.
  • Place per-session facts, such as the user's plan or region, after the shared instructions.

Caching works alongside agent memory and the context window: memory decides what goes into the prompt, and caching makes the repeated part cheaper.

What does prompt caching mean for cost and timeline?

Turning on caching is usually a small engineering task: reorder the prompt, mark the cacheable block where the provider needs it, and add monitoring. The larger work is the measurement that proves the savings and the discipline to keep prefixes stable.

  • Size and stability of the shared prefix.
  • Request volume and how close together requests arrive.
  • The provider and its pricing rules.
  • Number of prompts and agents to restructure.
  • Monitoring and evaluation needs after the change.

Check how model usage fits into a project budget on the LLM integration cost page, and run the free estimate to scope the whole feature.

How does StackLift approach prompt caching?

Cost control is designed in, not added later. On LLM integration and AI agent projects, prompts are laid out with stable content first, usage is tracked per task, and observability is part of the plan from sprint one.

A senior architect reviews every estimate, so model-usage assumptions, including caching and routing, are stated and checked before they become a commitment.

Questions people ask

Does prompt caching change the model's answers?

No. It reuses processing of the unchanged start of the prompt, but the model still generates a fresh reply to each request. Quality should be the same as without caching.

How long does a prompt cache last?

It varies by provider and settings, and is usually short, from minutes upward. Entries often stay alive when reused. Check your provider's current documentation.

Is prompt caching the same as response caching?

No. Prompt caching reuses work on a shared prompt prefix and still generates a new answer. Response caching stores whole answers and returns them for repeated identical questions.

Why is my cache hit rate low?

Usually something near the top of the prompt changes between requests, such as a timestamp, or the prefix is too short, or requests are too far apart. Look at the usage fields in responses to confirm.

Is prompt caching safe for private data?

Providers generally scope caches to your own account, but the content is still processed by them. Follow the same data rules as for any prompt, and keep per-user data out of shared prefixes.

Do all models support prompt caching?

Not all. Support depends on the provider and the model, and the rules change over time. Check the current documentation for the model you plan to use.

Can caching help with long documents?

Yes. If many questions go against the same long document, placing the document first lets later questions reuse its processing. A retrieval approach may be better when documents are very large or vary by question.

Should I always turn prompt caching on?

Turn it on where requests share a long, stable prefix and arrive often. For short or one-off prompts it adds little, and on providers that charge to write the cache, rarely reused prefixes can cost more than they save. Measure before and after, then keep it where the numbers show a gain.

Sources

Sources and further reading

The standards, rules, research and official documentation this page draws on.