KV cache
Expanded definition
During autoregressive decoding, each new token needs keys and values for all previous tokens. Caching them is what makes chat continuations affordable. Cache size grows with batch, layers, heads, and context length, which is why long conversations get expensive on GPU memory. Prompt caching at the API layer is related but not identical: it reuses a billed prefix across requests, while the KV cache is an inference-engine structure inside a single generation.
Related terms
Explore adjacent ideas in the knowledge graph.
KV cache FAQ
What is KV cache?
The KV cache stores attention keys and values from already processed tokens so the model does not recompute the whole prefix each step.
How is KV cache used in AI systems?
During autoregressive decoding, each new token needs keys and values for all previous tokens. Caching them is what makes chat continuations affordable. Cache size grows with batch, layers, heads, and context length, which is why long conversations get expensive on GPU memory. Prompt caching at the API layer is related but not identical: it reuses a billed prefix across requests, while the KV cach...
Related
Comparisons, tools, and models that connect to this idea.