Attention
Expanded definition
Scaled dot-product attention compares queries, keys, and values so each token can mix information from others in the window. Multi-head attention runs several of these mixes in parallel. Inference stores keys and values in the KV cache so later tokens do not recompute the whole prefix. Long context is expensive because attention scales with sequence length unless the implementation uses approximations or sparsity.
Related terms
Explore adjacent ideas in the knowledge graph.
Attention FAQ
What is Attention?
Attention is the mechanism that lets a transformer weigh which other tokens matter when computing the next representation.
How is Attention used in AI systems?
Scaled dot-product attention compares queries, keys, and values so each token can mix information from others in the window. Multi-head attention runs several of these mixes in parallel. Inference stores keys and values in the KV cache so later tokens do not recompute the whole prefix.
Related
Comparisons, tools, and models that connect to this idea.