Transformer
Expanded definition
Transformers replace recurrence with self-attention and feed-forward layers, enabling parallel training and long-range token interactions. Decoder-only models generate text autoregressively; encoder-decoder models are common in translation and some speech systems. Serving cost is driven by attention, KV cache, and context length, not only by parameter count. Mixture-of-experts variants keep the same family while activating a subset of parameters per token.
Related terms
Explore adjacent ideas in the knowledge graph.
Transformer FAQ
What is Transformer?
A transformer is a neural architecture that uses attention to relate tokens in a sequence; most frontier LLMs are transformers.
How is Transformer used in AI systems?
Transformers replace recurrence with self-attention and feed-forward layers, enabling parallel training and long-range token interactions. Decoder-only models generate text autoregressively; encoder-decoder models are common in translation and some speech systems. Serving cost is driven by attention, KV cache, and context length, not only by parameter count.
Related
Comparisons, tools, and models that connect to this idea.