Generative AI and LLM glossary

Deep Learning

Attention is the mechanism that lets a transformer weigh which other tokens matter when computing the next representation.

Inference

A context window is the maximum number of tokens a model can consider at once, including instructions, history, tools, and output.

Inference

The KV cache stores attention keys and values from already processed tokens so the model does not recompute the whole prefix each step.

Model Capabilities

Long context is an LLM's ability to accept a large token window containing prompts, documents, conversation history, tool results, and generated output.

Model Architecture

A mixture-of-experts model routes each token through a selected subset of specialized neural-network components instead of activating the full parameter set.

Product

Softmax converts a vector of logits into a probability distribution over classes or tokens.

Inference

Speculative decoding speeds generation by drafting tokens with a cheaper model and verifying them with the target model in parallel.

Inference

Temperature scales how randomly a model samples the next token: lower is more deterministic, higher explores more.

Inference

A token is the subword or symbol unit a model reads and writes; context limits, pricing, and latency are counted in tokens.

Natural Language Processing

The process of converting text into smaller pieces, called tokens.

Inference

Top-k sampling restricts next-token choice to the k highest-probability tokens before sampling.

Inference

Top-p, or nucleus sampling, draws the next token from the smallest set of tokens whose cumulative probability exceeds p.

Deep Learning

A transformer is a neural architecture that uses attention to relate tokens in a sequence; most frontier LLMs are transformers.