Quantization
Expanded definition
Quantization maps high-precision parameters to fewer bits (int8, int4, FP8, and similar). It can enable local serving of larger models and shrink KV cache, but it can also drop quality on hard tasks. Formats such as GGUF are common for local runtimes. Always eval the quantized build on your task, not only on a generic leaderboard. Training-time methods like QLoRA quantize during adapter training; inference-time quantization is a separate serving choice.
Related terms
Explore adjacent ideas in the knowledge graph.
Quantization FAQ
What is Quantization?
Quantization stores weights or activations in lower-precision formats such as 8-bit or 4-bit to reduce memory and often cost.
How is Quantization used in AI systems?
Quantization maps high-precision parameters to fewer bits (int8, int4, FP8, and similar). It can enable local serving of larger models and shrink KV cache, but it can also drop quality on hard tasks. Formats such as GGUF are common for local runtimes. Always eval the quantized build on your task, not only on a generic leaderboard. Training-time methods like QLoRA quantize during adapter training;...
Related
Comparisons, tools, and models that connect to this idea.