Quantization

← All terms · Local models and inference

Also called Model compression

A technique used to compress the massive size of an AI model by reducing the precision of its mathematical weights, making it faster and cheaper to run.

What it is

AI models are typically trained using high-precision 16-bit or 32-bit floating-point numbers. Quantization rounds these numbers down to lower precisions (like 8-bit or 4-bit). This significantly reduces the physical file size and the VRAM required to load the model into memory, with only a negligible loss in the model's actual intelligence.

When you would use it

You look for quantized models when your hardware lacks the massive memory required to run the original, uncompressed weights of a frontier open-weight model.

Common operations

• Fitting a 70-billion parameter model onto a single consumer GPU.
• Accelerating inference speeds on edge devices like smartphones.

Related terms

• GGUF
• Memory bandwidth (in inference context)
• llama.cpp

Where this is taught

• Running Your First Local Model