Skip to content
Orbit Docs

Quantization

Published Updated

Also called Model compression

What is Quantization?

A technique used to compress the massive size of an AI model by reducing the precision of its mathematical weights, making it faster and cheaper to run.

AI models are typically trained using high-precision 16-bit or 32-bit floating-point numbers. Quantization rounds these numbers down to lower precisions (like 8-bit or 4-bit). This significantly reduces the physical file size and the VRAM required to load the model into memory, with only a negligible loss in the model's actual intelligence.

How it relates to Orbit

Quantized models are what make local tiers practical: Orbit can route a tier to Ollama or LM Studio running one on your own hardware. See Complexity Tiers & Routing for details.

When you would use it

You look for quantized models when your hardware lacks the massive memory required to run the original, uncompressed weights of a frontier open-weight model.

Common operations

  • Fitting a 70-billion parameter model onto a single consumer GPU.
  • Accelerating inference speeds on edge devices like smartphones.

Related terms

Where this is taught