Skip to content
Orbit Docs

Memory bandwidth (in inference context)

Published Updated

Also called VRAM bandwidth

What is Memory bandwidth (in inference context)?

The speed at which data can be read from and written to a GPU's memory. It is the primary bottleneck for how fast an AI model can generate text.

During AI inference, generating a single token requires pushing the entire model's weights through the processor. Therefore, the speed of text generation is rarely limited by pure compute power (FLOPs); it is bottlenecked by how fast the hardware can move the model's massive data from VRAM into the compute cores.

How it relates to Orbit

Memory bandwidth decides how fast a local model answers when you route an Orbit tier to Ollama or LM Studio. See Complexity Tiers & Routing for details.

When you would use it

You consider memory bandwidth when designing deployment infrastructure, understanding that buying a GPU with faster memory will yield higher tokens-per-second than simply buying a GPU with more compute cores.

Common operations

  • Evaluating which hardware to purchase for a self-hosted AI cluster.
  • Using quantization to reduce memory size, thereby bypassing bandwidth limits.

Related terms

Where this is taught

No learning path uses this term yet. Browse the Learning Atlas for guided sequences through related ideas.