vLLM

← All terms · Local models and inference

Also called vLLM serving engine

A high-throughput, open-source library for serving LLMs in production, famous for pioneering PagedAttention to efficiently manage memory during text generation.

What it is

vLLM is an inference engine designed to maximize the speed and concurrency of serving open-weight models. It utilizes continuous batching and optimized attention kernels to process thousands of requests simultaneously. It is the industry standard for deploying self-hosted models at scale.

When you would use it

You deploy vLLM when transitioning an AI application from third-party APIs to self-hosted infrastructure, needing to squeeze maximum throughput and concurrency out of your available GPU hardware.

Common operations

• Serving high-volume enterprise API requests using self-hosted DeepSeek models.
• Optimizing GPU cluster utilization for AI application backends.

Related terms

• Self-hosted inference
• Quantization
• Memory bandwidth (in inference context)

Where this is taught

• Running Your First Local Model

Going deeper

• https://docs.vllm.ai/en/latest/