GGUF

← All terms · Local models and inference

Also called GPT-Generated Unified Format

A highly optimized file format specifically designed for storing and loading quantized large language models rapidly on consumer hardware.

What it is

Created by the developer of llama.cpp, GGUF replaces older formats to provide a single file that contains both the model weights and all necessary metadata (like prompt templates). It is designed to be mapped directly into memory (mmap), allowing massive models to load and execute incredibly fast on CPUs and Apple Silicon.

When you would use it

You specifically look for GGUF files when you intend to run a model locally on consumer hardware, particularly if you are using tools based on llama.cpp.

Common operations

• Downloading a single .gguf file to run a model in LM Studio.
• Distributing a highly compressed version of a model on Hugging Face.

Related terms

• Quantization
• llama.cpp
• LM Studio

Where this is taught

• Running Your First Local Model
• Notebook experiments to model assets