GGUF vs AWQ vs GPTQ vs FP8 vs NF4: LLM quantization choices
Choose LLM quantization from the runtime, hardware, and task you need to run. GGUF packages model tensors and metadata. AWQ and GPTQ reduce weight precision after training. FP8 uses eight-bit floating-point values. NF4 stores weights in four bits and is commonly used while training QLoRA adapters. These names describe different parts of the model-loading and execution process.
Choose the runtime and hardware first. Then choose a representation that the pinned runtime supports for the exact architecture. Benchmark task quality, memory, time to first token, throughput, and concurrency before accepting the smaller model files.
Check compatibility before comparing file size or bit width.
Decision table
| Goal | Starting point | Check before use |
|---|---|---|
| Local CPU, Metal, or portable llama.cpp inference | GGUF with a documented quantization | Architecture support, chat template, context behavior, quantization settings, and source revision. |
| GPU inference with calibrated weight quantization | AWQ or GPTQ | The serving engine, GPU, bit width, group size, kernels, and model architecture. |
| Supported data-center GPU serving | FP8 | Hardware capability, runtime implementation, calibration mode, and end-to-end quality. |
| Parameter-efficient training with frozen 4-bit weights | NF4 through bitsandbytes | Arithmetic precision, memory for training, and whether deployment uses an adapter or merged weights. |
| Simple Transformers prototype | bitsandbytes 8-bit or 4-bit | Backend support and whether the prototype path matches the production server. |
GGUF does not specify one quantization
A .gguf extension tells you how tensors and metadata are packaged, not the precision of every tensor. Names such as Q4_K_M identify llama.cpp quantization recipes, and those recipes can treat tensor groups differently. Record the original checkpoint, converter version, quantization command, and file hashes. Calibration uses representative inputs to choose quantization parameters. If conversion used those inputs, record the dataset too.
AWQ and GPTQ need a matching serving path
AWQ and GPTQ model files usually use Safetensors plus settings that describe the quantization method. A model can download successfully and still fall back to a slow kernel or fail on an unsupported architecture. Check the current support matrix for Transformers, vLLM, or the actual server version before selecting the model files.
FP8 and NF4 solve different problems
FP8 is attractive when the accelerator and serving stack implement it efficiently. NF4 is primarily used to store frozen model weights while training small adapters through QLoRA-style workflows. In bitsandbytes, the compute dtype sets the precision used for arithmetic; it can differ from the stored four-bit weights. Optional nested quantization also compresses the constants used to represent those weights. Neither label guarantees the same quality, speed, or memory use across models and hardware.
Evaluation checklist
Four-bit storage does not mean the whole process uses four bits per parameter. Quantization metadata, higher-precision tensors, temporary buffers, and the attention cache require more memory. Test the prompt length and concurrent requests you plan to serve.
- compare the same source checkpoint and prompt set
- include long-context and concurrent requests
- measure schema and tool-call correctness alongside perplexity
- record peak device and host memory
- verify time to first token and output throughput
- inspect rare or high-cost quality slices
- keep license, source revision, converter, recipe, and hash with the model files
Deeper reading
- LLM engineering guide: quantization places these choices beside memory, serving kernels, and GPU selection.
- Model Quantization Guide covers algorithms, calibration, kernels, and deployment trade-offs.
- Open-Weight LLM Variants separates checkpoint role, architecture, container, quantization, runtime, and hardware.
- Local LLMs on macOS applies these choices to Apple Silicon.