LLM quantization: what Q3 and Q4 actually mean
See what LLM quantization changes inside model weights, why Q3 is not three decimal places, and how lower-bit formats trade memory traffic for approximation.
In GGUF names such as Q3_K_M, Q3 and Q4 describe low-bit weight formats, not decimal places. Compact codes and shared metadata shrink the file by approximating weights; the label alone predicts neither whole-model size nor speed.
Picture a ruler with eight numbered marks and one shared note saying how far apart they are. For each weight, save the number of its nearest mark instead of the full value. The numbers take less room, but values between marks shift.
A Q3 model does not keep three digits after the decimal point. In common GGUF names, the number describes a low-bit weight format. Each compact code reconstructs an approximate weight using information shared with other weights in its block. The model file can be smaller, but the reconstructed values differ from the originals.
Bits, not decimal places#
A trained model contains large tensors full of numbers. A weight might be -0.9134, but the important question for storage is not how many decimal digits are printed. It is how many binary states the representation can encode. FP16 and BF16 each spend 16 bits on a value. A 3-bit code has only 2³ = 8 possible bit patterns. A 4-bit code has 16.
That is why “Q3 keeps three decimal places” is the wrong mental model. Decimal rounding would turn -0.9134 into something like -0.913 while still storing a floating-point number. Quantization instead asks a different question: which member of a small set should stand in for this weight, and what shared information lets us interpret that choice?
What a quantized block stores#
The simplest useful picture is a block of weights that shares one scale. Suppose, only for illustration, that a toy 3-bit codebook indexes eight levels from -4 through 3 and uses a scale of 0.25. Each original weight is mapped to the nearest level; the 3-bit code stores that level’s index, and inference reconstructs an approximation by multiplying the level by the shared scale.
| Original weight | 3-bit code | Level q | Reconstructed q × 0.25 |
|---|---|---|---|
-0.91 | 000 | -4 | -1.00 |
-0.56 | 010 | -2 | -0.50 |
-0.12 | 100 | 0 | 0.00 |
0.73 | 111 | 3 | 0.75 |
This is a teaching example, not the layout of a GGUF Q3_K block. Real schemes choose scales, codebooks, block sizes and special handling differently. The part worth keeping is the shape of the mechanism: many compact codes share a smaller amount of metadata. The code says which approximate value to use; the metadata says how to interpret it.
That is also why quantization can preserve much more useful information than chopping bytes off every float independently. The representation is designed around groups of weights and the distribution of values inside them.
Why Q3 is not exactly three bits per weight#
The numeral in a name such as Q3_K_M is useful, but it is not an exact file-size formula. In the GGML implementation checked here, one underlying Q3_K block holds 256 weights. Their three-bit codes occupy 96 bytes. Another 12 bytes store quantized scales, and a two-byte shared scale brings the block to 110 bytes: 3.4375 effective bits per weight. That is a property of this block layout, not the effective bit rate of every Q3_K_M model file.
A whole model can move farther from the headline number. llama.cpp calls its file-level choices “mostly” quantization types and exposes a –pure option specifically to disable K-quant mixtures. In other words, a recipe named Q3_K_M does not promise that every tensor in the checkpoint uses the same three-bit representation. The recipe can keep selected tensors at a different precision, so the file-level name is not a uniform encoding guarantee.
So when two downloads both have “Q3” in the name, compare their actual file size and format, not only the first numeral. The suffix is part of the recipe, and effective bits per weight are a measured property of that recipe rather than a decimal expansion of its name.
Why fewer bits can run faster#
During token generation, each new token needs operations that access model weights. If moving those weights through memory is the limiting step, a lower-bit representation can reduce the bytes moved for the same layer. That is one reason low-bit inference can be faster as well as smaller.
But the speed-up is conditional. Packed values have to be unpacked or dequantized into the form a kernel uses. Some processors have excellent kernels for one format and poor support for another. A model that fits entirely in fast memory at Q4 can behave very differently from the same model partly offloaded at Q5. And prompt processing, token generation and large batches stress hardware differently. The distinction between latency and throughput still applies.
That is why “Q3 is faster than Q4” is not a rule. It is a hypothesis to measure on one model, one runtime, one hardware configuration and one workload.
MoE active parameters are not model size#
Mixture-of-experts models add a second number that is easy to confuse with quantization. A label such as 30B-A3B can describe roughly 30 billion parameters in the checkpoint while routing each token through roughly 3 billion active parameters. The smaller number is primarily about computation per token. It is not the size of the checkpoint.
Qwen’s Qwen3-Coder-30B-A3B-Instruct is a concrete example: its model card lists 30.5 billion total parameters and 3.3 billion activated. Quantizing it changes how those stored weights are represented. A GGUF conversion still has to contain the experts that may be selected for another token; “3.3B activated” does not turn the checkpoint into a 3.3B model.
Resident memory is a separate question again. A runtime may memory-map a model, offload some layers to a GPU, or keep only a subset resident at a particular moment. That is operating-system and runtime behaviour, not a change in the model’s parameter count. Virtual memory explains why a mapped file and physical resident memory are not the same number.
What still needs memory#
Weight size is one large line in a local-inference memory budget, not the whole budget. The runtime also needs space for the KV cache that holds attention state for the current context, temporary activations and work buffers, model metadata, and its own allocations. Some runtimes can quantize the KV cache separately; doing that is a different decision from quantizing the model weights.
As a simplified example, a 20 GB model file does not mean that 20 GB of available memory is enough to run it comfortably. Context length matters. Cache precision matters. Offloading changes where the bytes live. Leave headroom for the rest of the workload rather than treating file size as the whole memory requirement.
Choose a quant by what fits and what survives#
For local use, pick the model first and the quantization second. Then choose the least aggressive compression that fits with enough headroom for the context and runtime you actually need. If Q4 fits comfortably, dropping to Q3 only because the number is smaller buys capacity you may not need. If Q4 forces heavy offloading or paging while Q3 fits, the smaller representation may be the practical choice even if it loses some model quality.
Do not compare formats with a single generated paragraph. Keep the model and runtime fixed, measure tokens per second separately for prompt processing and generation, and run a small set of tasks you care about. For a coding model that might be repository navigation, a multi-file edit, a bug diagnosis and a test repair. The useful question is not “Which quant is best?” but “At the memory budget I have, which representation preserves enough of this model’s behaviour for my workload?”
IF YOU REMEMBER ONE THING
Q3 and Q4 are compact representation families, not decimal precision settings. Read the format as a storage recipe, then measure the actual file, memory headroom, speed and task quality on the machine that will run it.
Questions people also ask
6 QUESTIONSDoes Q4 mean every model parameter uses exactly four bits?
No. Q4 names a quantization family or recipe. The low-bit weight codes may be four bits, but block scales, zero points or other metadata add overhead, and model-level recipes may keep selected tensors at a different precision. Effective bits per weight and final file size can therefore be higher than the numeral alone suggests.
Is Q3 just rounding model weights to three decimal places?
No. Decimal places describe how a number is printed in base ten. Quantization changes how values are represented for storage and computation. A 3-bit code can select only eight code values before any shared scale or other block metadata is applied.
Does a 30B-A3B mixture-of-experts model need memory for only 3B parameters?
No. The active-parameter count describes how much of the expert network participates in one token's computation. The checkpoint still stores the model's total weights. Runtimes can map or offload those weights differently, so resident memory varies, but the inactive experts do not disappear from the model file.
Is a lower-bit quantization always faster?
No. Moving fewer weight bytes can help when memory traffic is the bottleneck, but kernels, hardware support, dequantization work, context length and offloading all matter. Compare quantizations on the same model, runtime and workload rather than treating the Q number as a speed ranking.
Can I quantize an already quantized GGUF again?
Tools can allow it, but that does not restore information lost by the first quantization. llama.cpp warns that requantizing can reduce quality substantially compared with quantizing from 16- or 32-bit source weights. Start from the highest-precision source you have when quality matters.
Are activations quantized too?
They can be, but a Q4 model name usually describes the stored model-weight recipe, not every numeric value used at runtime. Weight precision, activation or compute precision, and KV-cache precision are separate choices in many inference stacks.