LLM Quantization for Local AI: Q4, Q5 and Q8 Explained

For everyday local AI chat, a supported Q4_K_M file is a sensible starting point when memory is tight. Try Q5_K_M of the same model if you have room. Choose Q8 only when its extra memory cost fits your setup and a comparison gives you a reason to keep it.

Those labels describe how model weights are stored. They cannot tell you whether a character will feel warmer or follow your instructions better. A larger download can preserve more of the original model’s behavior, but it still carries that model’s weaknesses.

The useful question is simple: which version of this model gives you satisfying replies while leaving enough memory for the conversation?

Quantization makes weights smaller, with a separate memory bill for chat

The short version: Quantization reduces the precision used to represent weights. Running the model also needs memory beyond those weights.

A language model contains learned numerical values called weights. Quantization stores them using a lower-precision representation, reducing their storage requirements. That makes models more practical to run on personal computers.

The llama.cpp quantization documentation explains the trade-off: quantization can reduce model size and speed up inference, with possible accuracy loss. Neither a speed improvement nor an invisible quality difference is guaranteed on every setup.

Once a chat starts, weights are only part of the memory bill. The runtime also needs working buffers and a KV cache, which holds attention information used when generating replies. Hugging Face’s local Llama memory discussion separates these costs explicitly.

Context length affects the cache budget. Some implementations preallocate it; others use different allocation strategies, including sliding windows. So it would be misleading to say that every new message always increases memory use. The KV cache documentation explains those differences.

For a hardware-focused explanation, see what fits in 8 GB of VRAM. Here, the goal is choosing between files of one model rather than repeating a hardware sizing table.

Q4, Q5 and Q8 describe precision rather than conversation scores

The short version: Higher nominal precision generally costs more space for the same model. The label alone cannot predict how much better a reply will be.

Compare quantizations only after fixing the underlying model and release. A Q8 file of one model and a Q4 file of another differ in more than precision.

These are qualitative comparisons, not benchmark results. The exact file contents matter.

The number in Q4 is not a rating out of ten. It also does not mean every stored value occupies exactly four bits. The llama.cpp quantization implementation uses higher-precision Q6_K storage for some tensors in Q4_K_M and Q5_K_M presets.

Q8 remains quantized. For example, GGML’s Q8_0 block definition stores a scale alongside integer values, rather than preserving the original floating-point weights. More precision gives the model more room to retain numerical detail; it does not make the file an untouched original.

The rest of the GGUF filename matters

The short version: Read the model identity and full quantization suffix before comparing downloads. The .gguf extension tells you the container format.

Consider this illustrative filename, not a recommendation for an actual download:

ExampleChat-8B-Instruct-Q4_K_M.gguf

• ExampleChat identifies the model family in this example. Check the repository for the exact release or revision.
• 8B describes the advertised parameter scale. It is not the file size or required RAM.
• Instruct indicates an instruction-tuned variant. Naming varies between publishers, so confirm it in the model card.
• Q4_K_M identifies the quantization preset. Keep the complete suffix when searching for another version.
• .gguf identifies the file format used by GGML-based executors.

The GGUF specification describes a container that carries model data and metadata. GGUF files can contain different tensor types; the extension alone does not specify precision.

Q4_0 and Q4_K_M are different schemes. Treating them as interchangeable because both start with Q4 hides a meaningful distinction. Nor does either suffix provide a universal quality score across models.

Check the publisher’s conversion notes when available. llama.cpp warns that requantizing already quantized weights can severely reduce quality compared with quantizing from 16-bit or 32-bit weights. A reassuring filename cannot establish how the file was made.

Higher precision cannot fix every awkward reply

The short version: A larger quant can reduce losses caused by quantization. It cannot repair weaknesses already present in the model or compensate for unsuitable chat settings.

Suppose a character keeps repeating your last sentence instead of moving the scene forward. Quantization could contribute, but that symptom alone does not diagnose it. The model’s training and the prompt deserve attention too.

An incorrect chat template can also make a capable model behave poorly. Check that the runtime uses the expected template before spending time downloading a larger file. Review the character instructions if they conflict with each other.

If Q4 and Q5 both ignore the same instruction, moving straight to Q8 is a weak next step. Try a clearer prompt or investigate whether the model suits the conversation you want.

This matters especially for roleplay. Natural dialogue involves choices that a precision label cannot summarize. A technically more faithful version can still produce a voice you dislike. The guide to small LLMs for roleplay on 8 GB RAM addresses model selection; comparing quants answers a narrower question after that choice.

Choose a supported file and leave room for conversation

The short version: Compatibility comes first. Then choose a quant that fits with your intended context settings, rather than squeezing weights into every available corner of memory.

A runtime must support the model architecture as well as the file’s quantization scheme. A familiar .gguf extension does not guarantee that an older application can load a newly released model.

For example, the Qwen llama.cpp guide documents architecture support requirements. Use your application’s current compatibility guidance; an old minimum version in a model guide is not a reason to install an old runtime today.

Before picking the largest file, check Local Waifu’s requirements and the model publisher’s instructions. Leave room for the context you intend to use. A model that loads for a short exchange can still be an uncomfortable choice at a longer configured context.

If you are unsure which repository asset to choose, follow the GGUF installation guide. Confirm the model variant before downloading a second quant. Otherwise, you could accidentally compare different releases and attribute their differences to Q4 versus Q5.

Compare two quants before keeping the larger download

The short version: Test two supported files of the same model under matching settings. Keep the larger one only if its benefit matters in your ordinary chat.

The following prompts are suggested checks, not a benchmark performed for this article. Start a fresh conversation for each file and use the same chat template. Keep the character prompt and sampling settings unchanged, with matching context limits.

Try prompts that reflect your actual use:

• Instruction following: “Reply in two sentences. Ask one question about my day, and do not offer advice yet.”
• Character voice: “You are a reserved friend sharing a quiet train ride. Respond to: I almost stayed home today. Avoid a speech about personal growth.”
• Context recall: Establish a small detail, such as a lost green notebook, earlier in the chat. Later ask which object the character is looking for, without repeating its name.
• Scene continuation: “We reached the closed cafe. Continue the scene with one small action, leaving my character’s response to me.”

Judge whether the response follows the request and feels usable. Record annoying repetition separately from factual or instruction errors. Also note whether waiting for replies interrupts the conversation.

One unusually good answer is weak evidence because generation varies. Repeat the checks and include a normal conversation, rather than choosing entirely from isolated prompts. If Q5 gives a consistent improvement you value and still runs comfortably, keep it. If the difference is hard to notice, keeping Q4 is a reasonable result. You do not owe the larger file a place on your machine.

Questions people ask

Back to the blog