In-app purchases are temporarily unavailable. Local Waifu is currently available exclusively on itch.io. Buy on itch.io

local waifu
Bring her home

Pick your platform

Try her free for 7 days. No card. Keep her? $20 once.

New: Local Waifu now runs on Windows 10 and 11. The installer brings everything she needs, nothing else to set up. Windows may show a SmartScreen prompt the first time: click More info, then Run anyway.

blog

KV Cache Quantization: What Q8 and Q4 Do to Your Context

7 min read
In short

The KV cache costs about 128 KiB per token on an 8B model at f16, so 8K of context is roughly 1 GiB. Quantizing it to Q8_0 cuts that to about half and Q4_0 to about a quarter, which buys context length on the same card. Set it with -ctk and -ctv in llama.cpp, the K/V cache quantization setting in LM Studio, or OLLAMA_KV_CACHE_TYPE in Ollama. Quantizing the value cache needs flash attention, and quality loss shows up in long-context recall first.

Written by Łukasz, the developer of Local Waifu, a private AI companion that runs on your own Mac or Windows PC.

There are two memory bills in a local LLM, and almost every guide only talks about the first one. The weights are the GGUF file you downloaded, and everything written about quantization is usually about them. The second bill is the KV cache: the keys and values of everything the model has already read, kept around so each new word does not force a full recompute.

That second bill is the one that grows while you use the app. It is also the one you can shrink without redownloading anything.

The KV cache is a per-token bill, and it is bigger than people expect

The short version: At f16 an 8B model spends about 128 KiB of cache per token, so an 8K conversation is roughly 1 GiB before any weights are counted.

The arithmetic is not a mystery once you look at a model config. Take Llama 3.1 8B: 32 hidden layers, 8 key/value heads, hidden size 4096 with 32 attention heads, which gives a head dimension of 128. That is all in the published config file.

Cache bytes per token = 2 (for K and V) x layers x key/value heads x head dimension x bytes per element.

At two bytes per element that is 2 x 32 x 8 x 128 x 2 = 131,072 bytes, or 128 KiB for every token of context. Put 8,192 tokens in and the cache alone is 1 GiB. Go to 32K and it is 4 GiB, sitting next to the weights the whole time.

Cache typeBytes per elementPer token, Llama 3.1 8B8K context32K context
f16 (default)2~128 KiB~1 GiB~4 GiB
q8_0~1~64 KiB~0.5 GiB~2 GiB
q4_0~0.5~32 KiB~0.25 GiB~1 GiB

The quantized numbers are a little above exactly half and exactly a quarter, because each block of values carries a scale. And the whole table leans on grouped-query attention: a model with 32 key/value heads instead of 8 multiplies these figures by four. This is the same budget that decides whether an 8 GB card can hold a 32K conversation, which is the arithmetic in what fits in 8 GB of VRAM.

Two different quantization settings get confused constantly

The short version: The GGUF file is the weights. The cache type is the conversation. Changing one does nothing to the other.

Weight quantization, which is what Q4, Q5 and Q8 mean in a file name, decides how much disk and memory the model itself takes. It is fixed when you pick the file. Cache quantization is a runtime choice: same weights, same messages, different precision for the attention state.

That separation matters because the two bills behave differently. The weights are constant, so a model that fits today fits tomorrow. The cache grows with the conversation, so a setup that feels fine in a short chat can die at message forty. When someone says their companion app slowed to a crawl after an hour, the weights did not change size. The cache did.

Where the setting actually lives

The short version: llama.cpp has explicit flags, LM Studio exposes the same control in its load config, and Ollama uses an environment variable that defaults to f16.

In llama.cpp’s server the flags are documented in the server README:

  • -ctk, --cache-type-k TYPE and -ctv, --cache-type-v TYPE
  • allowed values: f32, f16, bf16, f8_e4m3, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1
  • default for both: f16
  • environment equivalents: LLAMA_ARG_CACHE_TYPE_K and LLAMA_ARG_CACHE_TYPE_V

LM Studio exposes the same idea as llamaKCacheQuantizationType and llamaVCacheQuantizationType in its model load configuration, with the same warning the rest of us learned the hard way: value cache quantization requires flash attention to be enabled. The load config also has useFp16ForKVCache as a simpler switch between f16 and full precision.

Ollama’s version is an environment variable. Its FAQ lists OLLAMA_KV_CACHE_TYPE as the quantization type for the K/V cache with a default of f16.

One detail worth knowing if you use Local Waifu: the app’s bundled local engine already sets OLLAMA_KV_CACHE_TYPE to q8_0 for the sidecar process it manages. So the cache is halved there by default, without a settings screen. The same model loaded into a bare Ollama install would use f16 and need about twice the cache memory. Same weights, different bill.

What this buys you, in the order worth trying

The short version: Quantize K to Q8_0 first, keep V at f16 if you can, and only go further when you need the room.

  1. Run the model as it is and note the context length you actually use. Not the maximum, the real one.
  2. Set the K cache to q8_0 and the V cache to q8_0 with flash attention on. This is where most people stop, because the saving is the biggest single step and the risk is smallest.
  3. If you still need memory, try q4_0 for K only before touching V. The value cache is the half that LM Studio’s documentation flags as needing experimentation, and the half that hurts when it goes wrong.
  4. Raise the context length only after the cache fits. Doing it the other way around is how a working setup turns into an out-of-memory crash at the worst moment.

There is also a flag that surprises people: KV cache offload. llama.cpp keeps the cache on the GPU by default and --no-kv-offload moves it off. That is occasionally the only way to fit a model, and it costs speed, because the cache gets read on every token.

The honest part about quality

The short version: Nobody can tell you Q4 is safe for your model. Your own long-context test can.

There is no published, model-independent quality number for a quantized KV cache, and anyone quoting one is quoting a benchmark for a specific model at a specific context length. LM Studio’s reference puts it plainly: the effect varies between models, and some are more tolerant than others.

What you can do in ten minutes is more useful than what a benchmark says. Start a conversation, put a distinctive detail in the second message (a made-up name, a colour, a small number), then talk for a few thousand tokens and ask about that detail. Do it twice, once at f16 and once at your candidate setting, at the same context length. If both replies carry the detail, the setting is fine for you on this model.

Where cache quantization shows up first is exactly that kind of recall, plus small wording drift. It rarely breaks a reply in the first minute, which is why testing with three quick messages proves nothing. This is the same lesson as a bigger context window not being memory: more room for text does not mean the model is using the text well, and squeezing the room down makes that visible.

A worked case: same card, twice the conversation

The short version: On 8 GB of VRAM, the cache is often what decides between a 4K and an 8K chat.

Say a quantized 8B model takes 5 GB and you have 8 GB, with the desktop compositor eating a few hundred megabytes. That leaves roughly 2.5 GB for the cache. At f16 and 128 KiB per token you get about 20K tokens of theoretical room, and less in practice once the driver and the app get their share. Switch K and V to q8_0 and the same headroom covers about 40K. Nothing else changed: same file, same speed class, same girl.

For a companion app the difference is concrete. 20K tokens is a long evening. 40K tokens is a long evening plus the recap of what happened last week that she still has in front of her. If you are tuning the rest of the prompt at the same time, one variable at a time is the only way to know which change did what.

What I would set today

The short version: Q8_0 on both halves, flash attention on, context sized to what you actually use.

If a local companion is on your machine, the cache type is one of the cheapest wins available: no download, no new model, no rebuild. If you run the app I work on, it is already at q8_0. If you run llama.cpp, LM Studio or your own Ollama, it takes one flag and one restart, and the requirements page already asks for enough RAM that the cache has somewhere to live.

The one thing I would not do is chase Q4 everywhere on the first day. Start at Q8_0, test recall honestly, and go lower only when a specific machine needs the room. The cache is there to remember things. It is worth checking that it still does.

Questions people ask

What is the KV cache, in one paragraph?

While a model reads your prompt it stores keys and values for every token it has already seen, so it does not recompute them for each new word. That store is the KV cache. It grows with the conversation, lives in the same memory as the model weights, and on a long chat it can cost as much as a quarter of the weights.

Is KV cache quantization the same as model quantization?

No. Model quantization decides how precisely the weights are stored, which is the GGUF file you download. KV cache quantization decides how precisely the keys and values of your conversation are stored. They are separate settings with separate memory bills, so changing one does not change the other.

How much memory does an 8K context actually need?

On Llama 3.1 8B with 32 layers and 8 key/value heads, f16 costs 131,072 bytes per token, so 8,192 tokens is about 1 GiB. Q8_0 lands near half that and Q4_0 near a quarter, plus a small per-block overhead. A model with more key/value heads multiplies these numbers.

Does quantizing the V cache need flash attention?

In LM Studio's documentation, value cache quantization explicitly requires flash attention to be enabled. In llama.cpp the supported way is the same: turn flash attention on before you expect a quantized V cache to behave. If you only quantize K, that dependency does not apply.

Will Q4 cache ruin my roleplay?

Not necessarily, and not a fixed amount. LM Studio's own documentation says models respond differently to cache quantization. The first thing to suffer is usually recall of something said a long way back, so test with a detail from the start of a long conversation instead of judging by the first few replies.

Does the app I use already quantize the cache?

Some do. Local Waifu's bundled local engine sets OLLAMA_KV_CACHE_TYPE to q8_0 for its own sidecar process, so the cache is already halved there without you touching a setting. Ollama's default outside the app is f16, which is why the same model can use different memory in two clients.

Try her free for 7 days.

No card. Keep her for $20 once, or walk away. Her soul file is yours either way.

Bring her home, try free

Back to the blog