Why LLM Inference Caches K and V but Never Q
Most engineers know that LLM inference uses KV cache. Far fewer can precisely explain why K and V are worth caching but Q is discarded after every decode step. The answer comes from the causal mask — and it has direct implications for how you size and optimize serving infrastructure.