The KV cache is a temporary store inside a language model that keeps parts it has already computed, so each new word is generated faster.
Example
In a long answer, the model does not have to recompute the earlier text for every new word.
For pros
It stores the attention key and value vectors for all previous tokens. The cache grows with context length and takes a lot of GPU memory, so compression and sharing methods exist. Provider-side prompt caching builds on similar ideas.
