Latency is the waiting time until an AI starts answering or finishes.
Example
A voice assistant with high latency answers only after three seconds, and the conversation feels clumsy.
For pros
Key metrics are time to first token and tokens per second. Latency depends on model size, prompt length, reasoning effort, load and network. Smaller models, streaming, prompt caching and shorter prompts reduce it.
