Dictionary term
Latency
Also called time to first token · TTFT · response time
Latency is the time a model takes to process a prompt and return output. It is usually measured as time to first token (TTFT), the delay before output starts, plus output speed in tokens per second.
In plain English
Latency is how long you wait for the AI to reply, both before it starts and while it finishes.
Analogy
Like the wait at a restaurant: how long until the first dish arrives, and how quickly the rest of the meal follows.
Why it matters
In live chat, voice and SMS bots, slow replies lose people, so you pick faster models, trim prompts and output, use prompt caching, and stream the response. For overnight batch jobs latency barely matters and cost wins.
In practice
A website chat widget streams the reply so the first words appear within about a second, while a nightly lead-scoring job uses a slower, cheaper batch mode.
Related terms
Sources
Browse all terms →Last verified