RexaThe short answer
TTFT is the model-side number that matters for voice, because a streaming pipeline can begin synthesising as soon as the first words arrive. Total generation time barely affects the caller’s experience if the agent starts speaking promptly; a fast model with slow first-token latency still produces an awkward gap before every reply.
In detail
TTFT is dominated by prefill — the forward pass over the entire input before any output is produced — so it scales with prompt length. A long system prompt, a full conversation history and several tool schemas all lengthen prefill on every single turn, not just the first. This is the usual reason a voice agent gets slower as the conversation goes on.
The mitigations are mostly about input size and reuse. Prompt caching lets a provider reuse the computed state for an unchanged prefix, which is effective precisely because the system prompt is identical every turn. Trimming or summarising older history bounds growth. Loading only the tool definitions relevant to the current stage of the conversation avoids paying for schemas the agent will not use.
It is worth measuring TTFT separately from total generation latency, because they respond to different fixes and the aggregate hides the one that matters. A model that emits its first token quickly and then generates steadily is better for voice than one with lower total latency but a long initial pause, even though the second looks faster in a benchmark that measures completion time.
Related terms
Latency budget
A latency budget is an allocation of the maximum acceptable end-to-end response delay across each component that contributes to it.
System prompt
A system prompt is the standing instruction set given to a language model that defines its role, constraints and behaviour for every turn of a conversation.
Function calling
Function calling is the mechanism by which a language model requests execution of a defined external operation and incorporates its result into the conversation.
Retrieval-augmented generation
Retrieval-augmented generation is an architecture in which relevant documents are fetched from a knowledge store at query time and supplied to a language model as context for its answer.