Latency budget

The chain for a cascaded agent has more contributors than it first appears. Audio must reach the platform, which on a phone call includes carrier transit and jitter buffering. Endpointing must decide the caller has stopped, which is a deliberate wait rather than processing. Recognition must finalise. The model must produce enough output to begin speaking. Synthesis must render the first chunk. That audio must travel back.

Two of those are not compute at all. Endpointing delay is a policy choice — how long to wait before concluding the turn ended — and network transit is largely fixed by geography and carrier path. Shaving milliseconds from inference while running a conservative silence threshold and a distant media region is optimising the wrong term.

The technique that buys the most is overlap. Streaming recognition means transcription finishes almost when speech does. Streaming synthesis means playback starts on the first clause rather than the last. Speculative generation on interim transcripts can begin inference before the turn formally ends. Each converts a sequential segment into a concurrent one, which is worth more than making any single stage faster.

Perceived latency is also partly a design problem rather than an engineering one. A short acknowledgement before a slow lookup, or a filler phrase while a function runs, changes what the caller experiences without changing the measured number — which is why platforms that support tool calls usually let you configure what the agent says while it waits.