Speech-to-speech

In the cascaded architecture, speech recognition produces text, a language model reasons over that text, and text-to-speech renders the reply. Each stage adds its own delay and each boundary loses information. A speech-to-speech model instead tokenises audio directly, reasons in a space that retains acoustic detail, and generates audio tokens as output, which lets it respond to how something was said rather than only to what was said.

The practical advantages are latency and expressiveness. Because there is no wait for a finalised transcript and no separate synthesis pass, the response gap shortens. Because the model perceives prosody, it can react to hesitation or frustration, and it can produce laughter, hedging and interruption behaviour that a text pipeline has no representation for.

The disadvantages are the reason cascaded pipelines remain common in regulated work. Text is where guardrails, redaction, keyword detection and retrieval integration live, and without it those controls must be rebuilt. Debugging is harder when there is no transcript showing what the model believed it heard, and swapping one component for a better one is impossible when the components are a single model.