Automatic speech recognition

Modern ASR systems are neural sequence models that map audio frames directly to text, replacing the separate acoustic, pronunciation and language models of older pipelines. They run in one of two modes. Batch recognition takes a complete audio file and returns a final transcript. Streaming recognition consumes audio as it arrives and emits partial hypotheses that get revised as more context appears — the only viable mode for a live conversation.

Streaming output comes in two flavours that are easy to confuse. Interim results are provisional and may change on the next frame; final results are committed and will not be revised. A voice agent that acts on interim text will occasionally act on a word the recogniser then retracts, so the usual design is to display interim text for responsiveness but only send finalised segments to the language model.

Telephony audio is the hard case. The PSTN carries 8 kHz narrowband audio compressed by codecs designed for intelligibility rather than fidelity, which strips the high-frequency energy that distinguishes consonants like /s/ and /f/. Accents, background noise, hold music and overlapping speech all degrade accuracy further, which is why a model benchmarked on clean read speech will underperform on real calls.

Accuracy is normally reported as word error rate, but the number that matters operationally is entity accuracy: whether the recogniser got the phone number, the date, the postcode and the surname right. A transcript can score well overall while being useless because it mangled the one field the call existed to capture.